Originally posted on Github, I wrote I would place the list somewhere else if it gets too long, well, I can already see it getting too long so I placed a copy of it here on my blog.
I’m compiling a list of problem domains where Copilot suggests the right function and works perfectly, and where Copilot suggests the right function but doesn’t work out of the box, as well as those domains where Copilot completely fails to suggest a function that does the right thing.
This list is specific to Python, but I might do lists for other languages if there is interest.
- Basic string manipulation (reversing, justifying, etc.)
- Suggests the right functions
- Flawless
- Elementary math
- Suggests the right functions
- Flawless
- Matrix arithmetic using
numpy- Suggests the right functions
- A few errors
For example, typing def matrix_norm(x) will autosuggest something like
def matrix_norm(x):
return x.linalg.norm(1)which is not correct because the x should be the first argument, the number second, and the np sould be in the x‘s place.
- Zalgo text
- Suggests the right functions
- Works on paper but has a few errors
The problem here is that the function calls is not in any PyPI package so it becomes harder to locate their own definitions. Other times, the functions are cut off in places.
- Bitcoin private key generation
- Sometimes suggests the wrong functions
- Basically unusable
Specifically, private key hex generation mostly works, but WIF generation has about 1/4 chance of suggesting something from the bit library and the code does not run although it works on paper.
These are the things I have tested so far.
I will keep this list updated as I continue testing, but if it gets too long I’ll move it somewhere else.
