Posted: 02/03/2024
My Thoughts On Using GitHub CoPilot
Recently I enabled GitHub Co-Pilot in Visual Studio.
This was following a talk about AI with Silicon South.
I have heard loads about it, but had not had the urge to use it.
To give this post context, I work on a large existing code base (About 1 Million Lines Of Code) with a unique and bespoke platform, so I didnt feel it would help too much, hence the lack of fervour to try it.
Today I changed that, so here are my random thoughts when using it:
Day 1
Moving code around
Working on fixing some broken tests it offered nothing. Which was no surprise as I was moving around code.
Comments
The first time it fired, it wrote a comment for me that was spot on!
I typed:
//We have received a new snapsh
it said:
//We have received a new snapshot, so we need to update our internal list of users
It's comments are very generic, but they are a good place to start.
Existing Code versus New Code
It likes to create new content, but does not like to add to existing things.
e.g. Will create a unit test for you, but wont fill in a blank method
So this encourages a different way of writing than I am used to. I would normally plan by building out skeleton code and then gradually fill it in.
CoPilot wants to build it immediately!!
To make it start making suggestions within existing code, you need to "prompt" it as if it was chatGPT.
The best way of doing that is to start writing comments or pseudo code, and it then gets a much better idea.
Like a Newbie
It often gives incorrect code, which can be very confusing, as at first glance it looks correct.
It is like working with a person who understands how to code and the classes of the system, but not what it is supposed to do, or how it works. There is no
Domain Knowledge.
Day 1 Conclusion
After using it for a day, it saved me a small amount of work.
I look forward to using it on newer things to see its power, but right now, it wont be cost effective for legacy code maintenance.
Day 2
I spent the day adding new tests to existing complex code, that has no current tests.
I found that it is good at guessing, but it gets it wrong a LOT.
This actually slowed my flow down, as I kept having to look at what it was suggesting, and confirm if it was correct, and actively discarding it before writing what I needed.
Again, it felt like someone was really trying to help, but butting in with not thought out suggestions.
Just to be clear - if I was pair/mob programming with a real person, this is the kind of thing I would want to see, as it gives me an opportunity to help them learn.
BUT Copilot is a tool, and it is here to help ME, and not for ME to teach IT. So it comes off as annoying!!
The best thing it did, was save me typing when generating test data constructs though.
Final Conclusion
My biggest problem is it suggests things that are clearly (well to me) wrong.
This was mostly when suggesting parameters that are completely different types.
This is very annoying as it breaks concentration for me, because I am having to solve a problem that would not be introduced normally.
I liken this to a small child asking you questions while you are typing (Why is the sky blue? What is your favourite Carrot? What’s faster? Fire or dust?), sometimes you just end up typing what they are saying and have to redo it all again.
I am still open minded about Co-Pilot, ChatGPT and all the other AI tooling being developed, but for my current company project, it is not worth the cost to me, both financially and with my time.
Thanks for reading this post.
If you want to reach out, catch me on Twitter!
I am always open to mentoring people, so get in touch.