- GitHub says its AI-generated code is more readable, reliable and maintainable
- The test focused on a highly repetitive task – AI’s ultimate role
- Only 243 developers took part in the study
Software developer Dan Cîmpianu has criticized the quality of AI-generated code in a blog post targeted at GitHub’s claims about its Copilot AI tool.
More specifically, the Romanian developer slated the statistical accuracy and experimental design used by GitHub in a recent study, where it claimed that its Copilot-assisted code was “significantly more functional, readable, reliable, maintainable, and concise.”
However, the study focused on writing API endpoints for a web server, or Create, Read, Update and Delete actions (CRUDs), which Cîmpianu described as “one of the most boring, repetitive, uninspired, and cognitively unchallenged aspects of development.”
Is GitHub’s AI code actually that good?
The study compared GitHub’s OpenAI-backed AI-generated code with that of over 200 experienced developers, and found the AI code to perform better across multiple metrics.
However, Cîmpianu has criticized GitHub for using percentages to denote differences without actually providing the baseline metrics for comparison, which could artificially make the percentage values look higher than they are.
GitHub’s study also defines errors as “inconsistent naming, unclear identifiers, excessive line length, excessive whitespace, missing documentation, repeated code, excessive branching or loop depth, insufficient separation of functionality, and variable complexity,” meaning that bugs produced by its code were not included within the statistics of
Another criticism of the study is that, despite being a “home to 1 billion developers,” the study only uses a sample size of 243 developers.
Cîmpianu concluded: “This does not seem to be even attempting to [be] aimed towards developers, but rather has the perfume of marketing, catered to the C-suites with buying power.”
Moreover, the developer also highlighted the skill required to write strong code, stating that AI should be seen as a supplement and an aid rather than a substitute for continued training.