A “new milestone in competitive programming” was trumpeted by Google’s British AI subsidiary DeepMind earlier this month, when it unveiled AlphaCode, a system it claims can write fully fledged computer programmes that compare favourably to the work of humans. Software development has long been pinpointed as an area where AI can have a significant impact, and with the advances AlphaCode and other systems offer, is the prospect of machines replacing human coders a realistic one?
DeepMind says AlphaCode is capable of understanding a problem then writing a programme which solves that problem. It says it has tested the system against people who took part in coding contests and found that its results rank within the top 54% of human participants.
“Solving competitive programming problems is a really hard thing to do, requiring both good coding skills and problem-solving creativity in humans,” said Google software engineer Petr Mitrichev, who takes part in coding competitions. “I was very impressed that AlphaCode could make progress in this area, and excited to see how the model uses its statement understanding to produce code and guide its random exploration to create solutions.”
With AlphaCode displaying such impressive progress, Mitrichev and his colleagues could be forgiven for looking over their shoulders.
How significant is DeepMind’s AlphaCode?
DeepMind used a combination of unsupervised and supervised machine learning to train the AI models that underpin AlphaCode. First the system absorbed 715 gigabytes of data from coding repository GitHub, before the DeepMind team fine-tuned it using a database it had created of problem statements, test cases and submissions – correct and incorrect – from coding contests.
When presented with a new problem, AlphaCode trawls the information it has, comes up with thousands or, in some cases, millions of solutions, then filters these down to the ten best which it submits for the contest. If one of these turns out to be correct, the team considers it a success.
This process marks AlphaCode out from other AI coding tools, such as the OpenAI Codex, which are capable of generating specific code snippets on command. “This is a breakthrough in the sense that they are solving programming problems much more complex than earlier systems,” says Professor Ernest Davis, from the computer science department at New York University.
But, Professor Davis, says, it is not without its limitations. Because its method involves coming up with so many potential solutions to a problem (before filtering them down to find the best ones), the amount of training data required to write more complex programmes is likely to be enormous. “It is reasonable to expect that the number of samples required increases exponentially with the length of the program,” Professor Davis says. It is also reliant on the relevance of its training data to come with accurate solutions, something a human programmer could work around even if they had no knowledge of the subject.
AlphaCode is a useful contribution to AI research, but AI coding at the level of a skilled human is still light-years away.
Dzmitry Bahdanau, ServiceNow
And compared to DeepMind’s other AI accomplishments, AlphaCode is a relatively minor success, argues Dzmitry Bahdanau, a research scientist at workflow software vendor ServiceNow who has studied how AlphaCode works. “At this point in time, AlphaCode is a useful contribution to AI research, but AI coding at the level of a skilled human is still light-years away,” he says. “This is not AlphaGo in terms of beating humans, and not AlphaFold in terms of revolutionising an entire field of science.”
Will AI replace human programmers?
Even if an AI such as AlphaCode could overcome these limitations, the conditions in which it performed in testing are significantly different to the world of professional software development, says Dr Peter Schrammel, lecturer in computer science at the University of Sussex and CTO and co-founder of software company Diffblue. “Coding challenges are relatively specific as the assignments need to be as non-ambiguous as possible in order to be able to compare solutions in a fair manner,” he explains. “The description is quite well-structured and examples are given in a consistent way.”
Real-world development is “a different beast”, he argues. “Requirements are vague and determined in incremental iterations,” he says. “Software developers negotiate requirements in conversations with the stakeholders, implement a first version, then demonstrate it and discuss with the stakeholders what to improve for the next iteration.” Whereas in a coding challenge “it is perfectly known what the requirements are,” Dr Schrammel says, “in real-world software development, extracting the requirements from the stakeholders’ heads is the primary challenge.”
Bahdanau agrees, and says he does not believe many people in the industry “seriously view [AI replacing human programmers] as a goal”. “Humans understand the broader context, the actual real-world purpose of why the code needs to be written,” he says. “AlphaCode, Codex, and others are just very advanced statistical methods that have billions of gigabytes of code distilled in them. That said, AI has the potential to help human software developers accomplish their work faster.”
How will AI aid software development?
But with the ongoing tech skills shortage threatening to slow the pace of digital transformation, AI has a significant role to play in software development, aiding increasingly stretched IT teams. Tech Monitor has reported on the growing popularity of low-code and no-code systems, which enable non-technical staff to build digital functions to aid their work, with no-code expected to account for 65% of app development by 2024, according to Gartner.
For professionals, tools such as GitHub CoPilot, which suggests lines of code or functions to aid development, are likely to be the most productive use of AI in the short to medium term, argues Dr Schrammel. “Software development itself has largely benefitted from automation since its inception,” he says. “Nobody is manually crafting machine code anymore. Automated tools such as compilers and code generators transform high-level formal specifications of an application into machine-executable code.”
Software development itself has largely benefitted from automation since its inception.
Dr Peter Schrammel, University of Sussex
The rise of these compilers has seen “tedious tasks” automated, but not led to a decrease in the number of human developers, Dr Schrammel adds. “One reason could be that many software development activities such as requirements engineering rely on a wide range of domain knowledge and have to deal with large degrees of ambiguity,” he says. “These activities are much harder to automate away than activities that are more regular, such as coding from well-defined specifications and testing.”
Indeed, further automation of testing systems is likely to be the next frontier of AI in programming, Dr Schrammel says. As automation increases, Bahdanau believes an increasingly important role for human developers will be to ensure that the way AI generates code is understood. “Professional developers will need to understand and verify the algorithm that AI proposes,” he says. “For citizen developers using low-code or no-code programming interfaces, the bar for AI trustworthiness is going to be even higher, and they will need a developer experience that explains the logic the AI proposes.”
