AI's elusive coding speedup
Add Axios as your preferred source to
see more of our stories on Google.

Illustration: Allie Carl/Axios
A surprising new study finding that AI tools can reduce programmers' productivity is upending assumptions about the technology's world-changing potential.
Why it matters: Software runs our civilization, and AI is already transforming the business of making it — but no one really knows whether AI will decimate programming jobs, or turn every coder into a miracle worker, or both.
Driving the news: The study by METR, a nonprofit independent research outfit, looked at experienced programmers working on large, established open-source projects.
- It found that these developers believed that using AI tools helped them perform 20% faster — but they actually worked 19% slower.
- The study appears rigorous and well-designed, but it's small (only 16 programmers participated, completing 246 tasks).
Zoom out: For decades, industry visionaries have dreamed of a holy grail called "natural language programming" that would allow people to instruct computers using everyday speech, without needing to write code.
- As large language models' coding prowess became evident, it appeared this milestone had been achieved.
- "The hottest new programming language is English," declared AI guru (and OpenAI cofounder) Andrej Karpathy on X early in 2023, soon after ChatGPT's launch.
- In February, Karpathy also coined the term "vibe coding" — meaning the quick creation of rough-code prototypes for new projects by just telling your favorite AI to whip up something from scratch.
- The most fervent believers in software's AI-written future say that human beings will do less and less programming, and engineers will turn into some combination of project manager, specifications-refiner and quality-checker.
- Either that, or they'll be unemployed.
Zoom in: AI-driven coding tends to be more valuable in building new systems from the ground up than in extending or refining existing systems, particularly when they're big.
- While innovative new products get the biggest buzz and make the largest fortunes, the bulk of software work in most industries consists of more mundane maintenance labor.
- Anything that makes such work more efficient could save enormous amounts of time and money.
Yes, but: This is where the METR study found AI actually slowed experienced programmers down.
- One key factor was that human developers found AI-generated code unreliable and ended up devoting extra time to reviewing, testing and fixing it.
- "One developer notes that he 'wasted at least an hour first trying to [solve a specific issue] with AI' before eventually reverting all code changes and just implementing it without AI assistance," the study says.
Between the lines: The study authors note that AI coding tools are improving at a rapid enough rate that their findings could soon be obsolete.
- They also warn against generalizing too broadly from their findings and note the many counter-examples of organizations and projects that have made productivity gains with coding tools.
One notable caution that's inescapable from the study's findings: Don't trust self-reporting of productivity outcomes. We're not always the best judges of our own efficiency.
- Another is that it's relatively easy to measure productivity in terms of "task completion" but very hard to assess total added value in software-making.
- Thousands of completed tickets can be meaningless — if, for instance, a program is about to be discontinued. Meanwhile, one big new insight can change everything in ways no productivity metric can capture.
The big picture: The software community is divided over whether to view the advent of AI coding with excitement or dread.
- "Quitting programming as a career right now because of LLMs would be like quitting carpentry as a career thanks to the invention of the table saw," the AI developer and blogger Simon Willison recently wrote on Bluesky.
- Or "it could be like quitting agriculture as a career thanks to the invention of tractors and combine harvesters instead," industry veteran Tom Coates replied.
