In a 2025 trial, experienced open-source developers took 19 percent longer with AI tools, while believing they had been faster. The lesson is about measurement as much as AI.
Almost everyone who uses AI coding tools feels faster. In 2025, the research organisation METR set out to test that feeling with a randomised controlled trial, and the result surprised almost everyone, including the developers involved.
The trial
Sixteen experienced developers worked on 246 real issues in large open-source projects they had contributed to for years, repositories averaging more than 22,000 stars and a million lines of code. Each issue was randomly assigned either to allow AI tools, mainly Cursor with Claude 3.5 and 3.7 Sonnet, or to forbid them. Tasks took about two hours on average.
The result
With AI allowed, developers took 19 percent longer. Before starting, they had expected AI to speed them up by 24 percent. Afterwards, having been slowed down, they still believed it had sped them up by about 20 percent.
What it does and does not show
METR was careful about the limits. The study covered 16 developers, working in codebases they knew intimately, with early-2025 tools. It does not show that AI slows most developers, or that it would slow people in unfamiliar code, where help is worth more. METR has since said the results are out of date, as the tools have moved on.
The lasting lesson
The finding that has aged best is the gap between perception and measurement. Capable professionals were wrong about the direction of an effect on their own work, by a wide margin. That is a reason to measure AI's impact on real tasks, against a baseline, before rebuilding a team or a budget around how it feels.
- Pick a few representative tasks and time them with and without AI.
- Count review and rework time, not just the time to a first draft.
- Measure again as tools change; last year's result may not hold.
Sources
Written by Aevion Labs from the sources above. Figures are as reported by each source on the dates shown.