We are Changing our Developer Productivity Experiment Design – METR
March 6, 2026
METR previously published a paper which found the use of AI tools caused a 20% slowdown in completing tasks among experienced open-source developers, using data from February to June 2025.
To understand how AI is impacting developer productivity over time, we started a new experiment in August 2025 with a larger pool of developers using the latest AI tools.
Unfortunately, given participant feedback and surveys, we believe that the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools. The primary reason is that we have observed a significant increase in developers choosing not to participate in the study because they do not wish to work without AI, which likely biases downwards our estimate of AI-assisted speedup. We additionally believe there have been selection effects due to a lower pay rate (we reduced the pay from $150/hr to $50/hr), and that our measurements of time-spent on each task are unreliable for the fraction of developers who use multiple AI agents concurrently.
METR’s research last year into open source developer productivity using AI caused quite a stir because it suggested or even demonstrated in many people’s opinion that the uplift in developer productivity that many developers self-reported when using AI, AI based coding systems was perhaps illusory.
They are returning to this particular area of research, but as with many things, it is more complicated than it originally appeared.
If you’re interested in diving more deeply into this, there is a fantastic in-depth conversation between METR’s Joel Becker and Shawn Wang at Latent Space recently.
