That Anthropic Study on AI and Coding? It's Complicated

Anthropic just dropped a study claiming AI coding assistants make developers worse at their jobs. The headline finding: developers using AI scored 17 percentage points lower on comprehension tests than those who coded manually—nearly two letter grades. The debugging gap was even worse.

Sounds damning, right? Except when you actually dig into how this study worked, the picture gets a lot messier.

What They Actually Tested

AnthropicтАЩs researchers recruited 52 тАЬmostly junior software engineersтАЭ and had them complete coding tasks in Trio, a Python asynchronous programming library none of them had used before. Half got AI assistance via a sidebar chatbot. Half coded manually. Both groups then took a quiz on what theyтАЩd just built.

The AI group finished about 2 minutes faster on average (not statistically significant). But they bombed the quiz—50% average versus 67% for the manual coders. The study concluded that AI assistance might speed up coding while simultaneously preventing developers from actually understanding what theyтАЩre building.

HereтАЩs where it gets interesting: the setup matters way more than the headlines suggest.

The Environment Problem

These developers were thrown into a browser-based coding environment theyтАЩd never seen, using a library theyтАЩd never touched, with an AI tool they werenтАЩt familiar with, and told to work as quickly as possible. Oh, and they knew theyтАЩd be quizzed afterward.

That last part is crucial. The study explicitly encouraged participants тАЬto work as quickly as possible.тАЭ When youтАЩre racing against a clock with an unfamiliar tool, how much time are you realistically spending asking the AI to explain concepts versus just getting it to spit out working code?

The researchers found their answer: some participants spent up to 11 minutes—30% of the total time—just composing AI queries. And hereтАЩs the wild part: half the AI-using group spent most of their time manually retyping the AIтАЩs code suggestions instead of copy-pasting them.

Manually. Retyping. AI-generated. Code.

When you remove the retypers from the data, the AI users were 25% faster. But their quiz scores stayed just as low.

How People Actually Used the AI

The study tried to categorize interaction patterns, and this is where things get genuinely useful. The lowest-scoring approach—under 40% average—was what researchers called тАЬAI delegation,тАЭ where developers just let the AI write everything. These people finished fastest but learned nothing.

The highest-scoring pattern was тАЬgeneration then comprehension.тАЭ These developers asked the AI to generate code, then followed up with questions to understand what it had created. They scored 65% or higher on the quiz.

HereтАЩs the problem: only 2 out of 25 AI users did this. Two people. The researchers literally created a behavioral category from a sample size of two.

Why so few? Remember: they were told to work quickly. When youтАЩre a junior developer who already feels insecure about your skills, racing a clock, are you really going to slow down to ask тАЬhey AI, explain this async concept to me?тАЭ Or are you going to grab the working code and move on?

What This Study Actually Tells Us

The Anthropic research captures something real: if you use AI purely as a code vending machine, you wonтАЩt learn much. Shocking, I know.

But hereтАЩs whatтАЩs missing from the discourse around this study:

This was about first-time usage in artificial constraints. The researchers themselves admit тАЬwe expect AI would be more likely to significantly increase productivity when used on repetitive or familiar tasks.тАЭ Translation: this study measured what happens when junior developers encounter AI tools for the first time in an unfamiliar environment with unfamiliar libraries under time pressure.

ThatтАЩs useful data! But itтАЩs not proof that тАЬAI makes developers 20% dumber.тАЭ ItтАЩs evidence that if you donтАЩt know how to use a tool effectively and youтАЩre incentivized to rush, youтАЩll use it poorly.

The study canтАЩt tell us anything about experienced developers using AI tools theyтАЩve integrated into their workflow over months. It canтАЩt measure the compound productivity gains of someone whoтАЩs learned when to lean on AI versus when to code manually. It definitely canтАЩt account for developers who are shipping complex features in hours that would normally take days.

Because that happens. Real companies are seeing 30% year-over-year improvements in code output from developers using AI. Experienced engineers are having AI agents make architectural decisions that inform how theyтАЩd write code manually. This isnтАЩt hypothetical—itтАЩs happening right now in production environments.

The Real Question

What the Anthropic study does surface is something worth worrying about: the comprehension gap is real if youтАЩre not deliberate about how you engage with AI tools.

"The AI group averaged 50% on the quiz compared to 67% in the handcoded group, or the equivalent of nearly two letter grades. The largest gap in scores between the two groups was on debugging questions," the researchers found.

ThatтАЩs a meaningful signal. If developers are blindly accepting AI-generated code without understanding it, they wonтАЩt be able to debug it when it breaks. TheyтАЩll hit a ceiling where they can only build what the AI can generate for them.

But the solution isnтАЩt тАЬdonтАЩt use AI.тАЭ ItтАЩs тАЬlearn to use AI as a teaching tool, not just a code generator.тАЭ The two people in this study who did that—who generated code then asked for explanations—performed well on both speed and comprehension.

What we need now isnтАЩt more studies of junior developers speedrunning unfamiliar libraries. We need research on how developers build expertise over time when AI is part of their workflow from day one. We need to understand what тАЬgood AI-assisted learningтАЭ looks like at scale, not just in a 30-minute lab setting.

Because the technology isnтАЩt going anywhere. The only question is whether weтАЩre going to figure out how to use it without atrophying our own capabilities—or whether a generation of developers is going to wake up one day and realize they can only build what their AI tools already know how to make.

—Tyler Nakamura