Why AI Refactors Code Perfectly But Can't Count

Something shifted in December. Andrej Karpathy—former Tesla AI director, OpenAI co-founder, the person who essentially invented the term "vibe coding"—told a Sequoia Capital audience he'd never felt more behind as a programmer. Coming from someone who built self-driving systems, that statement deserves attention.

What changed wasn't incremental. Karpathy described a threshold moment when AI coding tools stopped producing snippets requiring human correction and started delivering complete, working applications end-to-end. "The chunks just came out fine," he explained at Sequoia's annual AI event. "I kept asking for more and it just came out fine. I can't remember the last time I corrected it."

This isn't hype. It's documentation of a phase transition in how software gets made. But it also exposes a deeper puzzle: why can these same models refactor 100,000-line codebases flawlessly while simultaneously suggesting you walk to a car wash to clean your vehicle?

The Verifiability Thesis

Karpathy's framework for understanding AI's jagged capabilities centers on a single concept: verifiability. "Traditional computers can easily automate what you can specify in code," he noted. "This latest round of LLMs can easily automate what you can verify."

The distinction matters. Software 1.0 required explicit instructions—step-by-step specifications of exactly what the computer should do. You write the rules, the machine follows them. Software 2.0, as Karpathy coined it years ago, meant programming through datasets and neural network architectures rather than explicit code. Now Software 3.0 shifts the lever again: programming happens through prompts, with the context window serving as your control surface over an LLM that interprets and executes.

In verifiable domains—math, code, logic problems with clear right answers—AI models demonstrate what looks like superhuman ability. Write code, compile it, run it: you immediately know if it works. Calculate 2+2: verification is instant and deterministic. Crucially, verification doesn't require human judgment.

That's why these models excel at coding. It's also why they stumble on tasks that seem simpler. The car wash problem—should you drive or walk 50 meters?—requires contextual understanding that can't be programmed as a verification function. The infamous strawberry problem (counting R's in the word) lacks a built-in reward signal during training.

"When Frontier Labs are training these LLMs, these are giant reinforcement learning environments," Karpathy explained. "They are given verification rewards and then because of the way that these models are trained, they end up basically progressing and creating these jagged entities that really peak in capability in verifiable domains like math and code."

The jaggedness isn't a bug. It's a feature of how these systems learn.

The Economics of Training

But there's a second layer. AI labs don't optimize models in a vacuum—they respond to economic incentives. Code generation became extraordinarily valuable because enterprise customers would pay significant premiums to move 10x or 100x faster in software development. That commercial reality shaped training priorities.

"Some things are significantly more valuable in the economy and end up creating more environments because the labs wanted to work in those settings," Karpathy observed. "Code is a good example of that."

Show me the incentive, and I'll show you the outcome. The labs focused training data, reinforcement learning, and optimization efforts on domains where customers would pay. Code topped that list. The combination of easy verifiability and massive economic value created the conditions for rapid capability gains.

This explains why Claude Opus 3.7 can find zero-day vulnerabilities yet fails common-sense reasoning tasks. The model wasn't optimized for the latter. The training environment didn't reward it. The verification mechanisms didn't exist.

The End-to-End Neural Network Future

Karpathy's more provocative claim concerns where this leads: toward end-to-end neural networks replacing traditional software architecture entirely. He described building a menu app using conventional programming—OCR for text, image generation for pictures, rendering logic to overlay them—then watching someone solve the same problem by simply telling a multimodal model: "Here's my menu photo. Overlay generated images of the dishes onto it."

No explicit steps. No traditional code. Just outcome specification and neural network execution.

"All of my menu gen is spurious. It's working in the old paradigm. That app shouldn't exist," Karpathy said. "The software 3.0 paradigm is a lot more raw. Your neural network is doing more and more of the work and your prompt or context is just the lever."

This connects to what's called "the bitter lesson" in AI research: never bet against end-to-end neural networks over human-designed heuristics. Elon Musk learned this at Tesla. For years, Autopilot combined neural nets with manually coded rules—if you see a red octagonal sign reading "STOP," it's a stop sign. But that approach required defining every edge case manually. When Tesla switched to pure end-to-end neural networks, performance improved dramatically while maintenance complexity dropped.

Karpathy worked on that transition. He knows the lesson empirically.

The implication: we may be watching the gradual obsolescence of traditional programming. Not immediately. Not universally. But the trend line points toward neural networks handling an expanding circle of tasks that currently require explicit code.

Consider how installation instructions are evolving. Traditional software requires bash scripts that balloon in complexity to handle different platforms. OpenClaw's installation "script" is now just text: "I'd like you to set up OpenClaw. Install as a skill if I have npm. If not, do this instead." The agent figures out the rest. Journey Kits follows the same pattern—outcome specification, not step specification.

"What is the piece of text to copy paste to your agent? That's the programming paradigm now," Karpathy said.

What This Means for Policy

From a regulatory perspective, this creates fascinating challenges. How do you audit or regulate software that isn't written through traditional code review processes? If neural networks increasingly handle tasks end-to-end, what transparency mechanisms apply?

The EU's AI Act attempts to regulate high-risk AI systems, but its frameworks assume something closer to Software 2.0—neural networks as components within larger systems. Software 3.0, where prompts and context windows become the primary interface, doesn't fit cleanly into existing regulatory categories.

Verification mechanisms that work for traditional software (code inspection, formal methods, testing protocols) don't translate directly. You can't review the "source code" of a neural network's decision-making in the same way. The model's weights aren't interpretable as traditional program logic.

Yet verifiability—Karpathy's central concept—might offer a regulatory foothold. If AI excels in verifiable domains, perhaps regulation should focus on ensuring outputs can be verified rather than trying to specify the process. For high-stakes applications (medical diagnosis, financial trading, legal analysis), build verification mechanisms into the deployment architecture rather than trying to audit the training process.

This inverts traditional software regulation, which often focuses on development practices and code quality. It's outcome-focused rather than process-focused. Whether that's actually workable remains an open question.

The Jaggedness Paradox

Karpathy's framework explains current AI capabilities better than most alternatives. But it also raises uncomfortable questions about what "intelligence" means when it's this domain-specific. The fact that state-of-the-art models demonstrate such extreme jaggedness—superhuman in narrow domains, subhuman in adjacent ones—suggests we're not close to artificial general intelligence, whatever the marketing claims.

"This is kind of the argument that we're not at AGI right now," Karpathy acknowledged. "Because if we were, the skills that it would have in code would be generalized beyond just code. The fact that there is such jaggedness is proof in itself that we do not have generalized intelligence yet."

Unless, he added, we simply don't know how to extract that generalization from models that possess it. That uncertainty cuts both ways for policy-makers trying to anticipate AI trajectories.

The December inflection point Karpathy described appears real. AI coding capabilities took a discrete jump in capability. But the same models still can't reliably count letters or apply basic contextual reasoning. Understanding why—verifiability combined with economic incentives shaping training priorities—helps predict where capabilities will develop next and where they'll remain frustratingly limited.

For anyone building on these systems, deploying them in production, or writing regulations to govern them, that distinction matters more than almost any other technical detail.

Samira Okonkwo-Barnes covers technology policy and regulation for Buzzrag.