Same Model, 30% to 95%: Why the Harness Beats the

Claude Opus scored 30% on ARC-AGI's closed private evaluation set. According to the host of last night's YC Paper Club, the same weights scored 95.5% when run through Prime Intellect's harness, and Nvidia's AVO reportedly hit 100% with a similar setup. No new weights. No fine-tuning. Just a better wrapper around the model.

I went into this session ready to roll my eyes. The event, covered in Why The Harness Matters More Than The Model, is titled like a manifesto, and the AI world has heard 'the scaffolding matters' before, usually from people selling scaffolding. The host opened by reading Reddit posts dismissing harness work as prompt engineering: 'Context engineering is not a research problem,' one commenter wrote. Wrapper engineering with a sneer attached.

Then the presenters started showing numbers, and my skepticism converted to curiosity somewhere around minute fifteen.

What a Harness Actually Is

Strip away the discourse and the idea is simple. A raw LLM is a function: tokens in, tokens out. A harness is everything wrapped around that function: tools, a code execution environment, memory, subagents, feedback loops, persistent state. Seth Karten, a Princeton researcher at Prime Intellect who built Prime Agent, described it as 'the layer between the LLM and the world, which adds things like steady state, tools, and computation.'

The historical framing the host offered is useful. The GPT-2 harness of February 2019 was just a loop until the end-of-sequence token. Then few-shot examples (2020), then chain of thought, then Toolformer and WebGPT's tool calling, then MemGPT's ability to read and write its own context, then Voyager's skills, then ReAct-style reflection, then recursive subagents. Each of those was an innovation in what the model could express, not in the weights themselves. Six years of capability gains, and a lot of them live in the wrapper.

Calling that work 'just prompt engineering' is like calling a compiler a macro. DSPy treats the system prompt as an optimizable artifact, searched with genetic programming over a small training set. Darwin-Gödel machines go further and let a meta-harness rewrite the harness code itself. The host called this shift, in roughly the last six months, the move from static bindings to self-improving ones.

Prime Agent and the Cache Metaphor

Karten's design philosophy is 'make the harness as expressive as possible.' His mental model borrows from hardware: the model weights are L1 cache, the active context is L2, and the file system is L3. The interesting engineering is cache management. A Python REPL lets the agent manipulate state programmatically instead of stuffing everything into its context window. Subagents persist as dormant sessions that can be messaged back to life with their context intact. Compression, garbage collection, refinement of skills and memories. His analogy: a raw LLM looks like a Turing machine; a harnessed agent looks like a von Neumann computer that can read and write external memory.

The ARC-AGI numbers I cited above come from Karten's talk. He was refreshingly honest about the messy path there: a first run hit 99.9%, which he discovered was a sandboxing error on his end; a proper rerun with GPT-4 got 78%; Opus got 95.5%. He also volunteered the unflattering numbers. Running Prime Agent on the MAgent benchmark burned about $5,000 with little performance improvement before they pulled the plug. 'Price-performance is very important,' he said, and being able to work with your context programmatically is what saves money.

That's the honest tension in all of this. The same harness that turns 30% into 95.5% on one benchmark can cost thousands of dollars to underperform on another. Karten himself said results on long-horizon auto-research experiments were 'highly dispersed' and that they can't attribute gains cleanly to tooling versus the model. If you've been following the harness-versus-model debate, our earlier breakdown of what actually drives performance covers the same tension from a different angle.

Open Jarvis and the 800X Claim

Jon Saad-Falcon, a Stanford PhD student working with Christopher Ré's group, presented Open Jarvis, a personal AI stack that runs entirely on-device. His argument: local models are now only 6-12 months behind the frontier (he cited Qwen 3.8 27B matching Claude 4.6 Opus as of August 2025, per the recording), and laptop accelerators are closing the gap further.

His team distilled the personal AI stack into five primitives: interface, agent logic, model, inference engine, and an optimization loop over the whole specification. The clever trick is letting a cloud model optimize the local stack. A frontier model diagnoses your local setup and rewrites its configuration; the optimized stack then runs locally for free. Saad-Falcon reported 800x lower execution costs than cloud inference, with reduced latency. He acknowledged that local models still fail on many tasks. The gap is shrinking monthly, but if you need frontier intelligence for a query, no local stack will save you this year.

The meta-observation is delicious: the harness is now good enough to use frontier models to optimize smaller models' harnesses. The big model is the compiler. The small model is the product.

QM, or What Happens When You Actually Deploy This

Josh France and Regan Bell from YC Labs presented QM, YC's open-source agent platform, and their talk was the most sobering of the night, because it's the only one describing production deployment to every employee of a real company.

Their architecture inverts the OpenClaw model: instead of an agent living inside its own computer, QM pulls the brain out of the sandbox. State lives in Postgres. Sandboxes become elastic resources the agent picks based on task difficulty, and the agent can even choose its own model provider to route around refusals.

Their failure modes are the ones nobody puts in demos. Agents give up too soon, so they built a 'grind tool' that budgets time and tokens against a goal (the agent isn't allowed to quit early). More importantly: 'Agents do not understand social context at all.' Tell a coworker something and they implicitly know where it can be repeated. An agent will happily leak private information into a context where it has no business being. France's conclusion: how much you can trust an agent's shared brain is limited by the granularity of your permission system, and most companies don't have a good one. Their automated self-improvement loop, where agents fix bugs with an LLM as judge, produced what he called 'protagonist syndrome': agents optimizing their tiny slice of the system while breaking the whole. Humans stay in the loop.

The pattern isn't unique to YC; agents that rewrite their own operational code run into the same oversight problem, and harness engineering has its limits when the failure is organizational rather than architectural.

What I Actually Think

The strongest version of the harness argument: benchmarks like ARC-AGI measure adaptation to novel problems, and a harness is precisely an adaptation machine. The model brings the priors; the wrapper brings the memory, tools, and feedback. Asking which 'matters more' is asking the wrong question, because the results are a product of both. But the marginal dollar right now buys more on the harness side. When the same weights swing from 30% to 95.5%, the bottleneck is upstream of the weights.

The caveats deserve equal billing. These are self-reported numbers from people presenting their own systems, mostly on benchmarks they chose. Cost scaling is hairy: the $5,000 MAgent run is a warning, not an anecdote. And self-improving harnesses raise the evaluation stakes: if the harness can rewrite itself, your benchmark is now training data, and 'protagonist syndrome' is what happens when the judge shares DNA with the defendant.

I came in expecting prompt engineering cosplay. I left thinking the field has been measuring perplexity when it should be measuring adaptability, as Trajectory's CEO argued in a presentation the host cited. The weights are the brain. The harness is the body, the tools, the habits, the colleagues. You can swap in a smarter brain, but until the body works, the smarts go nowhere.

The next YC Paper Club covers alternative compute paradigms. Given how fast the ground moved between Paper Club sessions this year, I'd start reading the literature now, or better yet, hand it to an agent with a good harness and a strict token budget.

Yuki Okonkwo covers AI and machine learning for BuzzRAG.