AI Agents for Science Hit a Wall: the Hypothesis

The benchmark results for AI research agents look impressive until you ask what problems they were actually solving. Coding puzzles. Toy optimization tasks. Supervised learning benchmarks where the data is clean, the objective is fixed, and "discovery" means finding a slightly better set of hyperparameters. That's not science. That's a very fast graduate student.

Sina Shahandeh, cofounder and CTO of Radicait, made this point plainly in a recent talk at the AI Engineer conference — and it's a distinction that tends to get steamrolled in the rush to announce that AI is now doing research. His actual argument is more specific, and more useful: AI agents are genuinely good at one part of the scientific process and structurally bad at another. Knowing which is which matters if you're trying to build something that works.

The part that's solved, and the part that isn't

The part that works: execution. Give an AI agent a well-defined objective and a codebase, and it will run experiments, track results, and iterate — faster and more tirelessly than any human. Shahandeh is direct about this. The agents are "very good at implementation of the code or running the experiments over lots of data," he says. That's real. It's also not where science gets stuck.

Where science gets stuck is at the moment a researcher has to invent a new frame for the problem. Not optimize within the current approach — abandon it and try something structurally different. Shahandeh calls this "research taste," which is an honest term for something that's genuinely hard to define: the instinct that the whole setup is wrong and needs to be rebuilt from a different premise. AI agents plateau there. They run out of ideas. The human researchers, especially the good ones, keep going.

This isn't a complaint about current model quality — it's a description of an architectural gap. The agents aren't failing because they're slow or inaccurate; they're failing because they're not generating the kind of ideas that produce step-change improvement. They're hill-climbing when the answer is in a different valley entirely.

What Radicait is actually building

To make this concrete, Shahandeh describes Radicait's core project: generating synthetic PET scans from CT scans. The clinical problem is real — when a patient has a suspicious nodule in their lung, doctors want to know if it's cancerous, which typically means a PET scan. PET scans measure metabolic activity; tumors tend to absorb more of the radioactive tracer and light up on the image. But PET scans are time-consuming and expensive to produce. The question Radicait is pursuing is whether an AI model can look at a CT scan — essentially a detailed structural X-ray — and infer what the PET scan would show.

The model doing this translation is a GAN, or generative adversarial network. The name sounds complex; the concept isn't. Two sub-models work against each other: one generates synthetic images, the other tries to distinguish the fakes from the real thing. Through that competition, the generator gets better. It's been a standard tool in medical imaging research for years.

The problem Shahandeh ran into wasn't building the model. It was improving it once it reached a plateau. The initial version treated CT scans in 2D — each cross-sectional slice processed independently, stacked like pages in a book. An autonomous agent iterating on that setup would tune parameters, adjust training procedures, try variations on the same basic architecture. It wouldn't, on its own, look at the problem and say: these are slices of a three-dimensional body, and we should be processing them in three dimensions. That idea requires stepping back from the current frame entirely. In Shahandeh's telling, he had to inject that idea himself — mid-loop, manually — before the agent could test and develop it.

That's the gap. Not capability. Structure.

The fix: force the agent to build a map first

Shahandeh's solution borrows from something called chain-of-thought prompting — a technique where you instruct a language model to work through a problem step by step rather than jumping to an answer. Research has shown this produces meaningfully better reasoning on complex tasks. The underlying idea is old: break the problem down, solve the pieces, build back up. What's interesting about Shahandeh's application is that he's using it not to improve the agent's final answer, but to expand the space of questions it can ask.

The mechanism is a forced decomposition step. Before the agent begins its optimization loop, it's prompted to map the entire problem as a hierarchy: top-level goal, major components, sub-components, all the way down to the specific code modules. The output looks like a structured knowledge graph — Shahandeh demonstrates it in Obsidian, a tool that renders linked documents as a visual network — with the problem statement at the top and actual code at the leaves.

Once that map exists, the agent's hypothesis generation changes character. Instead of proposing variations on whatever it was already doing, it can navigate to any node in the map and ask: what could be different here? Data preprocessing. Loss function. Architecture. Training procedure. Each component becomes a site for a candidate hypothesis. The agent can generate and evaluate a hundred distinct proposals across the full problem structure — including the proposal that the architecture should be three-dimensional instead of two-dimensional.

That's not a trivial improvement. It's the difference between an agent that tinkers and an agent that can, occasionally, think. Whether the map is rich enough to surface genuinely novel ideas — or just a more systematic version of the same search — is an open question. But as scaffold engineering goes, it's the most credible approach to the hypothesis problem I've seen described in practical terms. The co-evolving questions problem is real; Shahandeh's hierarchy at least gives the agent more places to look.

Where it gets genuinely complicated

The image registration example Shahandeh adds toward the end of the talk is worth dwelling on, because it illustrates a second layer of the problem that the hierarchy trick doesn't solve.

Image registration, in this context, means taking a CT scan and a PET scan of the same patient — scanned at different times, potentially on different equipment, with the patient in a different position — and aligning them spatially so they can be compared. It's another optimization problem, but one where many of the quality metrics are qualitative. You can't always express "this alignment is wrong" as a number. A trained radiologist looks at the images and knows. An agent, at least today, mostly doesn't.

Shahandeh's response is to bring in multimodal models — AI systems that can process images directly, not just text and code — as a quality-control layer within the loop. A model with strong visual reasoning looks at the registration output and flags whether it looks right. A stronger frontier model handles the higher-stakes hypothesis generation. The loop becomes a small team of specialized agents rather than a single generalist.

This is sensible, but Shahandeh is candid about where it still falls short: "No LLM today is able to identify these [subtle changes] very well because they are just not simply trained on scientific images and scientific data." A suspicious pulmonary nodule is a few pixels in a sea of gray. The clinical significance of that cluster depends on training that current general-purpose models simply don't have. You can give an AI agent a picture of a lung and ask if it looks right; you cannot yet expect it to see what a pulmonary radiologist sees.

That admission clarifies what Radicait's system is and isn't. It's an agent that can search more intelligently, hypothesize more broadly, and critique its own implementations with more nuance than a basic coding agent. It is not a system that can close the scientific loop independently, because closing the loop requires observation — and the models capable of observing scientific data at the level of a trained specialist don't yet exist for most domains.

Shahandeh is betting that better models will reduce the need for the scaffolding tricks: "I think going forward with the newer models that are much better post-trained to compartmentalize the problems and break down the problems, we probably need less and less of these tricks down the road." Maybe. The history of such predictions suggests caution. But the tricks are working now, on real problems, with real patient data — which is a more grounded claim than most of what circulates under the "AI for science" banner.

Building a startup around the gap between what AI agents can do today and what better models will eventually handle natively requires either intellectual honesty or a very patient business model — possibly both.

Bob Reynolds is Senior Technology Correspondent at Buzzrag.