Loop Engineering: Building AI Agents That Improve

Every technology platform shift eventually produces its own version of the same insight: shipping the thing is not the same as the thing working. We learned this with enterprise software in the nineties, with web applications in the aughts, with mobile in the twenty-tens. Each time, the industry spent a few years celebrating what it had built before sobering up to the harder problem of making it reliable. AI agents are now having that moment, and the people building them are starting to sound less like futurists and more like engineers who've been burned before.

Sydney Runkle, a product manager on LangChain's open source team and a former software engineer, ran a webinar recently laying out what she calls loop engineering — a design philosophy for building AI agents that don't just complete tasks but get better at completing them over time. It's worth taking seriously, and not just because LangChain has a significant stake in the problem. The underlying argument maps onto patterns that anyone who has run software in production for more than a few years will recognize immediately.

The core claim is straightforward: an AI agent — think of it as software that can plan, use tools, and make decisions without step-by-step human instruction — is only as reliable as the structure you build around it. The model sitting at the center has a fixed level of capability. What you can control is the harness. And the harness, Runkle argues, should be built out of loops.

She describes four of them, each wrapping around the previous one like the layers of an onion — or, as she puts it, like Russian dolls.

The first loop is the basic agent cycle: the model receives a task, calls whatever tools it needs (searching a database, writing code, sending a request to an external service), processes what those tools return, and continues until it decides the task is done. This is the part the industry has been talking about for two years. It is necessary but not sufficient. A model with good tools and a clear task can still produce work that is subtly wrong, and nothing in this basic loop will catch it.

The second loop is where reliability enters. Runkle calls it a verification loop, though the intuition is closer to what any software team would recognize as automated testing — except the tests run after every agent attempt, not just at deployment. After the agent completes a pass at a task, a separate grading process checks the output against a set of criteria. If the criteria aren't met, the agent tries again. If they are, the work is done. The practical upshot, which Runkle is candid about, is that this adds latency and cost. The trade-off is the same one you face when deciding whether to run a full test suite before every deployment: correctness is not free, and anyone who tells you otherwise is selling something.

The third loop is the one Runkle calls event-driven, and it addresses a failure mode that doesn't get enough attention: agents that nobody uses. An agent that requires a developer to manually trigger it will be used by developers. An agent that fires when you receive an email, or when a Slack message lands in a particular channel, or on a morning schedule — that's an agent that gets used. The loop here is less architectural and more behavioral: when an agent is embedded in the systems where decisions actually get made, it improves those systems, which generates more reasons to use the agent, which generates more data about how it performs. Runkle is talking about workflow integration, but the underlying point is one that anyone who has deployed enterprise software knows well. The tool that lives in the workflow gets used. The tool that requires a context switch gets ignored.

The fourth loop is the one that carries the most weight — and the most uncertainty. Runkle describes it as the hill climbing loop, the self-improvement loop, the learning loop. The mechanics: after an agent run completes, a separate background process analyzes the full execution trace — the step-by-step record of what the agent actually did, which tools it called, what it missed, where it went sideways. That analysis identifies patterns: a tool called with the wrong arguments, a preference the agent keeps ignoring, a step it skips when certain context is absent. The goal, as Runkle describes it, is for this background process to update the agent's prompts, memory, and learned behaviors so that the next run benefits from what the last one revealed.

LangChain refers to this background capability as Engine — described during the webinar as a component of its LangSmith platform. Runkle presents it as the mechanism for closing this outer loop, though she is careful about the scope: the kinds of updates being discussed are changes to how the agent reasons and what it remembers, not autonomous rewrites of production code. She's clear that harness changes significant enough to affect system architecture should still go through human review. That's a meaningful distinction, and it matters for anyone trying to decide how much of this to trust.

Here is where the pattern recognition kicks in, because this is not actually a new problem under a new name. The idea of systems that log their own behavior, analyze those logs for failure patterns, and update their configuration accordingly has a long history in operations — from self-tuning databases in the nineties to modern observability stacks that feed anomaly detection. What's changed is that the thing being tuned is non-deterministic. A database query either returns the right rows or it doesn't. An agent deciding how to respond to an email from a VP versus a colleague is making judgment calls that don't have clean ground truth. That's a genuinely different challenge, and Runkle doesn't paper over it. When asked during Q&A how to build evaluations for agents where you don't have a known correct answer, her response was direct: "This question might be above my pay grade." She then offered a partial answer — using agents to generate evaluation criteria from traces — while acknowledging she'd route the deeper research question to colleagues better positioned to answer it. That kind of precision about what's known and what isn't is rarer in vendor presentations than it should be.

The self-improvement loop also has an unsexy prerequisite that Runkle is appropriately emphatic about: evaluations. In software development, an eval suite is a set of tests you run to verify that your agent's behavior meets defined standards — think of it as the functional equivalent of a regression test suite, except instead of checking whether a function returns the right value, you're checking whether the agent handles a class of situations the way you want it to. Without good evals, Runkle argues, you cannot have confidence that any proposed improvement actually improves things. The hill climbing loop is only as good as the slope it's measuring. Teams that skip the eval infrastructure and jump straight to automation are not building self-improving agents — they are building agents that change in ways they cannot verify.

The human oversight question gets addressed honestly. Runkle's position is not that automation replaces human judgment but that it should be applied where human judgment adds clear value and not demanded where it doesn't. Flagging spam as spam doesn't need a human sign-off. Sending an email in someone's name, booking travel, or merging a code change — those do. The practical challenge, she notes, is making human review easy enough that it actually happens rather than becoming a bottleneck that agents pile up against. An agent doing good work but waiting days for approval is a workflow problem, not an agent problem.

What stays with me about this framework — and I mean this as observation rather than endorsement — is the direction the anxiety has shifted. Two years ago the question was whether agents could do useful work at all. Now the question is whether the work they do can be trusted, measured, and improved systematically. That is a more interesting and more tractable question. It's also the question that every mature software discipline eventually has to face. The agent industry has apparently reached that phase of adolescence.

Whether the specific tooling LangChain is building turns out to be the right answer is a different question. What's harder to argue with is the underlying structure: a system that can observe its own behavior, score it against criteria, and make targeted adjustments is more useful than one that can't. We already knew that. We just didn't have software that could plausibly do the observing.

By Bob Reynolds, Senior Technology Correspondent, BuzzRAG