Building Voice Agents: The Hard Engineering

There's a number that keeps appearing in conversations about voice AI, and it's not a benchmark score or a funding round. It's 500 milliseconds. Cross that threshold and users notice. Cross one second, and they hang up.

That single human-behavioral constraint — rooted in the fact that we respond to each other in about 300ms during normal conversation — is the organizing principle behind everything Rishabh Bhargava, who leads the voice AI team at Together AI, laid out at a recent AI Engineer conference. His talk is worth working through carefully, not because it hypes a product, but because it maps the actual engineering terrain with unusual specificity.

The headline: building a voice agent that works in the real world is not primarily a research problem anymore. It's a systems problem. And systems problems have different failure modes than research problems.

The pipeline nobody loves but everybody uses

The dominant production architecture today is a three-stage cascade: speech-to-text converts the user's audio into a transcript, an LLM processes that transcript and decides what to do, and a text-to-speech model converts the LLM's output back into audio. Conceptually simple. Operationally brutal.

Each stage has its own latency budget, and they're ruthlessly additive. Bhargava's rough rubric: the LLM consumes the largest share — targeting 200–300ms time-to-first-token — followed by TTS, with speech-to-text contributing least to both latency and cost. The speech-to-text target he cites is a P90 under 100ms with around a 6% word error rate, which sounds tolerant until you consider what errors propagate. Miss a drug name, a customer's name, a specific dollar amount — the LLM inherits the mistake, the TTS model voices it, and the caller hears something wrong. There's no spell-checker downstream to catch it.

The LLM piece has a particularly sharp constraint. Bhargava puts the useful model size range at 8–30 billion parameters. Go bigger and you blow the latency budget before you even account for network overhead. Go smaller and tool-calling quality degrades — which matters because tool calling is how voice agents actually do things in the world, like checking an order status or booking an appointment. You're not building a chatbot; you're building an agent, and agents need to act. That squeeze between capability and speed is where a lot of production deployments quietly suffer.

For TTS, there's a related metric worth understanding: real-time factor, which measures how much audio you can produce per second of processing time. A real-time factor below 1.0 means you're generating audio faster than it plays — that's the target. Above 1.0 and you're buffering, and users will feel it. As for TTS model architecture, the shift toward autoregressive transformers has improved naturalness significantly, though streaming remains genuinely difficult to get right.

The 75-millisecond problem nobody talks about in demos

Here's the part that doesn't show up in benchmarks: network latency.

The latency figures above — 100ms for STT, 200–300ms for LLM TTFT — measure engine latency, meaning the time the model itself takes to process. But if your agent orchestrator is in London and your LLM is running on servers in the US, you're adding a transatlantic round trip on top of that. Bhargava's numbers are concrete: 75ms of network latency in an already-optimized pipeline adds roughly 30% overhead. Co-locating everything in the same building drops that to around 5ms.

"Every 10 milliseconds matters," as he put it. That's not a figure of speech when your total budget is 500ms.

The implication is less obvious than it sounds. For developers building on top of frontier model APIs — OpenAI, Anthropic, others — co-location isn't really an option. The models live where the provider puts them. This creates a structural advantage for teams running open-source models on their own infrastructure, or using inference providers that let them deploy close to their orchestration layer. The voice agent market, in other words, has a geography problem baked into it.

Turn detection: the unsolved problem hiding in plain sight

Bhargava flags turn detection as "still somewhat unsolved, frankly," which is a notable admission given how central it is to basic conversational function. The challenge: when a user pauses, how does the system know whether they've finished their turn or are just thinking mid-sentence? Interrupt too early and you're that person who talks over everyone. Wait too long and the conversation feels laggy.

This is one of those problems where the solution in a controlled demo looks trivial — just set a silence threshold — and the solution in production looks much harder once you account for variable speaking styles, background noise, hedging language, and the social cost of a voice agent that repeatedly cuts people off.

It could be, as Bhargava notes, a 20-minute talk in itself.

Guardrails, and the problem of words you can't take back

A question from the audience surfaced something that deserves wider attention. A developer described introducing classifier models into their pipeline to catch unauthorized outputs — in their case, an LLM that was incorrectly offering an 8% discount it wasn't supposed to. How do you add that layer without blowing the latency budget?

Bhargava was honest: there are no easy answers. Every additional model in the chain is another latency cost and another potential failure point. Pre-LLM classifiers for routing, post-LLM guardrails before invoking TTS — both make sense architecturally, both add overhead.

The deeper issue someone in the audience named: with voice, you can't retract what's already been said. Text interfaces let you delete, edit, or quietly not send. Voice agents speak in real time, and a guardrail that catches an error after the TTS model has started producing audio is catching it too late. "You can't take back things that are spoken," one attendee observed, which is either obvious or clarifying depending on how much time you've spent thinking about it.

The thinker-talker pattern

One specific architectural pattern Bhargava described is worth naming because it's elegant. He calls it the "thinker-talker" model: a small, fast LLM handles the conversational flow — producing holding responses like "let me look into that" — while simultaneously issuing a single tool call to a larger, more capable model to actually do the complex reasoning. The larger model generates a cleaner, more authoritative response that then feeds into TTS.

The effect is that the user hears an immediate acknowledgment, not dead air, while the heavy computation happens in the background. Latency is managed on the fast path; quality is preserved on the slow path. It's a pattern borrowed loosely from how humans actually work — we buy time while we think — applied to a multi-model system.

It also illustrates a broader truth about production voice AI: the architecture is rarely the clean three-box diagram from the slides. In practice it accumulates classifiers, guardrails, secondary models, and routing logic. "We'll only add components," Bhargava noted, which sounds like a feature and functions like a warning.

The speech-to-speech horizon

The logical alternative to all of this orchestration is a single model that handles audio in and audio out — no text intermediary, no pipeline to orchestrate. OpenAI's real-time API and Nvidia's recently launched voice chat model gesture in this direction.

The appeal is real. A native speech-to-speech model preserves what gets lost in transcription: tone, hesitation, emotion. A user who sounds uncertain is processing differently than a user who sounds confident; that information evaporates the moment audio becomes text. Full-duplex communication becomes possible — the model can back-channel ("uh-huh," "I see") while still receiving audio, the way humans do.

The gap between appeal and reality, per Bhargava, is instruction-following and tool-calling. Speech-to-speech models currently struggle with both, which means they're fine for casual conversation and unreliable for any workflow where the agent needs to actually do something. "The real-world experience often looks like you'll try them and then spend a lot of time prompt engineering and hoping to fix issues," he said, "and eventually move to a pipeline architecture."

That's not a permanent verdict. These models will improve. But the gap between demo-able and production-deployable has a way of being larger than it looks from the outside, and voice is no exception.

The pipeline architecture's real cost isn't complexity — it's what it strips away. Every transcription is a lossy compression of human speech. The engineering question shaping the next few years is whether the industry can close that gap before the complexity of multi-model orchestration becomes its own ceiling.

Marcus Chen-Ramirez is a senior technology correspondent for Buzzrag covering AI, software development, and the intersection of technology and society.