Why Production Voice Agents Fail on Their First

Plivo says restructuring voice data collection around typed fields raised accuracy from roughly 30 percent to about 95 percent, without fine-tuning the underlying model.

That claim, made by founder and CEO Venky B. in an AI Engineer talk, points toward the production problem hiding behind polished voice AI demos. A convincing conversation is easy to recognize. A reliable system has to capture the correct phone number, interpret “next Wednesday at eight,” pronounce a customer’s name, tolerate an interruption and respond before silence becomes awkward.

The language model sits inside that machinery, but it does not own the whole job. Speech recognition, validation, normalization, synthesis, telephony and turn detection can each corrupt a call. Production voice AI therefore looks closer to a distributed system with an unusually impatient user than a chatbot somebody taught to speak.

The Most Valuable Output is Often a Field

Many commercial voice agents exist to collect information: an address, appointment time, account number or reason for calling. Free-form transcripts are a poor database interface because plausible text can still encode the wrong value.

Venky’s prescription is compact: “Decide your shape before you ask.”

A phone number becomes a typed field with an expected length and permitted characters. A date-time field gets a time zone, a reference date and rules for resolving ambiguity. A name can trigger spelling or confirmation. Tool calls perform the deterministic work while the model handles conversational phrasing.

This design also changes evaluation. Instead of replaying hundreds of complete calls whenever one extractor breaks, teams can test each field against accents, corrections, pauses and malformed input. The approach borrows from Pydantic models, TypeScript schemas and web forms. Voice AI has rediscovered input validation, wearing a headset.

According to the daily.dev account, Plivo identifies poor data collection as a major deployment failure and recommends typed, validated data models rather than raw transcript interpretation. That account supports the architecture described in the talk, though it does not independently validate the reported accuracy increase.

The 30-to-95-percent figure also needs boundaries. The available material does not specify the dataset, field mix, languages, sample size or definition of accuracy. Exact-match performance on ten-digit phone numbers differs from success on names and international addresses. Teams should treat the number as Plivo’s reported production experience, then reproduce the test against their own traffic.

Typed fields introduce their own policy decisions. Should E inside a phone number become 3 automatically? At high confidence, perhaps. For an account recovery call, silent correction could send sensitive information to the wrong person. Confidence thresholds, explicit read-back and human escalation belong in the schema beside the type definition.

Every Cleanup Layer Can Also Corrupt Meaning

Venky advises teams to “assume your transcriptions are going to be brittle.” The vulnerable tokens are often the ones a workflow cannot afford to lose: proper nouns, jargon, digits, street names and words moving between languages or scripts.

Code-switching makes the pipeline especially fragile. A caller may combine Hindi and English, or speak Hindi that the recognizer renders in Latin characters. The language model can continue from that representation, and the text-to-speech engine may pronounce the resulting output according to a different language’s rules. One local recognition choice propagates through the stack like a bad type cast.

Plivo proposes dynamic keyword boosting, transcript post-processing and transliteration. Dynamic boosting supplies the recognizer with likely terms only when the conversation reaches a relevant state. A clinic name might receive extra weight while the caller chooses a location, then disappear from context when the agent requests a phone number. This limits the chance that a giant keyword list starts pulling unrelated audio toward familiar terms.

An LLM can also repair a transcript using domain context. If a recognizer places a letter between digits, the model may infer the intended numeral. Yet post-processing creates a second source of error, one capable of producing cleaner and more convincing mistakes. Production systems need to retain the raw audio or transcript where policy permits, log transformations and distinguish inferred text from recognized text. Otherwise debugging becomes archaeology conducted through a beautified record.

Normalization continues on the output side. Model responses may contain markdown, emoji, abbreviations, currencies, dates or email addresses that a speech synthesizer reads badly. A separate layer can convert those structures into spoken forms and apply pronunciation dictionaries. Keeping that layer outside the model also makes switching synthesis providers easier.

Portability has a maintenance bill. Pronunciation dictionaries grow, language rules conflict, and every vendor exposes different controls. The modular architecture provides inspectable boundaries, but each boundary needs ownership, tests and an upgrade plan.

Speed Rewards Smaller Models and More Infrastructure Work

Plivo’s production target is under 550 milliseconds from the caller finishing a turn to the agent producing audio. Venky says many systems instead land between 750 milliseconds and 1.2 seconds, with slower calls prompting hang-ups. Those figures come from Plivo’s observations across customer deployments; the company says its telephony infrastructure carries more than one billion voice calls per month.

Call volume gives Plivo a broad operational vantage point, although the public material does not say how many of those billion calls involve AI agents or how the latency cohorts were measured. Plivo also sells programmable agents and a no-code agent studio, so its recommendations come from both infrastructure experience and a vendor’s product strategy.

The latency constraint collides with the direction of language-model development. As Venky puts it, “Pretty much all of your intelligence that’s been built in on the thinking layer is all off by default if you want it to be fast enough.”

A voice agent cannot pause for a long reasoning trace whenever somebody asks to reschedule an appointment. Plivo’s preferred compromise uses smaller, self-hosted models such as Gemma or Qwen, targeting an LLM response below 300 milliseconds. For multilingual systems, token fertility matters: a tokenizer that requires more tokens per word needs more generation steps before the user hears the same sentence.

This recommendation fits the broader engineering reality of voice systems, where model size, first-token latency and pipeline overhead compete inside a budget measured in fractions of a second.

Self-hosting also shifts responsibility. Teams gain control over capacity, model versions and latency variance. They inherit GPU scheduling, observability, failover, security updates and licensing review. “Open source model” can describe several licensing and governance arrangements, so deployers still need to inspect the terms attached to a model and its weights. Cheap inference on a slide can become an infrastructure rotation at 3 a.m.

A split-model architecture offers another option. A small model can produce conversational acknowledgements while a larger model performs tool selection or difficult reasoning. That preserves responsiveness, but now two models can disagree about state. The system needs a clear authority for what was promised to the caller and what action was actually executed.

Conversation is a Concurrency Problem

Turn-taking receives less attention than model quality because it looks trivial in a clean demo. Production audio includes hesitation, background speech, line noise, backchannels such as “uh-huh,” and callers who interrupt as soon as they know where an answer is going.

Voice activity detection must decide when a person has finished. Wait too long and the agent feels sluggish. Cut in early and it becomes the colleague who treats breathing as surrender. Barge-in adds another race: the system must stop generated audio, preserve any relevant state, recognize the interruption and decide whether an already-started tool call should continue.

Modular speech pipelines can implement these behaviors with explicit controls; a speech-to-speech model is not mandatory. Explicit controls improve inspectability and let teams tune interruption policy by workflow. An emergency line, restaurant booking service and collections call should not share one universal definition of an acceptable interruption.

End-to-end speech models may eventually absorb more of this coordination, reducing handoffs among transcription, language and synthesis components. They can also make failures harder to localize. A modular pipeline exposes more seams, while an integrated model hides more machinery. Neither architecture removes the need for call-level evaluation with noise, accents, interruptions and real task outcomes.

The decisive production metric may therefore be mundane: did the system store the right value, perform the authorized action and recover when speech got messy? A voice that sounds human for 30 seconds can pass a demo. A voice agent earns deployment one validated field, one interruption and one correctly pronounced name at a time.