GPT-Realtime-2: Voice AI That Actually Does Things

There's a version of voice AI that everyone is tired of—the kind where you say a thing, it says a thing back, and you both pretend that's useful. OpenAI's pitch with GPT-Realtime-2, which dropped last week and got a full live demo treatment in their Build Hour session, is that this model does something structurally different: it doesn't just respond, it operates.

That framing is doing a lot of work. Let's look at what's actually underneath it.

Three models, one drop

The release was actually a bundle of three models, each targeting a different layer of the voice stack:

GPT-Realtime-Whisper handles streaming speech-to-text with tunable latency as low as 200ms across 80 input languages. The pitch here is speed—faster transcription means earlier function calling, which means your app feels less like it's waiting and more like it's thinking. Good for captions, meeting notes, ambient context.

GPT-Realtime-Translate does live speech translation across 70+ input languages and 13 output languages. In the session, Teri Yu (multimodal API PM at OpenAI) demonstrated it live by speaking about breakfast foods and watching Spanish translation appear in real time. Someone in the chat confirmed it worked. The model also includes dynamic voice cloning with multi-speaker differentiation—so in a translated call, you can hear that two different people are speaking, not just one unified voice.

GPT-Realtime-2 is the main event. OpenAI describes it as bringing "GPT-4.5 class reasoning into voice," with particular strength in prompt adherence, tool calling, and multilingual performance. The headline upgrades: a 4x context window expansion to 128K tokens (roughly one hour of conversation), parallel tool calls instead of sequential ones, better domain vocabulary for specialized fields like healthcare, and what they're calling "preambles"—the model can say something like "let me check on that" before reasoning, which lands more naturally in conversation.

Controllable expressiveness is in there too. You can apparently tell it to whisper, be excited, or sound jealous. Whether that's a feature or a party trick probably depends on your use case.

The demo that actually made me pay attention

Solutions engineer Erika Kettleson ran two live demos, and the second one is the more interesting story.

The first—a voice shopping assistant for a fictional outdoor gear company called Supply Co—was impressive on its own terms. She asked the assistant to recall her shopping list, filter tents by price and capacity, read low-star reviews aloud, look up the Seattle weather forecast for a specific weekend, and add items to her cart. The model called external tools mid-conversation without breaking flow, chained multiple actions together, and remembered context across the entire interaction. It was handling 15-20 tools simultaneously, something Kettleson noted would have been unreliable with previous realtime models.

But the second demo flipped the perspective entirely. Same fictional company, except now she's the product manager, not the shopper. She's staring at an analytics dashboard showing a traffic drop in Europe, and instead of clicking through filters herself, she's just... talking to it.

"Let's filter by Europe. Look at the last seven days and compare to the seven days prior." The dashboard updates. "Can you point out other relevant filters?" It highlights voice search, first-time shoppers, and footwear. "Kick off the root cause investigation."

When she finally asked for a verbal summary to share with her engineering team, the model delivered this: "The investigation shows a mobile Safari-specific regression where the product detail page size selector validation doesn't update correctly. So first-time Europe footwear shoppers get stuck after choosing a size and can't add to cart. Chrome is near baseline, which points to the PDP release behavior on Safari rather than a broad traffic quality or search issue."

That's not a voice assistant. That's a colleague who read the data and knows how to talk to engineers.

The key design choice Kettleson emphasized: the model only spoke when asked. It took actions silently, updated the UI, and waited. "The model's good enough at instruction following that it only talked when I asked it to. I didn't need it to confirm with me every time it did a filter." This "voice-to-action" pattern—where the model operates the interface rather than narrating at you—is what she's calling a more durable workflow than the old speech-in, single-action-out approach.

What Sierra adds to the picture

The back half of the session featured Ken Murphy from Sierra, which builds customer service AI agents for Fortune 100 companies. His framing of the problem is worth sitting with:

"The challenge is not just can we make the agent sound natural, it's whether we can build, evaluate, constrain and operate agents that businesses trust to represent them directly with their customers."

Sierra is excited about GPT-Realtime-2 for its latency and reasoning improvements, but Murphy was clear that the model is one component in a larger harness Sierra built themselves. That harness handles guardrails, customer-specific workflow definitions, tool permissions, tracing, and—critically—their own custom-tuned VAD (voice activity detection) models. VAD is what decides when you've finished speaking, and apparently the off-the-shelf solution wasn't cutting it for production. A half-second pause in the wrong place "can feel awkward or broken," Murphy noted, and enterprise customers don't have patience for broken.

The Sierra demo featured an AI mortgage loan assistant named Jade—warm, context-aware, pulling in information from a previous conversation ("Last time we chatted in March, you were eyeing Noe Valley"). Jade knew when to hand off to a human banker and said so explicitly. It's a convincing artifact of where this is heading: AI that sounds natural enough that the friction point stops being how it sounds and starts being what it's allowed to do.

The honest questions

None of this is covered in the session, which is a developer-facing demo event—not a press conference—so that's fair. But they're worth raising.

The 128K context window is genuinely significant, but it also means the model is holding a lot more information about you across a conversation. Voice interactions are inherently more intimate than text—people say things differently when they're talking than when they're typing. What gets logged, retained, or used for model improvement from these extended voice sessions is a question that matters more than it might seem.

The "controllable expressiveness" feature—where you can make the model sound jealous or excited—sounds playful in a demo. In a customer service context where you're trying to engineer emotional responses in callers, it becomes something a bit more worth scrutinizing.

And the Sierra data point about 0.1% error rates becoming "real business risk" is honest in a way that's also quietly alarming. At enterprise scale, 0.1% is not a rounding error. The infrastructure Sierra built around GPT-Realtime-2—the harnesses, the evals, the custom VAD—is essentially a safety and reliability layer that OpenAI isn't providing natively. Which means the capability is accessible to anyone via the API, but the production-readiness is not. Small developers building on the raw API without that surrounding infrastructure are working with something that needs more than just good prompts to behave reliably.

What's actually new here

The voice-to-action pattern is the thing I keep coming back to. Not voice-to-voice, not the translation, not even the context window bump—but the idea that a voice model can sit quietly inside a product, take actions when asked, and only speak when it has something worth saying.

That's a different interface paradigm than the assistant-that-talks-back. Whether it's better probably depends on the context. But it's a more interesting design question than "does it sound natural?" which is largely solved at this point.

The next version of that question is: whose voice should it have, whose interests should it serve, and who gets to tune those parameters?

---

Yuki Okonkwo is Buzzrag's AI & Machine Learning Correspondent. She's self-taught in ML and mad about it in the best way.