Why Modern TTS Models Are Built Like LLMs

There's a question buried inside every voice agent interaction that feels smooth: how is it that the system starts talking before it's finished thinking? The answer turns out to be less mysterious than it sounds, and more architecturally interesting than most coverage suggests.

At the AI Engineer conference, Samuel Humeau—AI scientist at Mistral and former Facebook FAIR researcher—walked through why the text-to-speech field has largely converged on a single dominant pattern: autoregressive transformers generating audio token by token, in a structure that looks a lot like the LLMs powering the chat agents they're meant to serve. The occasion was the release of Mistral's first open-weight TTS model, which Humeau used as a running example throughout.

The talk is worth unpacking not because Mistral's model is necessarily the best one on the market—that's a separate conversation—but because Humeau is unusually clear about why the field ended up here, what the actual engineering constraints are, and where the honest uncertainty lives.

The information-density problem nobody talks about

Start with the fundamental mismatch that makes audio hard to model with a transformer.

Text is cheap. A human speaker—even a fast one—delivers maybe 15 bits per second of semantic content. A standard-quality MP3, by contrast, runs at around 200 kilobits per second. The gap between those numbers is not a rounding error; it's roughly four orders of magnitude. You cannot feed raw audio waveforms into a transformer and expect it to do anything useful. The sequence would be impossibly long, and the computation would be impossibly expensive.

The solution the field has landed on is neural audio codecs: learned compression systems that reduce audio to a manageable token rate while preserving the acoustic features that matter—voice identity, prosody, accent, the qualities that make a voice a voice rather than just a transcript. In Mistral's implementation, audio is chunked into 80-millisecond frames (12 frames per second), and each frame is encoded into 37 tokens. That brings the representation down to roughly 500 tokens per second—still substantial, but workable.

"One token of a vocabulary of a thousand, it's 10 bits of information," Humeau explained, "and the audio requires much much more." The codec isn't just compression for efficiency's sake; it's the bridge that makes the LLM-style architecture possible at all.

The comparison Humeau draws to text captioning is clarifying in a slightly uncomfortable way. A subtitle track drops nearly all the acoustic information and retains almost none of the bitrate. What a good audio codec tries to do is land somewhere between "full waveform" and "subtitle"—enough compression to be tractable, enough fidelity to reconstruct identity and tone. Whether any current codec actually hits that target for all use cases is a question the field is still working out.

Why autoregressive, and why now

The architecture story is straightforward once you accept the codec framing. If you've compressed audio into a token sequence, you now have something that looks a lot like a text sequence—and the transformer machinery the industry has spent years optimizing for text sequences becomes directly applicable.

So the dominant TTS pipeline in 2026 runs something like this: a large autoregressive transformer backbone (Mistral's is 4 billion parameters) processes the input context—reference audio of the target voice plus the text to synthesize—and generates audio tokens frame by frame. A separate smaller model, running at each frame step, reconstructs the full token set for that frame. A decoder converts tokens back to audio. The first audio packets come off the line well before the full generation is complete, which is where the perceived-latency trick lives.

Humeau clocked Mistral's model at 17 milliseconds to first audio on a single GPU, excluding network overhead. That number matters because in a voice agent pipeline, it determines whether the conversation feels like a phone call or a buffering spinner.

One thing Humeau flagged without overselling: Mistral's model departs from the "vanilla" pattern at the frame-reconstruction step. Instead of a standard decoder transformer for the 37 tokens per frame, they use a diffusion model—specifically a flow matching approach. He didn't drill into the technical details on stage, but pointed to the technical report for anyone wanting to follow the thread. The departure is notable mainly because it suggests the "dominant pattern" isn't fully locked in, even among labs that broadly subscribe to it.

Voice cloning: what's open and what isn't

The demo sequence Humeau ran on stage showed the model cloning a voice from a short reference recording, then synthesizing new text in that voice—including cross-language transfer, where a French speaker's voice (accent intact) generated an English sentence. The accent preservation was a deliberate design choice, not an artifact.

He also cloned his own voice live, which landed the way those demos tend to land: impressive, slightly uncanny, and worth pausing on.

"It's becoming so easy to impersonate a voice that it's becoming very easy to configure," Humeau noted—using "configure" in a sense that I'd translate as "misuse." He framed this as a brief digression before moving on, but it's probably worth more than a digression. Voice cloning from a few seconds of audio, running on open weights, is not a theoretical capability anymore. It's a conference demo.

The model weights are open. The voice cloning encoder is not—Mistral is serving that capability as a proprietary API for now. When asked why, Humeau was direct: "We just didn't want to give everybody the ability to clone any voice."

That's a reasonable line to draw, and it's notable that they drew it explicitly rather than pretending the capability doesn't have obvious misuse potential. But it's also worth noting that the line is architectural, not fundamental. The encoder withheld today is the kind of component that gets independently replicated, fine-tuned from other open models, or leaked. Anyone building threat models around voice authentication or deepfake detection should not be banking on this particular gate holding.

The secondary observation Humeau made—that companies will increasingly treat voice identity the way they currently treat visual brand identity—strikes me as both correct and underappreciated. If a few seconds of audio is enough to clone a voice with high fidelity, the concept of "your brand's voice" takes on a much more fraught meaning.

The unsolved problem: streaming text input

The part of Humeau's talk I found most interesting was the section he almost ran out of time to cover: what happens when the text input isn't a fixed block but a stream of tokens arriving in real time from an upstream LLM.

In a voice agent pipeline, the current pattern is: LLM generates full response text, TTS converts that text to audio, audio streams out. The latency savings come from streaming the audio output before the full generation finishes. But there's a further optimization available: if TTS can begin generating audio from the first tokens of the LLM output, before the LLM has finished generating the full response, you compress latency from two sequential processes into something closer to parallel.

The catch is that nobody has clearly won on the architecture for this. Humeau outlined two approaches: interleaving audio and text tokens in the same stream (so new text tokens get woven into the ongoing audio generation), and dual-stream architectures where audio and text run in parallel and get blended during inference. He was candid about where Mistral stands: "We actually don't know which one we'll choose."

That honesty is more useful than a confident answer would be. The streaming-text-input problem is where the next meaningful latency gains in agent pipelines will come from, and the field is genuinely still sorting it out. Labs that claim to have this solved cleanly deserve scrutiny.

The demo Humeau ran—a voice agent named "Paul" answering conference schedule questions—worked well, but the utterances were short enough that the distinction between "wait for full text then speak" and "speak as tokens arrive" wasn't really visible. For a full-paragraph response, the difference would be noticeable. That's the use case the next architecture generation is being built for.

What the field hasn't fully reckoned with is whether optimizing for lower and lower perceived latency eventually hits a wall that isn't engineering—it's human. There's a reason a thoughtful pause before a complex answer reads as competence rather than buffering. The most interesting voice agents of the next few years might be the ones that figure out when not to start talking immediately.

---

Rachel "Rach" Kovacs is Buzzrag's cybersecurity and privacy correspondent.