Speech and Realtime Voice

The previous chapters loosened generation from a left-to-right text order. Speech adds a different pressure: time. Audio has to be recognized, represented, generated, and sometimes answered while the other person is still speaking. A live voice interface has only a few hundred milliseconds before the reply feels late, and that single budget shapes the whole stack, from recognizer alignment to codec frame rate to whether the agent is a cascade or one full-duplex model. The same budget runs through the chapter's other choices: self-supervision and weak supervision lower the cost of labeled audio, neural codecs split early semantic tokens from later fidelity tokens, a few seconds of reference audio can clone a voice, and synthesis now divides between autoregressive and flow-matching routes.

Recognizing speech

automatic speech recognition (ASR) maps a long, unsegmented waveform, hundreds of frames, to a short transcript, with no per-frame label given. Training needs an alignment, and the field's three answers differ in how they supply it. connectionist temporal classification (CTC) introduces a blank token and sums the probability over every frame-level path that collapses to the target, computed by a forward-backward dynamic program (an efficient algorithm that sums over all alignments between audio frames and output symbols), but it assumes the outputs are conditionally independent given the audio, so it carries no internal language model and leans on an external one (Graves et al. 2006). To restore the dependency between outputs, the RNN transducer adds a prediction network, an autoregressive label model, and a joint network; the alignment stays monotonic and frame-synchronous, so it emits as frames arrive and is naturally streaming, which is why an all-neural transducer became the standard for on-device recognition (Graves 2012; He et al. 2019). The attention-based encoder-decoder learns the alignment implicitly through attention and assumes no independence, but its soft attention spans the whole utterance, so it is not naturally streaming (Chan et al. 2016). The encoder these losses sit on is the Conformer, which interleaves self-attention for global context with a convolution module for local acoustic detail (Gulati et al. 2020).

The expensive part was always labeled audio, and two ideas drove that cost down. Self-supervision came first, and it has a lineage. wav2vec 2.0 masks spans of a latent audio sequence and solves a contrastive task (training the model to tell a true continuation apart from distractor samples, which forces it to learn useful structure without labels) whose target is the quantized version of each masked step, reaching strong recognition from as little as ten minutes of labels on top of unlabeled pretraining (Baevski et al. 2020). In HuBERT the contrastive objective gives way to BERT-style masked prediction of offline k-means cluster targets (discrete labels made by clustering the audio features, used as stand-in training targets), the cluster labels refreshed by re-clustering on the model's own improving features, which decouples the target from the model and stabilizes training (Hsu et al. 2021). WavLM kept that objective but mixed in noise and overlapping speakers and trained the model to recover the clean primary speaker, which is what makes its representations strong on speaker and multi-speaker tasks, not just transcription (Chen et al. 2022). Scale came next: Whisper is an encoder-decoder transformer trained on 680,000 hours of weakly-supervised audio, and it folds language identification, transcription, translation, and timestamping into one model through a prefix of special tokens the decoder is conditioned on (Radford et al. 2023). Its contribution is zero-shot robustness (holding up on accents, noise, and domains it was never fine-tuned on) rather than the lowest number on any single benchmark, where a model fine-tuned in-domain can still beat it. The arc mirrors the text side of the book, self-supervised pretraining then large weakly-supervised scaling.

Audio as tokens

The move that folded audio into the rest of this book was to stop treating it as a continuous signal and start treating it as discrete tokens. A neural codec does this: an encoder downsamples a waveform to a latent at some frame rate, a residual vector quantizer maps it to a short sequence of codebook indices, and a decoder reconstructs it, all trained end to end with reconstruction and adversarial losses. SoundStream at 3 kbps outperforms the traditional Opus codec at 12 kbps and uses quantizer dropout so one model serves many bitrates (Zeghidour et al. 2021), and EnCodec pushed fidelity further with a streaming encoder-decoder and an entropy model over the codes (Défossez et al. 2023). Residual quantization is the key mechanism: each quantizer codes the leftover the previous ones missed, so depth trades bitrate against reconstruction error.

import numpy as np
rng = np.random.default_rng(0)

d, K, Q = 16, 256, 6            # frame dim, codebook size, RVQ depth (num quantizers)
codebooks = [0.5 * rng.normal(size=(K, d)) for _ in range(Q)]
x = rng.normal(size=d)          # one audio-frame embedding to compress

residual = x.copy(); tokens = []
for cb in codebooks:
    j = int(np.argmin(((cb - residual) ** 2).sum(1)))   # nearest entry to residual
    tokens.append(j)
    residual = residual - cb[j]                          # quantize the leftover
    print(f"{len(tokens)} quantizer(s) -> {len(tokens)} token(s), "
          f"recon error = {np.linalg.norm(residual):.3f}")
print("each extra quantizer codes the leftover residual: error falls, bitrate rises")

Not all audio tokens carry the same thing, and the distinction governs the rest of the chapter. Semantic tokens come from a self-supervised model like HuBERT or w2v-BERT (Chung et al. 2021) and capture linguistic content and long-range structure but discard fine acoustic detail; acoustic tokens come from a codec and capture speaker identity, room, and fidelity but lack long-range structure. The frontier folds the two together so a downstream language model gets meaning in its earliest tokens: SpeechTokenizer distills the first residual quantizer to match HuBERT's semantic tokens while later quantizers carry acoustics (Zhang et al. 2024), the Descript codec fixes codebook collapse to push universal 44.1 kHz audio to about ninety-fold compression (Kumar et al. 2023), and the Mimi codec inside Moshi runs at a 12.5 Hz frame rate with a split residual quantizer whose first token is semantic, distilled from WavLM (Défossez et al. 2024). The frame rate is the lever. For an autoregressive model the sequence length is the frame rate times the duration times the number of codebooks, so EnCodec's 75 Hz is hundreds of tokens per second while Mimi's 12.5 Hz is roughly six times shorter, which is the difference between a speech model that can run in real time and one that cannot. The tension is that fewer tokens cost reconstruction fidelity, so each codec spends its engineering on getting more quality per token.

Generating speech

With audio as tokens, generation becomes language modeling, but a flat model over raw codec tokens wanders semantically, because codec tokens are optimized for fidelity, not content. AudioLM fixed this with a hierarchy: model semantic tokens first to lock what is being said, then coarse acoustic tokens for speaker and prosody, then fine acoustic tokens for high-frequency detail, producing coherent speech and even piano continuations with no transcript (Borsos et al. 2023). text-to-speech (TTS), the task of synthesizing speech from text, followed the same turn from signal regression to token prediction. VALL-E treats TTS as conditional codec language modeling with a now-standard split: an autoregressive model predicts the first, coarsest codebook token by token, where duration and prosody live, and a non-autoregressive model fills the remaining residual codebooks in parallel; given a three-second enrollment recording of an unseen speaker it clones the voice zero-shot, carrying over the speaker's emotion and acoustic environment because the prompt is itself codec tokens (Wang et al. 2023). SoundStorm then made the acoustic stage fast with confidence-based parallel decoding, generating thirty seconds of audio in half a second (Borsos et al. 2023), and VALL-E 2 reached reported human parity by stabilizing the sampling that otherwise makes autoregressive codec models loop (Chen et al. 2024).

A parallel lineage rejects the autoregressive codec model for non-autoregressive continuous generation, the flow matching of Chapter 12 applied to speech. Voicebox frames synthesis as text-guided audio infilling and trains a flow-matching model that attends to both past and future context, reporting a lower word error rate (the fraction of words wrong, lower is better) and roughly twenty-fold speedups over VALL-E (Le et al. 2023), NaturalSpeech 3 factorizes speech into content, prosody, timbre, and detail and diffuses each (Ju et al. 2024), and the F5-TTS line strips the recipe to a bare flow-matching model with no duration predictor or alignment search (Chen et al. 2024). The split is the chapter's recurring one: the autoregressive codec model streams and samples diverse prosody but is prone to looping, while the flow or diffusion model is robust, fast, and bidirectional but needs the whole span up front, which makes it harder to stream. The same three-second cloning that makes this useful makes it a misuse vector, which is why audio watermarking arrived alongside it; AudioSeal trains a detector that localizes which samples of a clip are AI-generated in a single fast pass (San Roman et al. 2024).

Realtime, full-duplex voice

The hard part of a voice agent is not recognition or synthesis but the conversation loop: turn-taking, interruption, and latency. Human turn-taking gaps average around two hundred milliseconds across languages, and that ceiling, not any accuracy target, is the governing constraint. The obvious design is a cascade, automatic speech recognition into a text model into text-to-speech. It is modular, debuggable, and rides text-model progress, but it stacks each stage's latency serially and flattens everything through a text bottleneck that discards timing, overlap, and prosody. The alternative models speech to speech directly.

Moshi is a full-duplex speech-text model that removes explicit speaker turns. It runs on the low-frame-rate Mimi codec, models its own audio and the user's as two parallel streams so it is always listening while it speaks, and predicts a text "inner monologue" before each step's audio tokens, which improves linguistic quality and yields streaming recognition and synthesis for free; it reports about 160 ms theoretical and 200 ms practical latency (Défossez et al. 2024). The parallel-stream design is what makes interruption and backchannels native rather than a special case the way a half-duplex, silence-detecting agent must treat them. The first commercial instance was GPT-4o's voice mode in 2024, a single model trained end to end across text, vision, and audio that replaced the old three-model voice pipeline and reported an average 320 ms response (OpenAI 2024). Dedicated realtime speech-to-speech models then made this a standard product tier: OpenAI's gpt-realtime shipped with the Realtime API's general availability in August 2025 (OpenAI 2025), Google's Gemini Live added native-audio dialogue, and both architectures remain undisclosed. A parallel line unifies the tasks rather than the duplex loop: SeamlessM4T is one model for speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation plus recognition across roughly a hundred languages (Seamless Communication 2023), and its streaming successor adds low-latency simultaneous translation that preserves vocal style (Seamless Communication 2023).

Further reading