Speech and Realtime Voice

Speech is not text with a microphone attached. It is a timed signal: words arrive inside a voice, listeners respond before every sentence is complete, and two people may speak at once. A batch transcription system can wait for the recording to end. A live captioner must revise partial hypotheses without distracting the reader. A voice assistant must also decide when to answer, begin producing audio, and keep listening while it speaks. Streaming changes what counts as correct.

This chapter develops the stack in that order. It begins with the latent alignment between audio frames and text, then explains how learned representations and neural codecs reduce the amount of audio a model must process. It closes with speech generation and the engineering contract behind a usable conversation.

Treat speech as a timed stream

A speech system first converts the waveform into a sequence of short, overlapping frames. What happens next depends on the application:

The distinctions matter. A model can have an excellent final transcript while its partial text changes too often for captions. A synthesizer can render natural speech faster than real time yet take too long to produce its first audio frame. “Realtime” therefore needs a named measurement, not a single adjective.

Learn an alignment for recognition

automatic speech recognition (ASR) observes acoustic frames (\mathbf{x}=(x_1,\ldots,x_T)) and a shorter transcript (\mathbf{y}=(y_1,\ldots,y_U)), but no label says which frames produced each symbol. CTC and RNN-T sum over latent monotonic alignments. An attention-based encoder-decoder instead learns a soft correspondence between encoder frames and output steps.

connectionist temporal classification (CTC) adds a blank symbol and assigns probability to every length-(T) path that collapses to the transcript (Graves et al. 2006):

$$p_{\mathrm{CTC}}(\mathbf{y}\mid\mathbf{x}) = \sum_{\boldsymbol{\pi}\in B^{-1}(\mathbf{y})} \prod_{t=1}^{T}p(\pi_t\mid\mathbf{x}).$$

Here (\boldsymbol{\pi}) is a path over the output alphabet plus blank, and (B) merges consecutive repeated labels and removes blanks. A forward-backward dynamic program evaluates the sum without enumerating every path. Standard CTC factorizes the path emissions across frames, conditioned on the encoder output. It has no explicit label-history model, although a contextual encoder can still use broad acoustic context and an external language model remains optional.

RNN-T adds output history (Graves 2012). Its prediction network summarizes preceding nonblank labels, and its joint network combines that state with an acoustic encoder state. In the alignment lattice, blank advances acoustic time; a label advances output position. The model may therefore emit several labels at one encoder step. A causal or bounded-context encoder makes this factorization suitable for online decoding, but RNN-T by itself does not guarantee streaming. The original model used a bidirectional transcription network, while later work demonstrated a causal on-device system (He et al. 2019).

An attention-based encoder-decoder such as Listen, Attend and Spell models

$$p(\mathbf{y}\mid\mathbf{x}) =\prod_{u=1}^{U}p(y_u\mid y_{<u},c_u), \qquad c_u=\sum_t\alpha_{u,t}h_t,$$

where (h_t) is an encoder state and (\alpha_{u,t}) is its attention weight at output step (u) (Chan et al. 2016). The original system could attend over the whole utterance, so it was offline. Monotonic, chunked, or limited-look-ahead attention can trade future context for lower delay.

These alignment objectives can sit on several encoder families. Conformer is an encoder architecture, not an alignment objective. Each Conformer block combines convolution for local acoustic patterns with self-attention for longer-range interactions (Gulati et al. 2020). Its original full-context attention was also offline; streaming implementations constrain that context.

Pretrain representations before transcribing

Transcripts are expensive, while raw audio is plentiful. Self-supervision and weak supervision solve different data problems:

wav2vec 2.0 masks spans in a latent sequence. At a masked position (t), its context vector (c_t) must distinguish the corresponding quantized target (q_t) from a set (Q_t) containing that target and distractors (Baevski et al. 2020):

$$\mathcal{L}_t=-\log \frac{\exp(\operatorname{sim}(c_t,q_t)/\kappa)} {\sum_{\widetilde q\in Q_t} \exp(\operatorname{sim}(c_t,\widetilde q)/\kappa)}.$$

Here, the similarity is cosine similarity and (\kappa) is a temperature. These targets are matching masked steps, not future continuations. In one reported low-resource setting, a large model pretrained on 53,000 hours of Libri-Light and fine-tuned on ten labeled minutes reached 4.8 and 8.2 word error rate on LibriSpeech test-clean and test-other; the headline decoder also used a Transformer language model.

HuBERT replaces contrastive selection with masked prediction of offline cluster assignments (Hsu et al. 2021). It begins with clusters of handcrafted features, then can recluster representations produced by an earlier training stage. The targets stay fixed within a stage. The paper's important finding was that consistent targets can be useful even when they are imperfect phonetic labels. WavLM keeps this masked pseudo-label objective but sometimes adds noise or another speaker to the input. Its target remains the cluster label from the original primary utterance, so it learns to identify content through interference rather than reconstructing a clean waveform (Chen et al. 2022).

Whisper follows a different recipe: 680,000 hours of weakly supervised, multilingual audio-text pairs train one encoder-decoder Transformer (Radford et al. 2023). Its decoder sequence carries a language token, a transcription-or-translation task token, controls for timestamps, and the text; when enabled, timestamp tokens are interleaved around transcript spans. The main result is broad zero-shot transfer across held-out datasets, not universal superiority to a system tuned for one domain.

Turn waveforms into codec indices

A neural codec compresses a waveform into discrete indices. An encoder maps audio to one latent vector (z_t) per frame, a quantizer replaces each vector with codebook entries, and a decoder reconstructs audio. SoundStream established an end-to-end residual-vector-quantized design with quantizer dropout for multiple bitrates (Zeghidour et al. 2021). EnCodec combined a streaming encoder-decoder, residual vector quantization, adversarial training, and optional entropy coding (Défossez et al. 2023). Their quality claims are tied to the bitrates, datasets, and listening tests reported in those papers.

Residual vector quantization (RVQ) applies codebooks in sequence. For frame (t), let (r_t^{(0)}=z_t). At depth (j), choose the nearest entry and pass on the unexplained residual:

$$k_t^{(j)}=\arg\min_k\left\lVert r_t^{(j-1)}-e_k^{(j)}\right\rVert_2^2, \qquad r_t^{(j)}=r_t^{(j-1)}-e_{k_t^{(j)}}^{(j)}, \qquad \widehat z_t=\sum_{j=1}^{Q}e_{k_t^{(j)}}^{(j)}.$$

Here (e_k^{(j)}) is entry (k) in codebook (j), (k_t^{(j)}) is its index, and (Q) is the active depth. Later codebooks refine what earlier ones missed. This usually improves reconstruction, but it also sends more indices.

If the codec emits (f) frames per second and uses (Q) codebooks, its raw index traffic is

$$R_{\mathrm{idx}}=fQ.$$

If codebook (j) contains (K_j) entries and each index uses a fixed-width integer, the nominal payload is

$$R_{\mathrm{bit}}=f\sum_{j=1}^{Q}\left\lceil\log_2K_j\right\rceil.$$

Token rate is not bitrate. Neither number alone gives a generator's decoding cost: a model may predict codebooks sequentially, in parallel, or in groups.

The following deterministic example compresses one two-dimensional frame. Each codebook has four entries, so every selected index costs two bits.

from math import dist, log2

target = (1.0, -0.5)
codebooks = [
    [(0.0, 0.0), (0.75, -0.25), (-0.75, 0.25), (1.0, 0.0)],
    [(0.0, 0.0), (0.25, -0.125), (-0.25, 0.125), (0.0, 0.25)],
    [(0.0, 0.0), (0.0, -0.125), (0.0, 0.125), (0.125, 0.0)],
]
frame_hz = 50
residual = target

for depth, codebook in enumerate(codebooks, start=1):
    index = min(range(len(codebook)), key=lambda i: dist(residual, codebook[i]))
    chosen = codebook[index]
    residual = tuple(x - q for x, q in zip(residual, chosen))
    indices_per_second = frame_hz * depth
    bits_per_second = indices_per_second * int(log2(len(codebook)))
    print(
        f"depth={depth} error={dist(residual, (0.0, 0.0)):.3f} "
        f"indices/s={indices_per_second} bits/s={bits_per_second}"
    )

“Semantic” and “acoustic” are roles, not a clean partition. Tokens from a self-supervised speech model tend to preserve linguistic content while discarding some speaker and recording detail. Codec tokens must preserve enough information to reconstruct the signal, so they also carry content. Systems can encourage a division: SpeechTokenizer distills its first RVQ layer toward HuBERT units (Zhang et al. 2024), while later layers refine the audio. AudioLM uses w2v-BERT units for long-range structure and SoundStream codes for fidelity (Chung et al. 2021; Borsos et al. 2023). The Descript Audio Codec targets high-fidelity 44.1 kHz audio with improved quantizer use (Kumar et al. 2023). Mimi, used by Moshi, operates at 12.5 frames per second and combines a semantic quantizer with acoustic RVQ levels (Défossez et al. 2024). A low frame rate shortens the time axis, but quality, codebook depth, and the generator's schedule still determine the actual cost.

Choose a generation representation and schedule

AudioLM first models semantic units, then coarse and fine acoustic tokens. This hierarchy gave its samples longer-range coherence without asking one flat codec model to learn every timescale at once (Borsos et al. 2023). text-to-speech (TTS), the task of synthesizing speech from text, adds a transcript and often a short acoustic prompt that identifies the target voice.

VALL-E predicts EnCodec tokens conditioned on text and a three-second enrollment recording. Its autoregressive stage produces the first codebook, and its non-autoregressive stage fills residual codebooks one level at a time, with frame positions parallel within a level (Wang et al. 2023). SoundStorm uses confidence- based masked decoding across positions; in the authors' TPU-v4 setup, it generated thirty seconds of audio in half a second (Borsos et al. 2023). VALL-E 2 changed codec grouping and sampling to reduce repetition. Its “human parity” claim refers to the paper's preference tests on LibriSpeech and VCTK, not to every speaker, language, or recording condition (Chen et al. 2024).

Continuous generators offer another path. Voicebox uses conditional flow matching to infill masked mel-spectrogram spans, enabling synthesis, editing, denoising, and style transfer in one model (Le et al. 2023). NaturalSpeech 3 instead factorizes speech into content, prosody, speaker identity, and acoustic detail, then uses discrete masked diffusion for the generated factors (Ju et al. 2024). F5-TTS removes an explicit alignment search and duration predictor from training, but inference still needs a target duration to define the generated span (Chen et al. 2024).

None of these rows guarantees or forbids streaming. Chunking, causal context, look-ahead, and the vocoder determine when audio can leave the system.

Voice cloning also makes consent and provenance part of the technical design. AudioSeal jointly trains a watermark generator and a detector that can localize its embedded signal (San Roman et al. 2024). It is proactive rather than a universal detector: it can recognize content carrying that watermark, not prove that any unmarked recording is human-made.

Build conversation on incremental state

A turn-based cascade connects ASR, a text model, and TTS. Its components are easy to inspect and replace, and text offers a useful control boundary. It can also lose timing, overlap, pronunciation, and emotion unless those signals travel on separate channels. Direct speech-to-speech models preserve more of the signal but are harder to steer and diagnose.

For a purely serial cascade, time to first audio can be decomposed as

$$D_{\mathrm{first}}=D_{\mathrm{endpoint}}+D_{\mathrm{in}}+D_{\mathrm{ASR}} +D_{\mathrm{model}}+D_{\mathrm{TTS}}+D_{\mathrm{out}}.$$

The terms cover endpoint detection, inbound transport, recognition, response generation, synthesis, and outbound transport. Real systems overlap work, so the observed delay follows the critical path rather than always equaling this serial sum. Report time to first audio together with the endpoint rule and network conditions. Real-time factor, total generation time divided by audio duration, answers a different question.

Full duplex is a systems contract: capture and input processing continue while reply audio plays. That contract needs acoustic echo cancellation so the agent does not hear itself, voice-activity and endpoint logic, interruption detection, output cancellation, and a policy for rolling back text or tool actions produced after the user barges in. Backchannels such as “mm-hm” are outputs too; they must not accidentally claim a turn or trigger an action.

Moshi demonstrates one model-level implementation. It represents user and system audio as parallel streams and predicts a text “inner monologue” before the system audio at each step. The paper reports 160 ms theoretical latency and about 200 ms in its practical setup (Défossez et al. 2024). These figures describe that system, not a universal conversational threshold.

Human conversation is not a 200 ms service-level objective. Across ten languages, Stivers and colleagues observed response-time distributions with modes from 0 to 200 ms and median offsets from 0 to 300 ms, with substantial variation by language and response type (Stivers et al. 2009). People also use pauses as meaningful signals. A voice product should measure its own task: dictation, translation, customer support, tutoring, and casual conversation tolerate different delays.

End-to-end audio products provide useful system evidence but limited architectural detail. OpenAI reported 232 ms minimum and 320 ms average response time for GPT-4o audio, and described one network trained across audio, vision, and text (OpenAI 2024). That does not by itself establish full-duplex behavior. Speech translation is another neighboring workload: SeamlessM4T covers speech and text translation and recognition across up to one hundred languages, with coverage varying by task and direction (Seamless Communication 2023). Seamless Streaming uses monotonic attention for simultaneous translation; the same work also presents an expressive speech-to-speech model (Seamless Communication 2023).

Benchmark every boundary

An end-to-end score cannot explain which layer failed. Keep boundary metrics beside task-level listening tests:

For a reference transcript containing (N) words, word error rate is

$$\mathrm{WER}=\frac{S+D+I}{N},$$

where (S), (D), and (I) count substitutions, deletions, and insertions. WER does not measure punctuation, speaker attribution, prosody, naturalness, or whether a partial transcript arrived in time. Those need separate tests.

Further reading