Quantization and Kernels

The previous chapter reduced the number of target-model cycles needed for a token. This chapter asks what each remaining cycle costs. Quantization reduces the bytes used to represent weights, activations, or the key-value cache. IO-aware kernels reduce the bytes moved through the memory hierarchy. These ideas complement each other, but neither guarantees faster service by itself. A compressed artifact needs a kernel that can consume its exact layout, and the runtime must dispatch that kernel for the shapes produced by the workload.

That distinction is the chapter's organizing rule: keep the numeric contract, the kernel implementation, and the serving result separate. It explains why the same four-bit model can be fast in one engine and slow in another, why an FP8 path can help prefill more than decode, and why a smaller file may save capacity without improving latency.

Begin with the bytes

Let a model contain $P$ stored parameters. If the average stored weight uses $b_w$ bits, its weight footprint is approximately

$$M_{\mathrm{weights}} \approx P\frac{b_w}{8}+M_{\mathrm{meta}}.$$

Here $M_{\mathrm{weights}}$ is the number of stored bytes, $P$ is the parameter count, $b_w$ is the average number of data bits per parameter, and $M_{\mathrm{meta}}$ includes scales, zero points, padding, tensor headers, and any weights retained at another precision. Thus a 70-billion-parameter model stored entirely in FP16 needs about 140 GB for weights in decimal units. A nominal four-bit representation suggests 35 GB of weight data, but its real artifact is larger because metadata and mixed-precision tensors are not free.

Capacity is only the first constraint. During small-batch decode, many linear layers reuse each loaded weight too little to saturate the accelerator's arithmetic units. If a step reads $R_{\mathrm{step}}$ bytes from device memory and the achieved high-bandwidth-memory rate is $B_{\mathrm{HBM}}$, then a bandwidth lower bound is

$$t_{\mathrm{decode}}\gtrsim\frac{R_{\mathrm{step}}}{B_{\mathrm{HBM}}}.$$

Here $t_{\mathrm{decode}}$ is time for one decode step, $R_{\mathrm{step}}$ is the traffic that reaches high-bandwidth memory, and $B_{\mathrm{HBM}}$ is measured bandwidth for this kernel and workload, not the device's advertised peak. Weight quantization can lower both the resident footprint and $R_{\mathrm{step}}$. It lowers latency only if the executed kernel reads the compressed representation efficiently. A capacity saving is not automatically a latency saving.

The same point can be written as a roofline bound. If an operation performs $F$ floating-point operations while transferring $D$ bytes through device memory, its arithmetic intensity is $I=F/D$, and

$$t\gtrsim\max\!\left(\frac{F}{P_{\mathrm{kernel}}}, \frac{D}{B_{\mathrm{HBM}}}\right).$$

Here $P_{\mathrm{kernel}}$ is the measured or supported compute rate for the exact operand and accumulator types. Batch size can increase weight reuse and therefore $I$; a long decode can increase KV traffic; a mixture-of-experts model reads only its active experts; and distributed execution can instead be limited by communication. “Memory-bound decode” is a common regime, not a law.

Attention has a related, but shape-dependent, traffic problem. For query and key lengths $n_q$ and $n_k$, the score tensor for one head is

$$S\in\mathbb R^{n_q\times n_k},\qquad S=\frac{QK^\mathsf T}{\sqrt d}.$$

Here $Q\in\mathbb R^{n_q\times d}$ contains queries, $K\in\mathbb R^{n_k\times d}$ contains keys, and $d$ is the head dimension. During full-sequence prefill or training, $n_q$ and $n_k$ can both equal the sequence length, so materializing $S$ creates a quadratic intermediate. During ordinary decode, one new query attends to the retained keys, so $n_q$ is often one while $n_k$ grows with context. Decode therefore does not create a new full $n\times n$ score matrix at every decode step. It still benefits from kernels that fuse attention operations and avoid unnecessary reads, writes, and launches.

These two cost models point to different questions:

1. Which tensors occupy capacity and cross the memory interface?
2. Which numeric representation preserves the required quality?
3. Does the target hardware have a kernel for that representation and shape?

Quantization is a numeric contract

Integer quantization replaces real values with integer codes and the metadata needed to reconstruct approximations. For an element $x_i$ assigned to scale group $g$, a general affine rule is

$$q_i=\operatorname{clip}\!\left( \operatorname{round}\!\left(\frac{x_i}{s_g}\right)+z_g, q_{\min},q_{\max} \right), \qquad \hat{x}_i=s_g(q_i-z_g).$$

Every symbol in these equations belongs to the stored numeric contract:

• $x_i$ is the original real value and $\hat{x}_i$ is its reconstruction.
• $q_i$ is the stored integer code, bounded by $q_{\min}$ and $q_{\max}$.
• $s_g>0$ is the scale for group $g$, and $z_g$ is its integer zero point.
• $g$ may identify the whole tensor, one channel, or a small group of adjacent values. The choice is called the quantization granularity.
• $\operatorname{round}$ maps to an integer, and $\operatorname{clip}$ limits that integer to the representable range.

If $x_i$ lies inside the represented range and rounding is to the nearest code, then

$$|x_i-\hat{x}_i|\le\frac{s_g}{2}.$$

Clipping adds error beyond this bound. A finer granularity usually reduces the range within each group and therefore its step size, but it adds metadata and may require a more specialized kernel. The true storage cost includes metadata and padding, so “four bit” is a family of layouts rather than a complete format description.

Scale granularity also changes metadata overhead. Let $b$ be the data bits per value, $g$ the number of values in a group, $b_s$ the scale width, and $b_z$ the zero-point width. If each group stores one scale and one zero point, the ideal effective payload is

$$b_{\mathrm{eff}}=b+\frac{b_s+b_z}{g}$$

bits per value, before padding and higher-precision tensors. Four-bit values in groups of 128 with one FP16 scale and no stored zero point use 4.125 ideal bits per value, so the payload reduction from FP16 is about 3.88 times rather than exactly four. An all-zero symmetric group needs an explicit convention, such as storing $s_g=1$ and setting every code to zero, to avoid division by zero.

Symmetric quantization is the common special case $z_g=0$. A scale can be chosen from the largest absolute value in a group, but production recipes may optimize the scale or clipping threshold against calibration data instead. Floating-point formats use sign, exponent, and significand fields rather than a uniform integer grid. They provide a wider dynamic range, while block-scaled formats add a shared scale to a small block. Each choice changes both numerical error and what hardware can execute efficiently.

Outliers make granularity matter

Transformer activations in several studied model families contain channels whose magnitudes are much larger than the rest (Dettmers et al. 2022). With one scale for a whole tensor, an outlier widens the represented range and makes the grid coarse for ordinary values. Per-channel or per-group scales isolate that range, but at a metadata and kernel cost. This activation behavior motivated methods such as SmoothQuant, while weight-only methods use calibration activations to decide which weight errors matter (Xiao et al. 2023; Lin et al. 2024).

The runnable uses a deliberately small, synthetic vector. It is not a measured activation distribution. It shows only how a shared symmetric scale reacts to one large value.

def quantize(values, bits=4):
    qmax = 2 ** (bits - 1) - 1
    maximum = max(abs(v) for v in values)
    scale = maximum / qmax if maximum else 1.0
    codes = [max(-qmax, min(qmax, round(v / scale))) for v in values]
    return [scale * q for q in codes], scale

bulk = [0.7, -0.4, 0.9, -0.8, 0.3, -0.6, 0.5, -0.2, 0.75, -0.5, 0.6]
cases = [("no outlier", bulk), ("one outlier", [7.0] + bulk)]

for name, values in cases:
    reconstructed, scale = quantize(values)
    original_bulk = values if name == "no outlier" else values[1:]
    quantized_bulk = reconstructed if name == "no outlier" else reconstructed[1:]
    error = sum(abs(a - b) for a, b in zip(original_bulk, quantized_bulk)) / len(bulk)
    print(f"{name:12s} scale={scale:.3f}  bulk mean abs error={error:.3f}")

Weights, activations, and KV state are different targets

Notation such as W4A16 means that weights use roughly four data bits and activations use a 16-bit format. It does not specify group size, scale type, zero point, clipping rule, accumulator precision, or kernel layout. Those details are part of the numeric contract.

Weight-only quantization is often attractive for bandwidth-limited, small-batch decode. The kernel reads packed weights, dequantizes after loading compressed blocks, and performs the multiply with higher-precision activations and accumulators. Weight-and-activation quantization can use native low-precision matrix multiplication, which can matter more for large prefills or high-throughput batches. KV-cache quantization changes the state accounting from Chapter 32 and the inputs read by attention. These are three separate choices, not progressively stronger versions of one switch.

Here $L$ is the number of layers, $n_{\mathrm{kv}}$ the number of KV heads, $d_{\mathrm{head}}$ the head dimension, and $b_{\mathrm{kv}}$ the stored bytes per key or value element. The logical KV payload per retained token is

$$\kappa(b_{\mathrm{kv}})= 2L\,n_{\mathrm{kv}}\,d_{\mathrm{head}}\,b_{\mathrm{kv}}.$$

Here the factor two counts one key and one value. Changing two-byte BF16 or FP16 elements to one-byte values halves this logical element payload, before scale metadata. It does not double total admitted batch: weights, workspaces, allocator tails, reservations, and other state remain. KIVI also found different distribution behavior for cached keys and values and therefore used different quantization granularities in its evaluated two-bit scheme (Liu et al. 2024). The cache format, scales, model revision, and position policy must be part of cache identity, or a reused block can be decoded under the wrong numeric contract.

GPTQ, AWQ, and SmoothQuant solve different problems

GPTQ and AWQ produce weight-only artifacts. SmoothQuant prepares both weights and activations for W8A8 execution. Their names are not interchangeable format labels.

All three are post-training quantization (PTQ) methods: they start from trained weights and use a small calibration set to select scales or approximate the error of candidate weights. Quantization-aware training (QAT) instead exposes the optimization process to simulated or actual low-precision effects and updates model parameters. Activation scales may be static, estimated from calibration data, or dynamic, computed from the current input at runtime. A dynamic rule can adapt to range changes, but its reduction and scaling work is part of latency.

GPTQ minimizes layer-output reconstruction error. Let $\mathbf X$ denote calibration inputs, $\mathbf W$ the layer weight matrix, and $\mathcal Q$ the set of representable quantized matrices. The local objective is

$$\min_{\widehat{\mathbf W}\in\mathcal Q} \left\|\mathbf X\mathbf W-\mathbf X\widehat{\mathbf W}\right\|_F^2.$$

The method uses approximate second-order information to quantize weights in a sequence while updating remaining weights to compensate for error already introduced. The published experiments showed practical three- and four-bit post-training quantization at very large model scales (Frantar et al. 2023). The result still depends on the calibration inputs, layout, model, and runtime kernel.

AWQ uses activation evidence to protect salient weight errors. The AWQ paper observes that retaining about one percent of salient weights at high precision can greatly reduce error, but that mixed-precision representation is used to motivate the method. AWQ does not store that one percent in FP16. Instead, it searches for per-channel scaling that reduces the quantization error of salient weights while keeping a hardware-friendly weight-only layout. It needs calibration data, but not backpropagation or layer reconstruction (Lin et al. 2024).

SmoothQuant moves scale between activations and weights. For positive per-input-channel factors $\mathbf s$,

$$\mathbf X\mathbf W= \left(\mathbf X\operatorname{diag}(\mathbf s)^{-1}\right) \left(\operatorname{diag}(\mathbf s)\mathbf W\right).$$

Here $\mathbf X$ is the activation matrix, $\mathbf W$ is the weight matrix, $\operatorname{diag}(\mathbf s)$ is a diagonal matrix containing the channel scales, and both factors on the right have compatible inner dimensions. SmoothQuant leaves the full-precision layer algebraically unchanged. It makes activation channels easier to quantize by transferring part of their range to the corresponding weight rows, then applies W8A8 quantization. The scales are chosen from calibration statistics, often with a parameter that balances activation and weight ranges (Xiao et al. 2023).

None of these methods makes quantization lossless. Each chooses where to place approximation error and what calibration evidence to use.

A format is not an execution path

Several labels that appear together in model catalogs describe different layers of the stack:

• INT4 and INT8 describe integer element widths, but not group size, metadata, packing, or accumulator behavior.
• FP8 commonly refers to E4M3 or E5M2 encodings. E4M3 allocates more bits to the significand; E5M2 allocates more to the exponent (Micikevicius et al. 2022). Hardware support is device-specific.
• MXFP4 and NVFP4 are block-scaled four-bit floating-point schemes. Microscaling formats pair narrow elements with a shared block scale (Rouhani and others 2023). NVFP4 uses smaller scale groups and a hierarchy of block and tensor scales on supported NVIDIA hardware (NVIDIA 2025).
• GGUF is a container, not a numeric precision. A GGUF file stores tensors and model metadata; its tensors may use one of several quantization types. The llama.cpp ecosystem supplies kernels for many CPU and accelerator backends, but support and performance vary by type and backend (GGML project 2023).

The deployable path has three contracts: artifact, kernel, and runtime.

A runtime may accept an artifact by falling back to a generic conversion or kernel. That is functional compatibility, not evidence of acceleration. Supported does not mean fast. Before conversion, check the engine's current compatibility matrix for the exact model architecture, quantization recipe, device generation, tensor-parallel layout, and attention backend. Mutable engine documentation is useful for that deployment check, but paper results should not be treated as a compatibility guarantee.

FlashAttention removes an intermediate

Conventional attention can be implemented as separate kernels that write a score block to high-bandwidth memory, read it for softmax, write probabilities, and read them again for the value product. FlashAttention instead tiles the calculation so query, key, and value blocks fit in fast on-chip static random-access memory (SRAM). It updates the softmax and output online, without materializing the complete score tensor in high-bandwidth memory. This computes exact attention up to floating-point rounding; it is not a sparse or approximate attention rule (Dao et al. 2022).

For one query row, divide the keys and values into $J$ blocks. For block $j$, let

$$\mathbf S_j=\frac{\mathbf q\mathbf K_j^\mathsf T}{\sqrt d}, \qquad m_j=\max\!\left(m_{j-1},\max_r S_{j,r}\right), \\[4pt] \ell_j=e^{m_{j-1}-m_j}\ell_{j-1} +\sum_r e^{S_{j,r}-m_j}, \\[4pt] \mathbf o_j=e^{m_{j-1}-m_j}\mathbf o_{j-1} +\sum_r e^{S_{j,r}-m_j}\mathbf V_{j,r}, \qquad \mathbf O=\mathbf o_J/\ell_J.$$

Here $\mathbf q\in\mathbb R^d$ is the query row; $\mathbf K_j$ and $\mathbf V_j$ are the key and value rows in block $j$; $S_{j,r}$ is the score for row $r$ of that block; $m_j$ is the running maximum; $\ell_j$ is the running softmax denominator; $\mathbf o_j$ is the unnormalized output accumulator; and $\mathbf O$ is the final attention output. The initial state is $m_0=-\infty$, $\ell_0=0$, and $\mathbf o_0=\mathbf 0$. When a later block raises the maximum, the exponential factors rescale the earlier denominator and accumulator into the new numerical frame. This is the step missing from a naive block-by-block softmax.

Later versions keep the same IO-aware principle while changing work partition and hardware mapping. FlashAttention-2 reduces non-matrix operations and improves parallelism on A100-class hardware (Dao 2024). FlashAttention-3 overlaps data movement and matrix work on Hopper and adds an FP8 path (Shah et al. 2024). FlashAttention-4 redesigns the pipeline for Blackwell's different balance of tensor throughput, shared-memory traffic, and special functions (Zadouri et al. 2026). Their reported speedups belong to the stated hardware, precision, shapes, and baselines. They are not portable constants.

FlashAttention is also not the only fusion that matters. Quantized linear kernels commonly fuse weight unpacking, scale application, matrix multiply, and sometimes bias or activation work. Serving runtimes may fuse normalization, rotary position updates, cache writes, or sampling operations. Fusion removes traffic and launch overhead, but a larger fused kernel can also suffer from register pressure, poor occupancy, unsupported shapes, or extra compilation. Measure the executed path.

Choose and verify a deployment

Start from the workload, not the format name.

1. Record the baseline. Fix the model revision, tokenizer, sampling policy, engine version, kernel backend, hardware, tensor parallelism, prompt-length distribution, output-length distribution, concurrency, and service-level objective.
2. Locate the constraint. Measure resident weights, KV usage, allocator headroom, achieved memory bandwidth, arithmetic utilization, and kernel time. Separate prefill from decode and inspect representative matrix shapes.
3. Choose the tensor target. Use weight-only quantization when weight capacity or decode reads dominate; consider weight-and-activation formats when supported low-precision matrix multiplication dominates; quantize KV state when retained context limits capacity or attention traffic.
4. Record the complete numeric contract. Include method, bit width, element format, group size and axis, scale and zero-point types, clipping rule, calibration data, accumulator precision, tensors left unquantized, and the artifact version.
5. Prove the path executes. Confirm the runtime selected the intended kernels for prefill and decode. Profile unpacking, dequantization, conversions, graph breaks, compilation, and fallback kernels.
6. Compare at matched admitted load. Report time to first token, time per output token, inter-token latency, end-to-end latency, request and token throughput, goodput, peak device memory, energy or cost when relevant, and both median and tail behavior.
7. Re-run quality evaluation. Compare perplexity where useful, but also task-specific quality, long-context behavior, structured-output validity, safety checks, and any product-critical slice. Use identical prompts, decoding rules, and judge versions. Evaluations of quantized multilingual models, for example, have found task- and language-dependent changes that a single aggregate score can hide (Marchisio et al. 2024).
8. Keep a rollback. Fall back to the measured baseline when quality, compatibility, latency tails, or operational stability miss the declared threshold.

Test failures as well as the happy path. Exercise an unsupported layer, an odd matrix shape, a sequence near the maximum context, mixed prompt lengths, cache pressure, tensor-parallel boundaries, cancellation, and engine restart. Verify that loading rejects incompatible metadata instead of silently misinterpreting it. Confirm that a fallback is observable in metrics, because an unnoticed fallback can preserve correctness while erasing the expected speedup.

Payoff and boundary

Quantization reduces representation cost. Fusion and IO-aware algorithms reduce movement and launch overhead. The benefit reaches users only when the artifact, kernel, runtime, workload, and quality threshold agree. That full path is the unit of deployment.

Further reading