Edge and On-Device Deployment

Most of this book serves models from a datacenter. The other end of the wire is the device in a pocket. The on-device regime changes the design problem: models are architected small rather than shrunk after the fact, inference economics push them to be over-trained, quantization is driven past the server's int4 floor down to two bits and below, and ternary weights turn the energy budget into addition-heavy arithmetic. The edge is also fragmented in a way the CUDA datacenter is not, so the cloud-versus-edge split becomes part of the design rather than an afterthought.

Why run on the edge

Four pulls move inference onto the device. Privacy is the decisive one for messages, photos, and health: the data never leaves the phone. Latency drops too, because skipping the network round-trip lets a response start immediately. And the phone keeps working on a plane or in a dead zone, with no per-token serving bill owed to anyone. Against all four stands one hard wall. A phone has orders of magnitude less memory, compute, and power than a datacenter accelerator, no high-bandwidth memory, and a thermal budget measured in single-digit watts. The shape of that wall matters: autoregressive decoding is memory-bandwidth bound, because each token streams the entire weight set through the compute units, so on a device with no high-bandwidth memory the token rate is set by how many bytes must move per token, not by arithmetic throughput (Yuan et al. 2024). That single fact explains both halves of this chapter, why the model is kept small and why its weights are squeezed to the fewest possible bits.

Small by design, not shrunk

You cannot take a 70-billion-parameter model and shrink it onto a phone; the on-device tier is architected for the regime from the start. MobileLLM makes the case sharply: below a billion parameters the usual data-and-scale story yields to architecture, and depth matters more than width, with thirty-layer models beating twelve-layer ones at equal parameter count (Liu et al. 2024). It ties the input and output embeddings, reusing one matrix for both the input token lookup and the output projection, to reclaim the eleven-odd percent of a small model's parameters they would otherwise waste, spends grouped-query attention (sharing key and value projections across query heads to shrink the cache, see Chapter 8) to free more, reinvests all of it into depth, and shares weights between adjacent blocks so they stay resident in SRAM and are simply computed twice, avoiding the SRAM-to-DRAM weight movement that is the real latency cost. The result is a 350-million-parameter model that matches a LLaMA-2 7B on API-calling intent accuracy, a twenty-times-smaller model holding its own on a task that suits it. OpenELM pushes a complementary idea, allocating parameters non-uniformly across layers, narrow near the input and wider with depth, a layer-wise scaling that buys a 2.36 percent accuracy gain at a one-billion budget while using half the pretraining tokens of a uniform baseline (Mehta et al. 2024).

Two other levers define the tier. The first is data quality standing in for scale: the Phi line trained on filtered and synthetic "textbook-quality" data and reached coding and reasoning scores of models several times larger (Gunasekar et al. 2023). The second, and now the dominant production lever, is distillation. Gemma 2 trains its 2- and 9-billion models against a larger teacher's full soft-target distribution rather than one-hot next-token labels, and reports that this beats from-scratch training even at equal or fewer tokens (Gemma Team, Google DeepMind 2024). MiniCPM adds a training-schedule technique, the warmup-stable-decay scheduler whose loss falls only in a short final decay, so low-cost decay branches fork from one shared trunk and make small-model scaling experiments affordable (Hu et al. 2024).

The deepest reason these models are small is economic, and it reaches back into how long they are trained. The compute-optimal Chinchilla ratio of roughly twenty tokens per parameter minimizes training cost, but a deployed model pays inference cost on every request for its whole life. Amortized over a billion requests, total cost is minimized by a smaller model trained far past compute-optimal, because the per-request inference cost that scales with parameter count dwarfs the one-time training cost (Sardana et al. 2024). This is why every model in this tier is over-trained: Gemma 2's small models run past fifty times their compute-optimal token count, and MiniCPM measures its own optimal data-to-model ratio at around 192 to one, an order of magnitude past Chinchilla. The constraint arrow from serving cost, which justified over-training a frontier model in Chapter 5, bites hardest here.

The after-the-fact route exists but produces a different artifact. Sheared LLaMA prunes a 7B model down to 1.3 and 2.7 billion and continues pretraining on a fraction of the tokens (Xia et al. 2024), a real and efficient path, but it inherits a cloud model's architecture rather than gaining MobileLLM's cache-resident block sharing or the quantization-aware training the next section needs. The vendors ship the designed-small tier as a product. Gemini Nano is the on-device size: the original Gemini 1.0 Nano shipped at 1.8 and 3.25 billion parameters, distilled from larger Gemini models and 4-bit quantized (Gemini Team, Google 2023), and by mid-2026 the line has moved through Nano v3 to the Gemma-4-based Nano 4, multimodal across text, image, and audio, still served through Android's AICore (Android Developers 2026). Apple's on-device foundation model is about three billion parameters trained with 2-bit quantization-aware training and KV-cache sharing, with small low-rank adaptation (LoRA) adapters swapped in per feature, while a larger model waits in Private Cloud Compute for the hard requests (Apple 2025). The shared pattern is a small local model for the common case and an escape hatch to the cloud for the rest. Both of these are vendor-integrated; the designed-small tier you can ship yourself is the openly licensed one: Gemma 4's edge sizes E2B and E4B (Apache-2.0, and the base that Nano 4 builds on) and the sub-1B end of the Qwen3.5 ladder (Google 2026).

Quantization to the bone

Server quantization (Chapter 34) trades a little quality for throughput; on an A100 with memory to spare, int8 mostly buys more concurrent requests. On the edge the budget inverts: memory is the binding constraint, both the static footprint that must fit in a few gigabytes of unified RAM and the bandwidth that streams those weights each token. The governing identity is blunt, $\text{weight memory} = N_{\text{params}} \times b_{\text{bits}} / 8$, so a 3B model is 6 GB at 16 bits, 1.5 GB at 4 bits, and 0.6 GB at ternary, and because decode is memory-bound, fewer bits is also directly fewer bytes moved per token and so lower latency and energy. The same lever fits the model and speeds it up.

The standard floor is weight-only int4 post-training quantization, keeping weights at four bits and activations in higher precision, made accurate by methods like GPTQ's second-order error minimization (Frantar et al. 2022) and AWQ's scaling of the salient weight channels that multiply large activations (Lin et al. 2023). The reason the floor sits at the weights and not the activations is the outlier problem: a few emergent activation dimensions carry huge magnitudes that naive low-bit quantization destroys, which LLM.int8() handled by keeping those dimensions in higher precision (Dettmers et al. 2022) and SmoothQuant by migrating the difficulty from activations into weights with a per-channel rescale (Xiao et al. 2022). Weights quantize gracefully; activations do not, which is why the most aggressive edge schemes keep activations wider than weights.

Below four bits, post-training quantization falls off a cliff, because the rounding error is no longer a small perturbation the trained weights can absorb. The edge therefore reaches for quantization-aware training, which simulates the rounding in the forward pass while letting gradients flow through it with a straight-through estimator, which passes gradients through the non-differentiable rounding step as if it were the identity, so the network learns weights robust to being quantized rather than having them rounded after the fact. Apple's on-device model is the production instance, compressed to two bits per weight with quantization-aware training plus learnable weight clipping, the embedding table to four bits and the KV cache to eight, recovering the lost quality with adapters (Apple 2025).

The frontier is one bit. BitNet trains binary weights from scratch (Wang et al. 2023), and BitNet b1.58 relaxes them to ternary values in $\{-1, 0, 1\}$, about 1.58 bits each, reported to match a full-precision transformer of the same size and training tokens in both perplexity (how surprised the model is by held-out text, lower is better) and end-task accuracy from roughly three billion parameters upward (Ma et al. 2024). The mechanism is the payoff. With weights in $\{-1, 0, 1\}$ the dot product $\sum_i w_i x_i$ has no multiplies: every term is $+x_i$, $-x_i$, or zero, so the matrix multiply collapses into conditional additions, the zeros add free structured sparsity, and the kernels become integer-only, which is exactly what a CPU or NPU without strong floating-point throughput wants. The reported gains follow: at 3B, 3.55 times less memory and 2.71 times faster decode, and on 7nm silicon about 71 times less arithmetic energy for the matrix multiply. Follow-ups push activations to four bits (Wang et al. 2024) and ship an integer-kernel CPU inference stack (Wang et al. 2024). Whether one-bit-class training holds at frontier scale is unsettled, since the parity evidence tops out near three billion parameters and the recipe demands training from scratch with no post-training path, but the memory and energy case for the edge is direct: the device's budget is counted in bits per weight.

The other edge memory lever is the KV cache, which grows with context and can dwarf the weights at long context. Quantizing it to two bits, asymmetrically (the key cache per channel, the value cache per token, because their distributions differ), cuts peak memory by over half with little quality loss (Liu et al. 2024).

params = 3e9                        # a 3B on-device model
phone_budget_gb = 4.0              # memory you can spend on weights
for bits in [16, 8, 4, 2, 1.58]:
    gb = params * bits / 8 / 1e9
    fits = "fits" if gb <= phone_budget_gb else "too big"
    print(f"{bits:>4} bits/weight -> {gb:4.1f} GB weights  ({fits})")
print("memory is the budget: quantization is what makes a model fit on the edge")

The runtime

The pragmatic substrate is software like llama.cpp, a dependency-light C/C++ inference engine that runs on plain CPUs and on Apple silicon through Metal and NEON (ggml-org 2023), with the single-file GGUF format carrying the quantized weights and all the metadata needed to load them, architecture, tokenizer, and special tokens, so a model loads with no external config (ggml-org 2023). Its quality per bit comes from a k-quant scheme that quantizes in super-blocks at two to six bits and spends more bits on the tensors that matter most (Kawrakow 2023). Other stacks make different bets. Apple's MLX exploits the unified memory of Apple silicon, where arrays live in shared memory and dispatch to CPU or GPU with no copy (Apple 2023). MLC-LLM compiles one model to many backends, including the browser through WebGPU, on the TVM compiler (mlc-ai 2023; Chen et al. 2018), and ExecuTorch lowers a PyTorch model to run from phones down to microcontrollers (PyTorch Foundation 2024).

The dedicated accelerators are faster per watt but fragment the edge. Apple's Neural Engine, reached through Core ML, needs a model rewritten to map onto it (Apple Machine Learning Research 2022); Qualcomm's Hexagon NPU is reached through a separate SDK (Qualcomm Technologies, Inc. 2024); mobile GPUs from Arm and Qualcomm go through OpenCL or Vulkan; and Google's Tensor SoC carries its own TPU. Each has its own runtime, operator coverage, and quantization formats, so the edge is fragmented in a way the CUDA-centric datacenter, where one toolchain spans nearly every accelerator, is not. A portable GGUF model on the CPU is often the floor you can rely on everywhere, with an NPU path added where the hardware and toolchain allow.

Three techniques claw back speed on the device. Speculative decoding lets a tiny draft model propose several tokens that the larger model verifies in one parallel forward pass, turning a serial memory-bound decode into a partly batched one without changing the output distribution (Leviathan et al. 2023). KV-cache sharing across layers shrinks the second-largest on-device memory consumer (Apple 2025). And LoRA adapter swapping keeps one frozen base resident while small low-rank adapters specialize it per feature (Hu et al. 2022), the mechanism behind Apple's per-feature behavior. When a request does escape to the cloud, the privacy gap is closed by architecture: Apple's Private Cloud Compute runs the larger model under stateless computation, no privileged runtime access, and a verifiable transparency log so the code handling the data can be inspected (Apple Security Engineering and Architecture 2024).

Further reading