Edge and On-Device Deployment

An on-device model is not a small server placed inside a phone. It is one component of an application that must work across different processors, operating-system versions, memory conditions, battery states, and network conditions. The relevant unit is therefore a versioned device deployment: the model, tokenizer, prompt template, compiled graph, quantization scheme, runtime, routing policy, and supported device class that produced a result.

Local execution is useful when a feature needs an offline path, a short and predictable response loop, or a narrower data boundary. It is justified only when the complete feature also meets its quality, memory, energy, thermal, and support requirements. The model decision from Chapter 81 and the serving contract from Chapter 82 still apply; the difference is that qualification happens on a heterogeneous fleet rather than a controlled server pool.

Freeze the device support envelope

Start with the feature, not a parameter count or a runtime. A device support envelope states where the feature is supported and what “supported” means. It turns “run this model locally” into a claim that can be tested and withdrawn.

The support envelope is narrower than “all recent phones.” A useful device tier names the properties that affect execution: OS build, runtime version, system-on-chip family, memory class, storage headroom, and available backend. Marketing names alone are insufficient because devices sold under one family can have different memory or thermal behavior.

Represent the envelope as an acceptance matrix. Rows are supported device tiers; columns are relevant states such as cold and warm process, model absent and installed, normal and low storage, normal and low battery, foreground and background load, offline and weak network, and cool and thermally constrained. Each cell records pass, fail, or unsupported against the same feature contract. This makes gaps visible before a rollout turns them into user reports.

Count what must fit

Raw parameter size is only the first term in the memory budget. For a decoder model, a useful planning approximation is

$$\begin{aligned} M_{\mathrm{weights}} &\gtrsim \frac{N_{\mathrm{params}} b_w}{8} + M_{\mathrm{metadata}}, \\ M_{\mathrm{KV}} &= \frac{2 B L n_{\mathrm{kv}} d_h S b_{\mathrm{kv}}}{8}, \\ M_{\mathrm{peak}} &= M_{\mathrm{weights}} + M_{\mathrm{KV}} + M_{\mathrm{workspace}} + M_{\mathrm{runtime}} + M_{\mathrm{app}}, \\ M_{\mathrm{peak}} &\le M_{\mathrm{budget}}. \end{aligned}$$

Here $M_{\mathrm{weights}}$ is the stored weight footprint; $N_{\mathrm{params}}$ is the number of parameters; $b_w$ is the average stored bits per parameter; the constant $8$ converts bits to bytes; and $M_{\mathrm{metadata}}$ covers quantization scales, zero points, block headers, tensor alignment, and other format data. The inequality is deliberate: the packed-tensor calculation is a lower bound, not an installed-size or resident memory guarantee.

$M_{\mathrm{KV}}$ is the uncompressed planning size of the key-value cache; the factor $2$ accounts for keys and values; $B$ is the number of concurrent sequences; $L$ is the number of transformer layers; $n_{\mathrm{kv}}$ is the number of key-value heads per layer; $d_h$ is the head dimension; $S$ is the number of cached tokens per sequence; and $b_{\mathrm{kv}}$ is the stored bits per cache element. Runtime-specific padding, cache layout, sliding windows, or cross-layer sharing can change the measured value.

$M_{\mathrm{workspace}}$ covers activations, temporary buffers, compiled kernels, and backend workspaces. $M_{\mathrm{runtime}}$ is the inference runtime and its loaded libraries. $M_{\mathrm{app}}$ is everything else the application needs while inference runs. Their sum is peak resident memory $M_{\mathrm{peak}}$, which must remain below the memory available to the feature, $M_{\mathrm{budget}}$. The budget is not the device's advertised RAM: the OS, other processes, graphics, media, and the application already consume it.

A 3-billion-parameter model at four bits has a raw packed-weight payload of 1.5 GB. That arithmetic is useful for rejecting an impossible candidate, but it does not prove the candidate fits. Quantization metadata, tensor alignment, context-dependent cache growth, runtime allocations, and memory pressure can move the actual peak materially. Measure the exported artifact in the real process, including its longest supported context and its worst supported input.

KIVI illustrates why the cache deserves its own line item: its authors found different distributions in keys and values and evaluated per-channel key and per-token value quantization rather than treating the cache as ordinary model weights (Liu et al. 2024). The result belongs to the evaluated models and runtime; it is evidence for testing cache quantization, not permission to assume that a two-bit cache is lossless everywhere.

# Illustrative planning values; replace them with the candidate artifact.
params = 3_000_000_000
weight_bits = 4
batch = 1
layers = 28
kv_heads = 8
head_dim = 128
cached_tokens = 4096
kv_bits = 8
other_gb = 1.1  # runtime, app, workspaces, and measured format overhead

weights_gb = params * weight_bits / 8 / 1e9
kv_gb = 2 * batch * layers * kv_heads * head_dim * cached_tokens * kv_bits / 8 / 1e9
peak_gb = weights_gb + kv_gb + other_gb
print(f"packed weights: {weights_gb:.2f} GB")
print(f"planned KV cache: {kv_gb:.2f} GB")
print(f"planning total: {peak_gb:.2f} GB; verify peak RSS on every device tier")

Measure the path and the device state

The local and remote paths have different latency components. For an installed local artifact,

$$T_{\mathrm{local}} = T_{\mathrm{load}} + T_{\mathrm{prefill}} + \sum_{k=1}^{O} T_{\mathrm{decode},k} + T_{\mathrm{post}},$$

where $T_{\mathrm{load}}$ is model load and specialization time; $T_{\mathrm{prefill}}$ processes the input; $O$ is the number of output tokens; $T_{\mathrm{decode},k}$ is the time for output token $k$; and $T_{\mathrm{post}}$ covers validation, rendering, and any feature-specific post-processing. A remote path instead includes upload, network, provider queue, remote execution, and download time. Neither path is inherently faster. A cold local load can lose to a warm remote service, while a warm local model can avoid variable network delay.

For batch-one autoregressive decode, a bandwidth estimate can expose an upper bound:

$$R_{\mathrm{decode}} \lesssim \frac{BW_{\mathrm{eff}}}{D_{\mathrm{token}}}.$$

Here $R_{\mathrm{decode}}$ is generated tokens per second; $BW_{\mathrm{eff}}$ is effective memory bandwidth in bytes per second under the measured device state; and $D_{\mathrm{token}}$ is bytes moved from memory per generated token. This is a bandwidth ceiling, not a prediction. Prefill can be compute-bound, some weights or cache blocks may be reused, and unpacking, dequantization, unsupported operators, synchronization, or small kernels can become the bottleneck. Roofline analysis is a way to form a hypothesis about a specific workload. It is a measurement, not a promise that lower precision must increase speed (Yuan et al. 2024).

Measure at least cold start, warm start, time to first token, time per output token, end-to-end latency, peak resident memory, error rate, and quality on the same task set. Report p50 and p95 rather than one best run. Also measure energy per accepted task, not energy per token alone. A useful device-side estimate is

$$E_{\mathrm{task}} = \int_{t_0}^{t_1} \left(P(t)-P_{\mathrm{idle}}\right)\,dt,$$

where $E_{\mathrm{task}}$ is incremental task energy; $t_0$ and $t_1$ bound the complete feature interaction; $P(t)$ is measured device power at time $t$; and $P_{\mathrm{idle}}$ is the comparable idle baseline. If the platform exposes only energy proxies, label them as proxies and keep the measurement method stable across candidates.

Short bursts hide the device's central failure mode. Run a sustained run until latency, power, and temperature reach thermal steady state or the test limit. Record battery state, charging state, ambient condition, power mode, screen state, background load, and frequency throttling. Both Android and Apple expose thermal-state signals that applications can observe; those signals should be part of a controlled test and, where appropriate, a runtime admission decision (Android Developers n.d.; Apple n.d.). MLPerf Mobile likewise treats the model, software stack, device, accuracy target, and run rules as one benchmarked system (Reddi et al. 2022).

Compression is a system choice

Compression is valuable only when the resulting artifact retains task quality and the target backend executes its representation efficiently. Keep the weight format, activation format, and KV-cache format separate in the decision record. They affect different memory regions and require different kernel support. Chapter 34 develops the underlying representations and kernels; the concern here is qualifying their exported device artifact.

MobileLLM is useful evidence for designing within a small budget: its authors studied deep-and-thin sub-billion models, embedding sharing, grouped-query attention, and adjacent-block sharing rather than merely compressing a much larger checkpoint (Liu et al. 2024). That result motivates architecture search in the intended regime; it does not establish one universal mobile shape.

For post-training quantization, the calibration set is part of the artifact. It should cover real prompt lengths, languages, modalities, tool schemas, and rare but important request classes. AWQ uses observed activations to protect salient weight channels in a weight-only method (Lin et al. 2024). SmoothQuant moves some quantization difficulty from activations into weights to support W8A8 execution (Xiao et al. 2023). These methods solve different problems, so a paper's aggregate benchmark does not substitute for task-specific evaluation on the exact artifact and exact backend.

Quantization-aware training and low-bit pretraining become relevant when post-training quality is insufficient. One published Apple model used two-bit quantization-aware training and KV-cache sharing for an approximately 3-billion-parameter on-device model (Li et al. 2025). BitNet b1.58 reports ternary weights in $\{-1,0,1\}$ for models trained in that representation (Ma et al. 2024). In the mathematical dot product, a ternary weight replaces a general weight multiplication with selection, sign change, or zero. A deployed system still has scales, activation processing, attention, normalization, packing, and memory traffic. Zero-valued weights are not automatically structured sparsity, and a backend saves work only if its kernel exploits the representation. Smaller does not automatically mean faster.

Build a deployable artifact

A source checkpoint is not a mobile artifact. Export and lowering specialize a graph to an operator set, memory plan, tensor layout, precision, and backend. ExecuTorch, for example, separates export and ahead-of-time lowering from its device runtime and produces programs for selected backends (PyTorch Foundation n.d.). Core ML can schedule supported work across CPU, GPU, and Neural Engine, but the actual compute plan and compatibility still depend on the converted model and target OS (Apple n.d.). A container such as GGUF can carry tensors and model metadata, yet a runtime build must still implement the architecture, tokenizer semantics, quantization types, and required operators (ggml-org n.d.). Format is not compatibility.

Record a deployment fingerprint with at least:

• source checkpoint and immutable revision;
• tokenizer, special tokens, and prompt template;
• adapter set and merge state;
• export graph and exporter version;
• quantization configuration and calibration-set revision;
• compiled artifact format and cryptographic digest;
• runtime version and linked operator set;
• backend delegate, fallback partitions, and compute policy;
• supported OS build, ABI, and device class; and
• routing, safety, context, and sampling policies.

The fingerprint prevents a common diagnostic failure: comparing “the same model” when one build changed its template, exporter, quantization, or backend. It also supplies the key for every quality result and device metric.

Before performance testing, establish semantic parity. Run golden prompts through the source graph and compiled artifact; compare tokenization, structured output, stop handling, and task results within declared tolerances. Inspect the partition plan for unsupported operators. A silent CPU fallback may preserve correctness while destroying latency or energy, so the capability probe must report the actual backend delegate and every CPU fallback.

Make hybrid routing explicit

Hybrid execution is a policy decision, not a catch-all exception handler. Define the route before execution as

$$r(x,d,s,p) \in \{\mathrm{local},\ \mathrm{cloud},\ \mathrm{decline}\},$$

where $x$ is the request and its task class; $d$ is its data classification; $s$ is current device state, including model availability, memory, thermal, battery, and connectivity; $p$ is the versioned product policy; and $r$ is the selected route. “Decline” includes asking the user to download the model, wait, change a setting, or perform the task elsewhere.

For a local-only request, model unavailable, low storage, or thermal pressure must lead to a local error or decline path; never upload it. For a cloud-eligible request, require the declared legal and product basis and any explicit consent before transmission. If permission is absent, there is no silent fallback. If both routes are allowed, choose using measured quality and the remaining deadline, not a vague estimate that the request is “hard.”

Route before execution whenever possible. Falling back after partial output can duplicate text, change semantics, or expose data the user expected to remain local. If a local run fails after producing a partial output, terminate it with a typed reason and let the product contract decide whether the user may start a new cloud attempt. Never append a cloud continuation invisibly. Return execution provenance (local, cloud, or declined) so the UI, telemetry, and support tools can explain what happened.

Local execution is not automatically private

Keeping inference in the app can reduce transmission and server retention, but it is not, by itself, a privacy guarantee. Input or output may still reach temporary files, clipboard history, backups, keyboard services, shared caches, analytics SDKs, crash reports, telemetry, tools, or a cloud fallback. The model and its personalization data can also reveal sensitive behavior if stored without the app sandbox and platform protections.

Draw a data-flow map for the complete feature. For each surface, record purpose, data class, retention, protection, recipient, deletion path, and whether the surface works offline. Apply the same security review used for other mobile data: app sandbox boundaries, secure storage, authenticated network transport, least privilege, log redaction, and dependency review. The OWASP Mobile Application Security Verification Standard separates storage, network, platform, code, resilience, and privacy controls precisely because “processed locally” addresses only part of the system (OWASP Foundation n.d.). See Chapter 56 for the broader trust and authorization model.

Default fleet telemetry to content-free metrics: deployment fingerprint, route and reason, device tier, timing buckets, peak-memory bucket, thermal state, load error, cancellation, and quality outcome when it can be collected without content. Do not collect prompt or completion content by default. If a debugging program needs examples, make it separately authorized, narrowly sampled, redacted, access-controlled, time-bounded, and removable. OpenTelemetry's generative-AI conventions warn that model inputs and outputs are likely to contain sensitive information (OpenTelemetry n.d.).

Deliver model versions safely

Large artifacts are often downloaded separately from the application. That creates a software-update problem: incomplete downloads, corrupt storage, rollback attacks, incompatible runtimes, low disk, and mixed versions of model, tokenizer, and adapter. Treat the bundle as immutable and activate it only as a complete set.

Each release needs a signed compatibility manifest containing artifact names, sizes, cryptographic digest values, signature information, deployment fingerprint, supported app and runtime versions, target device classes, and expiry or retirement policy. The Update Framework formalizes signed metadata, hash and size verification, version progression, and defenses against rollback, freeze, and mix-and-match attacks; it is a useful model even when a platform's asset service performs the transport (The Update Framework n.d.).

Download beside the active bundle. Before activation, check free space and disk quota, verify size, cryptographic digest, signature, compatibility manifest, and license notices, then run a minimal load and golden-output smoke test. Use atomic activation so a process sees either the old complete bundle or the new complete bundle. Retain a last-known-good version until the rollout is healthy. Garbage collection must not delete the active or rollback target, and it must cope with an interrupted download.

Use a staged rollout by representative device tier, not only by account percentage. Stop or rollback when load failures, crashes, quality rejection, p95 latency, peak memory, energy, or thermal events cross their declared thresholds. A server-side kill switch can disable eligibility, but offline behavior must remain defined: an installed unsafe version may require local disablement encoded in the app or an expiry policy rather than a network response that never arrives.

Operate the fleet

The operating loop is short enough to make explicit:

1. Freeze the envelope. Record the task class, data boundary, quality threshold, supported device tiers, minimum OS, offline behavior, and fallback policy.
2. Build the fingerprint. Version the source checkpoint, tokenizer, prompt template, export graph, quantization configuration, compiled artifact, runtime, backend, and policy as one candidate.
3. Qualify the complete artifact. Check source-to-device parity, task quality, operator coverage, CPU fallback, storage, peak resident memory, and every acceptance-matrix cell.
4. Benchmark sustained behavior. Measure cold and warm latency, p50 and p95, energy per accepted task, memory pressure, and thermal steady state on representative device tiers.
5. Inject device failures. Exercise model unavailable, offline start, weak network, low storage, corrupt download, backend rejection, cancellation, background load, and thermal throttling.
6. Stage delivery. Verify the inactive bundle, activate atomically for a small device-tier cohort, and compare against the last-known-good release.
7. Observe and rollback. Monitor content-free route, quality, reliability, latency, energy, and thermal signals; rollback on the recorded thresholds.
8. Register each requalification trigger. Repeat the affected matrix when the model, tokenizer, template, quantization, exporter, runtime, backend, operating system, device policy, or feature contract changes.

NIST's generative-AI profile treats deployment-context evaluation, post-deployment monitoring, incident response, change management, and decommissioning as lifecycle work rather than one-time release checks (Autio et al. 2024). The same framing applies here. Chapter 87 provides the quality and observability method, while Chapter 89 develops staged release and rollback in more detail.

The output is a deployment decision record: the envelope, exact fingerprint, acceptance matrix, measured results, privacy data flow, routing policy, delivery plan, rollback thresholds, owner, and requalification triggers. It supports a local release, a hybrid release, or a documented decision not to ship on-device.

Further reading