Privacy, Provenance, and Unlearning

A model is a lossy compression of its training set, and some of what it compresses is a person. That is why a model leaks at serving time what it absorbed at training time, and why the only sound place to fix the leak is upstream or in a retraining run rather than at the output. When the law demands a fact be removed, machine unlearning, the approximate removal of a learned fact from trained weights, can promise some things and not others. Provenance for generated content is a separate problem from privacy, even though the same controls often touch both. Memorization is not a bug layered on top of learning but the same mechanism seen from the privacy side, so the controls that work are the ones applied before the weights set, and everything attempted after is mitigation against a fact already encoded.

What the weights remember

The first thing to establish is that the leak is real and not hypothetical. A large language model trained on web text will, under the right prompt, emit sequences from its training data verbatim: names, phone numbers, addresses, fragments of private correspondence that happened to be scraped. The founding demonstration extracted hundreds of verbatim training examples from a deployed model, including personally identifiable information, using only query access (Carlini et al. 2021). This is not the model reasoning about a person. It is the model returning a stored string.

How much a model memorizes is not random, and the follow-up work made the dependence precise. Verbatim memorization grows with model scale, with the number of times a sequence is duplicated in the training set, and with the length of context used to prompt for it (Carlini et al. 2023). All three knobs point upstream. A bigger model from Chapter 5 memorizes more. A corpus that escaped the deduplication of Chapter 6 holds its duplicated spans most stubbornly of all, and at inference a longer prompt draws more of them back out. The privacy exposure of the deployed system is set by decisions made before serving existed.

A weaker but more general attack does not need the model to regurgitate anything. membership inference asks only whether a given record was in the training set, by exploiting the fact that a model is more confident, lower loss (its prediction-error score), on data it trained on than on data it did not (Shokri et al. 2017). That is enough to violate privacy on its own: confirming that a particular patient's record was in a medical training set leaks the diagnosis category before a single token is regurgitated. Membership inference is the cheapest privacy attack and the hardest to fully close, because the signal it uses, the generalization gap (the difference in the model's confidence on training data versus unseen data), is a property of learning itself.

Privacy you can buy before the weights set

If exposure is set upstream, the cheapest control is also upstream, and it is the one already motivated for a different reason. Deduplication, introduced in Chapter 6 to stop the model wasting capacity on repeated text, is also the single most effective memorization control, because the sequences a model regurgitates are overwhelmingly the duplicated ones (Carlini et al. 2023). Privacy here is a free rider on an efficiency decision, which is the cleanest kind of control: it costs nothing extra and the constraint arrow runs the right way.

The principled control is stronger and more expensive. differential privacy (DP) bounds how much any single training record can change the trained model, by adding calibrated noise so that the output distribution is nearly identical whether or not that record was present. Applied to deep learning as DP-SGD, the recipe clips per-example gradients and adds Gaussian noise at each step (cap each example's influence, then add noise, the core of differential-privacy training), with a privacy budget that accounts across the whole run (Abadi et al. 2016). The guarantee is real and quantified, and it is the only training-time control that provably bounds memorization rather than merely reducing it. The cost is the problem. At frontier scale the noise and clipping degrade utility enough that no flagship general model is trained end to end under a tight DP budget, and DP is reserved for fine-tuning on a sensitive corpus or for models where the privacy guarantee is the product. Whether DP belongs in the pretraining loop at all, given the utility hit, is one of the open questions below.

PII handling and data residency are usually filed under deployment, the choice of which region a request runs in, and that framing undersells them. Read as privacy controls they are upstream interventions wearing operational clothing. Stripping or tokenizing personal identifiers before a record ever reaches the training set removes the string that could later be extracted; keeping a sensitive corpus inside a jurisdiction and out of the pretraining mix is a decision about what the weights are allowed to encode, not merely where a server sits. The same residency knob that Chapter 56 treats as a regulatory boundary is, from the privacy side, a choice about which facts the model is permitted to memorize.

Removing what was already learned

The hard case is when the fact is already in the weights and someone with a legal right asks for it to be gone. The baseline is brutal: the only way to guarantee a training record's influence is removed is to retrain the model from scratch on the corpus minus that record. For a frontier model that costs millions of dollars and weeks per deletion request, so in practice it is not a usable option. machine unlearning is the field that tries to approximate that guarantee for less.

The one approach that retains an exact guarantee buys it with architecture rather than cleverness. Sharded, isolated training, the SISA scheme, partitions the data into shards, trains a separate model per shard, and aggregates their predictions; deleting a record then requires retraining only the one shard it sat in, not the whole model (Bourtoule et al. 2021). The guarantee is exact because the deleted record provably never touched the other shards. The price is paid up front and permanently: sharding caps how much each sub-model sees, which costs capability, and the aggregate of small models is weaker than one model trained on everything. SISA is the clearest statement of the unlearning trade-off, that an exact deletion guarantee is something you architect for in advance, not something you bolt on after.

Everything cheaper is approximate, and approximate unlearning runs into a measurement problem before it runs into anything else. Suppose a method fine-tunes the model to raise its loss on the target record, or to match the behavior of a model that never saw it. How do you verify the fact is actually gone rather than merely suppressed on the prompts you tested? A model that refuses to recite a memorized address when asked directly may still complete it from an oblique prompt, or retain the influence in a way membership inference can still detect. Evaluating unlearning is close to evaluating a negative, and the field has no agreed test that a capability has been removed rather than hidden, which is the contested point below.

knowledge editing attacks a narrower target: not deleting a record but changing a specific fact the model will state, for instance updating who holds an office. Locating the factual association to a small set of mid-layer feed-forward weights and editing them directly, the ROME method, showed that a single fact can be rewritten with a rank-one update (a minimal, single-direction tweak to the weights) (Meng et al. 2022), and the follow-up scaled it to thousands of edits at once (Meng et al. 2023). The technique is precise and cheap, and that is also its limit. Edits are brittle: they hold for the exact phrasing targeted and can fail to propagate to paraphrases or logical consequences, so the model asserts the new fact in one form and the old one in another. Editing moves the surface a probe touches without guaranteeing the underlying representation moved with it, which connects directly to the open question in Chapter 54 of whether we can even locate a fact precisely enough to say it was changed.

Provenance is a different promise

Privacy asks what the model should not reveal. Provenance asks the opposite question about output: given a piece of text or media, can anyone tell that a model produced it, and trust that answer. The two belong in the same discussion because both are claims about the relationship between a model and a specific string, but they pull in opposite directions, and conflating them is a common error.

For text, the dominant idea is statistical watermarking: at generation time, bias the sampling toward a pseudo-randomly chosen "green" subset of the vocabulary at each step, so that text from the model carries a detectable statistical signature that ordinary text does not, recoverable without access to the model (Kirchenbauer et al. 2023). The scheme is elegant and adds little cost. Its weakness is robustness: paraphrasing the output, translating it and back, or having a second model rewrite it degrades the signal, and a determined evader can usually launder a watermark out. So a text watermark is a useful signal at population scale and a weak one against an adversary, which is exactly the wrong way round for the high-stakes attribution cases people most want it for. The population-scale reading is no longer hypothetical: Google DeepMind's SynthID-Text hardened the same sampling-bias idea for production and has served watermarked Gemini output since 2024, with the scheme published, open-sourced, and paired with a public detector (Dathathri et al. 2024).

For images, audio, and video, the more durable approach is not to watermark the pixels but to sign the provenance metadata. Content Credentials, standardized as C2PA, attach a cryptographically signed manifest recording how an asset was made and edited, so a viewer can verify the chain rather than guess from the content (Coalition for Content Provenance and Authenticity 2024). This inverts the watermarking bet: instead of hiding a signal in the content and hoping it survives, it binds a signature to the content and makes tampering detectable. The limit is adoption and stripping. A signed manifest only helps where the capture device, the editing tool, and the platform all participate, and an unsigned asset is not proof of anything, only the absence of a credential. Standards and law are now forcing that coordination rather than waiting for it: the specification is in the final stage of ISO standardization as ISO/DIS 22144 (International Organization for Standardization 2025), and the EU AI Act's synthetic-media transparency duty applies from August 2026 (Chapter 61), turning participation from voluntary to legally required in one major market. Stripping and the unsigned default remain the live limits, so provenance for the generated media of Chapter 16 is still a coordination problem as much as a cryptographic one.

Trust bought upstream

This layer is almost pure trust, bought against capability and efficiency. Capability is what memorization quietly adds and what differential privacy and unlearning take away: a model trained to forget, or trained under noise, knows less. The efficiency cost shows up in every real control here, from the deduplication that is nearly free to the full retrain that is the price of a guaranteed deletion, with DP and SISA in between as standing taxes on the training budget. Trust, in the end, is the whole point: whether a person can rely on the system not to leak them, and whether a viewer can rely on a provenance claim. The uncomfortable thesis is that trust at this layer is mostly purchased upstream and is expensive to retrofit, because a fact in the weights is not a record in a database that can be deleted with a query. The law that turns these engineering limits into obligations is Chapter 61, and the inference-time controls that try, and largely fail, to contain what the weights already hold are the runtime safety layer that follows.

Further reading