Privacy, Provenance, and Unlearning

Privacy, unlearning, and provenance answer three different questions. Training-data privacy asks what an observer can learn about the people and records used to train a system. Machine unlearning asks whether the influence of specified training data can be removed after training. Provenance asks what evidence accompanies an output about its origin and edit history. None of these promises follows from the others. A model can protect training records without attaching provenance, and a signed output can still disclose private information.

That separation matters because each promise needs a different test. Privacy needs a protected unit and an attacker model. Unlearning needs a retained-data retraining reference. Provenance needs a named mechanism, signer or key, and a verification policy. A refusal, an edited answer, or a badge may be useful, but none establishes all three.

Four events, not one leak

The word memorization often stands in for several different events:

• Memorization is a measured dependence between a trained model and a training example. Its meaning depends on the metric, such as exposure or likelihood relative to a reference model.
• Extractability means that an attacker can recover an unknown training sequence under a stated prompt, decoding strategy, access level, and query budget. A memorized sequence is not necessarily extractable.
• Disclosure means that released output reveals sensitive information about a person. An extracted sequence is not necessarily personal data, and disclosure can also come from a prompt, retrieval result, public source, or inference rather than memorization.
• Membership inference asks whether a supplied candidate belonged to a training dataset. Nothing must be reproduced; dataset membership can itself be sensitive.

These are different events. Carlini et al. recovered verbatim examples, including personally identifying information, from GPT-2 through black-box queries (Carlini et al. 2021). Later measurements found that extraction and memorization varied with repetition, capacity, and available context in the models studied (Carlini et al. 2023). Those results demonstrate attacks, not an assurance that every model or record can be extracted. Membership inference is similarly conditional: the original shadow-model attack showed that black-box outputs can reveal membership (Shokri et al. 2017), while later work found that distribution shift can make pretraining attacks appear more effective than they are (Duan et al. 2024).

Write the privacy threat model first

A privacy result is uninterpretable until its scope is recorded:

1. Privacy unit. Is the protected unit one token, example, document, account, or person? Record-level protection and user-level protection are different when one user contributes many records.
2. Adjacency. Which adjacent datasets count as one person's presence or absence? State whether one unit is added, removed, or replaced.
3. Attacker access. Record model version, prompts, output text, scores or log probabilities, auxiliary information, adaptive queries, and query budget.
4. Success criterion. Define exact or approximate recovery, membership classification, false-positive constraints, and the harm to the affected person.

The same model can pass one threat model and fail another. Restricting log probabilities may frustrate one membership attack but leave text extraction possible. A low average attack score can also conceal a small group of highly exposed examples. Report tail behavior and results by relevant language and record type, not only a global mean.

Reduce risk before and after training

Start with data minimization and purpose limitation: do not collect or retain a record merely because it might be useful later. The lineage practices in Chapter 6 should cover source data, derived datasets, tokenized copies, and fine-tuning data. They should also distinguish deletion from pseudonymization; replacing a name with a reversible identifier does not remove the underlying person's data.

Exact, near-duplicate, and repeated-span detection address different forms of duplication. In one set of language-model experiments, deduplication reduced measured memorization and improved evaluation hygiene (Lee et al. 2022). Related work measured substantially lower extraction risk after deduplicating the studied corpora (Kandpal et al. 2022). Deduplication reduced measured memorization in those settings; it does not provide differential privacy, catch every paraphrase, or remove a fact repeated across independent sources.

PII detection has false positives and false negatives. Measure precision and recall by identifier type, format, language, and document source. Names are not the only risk: combinations of occupation, location, and dates can identify a person. Filtering also has to reach derived datasets, tokenized copies, and fine-tuning data rather than only the first raw-data snapshot.

Serving-time controls still matter. Access control, rate limits, query and output-length limits, restricted score access, output filtering, and monitoring can reduce exposure or raise the cost of an extraction campaign. Buffer a high-risk output until a privacy gate has decided whether to release it, as in Chapter 57. These controls do not remove training influence, cannot recall information already disclosed, and should not be described as unlearning. Protecting prompts, retrieved context, and request logs is a separate confidential-inference problem covered by Chapter 60.

The formal promise of differential privacy

In this formula, $\mathcal{M}$ denotes a randomized training mechanism. For adjacent datasets $D$ and $D'$ and every measurable set of possible outputs $S$, let $p$ represent the first output probability:

$$p=\Pr[\mathcal{M}(D)\in S].$$

Likewise, let $p'$ represent the probability for the adjacent dataset:

$$p'=\Pr[\mathcal{M}(D')\in S].$$

The mechanism is $(\varepsilon,\delta)$-differentially private when the privacy-loss bound is

$$p\le e^{\varepsilon}p'+\delta.$$

Here the privacy unit determines what adjacency means. The parameter $\varepsilon$ is the multiplicative privacy-loss bound, so smaller $\varepsilon$ is stronger when the rest of the mechanism is fixed. $\delta$ permits a bounded probability of exceeding that multiplicative guarantee. The definition compares distributions of possible trained artifacts; it does not say that two training runs produce identical weights or that the model can never output personal information (Dwork and Roth 2014).

DP-SGD applies the definition to iterative optimization (Abadi et al. 2016). For a sampled batch of size $B$, let $g_{t,i}$ be the per-example gradient at step $t$. Clip each gradient to clipping norm $C$:

$$\bar g_{t,i}=g_{t,i}\min\left(1,\frac{C}{\lVert g_{t,i}\rVert_2}\right).$$

Then average the clipped gradients with Gaussian noise:

$$\begin{aligned} \widetilde{g}_t &=\frac{1}{B}\left( \sum_{i=1}^{B}\bar g_{t,i}\right. \\ &\qquad\left.+\mathcal{N}(0,\sigma^2 C^2 I) \right). \end{aligned}$$

Here, $\sigma$ is the noise multiplier and $I$ is the identity covariance shape. With learning rate $\eta$, the optimizer makes the update

$$\theta_{t+1}=\theta_t-\eta\widetilde{g}_t.$$

The batch size, sampling rate, number of steps, clipping norm, and noise multiplier all affect privacy and learning. Repeated access to private data causes composition: privacy loss accumulates across steps and other releases. A privacy accountant converts the sampling and noise schedule into the reported $(\varepsilon,\delta)$ bound under stated assumptions. Every tuning run or private-data-dependent release must be included in that accounting.

Differential privacy bounds one unit's influence on the mechanism's output distribution. It does not remove information repeated across many privacy units, protect private inference prompts, or guarantee useful accuracy. Utility must be measured for the deployment population and rare cases, not inferred from the privacy budget alone.

Define unlearning against retraining

machine unlearning, a process that aims to remove the influence of specified training data from a trained model, needs a counterfactual target. Here $\mathcal{A}$ denotes a randomized training algorithm, $D$ the original training dataset, $D_f$ the forget set, and $D_r=D\setminus D_f$ the retained dataset. The symbol $\mathcal{U}$ denotes an unlearning algorithm applied to an artifact produced by $\mathcal{A}(D)$. The retraining reference is

$$\begin{aligned} &\mathcal{U}(\mathcal{A}(D),D_f) \\ &\text{compared with} \\ &\mathcal{A}(D\setminus D_f). \end{aligned}$$

Exact unlearning makes the distribution of artifacts on the left match the distribution produced by retained-data training on the right. Approximate unlearning bounds a stated distance or distinguishability between those distributions (Guo et al. 2020). The target is a distribution because training and unlearning can both contain randomness. Matching one checkpoint or one answer is not the definition.

Full retained-data retraining is the general reference, not the only possible exact method. Some restricted learners support efficient exact or certified removal under their mathematical assumptions. Those results do not transfer automatically to a different architecture, optimizer, or training pipeline. Even exact unlearning removes the specified records' influence under the declared training algorithm; it need not erase a semantic fact also present in retained duplicates, correlated examples, or earlier pretraining.

SISA: deletion designed into training

SISA partitions the training data into disjoint shards, divides each shard into cumulative slices, trains isolated component models, and saves intermediate states. A deletion can retrain only affected components from a checkpoint before the relevant slice, then repeats the declared aggregation (Bourtoule et al. 2021). The training procedure must be designed in advance, and exactness is relative to that procedure.

The assumptions are operational as well as mathematical. Shard and slice assignments, randomness, checkpoints, and aggregation must be reproducible. Multiple affected shards require multiple retrains, and a learned aggregator may itself need deletion. The original evidence concerned supervised classification; it does not make arbitrary post-hoc editing exact or establish the same trade-off for a frontier language model.

Evaluate forgetting, retention, and recovery

An empirical evaluation should compare several unlearned artifacts with several retained-only retraining references. At minimum, report:

• Forget-set behavior: likelihood, exposure, extraction attempts, membership inference, direct prompts, paraphrases, and adaptive prompts.
• Retain-set utility: task performance, calibration, and behavior on neighboring knowledge. Measure collateral damage rather than hiding it in an average.
• Recovery: white-box probes where available, short fine-tuning or relearning attacks, and access to pre-deletion outputs or artifacts.
• Operations: deletion size, sequential requests, storage, latency, cost, and the complete affected-artifact inventory.

One refusal is not evidence of unlearning. A method may suppress the tested answer while retaining membership or enabling rapid relearning. Conversely, a retained-data retrain may still state the same fact because retained sources support it. TOFU provides synthetic author profiles, forget sets, retained references, and multiple metrics; its tested baselines did not jointly achieve effective forgetting and retention (Maini et al. 2024). It is a useful benchmark, not a universal certificate.

Deletion is a lineage operation

A deletion request should resolve from the data subject or record scope to every affected artifact:

• the source record, dataset snapshots, deduplicated copies, transformed and tokenized data, and queued future training runs;
• checkpoints, optimizer state, adapters, merged models, ensembles, and model registry copies;
• embeddings, retrieval indexes, evaluation fixtures, caches, logs, and backups governed by the same request; and
• a deletion tombstone that prevents accidental re-ingestion while preserving the minimum evidence needed for audit.

A checkpoint and adapter copy left outside the inventory can restore the old behavior after the primary model is replaced. The tombstone needs a stable scope identifier and expiry or review policy, without retaining the deleted content itself.

Editing is a different operation

Knowledge editing is not machine unlearning. It changes selected model behavior, often a subject-relation answer, and is commonly evaluated for direct efficacy, paraphrase generalization, and locality (Meng et al. 2022; Meng et al. 2023). It does not establish that a training record's influence is gone. A successful edit may fail on a paraphrase or logical consequence, cause collateral changes, or leave membership evidence intact. Use editing when the product promise is to change an answer; do not report it as record deletion without a corresponding unlearning argument and evaluation. This distinction also limits what mechanistic findings from Chapter 54 can establish.

Provenance provides evidence, not truth

A statistical watermark, a signed manifest, attribution to an author, and the truth of the content are four separate claims. A participating generator can embed a detectable signal without identifying the person who requested the output. A signer can attest to an edit history without proving that the depicted event happened.

Statistical watermarks

One family of text watermarks changes generation so tokens correlate with a secret keyed pattern. A detector computes a score and compares it with a detector threshold (Kirchenbauer et al. 2023). The operational report needs the false-positive rate, false-negative rate, abstention policy, text length, language, sampling configuration, model version, and measured quality. It also needs the evaluation base rate: even a small false-positive rate can dominate when watermarked text is rare.

Robustness is transformation-specific. Test truncation, ordinary edits, watermark paraphrase, translation, model rewriting, and mixed human/model text. A detector may retain confidence after one edit and lose it after another. Absence of a detectable watermark is not proof of human authorship; the text may come from a nonparticipating generator, be too short, or have been changed.

SynthID-Text is a production example with a different sampling construction from the original green-list scheme. Its large-scale evaluation covered about twenty million Gemini responses and measured detection and quality in that deployment (Dathathri et al. 2024). This evidence supports the tested mechanism and configuration, not a universal AI-text detector.

Signed Content Credentials

C2PA represents provenance claims in a signed manifest. The manifest contains assertions, an asset hash or other content binding, a claim, a claim signature, and information needed to evaluate the signing credential (Coalition for Content Provenance and Authenticity 2026). A valid signature and binding provide tamper evidence: they show that the signed claim was bound to this asset and has not been modified undetected since signing. Linked manifests can record an edit history when participating tools preserve or extend the chain.

Verification still requires policy. The verifier decides which trust anchors, credential status, algorithms, and assertion types it accepts. An unknown signer is not automatically malicious, but its claims may be outside the verifier's trust policy. A valid credential does not prove that the depicted event happened, that every edit was recorded, or that the assertions were honest.

Manifests can be embedded or stored externally, and soft bindings can help recover provenance after a permitted transformation. Complete removal remains possible. A stripped manifest produces missing evidence, not a successful verification; an unsigned asset is not evidence that the asset is synthetic. Watermarks and credentials can therefore complement one another, but neither turns absence into proof. C2PA's manifest architecture is also the subject of an ISO work item (International Organization for Standardization 2026). Current standards and disclosure duties are tracked in Chapter 61 because their status and scope change independently of these engineering guarantees.

Keep an operating record

The audit record should be machine-readable and versioned. A compact schema can use these fields:

data_subject_or_record_scope:
source_and_legal_basis:
dataset_and_transform_revisions:
privacy_unit_and_adjacency:
dp_mechanism_and_accountant:
training_job_and_artifact_revisions:
deletion_request_and_tombstone:
affected_artifact_inventory:
unlearning_method_and_retraining_reference:
forget_retain_and_privacy_results:
provenance_mechanism_and_signer:
detector_threshold_and_error_rates:
verification_and_exception_owner:

Do not let the schema become a claim generator. Empty fields, exceptions, and failed verification must remain visible. Access to the record should be restricted because lineage and deletion evidence may themselves identify a person.

Regression scenarios

Automate cases that exercise the assumptions, not only the happy path:

• a duplicated canary, one user across many records, a PII detector miss, and membership distribution shift;
• a checkpoint and adapter copy, a forget-set paraphrase, a relearning attack, and a retained-utility regression;
• a watermark paraphrase, a human-text false positive, a stripped C2PA manifest, and a tampered asset.

Each failure should name the affected person or content claim, the responsible owner, and whether the remedy changes data, training, serving, deletion, or verification.

Three promises, three kinds of evidence

Privacy is supported by a threat model, measured attacks, data controls, and, where used, a formal privacy budget. Unlearning is supported by lineage and a comparison with retained-data retraining. Provenance is supported by a watermark detector or verified signed assertions under a trust policy. Keeping these claims separate makes them more useful: an operator can state what was protected, what was removed, what was signed, and what remains unknown.

Further reading