The Production Data Engine

A production data engine is a governed evidence pipeline. It turns eligible product events into versioned data products for a declared purpose, such as monitoring quality, investigating failures, building an evaluation set, or acquiring training examples. It does not treat traffic as free raw material. Production traces are biased observations about a deployed system, and often a sensitive liability. A click, edit, regeneration, approval, or stop event is useful evidence, but it is not automatically a label.

This distinction changes the design. The pipeline must decide what may be collected, minimize it before storage, preserve the context needed to interpret it, sample according to purpose, obtain and check judgments, keep evaluation apart from training, and publish an immutable release with lineage. A model update is only one possible consumer. It still has to pass the evaluation and rollout gates in Chapter 89.

The data work is not a clerical tail on model development. Small problems in collection and labeling can cascade into failures that are expensive to locate later (Sambasivan et al. 2021). Dataset documentation exists for the same reason: creators need to tell consumers why data exists, how it was collected, what it contains, and which uses are appropriate (Gebru et al. 2021). The production version of that discipline begins at the product boundary.

Start with a data intake contract

Instrumentation should not begin with "log everything." Begin with the decision the data must support and the minimum evidence needed for that decision. A data intake contract defines what an eligible event means before the first event is stored.

The contract is a schema and a policy. Reject or quarantine events with missing identity, unknown tenant, absent use authorization, invalid versions, or a purpose that does not permit the proposed consumer. Apply data minimization and redaction before storage when possible. Encrypting an unnecessary copy does not make its collection necessary.

The system context matters because behavior is conditional. A regeneration after model release A, prompt version 7, retrieval version 3, and interface version 2 is not interchangeable with the same gesture after any of those components change. Chapter 91 defines correction events at the decision boundary. The data engine preserves those events and their context; it does not reinterpret them as grades.

Keep the evidence types separate

Several objects that look similar in a warehouse mean different things:

• A behavioral event records an action, such as accept, edit, regenerate, abandon, approve, reject, escalate, or stop.
• An observation joins eligible events with the system context and any later outcome that the intake contract permits.
• An annotation records a person's or machine's answer to a versioned task specification. It retains annotator identity or class, timestamp, guideline version, and confidence or abstention where relevant.
• An adjudicated label is a resolved answer that passed the defined quality process. Agreement is not truth: reviewers can agree because they share a bad instruction or the same missing evidence.
• A data product is an immutable, purpose-specific release of observations, annotations, or labels with a manifest, lineage, quality results, and allowed consumers.

The separation prevents an attractive but invalid shortcut. An accepted answer may be correct, merely convenient, or never inspected. An abandoned session may indicate a poor answer, a solved problem, an interruption, or a slow interface. Edits reveal a difference between before and after artifacts, not why the user made it. These signals can prioritize investigation, but ground truth requires a task-specific outcome or a qualified judgment.

Figure 92.1 shows the pipeline. The final gate notes that a model change alters future exposure, but there is no automatic train-and-deploy cycle.

Sample for the question being asked

One sampler cannot serve every purpose. Maintain separate streams and budgets:

Do not mix the streams in reporting. A queue enriched for difficult examples is valuable for diagnosis, but its raw failure rate cannot describe the served population. Conversely, representative monitoring may spend many labels on ordinary cases because those cases are part of the quantity being estimated. That is measurement, not waste.

If unequal-probability sampling is used for monitoring, retain each inclusion probability. For an eligible population of $N$ events, a probability sample $s$, an observed outcome $y_i$ for event $i$, and inclusion probability $\pi_i = P(i \in s)$, the Horvitz-Thompson estimate of the population mean is (Horvitz and Thompson 1952):

$$\widehat{\mu}_{HT} = \frac{1}{N}\sum_{i \in s}\frac{y_i}{\pi_i}.$$

Here, $N$ is the number of eligible events in the measurement window, $s$ is the set actually sampled, $y_i$ is the measured value for event $i$, and $\pi_i$ is that event's known positive chance of entering $s$. The inverse-probability weight is $1/\pi_i$. This estimator does not repair a broken eligibility rule, missing outcomes that depend on unobserved quality, or an unknown selection policy. Outcome delay and nonresponse need their own measurement treatment.

Use active learning as acquisition, not measurement

Active learning can direct scarce review toward examples expected to be useful for a learner (Settles 2009). Uncertainty sampling is one method, not the definition. Teams may also select by expected model change, diversity, novelty, error clusters, or slice coverage. Each method encodes a different notion of value.

The limits matter. Model uncertainty can be miscalibrated. A high-uncertainty queue can overselect outliers and underselect confident failures. For generated answers, disagreement among repeated samples is a diagnostic feature, not a calibrated probability of error. Consensus can be wrong, and deciding whether two free-form answers are semantically equivalent may itself require judgment. Keep an independent representative audit sample so the acquisition policy does not grade its own discoveries.

Weak supervision can expand a training candidate set with programmatic labeling functions (Ratner et al. 2016). Its label model relies on assumptions about labeling-function accuracy and labeling-function dependencies. Correlated rules can repeat the same mistake while appearing to provide several votes. Version the functions, test their coverage and conflicts, and compare their outputs with a held-out human audit. Weak labels are useful inputs to a release process, not automatically denoised truth.

Operate annotation as a production service

"Human feedback" hides a socio-technical system. The optimizer never sees the conversation that resolved an ambiguous instruction; it sees the resulting label. That makes the annotation operation part of model behavior, even though the people performing it are not the model.

A credible operation includes:

1. A versioned task specification with scope, definitions, counterexamples, evidence requirements, and an abstain or escalate option.
2. Annotator qualification for the domain and language, followed by practice on a calibration set with explained answers.
3. Seeded quality checks, blind relabel of a probability sample, and slice-level audits that remain separate from production scoring.
4. An adjudication process for disagreement and guideline defects. Agreement is a consistency measure, while adjudication and external outcomes address validity.
5. An operated queue with priority, queue age, throughput, a service-level objective, reviewer fatigue controls, and a bounded escalation path.

Data cascades often begin when these decisions are undocumented (Sambasivan et al. 2021). Benchmark label errors show why a familiar dataset is not exempt from review (Northcutt et al. 2021). Track error estimates and their uncertainty by task, guideline version, annotator cohort, language, and other material slices. Do not rank annotators solely by agreement with a majority that may itself be wrong.

Machine judgments need the same discipline

AI feedback can reduce cost and latency. InstructGPT is an example of training with human preference data (Ouyang et al. 2022), while Constitutional AI demonstrates a process that uses written principles and AI feedback for much of the supervision (Bai et al. 2022). Neither design makes a judgment source infallible.

Record a machine annotation with the judge model, judge prompt, decoding settings, policy or rubric version, input evidence, output, and abstention. Run expert calibration on held-out items, inspect disagreement by slice, and define when a person must review the case. A machine annotation is not ground truth. Changing the judge version creates a new measurement instrument and requires a bridge study before scores are compared across time.

Assign partitions before looking at content

An event cannot safely become both a training example and proof that training worked. Make a stable assignment before inspecting content or outcome, using an immutable split salt and a stable event identity. Typical destinations are:

• Monitoring and audit, for representative measurement and independent quality checks.
• Regression evaluation, for qualified cases that gate future releases.
• Training and weak supervision, for candidate examples and programmatic labels.
• Quarantine, for missing authority, corrupt records, suspected poisoning, unresolved incidents, or schema violations.

Entity-aware grouping may be necessary. Two events with different identities can still contain the same conversation, document, user, or generated variant. Check for exact and semantic duplicates across partitions. Record and block a split collision. Once a case has influenced model or prompt development, moving it into a supposedly held-out set creates adaptive leakage.

This dependency-free sketch demonstrates deterministic assignment. In a real system, the split salt is an access-controlled release parameter, and related entities are grouped before hashing.

import hashlib


def stable_partition(event_id, split_salt, eligible=True):
    """Assign an event before its content or outcome is inspected."""
    if not eligible:
        return "quarantine"

    key = f"{split_salt}:{event_id}".encode("utf-8")
    bucket = int.from_bytes(hashlib.sha256(key).digest()[:8], "big") % 10_000
    if bucket < 1_000:
        return "monitoring-and-audit"
    if bucket < 2_000:
        return "regression-evaluation"
    return "training-and-weak-supervision"


purpose = stable_partition("event-0042", "split-policy-v3")
assert purpose == stable_partition("event-0042", "split-policy-v3")
assert purpose in {
    "monitoring-and-audit",
    "regression-evaluation",
    "training-and-weak-supervision",
}
assert stable_partition("event-0042", "split-policy-v3", eligible=False) == "quarantine"

Turn a failure into a qualified test

A production failure is a lead, not a ready-made evaluation case. Before promotion into the suite from Chapter 52:

1. Minimize and sanitize the trace. Remove irrelevant personal or secret material while preserving the failure mechanism.
2. Reproduce the behavior in a controlled harness and record the model, prompt, retrieval, tool, policy, and interface version context.
3. Adjudicate what failed, which evidence was available, and whether the expected behavior is valid.
4. Freeze expected behavior at the right level. Prefer a property or outcome over a brittle exact string when several answers are acceptable.
5. Preserve provenance and independence in the evaluation manifest, and prevent the case and close variants from entering training.

The resulting test is a durable guard only while it remains valid, observable, and enforced. A stale test can encode an obsolete policy. A brittle test can reject a legitimate improvement. A team can also overfit to a visible suite or ignore a failing gate. Promotion therefore needs ownership, review dates, and retirement criteria as well as permanence.

Propagate rights through every derivative

The data engine inherits the privacy and provenance duties in Chapter 59. Its deletion lineage must connect a raw event to every derived feature, annotation, label, evaluation case, dataset shard, and training artifact that the policy says can be removed. A deletion request should create an auditable workflow with an owner, deadline, affected releases, completion evidence, and an explicit record of any lawful exception.

Purpose limits must travel with derivatives. Access to a raw event does not imply permission to place it in a training corpus, send it to an external labeler, or retain it in an evaluation case forever. Enforce use authorization at each promotion boundary, not only at collection. Dataset releases should name their allowed consumers so undeclared reuse is detectable. Hidden data dependencies and undeclared consumers are recurring sources of technical debt in production ML systems (Sculley et al. 2015).

Security controls also belong here. Deduplicate retries, validate event origin, rate-limit public feedback, separate tenant keys, scan for injected secrets, and quarantine suspected poisoned feedback. A user report is important evidence without being trusted code or a trusted label.

Measure the feedback loop, not just the dataset

Deployment changes which evidence will exist next. A recommendation changes what a person sees; a refusal changes what they try; a slow answer changes whether they wait. This is a performative feedback effect: the deployed system helps shape the future distribution it later observes (Perdomo et al. 2020).

Capture exposure, treatment assignment, model release, and serving policy version. Where a policy makes probabilistic choices, retain its propensity, the probability of the action under that policy. Without exposure context, absence of a click is uninterpretable because the user may never have seen the option.

A before-and-after improvement does not establish causality. The population, interface, season, or concurrent policy may also have changed. Use a randomized holdout where ethical and practical, a staged experiment, or a justified counterfactual design. Monitor guardrails as well as the target outcome. If an experiment is impossible, state the weaker observational claim and its likely confounders.

Useful operating metrics cover the pipeline rather than celebrating row count:

• eligible-event coverage, schema rejection, deduplication, redaction, and unknown-version rates;
• representative, inverse-probability weighted quality with uncertainty and slice coverage;
• acquisition yield, novelty, duplicate rate, and overlap with protected evaluation partitions;
• annotation disagreement, adjudication rate, blind-relabel error, abstention, queue age, service-level misses, and reviewer workload;
• deletion completion latency, unauthorized-use attempts, and unresolved lineage;
• downstream evaluation change, rollout guardrails, and experiment results.

No single metric certifies the engine. High annotation agreement can coexist with a wrong rubric, fast queue service with reviewer fatigue, and a larger dataset with worse coverage.

Release a data product, not a mutable table

A candidate release moves through explicit states: collect, validate, curate, assign, annotate, qualify, freeze, publish, monitor, and deprecate or delete. Publishing creates an immutable manifest. A later correction creates a new version rather than silently changing what an earlier model consumed.

Before release, exercise the pipeline with scenario tests for:

• a duplicate event and a retry carrying a conflicting payload;
• missing use authorization, a deletion request, and an unauthorized consumer;
• a stale outcome or an outcome joined to the wrong exposure;
• sampler outage, an incorrect inclusion probability, and biased slice coverage;
• schema drift, labeler drift, and an unreviewed guideline change;
• judge version change without expert calibration;
• a split collision, near-duplicate leakage, and adaptive leakage;
• poisoned feedback, adversarial instructions, and a cross-tenant record.

The production data release record is the handoff artifact. It names:

• release identity, owner, purpose, time window, and immutable manifest;
• intake schema, eligibility rule, versions, provenance, and source counts;
• sampling policies, inclusion probabilities, partition rule, and split salt identifier;
• annotation specification, workforce or judge versions, quality results, adjudication policy, and unresolved limitations;
• use authorization, allowed consumers, access control, retention, deletion lineage, and completed privacy review;
• scenario-test results, leakage checks, known incidents, and rollback or withdrawal procedure;
• downstream evaluation, experiment, and deployment decisions linked to the exact release.

Chapter 6 explains how to prepare a dataset for training, and Chapter 23 explains how to generate candidate examples. This chapter's engine governs the live evidence entering either process. The next chapter adds the operating contracts around cost, service levels, ownership, and tenant boundaries. A reliable production loop is therefore not a product harvesting its users. It is an evidence system whose purposes, uncertainty, rights, and release decisions can be inspected.

Further reading