Context Engineering

Context engineering decides what a model sees for one inference step. It does not replace the knowledge in the model's weights, and it is not a way to pour an application's entire state into a prompt. It constructs a bounded, versioned view of the current task from instructions, user input, retrieved evidence, conversation state, tool descriptions, and tool results. The useful question is therefore not “how much text fits?” but “which authorized tokens give this call the evidence and constraints it needs, without hiding conflicts or spending the space reserved for an answer?”

Prompt engineering remains part of that work: instructions still need clear language. The broader label became popular in 2025 as agent systems made the rest of the assembly visible (Anthropic 2025). The underlying mechanism is older. Brown et al. showed in 2020 that demonstrations in the input could change GPT-3's task behavior without a weight update (Brown et al. 2020). Retrieval, tools, and multi-turn agents then turned a hand-written prompt into a per-step data pipeline. Context engineering is the discipline of specifying and testing that pipeline, not a claim that prompting was renamed on a particular date.

Define the assembly contract

An application has more state than a model call should contain. The source corpus, event log, permissions, pending actions, and user profile remain in durable stores. The context is a temporary projection of that state. If the projection is wrong, a capable model can still answer the wrong question, follow stale instructions, cite obsolete evidence, or repeat an action that already happened.

Treat the projection as a versioned interface:

ContextSpec {
  model_revision, tokenizer_revision, chat_template_revision
  max_input_tokens, max_total_tokens, output_reserve
  instruction_policy_revision, tool_catalog_revision
  retrieval_spec_hash, compaction_policy_revision
  ordering_policy_revision, trust_policy_revision
  tool_server_identities, retention_policy_revision
}

ContextItem {
  item_id, kind, role, authority, content_hash
  source_uri, source_version, observed_at, expires_at
  tenant, acl_version, trust, token_count
  dependencies, supersedes, derived_from
  parent_call_id, mandatory
}

kind distinguishes instructions, task data, demonstrations, evidence, history, memory, tool schemas, and tool results. role records how the item is serialized for the model; authority records whose policy it may change. Those fields are deliberately separate. A database row or web page can appear in a user message or a tool result, but it does not thereby gain permission to override application instructions. Source and access-control fields make the item traceable to the authorized state from which it was derived.

The exact rendered request is part of the contract. A provider's chat template can add role markers, separators, tool-call records, and hidden formatting. Changing that template or tokenizer can change both behavior and token count even when the visible strings are identical. Hash the spec, record the ordered item IDs and content hashes, and make a new version whenever one of those transformations changes.

Count the request that will actually run

Token budgeting begins after serialization, not by adding word counts from the source documents. Let the final request be

$$\begin{aligned} X &= \operatorname{serialize}\!\left(F,\pi(S)\right), \\ T(X) &\le W_{\mathrm{in}}, \\ T(X)+R &\le W_{\mathrm{total}}. \end{aligned}$$

Here $F$ is the ordered sequence of mandatory items, such as the application instructions and active user request; $S$ is the selected subset of optional candidate items; $\pi$ is their ordering policy; and $\operatorname{serialize}$ is the model-specific chat and tool template. $X$ is the exact input sent to the model, $T(X)$ is its token count under the contracted tokenizer, $W_{\mathrm{in}}$ is the model or API's input limit, $W_{\mathrm{total}}$ is the combined input-and-generation limit, and $R$ is the output reserve. When an API exposes only one combined limit, the two constraints reduce to the second one.

The inequalities are admission checks, not targets. Filling every available token can raise prefill latency, enlarge the key-value cache, dilute useful evidence, and leave later tool results nowhere to go. A production policy normally sets smaller budgets for categories such as tool schemas, retrieved evidence, demonstrations, and history. Those allocations are policy choices to measure, not universal percentages. A multi-step tool loop also needs headroom for the assistant's tool request and the result that becomes input to the next call; reserving only the final prose answer can strand the run halfway through.

Never delegate overflow behavior to an undocumented backend default. Some interfaces reject an oversized request; others truncate or limit one field. The assembler should count the final payload, rebuild it under an explicit policy if it is too large, and fail visibly if mandatory content cannot fit. Silent clipping destroys reproducibility because the logged source items no longer match what the model received. Reconcile the preflight count with the provider's returned usage so tokenizer or template drift becomes observable.

Selection and placement are empirical policies

More context can help because the model can condition on task-specific information that its weights do not contain. It can also hurt because items compete for finite space and influence one another. In-context demonstrations are a simple example. GPT-3 established that zero-, one-, and few-shot prompts could produce useful behavior without gradient updates, but the effect varied substantially by task and model size (Brown et al. 2020). Later controlled experiments found that changing the order of the same four demonstrations could sharply change classification accuracy for the evaluated GPT-family models (Lu et al. 2022). Example identity, order, format, and label balance are therefore inputs to an evaluation, not decoration around a fixed method.

Irrelevant information is not neutral either. Shi et al. inserted distracting sentences into grade-school arithmetic problems and measured accuracy losses in the evaluated language models (Shi et al. 2023). That result does not prove that every extra passage lowers quality. It does show why retrieval score alone is insufficient: test distractors that are lexically related, plausible, stale, duplicated, or in conflict with the answer-bearing evidence.

Position can matter as well. Liu et al. varied the location of relevant information in multi-document question answering and synthetic key-value retrieval. Several models in that study performed best when the relevant item was near the beginning or end and worse when it was in the middle (Liu et al. 2024). “Lost in the middle” names that empirical pattern. It is not a law that every model, task, or context length follows, and it does not imply that the middle is unusable. Query placement, model training, task structure, and later architectures can change the curve.

The practical response is a permutation test, not a slogan such as “always put evidence at the edges.” Hold the selected items fixed, move the answer-bearing and conflicting items through the rendered request, and measure the change. Chat roles and templates may place text differently from the source order, so inspect the actual payload. Adopt an ordering rule only when that rule wins on the target workload and remains stable across important slices.

Preserve authority and provenance

The context is one token sequence to the model, but its contents do not have equal standing in the application. Platform policy constrains the application; application instructions constrain the task; the user supplies goals and data; retrieved documents and tool results supply evidence. The assembler must retain those distinctions even though the model can still misinterpret them.

Three rules follow:

1. Authorize before selection. Retrieval and memory lookup operate on the caller's authorized candidate set. Filtering after generation is too late; unauthorized content has already influenced the answer.
2. Keep data as data. Quote or structure retrieved text and tool output, attach its source, and state that embedded instructions are untrusted. Do not concatenate external text into a higher-authority instruction field.
3. Surface conflict instead of erasing it. Preserve source versions, timestamps, and disagreement among credible items. A summary that silently chooses one claim converts uncertainty into false certainty.

Roles, XML tags, and delimiters help the model parse these boundaries, but formatting alone is not a security boundary. Indirect prompt injection works precisely because untrusted data and executable instructions meet inside one model input. StruQ demonstrated a stronger separation by pairing a structured front end with a model specifically trained to ignore instructions in the data channel (Chen et al. 2025). Its result should not be generalized to an arbitrary model that merely sees tags. Authorization, least-privilege tools, confirmation for consequential actions, output validation, and the controls in Chapter 58 remain necessary.

Compaction is a lossy state transition

Long conversations and agent trajectories eventually exceed any fixed window. Dropping old turns loses state; carrying every turn forward consumes the budget; summarizing creates a new, fallible representation. Compaction is therefore a state transition with an explicit loss policy, not routine text cleanup.

Separate at least three kinds of state:

• Durable truth: source records, user approvals, tool effects, artifacts, and the event log remain outside the prompt. A summary may point to them but must not replace them.
• Working state: the current goal, completed steps, unresolved questions, constraints, and next action can be compacted into a checkpoint with source pointers.
• Ephemeral detail: exploratory prose, redundant tool output, and superseded drafts may be dropped once their useful result is represented elsewhere.

Every compacted checkpoint records its input range, policy version, content hash, source pointers, unresolved uncertainty, and actions already performed. Later calls can retrieve the original evidence when a decision depends on detail the summary omitted. This is the same separation used by Chapter 39: durable storage owns truth; the window holds a working set.

Compression quality is task-dependent. RECOMP trained extractive and abstractive compressors against downstream retrieval-augmented tasks and also learned when augmentation should be omitted (Xu et al. 2024). That is stronger evidence than assuming a generic summary preserves whatever the next step will need. Evaluate compaction on delayed questions, commitments, negations, multi-step dependencies, and recovery after failure. For irreversible actions, reconcile against the external system rather than trusting a prose summary.

Tools expand the candidate set, not the authority

A tool protocol determines how capabilities are described and invoked. It does not decide which capabilities a model should see, whether a call is authorized, or how much of a result belongs in the next context.

The Model Context Protocol (MCP) standardizes a client-host-server exchange for resources, prompts, and tools. In its architecture, the host coordinates clients, permissions, consent, and context aggregation; each client maintains a separate connection to a server (Model Context Protocol Contributors 2025). This removes bespoke message formats at an integration boundary, but it does not collapse all trust domains into one safe hub. Server identity, capability negotiation, authorization, schema versioning, result validation, and isolation still belong to the host and surrounding runtime.

A valid schema is not permission. Before execution, reauthorize the action, destination, and sensitive arguments against current policy. When a result returns, link it to the originating call ID, validate its error state, media type, size, and declared output schema, and retain the server and schema revision. Treat annotations supplied by a remote server as hints until local policy says otherwise.

Tool catalogs also consume tokens. Load the small, common set directly; expose larger catalogs through deterministic discovery; and fetch a full schema only when the task may need it. The discovery mechanism needs recall tests of its own. Hiding a necessary tool saves tokens and makes the task impossible, while showing many near-duplicate tools raises selection errors. Similarly, filter large results in code when the operation is deterministic, but keep the raw result and transformation trace outside the window so the filtered view can be audited.

Build one model call

A context assembler can now be stated as an algorithm rather than a prompt template:

Inputs:
  request q, caller identity a, durable state D, ContextSpec V

1. Resolve V to one model, tokenizer, chat template, and policy bundle.
2. Build mandatory items F from authorized instructions and the active request.
3. Collect candidate items C from history, memory, retrieval, and tool discovery.
4. Reject items outside a's tenant or access policy; preserve provenance on the rest.
5. Remove exact duplicates, mark conflicts and superseded versions, and expand
   required dependencies.
6. Select S from C under the category budgets; record why every item was kept,
   transformed, or omitted.
7. Order S with policy pi, serialize X = serialize(F, pi(S)), and count X with
   V's tokenizer and chat template.
8. If X violates an input or output-reserve constraint, rebuild under the next
   explicit reduction policy; never silently truncate mandatory content.
9. Reauthorize any proposed tool action at execution time; link each validated
   result to its call before assembling the next model request.
10. Emit X and a ContextManifest containing V's hash, ordered item IDs and hashes,
   token counts, transformations, omissions, and source pointers.

Here $q$ is the current task request, $a$ is the authenticated caller, $D$ is the durable application state, and $V$ is the selected context contract. $F$, $C$, $S$, $\pi$, and $X$ have the meanings defined in the budget formulation above. A ContextManifest is metadata for replay and diagnosis; sensitive raw content remains subject to its source retention and access policy.

The manifest makes absence observable. When a response lacks a fact, an operator can distinguish retrieval failure, authorization filtering, budget exclusion, compaction loss, serialization error, and model non-use. Without that record, all six failures look like “the model forgot.”

Evaluate the assembler, not just the answer

Freeze the model revision, tokenizer and chat template, context spec, corpus and permission snapshot, tool catalog, and evaluation cases. Compare against the current production policy, a no-added-context baseline, a full-context baseline, and, where possible, an oracle containing only the necessary authorized items. Then test four boundaries separately:

• Selection and use: task success, evidence recall after packing, supported claims, instruction adherence, and abstention when required evidence is absent.
• Robustness: permute relevant-item position; add duplicates, plausible distractors, stale versions, conflicting sources, long tool results, and demonstrations with different labels or order.
• Continuity and security: compact at different turns; resume from the checkpoint; revoke access; delete a source; inject instructions into retrieved pages and tool output; and verify that already completed actions are not repeated.
• Operations: serialized input and reserved output tokens, prefill and generation latency, cache reuse, retrieval and tool calls, context rebuilds, cost, and manifest coverage.

Use paired cases so only the context policy changes, and report uncertainty over tasks, the worst relevant position, and the mean rather than one average from unrelated workloads. Inspect material slices such as language, request length, tool family, permission cohort, and conversation age. A shorter context is not better if it drops decisive evidence; a longer one is not better if it adds latency and distractors without improving the task. Synthetic retrieval probes are useful diagnostics but do not establish long-context capability by themselves; HELMET combines recall, retrieval-augmented generation, many-shot learning, question answering, and summarization to expose different failure modes (Yen et al. 2025).

Context engineering closes the orchestration loop: retrieval, memory, tools, and agent state become one bounded, authorized input with a replayable manifest. The next part begins with Chapter 47 and asks what evidence is strong enough to show that this machinery works beyond a hand-picked example.

Further reading