Part VI · Chapter 39

Memory Systems

AuthorChangkun Ou

Reading time~21 min

An agent that runs longer than a single context window has to remember what it did and survive losing the machine it did it on. The state it leaves behind has three forms: the durable record of a session, the workspace it mutated, and the memory it carries across sessions. Session logs must record intent separately from result; branching a conversation is cheap while branching a workspace is not; and a shared vector store is a data-breach risk, not merely a relevance bug.

Figure 39.1. Schematic of useful recall as memory budget grows. Raw logs, summaries, and retrieval stores fail differently: the best memory system is not just the largest one. Idealized curves, not measured data.

A commit that runs twice

Start with the failure, because it names the whole subject. A runner prompts the model, the model asks to run git commit, the sandbox runs it and the commit lands, and then the runner crashes in the window between the effect firing and its record reaching durable storage. Recovery reads the last durable state, which is pre-commit, and the agent replays: it issues the same git commit again.

Figure 39.2. A crash between effect and persistence makes replay re-issue a non-idempotent commit, the intent-versus-result gap in its purest form.

This is not an exotic edge case. A harness has to survive any of four routine events: a harness crash, a replica reschedule, a sandbox loss, or a user resuming a session days later. All of them are the normal operating regime of a long-running agent, and reported agent retry rates of 15 to 30 percent make the replay above a regime to design for, not an accident (Fast.io 2025). The core difficulty it exposes is the gap between a side effect and its commit. A durable log records intent ("call git commit") and, separately, result ("commit succeeded"). If the harness crashes between those two writes, recovery re-issues the operation. For an idempotent operation that is harmless. For a non-idempotent one, a second commit, a duplicate ticket, a repeated payment, the store and the external world diverge silently, and nothing in the log can tell you it happened.

Surviving these events means writing state to a durable store, and the hard question is not where to put that state but what to record. The state an agent leaves behind comes in three shapes, and each fails in its own way. The session log is append-only intent, and its characteristic failure is the intent-versus-result gap above. Mutable sandbox state lives in the workspace, where durability is what breaks: a volume that cannot follow a pod across a zone boundary, a snapshot taken too rarely. That leaves long-term memory, the curated and retrievable store that outlives a session, whose failure is tenancy, because a retrieval that crosses a user boundary is not a wrong answer, it is a data breach. One small contract spans all three, append, read, snapshot, fork, applied to state shapes that are anything but uniform. The design discipline is stating plainly what each store does not guarantee. The three stores that follow keep that question in view: what each one buys, and what it refuses.

The session log: three ways to record a step

The session log can be built three ways, and the meaningful difference between them is what they record relative to the effect, which fixes where the intent-versus-result gap lands. Figure 39.3 places all three on that one axis.

Figure 39.3. Three durability models differ in what they record relative to the effect, which fixes where the intent-versus-result gap sits.

The append-only event log writes every user message, model response, tool call, and tool result to a durable store. The log is the sole recovery artifact: any replica can serve any session by replaying it. This buys clean replay, clean audit, and transport-agnostic recovery, and it is the model that exhibits the intent-versus-result gap in its purest form. From the log alone, "requested but never ran" and "ran but never recorded" are identical: a tool call with no result. That is the gap the opening crash walked into.

The durable step log closes the gap by adding a third record between intent and result: a durable step boundary written before the effect executes. Replay skips any step whose outcome is already durable, so "we asked to do X" and "X completed" become two separate, individually-durable facts, and replay can ask "did this step already complete?" instead of assuming it did not. This is the answer durable-execution frameworks settled on, and the evolution is worth tracing. The plain event log gave clean replay but blurred intent and result. Restate (Ewen et al. 2025), Temporal (Davis 2025), and Inngest (Poly 2026) journal each step, run the effect, then record its outcome, formalizing the distinction the event log blurs. Temporal's own framing is that probabilistic LLM behavior makes naive retry logic insufficient, which is exactly the case durable journaling is built for (WorkOS 2025). By 2025 this crossed into the mainstream: AWS Lambda durable functions, Cloudflare Workflows, and Vercel's Workflow DevKit all shipped with AI-agent use cases as primary framing (Poly 2026). The pattern is no longer framework-specific.

The filesystem-as-state model takes the opposite extreme: the working context is ephemeral and the filesystem is the sole authoritative record of task state (Zhou and others 2026). The event log demotes to a secondary log of intent; the filesystem holds the truth, usually through an opinionated convention (progress.md, plan.md) the agent maintains. This sidesteps the intent-versus-result gap for anything the filesystem can represent, the file either has the new content or it does not, but it relocates the gap rather than removing it: a git push, an external API call, an MCP server-side cursor advance are still outside the filesystem's reach.

From a list to a tree

The session log records steps, but its shape evolved separately from its durability. A linear append-only log treats a session as a list, and to back out you start over. Production harnesses moved past that. Rewind and truncate jump back to a checkpoint and discard the tail, rolling the workspace back to match, as in Claude Code's /rewind (Anthropic 2025) and Cursor's per-edit checkpoints (Cursor 2025). A forkable tree goes further: multiple live branches from any checkpoint, all preserved. LangGraph's time-travel model treats checkpoints as a DAG with explicit branch IDs (LangChain 2025), Claude Code's /fork spawns a child session from a shared history point (Anthropic 2025), and Replit Agent (Replit 2025) and OpenAI's Codex CLI (OpenAI 2026) ship their own variants. The session stopped being a list and became a tree, or at least a log with a movable head. That tree is cheap to build only because conversation state is a pointer structure, an assumption the workspace will break in the next section.

The workspace: partial durability, expensive branching

The workspace is the sandbox's mutable filesystem. It outlives individual tool calls but is not on the durable log, and that distinction is the source of two failures the session log does not have.

Workspace persistence is partial durability, not full recovery. Persisting the workspace across pod lifecycle events has its own failure modes. The common Kubernetes pattern mounts a PersistentVolumeClaim at /workspace; the dominant failure is zone scoping. A ReadWriteOnce PVC is backed by a zonal block device that is physically attachable only within its own availability zone, so when a pod reschedules across zones the volume cannot follow and recovery falls back to a full re-bootstrap (Rack2Cloud 2026; Kubernetes 2023). When a pod is deliberately destroyed (teardown, timeout, scale-down) the PVC is often deleted with it, and uncommitted work is gone: Gitpod issue #9544 is the archetype, a workspace timeout fired before the final sync completed and a user lost roughly two hours of work (Gitpod 2022). The alternative, checkpointing the complete state on a cadence, trades snapshot frequency directly against storage overhead, and the frequency knob is load-bearing because it sets the maximum work a crash can erase. Replit's snapshot engine captures filesystem and conversation context together and reports recovering from hourly OOM crashes without data loss (Replit 2025; Replit 2025). Any design that claims "sessions survive indefinitely" owes an explicit account of which failure modes it covers and which it does not.

Change the snapshot interval below and watch the two curves move in opposite directions: tighter snapshots erase less work on a crash but write more often.

import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(0)
intervals = np.array([1, 2, 5, 10, 15, 30, 60])  # minutes between snapshots
trials = 20000
lost = []
for iv in intervals:
    crash = rng.uniform(0, iv, trials)  # crash time within the interval
    lost.append(crash.mean())           # work since last snapshot, in minutes
lost = np.array(lost)
writes = 60 / intervals                 # snapshot writes per hour
for iv, l, w in zip(intervals, lost, writes):
    print(f"every {iv:2d} min: ~{l:4.1f} min lost, {w:5.1f} writes/hour")
fig, ax1 = plt.subplots()
ax1.plot(intervals, lost, "o-", color="#c0392b", label="expected work lost (min)")
ax1.set_xlabel("snapshot interval (min)"); ax1.set_ylabel("work lost (min)")
ax2 = ax1.twinx()
ax2.plot(intervals, writes, "s--", color="#2980b9", label="writes/hour")
ax2.set_ylabel("snapshot writes/hour")
fig.legend(loc="upper center"); plt.show()

Branching splits cheaply on conversation and expensively on the workspace. Branching conversation state is a pointer operation: the log is a tree of event IDs and forking a tree is cheap. Branching the workspace is not, because the workspace is a real filesystem and two branches cannot both own it.

Figure 39.4. Forking conversation state is a cheap pointer operation, while the shared workspace forces a choice between interfering branches and per-fork clone cost.

As Figure 39.4 shows, the choice collapses to a dilemma: forks that share the workspace race on it, branch A's writes are visible to branch B, and forks that clone it pay the clone cost. Three failure modes follow. A branch cannot fork the external systems it touches, git remotes, issue trackers, MCP cursors, so branchable and unbranchable state drift apart. Merge-back is structurally unsolved: files have a common ancestor at the checkpoint, so a three-way merge applies, but two divergent conversations have no merge base because the content is a reasoning trajectory, not editable lines, which is why the agent, not the harness, has to resolve a file merge. And orphan branches pin workspace and session-log resources until a TTL reclaims them, so without one the leak accumulates silently. Rewind alone suffices for interactive coding; forkable trees pay off when users explore alternatives in parallel or an evaluation pipeline replays a session with a changed prompt.

Constraint arrow

Whether branching and snapshotting are cheap or ruinous is decided one layer down, by the storage substrate, not by the harness. A PVC has no native cheap-clone primitive, so a fork on PVC must copy the whole workspace. Copy-on-write overlays and content-addressed snapshots duplicate only the divergent blocks, so a fork costs what diverged. Firecracker snapshots serialize an entire microVM and restore in tens to hundreds of milliseconds (Amazon Web Services 2025), which makes per-fork snapshotting affordable that would be ruinous on a PVC. The clone cost is bounded by the substrate, not by the harness, so the session-tree feature in the layer above is enabled or priced out by a storage choice made below it.

Long-term memory: useful because retrievable, dangerous for the same reason

A session is bounded by context compaction (Chapter 35): the context window fills, gets summarized, and the early turns fall out of the model's reach. An agent that is supposed to know a user across weeks therefore cannot rely on the context window alone. It needs a separate store, durable and queryable, that the harness loads into context on demand. That store is memory, and it is a different system from the session log: the log is append-only intent, memory is curated and retrievable.

The shape of memory

Memory has a structure worth naming. Short-term memory is the session context within one conversation. Long-term memory splits into episodic (past interactions), semantic (facts about the user or domain), and procedural (learned workflows). External knowledge, the corpora behind retrieval-augmented generation (Chapter 44) and knowledge graphs, sits alongside but is a different thing: memory is per-user, curated, and writable, where RAG retrieves from a shared corpus the agent does not own.

Figure 39.5. Memory splits into short-term and long-term (episodic, semantic, procedural), distinct from the shared external knowledge that RAG retrieves.

The base layer is a vector store plus embedding retrieval: each memory is turned into a vector (a list of numbers that captures its meaning), stored, and recalled by finding the nearest vectors to a query. Pinecone (Pinecone 2025), pgvector (pgvector 2025), and Chroma (Chroma 2025) are representative substrates. Above them sit opinionated memory frameworks, Letta (Letta 2025) (the successor to MemGPT (Packer et al. 2023)), Zep (Zep 2025), and Mem0 (Chhikara et al. 2025), that decide what to write, when, and how to compact it. Knowledge graphs model entities and relationships explicitly, which suits compositional queries but is harder to populate automatically, and a temporal knowledge graph adds the time dimension so a fact can be valid for a period and then superseded.

The category arrived the way the other two stores did, by a research artifact hardening into a product. MemGPT (Packer et al. 2023) framed the LLM as an operating system that pages memory in and out of a bounded context, and its successor Letta (Letta 2025) turned that into a hosted primitive. Zep (Zep 2025) built memory on a temporal knowledge graph, and Mem0 (Chhikara et al. 2025) optimized for low latency and token cost. The category is young enough that its benchmarks are still contested.

A now-mainstream alternative skips the index entirely and stores memory as plain files the agent reads and edits with its ordinary tools. Anthropic's memory tool gives the model a /memories directory it views, creates, and edits across sessions, with the application executing each file operation client-side (Anthropic 2025); the CLAUDE.md convention in coding agents is the same idea maintained by hand. Recall becomes an agentic read or grep instead of a similarity query, and curation becomes the model's own editing rather than a framework's write policy. The trades this chapter closes on shift with it: file memory needs no embedding-model commitment, so there is no reindex cliff, and a per-tenant directory is directly inspectable and erasable, at the cost of recall that reaches only what the agent thought to write down and chose to read back.

What's contested

Cross-vendor memory benchmarks are not yet trustworthy. Mem0 reports a 26 percent LLM-as-Judge gain over OpenAI memory on LOCOMO (Snap Research 2024) with large latency and token-cost reductions (Chhikara et al. 2025). Letta's counter-benchmark reports a higher LoCoMo score with disputed methodology (Letta 2025). Zep reports its temporal knowledge graph leading on a different benchmark (Zep 2025). The vendors agree on neither the metric, the dataset split, nor the baseline, so treat any cross-vendor memory benchmark as contested until it is independently reproduced. The useful signal is the shape of the trade, recall quality against latency and token cost, not the headline number.

What memory refuses to guarantee

Memory trades cost against simplicity, and write-time safety against read-time convenience. Memory must be either loaded ambiently into every prompt (simple, expensive) or exposed as a tool the agent queries on demand (cheaper, scales better, but the agent has to know when to ask). Changing the embedding model invalidates the existing index and forces an expensive reindex, so the embedding choice is a long-term commitment.

The sharpest trade is tenancy, and it is what makes a shared vector store a data-breach risk rather than a relevance bug. The same property that makes memory useful, that retrieved content lands directly in the model's context, makes it dangerous, because that content is indistinguishable from instructions the user or harness put there. A top-k similarity query (fetch the k closest stored vectors) against a shared vector store without an enforced tenant filter can return another tenant's data, and the defense has to live in the index, not in application code. Partition at write time, per-tenant namespaces, pgvector row-level security or separate schemas, so a query physically cannot reach another tenant's vectors. Attributing every chunk and filtering after the query is weaker, because filter-at-read-time is one bug away from a leak, whereas partition-at-write-time fails closed. This is where memory consumes the identity model directly (Chapter 56): tenancy is sourced from identity, and partitioning enforces it.

A second tenancy risk is specific to memory and has no analogue in the session log: because the agent writes memory based on user messages, a malicious user can craft messages that plant persistent misinformation. This is the prompt-injection problem moved into the memory store, where it survives across sessions. A misretrieved or poisoned record lands in the prompt and shifts the agent's behavior in ways that are hard to debug, because the trace shows retrieval succeeding: the injected text is not an error, it is a correct retrieval of wrong content.

Two properties to nail down before the first crash

Both of the following are invisible until exactly the wrong moment, which is why they belong in the design and not in the postmortem.

The first is the reconciliation gap, the operational form of the opening crash. If the sandbox holds mutable state outside the durable log, the harness must reconcile the two on recovery. For idempotent operations it does not matter; the result is the same whether the operation ran once or twice. For non-idempotent ones, append to a file, increment a counter, create a commit, call a side-effectful API, the harness needs either a step-log boundary or an idempotency key threaded through the tool. The journal records the step's intent and its key before the effect fires; on replay the harness performs check-then-act against that key, or against the external system's own dedup, git's ref state, a payment processor's idempotency header, rather than blindly re-issuing. The step log does not make external effects idempotent on its own. It gives you the hook to make them so.

# Idempotency-keyed step, conceptual sketch.
def run_step(journal, key, effect):
    if journal.has_outcome(key):     # replay: skip a completed step
        return journal.outcome(key)
    journal.record_intent(key)       # durable boundary BEFORE the effect
    result = effect()                # the non-idempotent side effect
    journal.record_outcome(key, result)
    return result

The second is erasure reach. A right-to-erasure request has to cascade across every store that holds a user's data, and memory is the painful case: framework-managed memory may not expose a delete API at all, so a store chosen for recall quality can quietly become a store you cannot erase. Partition-at- write-time helps here too, because a per-tenant namespace is a unit you can drop wholesale. The broader audit and compliance machinery, tamper-evident logs, data residency, the regulatory regimes, belongs to Chapter 56; the memory-specific obligation is to keep every record attributable to a tenant and erasable by tenant.

For the memory store itself, the base implementation is a vector index (pgvector (pgvector 2025) inside an existing Postgres is the low-operational-overhead default; Pinecone (Pinecone 2025) or Chroma (Chroma 2025) when a dedicated store is warranted), wrapped either by a framework (Letta (Letta 2025), Zep (Zep 2025), Mem0 (Chhikara et al. 2025)) that manages write and compaction policy, or by a thin custom layer when the policy is simple enough to own. The harness (Chapter 41) is what loads the result into context, ambiently or through a retrieval tool, closing the loop back to the session that the memory outlived.