Context Engineering: Compaction, Clearing, and Memory

The previous post separated an accepted input length from performance on a task. This post follows a conversation as it grows: what should remain in the next prompt, and what can live elsewhere?

A transcript contains decisions, source material, failed attempts and temporary tool outputs. Keeping all of it costs space and processing; removing the wrong detail can make a later question impossible. Context engineering is the work of making that choice explicit.

The application decides what to retain verbatim, what to summarize, and what to store for later retrieval. A model may help summarize or select evidence, but this input-management layer does not itself update the model’s weights.

Tokens aren't free

Context size affects cost and performance:

1. Input cost. Hosted APIs commonly bill by token, with separate cache rates.
2. Processing cost. Longer uncached inputs add work. Dense attention has a quadratic sequence-length term, while other parts are roughly linear; observed prefill latency depends on the implementation and hardware.
3. Task performance. More context can supply evidence or introduce distractions. Measure its effect on the questions the application must answer.

Appending every message is a workable starting point for a short task. Over a long session it can eventually exhaust the available input budget. Decide which evidence must remain available before choosing what to remove.

At the assumed turn-20 snapshot, only 302 tokens remain. A runtime should act before that point if its output reserve is larger. A fit check must count the actual serialized input and reserve the intended output capacity.

Prerequisites

This builds on context windows and prefix caching. A tool result is the data returned when the runtime executes a model-requested operation; Arc 10 develops that loop.

The core moves

Compaction. Replace a chunk of verbatim history with a summary. For example, replace 4,000 tokens of history with a 400-token summary. The summary saves space but may omit details needed later.

Clearing. Remove old data after preserving what remains useful, while retaining a way to fetch the original again. A flight-search response can often be replaced by a compact note, but that note may omit fare restrictions or dates needed later.

Isolation. Give a subagent a focused context and return a scoped result with supporting references. Its full exploratory transcript need not be copied into the parent prompt.

Retrieval. Keep a persistent store of past stuff outside the context window. On each turn, pull in just what's likely relevant.

Caching. Re-use the KV cache from the last turn so the prefix (system prompt, tool definitions, conversation so far) doesn't get re-prefilled. This isn't about reducing context size, but it's a lever on the latency and dollar costs of keeping context large.

Sliding-window summarization

One pattern keeps the latest complete turns or exchanges verbatim and summarizes older ones. “Complete” matters: a tool call and its matching result should not be split into an invalid message sequence.

The next summary should incorporate the previous summary as well as newly compacted history. Otherwise old decisions disappear after the second compaction. Updating a summary near the beginning of the prompt invalidates cache reuse after the first changed token; small edits do not keep the whole prefix cache-hot.

Batch updates when the budget requires them, then keep that summary stable between compactions. The exact threshold and retained window depend on output reserve, tool payloads and task requirements.

If turn 7 contained a specific decision ("let's go with option B") and your summary compresses that to "discussed options," the decision is gone from the model's view. This omission makes the next decision harder even if the summary reads fluently. A good one is biased toward preserving decisions, constraints, and named entities over narrative flow.

Tool-result clearing

Tool outputs can dominate a transcript, especially search results and file contents. Clearing is useful when those payloads are no longer needed verbatim.

After a tool result arrives, preserve task-relevant facts and source handles before removing its bulk from future prompts. The original can remain in storage even when it is absent from the model’s current input.

{
  "flights": [
    { "id": "UA123", "depart": "SFO",
      "arrive": "JFK", "price_usd": 412, ... },
    { "id": "DL456", "depart": "SFO",
      "arrive": "JFK", "price_usd": 398, ... },
    { ... 23 more rows ... }
  ],
  "rate_limit": { "remaining": 47 },
  "timing_ms": 1842
}

Anthropic’s context-editing API supports clearing older tool results under configurable triggers and retention rules. It can replace cleared content with placeholders.

A note can preserve the cheapest displayed flight while losing its baggage policy, currency, itinerary or freshness. Store enough provenance to inspect those omissions when needed.

Treat reasoning or scratchpad blocks according to the provider’s message requirements. They are not generally interchangeable with ordinary tool results, and indiscriminate removal can break a valid continuation.

Sub-agent context isolation

For a bounded investigation, the parent can dispatch a focused prompt such as “summarize the PRs merged yesterday.” The child runs its own tool loop and returns findings with source references. The harness decides how much parent context the child inherits and what comes back.

Include findings, unresolved questions and source references when those matter. Isolation reduces what is copied into the parent’s context, not total compute or billing.

Persistent memory

Everything above is about this conversation. But real systems often want state that survives across conversations. The user told you their name last week; you should remember that today without re-asking.

Persistent memory can use files, structured records, lexical search or vector retrieval. A memory’s relevance changes with the current request; similarity is not an enduring property of a note.

Anthropic’s memory tool exposes operations over a file-backed memory directory, with storage implemented by the client. The model can request reads and edits.

Rules and memory-bank files are other ways to supply persistent context. Some are loaded directly at session start; others are selected or read on demand.

For debugging, trace whether the right note exists, whether it was selected, and whether the model used it correctly. Those are separate failure points.

The protocol / implementation split, applied

The API accepts structured messages and tool results; the model ultimately processes a provider-specific representation. Message schemas, role handling and tool protocols differ across providers.

The implementation is everything that decides what goes into that window: your sliding-window logic, your summarizer, your context-editing rules, your memory retrieval, your sub-agent spawner.

Two applications using the same model can produce different results because they select different evidence and retain different decisions.

Log the assembled input and the transformations that produced it. That makes a forgotten decision easier to locate than treating the whole session as opaque model behavior.

Prompt caching as a context-engineering lever

If you keep a big system prompt and a long conversation history around, an uncached turn must prefill those tokens through the transformer. Prompt caching can reuse the KV cache for a matching prefix from an earlier prefill, reducing repeated work.

This is prefix-cache reuse from arc 5, surfaced as a user-level product feature. It has direct consequences for how you design context:

• Keep stable content first. Prefix reuse stops at the first change; appending content preserves more reusable work than editing an early summary.
• Batch necessary compaction. A changed summary invalidates the following prefix, so avoid regenerating it gratuitously. Budget and fidelity take precedence over a cache hit.
• Account for cache lifetime. Eviction, expiry and provider rules can turn a presumed hit into another write or ordinary prefill.

Worked example

Consider a support conversation where turn 7 selects fix B because it preserves the existing API. Later turns review code and test results.

There are several ways to assemble the next input:

1. Keep all available history, if it fits.
2. Keep only recent turns, losing the decision.
3. Keep recent turns plus a specific summary of the decision.
4. Keep recent turns plus a vague summary that says only “discussed fixes.”

Ask “what did we decide on turn 7?” and inspect whether each retained view supplies the decision and its reason.

Retain the source transcript so the application can recover details omitted from the summary.

Evaluate compaction with questions about actual decisions, constraints, identifiers and unresolved work. Compare multiple compaction cycles, since a detail can disappear when one summary is summarized again.

Implementation sketch

This provider-independent function compacts only at complete exchange boundaries. Its injected counter must count the complete serialized input, including system/tool framing, the summary and retained exchanges. The input budget is the total context limit minus reserved output capacity. The small word-count example below omits that framing for illustration. It returns only after checking the result; it raises if the retained text still does not fit.

def compact(summary, exchanges, budget, count_input, summarize, keep=2):
    """exchanges contains complete text exchanges, not individual tool messages."""
    if budget <= 0 or keep < 1:
        raise ValueError("positive budget and retained exchange count required")
    if count_input(summary, exchanges) <= budget:
        return summary, list(exchanges)
    if len(exchanges) <= keep:
        raise ValueError("recent exchanges alone need another budget strategy")
 
    old, recent = exchanges[:-keep], exchanges[-keep:]
    # Include the old summary so repeated compaction does not forget it.
    updated = summarize(summary, old)
    if not isinstance(updated, str):
        raise TypeError("summarizer must return text")
    if count_input(updated, recent) > budget:
        raise ValueError("summary plus recent exchanges still exceeds budget")
    return updated, list(recent)
 
# A deterministic teaching example using WORDS, not model tokens.
# Replace this counter with the deployed model's full-input token counter.
def word_count(summary, exchanges):
    return len((summary + " " + " ".join(exchanges)).split())
 
history = [
    "User asks which fix preserves the API. Assistant compares A and B.",
    "User chooses fix B because it preserves the API.",
    "Assistant runs compatibility tests. User asks for results.",
]
summary, recent = compact(
    "", history, budget=24, count_input=word_count, keep=1,
    summarize=lambda previous, old: "Decision: fix B; preserve API.",
)
assert word_count(summary, recent) <= 24
print(summary, recent)

A real summarizer call replaces the deterministic callback. Passing a length instruction to a model is not a fit guarantee: count its actual output with the retained exchanges. Add the new user turn once, include previous summary state, and validate complete tool exchanges before sending.

Misconceptions

"More context is always better." More context may add evidence or distractions. Measure task success and cost; retain enough detail to support the next action.

"Compaction happens automatically inside the model." The runtime or provider chooses what context to replace and when. A model may generate the summary, but the application still needs to preserve state and validate the resulting request.

"A stored memory must be remembered." A note outside the prompt helps only if it is retrieved or otherwise supplied and then used correctly. Check storage, selection and generation separately.

What's next

Arc 10 picks up on the output side, starting with a short history of agents from ReAct to 2026: how the loop around the model evolved from a single prompt-reply into the tool-calling, context-managing systems the rest of the arc is about to take apart.

Additional reading (and watching)

• Anthropic. (2025). Context editing and the memory tool. Announcement post for runtime-level context editing and the file-backed memory tool. The primary source for the tool-clearing and memory-tool patterns described above.

• Cline. (2025). Memory Bank documentation. A persistent notes pattern with instructions for reading and updating project context. The harness and prompt determine when those files are read.

• Anthropic. (2024). Prompt caching. Provider-level feature exposing KV-cache reuse across requests with the same prefix. The primary lever for making long stable prefixes cheap.

• Zheng, L., et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. Introduces RadixAttention, a tree-structured prefix cache that generalizes prompt caching across branching conversations.

• Liu, N. F., et al. Lost in the Middle. Task performance varies with evidence position; it is not a direct demonstration that summaries occupy high-attention positions.

• Anthropic. Context editing and memory tool. API behavior, triggers and client-managed file storage.