Context Engineering: Compaction, Clearing, and Memory

The previous post was about limits. How much context the model can in principle hold, and how much of that it can actually use once "lost in the middle" and context rot eat into it. This post is about what you do about it.

Because once you accept that effective context is a scarce resource, the whole game changes. You stop treating the context window as a bucket you pour stuff into and start treating it as a budget you spend deliberately. Every token in there either earns its place or it's costing you money and, more importantly, it's costing the model attention it could be spending on the thing you actually want answered.

That reframing is what people mean when they say context engineering. It's the layer of decisions about what goes into the prompt on this turn, what stays from last turn, what got compacted into a summary two turns ago, what's sitting in a persistent memory store waiting to be pulled in if it becomes relevant. None of this is about the model itself. The model is frozen. This is all stuff that happens in the runtime, in the scaffolding around the model, in the code you write.

Context engineering has quietly become a big deal. If you use Claude Code or Cursor or any serious agent harness in 2026, there's a lot of machinery running between you and the model that's doing this kind of work. An agent loop that just appends everything hits the wall fast.

Tokens aren't free

Here's the frame I use. Every token in the context window has three costs:

1. A dollar cost. You pay for input tokens. At scale this adds up fast. Prompt caching offsets this a lot (more on that later), but the base rate is still per-token.
2. A latency cost. Prefill scales roughly linearly with context length. A 100k-token prompt is meaningfully slower to first-token than a 10k one, even on a modern engine.
3. An attention cost. This is the big one. Models have a finite attention budget. The more stuff you stuff into context, the more the model has to split attention across. The previous post covered this in detail. The short version is that effective recall degrades with length, and not uniformly.

So when you write the prompt, every token is a choice. You're spending from a budget. The default agent loop, just append everything (every user turn, every assistant turn, every tool call, every tool result, forever), spends that budget badly.

Look at turn 20 in that bar. History has eaten most of the window. There's barely room for the model's response, and all that history is competing for attention with the one thing the user actually asked about on this turn. That's not a hypothetical. That's what happens to any long-running chat that doesn't do something about it.

Prerequisites

This post assumes you've seen context windows and know what lost-in-the-middle is. It helps to have read the tool-loop post from arc 10 too, because "tool result" is going to come up a lot. And the prompt-caching piece draws on the KV cache material from arc 5.

You don't need any new math. Context engineering is not a mathematical subject. It's a set of patterns for packing information into a budget.

The core moves

There are really only a handful of moves, and most real systems combine several of them.

Compaction. Replace a chunk of verbatim history with a summary. The full thing was 4,000 tokens; the summary is 400. You lose fidelity. You keep the shape of what happened.

Clearing. Drop stuff you don't need anymore. Tool results are the big example. You asked the model to search flights; the API returned a 3,000-token JSON blob; the model read what it needed and wrote down the answer. The raw blob can go.

Isolation. Spawn a sub-agent with its own focused context. Let it thrash around calling tools and burning tokens in its own window. When it's done, it returns one sentence to the parent.

Retrieval. Keep a persistent store of past stuff outside the context window. On each turn, pull in just what's likely relevant.

Caching. Re-use the KV cache from the last turn so the prefix (system prompt, tool definitions, conversation so far) doesn't get re-prefilled. This isn't about reducing context size, but it's a lever on the latency and dollar costs of keeping context large.

Each of these is a different answer to the same question: how do I keep the useful stuff in the window and push the rest somewhere else?

Sliding-window summarization

The most common pattern. You pick a window (last N turns, or last K tokens) that stays verbatim. Everything older gets summarized into a running digest.

The trick is that the summary itself is a living document. When a new turn falls out of the window, you don't just regenerate the whole summary from scratch (that'd be expensive and would churn the cache). You append: you call the model with the existing summary and the turn that just aged out, and ask for an updated summary that incorporates it. That's a small call, and it keeps the summary stable enough that the rest of the conversation stays cache-hot.

In practice, Claude Code does something like this when a conversation gets long enough. Cursor's "conversation history" in agent mode follows a similar pattern. The window size, the summary prompt, the compaction threshold, all of those are knobs.

Now. There's a tension here. Summaries lose detail. If turn 7 contained a specific decision ("let's go with option B") and your summary compresses that to "discussed options," the decision is gone from the model's view. This is the most common failure mode of compaction, and it's why the summarization prompt matters a lot. A good one is biased toward preserving decisions, constraints, and named entities over narrative flow.

Tool-result clearing

This one is easy to overlook, and it's probably the single highest-leverage thing you can do for agents that use tools a lot.

The pattern is that the model calls a tool. The tool returns some large structured response. The model reads the response and, in its next turn, produces some short artifact that captures what it needed from that response. At that point, the raw tool result is dead weight. It's sitting in the transcript consuming thousands of tokens, and the model has already extracted what it needs.

{
  "flights": [
    { "id": "UA123", "depart": "SFO",
      "arrive": "JFK", "price_usd": 412, ... },
    { "id": "DL456", "depart": "SFO",
      "arrive": "JFK", "price_usd": 398, ... },
    { ... 23 more rows ... }
  ],
  "rate_limit": { "remaining": 47 },
  "timing_ms": 1842
}

Anthropic shipped a feature in late 2025 called context editing specifically for this. The runtime can retroactively replace old tool results with a short marker like [tool_result cleared] once the model has written its own notes. From the model's perspective on later turns, it's as if the raw blob never existed, but the note it wrote is still there.

I want to flag something important about this move, because it comes back later. What you're doing is maintaining an invariant: the model's own writing is more compact than the raw data it read. If that's true (and for well-designed agent prompts, it is), then clearing old tool results is free compression. You lose nothing the model cared about. You free thousands of tokens.

This is the same pattern as scratchpads, really. A scratchpad is a temporary workspace where the model writes intermediate results (thinking-tokens-style). Once the final answer is produced, the scratchpad is discardable. Same invariant.

Sub-agent context isolation

The move here: the parent agent's context is sacred. Don't pollute it with tool thrashing.

Concretely, when the parent needs to do something that would require a bunch of exploratory tool calls (read a directory, grep for patterns, summarize a repo), it doesn't do those calls itself. It spawns a sub-agent with a focused prompt ("summarize the PRs merged yesterday"), lets the sub-agent run its own tool loop in its own context window, and takes back only the final one-line or one-paragraph answer.

This is the agent-architecture version of encapsulation. The parent's context stays small because the child absorbed all the mess and then got destroyed. You can spawn several sub-agents in parallel for independent sub-tasks and merge their results; you can chain them; you can have them spawn their own sub-agents. The context costs compose additively in time spent and dollars but not in the parent's window size.

Claude Code's task tool works this way. So does Devin. So does more or less every serious 2026 agent harness, for the same reason: the alternative (one giant monolithic context that carries every tool call the agent ever made) doesn't scale past a few dozen turns before context rot ruins it.

Persistent memory

Everything above is about this conversation. But real systems often want state that survives across conversations. The user told you their name last week; you should remember that today without re-asking.

The 2026 pattern for this is a persistent memory store, usually backed by a vector index, with explicit write and retrieve operations. On each turn, you embed the current user message, look up the top-K most similar memories, and pull those into the prompt. Memories below some relevance threshold stay in the store and don't show up.

The way Anthropic's memory tool works in late 2025 is roughly this, plus a twist. The tool is exposed to the model: the model can choose to write memories, retrieve memories, delete memories, all as explicit tool calls. The store itself is just a directory of files that the model reads and writes. That matters because it means the model gets to decide what's worth remembering, not some heuristic outside the model. The model might decide "this user's deadline constraints are important, let me save them" and the note lands in the store for next session.

Cursor and Cline have a similar pattern under names like "memory banks" or "user rules": persistent notes that stay in the user's config and get prepended or retrieved into each session.

There's a subtle thing to notice here. All three of these memory systems (Anthropic's, Cursor's, Cline's) leave the store outside the context window. The model doesn't see all the memories. It sees the ones retrieval picked this turn. This keeps the window clean. The cost is that retrieval quality becomes part of your product: if retrieval misses the relevant memory, the model behaves like it never knew.

The protocol / implementation split, applied

Here's a handle I think is worth hanging onto, because it comes up everywhere in this arc.

The protocol is what the model sees: a sequence of messages, tokens, tool results. Nothing more. From the model's perspective, it gets a context window and produces a response. End of story.

The implementation is everything that decides what goes into that window: your sliding-window logic, your summarizer, your context-editing rules, your memory retrieval, your sub-agent spawner. This layer is not the model's problem. It's your problem.

Claude, GPT-4.5, Gemini 2.5, they all speak roughly the same protocol: messages in, response out. What makes a production system good is almost entirely the implementation side. Two systems using the exact same model can have wildly different long-conversation quality because one of them does real context engineering and the other just appends.

This is why I said in the intro that context engineering quietly became a big deal. It's invisible. The model gets credit for what feels like a smart long session, but a lot of that smartness is the context engineering wrapped around it.

Prompt caching as a context-engineering lever

One more piece before the worked example.

If you keep a big system prompt and a long conversation history around, every turn means re-prefilling all those tokens through the transformer. Even with a fast engine, that's real latency and real cost. Prompt caching (which most major providers now expose as a first-class feature) says: if the prefix of this turn's prompt matches the prefix of a recent turn, reuse the KV cache from that earlier prefill.

This is the KV cache hydration trick from arc 5, surfaced as a user-level product feature. It has direct consequences for how you design context:

• Keep the prefix stable. If your summary at the top of the prompt changes every turn, you blow the cache every turn. Structure the prompt so the volatile parts (new user message, new tool results) are at the end, and the stable parts (system, tool definitions, compacted history) are at the beginning.
• Batch compaction. When you do re-summarize, you're going to invalidate the cache anyway. So do it all at once, not incrementally. A single summary update is better than constant mini-updates.
• Isolate volatile regions. SGLang's RadixAttention and similar tricks let you cache trees of prefixes, so multiple branching conversations can share a common root.

The mental model I use: prompt caching makes keeping-context-large cheaper, but context engineering decides whether keeping-context-large is even necessary. The two are complementary. Use caching to make the stuff you chose to keep cheap; use engineering to keep only what's worth keeping.

Worked example

Let me make this concrete.

Imagine a customer-support conversation that goes 20 turns. The user has described a bug, uploaded a stack trace, walked through reproduction steps, discussed three possible fixes, decided on one, asked for implementation help, reviewed a code diff, asked for tests, and approved them. About 12,000 tokens of transcript by the end.

Now imagine we need to run that conversation under a tight 3,000-token context budget. Three strategies:

1. Full 12k. Just include everything. Baseline.
2. Naive truncate. Keep the most recent ~3k tokens, drop the rest. The prefix of the conversation (including the original bug description) is gone.
3. Compacted 3k. A ~500-token summary of turns 1–15 at the top, then turns 16–20 verbatim. Total around 3,000 tokens.

Now we ask a probe question designed to require memory of an earlier turn: "what did we decide on turn 7?" Turn 7 was the decision to go with fix B out of three options.

The pattern across the published lost-in-the-middle and decision-probe results lines up here. Truncation loses about half of its decision-probes because those decisions happened early in the conversation and got dropped. Compaction lands much closer to the full-context baseline. Not equal to it, but close.

The reason is straightforward. A good summary preserves decisions, constraints, and named entities, even when it loses narrative flow. A truncation preserves nothing outside its window. For long conversations where the important stuff is distributed across time, compaction beats truncation by a lot at the same budget.

Implementation sketch

Here's roughly the shape of a real compaction step, minus error handling. The point is that it's not complicated at the code level. The complexity is entirely in the prompt you use for the summarizer and in when you decide to trigger compaction.

import anthropic
 
client = anthropic.Anthropic()
 
SUMMARIZER_SYSTEM = """You compress an older segment of a conversation into a
dense note. Preserve: decisions made, constraints stated, named entities,
open questions. Drop: greetings, restatements, small talk. Output plain text,
under 400 tokens."""
 
def compact_if_needed(history, budget_tokens=3000, window=8):
    """Return (summary, recent) where summary+recent fits the budget."""
    if token_count(history) <= budget_tokens:
        return "", history  # nothing to do
 
    # Keep the last `window` turns verbatim; summarize the rest.
    old, recent = history[:-window], history[-window:]
 
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=500,
        system=SUMMARIZER_SYSTEM,
        messages=[{
            "role": "user",
            "content": "Compress this transcript:\n\n" + render(old),
        }],
    )
    summary = resp.content[0].text
    return summary, recent
 
# --- On every new user turn ---
summary, recent = compact_if_needed(history)
prompt_messages = (
    [{"role": "user", "content": f"[earlier conversation]\n{summary}"}]
    if summary else []
) + recent + [new_user_turn]

The summarizer prompt in this sketch is doing most of the work: "preserve decisions, constraints, entities; drop small talk" is the difference between a summary that keeps you at 86% probe accuracy and one that keeps you at 55%. The window size (window=8 here) is a knob you tune. And the decision to only recompute the summary when over budget, rather than on every turn, is what keeps the prefix cache warm.

Misconceptions

"More context is always better." This is the default assumption and it's wrong in practice. More context means more cost, more latency, and more attention competition. Past a certain point, adding tokens actively hurts quality because the model gets distracted. The right question isn't "can I fit this?" but "does this token earn its place?"

"Compaction is the model's job." It's not. The model doesn't compact anything. The model produces a response. Compaction happens in the runtime, between turns. Treating compaction as something the model does in one call tends to produce confused designs where the same call is both answering the user and trying to summarize itself. The pattern that works is separation: one pipeline for the conversation, one explicit summarizer call that updates the summary when triggered.

"Memory means the model remembers you." The model doesn't remember anything between turns. What "memory" actually is, in every 2026 system I know of, is a persistent store outside the model that the runtime retrieves from and places into the prompt. The model sees the retrieved memories as part of its input on this turn, same as the system prompt. If retrieval misses, the model behaves exactly like it never knew. This matters for how you debug "why didn't the model remember X?": the answer is almost never about the model.

What's next

That closes out Arc 9. The arc started at cosine similarity over a vector store and ended at runtime systems that decide, turn by turn, which tokens the model gets to see. If you read it front to back, you now have the mental model for why retrieval-augmented generation looks the way it does in 2026, and why context engineering quietly became half the work.

Arc 10 picks up on the output side, starting with a short history of agents from ReAct to 2026: how the loop around the model evolved from a single prompt-reply into the tool-calling, context-managing systems the rest of the arc is about to take apart.

Additional reading (and watching)

• Anthropic. (2025). Context editing and the memory tool. Announcement post for runtime-level context editing and the file-backed memory tool. The primary source for the tool-clearing and memory-tool patterns described above.

• Cline. (2025). Memory Bank documentation. The pattern of a user-level persistent notes directory that gets loaded into the agent prompt each session. Cursor has an analogous "rules" feature.

• Anthropic. (2024). Prompt caching. Provider-level feature exposing KV-cache reuse across requests with the same prefix. The primary lever for making long stable prefixes cheap.

• Zheng, L., et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. Introduces RadixAttention, a tree-structured prefix cache that generalizes prompt caching across branching conversations.

• Liu, N., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. The paper that made the attention-degradation-with-length problem legible. Motivates why compaction works: short summaries land in high-attention positions.

• OpenAI. (2025). Responses API documentation. Server-side stateful conversation threads, with provider-managed context engineering as part of the abstraction.