Managing the Window

The context window as resource

The context window is not a buffer you fill and forget. It is a finite, priced resource that directly shapes the quality and cost of every inference call. How you fill the context window — what you include, in what order, at what granularity — is as important as the model you choose and the prompt you write.

Modern frontier models have large context windows — 200K tokens for Claude 3.5 Sonnet, 1M for Gemini 1.5 Pro. This creates a seductive mistake: treating the context window as a buffer into which you dump everything and let the model sort it out. Large windows are technically capable of this, but the costs compound quickly. At $3 per million input tokens, a 100K token context costs $0.30 per request before the model has produced a single output token. A system making 1,000 requests per day with this context spends $300/day on input tokens alone.

Beyond cost, context density affects quality. The lost-in-the-middle phenomenon — where the model attends less reliably to content in the middle of a long context — is well-documented. Filling a context window with irrelevant content degrades performance on the relevant content. Precision in what you include is not just a cost optimization; it is a quality optimization.

Tokens are not uniform

Not all tokens in the context window have equal weight. Content near the beginning and end of the context receives stronger attention than content in the middle. Instructions placed prominently are followed more reliably than instructions buried in the middle of a dense context. This has direct implications for how you structure your prompts: put what matters most at the top, and reinforce critical constraints near the end.

What to put in context

Context engineering is the discipline of deciding what to include in the context window, in what order, and at what level of detail. The goal is maximum relevant information per token while keeping total token count within budget.

The priority stack

1. Instructions. The system prompt and task-specific instructions. Never compressed. Always included. These establish the frame within which all other context is interpreted. If you have token budget pressure, compress everything else before touching the instructions.

2. Retrieved documents. Documents retrieved from a knowledge base or search system relevant to the current query. Include enough context around each retrieved chunk to make it comprehensible, but trim boilerplate aggressively. The first paragraph of a legal document is often a header and recitals that add nothing — start from the relevant section.

3. Conversation history. Prior turns in the conversation. The most recent turns matter most; distant history can often be summarized or dropped. If the conversation has been going for fifty turns and most of the early turns are no longer relevant to the current task, compress them.

4. Current request. The user's current input. Always included, always last. Placing the user's request immediately before the model's response position takes advantage of recency effects.

What not to include

Do not include information the model already knows. The model's training data contains extensive knowledge of common programming patterns, general domain knowledge, widely-known facts. Including this information in context adds tokens without improving performance.

Do not include entire documents when you need a section. Fetch the relevant section, not the whole document. A contract might be 50 pages; the relevant clause is one paragraph. Chunk granularly and retrieve precisely.

Prompt caching

Prompt caching is Anthropic's mechanism for reusing the computed KV cache of a prompt prefix across multiple requests. If your system prompt, tool definitions, and reference documents are the same across requests, you pay the full input token cost only on the first request. Subsequent requests with the same prefix hit the cache and pay 10% of the normal input token price for those tokens.

How it works

You add cache_control: { type: 'ephemeral' } to any content block to mark it as a cache breakpoint. The API caches everything up to and including that block. You can set up to four cache breakpoints per request. On subsequent requests, if the content before a cache breakpoint is byte-for-byte identical, the cache hits and you pay 10% of the input price for that prefix. The cache TTL is five minutes — any cache hit within the TTL resets the five-minute clock, keeping the cache warm for active sessions.

Cache breakpoint placement

Place cache breakpoints at the boundary between static and dynamic content. The canonical structure is: static system prompt → cache breakpoint → static tool definitions → cache breakpoint → static reference documents → cache breakpoint → dynamic conversation history → dynamic current message. The static prefix gets cached; the dynamic suffix re-executes on every request.

The critical constraint: everything before the cache breakpoint must be byte-for-byte identical across requests for the cache to hit. This means inserting any dynamic content — a user ID, a timestamp, a session variable — before the cache breakpoint will invalidate the cache on every request. Keep all dynamic content after the last cache breakpoint.

Cost math

Cache writes cost 25% more than normal input tokens — the first request that populates the cache is slightly more expensive. Cache reads cost 10% of the normal input price. For a 50K token static prefix at $3/MTok, that is $0.15 for the first request (cache write) and $0.015 for every subsequent request (cache read). The breakeven is at the second request. For any system that handles more than one request with the same prefix — which is every production system — prompt caching pays for itself immediately.

RAG vs long context

The availability of 200K+ token context windows has reopened the question of when retrieval augmented generation (RAG) is necessary. The answer is not "never" — RAG and long context serve different needs and complement each other in production systems.

When long context wins

Holistic understanding. Tasks that require reasoning across the full document — identifying contradictions between sections of a contract, understanding how early chapters of a codebase relate to later ones, synthesizing a complete picture from many parts. RAG retrieves fragments; long context processes the whole.

Unpredictable query patterns. If you do not know in advance which parts of a document are relevant to the user's query, RAG retrieval may miss important context. Long context avoids the retrieval gap.

Small, fixed document sets. If your knowledge base is small enough to fit in a context window and stable enough to cache, load it in full and cache it. You get complete coverage with the cost advantage of prompt caching.

When RAG wins

Large, dynamic document spaces. If your knowledge base has millions of documents that change frequently, you cannot load it in full. RAG retrieves the relevant subset per query.

Cost control at scale. For high-volume systems where every request hits a large knowledge base, RAG lets you include only the retrieved documents — typically 5–20K tokens — rather than the full corpus.

Precise attribution. RAG makes it easy to cite sources — you know exactly which retrieved chunks contributed to the response. With full document in context, attribution requires additional work to identify which sections the model relied on.

Truncation strategies

When conversation history grows long enough to threaten the context budget, you need a truncation strategy. Which strategy is right depends on the nature of the task and what information value old turns hold.

Summary truncation

When the conversation history exceeds a threshold, summarize the oldest turns into a compact summary and replace them with the summary. The summary is cheaper to process than the original turns and preserves the semantic content without verbatim quotes. This is the right strategy when continuity across the full conversation matters — a long-running customer support session, a multi-session research agent, a complex multi-step task.

Implement summary truncation carefully: the summary should be produced by the same model that will read it, and the summary prompt should specify what information to preserve versus what to compress. Key decisions, extracted facts, and unresolved questions should survive; small talk and verbose reasoning chains should be compressed.

Sliding window

Keep the most recent N turns and discard everything older. Simple to implement, no information extraction required. The right strategy when tasks are short-horizon — the current question does not depend on turns from fifteen exchanges ago — or when conversations are naturally episode-based with clean task boundaries.

The failure mode is dropping turns that introduced a constraint or decision that the current task depends on. If the user said "always respond in Spanish" on turn three and you drop turn three at turn twenty, the behavior changes silently. For sliding windows, re-inject critical facts from dropped history into the system prompt or a summary block.

Selective pruning

Not all turns are equal. A turn that extracted a key fact has lasting value; a turn where the model said "I understand, let me look into that" has nearly none. Selective pruning scores each turn for information density — either heuristically (low-information phrases, short acknowledgments) or by running a classification pass — and drops the lowest-value turns first.

Selective pruning is more expensive to implement but produces better results when information density is uneven — which it almost always is in real conversations. The investment pays off for high-stakes, long-running agent tasks where context loss is costly.