Context Window
Also known as: context length, token window
The context window is the maximum number of tokens a model can attend to in one request — system prompt, conversation history, retrieved documents, tool definitions, tool results and the response combined. It is a hard budget, and everything an agent knows in a given turn must fit inside it.
Last reviewed · Part of the Architecture Glossary
In practice
A long window is not the same as effective use of it. Two effects push against naive stuffing:
- Lost in the middle. Retrieval accuracy is highest for material at the start and end of the context and measurably worse in the middle. Ordering matters, so put the decisive material last.
- Cost and latency scale with input. Prefill is roughly linear in input tokens; a 100k-token context costs real money and real time on every turn of a conversation, not once.
The budget for an agent turn, allocated deliberately:
| Slice | Typical share | Notes |
|---|---|---|
| System prompt + tool schemas | 5-15% | Fixed cost per turn — cache it |
| Retrieved context | 20-50% | Rerank hard; more chunks is not better |
| Conversation history | 20-40% | Summarise or window it |
| Response headroom | reserve it | An overflow at generation time is a failed turn |
Prompt caching is the lever that makes long system prompts affordable: a stable prefix is cached, so only the changing suffix is billed at full rate.
When it matters
Any multi-turn agent, any RAG design decision about chunk count, and any cost model for an LLM feature.
Common mistake
Treating a bigger window as a substitute for retrieval quality. Filling 200k tokens with loosely relevant chunks lowers accuracy and raises cost simultaneously — see retrieval vs generation failure for how to tell which half is actually broken.
See also
- Tool-Use LoopThe tool-use loop is the core agent pattern: the model receives a goal and tool definitions, chooses a tool call, the runtime executes it and returns the result into the context, and the model decides again — repeating until it produces a final answer or hits a stop condition.
- RAG (Retrieval-Augmented Generation)RAG is the pattern of retrieving relevant documents at query time and placing them in the model's context so the answer is grounded in your data rather than in the model's parameters.
- EmbeddingAn embedding is a dense vector representation of text, an image or other content, positioned so that semantically similar items sit close together under a distance measure such as cosine similarity.