Context Windows: How Much Context Does an LLM Really Need, and What Are the Trade-Offs?

A large context window sounds like an obvious advantage. If a model can accept more tokens, it can read longer documents, retain more conversation history and work across larger collections of information without something being cut off.

In production systems, though, large context windows come with trade-offs. Every document, retrieved passage, instruction and prior interaction placed in context consumes space and adds processing cost. Long inputs can also make it harder for a model to reliably find the information that’s relevant to the task at hand. In the NoLiMa long-context benchmark, for example, researchers found that 10 of 12 tested models fell below 50% of their short-context performance at 32K tokens on tasks that required models to identify relevant information without obvious lexical overlap.

Context should be treated as a finite budget — with implications for quality, latency, cost and governance.

A context window is the amount of tokenized information a large language model (LLM) can consider when generating a response. Depending on the model and serving system, that context can include system instructions, a user’s current request, previous conversation turns, retrieved documents, tool results, code and other information supplied at inference time.

The limit is measured in tokens, rather than words or characters. A token represents a unit produced by the model’s tokenizer: a complete word, part of a word, punctuation or another text element. As a result, a 100,000-token context window doesn’t translate into a fixed number of pages or words. The exact relationship varies with the language, content and tokenizer.

The model uses the information within that window to determine what to generate next. If an application supplies instructions followed by five retrieved documents and a question, for instance, those components collectively consume context capacity. Generated output may consume part of the available token budget as well, depending on how the model and API define their limits.

Context is different from persistent memory. Once a model call ends, the context window itself doesn’t store the interaction for a future request. Persistent memory allows a chat application to create continuity by storing previous messages and supplying them again on subsequent calls, while an AI agent can persist state in a database or another external system and retrieve the relevant portions when needed.

This distinction is useful when thinking about context windows versus context engineering. The context window describes the capacity available to the model. Context engineering concerns what an application puts into that capacity — and, just as importantly, what it leaves out.

Learn how to build a high-quality RAG pipeline using LlamaParse and Snowflake Cortex. This walk-through is perfect for anyone looking to build question/answer systems on top of complex documents full of embedded images, charts and tables:

Increasing a context window changes the amount of information a model has to process during inference. With standard transformer attention, tokens interact with other tokens in the sequence, which makes processing long inputs substantially more computationally demanding than processing short ones.

The details depend on the model architecture and inference system. During the initial processing of a prompt, often called prefill, a longer sequence requires more computation and memory. During subsequent token generation, inference systems commonly use a key-value (KV) cache, which stores intermediate representations of earlier tokens so the model doesn’t have to recompute the entire sequence every time it generates another token.

That cache has its own resource cost. As context grows, so does the amount of cached state that the serving infrastructure may need to hold, placing additional pressure on accelerator memory. Techniques such as grouped-query attention, paged attention and optimized cache management can reduce those demands, while the underlying relationship remains: carrying more context requires resources somewhere in the inference stack.

For an application team, those infrastructure mechanics show up in familiar challenges. Longer prompts generally take more time to process, consume more input tokens and can increase the cost of each model call. If the same large body of context is repeatedly supplied across hundreds or thousands of requests, relatively small design choices about what gets included begin to compound.

Prompt caching can change that calculation when a stable prefix — a long set of instructions or a frequently reused document, for example — appears across requests. Cached input may be processed or priced differently by the model provider, reducing the cost of repeatedly supplying the same material. It doesn’t, however, make context capacity free: the application still has to decide what information belongs in the request and how much context the model needs to perform the task reliably.

Those trade-offs explain why maximum context length is a useful specification without being a straightforward measure of application quality.

The clearest benefit of a larger context window is simple: more task-relevant information can remain available to the model at the same time. For document analysis, that may mean supplying an entire report rather than splitting it into sections and processing them separately. A coding assistant can inspect more files from a repository at once, while an agent can retain a longer sequence of tool results, intermediate decisions and instructions as it works through a multistep task. In conversational applications, additional capacity allows more prior turns to remain available before older material has to be summarized or removed.

Larger windows also expand the amount of information available for in-context learning, in which examples and instructions supplied with a request influence model behavior without changing the model’s weights. More context leaves room for additional demonstrations, schemas, terminology or domain-specific reference material when the task benefits from them.

Some workloads depend on relationships spread across a large artifact. Analyzing a long contract, for example, may require connecting a definition near the beginning with provisions hundreds of pages later, and understanding a code change may require tracing dependencies across several files. Keeping the relevant material together can preserve relationships that would otherwise have to be reconstructed through retrieval or multiple model calls.

At the same time, the maximum supported window tells us only how much information the system permits us to provide. It doesn’t tell us how reliably the model will locate, connect and reason over every relevant piece of information inside it.

A more useful concept for AI application design is effective context. A model may be able to accept a very long context sequence, but its performance on a particular task declines well before the advertised context limit. This brings up another set of questions: where does that degradation come from, and what can go wrong before the context window is actually full?

A model that accepts 200,000 tokens doesn’t necessarily use information equally well across all 200,000. Researchers have observed this gap across long-context evaluations. In the widely cited “Lost in the Middle” study, models performed better when relevant information appeared near the beginning or end of a long input, while performance declined when the same information was placed in the middle.

Researchers have found degradation even when inputs remain far below models’ stated limits. The NoLiMa benchmark provides a particularly striking example. Rather than giving models an obvious lexical match to retrieve — a name or phrase repeated in both the question and source material — the benchmark required them to infer relationships between pieces of information. At 32K tokens, 10 of the 12 models tested performed at less than half their own short-context baseline.

The distinction between advertised context and effective context is important when evaluating models. The advertised window describes how many tokens the system accepts, while the effective context describes how much context a model can use reliably for the task you actually care about.

There isn’t one universal effective context number. A model might perform well across a long document when the task requires finding a distinctive phrase, yet deteriorate much earlier when it has to connect several dispersed facts. Document structure, the location of relevant evidence and the amount of distracting material can all influence the result.

Irrelevant context is particularly consequential. Adding another document or conversation turn gives the model more information to process, but it also introduces more material against which relevant information has to compete. Snowflake researchers observed this effect while evaluating retrieval strategies for financial documents, noting that even models with large context windows can experience “context confusion” when chunks contain too much irrelevant detail.

For application teams, the implication is practical: test models at representative input lengths using representative data. A benchmark result at 8K tokens says little about how the same model will perform when an application routinely supplies 80K, for example. Likewise, successfully submitting a request at 100K tokens establishes only that the request fits, not that the important details within it will influence the answer reliably.

Runtime behavior adds another complication. An application may encounter a configured context limit below the model’s maximum, while different serving systems might handle overflow by rejecting a request, truncating input or applying other context-management behavior. Unless that behavior is understood and monitored, information can disappear from the model’s usable context without producing an obvious application failure.

The operational limit is the real point where performance on the workload begins to deteriorate — which may occur considerably earlier than the number on the model’s specification sheet.

For many enterprise workloads, retrieval still has a substantial advantage. In the experiments mentioned earlier, Snowflake researchers tested retrieval and chunking strategies on question-answering tasks using curated SEC 10-K and 10-Q filings. The results showed large differences based on how source material was retrieved and prepared for the model. In one experiment, adding document-level context to moderate-sized chunks raised question-answering accuracy from roughly 50–60% to 72–75%. Oversized 14,400-character chunks, meanwhile, reduced accuracy by 10–20% relative to 1,800-character chunks.

These results get at the relationship between retrieval and context windows. Retrieval decides which information earns space in the context window. Instead of sending a large collection on every request, the application identifies material relevant to the question and supplies a smaller, more targeted set of evidence.

Chunking determines the granularity of that evidence. Very small chunks can separate a passage from information needed to interpret it, while very large chunks carry additional material that may have little bearing on the question. In Snowflake’s experiments, injecting broader document context into moderately sized chunks preserved useful information about the source without requiring the model to process an oversized passage on every request.

The generating model wasn't the largest determinant either. With tuned retrieval, the researchers moved Llama 70B from roughly 40–50% accuracy to more than 70%, approaching the performance of Claude 3.5 Sonnet in the same evaluation.

None of this makes long context unnecessary. Some tasks genuinely require a model to consider a large artifact as a whole. A contract analysis workflow may need clauses distributed throughout one document, for example, and an agent may need to retain the results of numerous previous actions while planning its next one. Breaking those inputs apart introduces other problems.

Retrieval is typically a good choice when the task involves repeatedly asking targeted questions of a corpus much larger than any individual request requires. Rather than placing a year’s worth of financial filings, support records or technical documentation into every model call, the retrieval layer can select evidence for the question at hand.

There’s an economic difference as well. A retrieval pipeline might supply a few thousand relevant tokens where a brute-force long-context approach supplies tens or hundreds of thousands. Multiply that difference across production request volume and context selection starts affecting both inference capacity and cost.

The question then is how much of the source material the task needs simultaneously. Whole-artifact reasoning may justify a large context. Repeated queries against a large corpus generally provide much more opportunity to retrieve selectively. In either case, the model can only work with the information the application supplies. Retrieval quality depends on the underlying corpus — its completeness, currency, structure and accessibility — just as long-context performance depends on the quality of the material placed directly into the window.

The cost of context shows up in several places. Larger prompts consume more input tokens, take longer to process and, at scale, use inference capacity that could otherwise serve additional requests. Caching and selective retrieval can change those economics considerably, which is why teams need to look beyond per-token pricing and measure how context behaves across the workload.

Start with the per-request math

Suppose an application sends an average of 100,000 input tokens per request. At 10,000 requests per day, it processes 1 billion input tokens daily. If more selective retrieval reduces the average context to 20,000 tokens, the number falls to 200 million input tokens before accounting for output.

For a model priced at P dollars per million input tokens, a simple estimate is:

Daily input cost = (average input tokens × daily requests ÷ 1,000,000) × P

That calculation is more useful when it uses token counts from representative requests rather than word estimates. JSON, code, tables, identifiers and multilingual text can tokenize very differently from ordinary English prose, so production estimates should come from the actual payloads the application expects to send.

Account for caching, output and latency

The basic formula is only the first layer. Prompt caching can change the economics substantially when requests share a stable prefix, because some providers price cached input differently from newly processed input. Applications with a large, repeated system prompt or reference block can therefore have a different cost profile from applications whose context changes on every call.

Output tokens add another component, and providers commonly price input and output at different rates. Agentic workloads introduce still more variation: a large tool result, query response or retrieved document may enter the context midway through a task and remain there across subsequent model calls unless the application summarizes or removes it.

Latency must also be considered. Before generation starts, the model has to process the supplied input, so longer prompts can increase time to first token even when the token bill remains acceptable. At higher concurrency, large contexts can also consume inference resources that would otherwise serve additional requests.

Reduce unnecessary context

Once teams understand where tokens are being spent, several levers can reduce context without removing information the model actually needs:

• Retrieve a smaller set of relevant passages
• Summarize older conversation turns
• Exclude unused fields from structured inputs
• Compress repeated prompt material
• Cache stable prefixes where the provider supports it

The right combination depends on the workload. A retrieval-heavy application may get the largest gain from improving chunk selection, while an agent may benefit more from summarizing tool history or pruning intermediate results.

Measure what happens in production

Request-level telemetry makes trade-offs visible. Average input size is useful, but distributions are more informative: a workload averaging 20K tokens can still have a small group of 150K-token requests driving a disproportionate share of cost and latency.

Tracking input tokens alongside task type, latency and model output quality also helps answer a more important question: whether the additional context is improving results enough to justify the resources it consumes.

Measurement becomes especially valuable as applications evolve. Context often grows incrementally — another retrieved source is added, tool output gets larger, conversation history runs longer — and no single change looks expensive on its own. Without per-request token telemetry, a prompt that began at 8K tokens can gradually reach several times that size before anyone notices the operational effect.

Once context is treated as a resource consumed by every request, deciding what enters the window becomes part of application architecture. In enterprise systems, that decision also has to account for which data the application is permitted to retrieve and expose.

Every piece of context has a source, an access history and, potentially, instructions embedded within it. As applications assemble prompts dynamically from enterprise data, retrieved documents and external content, teams also need to control what can enter a request, how the model treats it and what’s retained afterward.

Enforce access controls during retrieval

A retrieved passage doesn’t lose its access requirements when it enters an LLM request. Consider an assistant answering questions over customer records. A user may only have permission to see accounts from one region while the underlying data store contains records from every region. If retrieval searches the complete corpus without carrying the user’s authorization into that operation, the model can retrieve restricted records and synthesize their contents into a response that the user isn’t authorized to see.

Row-level access policies, column masking and other controls need to apply when source data is selected for context, rather than only when someone accesses the original table or document directly. For large enterprise corpora, that means retrieval architecture is a data governance decision as much as a relevance decision. The system needs to identify useful evidence while still respecting the permissions attached to the underlying data.

Treat retrieved content as untrusted input

Enterprise AI applications increasingly assemble prompts from material the application didn’t author: uploaded files, websites, emails, support tickets, retrieved documents and tool outputs. Some of that content may contain instructions that conflict with the application’s intended behavior, creating opportunities for indirect prompt injection.

As context grows, so does the amount of externally supplied material the model may have to interpret. Provenance can help teams distinguish trusted instructions from retrieved evidence and other untrusted content, while application controls can limit which tools or actions the model is permitted to invoke in response.

This is a separate concern from authorization. Access controls determine whether a user or application is allowed to retrieve particular information, while prompt-injection defenses address what retrieved content may cause the model to do once that information is already in context.

Preserve the context needed for an audit trail

For sensitive or regulated workflows, logging only the user’s question and the final response may not be enough to reconstruct an AI interaction. If the answer depended on dynamically retrieved material, teams may also need to retain which documents or records were retrieved, which policies applied, which model and version handled the request, and what other instructions or tool results were present in context.

That provenance helps answer practical questions after the fact: where did a claim in the response come from, was the requester authorized to access the underlying data and did the model receive information that could have influenced its behavior unexpectedly?

The exact audit record will depend on the application and its regulatory requirements, but designing for reconstruction early is usually easier than trying to recreate missing context after an incident or review.

The practical decisions around context — what to retrieve, how much to send and which model should process it — are easier to manage when the source data and AI services operate within the same governed environment.

In Snowflake, applications can assemble context from structured and unstructured data while retaining the governance policies attached to those sources. Rather than maintaining a separate copy of an enterprise corpus specifically for AI retrieval, teams can use data already managed in Snowflake, reducing the additional synchronization and access-control work that comes with another data store.

For workloads that need retrieval, Cortex Search provides the mechanism for selecting relevant information before it reaches the model. An application can retrieve a targeted set of passages for repeated questions over a large corpus rather than supplying the corpus itself on each request. Models available through Snowflake Cortex AI have different capabilities, context limits and inference costs, so an application doesn’t necessarily need the largest available window for every task. Snowflake has also developed specialized models for document workloads. Arctic-TILT and Arctic-Extract are designed for document understanding and extraction.

The objective is to give the model enough relevant context to perform the task reliably while avoiding context that contributes little to the result. Retrieval controls what enters the request, governance controls what the requester is permitted to retrieve and token telemetry shows what that design costs once it reaches production.

Context windows have grown quickly enough that applications can now place entire documents, lengthy conversations, extensive tool histories and, in some cases, substantial collections of data into a single model request. The engineering question has shifted accordingly. Having room for information doesn’t mean that the information should be included.

A good context strategy starts with the task. Determine which evidence the model needs, measure how reliably it uses that evidence at realistic input lengths and retrieve selectively when the workload repeatedly queries a larger corpus. Then account for what happens around the model call: token consumption, latency, access policies, provenance and the behavior of the serving runtime when context approaches its limit.

For some workloads, that process will justify a very large window because information distributed across an artifact genuinely needs to remain available together. For many others, a smaller collection of well-selected evidence produces a more efficient request and may produce a better answer as well.