AI Cost Optimization: How to Reduce LLM and Inference Costs in Production

The cost of running a capable language model has been falling at extraordinary speed. Between November 2022 and October 2024, the price of querying a model with GPT-3.5-level performance dropped from $20 to $0.07 per million tokens, according to Stanford’s 2025 AI Index — a decline of more than 280-fold.

Production AI costs have not followed the same curve, however. As applications move beyond single-turn interactions, one user request may set off a long sequence of model calls, retrieved context, tool use and intermediate reasoning. In a recent study of coding agents, agentic tasks consumed 1,000 times more tokens than code chat and code reasoning in the workloads evaluated; even repeated runs of the same task varied by as much as 30x.

Cheaper tokens, in other words, can still produce a significantly larger bill when applications use many more of them. Bringing that bill under control starts with understanding where the work is happening, then reducing unnecessary context, avoiding redundant model calls, matching tasks to appropriately sized models and improving inference efficiency and infrastructure utilization where relevant. Each change needs to be measured against the latency and output quality the application requires, which makes AI cost optimization an ongoing AI engineering practice rather than a single cost reduction exercise.

In many enterprise LLM workloads, the model processes far more text than it generates. A request may include a system prompt, retrieved documents, conversation history, tool results and user instructions, then return only a short answer. Snowflake AI Research observed an approximate 10:1 ratio between prompt and generated tokens across many enterprise workloads, which places much of the inference burden on processing the input.

That workload shape directs attention to the prefill phase. During prefill, the model processes the full prompt and creates the key-value cache (KV cache) used during generation. Decode follows, producing the response one token at a time. For applications with long prompts and relatively concise outputs, prefill can consume a substantial share of the compute required for each request.

Token pricing translates that processing into the API bill. Providers generally charge separately for input and output tokens, often at different rates, so both contribute to cost. Yet counting only the visible response can give teams the wrong impression: Retrieved context, repeated instructions and accumulated conversation history may account for most of the tokens sent through the model.

Leo Rodriguez, Principal Product Marketing Manager, AI/ML at Snowflake, says this is one of the easiest places for teams to underestimate cost: “If a task takes 20 prompts to complete, you’re not just paying for the 20th prompt. In many cases, the model is processing the prior prompts, long context windows and all the information it needs to decide where to look. That’s where teams can get surprised by the compounding cost of a single query.”

Inference creates the recurring bill

For teams building on foundation models, inference usually makes up the major continuing production expense. Every request initiates another round of prompt processing and generation, and multistep workflows can invoke a model several times before returning one result.

Agentic applications make call-level metrics especially important. A user may submit one request while the agent plans a task, retrieves information, calls tools, evaluates intermediate results and generates a final response. Measuring only the final call leaves most of the workflow’s consumption unaccounted for.

Two operational metrics provide a useful starting point:

• Cost per 1,000 API calls: Useful for services with a relatively consistent request pattern.
• Cost per workflow or agent run: More informative when one user action triggers multiple models, tools or retrieval steps.

Cost per successful workflow can add another layer of context. An inexpensive run that fails and must be repeated may cost more overall than a slightly more expensive run that completes reliably.

Self-hosting shifts the cost model

With self-hosted models, token counts still indicate workload volume, although GPU economics usually dominate the infrastructure bill. The selected GPU type, model size, batch size, memory footprint and utilization rate determine how much inference capacity the organization receives from each instance.

Idle time is particularly expensive. A GPU provisioned for peak demand may sit underused for much of the day, while an undersized serving pool can create queues, latency spikes and failed requests. Long context windows also consume KV-cache memory, limiting the number of requests a GPU can process concurrently.

Before changing the model or serving stack, teams need a baseline that connects infrastructure consumption to application activity: GPU-hours, utilization, throughput, latency and cost per completed request. Without that connection, lower infrastructure spending can conceal declining service quality.

No single technique addresses every source of AI cost. The right combination depends on request volume, context length, task complexity, latency requirements and how often prompts repeat.

Reduce prompt and context length

For workloads dominated by input tokens, prompt compression often offers the most direct savings. Teams can shorten system instructions, remove duplicated guidance, summarize older conversation turns and retrieve only the passages needed for the current request.

Research systems have demonstrated substantial compression, although results vary by task and by the amount of information that can safely be removed. Microsoft Research’s LLMLingua, for example, achieved up to 20x prompt compression with little performance loss in its evaluations. In a separate 2026 study focused on clinical prediction, Medical Token-Pair Encoding reduced input length by as much as 31% and inference latency by 34% to 63% while maintaining or improving performance across the tasks evaluated.

Production implementations need a more cautious target. Aggressive compression may remove a policy instruction, relevant record or example that the model needs. Start with deterministic reductions: eliminate duplicated context, cap retrieved passages and replace full documents with structured extracts where possible. After those changes, evaluate summarization or learned compression against representative requests.

Context engineering provides the broader discipline around deciding which instructions, data and prior state the model receives. Cost is one consideration within that design, while grounding, security and output quality remain part of the same decision.

Route requests according to task difficulty

A frontier model may be appropriate for complex reasoning, ambiguous requests or high-consequence decisions, but classification, extraction, formatting and straightforward summarization can often run on a smaller model at a lower token rate.

Rodriguez frames model selection as a practical cost-control discipline: “Cost optimization starts with the harness knowing which models or tools are right for the task. For complex work, you may need a more capable model. For simpler or less time-sensitive work, a less expensive model may deliver the same outcome at a much lower cost. In either case, a well-defined model harness can help route the task to the most appropriate model.”

Model routing, sometimes called model cascading, assigns each request to the least expensive model expected to meet its quality threshold. The router may use task type, prompt characteristics, confidence scores or the output of an initial model. When the smaller model cannot answer reliably, the workflow escalates the request.

Published results illustrate the potential range. The Hybrid LLM approach made up to 40% fewer calls to the larger model without reducing response quality in its experiments. BEST-Route reported cost reductions of up to 60% with less than a 1% performance decline on its evaluated data sets. These figures reflect controlled experiments, so teams should validate their own routing policy against production traffic and evaluation criteria.

A router also introduces an additional failure mode: A request may be sent to a model that lacks the required capability. Quality thresholds, fallback rules and periodic evaluation help keep those decisions aligned with application requirements.

Reuse responses through semantic caching

Exact-match caches work well when identical prompts recur. Semantic caching broadens the match by identifying prompts with sufficiently similar meaning, then serving a stored response instead of issuing another model call.

The savings depend on repetition within the workload. A customer-support assistant answering variations of the same policy questions may achieve a high cache-hit rate, while an analytical application receiving unique queries may find fewer reuse opportunities. In one study, a semantic cache reduced model API calls by as much as 68.8%, with positive-hit rates above 97% in the evaluated query categories. Another system reported reductions of up to one-third in inference cost for semantically similar requests.

Caching requires clear boundaries. Responses based on rapidly changing data need short expiry periods or cache invalidation rules. User-specific, permission-sensitive or nondeterministic outputs may be unsuitable for reuse. Similarity thresholds should also be tested carefully — a permissive threshold raises the hit rate, but it can return an answer generated for a meaningfully different question.

Right-size or specialize the model

A smaller model can lower cost in two ways: It may carry a lower API price and, when self-hosted, it generally requires less memory and compute. Distillation transfers behavior from a larger teacher model into a smaller student model, while fine-tuning can improve a compact model on a defined task.

This approach works best when the workload has stable boundaries. A specialized extraction model, for example, can learn a known schema and document type. A general-purpose assistant handling open-ended questions has a broader capability requirement and may need a larger model or an escalation path.

Before replacing a model, compare both average performance and important edge cases. Aggregate benchmark scores can hide failures on uncommon inputs, languages, long documents or requests requiring multiple reasoning steps.

Constrain output with structured schemas

Unbounded generation can produce explanatory text the application discards. When the downstream system needs a category, field set or tool argument, structured outputs can restrict the response to a defined schema.

A schema can reduce wasted output tokens, simplify parsing and lower retry rates caused by invalid formatting. The savings per call may be modest, particularly for already concise responses, but high-volume classification and extraction workloads can accumulate meaningful reductions.

Schema constraints don’t guarantee factual accuracy. The model may return a valid structure containing an incorrect value, so validation and evaluation remain necessary.

Batch compatible requests

GPUs process parallel work more efficiently than a stream of isolated requests. Batching groups requests so the serving system can run them together, increasing throughput and distributing infrastructure cost across more inferences.

Continuous batching improves this approach by adding and removing requests as generation proceeds, instead of waiting for every request in a fixed batch to finish. The benefit depends on concurrency and latency targets. Offline document processing can tolerate larger batches, while an interactive assistant may need smaller batches to protect time to first token.

The original vLLM research reported 2x to 4x higher throughput at comparable latency than the systems used as baselines, largely through PagedAttention and more efficient KV-cache memory management. Those gains represent serving-engine benchmarks, not a guaranteed reduction in a production bill, but they show how infrastructure efficiency can increase the useful output obtained from the same GPU capacity.

Use quantization and KV-cache optimization

Quantization stores model weights, and sometimes activations, at lower numerical precision. Moving model weights from 16-bit to 4-bit precision reduces their raw memory footprint by approximately 4x. Techniques such as activation-aware weight quantization (AWQ) are designed to preserve model quality under that lower-precision representation by protecting the weight channels that contribute most to the output. With less memory devoted to model weights, teams may be able to use lower-capacity hardware or leave more GPU memory available for KV caches and concurrent requests.

The quality trade-off depends on the model, quantization method and task. Reasoning-heavy workloads can be sensitive to aggressive quantization: One evaluation found sizable degradation on mathematical reasoning for some low-bit configurations. Test the quantized model against the actual application data, especially where small numerical or logical errors carry consequences.

KV-cache optimization addresses a different memory consumer. Each active request stores attention state for the tokens already processed, and long prompts can use considerable cache capacity. More efficient allocation, prefix reuse and cache compression can increase batch size or reduce the number of GPUs needed to support a given traffic level.

Stack compatible levers

Individual savings percentages cannot simply be added together. Compression changes the request sent to the router, routing changes which calls reach each model and caching removes some calls before inference begins. Infrastructure improvements then apply only to the requests that remain.

A practical sequence might compress prompt context, serve recurring queries from a semantic cache, route routine work to a smaller model and run the remaining self-hosted traffic through an efficient batching system. Measure the combined result at the workflow level, where interactions among the techniques are visible.

Watch Snowflake leaders discuss key considerations for enterprises moving AI into production:

AI cost optimization starts with attribution. Provider invoices show total consumption, while token counts and application telemetry explain which teams, features and workflows generated it.

At minimum, production monitoring should capture:

• Input and output tokens by model
• Model calls per user request
• Cache-hit and escalation rates
• Cost per 1,000 calls
• Cost per workflow or agent run
• Latency and completion rate
• Evaluation results for each model or routing path

Tagging usage by team, environment, feature and workflow supports showback or chargeback. A platform team can then identify whether a billing increase came from user growth, a new feature, longer prompts, repeated agent loops or a shift toward a more expensive model.

Budgets and alerts should use operational units as well as account-level spend. A monthly threshold may reveal a problem late in the billing cycle, whereas an alert on cost per workflow can surface a regression soon after deployment. Sudden increases in input tokens, tool calls or retry rates often indicate a code or configuration change that deserves investigation.

Evaluate every cost change

A lower bill doesn’t mean a successful optimization. Prompt compression may omit relevant context, a smaller model may fail on difficult requests and an overly broad cache may serve stale or mismatched answers.

Evaluation gates connect cost changes to application behavior. Before rollout, compare the proposed configuration with the current one on a representative evaluation set. Measure task-specific quality, policy compliance, latency and cost, then define the degradation the application can accept. In production, monitor the same signals for distribution shifts and uncommon failures.

This discipline helps teams avoid several recurring mistakes: assigning every request to a frontier model, overlooking repeated prompts, leaving token consumption unattributed, maintaining excess GPU capacity and assuming the cost of a workflow is fixed.

Snowflake Cortex AI provides managed inference, so teams can use supported models without provisioning and maintaining their own GPU serving infrastructure. Consumption is metered according to the Cortex AI service and model used, while Snowflake usage views expose token and credit consumption for monitoring. Cortex Agents, for example, are billed according to the tokens processed by the orchestration model, with additional consumption from the underlying services called during the workflow.

For batch-oriented work over tables, Cortex AI Functions run directly through SQL and are optimized for throughput. Teams can apply AI operations across many rows without building a separate request pipeline, while interactive applications can use the relevant REST APIs when latency carries greater weight.

Model selection provides another cost-control point. Simpler classification, extraction or summarization tasks can use an appropriately sized model, while more capable models remain available for requests that need them. Because Cortex AI model rates vary, the selected model directly affects the token cost of the workload.

For Snowflake workloads, Rodriguez says cost and quality improve when your AI data harness is purpose-built for the data environment where the work happens. “If you’re working with Snowflake data, using Snowflake CoCo gives you a better outcome in terms of accuracy and cost efficiency because the harness has been purpose-built for Snowflake. You’re not pointing a generic LLM at enterprise data and hoping it finds the right answer.”

Snowflake has also developed inference optimizations for prompt-heavy enterprise traffic. SwiftKV reuses hidden states from earlier transformer layers while generating the KV cache, reducing repeated prefill computation. Snowflake has reported up to 50% lower prefill compute and up to 2x higher throughput for enterprise workloads with long prompts. Earlier published testing showed approximately 40% to 50% throughput improvement in the configurations (compared to compression-only methods).

Keeping AI processing close to governed data can reduce additional architecture and data-transfer overhead. Applications can apply Snowflake access controls to the data used for inference, run Cortex AI Functions through SQL and inspect usage through account-level views without maintaining a separate copy solely for model processing. For workloads that otherwise export large context sets to another environment, reducing that movement can also lower transfer, storage and pipeline-management costs.

Across these controls, the underlying method remains consistent: Measure consumption at the workflow level, assign each task to suitable compute and validate every optimization against the quality the application is expected to deliver.

As applications add longer context, more tool calls and more autonomous workflows, the teams that control cost will be the ones that understand the full path of each request and tune it deliberately. AI cost optimization isn’t about doing less with AI, but rather matching every task to the right context, model and infrastructure so production systems can scale with both quality and financial confidence.