AI Engineering: How to Build and Operate Production AI Systems

Enterprise spending on generative AI reached an estimated $37 billion in 2025, according to Menlo Ventures. The largest share — $19 billion — went to the application layer: the products and software that put foundation models to work.

That distribution reflects where much of the technical work now sits. A production AI application needs more than access to a capable model. It has to assemble current, permissioned context; coordinate tools and multistep logic; evaluate variable outputs; meet latency and cost targets; and preserve enough state to continue work across calls.

AI engineering brings those responsibilities into a single discipline by treating the model as one component within a larger system whose data, runtime behavior and operating requirements all have to work together.

AI engineering is the discipline of designing, building, deploying and operating AI systems. It applies software-engineering practices — testing, versioning, observability, release controls and cost management — to applications whose core behavior depends on probabilistic AI models. The model supplies capabilities such as language understanding, generation or reasoning, while the surrounding system determines how those capabilities are used, constrained and evaluated.

The term is still used loosely, however. In some organizations, AI engineering includes traditional model development and training. But in most enterprises, the center of AI engineering work has shifted toward composing systems around pretrained models. The scope includes model access, retrieval and context assembly, orchestration, evaluation, deployment, inference optimization, monitoring and governance.

Each layer affects the others. A larger context may improve grounding while raising latency and cost, for example. A smaller model may handle routine extraction efficiently, but only if the routing policy recognizes requests that require more capability. The engineering challenge is to make trade-offs explicit and measurable.

A prototype often hides many of these concerns because traffic is low, context is hand-selected and a person is reviewing the output. Once an application is used repeatedly — by different users, over changing data and through downstream systems — reliability, governance, latency and cost determine whether it can remain in production.

AI engineering builds the production system around one or more AI models. The work typically includes model selection, retrieval and context assembly, application logic, tool integration, evaluation, serving and monitoring. Its primary artifact is the AI-enabled product or workflow that users interact with.

ML engineering typically focuses more directly on the model itself. ML engineers prepare features, select architectures, train or fine-tune models and improve their performance against defined objectives. In organizations that rely heavily on pretrained foundation models, some of that work may give way to model adaptation, but the model remains the central engineering artifact.

MLOps manages the operational lifecycle of models. It provides the pipelines and controls used to version, deploy, monitor and retrain them, much as DevOps supports the software lifecycle. AI engineering uses many of those practices, though its operating scope extends beyond the model endpoint to the retrieval, orchestration, evaluation and application layers around it.

Data science begins with the analytical question. Data scientists explore data, test hypotheses, develop statistical or predictive approaches and often create the first prototype. AI engineers take a model or prototype into a production application, where it has to work with live data, changing user requests, security requirements and downstream systems.

The boundaries between these disciplines aren’t rigid. An AI engineer may fine-tune a model or maintain a deployment pipeline, while an ML engineer may help integrate a foundation model into an application. The main difference is what each role is primarily responsible for: data science develops insight and prototypes, ML engineering develops models, MLOps operates the model lifecycle and AI engineering owns the behavior of the complete AI system.

AI engineering turns a model prototype into a reliable production system. Engineers define the task, choose a model, assemble the instructions and data it needs, and build the surrounding workflow for tool use, memory and control flow.

They then evaluate the system as a whole. Deterministic checks verify schemas, identifiers and permissions, while broader tests measure qualities such as groundedness, relevance and completeness. These evaluations help teams compare changes to models, prompts, retrieval and orchestration before release.

In production, the system must also meet latency, throughput and cost targets, recover from model or tool failures, and provide enough observability to trace problems to retrieval, context assembly, model behavior, tool execution or validation.

The resulting production stack can be understood as a sequence of connected responsibilities:

• Model access: Select or route to a model suited to the request
• Context and retrieval: Assemble the instructions, data, tools and state required for the current step
• Orchestration: Coordinate model calls, tool use, memory, retries and stop conditions
• Evaluation: Test whether the complete system meets quality and policy requirements
• Serving and optimization: Operate the workflow within latency, throughput and cost targets
• Monitoring and governance: Trace what the application accessed, produced and did over time

AI application architecture separates the model from the services around it. The model performs inference, while the application manages retrieval, memory, business logic, tools, validation, permissions and user interaction. This separation allows teams to change models without rebuilding the product and to enforce controls outside the model itself.

The architecture must also connect securely to existing data and systems. Reading governed data, searching documents, calling APIs or writing results back requires authentication, permission checks, schema validation, error handling and audit logs.

Deployment depends on the workload. Batch systems process records asynchronously, while interactive assistants and agents must account for response time, tool latency and concurrency. Many applications combine both approaches, using batch pipelines to prepare data that an interactive service retrieves at runtime.

Production design should make changes safe and reproducible. Models, prompts, retrieval settings and tool schemas should be versioned together, tested in staging and released gradually through canaries, shadow traffic or A/B tests. Observability should trace each request across retrieval, prompt assembly, model decisions, tool calls and validation so failures can be diagnosed and rolled back quickly.

For each model call, an AI application constructs a temporary working set. It may include system instructions, the user’s request, retrieved business data, conversation history, workflow state, tool definitions and an output schema. The quality of that working set often explains the quality of the result more directly than the nominal size of the model.

Context engineering designs the system that selects, structures and updates those inputs. A capable model can still produce an incorrect answer when retrieval returns the wrong customer record, an outdated policy outranks the current version or earlier state is dropped between steps. Conversely, a focused context assembled from authoritative sources can let a smaller model complete a bounded task reliably.

Leo Rodriguez, Principal Product Marketing Manager, AI/ML at Snowflake, frames the significance of context engineering this way: “The AI models are already intelligent. The problem is that they don’t automatically have all the business context to be trusted by the business users. To get accurate answers, you have to give the model the definitions, relationships, permissions and trusted sources it needs to reason over enterprise data.”

A production context stack commonly contains:

• System and task instructions that define the role, rules and immediate objective
• User input and identifiers associated with the current request
• Retrieved structured and unstructured data, filtered under the user’s permissions
• Short-term or long-term memory selected for the current task
• Tool definitions describing the functions the model may call
• Prior workflow state, including completed actions and unresolved decisions
• A structured-output schema for the downstream application

The context window places a hard budget around that material. Retrieval may return more passages than the model needs, tool schemas can consume substantial space and long conversations accumulate history quickly. Ranking, metadata filters, deduplication and compression help reserve capacity for the information most likely to affect the current step. Because information can be factually correct and still irrelevant, relevance has to be evaluated alongside recency, authority, identity and permissions.

Retrieval-augmented generation (RAG) and prompt engineering sit within this layer. RAG supplies external knowledge; prompt engineering shapes instructions and requested output. Context engineering coordinates those techniques with memory, state, tool exposure and token budgeting across the application.

Structured outputs provide a contract between the probabilistic model and deterministic code. A schema can limit the response to required fields and make malformed output easier to detect, though schema validity doesn’t establish factual accuracy. The application may still need to verify identifiers, compare values with source data or require approval before an action proceeds.

Learn how Cortex Search and Cortex Analyst can support retrieval-augmented generation over enterprise knowledge:

Production AI rarely consists of one prompt and one response. A request may pass through a gateway, route to one of several models, retrieve context, call a tool, inspect the result and continue until the task is complete. Orchestration coordinates that execution path.

An AI gateway provides a common entry point for model traffic. Depending on the architecture, it may centralize provider access, authentication, routing, rate limits, logging and cost attribution. Model routing then selects a model or fallback path based on the task, latency target, policy or confidence threshold, allowing routine work to use a smaller model while difficult requests escalate.

The agent harness provides the runtime orchestration around a model. It supplies tools, memory, context management, state persistence, control flow and guardrails, giving the model the working conditions required for multistep work. Two applications can use the same foundation model and produce very different completion rates because their harnesses expose different tools, preserve different state and handle failures differently.

Learn more about harness engineering >

Arun Agarwal, Snowflake’s Principal Product Marketing Manager, AI/ML, explains: “Agent orchestration lives in the agent harness, not the model. That separation is what lets customers keep pace as models get smarter. They can select the most appropriate model for their use case without rebuilding the agent.”

At the center of many harnesses is a control loop. The model proposes a next step, the harness validates and performs it, the result returns to the model and the cycle continues. That plan-act-observe pattern needs boundaries: iteration limits, timeout thresholds, retry rules and stop conditions prevent the application from repeating an unproductive action or consuming resources after the task can no longer be completed.

Guardrails can run before and after model inference. Input controls may reject prohibited requests or remove tools that the current user cannot access. Output controls may validate a schema, check grounding, enforce policy or block an unsupported action. Task-specific controls are generally more useful than a generic filter alone because a data engineering agent, a financial analysis agent and an internal support agent have different permissible actions.

The Model Context Protocol standardizes how applications connect to external tools and services, while agent orchestration coordinates work across multiple agents or workflow steps. The harness remains the per-agent runtime that governs how one agent uses its assigned tools, context and state.

AI performance is shaped by both the model and the serving path around it. Interactive applications track time to first token, which includes prompt processing and queueing, as well as time per output token during generation. Batch systems place more weight on throughput and cost per completed item. Agentic workflows add another unit: cost and latency per successful run, including every model call, retrieval step, tool invocation and retry.

The input side of inference deserves particular attention. During prefill, the model processes the full request and creates the key-value cache used during generation. In prompt-heavy enterprise workloads, long instructions, retrieved passages and conversation history can make prefill a substantial share of compute.

AI cost optimization starts with attribution. Provider invoices show total consumption, but application telemetry explains which feature, workflow or configuration produced it. Useful measures include input and output tokens, model calls per request, cache-hit rates, escalation rates, latency, completion rate and cost per successful workflow. When those values are tagged by team, environment and feature, a billing increase can be traced to user growth, longer context, repeated loops or a change in model selection.

Rodriguez says teams often underestimate AI cost when they look only at the final response rather than the full workflow that produced it. “If a task takes 20 prompts to complete, you’re not just paying for the 20th prompt,” he explains. “In many cases, the model is processing the prior prompts, long context windows and all the information it needs to decide where to look. That’s where teams can get surprised by the compounding cost of a single query.”

The main optimization levers affect different parts of the path:

• Context reduction removes duplicated instructions, caps retrieved passages and compresses history so the model processes fewer input tokens.
• Model routing assigns bounded work to the least expensive model expected to meet the quality threshold.
• Semantic caching reuses approved responses or intermediate results when a new request is sufficiently similar and the underlying data, permissions and freshness constraints still hold.
• Batching and continuous batching improve GPU utilization by processing compatible requests together.
• Quantization reduces model-weight memory, which can leave more capacity for concurrent requests or permit lower-capacity hardware.
• KV-cache optimization improves memory allocation or reuse for long and repeated prefixes.
• Structured outputs limit unnecessary generation and reduce retries caused by invalid formatting

These savings interact. Compression changes what reaches the router, caching removes some calls before inference and infrastructure improvements apply only to the traffic that remains. Evaluation has to accompany each change: a lower bill offers little value when compression drops a policy exception, a cache returns stale data or a smaller model fails on the requests routed to it.

See how Snowflake Cortex Analyst uses semantic context to let business users ask natural-language questions over governed Snowflake data:

AI systems often produce several acceptable answers, vary across repeated runs and depend on retrieval results that change over time. Evaluation makes that behavior tractable by defining representative tasks, acceptable outcomes and release thresholds. The evaluation should cover the complete pipeline.

Offline evaluation happens before release, using fixed or replayed examples. Online evaluation samples production traffic, gathers user feedback and detects distribution shifts or new error patterns. Together, they create a release loop: changes run through regression tests in continuous integration, canary or shadow deployments expose them to realistic traffic, and production monitoring shows whether behavior continues to meet the threshold.

Observability supplies the evidence behind the evaluation score. Traces should connect the final output with the model version, instructions, retrieved sources, tool calls, state changes, validation results, latency and token consumption. When performance drifts, engineers can then determine whether the cause is a model change, stale data, retrieval quality, a new request pattern or the surrounding runtime.

Enterprise AI systems derive much of their value from proprietary data: customer records, operational events, documents, metrics, policies and the relationships among them. Context carries the organization’s access rules and business definitions. When AI is built in a separate environment, teams may have to reproduce data pipelines, permissions, monitoring and governance for each application.

Snowflake brings AI services to governed data through Cortex AI, a suite of managed capabilities for building and operating AI applications within Snowflake. Teams can access foundation models, process structured and unstructured data, retrieve enterprise knowledge, build agents and evaluate application behavior while applying Snowflake’s existing security and governance controls.

Cortex AI Functions enable AI operations in SQL for tasks such as extraction, classification and summarization. Cortex Search retrieves relevant unstructured content, while Cortex Analyst interprets natural-language questions over structured data using semantic context. Cortex Agents coordinates these resources and custom tools for multistep work, and Snowflake CoWork provides a conversational interface through which business users can work with configured agents.

For custom systems, Snowpark Container Services supports application and model-serving components within Snowflake’s platform. AI Observability provides evaluation and tracing for generative AI applications, while Horizon Catalog supplies governance capabilities such as access control, lineage, data quality and AI guardrails. Together, these services map to the same production stack: access the model, assemble governed context, orchestrate work, evaluate behavior, operate the serving path and observe what happens.

Teams still define which data and tools an application may use, how semantic models represent business logic, which changes pass an evaluation gate and what level of cost or latency is acceptable. But the architecture does reduce the number of disconnected systems in which those decisions have to be reimplemented.

As foundation models improve and become easier to access, enterprise differentiation shifts toward the system around them: the quality of governed context, the specificity of tools, the evaluation data that reflects real work and the operating controls that keep the application reliable. AI engineering is the practice that assembles those pieces into a production system and keeps them aligned as models, data and requirements change.