This guide assumes you have already read Best Practices for Building Cortex Agents. The scope here is the evaluation cycle in detail, from early prototyping to production observability.

Evaluation helps you understand how agents perform across a variety of scenarios so you can systematically improve their quality.

There are several ways to construct your evaluation set depending on the resources available and your familiarity with the end use case. The best approach is to ask a willing set of end users what they hope to get out of your agent.

You can do this by having future users draft queries, or better yet, let them try out a prototype of your agent in a trial period. By allowing future users to try your agent, you get a more accurate sense of how they would actually use it. This also builds excitement around your agent to improve adoption when it is deployed in production.

Once users have had a chance to try your agent, you can query the logs to build a representative dataset using Cortex Code. Cortex Code will create the dataset from the logs and let you edit and define the ideal tool execution sequence for each query if it differs from what happened in the logs.

If it prompts you to select metrics, choose all of the metrics available so Cortex Code will add the expected tool uses to the dataset. This will be useful later.

Try it out by asking Cortex Code:

Once the dataset is generated, you can validate the expected responses in the dataset-curation skill. The skill walks you through each response one-by-one to validate.

If you can't find willing users to test your agent, you can still use Cortex Code to build your eval set. The resulting dataset will not be as representative as one produced from end users, which should affect your deployment strategy.

Once you have created your evaluation dataset, validate that the responses are what you expect.

Ground truth responses should be specific enough to meaningfully validate your agent's answers, but not so specific that non-determinism in LLM-generated responses causes false negatives. For example, for an input query like "How can I decrease my support time to resolution?", a ground truth of "Follow known best practices related to technical support" may be too generic and match partially correct answers. On the other hand, a ground truth that names specific entities, ticket counts, or resolution time targets risks penalizing correct answers that surface different but equally valid details from the underlying data. Aim for ground truth that captures the key facts and constraints of a correct answer without over-indexing on specific entities or figures that may vary between valid responses.

Be mindful of ground truth staleness - questions tied to real-time or frequently changing data (e.g., "what is the current stock price of Snowflake?") will produce ground truth that becomes incorrect over time. To avoid this, consider using input queries anchored to fixed points in time ("what was the closing price of Snowflake stock on January 31st, 2025?"), use ground truth that validates qualitative behavior rather than specific numeric values, or generate ground truth programmatically at evaluation run time so it reflects the current state of the data immediately before the eval executes.

If you have ground truth expected responses available (or can create them using the dataset-curation skill), answer correctness is the best single metric to evaluate response accuracy.

Logical consistency is a reference-free metric available out of the box that understands the entire execution trace of your agent. Because it does not rely on ground truth responses, it is easy to use. It can also catch agent mistakes that would be missed by looking at the response alone, such as an incorrect tool call or the agent not adhering to system instructions.

Custom evaluation metrics can be created by uploading a run configuration YAML to a Snowflake stage. Custom metrics are configured with a name, score ranges, and prompt:

Custom metric prompts can use template variables including input, output, ground_truth, tool information and more that are automatically populated from the evaluation run. These let you explicitly reference specific fields in your prompt.

The complete agent execution trace is always provided to the LLM judge as context, regardless of which template variables you use in your prompt. Template variables give you a way to explicitly call out specific fields in your scoring rubric, but the judge has access to the full trace — including all tool calls, tool outputs, intermediate reasoning steps, and span details — even if you don't reference them via variables.

See the documentation for all available template variables.

Once your evaluation metrics are well aligned with your human annotations, you can start improving your agent. This can be a much faster process because you get quick, automated feedback for each change.

Every improvement cycle should map cleanly to the agent version lifecycle:

For each iteration:

1. Make changes on the live version (ALTER AGENT)
2. Run evals against the live version or a committed candidate
3. Commit when you want a stable snapshot to compare against prior runs

For each new version of your agent, use ALTER AGENT. This ensures that evaluation runs are preserved. Avoid using CREATE OR REPLACE AGENT — it is destructive to past evaluation runs and monitoring traces.

To determine what changes to make, you can:

Try it out by asking Cortex Code:

Make one change at a time, and create a new agent evaluation run with each change. This ensures the effect of each change is properly isolated so you can measure its impact precisely. Be sure to not only look at overall results but understand trace-level regressions as well.

Continue to iterate through ALTER AGENT until reaching the pre-decided production threshold for all metrics.

To find the optimal performance for your use case, start with a powerful SOTA LLM such as the latest Claude Opus. Then, using the performance with Opus as a benchmark, create an evaluation run with a cheaper, less powerful LLM. Inspect the results and determine if the less powerful LLM can achieve similar performance.

By starting with a SOTA LLM, you determine the maximum performance of your agent given the tools available.

With programmatic evaluations, you can integrate agent quality checks directly into CI/CD pipelines. This brings LLMOps closer to traditional software development best practices.

Cortex Agent versioning offers the ability to create agent snapshots, allowing you to test, promote, and roll back across multiple versions.

Before setting up CI/CD, store your agent artifacts in version control:

Git history provides an audit trail of every agent change, and pull requests provide code review before any modification reaches a shared environment.

A typical GitHub Actions pipeline:

1. A developer opens a PR that modifies the agent spec
2. The CI workflow deploys the candidate spec to a staging agent
3. The workflow uploads the eval config YAML to a Snowflake stage and starts an evaluation run
4. The workflow polls until the evaluation completes, then retrieves results
5. A quality gate checks whether metrics meet your thresholds (e.g., answer_correctness >= 0.75)
6. If the gate passes, the PR is allowed to merge. If it fails, the PR is blocked and the developer can inspect the evaluation results to understand what regressed

Consider using progressive thresholds across environments: lenient and advisory in dev to avoid blocking experimentation, stricter hard gates in QA, and the highest thresholds in production paired with automatic rollback on failure. Set thresholds based on observed baselines from multiple eval runs rather than aspirational targets — thresholds set too aggressively create flaky gates that erode trust in the pipeline.

Depending on your confidence in your eval set and the stakes of your use case, you can choose a deployment strategy. In all cases, agent versioning is what makes safe deployment possible.

Cortex Agent versioning gives you a clean separation between development and production:

1. Develop on the live version

Iterate on prompts, tools, and configs. Test interactively or against evals.

2. Commit to create a production candidate

This creates a new immutable version (e.g. VERSION$4):

3. Test the new version explicitly

Run evals against VERSION$4. Optionally route internal traffic to it via a staging alias.

4. Promote the agent to production

All traffic pointing at production now uses the new version:

5. Rollback instantly if needed

This alias-based routing is what enables safe, reversible deployments.

At a defined cadence (hourly or daily), test your eval set with the production version of your agent to detect changes in performance. Choose the cadence based on your use case and your willingness to absorb the cost of running evaluations. We recommend testing no less than daily.

Your agent can change in unexpected ways due to changes in the underlying LLM, tools, or data. Regular evals give you a proactive signal before users notice degradation. While CI/CD evaluations protect against agent configuration changes, cadence-based testing catches issues from outside the agent configuration including model degradation, data changes, or tool configuration drift.

Store your eval dataset (YAML config) in a stage, call your evaluation procedure, and schedule it:

Start the task:

aside positive To help avoid unexpected regressions due to LLM changes, pin a specific model for orchestration rather than choosing auto. This ensures the LLM used is the one you have determined best for your use case.

Once your agent is in production, all usage lands in Snowflake's agent observability event logs. To analyze it, you can use the Cortex Code skill agent-observability-report. It pulls 110 metrics from Snowflake agent observability event logs: usage, latency percentiles, token economics, tool execution stats, user feedback breakdowns, and conversation depth.

Analyzing these metrics helps you deeply understand how your agent is being used and how well it is working. You can identify cases where your agent struggles by looking for low usage, high latency or token costs, reliability issues, low completion rates, or negative feedback.

Then, add these queries to your evaluation dataset and improve your agent by iterating either manually or by using agent optimization in Cortex Code.

Alerts can monitor metrics of interest from Snowflake agent observability event logs. Below are examples for evaluation accuracy, latency, reliability, and user feedback thresholds.

First, create a notification integration (one-time setup):

Agent evaluation threshold alert

Agent latency threshold alert

Agent reliability threshold alert

User feedback threshold alert

Congratulations! You now have a comprehensive understanding of how to evaluate, version, deploy, and monitor Cortex Agents in production.

1. Start with real user queries for your evaluation dataset — synthetic data is a fallback, not a default
2. Use ALTER AGENT and versioning instead of CREATE OR REPLACE AGENT to preserve evaluation history
3. Make one change at a time and evaluate after each change to isolate impact
4. Combine CI/CD evaluations with scheduled cadence-based testing for comprehensive coverage
5. Pin your orchestration LLM to ensure reproducible results
6. Set up production alerts for evaluation accuracy, latency, reliability, and user feedback to catch issues before users do