Whether you're building agents for a bank, a lender or any regulated business, this guide shows how to test a Cortex Agent like any other production system: with a ground-truth dataset, pinned versions and a gate that blocks a bad release before your customers see it.

You'll evaluate a lending operations agent for Northbank, a fictional UK consumer lender, and compare three versions of it. Along the way you'll see why answer correctness alone isn't enough:

This guide focuses on testing an agent for a regulated use case. For the basics of the evaluation framework, see Getting Started with Cortex Agent Evaluations and Best Practices for Evaluating Cortex Agents.

All data in this guide is synthetic.

A Cortex Agent answers a question by planning, calling tools (a semantic view through Cortex Analyst, a Cortex Search service, a stored procedure) and writing a response. Two answers can read equally well while one of them used the wrong date filter, counted duplicate applications or ignored a policy that was replaced last month.

In consumer lending that difference matters. Under the FCA Consumer Duty, a lender must avoid foreseeable harm and treat customers in vulnerable circumstances fairly. An answer that recommends a default notice for a customer who has just told you they're in hospital isn't a quality issue: it's a conduct issue.

Cortex Agent evaluations run a dataset of questions through a specific agent version, score each answer with built-in and custom metrics, and keep every trace for inspection in Snowsight. This guide uses:

Download northbank-agent-evals-quickstart.zip and unzip it. It contains the SQL scripts, the evaluation configs, the policy PDFs and the evaluation dataset.

Every script runs in a Snowsight SQL worksheet. Open each one, select Run All unless a step says otherwise, and run them in order.

Run sql/00_setup.sql. It creates:

In Snowsight, go to Ingestion » Add Data » Load files into a Stage, and upload into the root of each stage:

Then re-run the four statements at the end of 00_setup.sql to check that every file is in place.

Run these three scripts in order:

1. sql/01_synthetic_data.sql builds the loan book and the customer notes.
2. sql/02_documents_and_search.sql parses the PDFs with AI_PARSE_DOCUMENT, chunks them by heading with version and status metadata, and creates two Cortex Search services.
3. sql/03_semantic_view.sql creates the semantic view behind the loan_book tool.

The data generator is deterministic, and the expected answers depend on it. The last query in 01 should return:

Northbank's credit operations and collections teams use an agent with four tools:

Run sql/04_agent_versions.sql. It creates the custom tool and three committed versions of the agent, changing one thing at a time:

CREATE AGENT ... FROM SPECIFICATION commits the first specification as VERSION$1. Each later version is created by changing the live specification and committing it:

The script ends with SHOW VERSIONS. Note that VERSION$3 is now the default: committing a version moves the default to it. You'll come back to that in the step on pinning.

An evaluation only measures what its dataset makes difficult. So before writing a single question, the traps went into the data, modelled on how lending data really fails:

The evaluation set has 40 questions across core metrics, cross-tool reasoning, policy change, affordability, data quality, vulnerability (with non-vulnerable controls) and refusals. Every expected answer was computed in SQL against the source tables, never by asking the agent. Here's one row:

This row contains:

For refusal questions, such as "Forecast the Bank of England base rate", ground_truth_invocations is an empty list: calling any tool is wrong. A row without invocations silently drops out of TSA and TEA, so put them on every row.

Run sql/05_eval_dataset.sql. It loads the file into a table and registers two datasets with SYSTEM$CREATE_EVALUATION_DATASET: the full 40 questions, and a 5-question probe for cheap experiments.

An evaluation is described in a small YAML file. Here's eval_configs/version3.yaml, without the custom metrics:

It names the exact agent version, the dataset and each system metric pinned to a metric version.

Open sql/06_run_evaluations.sql and run it statement by statement, not with Run All, which would start every run at once. Start with the probe:

Snowflake invokes the agent once per question, records the trace and computes each metric. Check progress with the STATUS call in the script, or in Snowsight under AI & ML » Agents » LENDING_OPS_AGENT » Evaluations. Then start one full run per version (v1_r1, v2_r1, v3_r1).

In the original build, the baseline (VERSION$1) scored 0.92 on answer correctness. On that number alone you'd ship it. Tool selection was 0.84, and the gap is where the useful findings were:

TSA is computed as matched / max(expected, actual), so missing tools, wrong tools and redundant calls all cost points. Because it's deterministic, a change in TSA between runs means the agent behaved differently, not that a judge changed its mind.

The queries in 06 find these cases for you: correct answers whose tool selection wasn't perfect, and the judge's explanation for every low score.

VERSION$2 kept model A and rewrote the instructions to fix the baseline's failures: deduplicate by default, use the received date, one call per tool, no tools for out-of-scope questions, plus the vulnerability procedure and refusal rules.

It fixed what it aimed at. The declined question went from 0.33 to 1.0, and the cross-tool and base-rate questions reached TSA 1.0. Overall, though, it got worse:

The traces showed why. With the procedure written into its instructions, the agent stopped retrieving it. On vulnerability cases it read the notes but skipped the case record and the procedure document; on two refusals it called no tool at all. The answers still sounded right, so answer correctness moved only 0.024. TSA on the refusal tier fell to 0.33 and on the vulnerability tier to 0.55. In a regulated firm, an answer that cites no retrieved procedure isn't auditable, even when it happens to be correct.

To compare your own runs, select two or three runs in the Evaluations tab, select Compare and turn on Hide unchanged metrics. The by-tier query in 06 shows the same regression in SQL: averages hide a regression that's confined to one tier.

VERSION$3 kept VERSION$2's instructions and changed only the model. It's the best of the three.

Two things can change under you between runs: the agent and the judge.

The agent. Committing VERSION$3 made it the agent's default automatically. An evaluation targeting the default would have switched silently from model A to model B. In CI and scheduled runs, always target a committed VERSION$N, use LIVE only while iterating, and treat a change of default as a release event.

The judge. Metric versions fix the judge model, prompt and rubric. Three experiments on the same agent version showed what happens when you don't pin carefully:

Pin v3_0 for every system metric, keep it fixed across runs (scores from different versions aren't comparable), and upgrade deliberately when a deprecation is announced. Also make sure the judge isn't your agent's orchestration model; otherwise the agent is grading itself.

The built-in metrics don't know what the Consumer Duty requires. Custom metrics let you write the rubric yourself and choose the judge with the model key. The full-run configs add two, each scored 0, 5 or 10:

Here's an extract of the second one:

The {{tool_info}} placeholder passes the retrieved chunks to the judge, so it can check grounding against what the agent saw rather than what it remembered.

Both metrics were scored with two judges on identical traces (eval_configs/judge_comparison.yaml repeats the experiment). They agreed on 37 of 40 rows for each metric, which looks reassuring. But one judge gave consumer duty 10/10 on every row, so it never flagged anything. The other marked down answers that blocked a default notice but left out the Specialist Support referral. On policy grounding the roles reversed: the lenient judge scored one answer 0 for broadening the rule (the procedure requires a vulnerability check before a Collections referral or default notice; the answer said "before any collections action"), while the stricter one gave it 10.

Before trusting a custom metric, check that its scores spread across the range and read the explanations. Agreement between two judges proves nothing if one of them scores everything 10, and neither judge is uniformly the stricter one, so calibrate each metric on its own.

Each run re-invokes the agent, so a pinned agent and a pinned judge still produce new traces. Run VERSION$3 twice more with the variance runs in 06 (v3_r2, v3_r3). In the original build, the three runs looked like this:

The averages are stable, but individual questions aren't: between identical runs, 11 of 40 questions changed their TSA score, meaning the agent changed which tools it called. These are the questions to fix first, and they're invisible in the average. The last query in 06 lists them.

Run sql/07_ci_gate.sql (Run All is safe). It creates:

Once your runs from 06 have completed, run sql/07b_gate_your_runs.sql statement by statement. Its first query suggests floors from your own three runs; the rest scores your runs through the gate:

In the original build, VERSION$2 failed with "answer_correctness 0.893 < 0.925", plus the tool execution and tool selection floors, and all three VERSION$3 runs passed. A fourth VERSION$3 run, started by the Task itself, also passed.

The per-row rule earns its place. In a later rerun of VERSION$3, every average cleared its floor (answer correctness 0.942, tool selection 0.902), yet the gate blocked the run with consumer_duty_min 0 < 5. On question 30, a control case, the agent read the neighbour's bereavement as the customer's own and advised against a default notice that policy requires. Answer correctness, consumer duty and policy grounding all scored it 0. Your runs may or may not hit this: that's the variance you just measured.

Judge usage is attributable per run and metric by joining each evaluation span's inference_request_ids to CORTEX_REST_API_USAGE_HISTORY, which records input and output tokens but no credits, so you apply the list rate yourself. Agent invocation credits sit in CORTEX_AGENT_USAGE_HISTORY. Run sql/08_cost.sql for your own runs (ACCOUNT_USAGE views can lag by about 45 minutes).

Here's one 40-question VERSION$3 run from the original build, priced at on-demand list for an Enterprise account on AWS EU Dublin:

How it's priced (Snowflake Service Consumption Table):

Capacity contracts lower these prices.

Logical consistency sends the whole trace to its judge, so it takes 73% of judge tokens and 65% of judge spend, about 23 times the cost of answer correctness. Leaving it out of nightly runs and keeping it for release runs takes a run from about $12 to about $6.70. Tool selection adds no judge cost: once you've paid to invoke the agent, it's free to include on every run.

Evaluations don't pass session attributes, so an agent relying on row access policies keyed on session variables won't score what users see. MCP tools aren't called during an evaluation, and files written by the code execution tool aren't kept. Cover these with other tests.

Run sql/99_teardown.sql to suspend the Tasks and drop the database and warehouse. The evaluation runs and their results are stored with the agent and are removed with it.

Most of the work is in the dataset: expected answers computed independently, traps built from how your data really fails, and expected tool calls on every row. With that in place, the platform does the rest. From here, you can:

docs/adapt-to-your-use-case.md in the project maps each step of this guide to your own agent, with a checklist.