SnowPR: Benchmarking Coding Agents on Snowflake's 43-Million-Line Monorepo

Summary

• Public benchmarks don't look like our everyday work: SWE-bench tasks come from Python projects with about 3,000 source files each, and models are increasingly tuned to such sets. Our monorepo has 63,000 nontest source files and 17 million lines of nontest code, mostly Java and C++, built as one Bazel graph.
• SnowPR is real Snowflake work: 65 merged pull requests (PRs) across 11 product areas, from the SQL compiler to the C++ execution engine. Each is graded by the hidden tests that shipped with it, run with our own Bazel builds.
• Built with extra hardening: An LLM judge narrows thousands of merged changes, and available tasks are validated by human authors. Each task needs enough new and regression tests, passes with the real fix and fails without it, and survives 11 automatic checks and an adversarial review. Every stage but the description runs from a script, so SnowPR can follow our CI.
• Cortex Code's REM mode* helps: REM mode exposes the agent's file, shell and SQL operations as Python functions inside one REPL tool. With opus-5, it matched the pass rate of code mode*, Claude Code and Cursor at the lowest cost, and solved the most tasks of any agent, 62 of 65. With haiku-5.5, it passed more trials than Claude Code, at $0.24 a trial.
• Monorepo features make tasks hard: Changes that span Java and C++ pass 55% of trials, against 80% for Java alone.

* REM mode (REPL Execution Model): a Cortex Code mode that exposes the agent's file, shell, search and SQL operations as Python functions inside one REPL tool. It is internal only for now and not yet available to customers.

* Code mode: a Cortex Code mode that gives the agent only file, shell and code-search tools (documentation).

At Snowflake, engineers write more and more of their code with AI coding agents, and the number of agents and models to choose from grows every month. Although public benchmarks help us follow that market, they cannot tell us which agent works best on the actual code our engineers change every day.

Most public benchmarks draw their tasks from a few dozen open source projects, and models are increasingly tuned to score well on them. A high score there says little about our 43-million-line Java and C++ monorepo, built with Bazel, where many tests start their own FoundationDB. So we built SnowPR, a benchmark made from our own merged pull requests and graded by the tests that shipped with them.

SnowPR has 65 tasks across 11 product areas. Each task is a real pull request that Snowflake engineers reviewed and merged: 62 come from our monorepo and three from a separate Go repository for developer tools. The agent gets the code from before the change and a description of what to build. As in SWE-bench, the hidden tests that shipped with the PR decide pass or fail, run with our own Bazel builds on our remote build farm.

SnowPR also has a second job. Snowflake builds its own coding agent, Cortex Code, so SnowPR is more than a scoreboard for outside agents. It is also a test bench for our own harness. We used it to compare Cortex Code's two modes, code mode and REM mode, with Cursor, OpenAI's Codex and Anthropic's Claude Code, on the same models where we could. REM mode is internal only for now.

An agent here is a harness plus a model, working inside our repository with the context we keep there for coding agents, including skills. We ran multiple models with different harnesses (Figure 1). Each tried each task three times. pass@1 is the share of all trials that pass, and pass@3 is the share of tasks solved at least once.

Figure 1: Pass@3 against cost per trial for seven agents: opus-5 in four harnesses and haiku-5.5 in three, on 65 tasks with three trials each. The x-axis breaks between $0.35 and $1. The line joins the agents that no other agent shown beats on both cost and pass@3. coco-rem is Cortex Code in REM mode and coco-code is Cortex Code in code mode. Cost per trial includes subagents and is billed or token-priced.

What is in SnowPR

SnowPR covers the range of work our engineers do, not one kind of bug (Figure 2). Of its 65 tasks, 52 change the Java control plane: the services that compile queries, manage metadata and run features such as Apache Iceberg™ tables, data sharing and Snowflake Cortex AI. The SQL compiler and the metadata layer lead with 10 tasks each. Eight tasks change the C++ execution engine, two change hybrid tables across Java and C++, and three change Go developer tools.

Figure 2: SnowPR's 65 tasks by product area, split by difficulty we predicted when building each task. Areas above the line are in the Java control plane.

When we built the tasks, we predicted 43 to be hard, 19 medium and three easy. Most hard labels mark tasks from our hard-PR mining pass (Figure 5).

A SnowPR task is also bigger and reaches further than a typical public task (Figure 3).

• Scale and depth: Counting only nontest sources, the way SWE-bench does, our monorepo has 21 times more files and 39 times more lines than the average SWE-bench repository. At the median, the files a task changes sit nine folders deep.
• Larger changes: The reference fix changes 4.0 files and 262 lines on average, against 1.7 files and 33 lines in SWE-bench. In 34 tasks, it touches two or more modules.
• Two languages, one build graph: Java and C++ live in a single Bazel graph. A C++ task runs about 21,700 build actions on our remote build farm before the agent starts, and needs 32 GB of memory.
• Tests far from the code: The control plane keeps most of its tests in one shared test tree. In 33 tasks, the graded test lives there while the code to change lives in a separate module. Grading for 30 tasks runs through two or more Bazel targets.
• Stateful metadata: In 39 tasks, the graded tests start their own FoundationDB, the database that holds Snowflake's metadata. These tests check behavior such as which table version a statement may see while other transactions commit.
• Private code: Every task comes from Snowflake's own history, so none of it is in public training data.

Figure 3: SnowPR next to SWE-bench. Codebase counts are for our monorepo, where 62 of the 65 tasks come from, and cover nontest source the way SWE-bench reports it. SWE-bench values are the means in Table 1 of its paper. SnowPR values are means over its 65 tasks.

Public benchmarks span more projects. SnowPR goes deeper into one company's code, almost all of it from a single private monorepo. Grading works the same way as in the public sets: hidden tests decide pass or fail.

How we built SnowPR, and where we were careful

A benchmark made from real PRs is only as good as its tasks. Every task follows the same recipe (Figure 4), and most steps carry a rule that came from a mistake we made, or nearly made, in early runs.

Figure 4: How a SnowPR task is built and graded. The agent never sees the PR's tests or its code change, and git history stops at the commit before the PR.

Choosing PRs: an LLM narrows, people decide

Most merged changes make poor tasks: mechanical sweeps, tests that check structure rather than behavior, or services a task container cannot start. Size does not separate them, and our changes rarely carry a written explanation, so judging one means reading its diff and its tests.

So an LLM does the first read (Figure 5). In our largest mining pass, simple rules cut 13,965 merged changes from one month to 4,218 that change Java source and add at least one new-behavior test case. An open-weight judge on our own GPUs then scored each one for hardness, meaning the depth of reasoning along one code path rather than patch size, and for gradeability. The judge also rejected PRs whose tests are proxies, pin exact wording or need more than a description can specify. AI agents inspected 384 of the top-ranked candidates, and 84 passed. People then reviewed every task built from them, so every SnowPR task is human-verified.

Figure 5: Our largest mining pass, over one month of control-plane changes. Rules cut the list first, an LLM judge and AI agents narrow it, and people review every task built from what is left. The other SnowPR tasks come from smaller passes, including one for the C++ execution engine.

The LLM gates what enters SnowPR, but never computes a score. It is a filter, not a ranker: In a later round, a judge's predicted pass rates barely tracked the measured ones (Spearman +0.18 over 51 tasks).

Tests, checks and grading

We grade the way SWE-bench does: Test cases the PR added must go from failing to passing, and cases that already existed must keep passing. A new task needs at least one new case on the PR's main change and at least 10 regression cases. When a PR ships fewer, we borrow existing tests that pass before and after the real change, or drop the PR. After the agent stops, the grader injects the hidden tests and runs them with our own bazel test on our remote build farm. Across the 65 tasks, that is 147 Bazel targets with 676 new-behavior and 2,890 regression cases. A trial passes only if every graded test passes.

Before a task ships, the real fix must pass and an empty change must fail. Eleven automatic checks confirm, for example, that the description names every new class or method the hidden tests call. An adversarial review then confirmed 84 findings on 49 tasks before release, eight of them blocking. Descriptions turned out to be part of the measurement too. With the same hidden tests and fix, 10 agents passed 83% of trials on one engineer's write-up of a PR and 57% on another's.

We cannot simply ask Bazel which tests a change affects. For one real task, that query returns 23,946 of the 24,028 control-plane test targets. An agent's edited tree misses our remote cache, so all of them would run in every trial. SnowPR therefore grades with the tests that shipped with the PR.

Now, the process of mining-to-task conversion is fully automated, so we can keep our tests fresh and updated.

REM mode matches the best score at the lowest cost

Cortex Code runs in two modes. Code mode gives the model separate file, shell and code-search tools. REM mode exposes the same operations as Python functions inside one REPL tool, so the model can chain several operations in one step.

With opus-5, all four harnesses pass 84.1% to 85.1% of trials, a spread too small for 65 tasks to detect (Figure 1). They differ in cost instead: Per trial, REM mode cost $6.34, Claude Code $6.51, Cursor $6.93 and code mode $8.19. Against code mode, REM mode was 23% cheaper and used 14% fewer new tokens, meaning input not served from cache plus output (Figure 6). It also solved the most tasks at least once, 62 of 65.

haiku-5.5, a much cheaper model, tells the same story. In REM mode, it passed 70.8% of trials for $0.24 a trial, against 64.1% for $0.26 in Claude Code. Codex passed 74.4%, a gap 65 tasks cannot detect, at almost nine times the cost. The model still sets the tier: Every opus-5 agent passed 10 to 21 points more trials than every haiku-5.5 agent.

Figure 6: REM mode against code mode in Cortex Code with opus-5, at the same effort on the same 65 tasks. REM mode matched code mode's pass rate at lower cost and with fewer new tokens, but took longer. New tokens are input not served from cache, plus output.

Monorepo features are what make tasks hard

The properties that set our monorepo apart are also the ones that predict failure (Figure 7). Averaged over the seven agents in Figure 1, the two tasks that change Java and C++ together pass 55% of trials. Tasks that change three or more source files pass 70% of trials, against 89% for one or two files. Tasks graded against FoundationDB pass 75% against 83% without it, and they make up eight of the 10 tasks that agents pass less than half the time.

Two results surprised us. First, C++ alone is not much harder than Java: 76% against 80% of trials. The hard part is crossing the language boundary, not the language. Second, the difficulty we predicted when building tasks separates the hard tasks from the rest but runs high: Agents still pass 74% of trials on the 43 tasks labeled hard.

What's next

SnowPR gives us an agent ranking we trust at the tier level, and a test bench for our own harness. Next, we want it wider and fresher.

• End-to-end tests: Snowfort is our end-to-end testing system — a test framework plus an execution environment that runs tests against an accurate replica of production Snowflake. Its tests cover about a quarter of the PRs that ship a test, including sharding, replication and data sharing, but a task container cannot start it yet.
• More tasks, refreshed often: About 200 tasks would let us detect a seven-point gap instead of 12. The automated pipeline grows SnowPR from recent PRs, and versioned releases keep results comparable as we revalidate, repair or retire tasks.
• Open models in REM mode: We are setting up a lane that runs SnowPR in REM mode with open models served on our own GPUs.
• Choosing the right model for each task: No agent wins every task. Per-task results like these let us match the model to the complexity of the task, across our product surfaces and for our own engineers' daily work.
• Grading beyond tests: Tests decide correctness, but not qualities such as design or the clarity of an explanation. We plan to add rubric-based LLM judges for those, calibrated on human-reviewed examples, next to the tests.
• More kinds of work: Refactoring, writing tests and answering questions about the codebase are large parts of engineering work that SnowPR does not measure yet.

Any team with a history of merged PRs and tests can build their own SnowPR version. Ours taught us that the benchmark itself needs as much testing as the agents it measures.