Introducing Data-eng-bench: Why You Need "Data-Native" Harnesses for Data Engineering

As AI agents move from writing individual functions to owning end-to-end workflows, data teams face a harsh reality: general-purpose coding agents, regardless of their proficiency in Python or SQL, can struggle with production-grade data engineering. They often fail to complete tasks or incur high costs. This is a demanding test of an agent's ability to navigate a large warehouse, reason about business logic and handle edge cases. This is precisely the kind of work that has historically been difficult to measure.

To measure this capability, we're open sourcing data-eng-bench, a benchmark for repository-level data engineering created in joint work with Bespoke Labs. Tasks in data-eng-bench hand an agent a live dbt project connected to an enterprise-scale data warehouse and ask it to build and fix real data pipelines. Resulting dbt models are then evaluated against the business rules and edge cases a working pipeline must satisfy. Relative to ADE-Bench, one of few other open benchmarks in this domain, data-eng-bench offers higher scale (103 tasks versus 63 tasks), robust tests that check invariants of the output data pipeline, and more complex task specifications.

Our testing on data-eng-bench reveals a clear divide between generic agent harnesses, including Claude Code and OpenAI Codex, and the data-native agent harness Snowflake CoCo. Specifically, CoCo leverages its understanding of the data platform to consistently offer higher quality (that is, task completion rate) while incurring significantly lower cost. In more detail:

• The harness matters for quality; the effect varies by model: Holding the harness fixed at CoCo, Pass@1 varies drastically across models: from 73.8% with Opus 5 to 64.1% with GPT 5.6 Sol to 56.6% with Sonnet 5, a 17-point spread. The impact of the harness on quality depends on the model: Opus 5 performs the best with CoCo, dropping by ~4pp in Pass@1 with Claude Code; Sonnet 5 performs equally well with both CoCo and Claude Code; GPT 5.6 Sol performs the best with CoCo, dropping by 3.6pp with Codex.
• The harness matters significantly for cost efficiency: CoCo achieves higher quality at lower cost than other harnesses: With Opus 5, CoCo reports a 4pp higher Pass@1 at 3.9x lower cost than Claude Code. With Sonnet 5, CoCo reports the same Pass@1 at 2.3x lower cost than Claude Code. With GPT 5.6 Sol, CoCo reports a 3.6pp higher Pass@1 than Codex with Codex incurring 1.5x the cost of CoCo. CoCo completes tasks with 1.5x fewer tool operations and 2.2x fewer agent steps than Claude Code. Task solving patterns show that CoCo adopts a more efficient exploration and validation strategy, requiring 1.7x fewer SQL queries and 1.2x fewer file reads during these phases. CoCo also stays on scope in 2.2x more instances, skipping unnecessary DuckDB cross-validation that Claude Code performs by default.

While these results indicate that frontier agents have made big strides in tackling data engineering tasks, there is still headroom for further improvement. The strongest configuration we benchmarked, Snowflake CoCo with Opus 5, successfully solves 73.8% of tasks on the first attempt (average Pass@1 across 3 trials), but only 64.1% of tasks pass on all three runs (Pass^3). Sonnet 5 reports Pass^3 at 40.8% and GPT 5.6 Sol reports Pass^3 at <56% across harnesses.

In the rest of this blog, we provide an overview of the data-eng-bench benchmark, report how frontier agents perform on it in terms of quality, token and cost efficiency, and what it means for teams adopting coding agents for data engineering tasks.

Get the benchmark.

The data-eng-bench benchmark

We designed data-eng-bench to mirror how enterprise data engineering teams actually operate: many source systems, layered staging-to-mart development and a shared project that must be maintained by many users. Here is a breakdown:

One shared data warehouse: Every task runs against a single, persistent retail data warehouse — 579 source tables across 19 schemas, roughly 8,000 columns in total — spanning orders, finance, procurement, marketing, inventory and more. That's broader than prior data engineering benchmarks, and each task requires navigating the same schema with different requirements.

103 tasks of two variants: Each task gives the agent an instruction in natural language, a starting dbt project, and the data warehouse, then asks it to produce or correct models that materialize the right tables. The two variants are as follows:

• Build (84 tasks): Author new models and keep the existing pipeline running. Build tasks can be further divided into (a) Greenfield: Scaffold a brand new dbt project from an empty state; and (b) Brownfield: Add new models into an existing multi-layer project while reusing and preserving dozens of production models.
• Fix (19 tasks): Diagnose and repair a subtly broken production model.

Real dbt mechanics, not free-form SQL: Agents work through dbt primitives: source declarations (45 tasks), reusable macros (13 tasks) and per-model materializations across staging, intermediate and mart layers. And 82% of gold solutions wire models together through explicit ref dependencies, averaging roughly nine ref calls each, which the agent must resolve into a coherent, compilable DAG. Those DAGs are meaningfully large: a median of four models per solution, and up to 42 for the biggest pipelines.

Diverse, high-difficulty business rules: The tasks live in domains where data engineering is proven difficult — finance (ledger reconciliation, revenue recognition, multi-currency settlement), inventory (LIFO/FIFO costing, turnover, stockout risk), marketing (multi-touch attribution, campaign ROI) and customer analytics (RFM segmentation, churn, lifetime value). The business rules depend on the order and timing of events, accumulated or allocated values across many interdependent tables, and edge cases that should be handled exactly as defined.

Graded by what the pipeline does, not what it looks like: Each task ships with a hidden verifier suite of 10–50 assertions that materializes the agent's models and interrogates the resulting tables directly. Those assertions encode the invariants a correct solution must satisfy (for example, output grain and column contracts, formula-level correctness, edge-case handling, idempotency across re-runs) and for the hardest tasks, independently recompute the expected result in Python.

Scoring is two-fold: At the test level, assertions provide partial credit (the fraction that pass). At the task level, a task counts as resolved only if every assertion passes — so a pipeline that's directionally right but wrong at the edges earns no task-level credit. This lets us separate "can generate a plausible model" from "got the whole pipeline correct" and pinpoint exactly where solutions fail.

Measuring agent quality and cost

We evaluate a combination of harnesses and models on data-eng-bench, isolating how much of an agent's performance comes from the scaffold versus the underlying model.

We test three models spanning proprietary families — Opus 5 (Anthropic), Sonnet 5 (Anthropic) and GPT 5.6 Sol (OpenAI), run with several harnesses that differ in context management, planning and tool interfaces: Snowflake CoCo, Claude Code and Codex. We recommend Code mode (cortex --mode code via CLI) while using CoCo on dbt tasks for quality/cost balance. For every combination, we report two quality metrics: Pass@1 and Pass^3 rates. Pass@1 and Pass^3 capture complementary dimensions of quality, measuring the capability ceiling and consistency across runs respectively. We also report the average cost per trial (in $) and the average number of total tokens (across input, cache and output) per trial.

Tab 1. Strict, task-level resolve rates (%). Pass@1 = mean single-attempt pass rate (averaged across three attempts); Pass^3 = resolved if all three attempts succeed. Cost multipliers are set to 1 for each combination of CoCo with the 3 models (Opus 5, Sonnet 5, GPT 5.6 Sol); the cost multiplier for other harnesses in conjunction with the corresponding model is compared against this baseline. For example, Claude Code with Opus 5 is 3.9x more costly than CoCo with Opus 5.

A key takeaway is that the harness matters significantly for quality and cost efficiency. On data-eng-bench, CoCo consistently achieves higher quality at lower cost than other harnesses:

• With Opus 5, CoCo reports a 4pp higher Pass@1 at 3.9x lower cost than Claude Code.
• With Sonnet 5, CoCo reports the same Pass@1 at 3x lower cost than Claude Code.
• With GPT 5.6 Sol, CoCo reports a 3.6pp higher Pass@1 at 1.5x lower cost than Codex.

We now delve into why CoCo is more cost efficient than Claude Code.

With Opus 5, CoCo requires 1.5x fewer tool operations and 2.2x fewer agent steps than Claude Code to solve a task on average. The agent spends most of its time on the predevelopment exploration and postdevelopment validation phases, under both harnesses. Overall, CoCo issues 1.7x fewer SQL queries, concentrates file reads to the exploration phase and produces 1.9x fewer file writes. The tool call patterns in different phases on task execution reveal the strategies adopted by Opus 5 under the two harnesses:

1. Phase 1 — Predevelopment setup and exploration: CoCo requires 1.3x fewer tool operations before writing or editing the first SQL file. Both agents spend similar effort reading existing models and project config (6.6 vs 7.1 file reads). The divergence is in SQL: CoCo uses 2x fewer queries for schema exploration and source data profiling before writing.
2. Phase 2 — dbt model development: Both agents primarily write SQL files (4.4 vs 5.6 file writes). Claude Code additionally reads more files during this phase — consulting existing staging models and macros mid-draft — while CoCo completes all reading in Phase 1 and writes without look-back.
3. Phase 3 — Build, validate and iterate on Snowflake: CoCo requires 1.7x fewer tool operations, with the gap driven primarily by Claude Code issuing 1.5x more SQL queries to verify the built models and 3x more shell setup commands around each build iteration.
4. Phase 4 — Cross-validation with DuckDB: Interestingly, Claude Code performs DuckDB cross-dialect validation in 2.25x more trials than CoCo. This step is unnecessary given the backend is specified as Snowflake and evaluates the agent's ability to avoid irrelevant details in the task instructions.

These differences reflect two distinct strategies. CoCo follows a plan-then-execute approach — it front-loads exploration, writes directly, verifies once and stops. Claude Code follows an explore-and-refine approach — it interleaves reading with writing throughout development, wraps each phase with additional verification passes and treats cross-dialect validation as a default rather than an optional step.

Get started

Data-eng-bench is open source. Whether you build agents, harnesses or the models underneath them, it's a realistic, hard-to-saturate testbed for measuring autonomous data engineering.

• Explore the benchmark: here
• Contribute: add tasks, harnesses or model results and tell us what you find.

We're excited to see how far agents can go on the work that quietly powers every analytics stack. Try data-eng-bench, run your own model–harness combinations and share your results!