Beyond Adoption: How to Measure Whether AI Is Changing the Work That Matters

Consider two organizations that each log 1,000 AI interactions in a week. In the first, most of those interactions are explanations and lookups. In the second, people are building data pipelines, investigating anomalies and running multistep workflows.

Most enterprise AI dashboards would score those two organizations identically. The first generation of AI developer metrics measured access and activity: how many people have a license, how many are active and how many prompts they send. But these metrics cannot tell you whether AI is driving meaningful work outcomes.

Answering that question means tracking what people use AI for and whether that is moving. Here is the framework we use to measure AI value and what we found when we applied it to two of our own teams.

A four-part framework for measuring AI value

1. Start with the work AI is doing

Asking an assistant to explain a SQL statement is a different request from asking an agent to build a pipeline or investigate a failure. Group the work the AI is doing into three levels:

• Foundational work: The AI explains code, answers questions and helps someone understand an unfamiliar concept.
• Applied development: The AI writes and modifies code, explores data, troubleshoots and builds reports or applications.
• Agentic work: This is a multistep process. The AI reasons across context, uses tools and helps finish an outcome instead of answering just one question.

This breakdown describes the work rather than the volume, which gives you something to check. If AI is creating real capacity, the hours saved should show up in the agentic level.

2. Measure adoption and fluency separately

The next question is whether higher-value usage has spread beyond just a few people. Two dimensions cover that:

• Adoption: This measures how broadly AI is used across the eligible population.
• AI fluency: This shows how sophisticated the work that people delegate to AI has become.

Together these dimensions describe four positions.

Figure 1: Measuring AI adoption and AI fluency 

3. Use industry benchmarks for context

A number on its own is hard to act on. If 35% of your eligible developers actively use an AI coding agent, that figure alone will not tell you whether 35% is good. It depends on what comparable organizations are doing, and expectations vary by industry. A software company and a regulated insurer are not working from the same baseline.

Peer benchmarking gives you a relative view on both dimensions. Instead of reporting that adoption is 35%, it can tell you whether that puts you among the emerging, developing, proficient or leading organizations in your industry.

The same applies to fluency. One organization may have broad adoption but little agentic work. Another may have a handful of users doing sophisticated work that has not spread. Those two findings call for different responses.

4. Look at the gap between your best users and everyone else

Your most advanced users are the signal worth watching.

In most organizations, a few people find effective AI workflows before their colleagues do. They learn to hand over larger tasks, provide better context and combine tools. Treat those patterns as a roadmap, not a collection of anecdotes.

The practical question is how to move a behavior from one person to 100. Four tactics worked for us:

• Have power users teach, not just demo: A hands-on workshop led by someone on the team who already gets results beats a vendor training session. Watch for whether weekly usage holds a month later, not whether the room was full. A workshop that changes habits looks different from one that produces a spike.
• Turn their workflows into recipes and templates: When someone solves a problem well, have them write it down as something a colleague can run rather than read. The test is whether the next person executes it without asking the author a question.
• Run a regular show-and-tell: Provide a standing slot where people walk through a workflow they built, including the things that did not work. The failures travel better than the successes, because they tell everyone else which problems are worth attempting.
• Make the library findable: This is the step most teams skip. Once you have dozens of reusable workflows, discovery, rather than creation, becomes the constraint, and a library nobody can search is a library nobody reuses.

The aim is to move proven practice from a few people to many, and the mechanism matters more than the enthusiasm.

What to do with the answer

Measurement earns its keep when it points to an action instead of a status. There are three common positions, and each suggests a different course of action:

• Sophisticated but narrow: Early adopters use agents for advanced work, and most eligible people have not started. Remove access friction, share internal examples, and expand enablement.
• Broad but shallow: Many people use AI, mostly for explanations and simple code generation. Show them how to hand over larger tasks through examples, templates and hands-on training.
• Broad and sophisticated: More adoption is no longer the priority. Find the repeatable workflows, build reusable agents, and measure outcomes such as development velocity or operational efficiency.

The same metrics support different recommendations depending on where you sit.

Applying the framework to our own teams

We spent two quarters using this framework on Snowflake CoCo, our agentic AI development assistant, inside Snowflake. Two teams with very different work landed in a similar place, and in both cases the gains came from agentic workflows rather than foundational use.

Product data science: Automating a diagnostic runbook

Our product data science team runs thousands of data pipelines. In the first quarter of FY 27, those pipelines needed attention 960 times, which is normal at that scale. Each case took a skilled engineer about 30 minutes to diagnose. That works out to roughly 160 hours a month of expert time spent on diagnosis. 

CoCo now performs that diagnosis. Across the Q1 and Q2, working alongside the team CoCo has: 

• Improved more than 500 pipelines
• Generated more than 100 merged PRs
• Brought the time from incident to a tested and reviewable fix to under 16 minutes
• Reduced hundreds of daily wasted credits

Optimizing an expensive pipeline isis valuable and rarely urgent, so it is the work that waits indefinitely. An agent that proposes those changes as a matter of course does not need a quiet week to get to them.

The team had done the important work before CoCo was involved. They had a rigorous diagnostic runbook, written over hundreds of incidents and general enough to be useful in getting to the bottom of many different sorts of incidents.. That discipline is why incidents were resolved consistently. The runbook was already good enough to scale. What it needed was a way to run without waiting for someone to be available.

So the team gave the runbook to CoCo and had an alert trigger it instead of a keystroke. Same steps, same standard — but it ran on the incident's schedule instead of the team's.

That is the frontier-to-mainstream pattern in its most literal form. A method that lived with one team became a workflow that runs on its own. If a method is clear enough to onboard a new hire with, it is clear enough to hand off to AI. And if it currently lives in one expert's experience, writing it down is worthwhile on its own terms.

How review works

CoCo proposes and people decide. Every change arrives as a pull request that a named owner reviews and approves. Merging and deploying stay with the team.

That boundary is deliberate, and it moves to where the work sits. Reviewer attention becomes the thing to manage, and a team that produces more proposals than it can read has relocated its constraint rather than removed it. In exchange, the engineers moved 160 hours of diagnosis into the judgment call at the end, which is the part that needed them.

The practical lesson: Design the work so its output is always something a person can review. Review becomes the control, and that is what makes it straightforward to run against production pipelines.

Marketing: from 11.6% confident to 70% weekly use

The clearest evidence that this generalizes came from a part of the company with almost no engineers in it. Snowflake's marketing organization is nearly 600 people, most of them more comfortable with a campaign slogan than a SQL statement.

In February 2025, an internal survey of the North America marketing team found 11.6% felt very confident using AI in their role. When Snowflake CoCo arrived as a research preview late that year, eight North America marketers used it weekly. 

By May 2026, weekly CoCo usage across the global marketing team had reached 70%. Separately, 93% said they use AI tools daily. The first is observed behavior and the second is self-reported, which is why we lead with the first.

What they built:

A motion graphics export pipeline that turned a full week of Summit work into 20 minutes.

A win-loss reporting app on CoCo and Streamlit that schedules interviews and generates the quarterly leadership deck. Three hours a month of candidate identification plus a week of deck work is now roughly 90% automated.

An inbound automation engine that handles 2,000 to 3,000 daily email replies to global campaigns with no human intervention, replacing a planned software purchase.

Marketers have since created more than 600 personal and project-level skills, and 100 to 200 have built their own from scratch. The median user runs seven skills a week. 

Both dimensions moved. Adoption went from a small confident minority to 70% weekly use, and fluency went from asking questions to authoring workflows that run without their author present. A dashboard watching only adoption would have missed half of it.

That came from quarterly AI Days, regional hackathons, a newsletter and office hours, all peer-led.

Why platform context matters

Any capable AI assistant can read an error message. These results hold up in production because CoCo works from inside the data platform, where the context that a real fix depends on is already there.

It reads the data catalog and lineage, checks whether data is current and sees which parts of the business depend on a given table. In interactive use it runs with the permissions of the person using it, so its scope matches theirs. That grounding is what lets a runbook written for people transfer to an agent.

We published the measurement rather than asserting it. Snowflake AI Research built data-eng-bench, an open benchmark of 103 real data engineering tasks graded on whether the resulting pipeline works. On that benchmark, CoCo leads the general-purpose coding assistants we tested on quality for cost.

How to scope a first workflow

The pattern behind these results is narrower than "adopt AI," and that is why it worked:

• Start with work that is repeatable and well specified: Respond to a known category of alert, routine cleanup, the recurring analysis someone runs every Monday. The prerequisite is a documented method, not a particular technology.
• Keep judgment-heavy work interactive: New problems are where a person and an assistant together beat either alone, and knowing which category a task falls into is most of the skill.
• Track consistent use rather than volume: Track consistent use rather than volume. The number of days someone chooses to use an assistant tells you more than the number of prompts they send, and the signal is strongest when the work produces something reusable. 

We are still learning where this pattern applies. We report pipelines improved and issues resolved. The ratio of proposals raised to proposals accepted is the next figure we plan to publish, because it shows how much reviewer attention the pattern asks for in return. We are instrumenting it now.

Usage, understanding, benchmarking, action, impact

Prompt counts, active users and consumption still matter as a starting point. A fuller picture answers four questions:

• Who is using AI?
• What work do they use it for?
• How sophisticated is that work, compared with the work of peers and your own best users?
• What should you do next?

Those four steps run in a sequence: usage, understanding, benchmarking, action, impact. Each one depends on the one before it.

That sequence also explains the two organizations we opened with. The difference between them was never the interaction count. One had moved a documented method into agentic work and the other had not, and only the second kind of measurement makes that visible.

So pick one repeatable, well-documented process your team already runs, and measure what it costs you today.

See how we measure agent quality and cost on real data engineering work in data-eng-bench, Snowflake AI Research's open benchmark of 103 production-grade tasks.