How Encoding Observability Expertise in APIs Raised Agent Investigation Success from 7.8% to 69%

What we learned making Observe by Snowflake agent-ready

At Observe by Snowflake, better APIs for finding and interpreting telemetry raised agent pass rates from 7.8% to 69% across 56 curated observability questions in an internal benchmark.

LEARN HOW OBSERVE BY SNOWFLAKE TURNS TELEMETRY INTO INSIGHTS, FAST

When AI agents investigate observability data, they should spend their token budget producing answers engineers can act on, not figuring out where to look.

Here's what we built, how we measured it and what we learned about designing APIs for agents.

Key takeaways at a glance:

• Telemetry access alone isn’t enough. With raw queries and keyword search, Claude Code passed 7.8% on our internal eval of 56 observability questions.
• Grounding APIs raised the initial pass rate to 26.2% by helping agents identify entities and locate their data.
• Domain APIs raised it again to 69% by answering service health and dependency questions directly. They made the biggest difference.
• The full stack reduced cost: It cut latency and token use by 58% each and needed six tool calls per-investigation instead of 15.
• The lesson: Start with user questions, use evals to find agent failures and encode that expertise in your APIs.

Telemetry access alone isn’t enough

Take a question an on-call engineer might ask: “Why are payments slow for acmecorp in the last hour?” To a person, it’s a single, clear question. To an agent, it’s a pile of unknowns: what “payments” and “acmecorp” each refer to, where that service’s data lives, whether it’s healthy and how to chase “slow” to its cause.

Agents field many types of observability questions. “What services are running in production, and which are busiest?” “Is checkout starting to degrade against its usual baseline?” Different questions, but the same underlying work: identify what the terms mean, find the telemetry that covers them and make sense of the answers.

We wanted to understand how well an agent could answer these questions, so we evaluated Claude Code, a coding agent many of our customers already use for observability questions, against investigation prompts with ground-truth answers. We tracked quality, speed and cost: whether the agent reached a correct answer, how long it took and how many tokens it used. The failures showed where the agent struggled: figuring out what names referred to, finding the relevant telemetry and constructing queries to answer common questions.

Those failures showed us where the agent needed help and guided the experiments that followed. We tried resolving vague terms to specific entities, building inverted indexes that connected those entities to their datasets and metrics, and describing the data to the agent in advance. Some approaches helped; others made little difference. Through that trial and error, we learned that making the data available wasn’t enough. The agent needed help finding the right data and using it to answer the question.

Our existing APIs already exposed the underlying telemetry. But agents still had to determine where to look, how to query and how to interpret the results. These experiments helped us decide which parts of that work to encode in the APIs.

Reliable investigations require grounding and domain context

We grouped four capabilities into two API layers, both powered by the same underlying telemetry.

Grounding APIs identify the entities in a question and locate their data.

Domain APIs expose concepts such as service health and dependencies, encoding the observability expertise needed to derive them from telemetry and carrying context into deeper investigation.

Grounding APIs help agents find the right data

Entity resolution: The agent must turn vague terms into identifiers it can use in the telemetry. Our tag values API returns matching key-value labels: an exact lookup can resolve “acmecorp” to customer=acmecorp, while a semantic search can resolve “payments” to service=checkout. We use Cortex Search, Snowflake's semantic search, for the match by meaning, so the agent can find the relevant entity even when the question uses a different term.

A path to the right data: Knowing the entity isn't enough. The agent needs the logs, metrics and traces where that entity actually shows up. We built inverted indexes that connect tag values to the datasets and metrics that carry them. For the payments question, those connections identify where checkout and acmecorp appear, so the agent can query the relevant telemetry without scanning every dataset. Together, entity resolution and these connections expose our Observability Context Graph through APIs.

Domain APIs turn telemetry into observability insights

Domain operations: For common observability questions, such as “Is checkout healthy? What does it depend on?”, the agent shouldn't have to write a query at all. We encoded that expertise into the API: one call returns service health or the invocation graph, instead of the agent hand-writing a query over raw telemetry. For checkout, that gives the agent health signals and service dependencies to investigate before querying raw telemetry.

Composability: A service-health answer may show that checkout is slow without explaining why. Our responses carry the relevant dataset, metric and time window forward, so the agent can query the telemetry behind a finding without repeating discovery. That context also supports engineers’ follow-up questions in conversations with Observe’s AI Site Reliability Engineer.

A consistent contract across APIs

A shared contract underpins every API, so an agent can learn one consistent interaction model and apply it across the platform:

• Search, by meaning or by pattern, is the same across every search API.
• Filtering with Common Expression Language (CEL), so the agent can request only critical alerts for services whose names start with “checkout.”
• Structured errors, so a failed call says what went wrong and how to fix it and the agent recovers instead of stalling.
• Time ranges, to scope to the query window that matters, like the last hour.

The same APIs power all our agentic surfaces: the MCP server, the CLI and our own AI SRE. Because the surfaces share the same APIs, improvements can benefit all three.

Here is an example of the payments prompt answered two ways: with raw-query access and keyword search alone, and with grounding and domain APIs added.

With raw-query access and keyword search alone, the agent identifies slow checkout; with grounding and domain APIs, it goes to the orders database, where a targeted query reveals an exhausted connection pool.

Measuring what each capability contributed

Evals are central to our development process and our test suite continues to grow with the product. To measure the impact of these API changes, we evaluated Claude Code using our MCP server on 56 curated observability questions, each with a rubric for its known-good answer. The prompts ranged from finding data and checking health to investigating failures, covering questions such as “What datasets have data for the payment service?”, “Which frontend route has the highest latency?” and “Why is the ad service failing?” The agent ran claude-opus-4-7 at high reasoning effort, with three runs per prompt, or 168 trials per configuration.

A separate LLM judge, Cortex gpt-5.4, evaluated each answer against the corresponding reference answer on a five-point scale. We treated scores of four or five, indicating that the answer was substantially correct, as passes. For each configuration, we report the share of all runs that passed, alongside latency, token use and tool calls. We also run this evaluation suite in continuous integration (CI), where it must meet our quality threshold before we can merge an API change.

What grounding and domain APIs added

We evaluated a series of configurations, starting with a baseline of raw queries and keyword search and building up to the full stack. The figure below shows success, token use and latency as grounding and domain APIs are added cumulatively to the baseline, labeled “Floor.” The ablation chart later shows the effect of removing individual capabilities from the full stack. The two figures present different analyses of the same evaluation results.

Adding grounding APIs for entity resolution and a path to the relevant data raised the agent’s investigation pass rate from 7.8% to 26.2%. These APIs helped the agent find relevant telemetry more reliably, but it still had to construct queries to interpret the results. Adding domain APIs on top of that grounding, with domain operations and investigation context, raised the pass rate further to 69.0%.

Domain operations made the largest difference

We then removed each API capability, one at a time, while keeping the others available. Each resulting pass rate is compared with the full configuration’s 69.0%. The drop shows how much the agent relied on that capability in this evaluation. These effects can overlap, so the drops should not be added together.

Removing domain operations produced the largest drop, returning the agent to the grounding configuration shown above. It retained entity resolution, connections to relevant data and raw-query tools, but lost the APIs for listing environments, assessing service health and mapping dependencies. The agent had to reconstruct those concepts from raw spans and metrics instead. Finding the relevant telemetry was not enough: without domain operations the agent could not reliably turn that data into answers.

The two grounding capabilities also addressed different problems. Removing the path to data left entity lookup available, but the agent could no longer limit dataset and metric searches to sources containing data for that entity. The agent’s success rate fell to 51.2%, total token use rose by about 30% and all nine dataset-search prompts fell below the passing threshold. Removing entity resolution instead lowered the success rate to 54.8%. In other words, the agent needs to solve two separate problems: determining what an engineer means and finding where that entity’s data lives.

Removing composability had a smaller effect on the agent’s investigation pass rate: it lowered success from 69.0% to 65.9%, increased token use by about 6% and added one tool call per investigation. In this benchmark, preserving context between investigation steps produced a modest improvement but not the substantial gain we saw when we added domain operations.

The full API stack improved accuracy while reducing agent effort

The full stack’s higher agent investigation pass rate also required less work. Compared with the raw-query-and-keyword-search baseline, mean latency fell by 58%, token use by roughly 58% and tool calls per investigation fell from 15 to six.

The baseline used more tokens while producing correct answers for far fewer questions. In the runs we inspected, the agent spent tool calls searching for relevant data and constructing queries, but often did not reach a correct answer. Grounding APIs reduced the effort required to locate the relevant telemetry, while domain APIs reduced the effort required to derive service health and dependency insights from that telemetry.

In this benchmark, removing domain operations caused the largest drop in the agent’s investigation pass rate. That supports encoding common observability calculations and relationships in APIs rather than requiring the agent to reconstruct them from raw telemetry.

Composability remains an area for further evaluation. We have not yet measured its value across follow-up questions from an engineer, where preserving investigation context between steps may matter more. At 69% success, there is still substantial room to improve the investigations.

Build APIs around the questions your users ask

At Observe, exposing telemetry was only the starting point. Grounding APIs helped agents find relevant data, while domain APIs encoded the expertise needed to assess service health and understand dependencies. Together, they improved the agent’s investigation pass rate while reducing latency and token use. They also gave the agent enough context to decide where to investigate next and which underlying telemetry to query.

If you’re building APIs for agents, start with the questions your users ask and identify where the agent struggles. Use those failures to guide what you build, then measure whether each change improves answer quality, speed and cost. We developed our design through iterative testing, and our evaluations showed which capabilities made the biggest difference. The same approach can help you improve your own product.

Learn more and get started with Observe by Snowflake.