Beyond the Prompt: Dangerous Activity Detection for AI Agent Trajectories

Enterprises are deploying autonomous agents to carry out complex, multi-step tasks across applications, databases and operational systems. This ability to act across systems can automate substantial work, but it also creates opportunities for unsafe or unauthorized behavior. 

Traditional guardrails that monitor single actions or individual prompts are insufficient at detecting such risks; a new class of guardrails that analyzes multi-step agent trajectories is needed to address these specific risks. We introduce Dangerous Agent Activity Detection (DAAD) as a two-stage engine to analyze agent trajectories to detect dangerous activities stemming from agent overreach or user-directed misuse across long horizons. We have deployed DAAD internally and will have it available shortly as a feature in Cortex AI Gateway.

We cover the following four key areas:

• Framework to detect dangerous agent activity: Outlines how we categorize dangerous activities into agent overreach and user-directed misuse across threat domains (such as cyber, bio and enterprise policy deviations).

• Core architectural principles: Specifies the key design goals and the architecture of the two-stage engine.

• Benchmark evaluations and key takeaways: Presents the empirical results from internal enterprise cohort testing and demonstrates how adding context improves detection accuracy.

• Integration with OpenAI Private Safety Processing (PSP) safety signals: Explains how DAAD can ingest OpenAI PSP signals to strengthen the safety pipelines.

Framework to identify dangerous agent activity

Dangerous agent activity can arise when an agent exceeds a legitimate objective or when a user deliberately pursues a harmful one. Either source can lead to harm across multiple domains. Our framework therefore analyzes risk through two complementary lenses: what drives the unsafe behavior and the threat domain in which it occurs.

The two primary sources of risk are:

• Agent overreach: Autonomous agents misinterpret or exceed legitimate objectives, taking unsafe or unauthorized actions.

• User-directed misuse: Malicious users, including rogue insiders, deliberately steer agents toward dangerous activities, using multi-step execution paths to obscure their intent.

Across both sources of risk, the framework covers several threat domains, including cyber threats, biological threats and enterprise policy deviations:

Cyber threats: Covers agent trajectories that enable cyber harm, including unauthorized exploitation or activity that exceeds approved testing boundaries. Automated pentesting and vulnerability research require context to distinguish legitimate work from dangerous activity.

• Agent overreach example: While investigating a service outage, an administrative agent moves beyond approved diagnostics and attempts to exploit another system without authorization.

• User-directed misuse example: A malicious user directs an agent to compromise a third-party system, distributing the objective across seemingly benign requests.

Biological threats: Covers agent trajectories that provide actionable assistance for biological harm, including biological weapons development or evasion of biological safety screening.

• Agent overreach example: Asked to summarize published biological research, an agent goes beyond the informational task and generates actionable guidance that could enable biological harm.

• User-directed misuse example: A malicious user steers an agent through a series of apparently benign requests to obtain assistance for evading biological safety screening.

Enterprise policy deviations: Covers agent trajectories that violate organizational controls, including unauthorized disclosure of sensitive data, bypassing required approvals or using tools outside permitted boundaries.

• Agent overreach example: While resolving a support request, an agent retrieves restricted financial data and includes it in a response to a user who is not authorized to receive it.

• User-directed misuse example: A malicious insider directs an agent to bypass required approvals and transfer sensitive files to an unauthorized destination.

Core architectural principles

We designed DAAD around three core principles:

• Detection effectiveness: Target high true positive rates (TPRs) and low false positive rates (FPRs) to detect dangerous activity while limiting false alerts and alert fatigue.

• Actionable evidence and transparency: Provide security administrators with clear explanations and supporting evidence from agent trajectories to help them investigate detections and determine appropriate remediation.

• Cost-efficient analysis: Use lower-cost models for initial screening and selectively escalate trajectories requiring deeper analysis to more capable models, optimizing inference costs without compromising detection goals.

To put these principles into practice, DAAD uses a two-stage engine that combines fast initial screening with selective, context-enriched analysis:

• Stage 1 — Fast screening: A lightweight model screens agent trajectories, filtering out those assessed as benign and flagging potentially dangerous trajectories for further review.

• Stage 2 — Context-enriched assessment: A deeper reasoning model evaluates the flagged trajectories using additional context to determine whether they represent dangerous activity.

Trajectories undergo redaction to remove sensitive information, such as access keys and user identities, while preserving patterns relevant to safety assessment. Both stages return schema-constrained outputs containing a decision, supporting evidence, a concise rationale and a confidence score.

Benchmark evaluations and key takeaways

Over a three-week internal trial, we evaluated DAAD on approximately 6,000 sessions from a red-teaming and penetration-testing cohort. DAAD initially identified potentially risky activity associated with every participating user but did not issue alerts after accounting for the authorized scope of their testing.

In a separate trial to evaluate the performance against a negative baseline, we analyzed approximately 12,000 agent trajectories from engineering staff serving as the control group. On a per-user basis, DAAD flagged three individuals. Further analysis confirmed one instance of rogue-agent activity, which terminated the process across all users of a cloud workspace; the remaining two flagged cases involved users performing software development tasks that resembled network enumeration.

As an additional evaluation step, three reasoning models independently reviewed every trace, including those DAAD assessed as benign. Cases involving disagreement among the models were referred for human review.

A key takeaway was the importance of context in distinguishing legitimate, authorized activity from dangerous behavior. Contextual signals such as authorization scopes and organizational roles enabled DAAD to evaluate whether a given action aligned with the user's permissions and the agent's designated task, and adding them can further reduce alert noise. 

We have internally deployed DAAD in production and process approximately 18,000 internal agent trajectories daily across multiple agentic surfaces deployed in Snowflake®, with a flag rate of less than 0.2%. We aim to make it available as a feature in Cortex AI Gateway shortly. 

Integration with OpenAI PSP safety signals

OpenAI recently introduced PSP, which enables offline, automated safety reviews without requiring OpenAI to retain customer content. It is built around three core principles:

• Customers controlling their content and its storage

• No human review by third parties

• Safety reviews conducted in partnership with customers, who can investigate alerts and choose what information to share

PSP provides an opportunity to ingest safety signals as additional context into DAAD. This allows for evaluation of safety signals alongside trajectory and enterprise context, helping security teams prioritize high-risk dangerous activity and increasing confidence in detections.

We collaborated closely with OpenAI as a technical design partner in the development of PSP. We strongly believe that ingesting PSP safety signals will make DAAD more effective for enterprises at scale. This establishes an early blueprint for how OpenAI and its partners, such as Snowflake, can collaborate to identify and respond to misuse.

Cortex AI Gateway rollout and call to action

We are excited to share that DAAD is scheduled to be available in preview natively within the Cortex AI Gateway soon. 

Forward-looking statements

This article contains forward-looking statements, including about our future product offerings, and are not commitments to deliver any product offerings. Actual results and offerings may differ and are subject to known and unknown risk and uncertainties. See our latest 10-Q for more information.