AI Agents: A Guide to Agentic AI Architecture and Governance

Enterprise AI is moving past the era of the isolated prompt. Even just a year ago, many AI initiatives focused on giving employees access to chatbots: upload a file, request a summary, ask a question. Now, organizations want agentic AI systems that can work across data, tools, codebases and business workflows with enough context and control to complete multi-step tasks.

However, an AI agent can’t be treated like a chatbot with more permissions. It needs access to context, but not unrestricted access to every system. It needs access to tools, but not the freedom to use just any tool in any way. It needs enough autonomy to work through a task, but with enough control for the organization to see what it did, which data it used and why it took a given action.

By 2028, Gartner predicts that 33% of enterprise software applications will include agentic AI, a dramatic leap from less than 1% in 2024.¹ As agents become part of the enterprise software layer, the practical challenge is how to connect agents to data, tools and workflows through a governance layer that can work across vendors, preserve permissions and keep actions auditable.

An AI agent is a software system that uses AI to pursue a goal, make decisions and take action through applications or other systems. It works through a task, first interpreting the request and gathering context, then deciding what to do next, calling the right tool(s), checking the results and continuing until it reaches an outcome or hands the task to a person or another system.

Most current enterprise agents use large language models (LLMs) as their reasoning layer. But an AI agent is much more than its model. It includes the orchestration logic around the model: the instructions that shape its behavior, the planning loop that determines the next step, the memory or task state it can reference, and the connectors that let it call approved tools and data sources.

Agentic AI vs. generative AI vs. ML

While traditional machine learning (ML) can sit inside automated workflows, and generative AI can be connected to tools, neither is agentic on its own. What makes an AI system agentic is the goal-directed loop around the model: context, connectivity, tool use, state and enough autonomy to work through a multi-step task.

• Traditional ML predicts or classifies: Machine learning systems are usually trained to recognize patterns in data and return a prediction, classification, score or recommendation. The model output might inform a workflow, but the workflow itself is defined outside the model.
• Generative AI creates new content: Generative AI models produce text, images, code, audio or other outputs from a prompt or input. A user asks a question, requests a summary or provides instructions, and the model generates a response. The interaction may be conversational, but the system is typically limited to producing an output for the user.
• Agentic AI acts toward a goal: Agentic AI uses models inside a broader system that can retrieve context, choose next steps, call tools, observe results and continue until it reaches an outcome or hands the task off. The model may provide reasoning or generation, but the agentic system gives that model access to context, a path for taking action and a process for deciding what to do next.

AI agents vs. data agents

A data agent is a specialized AI agent focused on data work. It connects natural language, business context and data systems so people can ask questions, generate queries, inspect metadata, check quality, summarize data sets or investigate anomalies without manually moving between tools.

A data agent needs governed access to data. It needs to understand metric definitions, table freshness, column-level lineage, ownership, access policies and approved query paths. If a user asks for customer churn by region, for example, the agent should know which churn definition is approved, which customer table is current, whether the user can see row-level details and whether the answer should be aggregated.

“Adding context is the battlefield right now,” explains William Allen, Snowflake’s Head of Product for CoWork & Agents. “The question is how you give the agent context at every turn, without overwhelming it and without leading it down random paths.”

Data agents also need to work across different kinds of data. A revenue investigation might involve a SQL table, a contract PDF, a support transcript and a dashboard definition. An AI data agent brings those objects together through a governed interface, so the model can reason over the business question without bypassing access controls.

This is why enterprise AI agent adoption involves decisions around data architecture: the agent’s answer depends on the quality, freshness, semantics and permissions of the underlying data.

The value of an AI agent over a standalone model comes from its ability to plan steps, retain or retrieve context, and use tools to act on information beyond the model itself.

• Reduce manual orchestration: A user doesn’t have to retrieve context, paste it into a prompt, ask for a summary, copy the answer into another system, re-prompt when something is missing and manually verify every step. The agent handles more of the retrieve, reason, act and verify loop inside the workflow.
• Shorten the path from question to data-backed answer: Someone asking why pipeline costs increased shouldn’t have to know which table stores usage history, which view has the approved cost allocation logic or which dashboard owner changed the calculation. A governed data agent can translate the question into data tasks, retrieve the right context and surface the answer in business language.
• Make repetitive decision points more consistent: Ticket triage, data quality checks, log review and account-risk summaries all depend on applying the same reasoning process across large volumes of work. Human reviewers still matter, especially for exceptions, but the agent handles the repeatable parts with a consistent path.
• Enable adaptive automation: A rule-based workflow stops when the input changes unexpectedly. An agent can inspect the new condition, revise the plan and choose another approved path. That adaptability is the reason agents are useful — and the reason governance has to sit close to the execution layer.

The best early use cases for AI agents aren’t necessarily the flashiest ones. They’re usually workflows where the work is repetitive, context-heavy and spread across multiple systems.

Customer service

In customer service and support, agents triage tickets, retrieve account context, classify issues, draft responses and route exceptions to a human reviewer. The agent doesn’t replace the escalation path; it makes the path more selective by handling the repetitive context gathering and first-pass resolution work.

Software development

In software development, coding agents plan, write, test and revise across a codebase. The difference from autocomplete is scope. A coding agent needs repository context, issue context, dependency information, test results and project-specific instructions. It also needs guardrails around what code it can change, which commands it can run and what requires human approval.

Data operations

In data operations, agents monitor pipelines, inspect data quality issues, trace lineage and identify affected downstream assets. A failed freshness check becomes more useful when the agent can show which source table changed, which transformation depends on it, which dashboard is affected and who owns the workflow.

Sales operations

In sales and revenue operations, agents synthesize customer relationship management (CRM) data, call transcripts, account history and product usage to flag at-risk accounts or draft outreach. The useful output isn’t a generic account summary. It’s a recommendation grounded in current customer context, with the source signals visible.

IT and security operations

In IT and security operations, agents review logs, correlate alerts, retrieve asset context and help triage incidents. These workflows need tight governance because the agent may touch sensitive systems, security telemetry and high-impact actions. The safest early deployments often keep humans in the approval loop while using agents to reduce context-gathering time.

Watch leading AI researcher Andrew Ng explore the rise of AI agents and agentic reasoning:

Most AI agents follow a loop: sense, reason, plan, coordinate, act and improve. The labels vary across frameworks, but the sequence captures the same operating pattern.

Sense

The agent first senses the environment. That might mean reading a user prompt, retrieving a document, inspecting a table, receiving an event from a workflow system or observing the output of a tool call. The input is rarely just a sentence; in enterprise workflows, it often includes metadata, permissions, source context, business rules and prior task state.

Reason

The agent then reasons over the goal and available context. It decides what information matters, what is missing and what constraints apply. For a data agent, that might include the meaning of a metric, the freshness of a table, the user’s access privileges and whether the requested answer requires joining sensitive data.

Plan

Planning turns that reasoning into a path. The agent decides which steps to take, which tools to call and what order to follow. In a fixed workflow, the path may be predefined. In a more autonomous workflow, the agent chooses the next step based on what it has already observed.

Coordinate

Coordination becomes important when multiple tools, agents or approval steps are involved. One agent might retrieve relevant documents while another checks policy constraints. A supervisor agent might delegate subtasks, compare results and decide whether a human reviewer needs to approve the next action.

Act

The agent then acts. It calls a function, runs a query, updates a ticket, sends a message, drafts code, triggers a workflow or returns an answer. In governed environments, the action should be logged with enough detail to reconstruct what happened: the request, the data accessed, the tool used, the output returned and the decision path that led there.

Improve

Finally, the agent learns in the operational sense: it incorporates feedback, updates task state, stores approved memory or adjusts future steps. That learning may not mean retraining the foundation model. More often, it means improving the system around the model through memory updates, evaluation results, prompt changes, policy tuning or better tool routing.

AI agents can be designed to reason and act in different ways. Some workflows ask the model to think through a problem before responding. Others let the agent interact with tools, observe the results and revise its next step. The right pattern depends on how predictable the task is, how much feedback the environment provides and how much control the organization needs over the workflow.

• Step-by-step reasoning: Early agent design borrowed from prompting techniques developed for LLMs. One precursor idea was step-by-step reasoning, where the model is prompted to break a problem into intermediate steps before producing an answer. In that setup, the reasoning happens before the final response or action. The model doesn’t revise its path against external feedback unless the workflow gives it another turn.
• ReAct: ReAct (short for reasoning and action) changes the sequence. Instead of reasoning once upfront, the agent alternates between reasoning, acting and observing. It might decide to query a table, inspect the result, realize the metric definition is ambiguous, retrieve metadata, then revise its next step. The departure from chain-of-thought isn’t simply “more reasoning,” but reasoning that changes after the agent interacts with a tool or environment.
• Reflection: Reflection adds another check. The agent reviews its own answer, plan or tool result before finalizing the output or continuing the workflow. A coding agent might generate a test, run it, inspect the failure and critique the code it wrote. A data agent might produce a SQL query, compare the result with the user’s request and identify that the query filtered on booking date when the metric definition requires transaction date.
• Planning-based and reactive: Planning-based and reactive agents draw another dividing line. A planning-based agent builds a multi-step plan before acting, often with checkpoints and dependencies. A reactive agent chooses one step at a time as conditions change. The planning module defined in the architecture determines how much structure exists before the agent begins and how much freedom it has to revise the path.
• Prompt chaining: Prompt chaining decomposes a task into a sequence of smaller prompts. One prompt extracts entities, the next retrieves context, the next summarizes the results and the next formats the final answer. Chaining is typically more fixed and sequential than ReAct. It’s useful when the workflow is known in advance and each step feeds the next.

Most production agents use a combination of these paradigms. A workflow might begin with a plan, use ReAct to gather missing information, apply reflection before a high-risk output and rely on structured tool invocation to call approved functions, APIs, workflow nodes or Model Context Protocol (MCP)-connected systems. The design question isn’t which paradigm is best, but how much flexibility, feedback and control the task requires.

AI agents are often described through five canonical types: simple reflex agents, model-based reflex agents, goal-based agents, utility-based agents and learning agents. These categories come from earlier AI research, but they still help explain the design choices behind modern agent systems.

• A simple reflex agent responds to the current state using predefined rules. If the condition is true, it takes the matching action. This design works for narrow tasks with stable conditions, but it doesn’t maintain a model of the world or reason over future consequences.
• A model-based reflex agent keeps an internal representation of the environment. Instead of reacting only to the current input, it uses stored state to decide what’s happening. For example, an operations agent might track the last known status of a pipeline, the most recent schema change and the owner attached to the affected table before deciding which alert to raise.
• A goal-based agent evaluates actions against a desired outcome. It’s not only matching conditions, but also choosing steps that move the system toward a goal. A customer support agent trying to resolve a billing dispute might gather account history, compare policy rules, draft a response and escalate only when the task exceeds its authority.
• A utility-based agent weighs different outcomes. It might compare speed, cost, confidence, risk and business value before selecting an action. In an enterprise setting, utility matters when the fastest answer isn’t necessarily the safest one, or when a more expensive reasoning path is justified only for high-risk decisions.
• A learning agent improves through feedback. That feedback might come from users, evaluators, monitoring systems or prior task outcomes. In production, the improvement often happens through updated prompts, memory, routing logic, policy rules and evaluation sets rather than continuous retraining of the underlying model.

Two advanced types of agents are especially relevant to enterprise workflows:

• Hierarchical agents split work across levels, with a supervisor agent delegating subtasks to specialist agents.
• Multi-agent systems coordinate several agents that may work sequentially, in parallel or under a shared orchestrator.

In practice, many enterprise deployments also classify agents by autonomy: fully autonomous agents perform tasks with limited human intervention, while semi-autonomous agents request approval before high-risk actions. Domain-specific agents narrow the task surface further, such as a data agent, coding agent, security agent or sales operations agent.

In addition to a foundation model, an AI agent also needs an orchestration layer, a planning module, memory architecture, tool integrations and a control plane that governs execution.

Foundation model

The foundation model provides flexible interpretation, reasoning and generation. Many current systems use LLMs, although agents don’t have to be limited to language models. The model supplies flexible reasoning, while the surrounding architecture determines what the agent is allowed to know, which tools it can use and what happens when the task changes.

Model choice needs to stay flexible. As enterprises build agents across different workflows, locking every use case to a single foundation model can limit performance, cost control and adaptability. “If you get locked into any one vendor, a few months down the road you could be in trouble,” says Allen. “Maybe another vendor comes out of nowhere with some amazing new innovation and you’d miss out. We think optionality on model routing is foundational.”

Orchestration layer

The orchestration layer manages the agent loop. It passes context to the model, routes tool calls, stores intermediate state, handles retries and determines whether the workflow should continue, stop, escalate or hand off to another agent.

Planning module

The planning module breaks a goal into smaller steps. For a request such as “find why revenue is down in the Northeast region,” the agent might plan to identify the relevant metric definition, query recent revenue by segment, compare it with prior periods, inspect pipeline freshness and check whether a source table changed. In a simpler agent, planning might happen one step at a time. In a more complex one, the agent builds a multi-step plan upfront and revises it as tool results come back.

Memory architecture

Memory gives the agent continuity. Short-term memory holds the immediate task state: the user’s request, tool outputs, constraints, retrieved documents and prior turns in the conversation. Long-term memory stores reusable knowledge, such as user preferences, previous decisions, organization-specific definitions or resolved issues. Memory architecture determines what gets stored, how it’s retrieved, how long it persists and which parts of memory are available to which agent.

Tool integration

Tool integration connects the agent to the systems where work happens. These tools might include databases, search indexes, code repositories, ticketing systems, CRM applications, workflow engines, document stores or internal APIs. Function calling gives the model a structured way to invoke those tools, while standards such as MCP are emerging to standardize how agents discover and use external capabilities.

Control plane

The control plane is the governed platform layer that connects data, models, tools and enterprise workflows, while enforcing permissions, routing actions, logging activity and managing how agents operate across systems. It also sets cost and latency limits and manages human approvals. Even when an agent operates on governed data inside a platform such as Snowflake, retrieved content, user prompts and tool outputs should still be treated as inputs to an agentic workflow, not as instructions the model can blindly follow.

For enterprise AI agents, this layer isn’t optional plumbing. It’s what turns a model-driven workflow into something an organization can govern with confidence.

Evaluation and optimization harness

An evaluation and optimization harness tests how an agent performs across real tasks, tracks where it fails and helps teams improve the system without relying on one-off manual tuning. Instead of judging the agent only by a final answer, the harness can evaluate the plan, retrieved context, tool calls, intermediate outputs, cost, latency and final result.

In more advanced implementations, the harness can also propose changes to the agent’s instructions, tools, semantic models, prompts or workflow configuration, then test those changes against regression and evaluation sets before they’re promoted. This turns agent improvement into a governed loop: evaluate, diagnose, modify, test and keep only changes that improve performance without breaking previously working behavior.

A multi-agent system uses more than one agent to complete a task. The value is specialization: one agent retrieves context, another reasons over policy, another writes code, another validates the result and an orchestrator coordinates the workflow.

That specialization is often the reason to use a multi-agent system in the first place. Instead of asking one broad agent to handle every function, teams can break the work into narrower agents with clearer responsibilities.

The trade-off is coordination overhead. Every handoff needs state, permissions, error handling and a clear record of what each agent did. In practice, multi-agent design is less about adding more agents and more about deciding where specialization improves the outcome enough to justify the additional complexity.

Sequential orchestration

Sequential orchestration works like a pipeline. One agent completes a step and passes its output to the next. This is the multi-agent equivalent of prompt chaining, and it works well when the task order is predictable.

A document-processing workflow, for example, might extract entities, classify the document, check compliance rules and draft a summary in a fixed sequence. Each step depends on the output of the step before it, so the workflow can be inspected as a chain of intermediate decisions.

Hierarchical orchestration

Hierarchical orchestration uses a supervisor agent. The supervisor receives the goal, delegates subtasks to specialist agents and assembles the final result.

Frameworks such as LangGraph support this kind of production workflow because graph-based state models map well to checkpoints, retries and recovery points. Role-based frameworks such as CrewAI are often used for prototypes where teams want to assign tasks to named agents quickly before hardening the workflow.

Parallel orchestration

Parallel orchestration sends the same problem to multiple agents at once, then merges or compares the outputs. This pattern is less common in production today because it increases cost, latency and reconciliation complexity.

It can still be useful in evaluation-heavy workflows where independent answers improve confidence. For example, multiple agents might review the same proposed answer, test different assumptions or check the result against separate sources before a final response is assembled.

Standards for tool and agent communication

MCP and emerging agent-to-agent communication protocols are part of this orchestration layer. MCP standardizes how agents and AI applications connect to tools and data sources. Agent-to-agent communication protocols focus on how agents exchange messages, state and task outputs with one another.

These standards help teams answer the question of whether and how the organization can govern which agent called which tool, with whose permissions, for what task and with what result.

Autonomy is a spectrum. An AI agent can be autonomous in one part of a workflow and tightly constrained in another. It might retrieve documents, inspect metadata and draft a recommendation on its own, then require human approval before it updates a system, sends a message or makes a change to a pipeline. The balance will also vary by use case.

The level of guardrailing a workflow needs depends on both the consequence of failure (what happens if the agent makes a mistake and how difficult it is to reverse) and the point at which failure is likely to occur. A broad approval rule may reduce risk, but it may also slow down low-risk steps the agent already handles well. A narrower guardrail, applied to the point where the agent tends to retrieve weak context, choose the wrong tool or overstate confidence, can improve reliability without turning the whole workflow into manual review.

Agent governance has to account for consequences, not just outputs. Once an AI system can retrieve data, call tools and act across workflows, failures are no longer limited to an inaccurate response. The agent might use unauthorized context, disclose sensitive information or take an action with downstream effects that are costly or impossible to reverse.

Accuracy

Accuracy is a primary concern. In finance, engineering, healthcare, security and other high-risk domains, the agent must retrieve the right context, use the right definition, apply the right policy and know when its confidence is too low to proceed. Evaluation has to reflect the task. For example, a generic LLM benchmark won’t show whether an agent correctly applies an organization’s revenue-recognition rules or escalates the right security alert.

Governance

Governed data access is also a high priority. Agents often need to work across structured data, documents, audio, logs, images, tickets and code. Those sources sit behind different access models, and the agent’s ability to act shouldn’t exceed the user’s authority. A governed agent needs identity propagation, policy enforcement, query controls and audit trails that show what data was used.

Allen argues that organizations shouldn’t wait until every governance question feels settled before they begin, however. “You need to set the policies, steer it and govern it. But don’t be afraid to move faster — just do it with a trusted partner who understands the governance side.”

Trust and security

Trust and security become more complex once agents start using tools. Prompt injection, excessive permissions, unsafe tool calls, data leakage and unauthorized write actions all become practical risks. The security framework has to account for both the model’s behavior and the systems it can reach.

Cost and latency

Cost and latency are often the first operational roadblock. A single prompt-response interaction might require one model call. A multi-step agentic workflow might include planning, retrieval, tool calls, reflection, another retrieval step and a final response. Each step adds cost and time. Reflection, ReAct loops and multi-agent orchestration are useful when the task justifies them, but there are deliberate cost and accuracy tradeoffs.

Before an agent can act in production, teams need to define the data it can access, the tools it can use, the actions it can take, the approvals it needs and the evidence required to audit what happened. The goal isn’t simply to connect a model to more systems, but to create a governed execution path from user request to agent action.

Start with data access

Before an agent touches production data, define what it’s allowed to read, what it is allowed to write and which interface it must use. In a governed architecture, the agent shouldn’t receive direct, unrestricted table access. It should operate through approved query layers, APIs, semantic models or retrieval systems that enforce identity, policy and context.

Define governance at the control-plane level

The control plane should enforce permissions, route tool calls, log actions, manage approvals and stop workflows that exceed defined limits. If an agent can update a ticket, send a message, modify a pipeline or run a query against sensitive data, the approval and audit path should be clear before production use begins.

Evaluation comes before deployment

The agent should be tested against task-specific benchmarks that reflect the work it will perform. For a data agent, that might include questions with ambiguous metric definitions, stale tables, conflicting documents, access-restricted fields and known edge cases. Accuracy thresholds should match the risk of the workflow. An internal knowledge assistant may tolerate more uncertainty than an engineering agent that changes infrastructure or a finance agent that informs reporting.

Monitoring continues after deployment

Teams need traces that show each step in the agent’s workflow: retrieved context, tool calls, intermediate outputs, approvals, latency, cost and final actions. Drift detection, action audit trails and human-in-the-loop escalation paths help teams understand when the agent’s behavior changes, when its inputs degrade or when a low-confidence decision needs review.

In production agent systems, teams also need a harness that can show which questions, tools, retrieval paths or workflow steps are causing failures, then test proposed fixes against both targeted cases and regression sets. Snowflake’s CoCoEvolve is one example: the system wraps around an AI artifact, proposes changes, evaluates whether they help, keeps successful changes and discards unsuccessful ones.

The future of AI agents will likely be shaped less by isolated model improvements and more by the systems that supply context, govern action and coordinate work across tools. As agents move from chat windows into operational workflows, enterprises will need architectures that expose the right data at the right time, preserve user permissions and record what happened at each step.

For enterprise teams, that architecture often depends on bringing AI closer to governed data rather than copying data into a separate AI environment. “You need to be able to bring AI to the data, and not move that data out because of all the governance,” Allen says. “Keep your data where it is, and then leverage AI to build agents that have that context.”

That requirement is also why protocols such as MCP are becoming important. Agents need a consistent way to discover and use tools, data sources and applications without every connection relying on a custom integration. Anthropic introduced MCP in November 2024 as an open standard for connecting AI applications to external systems, then donated it to the Agentic AI Foundation under the Linux Foundation in December 2025. Since then, MCP has moved beyond local tools and is being used in production by a growing range of organizations.

MCP’s growth points to a more connected agentic ecosystem, but connection alone doesn’t solve enterprise challenges. The next stage will also depend on governed interoperability: agents that can discover tools, use them with the right identity, respect policy, preserve context and hand off work across systems without losing the audit trail.