Reinforcement Learning: How Agents Learn from Rewards

Reinforcement learning (RL) once was limited primarily to robotics, game-playing systems and control systems. Now, its growing role in artificial intelligence, particularly LLM post-training and tool-using AI agents, is pulling it into the center of ML model development.

The same core requirements still apply: useful feedback, an environment that produces experience, records of the resulting interactions and infrastructure that converts those interactions into policy updates. In LLM and AI agent applications, however, those requirements reach across language-model rollouts, tool calls, database queries and longer, less predictable workflows.

For teams applying reinforcement learning in these settings, implementation is as much a systems engineering undertaking as an algorithmic one. Understanding how agents learn from rewards is the starting point, but reliable training also requires coordination across data, evaluation and training infrastructure.

Reinforcement learning is a machine learning approach in which a decision-making system, called an agent, learns by taking actions in an environment and receiving rewards based on the results.

Over repeated interactions, the agent adjusts its policy — the method it uses to choose actions — to increase the cumulative reward it expects to earn.

Unlike supervised learning, reinforcement learning doesn’t usually provide a correct answer for every training example. Its feedback is evaluative: the environment indicates how useful an action or outcome was, and the agent must infer which behavior contributed to that result.

As Snowflake Machine Learning Engineer Thong Nguyen explains it, “Reinforcement learning lets you define a quality signal that captures what you care about, then systematically optimizes a small model against it.” The difficulty lies in making sure that signal actually represents the intended result.

A reinforcement learning system includes five core components:

• Agent: The system choosing actions
• Environment: The setting in which the agent acts
• State or observation: The information available when the agent makes a decision
• Action: A choice available to the agent
• Reward: The feedback returned after an action or outcome

(In this context, agent is the general RL term for the decision-making system. It doesn’t necessarily refer to a tool-using AI agent. A robot, recommendation policy, game-playing system or language model can all serve as the agent.)

Consider an inventory system deciding how much stock to reorder. The agent receives information about current inventory, recent demand, supplier lead times and outstanding orders, then selects a reorder quantity. Subsequent sales, storage costs and stockouts provide the consequences it uses to learn.

The reward reflects whichever parts of the operating objective the system has been designed to measure. Because one order changes the inventory available for the next decision, the policy must account for consequences that unfold across several weeks.

For example, minimizing this week’s storage cost may produce a stockout later, while carrying additional inventory now might support a higher cumulative return across the full planning period. The reward function plays a central role in reinforcement learning design.

In this video, Sukanya Joshi, AI/ML Solutions Consultant at Snowflake, guides you through an end-to-end MLOps workflow — from model development to deployment:

A reinforcement learning system moves through a recurring loop:

1. The agent observes the current state or available information.
2. Its policy selects an action.
3. The environment processes the action.
4. The environment returns a reward and an updated state.
5. The learning algorithm adjusts the policy or its estimates of value.

Represent sequential decisions

For an RL policy to choose the right action, the current state has to contain all the information that affects what happens next. Consider the inventory example: If the state shows current stock but omits a shipment due tomorrow, the system will likely choose an inappropriate reorder quantity.

A Markov decision process formalizes this relationship among states, actions, transitions and rewards. Under the Markov assumption, the current state contains enough information to estimate the consequences of the next action.

Real systems often fall short of that ideal. A customer history may be incomplete, for example, or an external application may not be updated. A tool-using AI agent might see a request and its available tools, but be unaware of events outside its context or permissions.

In those cases, the system is partially observable: the agent acts on the information available to it rather than a complete representation of the underlying state.

Learn from delayed consequences

Reinforcement learning often evaluates an action through what happens several steps later. The sequence of observations, actions and rewards produced during an interaction is called a trajectory. When that interaction has a defined beginning and end, it’s typically described as an episode.

For an AI agent preparing a report, one trajectory might include choosing a data source, constructing a query, inspecting the result, revising the query and producing a final answer. The environment may withhold the most meaningful reward until the workflow ends.

That delay creates a credit-assignment problem. If an agent receives a positive reward after a 20-step workflow, the training process needs to determine which earlier actions contributed positively to the result. Some may have moved the task forward, while others were unnecessary. Some may have even introduced errors that it needed to correct later.

A value function estimates the cumulative reward expected from a state or state-action pair, helping the policy evaluate choices whose payoff arrives later. The discount factor controls how much influence future rewards have on the current decision.

In the inventory example, a policy focused heavily on immediate returns may avoid carrying additional stock. A policy trained across a longer horizon may accept that short-term cost to reduce the likelihood of a larger stockout later.

Balance exploration and exploitation

An agent rarely knows the best action at the start of training. It must choose between moving ahead with an action that appears valuable or exploring alternatives whose outcomes remain uncertain.

Too little exploration may leave the policy with a mediocre strategy because it never gathered evidence about better options. But too much can expose the environment to risk and add expense.

Methods such as epsilon-greedy selection, upper confidence bound and Thompson sampling handle this trade-off differently. In production-oriented applications, however, the practical constraint is often how safely and reversibly the environment supports experimentation.

Teams typically begin with logged data, sandboxes or simulations. Before deployment, they may narrow the action space, test against held-out scenarios and introduce approval or rollback controls. Even then, rare events and differences between test and live systems may expose the policy to unfamiliar states, so production exploration tends to remain limited and closely monitored.

Compare the main algorithm families

Reinforcement learning algorithms differ in what they learn and how they derive a policy. Value-based methods estimate which actions are likely to produce strong cumulative returns. Policy-based methods optimize action selection directly, which is useful when actions are continuous or the policy should preserve some variation. Actor-critic methods combine direct policy optimization with a learned value estimator that helps stabilize training.

These families continue to shape newer RL methods, although individual algorithms don’t always fit neatly into one box. Two methods are especially relevant to language-model post-training:

• Proximal policy optimization (PPO) is an actor-critic method that uses a learned critic to help estimate the value of generated responses. It became widely associated with reinforcement learning from human feedback.
• Group relative policy optimization (GRPO) is a PPO-style policy-optimization method that removes the separate learned critic. Instead, it calculates a relative baseline from groups of responses generated for the same input, reducing some of the memory and compute associated with critic training.

During language-model post-training, a model produces an output, receives a reward or score, and adjusts its policy so that higher-scoring outputs become more probable. Prominent approaches differ in how that training signal is produced.

Reinforcement learning from human feedback

Reinforcement learning from human feedback (RLHF) uses human preferences or evaluations to construct the reward signal used during policy optimization. The workflow typically begins with supervised fine-tuning. Human evaluators then compare candidate responses, and those rankings train a reward model that predicts which outputs people are likely to prefer. The language-model policy is subsequently optimized against that learned score, classically using PPO.

The reward model allows the system to evaluate more outputs than people could review directly. Its predictions, however, reflect the examples and instructions used to train it. If evaluators consistently favor confident or detailed answers, the model may treat those features as signs of quality even when they don’t indicate correctness.

Reinforcement learning from AI feedback

Reinforcement learning from AI feedback (RLAIF) uses an AI evaluator for some or all of the scoring that guides policy updates. The evaluator may compare responses, apply a rubric or judge whether an output satisfies task-specific criteria. Deterministic checks often operate alongside it. A structured extraction task, for example, might verify the required format before asking an AI judge to assess whether the extracted content is accurate and complete.

The resulting policy learns from the evaluator’s judgments, including any blind spots, stylistic preferences or inconsistencies it carries. A different judge, model version or rubric may produce a different training signal from the same output.

Reinforcement learning with verifiable rewards

Reinforcement learning with verifiable rewards (RLVR) uses a verifier to check whether an output or action produced the required result. The verifier might run unit tests against generated code, compare a mathematical answer with a known result, execute SQL, submit code to a compiler or confirm that a workflow reached an expected final state.

A verifier also leaves an inspectable record — teams can trace the reward back to a test result, query output or state condition. That transparency doesn’t guarantee completeness, however. A test suite may miss an edge case, or a workflow may satisfy its final-state check after bypassing an unstated requirement.

A reinforcement learning system learns from the objective it’s been given, which means reward design has to capture enough of the goal to guide useful behavior. When the measured reward leaves out something important, the policy may discover behavior that performs well against the metric but produces a poor result in practice.

That gap is known as reward misspecification. If the policy learns to exploit it, the result is reward hacking or, more broadly, specification gaming: behavior that satisfies the formal objective without fulfilling its intent.

The risk appears in various forms across RL applications. A language model whose evaluator favors confident responses may learn to state uncertain claims more assertively, for example, or an AI agent judged only on final state could complete a requested change while skipping a required validation step.

As policies grow more capable, reward design has to account for a wider range of possible strategies, such as identifying weaknesses in tests, learned judges or scoring rules. In RLHF, for example, a reward model that consistently associates agreement with helpfulness may reinforce sycophantic responses.

Practitioners also have to decide when to provide feedback. A reward issued only at the end of a long workflow leaves the training process with little information about which earlier actions helped or hurt. Adding intermediate rewards — a technique known as reward shaping — gives the policy more guidance, perhaps by rewarding a correct tool choice or a required workflow step.

Additional guidance introduces its own trade-off, however. Every intermediate reward adds another assumption about what productive behavior looks like, and each assumption creates another surface the policy could optimize too literally. For this reason, many systems combine outcome rewards, which evaluate the final result, with process rewards, which evaluate selected intermediate actions or reasoning steps. The goal is to give the policy enough information to learn efficiently without turning the reward into a collection of proxies that are easier to satisfy than the task itself.

Agentic RL applies reinforcement learning to tool-using AI agents across multistep interactions. Because one action changes the conditions for the next, training requires an environment that responds to tool calls, preserves state and records what happened throughout the workflow.

Consider an AI agent updating a business record. Success depends on the agent identifying the correct record, using the appropriate tool, preserving required fields and leaving the underlying system in the intended state. A plausible summary can’t establish that those steps happened correctly.

For this reason, the training data has to capture the full trajectory: what the agent observed, which actions it selected, the arguments passed to tools, the results those tools returned, how the environment changed and the reward associated with the outcome. That record gives the training process evidence about both the final result and the decisions that produced it.

Generating enough trajectories for the agent to learn from is difficult in live systems. Production environments may contain sensitive data, serve active users or enable actions that are costly or difficult to reverse. Tool calls may also depend on external services with latency, rate limits or inconsistent availability. For training, teams therefore often rely on controlled or generated environments that reproduce the relevant tools, state transitions and task conditions without exposing production systems to unrestricted exploration.

Those environments need enough variation to prevent the policy from simply learning one schema, tool inventory or workflow pattern. Different data structures, task conditions and tool combinations expose the agent to a broader range of trajectories and provide a better test of whether the learned behavior transfers.

Scale alone, however, doesn’t make an environment useful. The tools have to work, the tasks have to be solvable and the success criteria have to reflect the intended outcome. A generated environment with incomplete tests may reinforce the wrong behavior just as easily as a poorly designed reward.

For this reason, the verifier is part of the training environment itself. It may inspect query results, compare state before and after an action, confirm that required steps occurred or evaluate the completed workflow. Its job is to provide a trustworthy account of what the agent accomplished, so the reward reflects the underlying task rather than only the response the agent produced at the end.

Snowflake AI Research took this approach with DARE-Bench, a benchmark of 6,300 executable data science tasks built around deterministic, programmatic ground truth. Agents execute code in a sandbox and their outputs are checked against reproducible task-specific criteria, providing a verifiable reward signal for training and evaluation.

The first question is whether the problem requires learning from the consequences of actions over time. If each prediction is largely independent and the correct output can be labeled directly, supervised learning is usually a simpler fit. A fraud classifier, for example, can learn from labeled transactions without modeling how one prediction changes the conditions for the next.

Some decision problems require feedback from actions without requiring the system to reason across a long sequence. Contextual bandits, a simpler form of reinforcement learning, handle this kind of problem by choosing an action based on the current situation and learning from the immediate reward. Because each decision is treated largely as a one-step interaction, the system doesn’t model how that action changes a series of future states.

Sequential reinforcement learning is appropriate when those downstream effects are part of the problem. If an inventory decision changes future stock levels, for example, or an AI agent’s tool call changes the information available for its next action, the policy has to learn how decisions and rewards unfold across multiple steps.

A reinforcement learning pipeline needs more than a training data set and model compute. It also requires interaction histories, versioned rewards, rollout generation, checkpoints and evaluation across nondeterministic runs.

Replayable interaction data

A useful record captures what the policy saw, what it selected and what followed. Depending on the system, this may include the state, available actions, chosen action, environment response, reward, policy version, tool calls and final outcome.

For AI-agent trajectories, tool arguments, returned errors and state changes are also typically included. Recording only the final response removes information needed for credit assignment, debugging and retrospective reward computation.

Reward and verifier lineage

A reward reflects a particular function, model, rubric or test. Teams should be able to trace the reward-function version, AI judge, evaluation rubric, verifier, policy checkpoint, evaluation data set and environment configuration associated with a score.

Changing a judge prompt or test suite may alter the evaluation even when the policy remains the same. Versioning these components helps separate policy improvement from evaluation drift and supports retrospective scoring when a better verifier is introduced.

Rollout throughput

Reinforcement learning alternates between generating experience and updating the policy. Rollout workers produce trajectories, training workers calculate updates and evaluators score the completed outputs or actions.

For agentic workloads, code execution, database queries and tool calls introduce latency between model calls. Additional training GPUs provide limited benefit when the environment cannot return trajectories quickly enough.

Long runs also require frequent checkpoints and fault recovery. The stored state should support resuming training and rolling back to a known-safe policy when later updates produce unstable behavior.

Evaluation and governance

Average reward alone doesn’t establish whether a policy performed the intended task reliably. Evaluation should also consider end-to-end task success, verifier pass rates, held-out environments, tool errors, latency, cost and variation across repeated runs.

For policies that take external actions, permission boundaries, approval workflows and reversible operations constrain what the agent may do. An audit trail should preserve what the policy observed, which action it selected, what the environment returned and which feedback reinforced the behavior.

Reinforcement learning connects training to a larger operational loop: policies generate interactions, environments return outcomes, reward systems evaluate them and the resulting data feeds the next policy update. Keeping those components close to the underlying data makes it easier to preserve the context needed to reproduce a run, compare policy versions and trace a reward back to the interaction that produced it.

Snowflake provides a governed foundation for those workflows. Teams can retain trajectories, model evaluation results, reward metadata and other training records alongside the enterprise data used to construct environments or evaluate outcomes, rather than separating model development from the data and governance controls around it. This is particularly useful for agentic RL, where a single trajectory may span database queries, tool results and multiple changes in state.

The machine learning stack can then support different parts of the model training lifecycle. Snowpark ML provides APIs for building data and model pipelines, while Snowflake Notebooks gives practitioners a development environment close to governed data. For compute-intensive training, Container Runtime for ML supports GPU-based workloads. Snowflake ML also provides model lifecycle and registry capabilities for tracking the models and versions produced by those workflows.

Reinforcement learning works when the system can connect actions to consequences well enough to improve the next decision. In modern LLM and AI-agent applications, that depends not only on the optimization method, but on the quality of the environments, rewards, interaction data and evaluation mechanisms surrounding it.

For practitioners, the practical question is less about choosing RL in the abstract and more about whether the problem, feedback and infrastructure support a reliable learning loop. When they do, reinforcement learning provides a way to improve behavior across decisions whose effects unfold over time.