Gradient Descent: How Machine Learning Models Learn From Error

When a machine learning model produces a wrong prediction, how does it determine which of its millions of weights contributed to the error — and how each one should change to improve performance?

Gradient descent provides the basic rule. During training, the model makes a prediction, the loss function measures how far that prediction is from the expected result, and backpropagation calculates how the loss would change in response to a small change in each weight. The optimizer then uses those gradients to adjust the weights in a direction expected to reduce the loss.

The rule is simple, but applying it well is not. The learning rate controls the size of each update. The batch determines which training examples contribute to the gradient. The optimizer determines how the current gradients, and in many cases information from earlier updates, translate into changes to the weights. Together, those choices influence whether the loss falls steadily, fluctuates, stalls or becomes unstable.

Modern frameworks calculate gradients automatically, so practitioners rarely implement gradient descent themselves. They configure the process around it — choosing batch sizes, learning-rate schedules and optimizer variants, then interpreting the loss curve to determine whether those choices are working.

Gradient descent is an optimization algorithm that adjusts a model’s weights to reduce its loss, meaning the numerical measure of how far its predictions are from the expected result. It does this iteratively, using the gradient of the loss to guide each update.

Those weights determine how the model transforms an input into an output. In a linear regression, for example, they define the slope and intercept of the fitted line. In a neural network, the same basic idea extends across many layers and potentially billions of parameters.

Gradient descent should be distinguished from gradient boosting. Gradient descent updates parameters within a model. Gradient boosting builds an ensemble by adding models sequentially, with each one fitted to the errors left by the models before it.

Watch this Snowflake BUILD session to learn how teams are using Snowflake ML to build and operationalize large-scale models:

To understand the full process, consider a simple linear regression model that predicts a house’s sale price from its square footage. The model uses two trainable parameters: a weight that controls how sharply predicted price rises with square footage and a bias that sets the starting level of the fitted line. During training, gradient descent adjusts those values so the line fits the observed sales data more closely.

Suppose the model begins with poorly chosen weights. For a 1,500-square-foot house that sold for $500,000, it predicts $400,000. Training now needs to measure the error, determine how the current weights contributed to it and revise them before the model predicts again.

1. The forward pass produces a prediction

During the forward pass, the model applies its current weights to the input and produces a predicted sale price.

A real training step would typically process a mini-batch rather than one house at a time. The model might predict prices for 256 properties in parallel, using the same current weights for every example in the batch.

2. The loss function measures the error

The loss function compares those predictions with the recorded sale prices. A regression model commonly uses mean squared error, which calculates the difference between each predicted and observed value, squares those differences and averages them across the batch.

Squaring prevents positive and negative errors from canceling each other out, while giving larger misses more influence on the final score. The result is one loss value representing how well the current weights performed on that batch.

At this point, the training process knows how large the error is. The loss value alone, however, doesn’t show how each weight should change.

3. Backpropagation calculates the gradients

Backpropagation calculates the gradient of the loss with respect to each trainable weight. For a given weight, the gradient describes how the loss would respond to a small change in its value.

The gradient’s sign indicates the direction of the local change. A positive gradient means that a small increase in the weight would increase the loss, while a negative gradient means that a small increase would reduce it. Its magnitude shows how sensitive the loss is to that weight at the model’s current settings.

In the linear regression example, the calculation involves only a few operations, while a neural network may contain millions or billions of weights connected across many layers. Backpropagation handles that larger calculation by applying the chain rule from the loss backward through every operation that contributed to the prediction.

4. The optimizer updates the weights

For plain gradient descent, the weight update can be written as:

wt+1​=wt​−η∇L(wt​)

The current weight is wt​, the gradient is ∇L(wt​), and the learning rate η controls the size of the change.

Because the optimizer subtracts the gradient, it moves the weight in the direction expected to reduce the loss. A positive gradient lowers the weight, while a negative gradient raises it.

In the house-price example, suppose the model is underestimating how strongly prices rise with square footage. Backpropagation may produce a negative gradient for the weight controlling that relationship, indicating that increasing the weight should reduce the loss. The optimizer raises it by an amount determined by the learning rate.

Some optimizers use more elaborate update rules, but they begin with the same gradients. They differ in how they scale, combine or transform that information before changing the weights.

5. The model repeats the loop

With the revised weights, the model processes the next mini-batch. It produces new predictions, calculates another loss, computes new gradients and updates the weights again.

The loss doesn’t have to decline after every batch. Each mini-batch contains a different sample of the training data, so an update that improves the model overall may still produce a slightly higher loss on the next group of examples. Practitioners look for a downward trend across many updates rather than expecting a perfectly smooth sequence.

After the model has processed every training example once, it has completed what is called an epoch. Training usually continues for multiple epochs, allowing successive updates to refine the weights as the model encounters the data again.

Modern frameworks automate most of this process. PyTorch, JAX and TensorFlow record the operations used during the forward pass and apply automatic differentiation to calculate the gradients. Practitioners define the model and loss function, select an optimizer and configure settings such as the learning rate and batch size.

A gradient can be calculated from the entire training data set, from one example or from a subset of examples. The choice changes both the cost of an update and the amount of variation in the direction it provides.

Batch gradient descent

Batch gradient descent uses every training example before updating the weights. In the house-price example, the model would predict a value for every property, calculate one loss across the full data set and derive a gradient from all of those observations.

Because every example contributes, the gradient represents the direction that reduces loss across the complete training set. The update is relatively stable, but each step requires a full pass through the data. At large data volumes, that approach is expensive and difficult to fit into memory.

Stochastic gradient descent

Stochastic gradient descent, in its strict mathematical definition, updates the weights after a single example. Each step is inexpensive, although one observation may point in a direction that differs considerably from the direction favored by the data set as a whole.

The resulting path is noisy. Loss may rise on one update and fall on the next even while the longer-term trend improves. Single-example updates also make poor use of the parallel processing available on modern accelerators.

Mini-batch gradient descent

Mini-batch gradient descent uses a subset of examples for each update. A batch might contain 32, 256 or several thousand records, depending on the model, data and available hardware. This gives the framework enough work to process efficiently in parallel while preserving frequent updates.

In everyday deep-learning usage, the term stochastic gradient descent (or SGD) often includes mini-batch training because each batch provides a sampled estimate of the full-data gradient.

Mini-batch noise isn’t purely a cost. Each batch provides a slightly different estimate of the full-data gradient. That variation can sometimes help the optimizer move out of shallow or low-gradient regions, although it can also make the training path less stable. Under some conditions, the noise acts as a form of implicit regularization and contributes to solutions that perform better on unseen data.

Batch size therefore affects more than speed. Larger batches produce smoother, lower-variance gradient estimates. Smaller batches yield more frequent and variable updates. Changing the batch size also changes the number and character of the updates, so practitioners usually revisit the learning rate and schedule at the same time. Larger batches can sometimes support larger learning rates, but the relationship depends on the optimizer, architecture and training setup.

Available memory places an upper bound on the batch. Inputs, activations, gradients, model weights and optimizer state must all coexist during training, with activations frequently consuming a large share of GPU memory.

The learning rate determines how far the weights move during each update. Even when the model, data and gradient remain unchanged, a learning rate of 0.1 produces a much larger adjustment than a learning rate of 0.001.

When the rate is too high, the weights may leap past lower-loss values. One update crosses a useful region, the next crosses back and the loss begins to oscillate. With still larger steps, the calculations may become unstable and produce an infinite or NaN loss.

A very small rate creates slower progress. The loss may decline, but each adjustment changes the model so little that training requires far more steps than the available budget supports. In a flat part of the landscape, the curve may appear stalled.

Returning to the regression model, suppose the fitted line lies below most of the observed house prices. The gradient indicates that the slope should increase. A moderate learning rate moves the line closer over several updates. An excessive rate sends the slope past the useful range, while a very small one barely changes it.

The appropriate step size can shift during training. Early in a run, weights may be far from a useful solution and gradients may vary sharply. Later, after the loss has declined, smaller updates can refine the parameters without repeatedly crossing low-loss regions.

Large neural-network training runs often use a learning-rate schedule. Warmup begins with small steps and increases the rate over an initial sequence of updates. Once the model and optimizer state have stabilized, the schedule reaches a peak and later reduces the rate.

Cosine decay is one common approach. It lowers the learning rate along a smooth cosine-shaped curve, allowing larger updates during the main phase of training and finer adjustments near the end. Other schedules use linear decay, stepwise reductions or performance-based changes.

A schedule still requires sensible starting values. The peak rate, warmup duration and final rate interact with the batch size, optimizer and architecture. Changing the batch without revisiting the learning rate can turn a stable training configuration into a slow or unstable one.

When loss rises sharply, oscillates or becomes NaN, an excessive learning rate is one of the first causes to investigate. The diagnostic section later in this article covers the other conditions that can produce similar patterns.

Practitioners commonly test several learning rates and batch sizes rather than treating either as an isolated choice. Hyperparameter optimization tools such as Optuna and Ray Tune can run candidate configurations concurrently, although each candidate still incurs its own training cost.

Plain gradient descent scales the current gradient by the learning rate and applies it directly. Modern optimizers change that calculation by incorporating earlier gradients, adapting the step for individual parameters or using structural information from the model’s weight matrices.

Momentum methods

Momentum maintains a running direction based on recent updates. When several gradients point the same way, their effect accumulates. A single mini-batch that points sharply elsewhere has less influence because the optimizer carries information from the preceding steps.

This is useful on uneven loss landscapes. In a long, narrow valley, plain gradient descent may repeatedly cross from one wall to the other. Momentum preserves more motion along the valley while damping some of that lateral oscillation.

Coordinate-wise adaptive methods: AdaGrad, RMSprop, Adam and AdamW

Adaptive methods adjust the effective learning rate for each parameter. AdaGrad introduced parameter-specific rates based on accumulated squared gradients, although its rates can shrink too aggressively over a long run. RMSprop uses a moving average instead, allowing recent gradient behavior to carry more weight.

Adam combines a momentum-like estimate of the gradient’s direction with a moving estimate of its squared magnitude. Each parameter receives an update informed by both. This often makes Adam easier to tune than plain stochastic gradient descent, especially when gradients vary widely across parameters, although the learning rate still has a major effect on stability and final performance.

AdamW separates weight decay from Adam’s adaptive gradient update. In the original Adam formulation, adding an L2 penalty to the loss doesn't behave exactly like conventional weight decay because the optimizer’s parameter-specific scaling also affects that penalty. Applying weight decay separately makes its effect easier to interpret and gives practitioners more direct control over optimization and regularization.

AdamW is a common default for transformer training, while stochastic gradient descent with momentum remains competitive in areas such as computer vision. A carefully tuned SGD configuration can generalize well and uses less optimizer state, but it often demands more experimentation.

Memory is part of that choice. Adam and AdamW typically maintain two additional values for every trainable parameter. On a large model, those moment estimates can occupy a substantial share of the total memory footprint alongside the weights and gradients.

Structured or matrix-aware methods: Shampoo, SOAP and Muon

Structured optimizers take advantage of the fact that many neural-network parameters are organized as matrices or higher-dimensional tensors. Rather than adapting every scalar entry independently, these methods use relationships across rows, columns or tensor dimensions when constructing the update.

Shampoo maintains preconditioning information for each dimension of a tensor, while SOAP applies Adam-style adaptation within Shampoo’s preconditioned coordinate system. These methods use richer structural information than coordinate-wise optimizers and can capture some of the benefits associated with curvature-aware optimization.

Muon takes a different approach. It transforms momentum updates for two-dimensional weight matrices through an orthogonalization step before applying them, rather than directly estimating the curvature of the loss.

Interest has grown as large-model experiments have shown that a more effective update may reduce the training required to reach a target quality. In one 1.5-billion-parameter experiment, Muon reached GPT-2 XL-level performance approximately 25% faster than AdamW. The trade-off is workload-specific: additional computation or optimizer state during each step must be balanced against any reduction in the total number of steps.

For most training projects, the practical starting point remains an optimizer already proven for the model family. Learning rate and schedule usually deserve attention before replacing a well-supported default. At the scale of frontier-model training, however, optimizer choice has reemerged as a meaningful compute and performance decision.

A loss curve summarizes how the model’s error changes across updates. Its shape can’t identify a cause by itself, but it can show whether training is progressing, becoming unstable or separating from validation performance.

A healthy curve often drops quickly at first, then declines more gradually. Mini-batch variation may make the raw line uneven, especially with smaller batches, so practitioners frequently examine both raw and smoothed values.

Loss is flat from the beginning

When loss barely changes from the first updates, the model may not be receiving a useful signal. The learning rate could be too small, the gradients could be close to zero or the inputs and labels could be misaligned.

Vanishing gradients are one possible cause. As backpropagation moves through a deep network, the sequence of derivatives can repeatedly weaken the signal. The gradients reaching earlier layers may then become too small to change their weights meaningfully.

Activation functions, initialization and architecture all influence gradient flow. ReLU-family activations preserve stronger gradients across much of their range than saturating functions. Initialization methods scale starting weights according to the number of incoming or outgoing connections, while residual connections provide shorter paths through deep networks.

Gradient norms help distinguish among causes. Near-zero values across many layers suggest weak gradient flow or a disconnected computational graph. Healthy gradients paired with a flat loss point toward the learning rate, loss calculation or data.

Loss rises, oscillates or becomes NaN

Aggressive updates often produce a rising or sharply oscillating curve. Lowering the learning rate is a sensible first test, followed by checks for exploding gradients, invalid input values and numerical instability.

Exploding gradients occur when the chain of operations used during backpropagation repeatedly amplifies the gradient. In deep or recurrent networks, those effects can compound until the resulting updates become extremely large. Gradient clipping can limit the damage. One common form, global-norm clipping, rescales the full set of gradients when their combined norm exceeds a threshold.

Clipping is especially useful in architectures prone to occasional spikes, although frequent clipping indicates that the learning rate, initialization, numerical precision or model design may still need attention.

Loss declines and then plateaus

A plateau may indicate that the model is approaching the best solution available under the current configuration. It can also appear when the learning rate has decayed too far, the model lacks capacity or the optimizer has entered a flat or poorly conditioned region.

Saddle points are one possible explanation, although the loss curve alone cannot establish that diagnosis. Along some directions, the loss curves upward; along others, it curves downward. Near the center, the gradient may be small even though lower-loss routes still exist. Momentum, adaptive updates and mini-batch variation can sometimes help the optimizer continue moving.

A plateau can also reflect limits in the data. Noisy labels, missing predictors or irreducible variation place a floor under the loss that optimization alone cannot remove.

The current learning rate, gradient norms and validation behavior provide more context than the training curve alone. If both training and validation loss flatten at acceptable levels, further optimization may offer little value.

Training and validation loss diverge

When training loss continues to fall while validation loss rises, the optimizer is still reducing error on the training examples, but the model’s performance on unseen data is deteriorating.

This pattern points toward overfitting rather than a failure of gradient descent. Regularization, more representative training data, augmentation or an earlier stopping point may improve generalization.

Normalization can stabilize training

Normalization can improve optimization stability by keeping intermediate activations within a more manageable range. Batch normalization uses statistics calculated across a mini-batch, often supporting higher learning rates and smoother training. Its original explanation focused on reducing internal covariate shift, while later research has placed greater emphasis on smoothing the loss landscape.

At very small batch sizes, batch-level statistics become less reliable. Transformers therefore typically use layer normalization, which normalizes across each token’s hidden features and doesn’t depend on the other examples in the mini-batch.

The curve supplies a starting point for investigation. Gradient measurements, validation metrics, data checks and system logs establish which part of the training process actually needs adjustment.

Each gradient update begins only after the training system has prepared another mini-batch. Reading records, applying transformations, moving tensors into accelerator memory and running the forward and backward passes all contribute to the time required for one step.

At production scale, several constraints tend to determine how efficiently that loop runs:

• Data loading and preprocessing: When the input pipeline cannot supply examples as quickly as the accelerator can process them, the GPU waits between updates. Tokenization, image decoding, augmentation and feature construction often run on CPUs, so loader concurrency, storage throughput and preprocessing design can affect training time even when the loss curve looks normal.
• Accelerator memory: Training must hold model weights, forward-pass activations, gradients and optimizer state at the same time. Larger batches increase activation memory, while optimizers such as AdamW maintain additional tensors for every parameter. When the workload no longer fits, practitioners may reduce batch size, accumulate gradients across smaller batches, recompute selected activations during backpropagation or use reduced-precision formats.
• Distributed execution: Larger models and batches may require parameters, gradients or optimizer state to be spread across multiple devices. Under data parallelism, each GPU processes a different mini-batch, then participates in an all-reduce operation so the workers apply a consistent update. As device counts grow, synchronization and interconnect bandwidth can consume a larger share of each step.
• Utilization and throughput: Adding GPUs doesn't produce a proportional reduction in run time. In a large-scale Megatron-LM study, a 1-trillion-parameter training run reached 52% of theoretical per-GPU peak performance across 3,072 A100 GPUs, illustrating how memory access, communication and other overhead remain significant even in a highly optimized system. Step time and examples processed per second often provide a clearer operational view than device count alone.
• Checkpointing and recovery: Long training runs periodically save model weights, optimizer state and progress metadata so they can resume after an interruption. Frequent checkpoints reduce the amount of work at risk, but writing large states too often adds storage traffic and delays subsequent steps.

Optimization choices determine how many updates the model needs. Data throughput, memory use and distributed execution determine how long those updates take.

Snowflake ML brings training code, governed data and CPU or GPU compute into the same environment. Within Container Runtime, practitioners can use familiar open source frameworks while Snowflake manages the infrastructure that runs them. Teams choose the workload configuration, dependencies and resource requirements, without having to build and maintain the underlying cluster themselves.

Supply the training loop with data

Container Runtime provides optimized access to Snowflake tables and converts query results into objects used by common ML libraries. Training can operate near data already held and governed in Snowflake, reducing the need to export and maintain a separate training copy. In one Snowflake test, Container Runtime loaded an 81 GB training data set in about 50 seconds, compared with approximately 9.5 minutes in the external environment used for the comparison.

Snowflake Notebooks provide an interactive environment for exploring the data, developing features and testing training code. Since the notebook runs against Snowflake data and compute, practitioners can move from query results to framework-native training objects within the same workflow.

Match compute to the workload

Compute pools supply the CPU or GPU resources used by Container Runtime. Teams can select an environment suited to the model’s memory and processing requirements without provisioning and maintaining the underlying cluster themselves.

For models that require more than one worker, Snowflake distributed trainers coordinate execution across nodes and GPUs for supported model families and frameworks, including PyTorch-based neural networks and distributed XGBoost workloads. The training code continues to define the model, loss function and optimizer, while Snowflake handles worker setup and resource orchestration.

The framework still controls techniques such as gradient accumulation, mixed-precision training and model sharding. Container Runtime provides the managed environment in which those techniques run.

Run repeatable jobs and compare experiments

ML Jobs lets teams submit training workloads to Snowflake compute from Snowflake Notebooks or an existing development environment. This separates interactive development from repeatable execution while preserving the same underlying code.

When teams test several learning rates, batch sizes or optimizer configurations, parallel ML Jobs can run the candidates concurrently. Snowflake ML Experiments records parameters, metrics and artifacts across those runs, giving practitioners a consistent basis for comparison.

Once training is complete, the model can be logged in the Snowflake Model Registry. The registry stores model versions and metadata as governed Snowflake objects and provides a path into inference and downstream lifecycle management.

Gradient descent gives model training its basic feedback loop: make a prediction, measure the error, calculate the gradients and update the parameters. Whether that loop works well depends on the choices around it. Batch size shapes the gradient, the learning rate controls the size of each step and the optimizer determines how those signals become an update. The loss curve shows the result, while gradient norms, validation metrics and data checks help explain what is happening.

At production scale, training is also a systems problem. An effective configuration reduces the number of updates needed, while efficient data loading, memory use and distributed execution determine how quickly those updates can run.