Gradient Descent

An optimization algorithm used for minimizing the cost function in various machine learning algorithms.

How it works

Gradient descent computes the gradient (partial derivatives) of the loss function with respect to every model parameter. This gradient points in the direction of steepest increase of the loss. The algorithm takes a small step in the opposite direction — scaled by the learning rate — to nudge all parameters toward lower loss. Stochastic Gradient Descent (SGD) estimates the gradient from a single example; mini-batch SGD (the standard in deep learning) estimates it from a small batch, which is faster and more stable than either extreme.

Why it matters

Gradient descent is the universal training algorithm underlying virtually all neural network learning. Without it, there would be no efficient way to adjust billions of model parameters to reduce prediction error. Modern variants like Adam and AdamW add adaptive learning rates per parameter, momentum, and weight decay, making training dramatically more stable and efficient than vanilla SGD. Every AI model in production was shaped by some variant of gradient descent.