Hyperparameter

A parameter whose value is used to control the learning process of a machine learning algorithm.

How it works

Unlike model parameters (weights) which are learned from data, hyperparameters are set before training begins and control how learning happens. Examples include the learning rate (how large each gradient step is), batch size (how many examples per update), number of layers, dropout rate, and number of training epochs. Hyperparameter tuning — finding the best combination — is typically done via grid search, random search, or Bayesian optimisation on a validation set.

Why it matters

The same model architecture with different hyperparameters can produce wildly different results. A learning rate that is too high causes training to diverge; too low and training is painfully slow. Getting hyperparameters right is often the difference between a working model and one that fails to train at all. At the frontier, the exact hyperparameter schedules (warmup steps, cosine decay, batch size ramp) used during training are closely guarded as competitive advantages.