Activation Function

A mathematical function applied to a neural network node to help it learn complex patterns.

How it works

After a neuron computes a weighted sum of its inputs, the activation function decides whether and how strongly that signal is passed on to the next layer. Without activation functions, a neural network would simply be a linear regression, incapable of learning non-linear relationships. Functions like ReLU output zero for negative inputs and pass positive values through unchanged, introducing non-linearity cheaply. Others like sigmoid squash values into a 0–1 range, useful for binary classification outputs.

Why it matters

Activation functions are what give neural networks the ability to model the complex, non-linear patterns found in language, images, and sound. Choosing the right one — and avoiding problems like vanishing gradients — directly determines how deep a model can be trained and how quickly it converges. Modern choices like SwiGLU and GELU power the quality improvements seen in today's frontier language models.