Residual Connection

A skip connection that adds a sublayer’s input to its output, helping deep Transformers train stably by letting gradients flow through the network.

How it works

A residual connection takes the input to a sublayer (say, attention or FFN) and adds it directly to the output of that sublayer: output = sublayer(x) + x. This means the sublayer only needs to learn the 'residual' — the difference from the identity function — rather than the full transformation. During backpropagation, gradients can flow directly through the residual path without passing through the sublayer's weights, preventing the vanishing gradient problem in very deep networks.

Why it matters

Residual connections, introduced in ResNet (2015), made it possible to train networks with hundreds of layers without gradients vanishing before they reached early layers. Every Transformer block uses residual connections around both the attention and feed-forward sublayers. Without them, models with 80 to 120 layers — like modern LLMs — would be untrainable. They are one of the most impactful and simple architectural innovations in the history of deep learning.