Causal Attention

An attention mask used in autoregressive language models so each token can only attend to itself and earlier tokens, never to future tokens.

How it works

In a standard Transformer, each token can attend to all other tokens. For language generation, this would allow tokens to 'see' future tokens, which is invalid during inference (where future tokens don't exist yet). Causal attention applies a triangular mask to the attention score matrix: all positions above the diagonal are set to -infinity before softmax, resulting in zero attention weight. This enforces a strict left-to-right information flow, making training consistent with inference.

Why it matters

Causal masking is the mechanism that makes decoder-only Transformer training parallel. Without it, sequential text generation could not be trained efficiently — each token would need to wait for all previous tokens to be generated before it could be trained. With causal masking, all positions in a sequence can be trained simultaneously on a single forward pass. This is what makes the pre-training of GPT-style models on billions of tokens computationally feasible.