Attention Mechanism

A technique in neural networks that allows the model to focus on specific parts of the input when generating an output, heavily used in Transformers.

How it works

Attention computes three projections for each token: a Query, a Key, and a Value. The dot product between a token's Query and every other token's Key produces a score reflecting relevance. These scores are passed through a softmax to become weights, and then used to create a weighted sum of the Value vectors. The result is a context-aware representation of each token that accounts for its relationship to every other token in the sequence.

Why it matters

Before attention, recurrent networks processed sequences step-by-step and struggled to connect information separated by many tokens. Attention allows every token to directly interact with every other token in a single operation, solving the long-range dependency problem. This breakthrough made Transformers possible and ultimately led to GPT, BERT, and the entire current generation of language models.