Self-Supervised Learning

A machine learning process where the model generates its own labels from the input data, often used in large language models.

How it works

Self-supervised learning creates supervision signals from the structure of the data itself, without human annotation. For LLMs, the pretext task is next-token prediction: the label for each token is the next token in the sequence, which is already present in the data. For vision models, pretext tasks include predicting masked image patches (MAE) or matching different views of the same image (contrastive learning). The model learns rich representations that transfer to downstream tasks.

Why it matters

Self-supervised learning is what makes training on internet-scale data feasible. Human annotation is the bottleneck for supervised learning — you can only label as much data as humans can process. Self-supervision eliminates this bottleneck by exploiting the natural structure of text, images, and audio. GPT, BERT, and all modern vision foundation models are self-supervised. This paradigm shift enabled the current era of large, capable models that generalise across tasks.