Pretraining Annealing

Near the end of pretraining, gradually shifting the data mix toward higher-quality or instruction-style examples so the model settles into better behaviour before fine-tuning.

How it works

In standard pre-training, the data mix is held roughly constant throughout. In pretraining annealing, the learning rate is cosine-decayed to near-zero while simultaneously shifting the training data mix toward higher-quality sources — curated books, academic papers, synthetic instruction data, or code. This final phase, lasting perhaps 1-5% of total training compute, allows the model to 'settle into' better representations without catastrophic forgetting, producing a stronger base model before SFT begins.

Why it matters

Pretraining annealing is an under-discussed but important detail in modern LLM training recipes. Llama 3 and other recent models have highlighted annealing as a key contributor to their base model quality. By front-loading noisy web data for broad knowledge and reserving high-quality data for the final phase, labs get the benefits of both: broad generalisation from large-scale pre-training and precise, high-quality behaviour from the annealing phase. It reduces the work required in subsequent SFT stages.