DPO (Direct Preference Optimization)

A streamlined alternative to RLHF that directly optimizes a language model to align with human preferences without needing a separate reward model.

How it works

DPO reformulates the RLHF objective mathematically to show that the optimal policy under a KL-constrained reward maximisation problem can be found by directly fitting the language model to preference data, without training a separate reward model. Given pairs of (preferred, rejected) responses, DPO increases the probability of preferred outputs and decreases the probability of rejected ones using a binary cross-entropy loss computed directly on the policy.

Why it matters

DPO dramatically simplified the RLHF alignment pipeline. Training a separate reward model, then running PPO, requires significant engineering complexity, compute, and careful hyperparameter tuning. DPO collapses these into a single fine-tuning stage on preference pairs, making alignment accessible to teams without dedicated RL infrastructure. Most open-source alignment methods today use DPO or its variants (IPO, SimPO, ORPO), making it the dominant practical alignment technique outside of OpenAI and Anthropic.