PPO (Proximal Policy Optimization)

A popular reinforcement learning algorithm used in the RLHF phase of training large language models to balance stability and training speed.

How it works

PPO is a policy gradient RL algorithm that updates the model's policy (the language model) to maximise expected reward while constraining how much the policy changes per step. It introduces a clipped surrogate objective that prevents large policy updates which could destabilise training. In RLHF, a frozen reward model scores generated responses, and PPO uses these scores as rewards to push the LM toward generating higher-scoring responses. A KL penalty term also prevents the LM from deviating too far from its SFT baseline.

Why it matters

PPO was the workhorse of RLHF training at OpenAI, Anthropic, and others, and is directly responsible for the aligned, helpful behaviour of ChatGPT, Claude, and Gemini. Its stability advantages over earlier policy gradient methods made large-scale RLHF feasible. DeepSeek-R1's use of GRPO (a PPO variant without the value network) showed that even simpler RL formulations could produce excellent reasoning models, prompting re-evaluation of how much complexity the alignment RL process actually requires.