Constitutional AI

An alignment approach (popularised by Anthropic) where a model critiques and revises its own outputs against a written set of principles, instead of relying only on human preference labels.

How it works

Constitutional AI has two phases. In the supervised learning phase, the model is asked to critique its own potentially harmful responses according to a set of written principles (the 'constitution'), then revise them based on that critique. In the RL phase, an AI-generated preference dataset — created by having the model compare pairs of its outputs against the constitutional principles — is used to train a reward model, which then guides further fine-tuning via RLHF. This allows alignment to scale without requiring human labellers to evaluate every output.

Why it matters

Constitutional AI addresses a key scalability bottleneck in AI alignment: collecting high-quality human preference data is expensive and slow. By using the model itself as a critic guided by explicit principles, Anthropic can scale alignment to more diverse situations than human raters could cover. It also makes alignment decisions more transparent and auditable, since the principles are explicit text rather than opaque human intuitions.