Muon Optimizer

A training optimizer used in Moonshot’s Kimi K2 work that aims to get more model quality from the same token budget than a plain AdamW-style setup.

How it works

Muon is an optimiser based on the Nesterov accelerated gradient with an orthogonalisation step applied to the gradient matrices using Newton-Schulz iterations before the weight update. This orthogonalisation normalises the update direction, preventing some parameters from being updated far more than others — a common issue with AdamW that causes training instability or inefficiency in large models. Muon is applied to the non-embedding Transformer weights; Adam is still used for embeddings and scalar parameters.

Why it matters

Moonshot AI reported that Muon improved training efficiency for Kimi K2 compared to AdamW, squeezing more model quality from the same compute budget. In the highly competitive world of frontier model training where every percentage point of efficiency matters, a better optimiser can translate to significant cost savings or capability improvements. Muon's use in a frontier model has drawn significant attention to second-order-inspired optimiser research as a practical lever for improving training efficiency.