QK-Clip

A training stability technique from Moonshot’s Kimi K2 report that clips extreme query-key attention scores to keep long-context training stable.

How it works

During attention computation, query-key dot products can grow very large in magnitude, especially for long sequences, causing attention weights to become numerically unstable (the softmax saturates to near-zero or near-one). QK-Clip addresses this by clipping the raw QK dot product values to a maximum absolute value before applying softmax. This prevents attention scores from reaching extreme values that could cause NaN gradients or training instability, particularly important when training on very long context windows.

Why it matters

Training stability is one of the most challenging practical problems in large-scale LLM training. A single training instability event — a spike in loss or a gradient NaN — can invalidate days or weeks of expensive GPU compute. QK-Clip is a simple but effective technique to prevent one specific class of instability (attention score explosion) that becomes more common as context length increases. Its inclusion in Kimi K2's training pipeline signals the kind of engineering detail that separates successful frontier model training from failed attempts.