SwiGLU

A gated activation function used in many modern LLM feed-forward blocks: SiLU on one projection multiplied by another linear gate, giving stronger expressiveness than a single linear layer plus ReLU.

How it works

SwiGLU computes FFN(x) = (SiLU(xW₁) ⊙ xW₂)W₃, where ⊙ is element-wise multiplication. The SiLU (Sigmoid Linear Unit, also called Swish) branch provides a smooth non-linearity, while the second linear gate learns to selectively suppress or amplify different activation dimensions. This gating mechanism makes the FFN more expressive than vanilla ReLU or GELU activations. PaLM and LLaMA pioneered its use in LLMs; it is now standard in most modern Transformer architectures.

Why it matters

SwiGLU consistently outperforms ReLU and GELU activations in language model benchmarks for the same number of parameters and compute. Its adoption by LLaMA, Mistral, Gemma, and DeepSeek makes it the de facto standard for modern open-source LLMs. Understanding SwiGLU is important for implementing custom model architectures correctly and for interpreting published model configurations, where FFN variant choice is a key architectural decision.