Feed-Forward Network (FFN)

The fully connected layers inside each Transformer block that transform attention outputs. In MoE models, these FFNs are often the routed experts.

How it works

Each Transformer block contains an FFN sublayer that applies two linear transformations with a non-linear activation between them: FFN(x) = activation(xW₁ + b₁)W₂ + b₂. In most modern LLMs, the FFN uses a gated variant like SwiGLU: FFN(x) = (SiLU(xW₁) ⊙ xW₃)W₂. The FFN's hidden dimension is typically 4x the model dimension, so FFN weights account for ~66% of a Transformer's total parameters. In MoE models, this FFN is replaced with N expert FFNs, with a router choosing the active ones.

Why it matters

FFNs are where most of a Transformer's knowledge is stored. Mechanistic interpretability research has found that FFN layers act as associative memory, encoding factual associations like (Barack Obama, born in, Hawaii). This makes FFNs the primary target for knowledge editing and continual learning research. The FFN also dominates memory usage in MoE architectures — the inactive expert weights must be stored somewhere even if not used, which is why MoE models have large total parameter counts.