LoRA (Low-Rank Adaptation)

A highly efficient fine-tuning technique that drastically reduces the number of trainable parameters by injecting small rank-decomposition matrices into each layer of the Transformer architecture.

How it works

LoRA freezes all original model weights and inserts two small matrices A (d×r) and B (r×d) into each attention layer, where r is the rank (typically 4-64, much smaller than d). Only A and B are trained. Their product BA approximates the weight update that full fine-tuning would learn. At inference, the LoRA matrices can be merged into the original weights at zero additional cost, or kept separate to allow hot-swapping between LoRA adaptors for different tasks.

Why it matters

LoRA made fine-tuning of large models practical on consumer hardware. Fine-tuning a 7B-parameter model with LoRA requires a fraction of the VRAM and compute of full fine-tuning. This enabled the community to fine-tune LLaMA, Mistral, and other open models on consumer GPUs, producing thousands of specialised model variants. LoRA adaptors are small (∼10-100MB) and swappable, making it practical to maintain many task-specific adaptors for a single base model in production.