GGUF

A binary file format used for storing models for inference (especially via llama.cpp), allowing them to run efficiently on standard consumer hardware, including CPUs.

How it works

GGUF (GPT-Generated Unified Format) is a self-contained binary format that stores model weights, tokeniser vocabulary, model metadata, and quantisation parameters in a single file. It supports multiple quantisation levels (Q4_0, Q4_K_M, Q8_0, etc.) out of the box. The llama.cpp inference engine reads GGUF files and runs models efficiently on CPUs, Apple Silicon (Metal), and GPUs using hand-optimised SIMD kernels, enabling model serving without any Python or CUDA dependency.

Why it matters

GGUF made running LLMs locally accessible to virtually anyone with a modern laptop. A quantised 7B model in GGUF format can run at multiple tokens per second on a MacBook Air with no GPU. This democratised private, offline AI inference and spawned tools like Ollama and LM Studio that let non-technical users run powerful models locally. GGUF has become the standard format for the offline AI community, with models on Hugging Face commonly released in both standard and GGUF versions.