VRAM (Video RAM)

The memory located on a graphics card (GPU). In AI, sufficient VRAM is critical for loading large models and their context windows entirely into GPU memory for fast inference.

How it works

VRAM is the on-chip memory of a GPU, distinct from system RAM and far faster than it for the massively parallel matrix operations that underpin deep learning. During inference, model weights are loaded into VRAM (e.g., a 7B parameter model at BF16 requires ~14GB). During generation, the KV cache for active conversations also occupies VRAM. If VRAM is exhausted, the model either cannot run or must offload to slower system RAM, dramatically reducing throughput.

Why it matters

VRAM is the single most important hardware constraint for running large language models. It determines which models are feasible on a given machine and at what batch size. An H100 has 80GB VRAM; a consumer RTX 4090 has 24GB. This is why quantisation (fitting 70B models into ~35GB with INT4) and MoE architectures (activating only a fraction of parameters per token) are so valuable — they multiply effective VRAM capacity. VRAM availability directly caps the context length and batch size of production serving systems.