Inference

The process of using a trained machine learning model to make a prediction or generate text.

How it works

During inference, a trained model's weights are frozen. The input is tokenised, converted to embeddings, and passed through the model's layers in a single forward pass (for classification) or repeatedly token by token (for text generation). For LLMs, the KV cache stores intermediate computations from previous tokens so they don't need to be recomputed at each step. Hardware accelerators like GPUs and TPUs are used to perform the large matrix multiplications that dominate inference compute.

Why it matters

Inference is where AI creates business value — every API call, every chatbot response, every recommendation is an inference. Inference efficiency directly determines the cost per query, which in turn determines whether an AI product is economically viable. Innovations like quantisation, speculative decoding, continuous batching, and the KV cache all exist to make inference faster and cheaper, and they represent a multi-billion-dollar engineering focus across the industry.