TPOT (Time Per Output Token)

Average time between successive generated tokens after the first one. Together with TTFT, it describes how fast a chat feels once streaming has begun.

How it works

After the first token is generated (TTFT), the model continues in the decode phase, generating one token at a time. TPOT measures the average wall-clock time between successive tokens during this phase. Decode is memory-bandwidth-bound: at each step, all model weights must be read from VRAM once, and the bottleneck is how fast the GPU can transfer bytes from memory — not how fast it can compute. Larger models have lower token generation rates (higher TPOT) because they require more memory bandwidth per token.

Why it matters

TPOT determines the streaming experience once text has started appearing. At 40ms TPOT (25 tok/s), text appears faster than humans can comfortably read. At 200ms TPOT (5 tok/s), the experience feels slow and frustrating. Optimising TPOT motivates quantisation (smaller weights = less memory bandwidth = lower TPOT), speculative decoding (multiple tokens per memory pass), and continuous batching (amortising memory reads across concurrent requests). It is the primary throughput metric for AI serving infrastructure teams.