TTFT (Time To First Token)

Latency from sending a request until the first generated token arrives. A key user-facing metric for chat apps — how quickly the reply starts streaming.

How it works

TTFT encompasses the end-to-end latency from when a client sends a request to when the first token of the response begins to arrive. It includes network round-trip time to the inference server, prefill time (processing the full prompt through all Transformer layers), sampling the first output token, and the server's response write time. For long prompts, prefill dominates TTFT since it scales linearly with input length. TTFT is measured separately from TPOT because they have different bottlenecks and optimisation strategies.

Why it matters

TTFT is the primary determinant of how 'snappy' an AI application feels. Users perceive a chatbot as responsive if text starts appearing within 1-2 seconds, regardless of how fast subsequent tokens arrive. A high TTFT — caused by a long system prompt, a busy server, or a slow prefill — makes the product feel sluggish even if TPOT is fast. Optimising TTFT motivates techniques like chunked prefill (processing prefill in chunks to interleave with decode), GPU batching strategies, and network co-location with compute.