Sticky Session

Routing successive requests from the same user or dialogue to the same server so cached state (such as a KV cache) can be reused and hit rates stay high.

How it works

In stateless LLM serving, each request is routed to any available server, and the KV cache from previous turns must be recomputed from scratch (or fetched from shared storage). With sticky sessions, a load balancer routes all requests from the same session ID (user, conversation, or prefix) to the same server instance. This server can keep the KV cache of the conversation's history in its local VRAM, allowing new turns to be served with only incremental computation rather than full context recomputation.

Why it matters

Sticky sessions can dramatically reduce serving costs for multi-turn conversations. A conversation with 50 previous turns requires the model to process all 50 turns' tokens on every new request if stateless. With a warm KV cache from sticky session routing, only the new turn's tokens need processing. The cost savings compound as conversations grow longer. This is particularly valuable for long-running agentic workflows and enterprise chatbot deployments with long system prompts that benefit from KV cache prefix sharing.