An AI agent can spend most of a model call reading a history it has already processed. The KV cache preserves the keys and values that a causal Transformer computed for earlier tokens, at each attention layer. Later tokens can consult those saved states instead of running the historical tokens through the model again. The model's weights stay unchanged; this is reusable inference state.
DeepSeek explicitly connects cache-hit charges to a large share of its agent workloads' costs, making the storage and reuse of that state a product-cost concern as well as a serving optimization. DeepSeek's release explanation
Consider the shape of the work. A short chat might contain 500 input tokens and produce 500 output tokens. A coding agent can accumulate system instructions, repository files, earlier decisions, dozens of tool calls, and long tool results. It may then read 200,000 tokens to produce just 200 more before calling another tool.
Must the next call rebuild everything the model already computed?
It need not. But preserving the work creates a second question: where is the saved state, and how much of it must move before the agent can continue?
Three different savings sit behind that question. A smaller representation reduces the bytes needed for each remembered position. Prefix caching reuses computation from an earlier call. Keeping the reusable state near the executing GPU avoids restoring it across a slower connection. These savings interact, but they are not interchangeable.
Here is the useful starting distinction: a cache hit can eliminate the work of rebuilding an old token without eliminating the work of consulting it. A newly arrived tool result still needs to interact with the previous context. Every new generated token may consult that context again. A cheap input-cache hit therefore does not imply cheap generation from a long history.
We'll follow those saved states from attention through head sharing, latent compression, paging, persistent storage, and separate prefill and decode workers. Then we'll account for a 50-turn agent and use a separate timing example to decide when retaining, restoring, or rebuilding state pays off.
Table of contents
The work a KV cache saves
When bytes limit decoding
Shrink the state per token
Reuse the prefix without copying it
Keep, move, or recompute
Account for a 50-turn agent
Choose a retention policy
Design models around the cache
