CUDA Graphs can speed up repeated GPU work by preparing its execution plan once and replaying it with new data. They are a feature of NVIDIA’s CUDA platform. In LLM inference, that work can be the sequence of operations used to generate the next token. Replay launches those operations again; it does not return an earlier answer or avoid computing the new token.
Serving engines such as vLLM expose full and piecewise graph modes, making replay coverage a practical choice about latency and GPU memory.
How can a reusable plan handle an LLM whose tokens, context lengths, and active requests keep changing?
The useful distinction is between the description of the work and the data that work reads. A captured operation can keep the same input address while the values stored at that address change. The GPU then follows the prepared dependencies and computes a fresh result.
A fixed execution plan does not require a fixed answer. An operation that doubles the numbers in an input buffer produces a different output after those numbers change, even though the operation and its input address are unchanged. Replacing the buffer itself is a different matter: the prepared operation may still point to the original storage.
LLM serving builds on that distinction. New tokens, positions, and cache locations can become updated data inside persistent buffers. Changes to the amount or arrangement of work may instead require another prepared graph, a smaller captured region, or ordinary execution.
We will follow that boundary from GPU launches to growing token histories, then examine when the cost of keeping several graph variants ready outweighs the launch savings.
Table of contents
Where a decoding step spends time
Capture a plan, replay fresh work
Keep addresses stable while values change
Fit changing LLM state into a replay
Choose the right graph for the batch
Decide whether replay pays
