Speculative decoding can make a coding assistant respond faster by having a small model propose several future tokens for a larger model to check together. The larger model, called the target, still determines the output probabilities. The surprising part is that the small model can guess wrong without changing those probabilities.
In the two-model version we will follow, the target still does its full computation. The saving comes from arranging that computation differently, so one pass through its layers can advance the answer by several tokens. Whether that saves time depends on how much of the draft survives.
