Reinforcement learning trains an LLM by scoring answers it generates and using those scores to change its weights. The model samples a response, a checker or reward model evaluates it, and training adjusts the probabilities of the tokens that produced it. Those changes remain when the model answers another question. Selecting the best of several answers at inference time does not, by itself, perform this training.
Ahead of AI's recent reasoning-model course includes implementing GRPO, making the path from answer scores to weight updates a concrete part of learning how reasoning models are trained.
How can one score at the end of an answer tell the model how to change thousands of token probabilities?

The useful starting point is reward relative to an expectation. A training algorithm constructs an advantage: a signed estimate of how much better or worse sampled behavior was than a baseline. Positive advantage encourages that behavior; negative advantage discourages it. The difficult choice is what evidence supplies the baseline and how strongly to weight the difference.
We will follow three policy-gradient methods, which train the model through the probabilities of its sampled actions:
PPO, Proximal Policy Optimization: in its common actor–critic form, a learned value model predicts the future reward available from each unfinished response. The training signal compares observed progress with those predictions.
RLOO, REINFORCE Leave-One-Out: other independently sampled answers to the same prompt provide the baseline. There is no learned value model to fit.
GRPO, Group Relative Policy Optimization: answers are compared with their group mean, and the differences are divided by the group's reward spread.
This organizing axis is the source and scale of the advantage. PPO also names an update-control rule, clipping, that can be combined with group-based advantages. These are partly interchangeable components, not three sealed systems.
One immediate consequence: an answer with reward zero can still produce a useful negative training signal if the baseline is positive. Conversely, a successful answer need not receive a large positive signal when success was already strongly expected. The score describes the outcome; the advantage determines its relative push on the model. Neither tells us that every token in a successful response was necessary or correct.
Contents
From reward to a weight update
PPO learns a value for each prefix
RLOO compares separate attempts
GRPO standardizes the group
Control how far an update moves
Choose the evidence your training needs



