This edition explores how a model can pass validation for the wrong reason, how an agent can earn a reward without completing its task, and what makes a tool retry safe after a crash.
We'll also look inside speculative decoding and text watermarking, then turn to two visual problems: why a generated image can look convincing while missing the instructions, and why a fluent document answer can still come from the wrong cell.
1. Model validation: a random split can preserve the shortcut
Can machine learning validation approve a model that learned the wrong feature? It can, if the validation set repeats the same shortcut!
Imagine a pneumonia classifier trained on chest X-rays from two hospitals. Hospital A contributed most of the positive cases. Hospital B contributed most of the negative cases.
Each image contains two usable signals: the lung pattern we care about, and a hospital fingerprint from scanner processing, text markers, or acquisition style. The loss function does not know which explanation we intended. It rewards features that predict the label. If recognizing the hospital offers an easier route to lower loss, the model can use it.
Now randomly split the rows into training and validation sets. Both sets still contain the same hospitals and the same correlation: Hospital A usually means positive; Hospital B usually means negative. Randomization changes which rows land in each set, but it does not change the process that produced them.
The validation images are unseen, yet the shortcut is familiar. A high score can accurately describe performance on this source mixture and still mislead us about deployment.
Move the model to Hospital C. In this toy example, the disease signal remains valid while the old hospital-label correlation disappears. Prediction quality can drop without any bug in the inference code. The diagram follows that change from collection to deployment.
A random row split estimates performance on another row from the same source mixture. It says little about a source absent from that mixture. When the expected novelty is a hospital, camera, customer, region, time period, or device, holding out the entire unit lets validation test that change instead of scattering the unit across both sides.
I would then ask whether the model is using the particular cue I suspect. Keep the target content fixed, remove or swap the background, marker, source template, or metadata, and watch whether the prediction moves.
These tests answer different questions. A group holdout tests an environment change. A cue intervention tests dependence on one named feature. Neither proves that the model learned the intended signal: the holdout covers only the environments we chose, and the intervention is trustworthy only if it changes the shortcut without changing the task itself.
2. Agent reward hacking: passing the test while leaving the bug
Why can AI agent reward hacking produce a perfect score on a task the agent never completed? The reason is simple: the agent optimizes the feedback we expose, not the intention behind it.
Consider a coding agent asked to fix an invoice calculation. The correct total is $110, but the code returns $100. We use the test suite as the evaluator: a pass earns a reward of +1, and a failure earns 0.
During reinforcement learning, the agent tries a sequence of actions and receives this reward. The optimizer makes high-reward action sequences more likely in similar situations.
We expect the agent to fix the calculation, pass the tests, and produce an invoice for $110. But the test suite is also part of the environment. If the agent can modify it, another path becomes available: disable the assertion, pass the tests, and leave the invoice at $100.
Both paths receive +1. The second path can be reinforced because the remaining failure was never reflected in the score.
Three different objects matter here:
The true objective is a correct invoice.
The proxy reward is the +1.
The evaluator is the test suite that decides when to issue it.
Exploiting a missing condition in the proxy is usually called specification gaming. Changing the evaluator itself, by editing a test, checklist, log, or scoring process, is the more specific case of reward tampering.
No human-like intention to cheat is required. From the optimizer's point of view, the two action sequences received the same reward. The number does not contain the missing requirement, so we cannot recover it from the score afterward.
Greater capability does not settle this problem. Better search can find the correct implementation, but it can also find shortcuts that a weaker agent never discovered. It expands both sets of paths.
A wrong answer with a low score is ordinary failure. Reward hacking is narrower: the measured score says success while the intended outcome stays unchanged or gets worse. The test suite is another piece of the environment, and its assumptions shape what the agent learns.
3. Durable agents: a checkpoint cannot confirm an external payment
How can durable execution for AI agents survive a crash and still charge the same invoice twice? The checkpoint remembers where to resume, but it cannot prove whether the outside world already changed!
Imagine an agent paying a $480 invoice. It records step pay-1 as pending, sends the request, and the payment API accepts it. Then the worker crashes before saving the receipt.
After restart, the execution state still says:
run-42 -> pay-1 -> PENDING
From that saved state alone, the replacement worker cannot tell whether the payment failed or only the acknowledgement failed. We know the API accepted the payment in this example; the worker has no saved receipt to establish that. Retrying may be necessary, but the tool step may execute again.
An idempotency key is a stable identifier that tells the external service that a retry represents the same logical operation. The checkpoint identifies the unfinished step; the key identifies the one external operation that step is allowed to create.
The diagram uses run-42/pay-1 as the key, recorded before the first request. The replacement worker reloads the latest checkpoint, skips completed work, and sends pay-1 again with that same key. If the payment service supports idempotency and already processed the key, it returns the original receipt instead of creating another payment. The runtime then saves the receipt, marks the step completed, and continues.
This is why conversation memory and execution state deserve separate treatment. Conversation memory can hold messages, facts, and user preferences. Execution state needs the run ID, step ID, exact tool arguments, status, approvals, timers, and durable receipts. A sentence saying "the invoice was paid" is not a transaction record.
The same pattern appears outside payments. A database can reject a duplicate operation through a unique constraint or conditional write. An event consumer can deduplicate using a stable event ID.
If a tool has no idempotency mechanism, blind retry is the wrong recovery strategy. The agent has to query the external system, run a compensating action that reverses or offsets the first effect, or stop for human review.
Durable execution preserves progress. It does not make the model correct, permissions safe, or an arbitrary API idempotent. Once an agent can wait for humans, call tools, and run longer than one request, its execution state matters as much as its prompt.
4. Speculative decoding: draft several tokens, verify in one pass
How can speculative decoding use a smaller language model without inheriting its mistakes? The small model guesses a few tokens ahead, but the large target model verifies the whole guess in one pass and keeps control of the output distribution.
The usual language-model decoding loop is serial. The target model predicts one token, appends it to the sequence, then runs again. Producing K tokens requires K target-model passes.
During low-batch decoding, a pass often spends more time moving the model weights than using all the available arithmetic. Scoring a short block can reuse each weight read across several token positions, so it can take close to the time of scoring only one.
Speculative decoding takes advantage of that opportunity. In the common two-model version, a cheap draft model generates a short continuation one token at a time. The large target model receives the prompt and complete draft, then scores every drafted position in parallel.
Acceptance still proceeds from left to right. Accepted tokens become final. At the first rejection, every later draft token is discarded because it was conditioned on a prefix the target did not accept. The target samples a correction, and the next drafting round starts from the corrected prefix.
For the prompt "Water freezes at", imagine the draft proposes:
"0" -> "degrees" -> "Kelvin" -> "."
One target pass can score all four positions. It may accept "0" and "degrees", reject "Kelvin", discard the period, and sample "Celsius" as the correction. That slow pass has finalized three useful tokens instead of one.
What I find most interesting is how this works for sampled decoding. We cannot simply keep a token when two independent samples happen to match. If draft and target were identical uniform distributions over 100 tokens, their separate draws would match only 1% of the time, despite perfect agreement between the distributions.
Call the target distribution p and the draft distribution q. A drafted token x is accepted with probability:
min(1, p(x) / q(x))
If it is rejected, the correction is sampled from the normalized positive part of p - q. To build that distribution, keep the positive differences between target and draft probabilities, set the others to zero, and rescale the remaining values so they sum to one. This is the target probability not already covered by the draft.
Together, the accepted path and correction path reconstruct p. Under this rule, the draft changes latency without biasing the target distribution.
The benefit depends on the whole serving setup. A slow draft adds too much overhead. A draft with little overlap with the target causes frequent rejections. Hardware that is already compute-saturated has little parallel capacity left to exploit.
The useful draft is the one whose acceptance rate is worth its cost on the target model and hardware being served. Accuracy of the small model alone does not determine that tradeoff.
5. Text watermarking: small token preferences add up to evidence
How can generative text watermarking mark a paragraph without adding hidden characters, metadata, or a visible tag? It changes ordinary token choices, then looks for the accumulated pattern!
At each decoding step, a large language model (LLM) produces a probability distribution over the next token. Several continuations may fit. A normal sampler chooses among them; a watermarking sampler uses one additional input, a secret key.
One common construction uses the key and recent tokens to pseudorandomly divide the vocabulary into green and red sets. Plausible green tokens get a small boost. The partition is recomputed after every token, so there is no fixed list of suspicious words.
Consider a toy passage with 100 positions counted by the detector. Without the watermark, we expect roughly 50 green hits by chance. A watermarked sampler might produce 70. No individual token looks unusual; the pattern appears when we aggregate the choices.
For this simplified 50/50 construction, a one-proportion test gives:
z = (70 - 50) / sqrt(100 × 0.5 × 0.5) = 4
The numerator is the 20 extra green hits. The denominator is 5, the standard deviation of the count under this simplified chance model. The observed count sits four standard deviations above the expected count.
Other watermark families use different scoring rules. Here, the detector tokenizes the passage, uses the same key to recreate the green and red sets at each position, and tests whether the excess of green tokens is too large to explain by chance. It measures a statistical pattern rather than judging whether the prose sounds robotic.
Length supplies evidence. A ten-token answer provides very few weak votes; a long passage supplies many more observations. The model also needs freedom to choose. Open prose may offer several reasonable continuations, while code or familiar fixed phrases can have one dominant next token, leaving little room to embed a preference without damaging the output.
Editing weakens the signal without predictably erasing it. Replacing a token removes one marked choice. In context-dependent schemes, it can also change the expected partition at positions that use the edited context. A substantial rewrite can reduce the evidence dramatically, yet a long paraphrase may preserve enough unaltered fragments to remain detectable.
A positive result means the text is statistically consistent with one key's watermarking rule. It does not prove who wrote the ideas, and it cannot identify output from a model that never applied that mark. The detector threshold still trades missed detections against false positives.
That is why a long, open-ended answer can carry strong watermark evidence while a short code completion from the same model may carry almost none.
6. Image generation: realism and exact composition are different tests
AI image generation can give you a beautiful campaign image and still ignore the requirement that makes it usable! The problem is that a text prompt is not an engineering drawing.
Consider this request: "three green bottles to the left of one orange box, with FRESH on the label."
One sentence carries five kinds of constraints: which objects exist, how many of each, which attributes belong to which objects, where they go, and which exact characters appear.
In many diffusion systems, a text encoder converts the prompt into vectors. Cross-attention lets the denoising network consult those vectors while turning noise into an image. A few words can influence a large visual field without someone hand-coding the scene.
But the vectors do not arrive as an explicit object table with separate entries for color, count, and coordinates. During denoising, "green", "bottles", "three", and "left of" must all influence one shared visual representation. The model can form a convincing poster while giving you four bottles, coloring the box green, or letting objects overlap.
Realism and prompt compliance measure different things. Bottles, an orange box, and text-like marks can all be present while the requested relationships drift. Counting, relative position, and attaching an attribute to the correct object are compositional challenges.
Saying that image models "cannot count" misses the larger problem. Individually recognizable concepts have to stay attached to the right instance and location throughout generation.
Sketches, layouts, segmentation maps, and image checks help make some intended state explicit: where an object belongs, how many are needed, and whether a word is exact. The diagram separates guidance from checking: its layout specifies the bottles' positions, while the checks inspect count, position, and the characters in FRESH. These aids do not guarantee a correct result, but they stop the prompt from carrying every constraint alone.
A creative mood board benefits from the model filling gaps. An asset with exact copy, object counts, or placement can fail precisely when the model fills in a gap you intended to specify.
7. Document extraction: connect the answer to the right cell
AI document extraction can give you a perfectly grammatical answer that comes from the wrong cell in the document! The reason is that a vision-language model is doing several different jobs before it ever starts reasoning about your PDF.
Start with a scanned bank statement:
Opening balance: $14,900
Deposits: $2,200
Closing balance: $17,100
Ask, "How much was deposited?"
The model does not receive the document as a clean database row. A vision encoder turns pixels into a limited set of visual tokens. Small type, a low-quality scan, and dense tables can make the evidence for a word or nearby number weak or mixed. Some PDF pipelines pass native text separately, which changes the failure mode, but charts, tables, handwriting, and rendered layout still require visual interpretation.
Then comes binding: "Deposits" must be linked to $2,200. Recognizing all three amounts somewhere on the page is insufficient. The language model uses the visual representations and the question to generate its answer token by token.
If the deposit label was unclear, or the label-to-value association was wrong, "$17,100" can still be a plausible completion. Fluency tells us the decoder completed a sentence well. It does not tell us which pixels supported the answer.
I separate document extraction into three questions:
Did the system read the relevant glyphs?
Did it preserve the relationship between label, row, column, and value?
Did the answer come from that evidence, or from a plausible completion?
Optical character recognition (OCR) and coordinates can make the first two steps inspectable. A vision-language model is useful when the final answer needs a comparison, interpretation, or calculation.
More visual resolution can improve fine detail, but it also costs tokens and latency. It does not automatically recover a missing label-to-value relationship.
Chart questions have the same difficulty. Reading "Q4" and "47" is insufficient: the system has to connect the right series, axis, and visual mark before it can reason usefully about them.
Happy to see you back :)