Ahead of AI's recent reasoning-model course teaches distillation as a route to smaller reasoning models, making the transfer of useful behavior into a cheaper model a concrete training choice.

How does that transfer happen? Through three decisions: what the teacher supplies, where the student practices, and which differences the training loss penalizes. Those decisions determine what the student has an opportunity to learn. They do not guarantee that it acquires everything the teacher can do.

A useful distinction comes before any equations. Distillation changes a model during training. It does not normally keep a larger model beside it to check every answer during deployment. In speculative decoding, a small model proposes tokens and a larger model verifies them. After distillation, the smaller model's own predictions determine its output. Removing the teacher from that loop creates the potential saving, and also makes the student's independent quality the relevant test.

Nor does distillation automatically make a model smaller. We choose a student architecture, usually start from an already pretrained checkpoint, and train it to reproduce selected behavior. A smaller student might learn a specialized task well while losing some of the teacher's breadth. Distillation can also use a student of the same size; here we will focus on transferring behavior into a smaller model.

We will follow the training signal from recorded answers to probability targets, then change who produces the prefixes used for training. That will let us explain why a teacher can provide useful feedback on student mistakes without becoming a correctness oracle.

Contents

• What moves from teacher to student

• Learn from a teacher's answers

• Learn from a teacher's probabilities

• Train where the student actually goes

• Choose what the student should match

• Check what transferred

Read more

Serving engines such as vLLM support several quantization methods with different hardware requirements, making a smaller checkpoint a choice about execution as well as storage.

How can we remove precision without removing the computation that makes the model useful?

The central idea is to preserve what the numbers do together. A weight contributes to an output by multiplying an activation. A small weight error can matter greatly when its input is large. Errors in different weights can also reinforce or cancel one another.

We will follow three members of the post-training quantization family, organized by where they absorb that error:

• GPTQ rounds a weight, then adjusts weights that have not yet been rounded to compensate for its effect.

• AWQ, activation-aware weight quantization, rescales input channels so consequential weights get a more useful rounding grid.

• SmoothQuant redistributes large activation values into the weights, making it easier to quantize both operands of a matrix multiplication.

These choices are related, but they are not three interchangeable file formats. GPTQ and AWQ are commonly used for weight-only compression. SmoothQuant addresses the additional difficulty of quantizing activations, and its transformation can be followed by GPTQ.

Here is the first useful distinction: the largest weight is not automatically the most important weight to preserve. Its effect depends on the input it multiplies and the other contributions to the output. That is why a conversion method may run calibration text through a trained model before deciding how to round its weights. It needs examples of the computation it is trying to preserve.

Table of contents

• What rounding changes

• GPTQ compensates across weights

• AWQ changes the effective grid

• SmoothQuant balances two operands

• From compressed weights to execution

Read more

Serving engines such as vLLM expose full and piecewise graph modes, making replay coverage a practical choice about latency and GPU memory.

How can a reusable plan handle an LLM whose tokens, context lengths, and active requests keep changing?

The useful distinction is between the description of the work and the data that work reads. A captured operation can keep the same input address while the values stored at that address change. The GPU then follows the prepared dependencies and computes a fresh result.

A fixed execution plan does not require a fixed answer. An operation that doubles the numbers in an input buffer produces a different output after those numbers change, even though the operation and its input address are unchanged. Replacing the buffer itself is a different matter: the prepared operation may still point to the original storage.

LLM serving builds on that distinction. New tokens, positions, and cache locations can become updated data inside persistent buffers. Changes to the amount or arrangement of work may instead require another prepared graph, a smaller captured region, or ordinary execution.

We will follow that boundary from GPU launches to growing token histories, then examine when the cost of keeping several graph variants ready outweighs the launch savings.

Table of contents

• Where a decoding step spends time

• Capture a plan, replay fresh work

• Keep addresses stable while values change

• Fit changing LLM state into a replay

• Choose the right graph for the batch

• Decide whether replay pays

Read more

Cohere's release of Rerank 5 for retrieval-based answers and agents makes this a concrete application choice: a pipeline can spend additional computation deciding which evidence reaches the answering model.

How does that extra computation produce a better order? The main distinction is what the model judges together:

• Pointwise: read the question with one passage, produce a score, then sort the scores.

• Pairwise: read the question with two passages, choose the better one, then assemble those preferences into an order.

• Listwise: read the question with a group of passages and return their order, sometimes one overlapping group at a time.

These methods buy different kinds of judgment. Independent scores can be computed in parallel. Comparisons let the model weigh competing evidence directly, but the application must decide how to reconcile those judgments. A listwise answer orders several candidates at once, at the cost of a longer input and a more demanding output.

The names describe the ranking procedure. The underlying model architecture and training objective are separate choices. We will separate those dimensions, follow the numbers and comparisons into an actual ordering, and see where the methods can work together.

Contents

• What a reranker changes

• Pointwise: score each passage

• Pairwise: compare two passages

• Listwise: order a group

• Choose the work your query needs

There is one useful boundary to establish immediately. A reranker cannot promote a passage that never entered its shortlist. If the answer is present but buried, reordering may help. If the necessary evidence is absent, the pipeline needs another search or a different way to retrieve it. This distinction tells you where to investigate a bad answer before choosing a more expensive ranking model.

Read more

Open-weight models such as Huginn expose the number of recurrent passes as an inference setting, making model size and computation per token separate design choices.

How can applying the same weights again produce useful new work?

The answer is in what gets passed around the loop. Restart a deterministic computation from the same state and it returns the same result. Feed its result into the next pass and the computation can advance. A query can change after retrieving a fact, an intermediate result can become the input to another operation, or a partial representation can be refined.

This gives us a useful first distinction. Suppose a core contains two layers, A and B, and we run it three times. The execution is A, B, A, B, A, B. We store two sets of layer weights but perform six layer applications. The copies in an unrolled drawing represent uses of the same parameters.

That can provide more computation without storing six independent transformations. It does not make the extra applications free, and it does not establish that the sixth application produces a better answer. The state must contain something useful for the next pass, training must encourage useful repeated transformations, and the model needs a sensible place to stop.

We will follow those dependencies from attention and changing hidden states through training, halting, and memory. Along the way, a small constructed attention block will make the intermediate operations visible. Its calculations illustrate what shared weights can do; they are not a claim about the internal variables of a trained language model.

Table of contents

• What repeats inside the model

• Why another pass can change the answer

• Training the shared computation

• Deciding when to stop

• What looping costs

• Depth and reasoning tokens are different budgets

Read more

DeepSeek explicitly connects cache-hit charges to a large share of its agent workloads' costs, making the storage and reuse of that state a product-cost concern as well as a serving optimization. DeepSeek's release explanation

Consider the shape of the work. A short chat might contain 500 input tokens and produce 500 output tokens. A coding agent can accumulate system instructions, repository files, earlier decisions, dozens of tool calls, and long tool results. It may then read 200,000 tokens to produce just 200 more before calling another tool.

Must the next call rebuild everything the model already computed?

It need not. But preserving the work creates a second question: where is the saved state, and how much of it must move before the agent can continue?

Three different savings sit behind that question. A smaller representation reduces the bytes needed for each remembered position. Prefix caching reuses computation from an earlier call. Keeping the reusable state near the executing GPU avoids restoring it across a slower connection. These savings interact, but they are not interchangeable.

Here is the useful starting distinction: a cache hit can eliminate the work of rebuilding an old token without eliminating the work of consulting it. A newly arrived tool result still needs to interact with the previous context. Every new generated token may consult that context again. A cheap input-cache hit therefore does not imply cheap generation from a long history.

We'll follow those saved states from attention through head sharing, latent compression, paging, persistent storage, and separate prefill and decode workers. Then we'll account for a 50-turn agent and use a separate timing example to decide when retaining, restoring, or rebuilding state pays off.

Table of contents

• The work a KV cache saves

• When bytes limit decoding

• Shrink the state per token

• Reuse the prefix without copying it

• Keep, move, or recompute

• Account for a 50-turn agent

• Choose a retention policy

• Design models around the cache

Read more

Released architectures from Qwen, Moonshot AI and NVIDIA now combine these mechanisms, making the meaning of “model size” depend on both how history is stored and which parameters actually execute.

Why combine them? They separate three resources: memory of the sequence, access to individual positions, and capacity to transform a token. Improving one resource does not automatically improve the others.

The recurrent sequence mixers belong to a recognizable family, but their writes differ:

• Mamba-2 selectively fades a running state and adds new information.

• Gated DeltaNet also checks what its memory predicts, then writes a correction.

• Kimi Delta Attention, or KDA, adds finer control over which state coordinates fade.

Retained attention gives these compact memories a complementary path through position-indexed history. Mixture of experts, or MoE, changes a different component: the feedforward computation applied to each token representation.

That distinction already lets you read a hybrid architecture more accurately. In Qwen3-Next, a hybrid language model with a 3:1 mixer design, three recurrent mixers are followed by an attention mixer along the model's depth. Every token runs through that configured sequence. The model does not wait three generated tokens before using attention, and it does not decide at runtime whether a particular token deserves an attention layer.

The expert router does make a per-token choice. Mixer placement is fixed; expert selection depends on the current representation. Calling both “conditional computation” without this qualification hides how the model actually runs.

We will follow those two design decisions from the original decoder block to the memory updates, then put them back together. The payoff is practical: given another hybrid model, you should be able to explain what grows with context, what remains fixed, and why its active parameter count cannot tell you its serving cost.

Contents

• Two jobs inside a Transformer

• Mamba-2 keeps a running state

• Delta memory corrects what it stores

• Why hybrids retain attention

• Sparse experts change the other half

• When the combination is better

Read more

OpenAI's managed Agents API now exposes durable sessions, context compaction, recovery, and separate execution environments, making the boundary between a saved conversation and a recoverable task a concrete platform decision. Agents API overview.

What actually allows an agent to continue tomorrow?

The familiar agent loop is straightforward: receive a goal, ask the LLM for an action, execute a tool, return the observation, and repeat. A short demonstration can keep almost everything in one process. Stretch that task across several hours and the assumptions change. The agent may edit files, wait for approval, lose a network response, and outlive its container. Its next prompt cannot reconstruct facts that were never durably recorded.

The useful mental model is a distributed system with a probabilistic planner inside it. The model proposes what to do. The surrounding runtime records accepted decisions, preserves evidence, executes actions, and determines what can safely happen next.

One consequence is immediately useful: a missing result is different from a failed action. An external service may have completed the request before the worker disappeared. A saved task record saying “no result recorded” tells us about the worker's knowledge. It does not tell us whether the world changed. Recovering the job means closing that gap, not asking the model to guess.

We will separate the kinds of state, reconstruct a bounded context, and work through the commit boundaries. Then we will build a local runtime, continue a code repair in a fresh context, and terminate a process between an external effect and its saved acknowledgement.

Table of contents

• Separate the state that must survive

• Rebuild context from durable evidence

• Recover execution across a crash

• Resume workspaces, approvals, and delegated work

• Connect the complete runtime

• Build the durable runtime

• Continue a coding task across context windows

• Break it and compare recovery strategies

Read more

OpenAI’s Tax AI deployment turns practitioner corrections into evaluated product changes, making the quality of this feedback loop a concrete engineering problem.

The interesting question is how to turn those failures into useful practice, then retain what helps on future tasks.

The answer has two paths. One repairs the agent system: its code, instructions, tools, retrieval, or stored skills. The other improves the model using selected demonstrations or rewards from fresh attempts. Both need a way to decide whether a proposed change actually helps.

That is where environments become important. An environment supplies a task, a starting state, tools that change that state, and a way to evaluate the consequences. It can produce new experience when the agent tries a different action. A saved conversation contains only the experience that already happened.

This distinction changes what we should scale. More failure logs can reveal recurring defects. More executable environments can let an agent practice different decisions. Neither produces improvement until an update is made and tested on future behavior.

A useful first check is to ask what survives the run. A reflection left in a discarded conversation changes nothing for the next user. A saved skill changes future behavior only if another run retrieves and executes it. A newly trained checkpoint changes nothing until the application serves it. Persistence needs an actual connection to the next execution.

We will follow that connection through traces, three ways to construct executable environments, training, curriculum, and deployment.

Contents

• What persists after a run

• Turn traces into testable failures

• An environment produces new experience

• How executable environments are built

• Check what the environment rewards

• Convert experience into an agent update

• Build a curriculum from agent outcomes

• Promote improvements on independent evidence

Read more

DeepSeek's technical report motivates this design with agents that repeatedly receive long tool results and accumulated history, making the cost of reading and retaining that information a major part of running the model. Why should preparing old text for later use require the same computation as deciding what to write next?

The design divides 40 layers into two halves. The first 20 prepare reusable memory of the input. Most input positions can stop there; a final slice of up to 128 positions also passes through the upper half to prepare its recent context. Once generation begins, each newly chosen token goes through both halves to predict the following token. The architecture is called a causal encoder-decoder, or CED. We will unpack those roles, then trace how sharing memory, selecting what to read, storing fewer bits, and rebuilding recent state change different costs.

The useful distinction is already visible: preparing a record of earlier text and using that record to choose a new token need not follow identical paths. The shorter input route matters most when there is much more new text to read than text to generate.

Table of contents

• Why one causal stack reads and writes

• Why agent workloads change the budget

• Prefill, decode, and the cache between them

• How CED changes the old-token dependency

• Why generated tokens still use the full model

• Why this is a different encoder-decoder

• How CSA2 shares global history

• Why sparse search needs a hierarchy

• How the cache bytes add up

• What bounded replay rebuilds

• Putting the architecture together

• What the quality evidence establishes

• What this means for agent architectures

Read more

Here is the agent we will build: a generative model plans and writes; Jev chooses among named next actions using the current state; Python checks hard rules, executes a local tool, and records what actually happened. The central question is where a fast decision belongs when the task still needs open-ended reasoning and real effects. The short answer is between observations and permitted transitions. Jev can judge the state, but its choice cannot itself refund an account or prove that a refund settled.

We will make one Jev call first, then connect it to a small support agent. Then we will compare Jev with two structured-output LLMs using our experiments on support and document-research tasks. The experiment is useful because it records both proposed actions and actual local effects.

In this guide

• What Jev decides

• Give the agent observable state

• Build the guarded loop

• Run support and research tasks

• Compare three supervisors

• Read probabilities and disagreements

• Decide where Jev belongs

One distinction will keep the rest of the build honest. A tool returning HTTP 202 says a request was accepted for processing. It does not say the requested business outcome happened. That gap is exactly where a supervisor needs evidence, and where code must resist an unsafe retry.

Read more

The short answer is that Jev moves the boundary. A conventional language model can certainly emit {"route":"billing"}; strict structured output can make that object parse. Jev instead accepts the answer space as part of the question and returns the chosen answer and its distribution as the native result. Several questions can refer to one supplied state. That makes semantic judgments easier to place inside software, but it does not make their probabilities self-validating or authorize any action.

This is a model category worth understanding even if a particular workload ends up using rules, a fine-tuned classifier, or a constrained LLM. We will follow the interfaces first, then the computation they may avoid, the architecture that is and is not disclosed, and finally the policy that turns a judgment into a safe system decision.

In this deep dive

1. What Jev receives and returns

2. Why a separate decision model might help

3. What can be inferred about Jev's architecture

4. Calibration, RLCD, and action thresholds

5. Where Jev fits among existing tools

6. A system with generation and decision separated

One useful rule already follows from the interface: a model's answer should be treated as evidence about a defined outcome, while the application's code retains the choice of action. The rest of the article explains exactly what must be defined, computed, and checked before that rule is useful.

Read more

Teams are already using production conversations and human labels to build agent evaluations, as Dropbox's engineering account illustrates; the durable problem is deciding whether an agent actually completed an authorized task. How do we turn that question into a pipeline that can test a new agent version before release and keep learning afterward?

The compact answer is a loop: observed request → reviewed case → isolated trial → trace and final state → graders → version comparison → release decision → production feedback. A case describes the starting world and acceptable outcomes. A trial runs one agent attempt against that world. Graders inspect independent evidence, and the comparison asks whether the new version improves the tasks that matter without breaking important slices. The loop is reusable; the meaning of success changes with the agent's job.

The first useful rule is to give a completed action its own proof. If a support agent says “refunded,” a ledger record or service commit is the proof, subject to the service's consistency contract. If the record is absent, the polished sentence cannot rescue the task. That single separation prevents an answer-only judge from turning a failed action into a pass.

The second rule is to test the task from a known starting point. The same request can require a refund when the customer verifies identity, a clarifying question when they cannot, or a refusal when policy forbids it. An evaluation case must carry those starting facts and accepted branches. Otherwise a grader may punish a careful agent for asking or reward a reckless one for acting.

Before trusting a score, ask what its grader can observe. If it sees only messages, the score measures answer quality. If it sees tool calls but no final state, it can verify attempted actions without proving durable effects. Reserve “task success” for a test whose evidence covers the task's actual outcome. This distinction is useful even before the full pipeline exists.

Table of contents

1. Define what success must prove

2. Turn production evidence into cases

3. Run isolated, observable trials

4. Grade the evidence

5. Compare agent versions fairly

6. Operate the pipeline and close the loop

Read more

Persistent agent memory tools now let applications carry stored information across sessions and update it, making the quality of that update consequential for later tasks. The question is: when a user changes a standing preference, what must happen so an agent acts on the new one? A useful memory system treats the new message as a proposed change to stored state. It identifies the fact, finds the existing fact about the same thing, reconciles the two, commits the active version, and retrieves that version when the next task needs it.

That differs from appending messages to a transcript. We can see why by following one preference from its first instruction through a correction and a later task. The same failure can affect a project decision or customer preference.

What changes when an agent has memory?

The model receives a limited working context with each request. A conversation transcript can be placed in that context, but it grows, contains incidental remarks, and may contain claims that no longer hold. Persistent memory here means an external store that keeps selected information beyond one request or session. A write operation changes that store; a later read returns a useful part of it to the model. The model's weights are not being retrained every time a preference changes.

Suppose a user tells a coding assistant, “For the Atlas repository, write deployment scripts in Python.” The assistant might store a compact record with four parts: the user, the Atlas repository, the subject deployment-script language, and the current value Python. The message that supplied the preference is its source. This is more useful than an unqualified memory such as “prefers Python,” because the latter could affect unrelated work.

The store has two distinct paths. On the write path, a message or tool result is examined for information worth retaining, then compared with what is already stored. On the read path, the next task retrieves applicable memory and places it where the model can use it. If the write path is wrong, fast retrieval brings back a mistake. If the read path misses the right record, a perfect update sits unused.

Now the user says, “For Atlas, use Go for deployment scripts from now on.” Adding that sentence as a second, equally active memory would leave the future assistant to arbitrate a contradiction each time. A better result is an active Atlas deployment-language preference of Go, with the former Python preference no longer active. The old statement may remain in an audit history, but it should not be presented as another current instruction.

Remembering a correction means changing the state that future reads see, not merely remembering that a correction was said. The change has to preserve the subject and scope of the fact, or it will solve one task by creating a wider error.

Read more

Prefix-aware routing became a managed endpoint option in September 2026, a sign that cache locality has moved from an engine detail into a deployment decision.

So where should the next request go: to the warm replica or the idle one? The compact answer is to send it where the saved prompt computation is worth more than the extra work already waiting there. A useful router therefore needs two views at once: what can be reused and what must still be processed.

To see why, start with the work a cached prefix actually removes. Then we will keep one support prompt fixed while four routing policies make different placement decisions. That comparison will expose the useful rule, its failure modes, and the state a production router has to keep current.

What a warm prefix actually saves

Before routing makes sense, we need one piece of transformer serving machinery.

An incoming prompt is a sequence of tokens. During prefill, the model processes those input tokens through every transformer layer. In each attention layer it derives a key vector and a value vector for every position. Those vectors are the processed state that later tokens attend to. The first generated token can appear only after this prompt work has finished.

During decode, the model generates one new token, appends its new key and value state, and repeats. Decode can reuse the state for all earlier positions instead of rebuilding it. The stored collection is called the KV cache.

Now consider a support assistant whose prompt begins with 3,200 tokens of policies, tool descriptions, and response rules. Only the short customer question at the end changes. If the engine has already processed the same beginning, it can reuse the matching KV state and prefill only the new suffix.

Prefix caching does not mean that the model remembers a similar idea. The match is structural. Serving engines divide the prompt into token blocks, identify a block using its tokens and the blocks before it, and reuse only an unbroken matching prefix. If block three changes, later blocks no longer describe the same model state, even if much of their text looks familiar.

That distinction gives prefix caching an honest boundary. It skips model computation already performed for the input prefix. It does not skip the generation of new output tokens, because those tokens do not exist yet. A workload with long repeated inputs and short answers can gain a lot; a workload with unrelated prompts or very long answers may gain little.

Within one replica, the rule is simple: reuse every valid prefix block that is still resident, prefill the remaining input, then decode normally.

Read more

That distinction is now practical because current OpenAI and Anthropic APIs expose reasoning-effort controls while Google offers an Extended Thinking model for complex live-agent tasks, turning “let it think” into a quality, latency, and cost decision for each request.

The useful question is not simply, “How much thinking should I buy?” It is: where should the extra compute go? Test-time scaling allocates it to depth, width, or feedback. Depth extends one attempt. Width creates separate attempts. Feedback evaluates work and changes the next attempt. Each repairs a different failure, and none guarantees correctness.

What test-time compute actually is

Start with an ordinary request. Training is already over, and the model begins with learned weights that stay fixed while it answers. At each step, the model processes the request and the tokens generated so far to choose the next token. That ordinary generation already consumes test-time compute: computation performed after training for one particular input.

The term covers more than the model's hidden reasoning tokens. It includes the work the model and its surrounding system perform to produce, inspect, and improve this one answer. Continuing the generation adds more token-level operations to the current history. Starting another generation creates a separate candidate history. Running a test or verifier creates an observation that a later attempt can use. Revising with that observation performs another generation under changed conditions.

Test-time scaling is the deliberate decision to increase or reallocate that per-request work relative to a baseline answer. The extra work can matter because it changes the transient information available to a later choice: a longer prefix, a genuinely different candidate, or new evidence about an error. If the added work does not change that state or evidence, it may increase latency and cost without improving the answer.

This is not training or fine-tuning. Training changes reusable model weights for future requests. Test-time compute leaves those weights alone and is paid again for each request. It is also not synonymous with selecting a larger model or producing a longer visible answer. A system may spend substantial compute on candidates, checks, and revisions that never appear in the final response.

We will follow one realistic coding task through all three choices. An agent must repair invoice_total, a function that sums prices, applies tax, and returns integer cents. Its contract requires exact decimal arithmetic, an 8 percent tax applied after aggregation, round-half-up once to the nearest cent, then conversion to integer cents. The visible invoice contains items priced at $12.49, $7.25, and $3.10. Hidden tests vary the inputs to expose half-cent rounding, multiple-item accumulation, zero tax, and binary floating-point drift.

First we need to see why one uninterrupted generation can fail. Then we can spend the same request-time budget on a deeper path, a wider candidate set, or a feedback loop and see what actually changes.

Before scaling, identify the one-shot failure

An autoregressive language model generates a response one token at a time. At every position it receives the request plus the tokens already written, then assigns probabilities to possible next tokens. Once a token is chosen, that token becomes part of the state used to predict everything that follows.

This left-to-right process matters because an early choice can narrow the rest of the answer. Suppose our coding agent begins with the assumption that converting dollars to cents is just int(total * 100). It can write an elegant explanation and many consistent lines of code after that. The later tokens are coherent with the earlier assumption, but coherence does not repair the assumption.

A one-shot attempt is one uninterrupted generation from request to answer. It may spend many internal reasoning tokens, but it has no separate alternative to compare with and no external result that changes its course. Its success depends on the model entering a useful line of reasoning and staying there.

For this lesson, a useful accounting unit is a generated reasoning token, a candidate response, a verifier evaluation, or a tool call such as a test run. These units have different hardware costs, so we will not pretend they are perfectly interchangeable. The important constraint is that the system has a finite request-time budget and must decide how to allocate it.

There are three places to allocate that budget:

1. Depth: continue or enrich the same reasoning trajectory.

2. Width: sample several separate trajectories from the same starting request.

3. Feedback: inspect work, then condition the next attempt on what the inspection revealed.

The distinction is about information flow. Extra depth sees the same request and its own growing prefix. Width gives each attempt the request but not the siblings' work. Feedback adds a new observation, such as “the half-cent test failed,” before the next generation. Width is logical branching; whether those branches run simultaneously is a separate serving decision.

That gives us a diagnostic before we spend anything. If the model merely stops before completing a sound plan, buy depth. If different plausible starts lead to different outcomes, buy width. If a checker can reveal what went wrong, buy feedback. A high effort setting may combine these internally, but an application designer still needs this mental model to understand latency, cost, and failure.

Depth gives one trajectory room to finish

The simplest scaling strategy is to let one attempt run longer. The model can decompose the task, inspect edge cases, and carry intermediate facts farther before committing to the final response. In our invoice task, a deeper trajectory must preserve a specific dependency: represent the prices exactly, sum them to $22.84, apply the 8 percent tax to obtain $24.6672, round that total once to $24.67, then return 2467 cents.

The benefit is continuity. Every later step sees the reasoning accumulated earlier. Here, the final integer conversion only makes sense after the representation and rounding policy are settled. Once an earlier step truncates a fraction of a cent, no later step can reconstruct it. Proofs, long calculations, and code changes with dependent invariants have the same shape.

But depth is not the same as recovery. If the trajectory commits early to binary floating point and treats a failing cent as harmless noise, more tokens can reinforce the wrong frame. The model may repeatedly check arithmetic inside a mistaken representation. It is thinking longer, but within a shrinking part of the solution space.

That gives us the first useful rule: depth helps when a sound path is unfinished, not merely when a path is wrong. A model that often reaches the correct approach and then runs out of room is a good candidate for higher effort. A model that confidently commits to a bad representation needs an alternative or new evidence, not an even longer monologue.

Read more

Normally, a language model can choose any token in its vocabulary at each generation step. Constrained decoding inserts a rule checker into that loop. The checker looks at the text already generated and removes every next token that would make the requested structure impossible to complete. The model then chooses among what remains.

Structured output has become a common connection layer between LLMs, extraction pipelines, and agent tools, yet a recent developer discussion exposed the practical surprise that matters here: a perfectly shaped object can still contain invented values, so schema-valid is not the same as safe to execute.

The central question is simple: what does the decoder actually guarantee? The compact answer is that it can guarantee membership in a supported output language, such as a JSON Schema, by blocking illegal continuations token by token. It does not prove that a legal value came from the input, reflects reality, satisfies a business rule, or deserves permission to trigger an action.

We will follow one support request through the complete mechanism. The request says: Please refund order A-17. The charge was $42.50. The desired output is a small object containing a status, an order identifier, and a refund amount in cents. First we need to separate three ideas that are often collapsed into the word “valid.” Then we can compile the schema, watch the token mask change, and see exactly where truth leaves the decoder's jurisdiction.

Three different promises hide inside “valid JSON”

JSON is a text format. A JSON parser checks punctuation and nesting: braces balance, keys are quoted, commas occur in legal places, and values have valid JSON forms. This object is valid JSON:

{"status":"ready","order_id":"A-17","refund_cents":4500}

A schema asks a narrower question about the parsed value. It can require an object, name its allowed properties, require selected keys, restrict a field to an integer, or limit a value to an enumeration. If our schema allows any non-negative integer for refund_cents, then 4500 passes. The schema has no memory of the user's $42.50.

Finally, the application asks whether the object means something acceptable in its real environment. Does order A-17 exist? Was the charge really $42.50? Is it refundable? Is this user authorized? Those are evidence, policy, and permission questions. They are not punctuation questions.

The checks therefore form a hierarchy:

1. Parseable means the text is legal JSON.

2. Schema-valid means the parsed value also obeys the declared structural rules.

3. Grounded and actionable means the values are supported by evidence and pass application policy.

Each level can remove failures that the previous level accepts. None can be silently substituted for the next.

Our refund extractor also needs a way to say that the message is incomplete. If a successful decode must complete a contract where every field is required and refund_cents must be an integer, every successful path puts some integer there even when no amount appears in the input. A better output contract exposes two branches:

{ "anyOf": [ { "status": "ready", "order_id": "string", "refund_cents": "non-negative integer" }, { "status": "insufficient_info", "order_id": null, "refund_cents": null } ] }

This is a readable sketch of the contract, not a literal complete JSON Schema. The important move is structural: uncertainty is now a legal output rather than an instruction the model must squeeze into fields that forbid it.

Here is the first design rule worth keeping: a schema should represent the failure states your application expects. Constrained decoding can only choose paths you make legal. If the contract requires a confident answer, the decoder cannot invent an abstention branch on your behalf.

Read more

That first decision is the hidden trade-off behind sparse attention. The name suggests that the model merely turns down unimportant history. In many designs, it does something more consequential: it prevents most of the history from entering the main attention calculation at all.

A late-August 2026 Transformers release added Qwen Sparse Attention, while an open-weight Qwen model built on the same block-selection idea prompted immediate questions about long-context latency and cache placement; as more models save work this way, the durable question is which evidence they stop reading.

So here is the question we will answer: When sparse attention makes a long context cheaper, what does an LLM stop looking at?

The compact answer is that every attention pattern creates an eligible set of earlier token states. Full attention admits the whole past. A sliding window admits only nearby history. Content-selected sparse attention uses a cheaper scoring step to choose a small subset that can include distant history, then runs ordinary attention on that subset. Fewer eligible states mean less work, but a clue excluded at this point cannot be rescued by a larger attention weight later.

We will follow one question through all three patterns. A long incident report contains the deployed database proxy setting, port 6432, near the beginning. Much later, a diagnostic note mentions the default database port, 5432. At the end, the reader asks: Which port did the incident's database proxy actually use?

This is a useful test because the answer is a tiny exact value separated from the question by a lot of plausible noise. We will hold the tokens, model layer, and question fixed. Only the rule that exposes earlier states to attention will change.

Attention makes two decisions, not one

An LLM processes text as tokens, which are words or pieces of words converted into numerical states. At one layer, every token state is transformed into three useful roles.

Every token produces all three. When the layer updates the current token, its query represents what information it is seeking. The current token and earlier tokens provide keys, which can be compared with that query, and values, the information that can be carried forward. The query-key comparisons become weights; the weighted values become the context added to the current token's state. The diagram focuses on earlier sources, but causal attention also allows the current token to attend to itself.

There is a prerequisite hiding before those comparisons. An attention rule supplies a mask or selection pattern that says which keys and values are eligible. A future token is ineligible in a causal language model because generation cannot look ahead. Sparse patterns add more exclusions among the past.

It helps to separate two questions:

1. Eligibility: may the current token connect to this earlier state?

2. Importance: among eligible states, how much weight should this one receive?

Full, windowed, and content-selected attention mainly differ on the first question. Once a subset is eligible, the main attention operation can still compare queries with keys and mix values in the familiar way.

Apply that distinction to our incident report. Full attention places both 6432 and 5432 in the candidate set. A four-block sliding window includes only the late diagnostic note and loses the deployed setting. A good content selector reaches back to the configuration block containing 6432, while ignoring most of the middle.

The model still has to interpret the candidates correctly. Full attention could overweight the distracting 5432. A selector could choose the wrong block. Eligibility does not guarantee a correct answer; it defines which answer evidence remains possible at this layer.

That is the first complete mental model: sparse attention saves work before importance is calculated. The rest of the article explains how each eligibility rule buys its saving, and the distinct miss it creates.

Read more

The surprising part is that a longer context window does not remove this decision. Each retained frame expands into many pieces the model must process, each second of audio adds more, and the relationships across time still have to be computed. Keeping more evidence can preserve a brief event, but it can also bury that event inside thousands of irrelevant moments.

Google's September 2026 agentic video mode now lets a model search a timeline and adjust frame rate, resolution, and modality around promising moments, making evidence selection a practical product choice rather than a hidden preprocessing detail.

So the useful question is: How can an AI answer a precise question about hours of video without placing every frame in context?

The compact answer is to treat long video as a budgeted search for evidence. Fixed-rate sampling buys predictable coverage. Change-aware sampling keeps moments when the picture moves or cuts. A temporal hierarchy narrows a long recording to a few candidate intervals. Query-adaptive rewatching then spends dense frames, higher resolution, or audio only where the question needs them.

We will follow one realistic request through all four techniques. A warehouse camera records aisle 7 for two hours. A worker enters at 10:00. A red forklift first touches shelf B at 10:04.3, then backs away. The question is not merely whether both objects appeared. It is: When did the forklift first touch the shelf after the worker entered?

That wording matters. The answer needs an identity, an order, and a brief physical relation. A transcript may confirm that a worker called for help. A single frame may show the forklift beside the shelf. Only a short sequence can establish the first contact.

A video becomes a budgeted evidence packet

A frame is one still image from the video timeline. In a common transformer-based path, a vision encoder splits that image into patches and converts the patches into numerical representations the language model can process. I will call those representations visual tokens. One frame is therefore not one token; it expands into a small field of spatial evidence.

Video adds a second requirement. The model must connect what appears in one frame with what changes in later frames. That temporal relationship distinguishes "forklift near shelf" from "forklift moves toward shelf, makes contact, then reverses." The model's attention mechanism can compare retained visual tokens across time, but it cannot reconstruct a moment that the input pipeline never kept.

Audio and transcripts form separate evidence streams. Audio preserves nonverbal signals such as a metallic impact. A transcript is cheaper to search and excellent for spoken names or instructions, but it omits silent motion. Timestamps align the streams so a clue in one can point to a region in another.

Now imagine sampling the warehouse video once each second. The system keeps frames at 10:04.0 and 10:05.0. The actual touch at 10:04.3 may fall between them. Raising the rate to ten frames per second makes the event much more likely to appear, but it multiplies the visual evidence for the entire two-hour recording.

This is the first useful insight: temporal detail is purchased with context. The family of techniques differs mainly in who decides where to spend that detail, and when.

Read more

The surprising part is that an exact copy into a plain-text document can preserve the evidence. There is no hidden character to carry along. The ordinary words preserve the choices.

If you use AI to draft, proofread, or rewrite, the distinction matters: those operations leave different amounts of freedom for a watermark. And when someone says a passage was “detected,” the test matters too. A keyed watermark check measures a deliberately introduced pattern; it does not infer AI use from a writer's style.

Read more