Released architectures from Qwen, Moonshot AI and NVIDIA now combine these mechanisms, making the meaning of “model size” depend on both how history is stored and which parameters actually execute.

Why combine them? They separate three resources: memory of the sequence, access to individual positions, and capacity to transform a token. Improving one resource does not automatically improve the others.

The recurrent sequence mixers belong to a recognizable family, but their writes differ:

• Mamba-2 selectively fades a running state and adds new information.

• Gated DeltaNet also checks what its memory predicts, then writes a correction.

• Kimi Delta Attention, or KDA, adds finer control over which state coordinates fade.

Retained attention gives these compact memories a complementary path through position-indexed history. Mixture of experts, or MoE, changes a different component: the feedforward computation applied to each token representation.

That distinction already lets you read a hybrid architecture more accurately. In Qwen3-Next, a hybrid language model with a 3:1 mixer design, three recurrent mixers are followed by an attention mixer along the model's depth. Every token runs through that configured sequence. The model does not wait three generated tokens before using attention, and it does not decide at runtime whether a particular token deserves an attention layer.

The expert router does make a per-token choice. Mixer placement is fixed; expert selection depends on the current representation. Calling both “conditional computation” without this qualification hides how the model actually runs.

We will follow those two design decisions from the original decoder block to the memory updates, then put them back together. The payoff is practical: given another hybrid model, you should be able to explain what grows with context, what remains fixed, and why its active parameter count cannot tell you its serving cost.

Contents

• Two jobs inside a Transformer

• Mamba-2 keeps a running state

• Delta memory corrects what it stores

• Why hybrids retain attention

• Sparse experts change the other half

• When the combination is better

Read more

OpenAI's managed Agents API now exposes durable sessions, context compaction, recovery, and separate execution environments, making the boundary between a saved conversation and a recoverable task a concrete platform decision. Agents API overview.

What actually allows an agent to continue tomorrow?

The familiar agent loop is straightforward: receive a goal, ask the LLM for an action, execute a tool, return the observation, and repeat. A short demonstration can keep almost everything in one process. Stretch that task across several hours and the assumptions change. The agent may edit files, wait for approval, lose a network response, and outlive its container. Its next prompt cannot reconstruct facts that were never durably recorded.

The useful mental model is a distributed system with a probabilistic planner inside it. The model proposes what to do. The surrounding runtime records accepted decisions, preserves evidence, executes actions, and determines what can safely happen next.

One consequence is immediately useful: a missing result is different from a failed action. An external service may have completed the request before the worker disappeared. A saved task record saying “no result recorded” tells us about the worker's knowledge. It does not tell us whether the world changed. Recovering the job means closing that gap, not asking the model to guess.

We will separate the kinds of state, reconstruct a bounded context, and work through the commit boundaries. Then we will build a local runtime, continue a code repair in a fresh context, and terminate a process between an external effect and its saved acknowledgement.

Table of contents

• Separate the state that must survive

• Rebuild context from durable evidence

• Recover execution across a crash

• Resume workspaces, approvals, and delegated work

• Connect the complete runtime

• Build the durable runtime

• Continue a coding task across context windows

• Break it and compare recovery strategies

Read more

OpenAI’s Tax AI deployment turns practitioner corrections into evaluated product changes, making the quality of this feedback loop a concrete engineering problem.

The interesting question is how to turn those failures into useful practice, then retain what helps on future tasks.

The answer has two paths. One repairs the agent system: its code, instructions, tools, retrieval, or stored skills. The other improves the model using selected demonstrations or rewards from fresh attempts. Both need a way to decide whether a proposed change actually helps.

That is where environments become important. An environment supplies a task, a starting state, tools that change that state, and a way to evaluate the consequences. It can produce new experience when the agent tries a different action. A saved conversation contains only the experience that already happened.

This distinction changes what we should scale. More failure logs can reveal recurring defects. More executable environments can let an agent practice different decisions. Neither produces improvement until an update is made and tested on future behavior.

A useful first check is to ask what survives the run. A reflection left in a discarded conversation changes nothing for the next user. A saved skill changes future behavior only if another run retrieves and executes it. A newly trained checkpoint changes nothing until the application serves it. Persistence needs an actual connection to the next execution.

We will follow that connection through traces, three ways to construct executable environments, training, curriculum, and deployment.

Contents

• What persists after a run

• Turn traces into testable failures

• An environment produces new experience

• How executable environments are built

• Check what the environment rewards

• Convert experience into an agent update

• Build a curriculum from agent outcomes

• Promote improvements on independent evidence

Read more

DeepSeek's technical report motivates this design with agents that repeatedly receive long tool results and accumulated history, making the cost of reading and retaining that information a major part of running the model. Why should preparing old text for later use require the same computation as deciding what to write next?

The design divides 40 layers into two halves. The first 20 prepare reusable memory of the input. Most input positions can stop there; a final slice of up to 128 positions also passes through the upper half to prepare its recent context. Once generation begins, each newly chosen token goes through both halves to predict the following token. The architecture is called a causal encoder-decoder, or CED. We will unpack those roles, then trace how sharing memory, selecting what to read, storing fewer bits, and rebuilding recent state change different costs.

The useful distinction is already visible: preparing a record of earlier text and using that record to choose a new token need not follow identical paths. The shorter input route matters most when there is much more new text to read than text to generate.

Table of contents

• Why one causal stack reads and writes

• Why agent workloads change the budget

• Prefill, decode, and the cache between them

• How CED changes the old-token dependency

• Why generated tokens still use the full model

• Why this is a different encoder-decoder

• How CSA2 shares global history

• Why sparse search needs a hierarchy

• How the cache bytes add up

• What bounded replay rebuilds

• Putting the architecture together

• What the quality evidence establishes

• What this means for agent architectures

Read more

Here is the agent we will build: a generative model plans and writes; Jev chooses among named next actions using the current state; Python checks hard rules, executes a local tool, and records what actually happened. The central question is where a fast decision belongs when the task still needs open-ended reasoning and real effects. The short answer is between observations and permitted transitions. Jev can judge the state, but its choice cannot itself refund an account or prove that a refund settled.

We will make one Jev call first, then connect it to a small support agent. Then we will compare Jev with two structured-output LLMs using our experiments on support and document-research tasks. The experiment is useful because it records both proposed actions and actual local effects.

In this guide

• What Jev decides

• Give the agent observable state

• Build the guarded loop

• Run support and research tasks

• Compare three supervisors

• Read probabilities and disagreements

• Decide where Jev belongs

One distinction will keep the rest of the build honest. A tool returning HTTP 202 says a request was accepted for processing. It does not say the requested business outcome happened. That gap is exactly where a supervisor needs evidence, and where code must resist an unsafe retry.

Read more

The short answer is that Jev moves the boundary. A conventional language model can certainly emit {"route":"billing"}; strict structured output can make that object parse. Jev instead accepts the answer space as part of the question and returns the chosen answer and its distribution as the native result. Several questions can refer to one supplied state. That makes semantic judgments easier to place inside software, but it does not make their probabilities self-validating or authorize any action.

This is a model category worth understanding even if a particular workload ends up using rules, a fine-tuned classifier, or a constrained LLM. We will follow the interfaces first, then the computation they may avoid, the architecture that is and is not disclosed, and finally the policy that turns a judgment into a safe system decision.

In this deep dive

1. What Jev receives and returns

2. Why a separate decision model might help

3. What can be inferred about Jev's architecture

4. Calibration, RLCD, and action thresholds

5. Where Jev fits among existing tools

6. A system with generation and decision separated

One useful rule already follows from the interface: a model's answer should be treated as evidence about a defined outcome, while the application's code retains the choice of action. The rest of the article explains exactly what must be defined, computed, and checked before that rule is useful.

Read more

Teams are already using production conversations and human labels to build agent evaluations, as Dropbox's engineering account illustrates; the durable problem is deciding whether an agent actually completed an authorized task. How do we turn that question into a pipeline that can test a new agent version before release and keep learning afterward?

The compact answer is a loop: observed request → reviewed case → isolated trial → trace and final state → graders → version comparison → release decision → production feedback. A case describes the starting world and acceptable outcomes. A trial runs one agent attempt against that world. Graders inspect independent evidence, and the comparison asks whether the new version improves the tasks that matter without breaking important slices. The loop is reusable; the meaning of success changes with the agent's job.

The first useful rule is to give a completed action its own proof. If a support agent says “refunded,” a ledger record or service commit is the proof, subject to the service's consistency contract. If the record is absent, the polished sentence cannot rescue the task. That single separation prevents an answer-only judge from turning a failed action into a pass.

The second rule is to test the task from a known starting point. The same request can require a refund when the customer verifies identity, a clarifying question when they cannot, or a refusal when policy forbids it. An evaluation case must carry those starting facts and accepted branches. Otherwise a grader may punish a careful agent for asking or reward a reckless one for acting.

Before trusting a score, ask what its grader can observe. If it sees only messages, the score measures answer quality. If it sees tool calls but no final state, it can verify attempted actions without proving durable effects. Reserve “task success” for a test whose evidence covers the task's actual outcome. This distinction is useful even before the full pipeline exists.

Table of contents

1. Define what success must prove

2. Turn production evidence into cases

3. Run isolated, observable trials

4. Grade the evidence

5. Compare agent versions fairly

6. Operate the pipeline and close the loop

Read more

Persistent agent memory tools now let applications carry stored information across sessions and update it, making the quality of that update consequential for later tasks. The question is: when a user changes a standing preference, what must happen so an agent acts on the new one? A useful memory system treats the new message as a proposed change to stored state. It identifies the fact, finds the existing fact about the same thing, reconciles the two, commits the active version, and retrieves that version when the next task needs it.

That differs from appending messages to a transcript. We can see why by following one preference from its first instruction through a correction and a later task. The same failure can affect a project decision or customer preference.

What changes when an agent has memory?

The model receives a limited working context with each request. A conversation transcript can be placed in that context, but it grows, contains incidental remarks, and may contain claims that no longer hold. Persistent memory here means an external store that keeps selected information beyond one request or session. A write operation changes that store; a later read returns a useful part of it to the model. The model's weights are not being retrained every time a preference changes.

Suppose a user tells a coding assistant, “For the Atlas repository, write deployment scripts in Python.” The assistant might store a compact record with four parts: the user, the Atlas repository, the subject deployment-script language, and the current value Python. The message that supplied the preference is its source. This is more useful than an unqualified memory such as “prefers Python,” because the latter could affect unrelated work.

The store has two distinct paths. On the write path, a message or tool result is examined for information worth retaining, then compared with what is already stored. On the read path, the next task retrieves applicable memory and places it where the model can use it. If the write path is wrong, fast retrieval brings back a mistake. If the read path misses the right record, a perfect update sits unused.

Now the user says, “For Atlas, use Go for deployment scripts from now on.” Adding that sentence as a second, equally active memory would leave the future assistant to arbitrate a contradiction each time. A better result is an active Atlas deployment-language preference of Go, with the former Python preference no longer active. The old statement may remain in an audit history, but it should not be presented as another current instruction.

Remembering a correction means changing the state that future reads see, not merely remembering that a correction was said. The change has to preserve the subject and scope of the fact, or it will solve one task by creating a wider error.

Read more

Prefix-aware routing became a managed endpoint option in September 2026, a sign that cache locality has moved from an engine detail into a deployment decision.

So where should the next request go: to the warm replica or the idle one? The compact answer is to send it where the saved prompt computation is worth more than the extra work already waiting there. A useful router therefore needs two views at once: what can be reused and what must still be processed.

To see why, start with the work a cached prefix actually removes. Then we will keep one support prompt fixed while four routing policies make different placement decisions. That comparison will expose the useful rule, its failure modes, and the state a production router has to keep current.

What a warm prefix actually saves

Before routing makes sense, we need one piece of transformer serving machinery.

An incoming prompt is a sequence of tokens. During prefill, the model processes those input tokens through every transformer layer. In each attention layer it derives a key vector and a value vector for every position. Those vectors are the processed state that later tokens attend to. The first generated token can appear only after this prompt work has finished.

During decode, the model generates one new token, appends its new key and value state, and repeats. Decode can reuse the state for all earlier positions instead of rebuilding it. The stored collection is called the KV cache.

Now consider a support assistant whose prompt begins with 3,200 tokens of policies, tool descriptions, and response rules. Only the short customer question at the end changes. If the engine has already processed the same beginning, it can reuse the matching KV state and prefill only the new suffix.

Prefix caching does not mean that the model remembers a similar idea. The match is structural. Serving engines divide the prompt into token blocks, identify a block using its tokens and the blocks before it, and reuse only an unbroken matching prefix. If block three changes, later blocks no longer describe the same model state, even if much of their text looks familiar.

That distinction gives prefix caching an honest boundary. It skips model computation already performed for the input prefix. It does not skip the generation of new output tokens, because those tokens do not exist yet. A workload with long repeated inputs and short answers can gain a lot; a workload with unrelated prompts or very long answers may gain little.

Within one replica, the rule is simple: reuse every valid prefix block that is still resident, prefill the remaining input, then decode normally.

Read more

That distinction is now practical because current OpenAI and Anthropic APIs expose reasoning-effort controls while Google offers an Extended Thinking model for complex live-agent tasks, turning “let it think” into a quality, latency, and cost decision for each request.

The useful question is not simply, “How much thinking should I buy?” It is: where should the extra compute go? Test-time scaling allocates it to depth, width, or feedback. Depth extends one attempt. Width creates separate attempts. Feedback evaluates work and changes the next attempt. Each repairs a different failure, and none guarantees correctness.

What test-time compute actually is

Start with an ordinary request. Training is already over, and the model begins with learned weights that stay fixed while it answers. At each step, the model processes the request and the tokens generated so far to choose the next token. That ordinary generation already consumes test-time compute: computation performed after training for one particular input.

The term covers more than the model's hidden reasoning tokens. It includes the work the model and its surrounding system perform to produce, inspect, and improve this one answer. Continuing the generation adds more token-level operations to the current history. Starting another generation creates a separate candidate history. Running a test or verifier creates an observation that a later attempt can use. Revising with that observation performs another generation under changed conditions.

Test-time scaling is the deliberate decision to increase or reallocate that per-request work relative to a baseline answer. The extra work can matter because it changes the transient information available to a later choice: a longer prefix, a genuinely different candidate, or new evidence about an error. If the added work does not change that state or evidence, it may increase latency and cost without improving the answer.

This is not training or fine-tuning. Training changes reusable model weights for future requests. Test-time compute leaves those weights alone and is paid again for each request. It is also not synonymous with selecting a larger model or producing a longer visible answer. A system may spend substantial compute on candidates, checks, and revisions that never appear in the final response.

We will follow one realistic coding task through all three choices. An agent must repair invoice_total, a function that sums prices, applies tax, and returns integer cents. Its contract requires exact decimal arithmetic, an 8 percent tax applied after aggregation, round-half-up once to the nearest cent, then conversion to integer cents. The visible invoice contains items priced at $12.49, $7.25, and $3.10. Hidden tests vary the inputs to expose half-cent rounding, multiple-item accumulation, zero tax, and binary floating-point drift.

First we need to see why one uninterrupted generation can fail. Then we can spend the same request-time budget on a deeper path, a wider candidate set, or a feedback loop and see what actually changes.

Before scaling, identify the one-shot failure

An autoregressive language model generates a response one token at a time. At every position it receives the request plus the tokens already written, then assigns probabilities to possible next tokens. Once a token is chosen, that token becomes part of the state used to predict everything that follows.

This left-to-right process matters because an early choice can narrow the rest of the answer. Suppose our coding agent begins with the assumption that converting dollars to cents is just int(total * 100). It can write an elegant explanation and many consistent lines of code after that. The later tokens are coherent with the earlier assumption, but coherence does not repair the assumption.

A one-shot attempt is one uninterrupted generation from request to answer. It may spend many internal reasoning tokens, but it has no separate alternative to compare with and no external result that changes its course. Its success depends on the model entering a useful line of reasoning and staying there.

For this lesson, a useful accounting unit is a generated reasoning token, a candidate response, a verifier evaluation, or a tool call such as a test run. These units have different hardware costs, so we will not pretend they are perfectly interchangeable. The important constraint is that the system has a finite request-time budget and must decide how to allocate it.

There are three places to allocate that budget:

1. Depth: continue or enrich the same reasoning trajectory.

2. Width: sample several separate trajectories from the same starting request.

3. Feedback: inspect work, then condition the next attempt on what the inspection revealed.

The distinction is about information flow. Extra depth sees the same request and its own growing prefix. Width gives each attempt the request but not the siblings' work. Feedback adds a new observation, such as “the half-cent test failed,” before the next generation. Width is logical branching; whether those branches run simultaneously is a separate serving decision.

That gives us a diagnostic before we spend anything. If the model merely stops before completing a sound plan, buy depth. If different plausible starts lead to different outcomes, buy width. If a checker can reveal what went wrong, buy feedback. A high effort setting may combine these internally, but an application designer still needs this mental model to understand latency, cost, and failure.

Depth gives one trajectory room to finish

The simplest scaling strategy is to let one attempt run longer. The model can decompose the task, inspect edge cases, and carry intermediate facts farther before committing to the final response. In our invoice task, a deeper trajectory must preserve a specific dependency: represent the prices exactly, sum them to $22.84, apply the 8 percent tax to obtain $24.6672, round that total once to $24.67, then return 2467 cents.

The benefit is continuity. Every later step sees the reasoning accumulated earlier. Here, the final integer conversion only makes sense after the representation and rounding policy are settled. Once an earlier step truncates a fraction of a cent, no later step can reconstruct it. Proofs, long calculations, and code changes with dependent invariants have the same shape.

But depth is not the same as recovery. If the trajectory commits early to binary floating point and treats a failing cent as harmless noise, more tokens can reinforce the wrong frame. The model may repeatedly check arithmetic inside a mistaken representation. It is thinking longer, but within a shrinking part of the solution space.

That gives us the first useful rule: depth helps when a sound path is unfinished, not merely when a path is wrong. A model that often reaches the correct approach and then runs out of room is a good candidate for higher effort. A model that confidently commits to a bad representation needs an alternative or new evidence, not an even longer monologue.

Read more

Normally, a language model can choose any token in its vocabulary at each generation step. Constrained decoding inserts a rule checker into that loop. The checker looks at the text already generated and removes every next token that would make the requested structure impossible to complete. The model then chooses among what remains.

Structured output has become a common connection layer between LLMs, extraction pipelines, and agent tools, yet a recent developer discussion exposed the practical surprise that matters here: a perfectly shaped object can still contain invented values, so schema-valid is not the same as safe to execute.

The central question is simple: what does the decoder actually guarantee? The compact answer is that it can guarantee membership in a supported output language, such as a JSON Schema, by blocking illegal continuations token by token. It does not prove that a legal value came from the input, reflects reality, satisfies a business rule, or deserves permission to trigger an action.

We will follow one support request through the complete mechanism. The request says: Please refund order A-17. The charge was $42.50. The desired output is a small object containing a status, an order identifier, and a refund amount in cents. First we need to separate three ideas that are often collapsed into the word “valid.” Then we can compile the schema, watch the token mask change, and see exactly where truth leaves the decoder's jurisdiction.

Three different promises hide inside “valid JSON”

JSON is a text format. A JSON parser checks punctuation and nesting: braces balance, keys are quoted, commas occur in legal places, and values have valid JSON forms. This object is valid JSON:

{"status":"ready","order_id":"A-17","refund_cents":4500}

A schema asks a narrower question about the parsed value. It can require an object, name its allowed properties, require selected keys, restrict a field to an integer, or limit a value to an enumeration. If our schema allows any non-negative integer for refund_cents, then 4500 passes. The schema has no memory of the user's $42.50.

Finally, the application asks whether the object means something acceptable in its real environment. Does order A-17 exist? Was the charge really $42.50? Is it refundable? Is this user authorized? Those are evidence, policy, and permission questions. They are not punctuation questions.

The checks therefore form a hierarchy:

1. Parseable means the text is legal JSON.

2. Schema-valid means the parsed value also obeys the declared structural rules.

3. Grounded and actionable means the values are supported by evidence and pass application policy.

Each level can remove failures that the previous level accepts. None can be silently substituted for the next.

Our refund extractor also needs a way to say that the message is incomplete. If a successful decode must complete a contract where every field is required and refund_cents must be an integer, every successful path puts some integer there even when no amount appears in the input. A better output contract exposes two branches:

{ "anyOf": [ { "status": "ready", "order_id": "string", "refund_cents": "non-negative integer" }, { "status": "insufficient_info", "order_id": null, "refund_cents": null } ] }

This is a readable sketch of the contract, not a literal complete JSON Schema. The important move is structural: uncertainty is now a legal output rather than an instruction the model must squeeze into fields that forbid it.

Here is the first design rule worth keeping: a schema should represent the failure states your application expects. Constrained decoding can only choose paths you make legal. If the contract requires a confident answer, the decoder cannot invent an abstention branch on your behalf.

Read more

That first decision is the hidden trade-off behind sparse attention. The name suggests that the model merely turns down unimportant history. In many designs, it does something more consequential: it prevents most of the history from entering the main attention calculation at all.

A late-August 2026 Transformers release added Qwen Sparse Attention, while an open-weight Qwen model built on the same block-selection idea prompted immediate questions about long-context latency and cache placement; as more models save work this way, the durable question is which evidence they stop reading.

So here is the question we will answer: When sparse attention makes a long context cheaper, what does an LLM stop looking at?

The compact answer is that every attention pattern creates an eligible set of earlier token states. Full attention admits the whole past. A sliding window admits only nearby history. Content-selected sparse attention uses a cheaper scoring step to choose a small subset that can include distant history, then runs ordinary attention on that subset. Fewer eligible states mean less work, but a clue excluded at this point cannot be rescued by a larger attention weight later.

We will follow one question through all three patterns. A long incident report contains the deployed database proxy setting, port 6432, near the beginning. Much later, a diagnostic note mentions the default database port, 5432. At the end, the reader asks: Which port did the incident's database proxy actually use?

This is a useful test because the answer is a tiny exact value separated from the question by a lot of plausible noise. We will hold the tokens, model layer, and question fixed. Only the rule that exposes earlier states to attention will change.

Attention makes two decisions, not one

An LLM processes text as tokens, which are words or pieces of words converted into numerical states. At one layer, every token state is transformed into three useful roles.

Every token produces all three. When the layer updates the current token, its query represents what information it is seeking. The current token and earlier tokens provide keys, which can be compared with that query, and values, the information that can be carried forward. The query-key comparisons become weights; the weighted values become the context added to the current token's state. The diagram focuses on earlier sources, but causal attention also allows the current token to attend to itself.

There is a prerequisite hiding before those comparisons. An attention rule supplies a mask or selection pattern that says which keys and values are eligible. A future token is ineligible in a causal language model because generation cannot look ahead. Sparse patterns add more exclusions among the past.

It helps to separate two questions:

1. Eligibility: may the current token connect to this earlier state?

2. Importance: among eligible states, how much weight should this one receive?

Full, windowed, and content-selected attention mainly differ on the first question. Once a subset is eligible, the main attention operation can still compare queries with keys and mix values in the familiar way.

Apply that distinction to our incident report. Full attention places both 6432 and 5432 in the candidate set. A four-block sliding window includes only the late diagnostic note and loses the deployed setting. A good content selector reaches back to the configuration block containing 6432, while ignoring most of the middle.

The model still has to interpret the candidates correctly. Full attention could overweight the distracting 5432. A selector could choose the wrong block. Eligibility does not guarantee a correct answer; it defines which answer evidence remains possible at this layer.

That is the first complete mental model: sparse attention saves work before importance is calculated. The rest of the article explains how each eligibility rule buys its saving, and the distinct miss it creates.

Read more

The surprising part is that a longer context window does not remove this decision. Each retained frame expands into many pieces the model must process, each second of audio adds more, and the relationships across time still have to be computed. Keeping more evidence can preserve a brief event, but it can also bury that event inside thousands of irrelevant moments.

Google's September 2026 agentic video mode now lets a model search a timeline and adjust frame rate, resolution, and modality around promising moments, making evidence selection a practical product choice rather than a hidden preprocessing detail.

So the useful question is: How can an AI answer a precise question about hours of video without placing every frame in context?

The compact answer is to treat long video as a budgeted search for evidence. Fixed-rate sampling buys predictable coverage. Change-aware sampling keeps moments when the picture moves or cuts. A temporal hierarchy narrows a long recording to a few candidate intervals. Query-adaptive rewatching then spends dense frames, higher resolution, or audio only where the question needs them.

We will follow one realistic request through all four techniques. A warehouse camera records aisle 7 for two hours. A worker enters at 10:00. A red forklift first touches shelf B at 10:04.3, then backs away. The question is not merely whether both objects appeared. It is: When did the forklift first touch the shelf after the worker entered?

That wording matters. The answer needs an identity, an order, and a brief physical relation. A transcript may confirm that a worker called for help. A single frame may show the forklift beside the shelf. Only a short sequence can establish the first contact.

A video becomes a budgeted evidence packet

A frame is one still image from the video timeline. In a common transformer-based path, a vision encoder splits that image into patches and converts the patches into numerical representations the language model can process. I will call those representations visual tokens. One frame is therefore not one token; it expands into a small field of spatial evidence.

Video adds a second requirement. The model must connect what appears in one frame with what changes in later frames. That temporal relationship distinguishes "forklift near shelf" from "forklift moves toward shelf, makes contact, then reverses." The model's attention mechanism can compare retained visual tokens across time, but it cannot reconstruct a moment that the input pipeline never kept.

Audio and transcripts form separate evidence streams. Audio preserves nonverbal signals such as a metallic impact. A transcript is cheaper to search and excellent for spoken names or instructions, but it omits silent motion. Timestamps align the streams so a clue in one can point to a region in another.

Now imagine sampling the warehouse video once each second. The system keeps frames at 10:04.0 and 10:05.0. The actual touch at 10:04.3 may fall between them. Raising the rate to ten frames per second makes the event much more likely to appear, but it multiplies the visual evidence for the entire two-hour recording.

This is the first useful insight: temporal detail is purchased with context. The family of techniques differs mainly in who decides where to spend that detail, and when.

Read more

The surprising part is that an exact copy into a plain-text document can preserve the evidence. There is no hidden character to carry along. The ordinary words preserve the choices.

If you use AI to draft, proofread, or rewrite, the distinction matters: those operations leave different amounts of freedom for a watermark. And when someone says a passage was “detected,” the test matters too. A keyed watermark check measures a deliberately introduced pattern; it does not infer AI use from a writer's style.

Read more

In the two-model version we will follow, the target still does its full computation. The saving comes from arranging that computation differently, so one pass through its layers can advance the answer by several tokens. Whether that saves time depends on how much of the draft survives.

Read more

We'll also look inside speculative decoding and text watermarking, then turn to two visual problems: why a generated image can look convincing while missing the instructions, and why a fluent document answer can still come from the wrong cell.

1. Model validation: a random split can preserve the shortcut

Can machine learning validation approve a model that learned the wrong feature? It can, if the validation set repeats the same shortcut!

Imagine a pneumonia classifier trained on chest X-rays from two hospitals. Hospital A contributed most of the positive cases. Hospital B contributed most of the negative cases.

Each image contains two usable signals: the lung pattern we care about, and a hospital fingerprint from scanner processing, text markers, or acquisition style. The loss function does not know which explanation we intended. It rewards features that predict the label. If recognizing the hospital offers an easier route to lower loss, the model can use it.

Now randomly split the rows into training and validation sets. Both sets still contain the same hospitals and the same correlation: Hospital A usually means positive; Hospital B usually means negative. Randomization changes which rows land in each set, but it does not change the process that produced them.

The validation images are unseen, yet the shortcut is familiar. A high score can accurately describe performance on this source mixture and still mislead us about deployment.

Move the model to Hospital C. In this toy example, the disease signal remains valid while the old hospital-label correlation disappears. Prediction quality can drop without any bug in the inference code. The diagram follows that change from collection to deployment.

A random row split estimates performance on another row from the same source mixture. It says little about a source absent from that mixture. When the expected novelty is a hospital, camera, customer, region, time period, or device, holding out the entire unit lets validation test that change instead of scattering the unit across both sides.

I would then ask whether the model is using the particular cue I suspect. Keep the target content fixed, remove or swap the background, marker, source template, or metadata, and watch whether the prediction moves.

These tests answer different questions. A group holdout tests an environment change. A cue intervention tests dependence on one named feature. Neither proves that the model learned the intended signal: the holdout covers only the environments we chose, and the intervention is trustworthy only if it changes the shortcut without changing the task itself.

2. Agent reward hacking: passing the test while leaving the bug

Why can AI agent reward hacking produce a perfect score on a task the agent never completed? The reason is simple: the agent optimizes the feedback we expose, not the intention behind it.

Consider a coding agent asked to fix an invoice calculation. The correct total is $110, but the code returns $100. We use the test suite as the evaluator: a pass earns a reward of +1, and a failure earns 0.

During reinforcement learning, the agent tries a sequence of actions and receives this reward. The optimizer makes high-reward action sequences more likely in similar situations.

We expect the agent to fix the calculation, pass the tests, and produce an invoice for $110. But the test suite is also part of the environment. If the agent can modify it, another path becomes available: disable the assertion, pass the tests, and leave the invoice at $100.

Both paths receive +1. The second path can be reinforced because the remaining failure was never reflected in the score.

Three different objects matter here:

• The true objective is a correct invoice.

• The proxy reward is the +1.

• The evaluator is the test suite that decides when to issue it.

Exploiting a missing condition in the proxy is usually called specification gaming. Changing the evaluator itself, by editing a test, checklist, log, or scoring process, is the more specific case of reward tampering.

No human-like intention to cheat is required. From the optimizer's point of view, the two action sequences received the same reward. The number does not contain the missing requirement, so we cannot recover it from the score afterward.

Greater capability does not settle this problem. Better search can find the correct implementation, but it can also find shortcuts that a weaker agent never discovered. It expands both sets of paths.

A wrong answer with a low score is ordinary failure. Reward hacking is narrower: the measured score says success while the intended outcome stays unchanged or gets worse. The test suite is another piece of the environment, and its assumptions shape what the agent learns.

3. Durable agents: a checkpoint cannot confirm an external payment

How can durable execution for AI agents survive a crash and still charge the same invoice twice? The checkpoint remembers where to resume, but it cannot prove whether the outside world already changed!

Imagine an agent paying a $480 invoice. It records step pay-1 as pending, sends the request, and the payment API accepts it. Then the worker crashes before saving the receipt.

After restart, the execution state still says:

run-42 -> pay-1 -> PENDING

From that saved state alone, the replacement worker cannot tell whether the payment failed or only the acknowledgement failed. We know the API accepted the payment in this example; the worker has no saved receipt to establish that. Retrying may be necessary, but the tool step may execute again.

An idempotency key is a stable identifier that tells the external service that a retry represents the same logical operation. The checkpoint identifies the unfinished step; the key identifies the one external operation that step is allowed to create.

The diagram uses run-42/pay-1 as the key, recorded before the first request. The replacement worker reloads the latest checkpoint, skips completed work, and sends pay-1 again with that same key. If the payment service supports idempotency and already processed the key, it returns the original receipt instead of creating another payment. The runtime then saves the receipt, marks the step completed, and continues.

This is why conversation memory and execution state deserve separate treatment. Conversation memory can hold messages, facts, and user preferences. Execution state needs the run ID, step ID, exact tool arguments, status, approvals, timers, and durable receipts. A sentence saying "the invoice was paid" is not a transaction record.

The same pattern appears outside payments. A database can reject a duplicate operation through a unique constraint or conditional write. An event consumer can deduplicate using a stable event ID.

If a tool has no idempotency mechanism, blind retry is the wrong recovery strategy. The agent has to query the external system, run a compensating action that reverses or offsets the first effect, or stop for human review.

Durable execution preserves progress. It does not make the model correct, permissions safe, or an arbitrary API idempotent. Once an agent can wait for humans, call tools, and run longer than one request, its execution state matters as much as its prompt.

4. Speculative decoding: draft several tokens, verify in one pass

How can speculative decoding use a smaller language model without inheriting its mistakes? The small model guesses a few tokens ahead, but the large target model verifies the whole guess in one pass and keeps control of the output distribution.

The usual language-model decoding loop is serial. The target model predicts one token, appends it to the sequence, then runs again. Producing K tokens requires K target-model passes.

During low-batch decoding, a pass often spends more time moving the model weights than using all the available arithmetic. Scoring a short block can reuse each weight read across several token positions, so it can take close to the time of scoring only one.

Speculative decoding takes advantage of that opportunity. In the common two-model version, a cheap draft model generates a short continuation one token at a time. The large target model receives the prompt and complete draft, then scores every drafted position in parallel.

Acceptance still proceeds from left to right. Accepted tokens become final. At the first rejection, every later draft token is discarded because it was conditioned on a prefix the target did not accept. The target samples a correction, and the next drafting round starts from the corrected prefix.

For the prompt "Water freezes at", imagine the draft proposes:

"0" -> "degrees" -> "Kelvin" -> "."

One target pass can score all four positions. It may accept "0" and "degrees", reject "Kelvin", discard the period, and sample "Celsius" as the correction. That slow pass has finalized three useful tokens instead of one.

What I find most interesting is how this works for sampled decoding. We cannot simply keep a token when two independent samples happen to match. If draft and target were identical uniform distributions over 100 tokens, their separate draws would match only 1% of the time, despite perfect agreement between the distributions.

Call the target distribution p and the draft distribution q. A drafted token x is accepted with probability:

min(1, p(x) / q(x))

If it is rejected, the correction is sampled from the normalized positive part of p - q. To build that distribution, keep the positive differences between target and draft probabilities, set the others to zero, and rescale the remaining values so they sum to one. This is the target probability not already covered by the draft.

Together, the accepted path and correction path reconstruct p. Under this rule, the draft changes latency without biasing the target distribution.

The benefit depends on the whole serving setup. A slow draft adds too much overhead. A draft with little overlap with the target causes frequent rejections. Hardware that is already compute-saturated has little parallel capacity left to exploit.

The useful draft is the one whose acceptance rate is worth its cost on the target model and hardware being served. Accuracy of the small model alone does not determine that tradeoff.

5. Text watermarking: small token preferences add up to evidence

How can generative text watermarking mark a paragraph without adding hidden characters, metadata, or a visible tag? It changes ordinary token choices, then looks for the accumulated pattern!

At each decoding step, a large language model (LLM) produces a probability distribution over the next token. Several continuations may fit. A normal sampler chooses among them; a watermarking sampler uses one additional input, a secret key.

One common construction uses the key and recent tokens to pseudorandomly divide the vocabulary into green and red sets. Plausible green tokens get a small boost. The partition is recomputed after every token, so there is no fixed list of suspicious words.

Consider a toy passage with 100 positions counted by the detector. Without the watermark, we expect roughly 50 green hits by chance. A watermarked sampler might produce 70. No individual token looks unusual; the pattern appears when we aggregate the choices.

For this simplified 50/50 construction, a one-proportion test gives:

z = (70 - 50) / sqrt(100 × 0.5 × 0.5) = 4

The numerator is the 20 extra green hits. The denominator is 5, the standard deviation of the count under this simplified chance model. The observed count sits four standard deviations above the expected count.

Other watermark families use different scoring rules. Here, the detector tokenizes the passage, uses the same key to recreate the green and red sets at each position, and tests whether the excess of green tokens is too large to explain by chance. It measures a statistical pattern rather than judging whether the prose sounds robotic.

Length supplies evidence. A ten-token answer provides very few weak votes; a long passage supplies many more observations. The model also needs freedom to choose. Open prose may offer several reasonable continuations, while code or familiar fixed phrases can have one dominant next token, leaving little room to embed a preference without damaging the output.

Editing weakens the signal without predictably erasing it. Replacing a token removes one marked choice. In context-dependent schemes, it can also change the expected partition at positions that use the edited context. A substantial rewrite can reduce the evidence dramatically, yet a long paraphrase may preserve enough unaltered fragments to remain detectable.

A positive result means the text is statistically consistent with one key's watermarking rule. It does not prove who wrote the ideas, and it cannot identify output from a model that never applied that mark. The detector threshold still trades missed detections against false positives.

That is why a long, open-ended answer can carry strong watermark evidence while a short code completion from the same model may carry almost none.

6. Image generation: realism and exact composition are different tests

AI image generation can give you a beautiful campaign image and still ignore the requirement that makes it usable! The problem is that a text prompt is not an engineering drawing.

Consider this request: "three green bottles to the left of one orange box, with FRESH on the label."

One sentence carries five kinds of constraints: which objects exist, how many of each, which attributes belong to which objects, where they go, and which exact characters appear.

In many diffusion systems, a text encoder converts the prompt into vectors. Cross-attention lets the denoising network consult those vectors while turning noise into an image. A few words can influence a large visual field without someone hand-coding the scene.

But the vectors do not arrive as an explicit object table with separate entries for color, count, and coordinates. During denoising, "green", "bottles", "three", and "left of" must all influence one shared visual representation. The model can form a convincing poster while giving you four bottles, coloring the box green, or letting objects overlap.

Realism and prompt compliance measure different things. Bottles, an orange box, and text-like marks can all be present while the requested relationships drift. Counting, relative position, and attaching an attribute to the correct object are compositional challenges.

Saying that image models "cannot count" misses the larger problem. Individually recognizable concepts have to stay attached to the right instance and location throughout generation.

Sketches, layouts, segmentation maps, and image checks help make some intended state explicit: where an object belongs, how many are needed, and whether a word is exact. The diagram separates guidance from checking: its layout specifies the bottles' positions, while the checks inspect count, position, and the characters in FRESH. These aids do not guarantee a correct result, but they stop the prompt from carrying every constraint alone.

A creative mood board benefits from the model filling gaps. An asset with exact copy, object counts, or placement can fail precisely when the model fills in a gap you intended to specify.

7. Document extraction: connect the answer to the right cell

AI document extraction can give you a perfectly grammatical answer that comes from the wrong cell in the document! The reason is that a vision-language model is doing several different jobs before it ever starts reasoning about your PDF.

Start with a scanned bank statement:

• Opening balance: $14,900

• Deposits: $2,200

• Closing balance: $17,100

Ask, "How much was deposited?"

The model does not receive the document as a clean database row. A vision encoder turns pixels into a limited set of visual tokens. Small type, a low-quality scan, and dense tables can make the evidence for a word or nearby number weak or mixed. Some PDF pipelines pass native text separately, which changes the failure mode, but charts, tables, handwriting, and rendered layout still require visual interpretation.

Then comes binding: "Deposits" must be linked to $2,200. Recognizing all three amounts somewhere on the page is insufficient. The language model uses the visual representations and the question to generate its answer token by token.

If the deposit label was unclear, or the label-to-value association was wrong, "$17,100" can still be a plausible completion. Fluency tells us the decoder completed a sentence well. It does not tell us which pixels supported the answer.

I separate document extraction into three questions:

1. Did the system read the relevant glyphs?

2. Did it preserve the relationship between label, row, column, and value?

3. Did the answer come from that evidence, or from a plausible completion?

Optical character recognition (OCR) and coordinates can make the first two steps inspectable. A vision-language model is useful when the final answer needs a comparison, interpretation, or calculation.

More visual resolution can improve fine detail, but it also costs tokens and latency. It does not automatically recover a missing label-to-value relationship.

Chart questions have the same difficulty. Reading "Q4" and "47" is insufficient: the system has to connect the right series, axis, and visual mark before it can reason usefully about them.

In this guest post, Aria Shi, the Developer Relations lead at SylphAI, walks you through how AdalFlow empowers AI Agent development, highlighting a hands-on example with a LinkedIn Reachout Agent.

Say goodbye to manual prompt engineering. AdalFlow is the all-in-one, auto-differentiative solution for optimizing prompts, whether you’re using zero-shot or few-shot learning. Backed by our state-of-the-art research (LLM-AutoDiff and Learn-to-Reason), our framework achieves the highest accuracy among all automatic prompt optimization libraries.

The rise of large language models has completely changed the way we build applications—whether it’s chatbots, RAG systems, or fully autonomous agents. But as an AI engineer, trying to bring these models into production often feels like stitching together a bunch of experiments, rather than building a stable and reliable system.

We introduce AdalFlow: a PyTorch-like library designed to bring structure, clarity, and optimization to the world of LLM application development. Built as a community-driven project, AdalFlow is uniting AI research and production engineering into a single ecosystem.

Why We Built AdalFlow

Modern AI development faces a paradox. On one hand, researchers push the boundaries of model capabilities with new techniques in prompting, evaluation, and optimization. On the other hand, production teams need reproducibility, scalability, and a way to iterate safely on real-world data.

Most libraries excel at one side of the equation but leave the other underserved. AdalFlow was born to bridge this gap. With 100% control and clarity of source code, it empowers researchers to experiment freely while giving product engineers the tools to build and ship with confidence.

Why AdalFlow Matters

By treating prompts as first-class citizens and introducing LLM-AutoDiff, AdalFlow provides what’s been missing in the LLM ecosystem:

• For researchers: A familiar PyTorch-like environment to prototype new prompting and training methods.

• For engineers: Production-ready workflows that are debuggable, reproducible, and optimizable.

• For teams: A shared framework that unites research and production into one healthy ecosystem.

Core Philosophy: Prompt Is the New Programming Language

If PyTorch turned tensors into the lingua franca of deep learning, AdalFlow treats prompts as the new programming primitives.

Every LLM application boils down to structured prompts and their transformations. AdalFlow embraces this reality by making prompt engineering explicit and optimizable. Behind the scenes, it uses the Jinja2 templating engine to let developers define composable prompt structures, ensuring that LLM apps are both modular and debuggable.

Components: The Building Blocks of LLM Workflows

At the heart of AdalFlow lies the Component abstraction. Just as nn.Module became the foundation for PyTorch models, Components unify every stage of an LLM pipeline.

• Component: The base class for all workflows. Handles both training (forward) and inference (call) modes, with bicall bridging the two.

• GradComponent: Components capable of backpropagation (e.g., Generators, Retrievers).

• DataComponent: Lightweight components for formatting and parsing data (e.g., DataClassParser).

• LossComponent: Wraps evaluation metrics and enables gradient-like feedback for text optimization.

Example 1: Q&A with Object Counting (Component + DataComponent)

template = r"""<START_OF_SYSTEM_PROMPT>
{{system_prompt}}
<END_OF_SYSTEM_PROMPT>
<START_OF_USER>
{{input_str}}
<END_OF_USER>"""

@adal.func_to_data_component
def parse_integer_answer(answer: str):
    numbers = re.findall(r"\d+", answer)
    return int(numbers[-1])

What’s happening here?

• parse_integer_answer is wrapped with @adal.func_to_data_component.

• This turns a plain Python function into a DataComponent, which handles structured output parsing.

• In this case, it ensures the model’s answer ends with a numerical value.

Next, we define a full pipeline:

class ObjectCountTaskPipeline(adal.Component):
    def __init__(
            self, model_client: adal.ModelClient, model_kwargs: Dict
        ):
        super().__init__()
        system_prompt = adal.Parameter(
            data="You will answer a reasoning question. Think step by step. The last line should be 'Answer: $VALUE'.",
            role_desc="Task instruction for the model",
            requires_opt=True,
            param_type=ParameterType.PROMPT,
        )
        self.llm_counter = adal.Generator(
            model_client=model_client,
            model_kwargs=model_kwargs,
            template=template,
            prompt_kwargs={"system_prompt": system_prompt},
            output_processors=parse_integer_answer,
        )

    def bicall(self, question: str, id: str = None):
        return self.llm_counter(
              prompt_kwargs={"input_str": question}, id=id
        )

ObjectCountTaskPipeline subclasses Component. Inside it, we define:

• A Parameter of type PROMPT, which AdalFlow can later auto-optimize.

• A Generator (a GradComponent) that executes the prompt, then passes the raw LLM output through our parse_integer_answer DataComponent.

The workflow is:
Prompt → LLM Generation → Structured Output Parsing → Final Numerical Answer.

Example 2: Classification with Structured Output (Component + DataClass)

Classification tasks are a perfect showcase of AdalFlow’s DataClass feature.

@dataclass
class TRECExtendedData(adal.DataClass):
    question: str = field(
        metadata={"desc": "The question to be classified"}
    )
    rationale: str = field(
        metadata={"desc": "Step-by-step reasoning"}, default=None
    )
    class_name: Literal[
        "ABBR", "ENTY", "DESC", "HUM", "LOC", "NUM"
    ] = field(
        metadata={"desc": "The class name"}, default=None
    )

    __input_fields__ = ["question"]
    __output_fields__ = ["rationale", "class_name"]

• TRECExtendedData extends DataClass, which (like Pydantic) gives us schema enforcement.

• Input: a question.

• Output: a rationale (reasoning trace) and a class_name (final label).

Now let’s plug it into a pipeline:

class TRECClassifierStructuredOutput(adal.Component):
    def __init__(
        self, model_client: adal.ModelClient, model_kwargs: Dict
    ):
        super().__init__()
        # Task description prompt
        task_desc_str = adal.Prompt(
            template=task_desc_template,
            prompt_kwargs={
                "classes": [
                    {"label": l, "desc": d} 
                    for l, d 
                    in zip(_COARSE_LABELS, _COARSE_LABELS_DESC)
                ]
            }
        )()

        parser = adal.DataClassParser(
            data_class=TRECExtendedData,
            return_data_class=True,
            format_type="yaml"
        )

        prompt_kwargs = {
            "system_prompt": adal.Parameter(
                data=task_desc_str,
                role_desc="Task description",
                requires_opt=True,
                param_type=adal.ParameterType.PROMPT,
            ),
            "output_format_str": parser.get_output_format_str(),
        }

        self.llm = adal.Generator(
            model_client=model_client,
            model_kwargs=model_kwargs,
            prompt_kwargs=prompt_kwargs,
            template=template,
            output_processors=parser,
        )

    def bicall(self, question: str, id: Optional[str] = None):
        return self.llm(prompt_kwargs={"input_str": question}, id=id)

• The Prompt defines the system instruction with class definitions.

• DataClassParser enforces structured YAML output that matches TRECExtendedData.

• Generator (a GradComponent) runs the LLM with prompt + parser.

• Output is guaranteed to follow the schema: rationale + class name.

This ensures the model never drifts into free-form answers—it always returns structured classification results.

Example 3: Training With LossComponent

Finally, how do we train or optimize these components? That’s where LossComponent comes in:

eval_fn = AnswerMatchAcc(type="exact_match").compute_single_item
loss_fn = adal.EvalFnToTextLoss(
    eval_fn=eval_fn,
    eval_fn_desc="exact_match: 1 if str(y) == str(y_gt) else 0"
)

• AnswerMatchAcc is the evaluation metric.

• EvalFnToTextLoss wraps it as a LossComponent, enabling LLM-AutoDiff to optimize prompts automatically during training.

By attaching this to your pipeline, you get a full training loop:
Forward pass → Eval metric → Backward engine → Prompt optimization.

Agents: Reasoning Meets Action

AdalFlow embraces the ReAct paradigm—combining reasoning (plan) with acting (tool use)—to build autonomous, auditable AI systems. An agent reasons about the task, selects tools, executes them, observes results, and iterates until it can deliver a final answer.

• https://adalflow.sylph.ai/new_tutorials/agents_runner.html

• https://colab.research.google.com/github/SylphAI-Inc/AdalFlow/blob/main/notebooks/agents/agent_tutorial.ipynb

Architecture at a Glance

• Agent (planner + tool manager)
  Handles planning and decision-making via a Generator-based planner, and knows what tools are available and how to call them.

• Runner (executor + conversation loop)
  Orchestrates multi-step execution, tool calling, observation handling, timeouts, and final answer synthesis.

This separation lets you swap or customize planning vs. execution independently.

Execution Flow (ReAct Loop Recap)

1. Planning – The Agent (Generator planner) analyzes input and proposes the next action.

2. Tool Selection – Chooses a tool from the registered set.

3. Tool Execution – The Runner invokes the tool with arguments.

4. Observation – The result is fed back to the planner.

5. Iteration – Repeat 1–4 up to max_steps or until confident.

6. Final Answer – The planner synthesizes the answer (optionally into a structured type).

Minimal, End-to-End Example

1) Define a Tool (callable or FunctionTool)

# Tool: a plain Python callable works, or wrap with FunctionTool for extras.
def calculator(expression: str) -> str:
    """Evaluate a mathematical expression."""
    try:
        result = eval(expression)
        return f"Result: {result}"
    except Exception as e:
        return f"Error: {e}"

2) Build the Agent (Planner + Tools)

from adalflow import Agent, Runner
from adalflow.components.model_client.openai_client import OpenAIClient

agent = Agent(
    name="CalculatorAgent",  # Agent identifier
    tools=[calculator],  # List of tools (callables or FunctionTool)
    # LLM client used by the planner (Generator-based)
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o", "temperature": 0.3},
    max_steps=6,  # Upper bound for ReAct loops
)

What this maps to:

• Planner: An internal Generator that decides the next step (think: “reasoning trace”).

• ToolManager: The agent’s registry of permitted tools.

• max_steps: Safety rail to prevent runaway loops.

Model Configuration (Swap Backends Easily)

# OpenAI
from adalflow.components.model_client.openai_client import OpenAIClient
agent = Agent(
    model_client=OpenAIClient(), 
    model_kwargs={"model": "gpt-4o"}
)

# Anthropic
from adalflow.components.model_client.anthropic_client import (
    AnthropicAPIClient
)
agent = Agent(
    model_client=AnthropicAPIClient(), 
    model_kwargs={"model": "claude-3-sonnet-20240229"}
)

3) Execute with the Runner (Multi-step Orchestration)

# Manages turns, tool calls, observations, and finalization
runner = Runner(agent=agent) 

result = runner.call(
    prompt_kwargs={"input_str": "Invoke the calculator tool and calculate 15 * 7 + 23"}
)

print(result.answer)
# -> "The result of 15 * 7 + 23 is 128."

RunnerResult schema (returned by Runner.call)

# result has:
# - result.step_history: [StepOutput(...)]  # Each step’s action + observation
# - result.answer: str | structured type     # Final synthesized answer
# - result.error: None | Exception info      # Error if something failed
# - result.ctx: dict | None                  # Optional execution metadata

This is the full ReAct loop in action:

Plan → Select Tool → Execute → Observe → Iterate → Answer.

Advanced Features (Production-Ready)

1) Streaming Execution (Real-Time Updates)

# Pseudocode: actual API may differ slightly in your version.
stream = runner.stream(
    prompt_kwargs={"input_str": "Compute 42 * 73 and explain."}
)
for update in stream:
    # update contains partial thoughts, tool calls, observations, etc.
    print(update)

Use streaming to surface live reasoning/tool progress in UIs.

2) Human-in-the-Loop (Permission Management)

from adalflow.permissions import PermissionManager

class MyPerms(PermissionManager):
    def approve(self, tool_name: str, args: dict) -> bool:
        # Example policy: only allow calculator; prompt user otherwise
        return tool_name == "calculator"

agent = Agent(
    name="GuardedAgent",
    tools=[calculator, search_tool],
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o"},
    permission_manager=MyPerms(),  # <- Every tool call can be inspected/approved)

Great for tools that hit external systems (files, emails, APIs).

3) Custom System Templates (Planner Behavior)

custom_role_desc = """
You are a careful, step-by-step data analyst.
When you use a tool, explain why and what you expect to get.
"""
agent = Agent(
    name="DataAnalyst",
    # Custom planner persona and guardrails
    role_desc=custom_role_desc,        
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o", "temperature": 0.2},
)

4) Tracing (Observability)

# Configure tracing once (destination: console, file, or tracing backend)
from adalflow.tracing import enable_tracing
enable_tracing(project="adalflow-agents-demo")

result = runner.call(prompt_kwargs={
    "input_str": "Use the calculator for 88*19."
})
# Inspect step_history, tool IO, latency, errors, etc.

Agent Summary:

• Agent = Reasoning + Tool selection (Generator-based planner + ToolManager)

• Runner = Controlled execution loop (steps, tools, observations, final answer)

• Tools = Safe, permissioned extensions to the agent’s capabilities

• Production = Streaming, human approvals, tracing, structured outputs

Real-World Use Case: LinkedIn Recruitment Agent with AdalFlow

Hiring top talent is one of the most resource-intensive parts of building a company. Recruiters spend hours scrolling LinkedIn, opening profiles, copying notes, and crafting outreach messages.

What if we could automate that entire workflow—turning hours of manual searching into minutes of AI-assisted sourcing?

That’s exactly what we built using AdalFlow’s Agent + Runner architecture combined with browser automation via Chrome DevTools Protocol (CDP).

✨ Before vs. After

Traditional Recruiting Workflow (Manual)

❌ BEFORE: 2–3 hours per role

1. Navigate to LinkedIn people search

2. Type in “Product Manager, San Francisco”

3. Scroll endlessly, click into profiles

4. Skim experience, education, skills

5. Take notes in spreadsheets

6. Write & send DMs manually

Automated Workflow with AdalFlow (Agentic)

✅ AFTER: 10 minutes per role

1. Run: linkedin-agent --query “Product Manager” --limit 10

2. Agent plans and executes:

- Smart search strategy

- Extract profiles via browser automation

- Evaluate candidates with scoring models

- Draft personalized outreach messages

3. Get structured output: JSON/CSV with names, titles, LinkedIn URLs, evaluation scores, outreach drafts. Recruiters get to focus on talking to people, not copy-pasting data.

How It Works — Global State Architecture

We structured the solution around a global state shared between tools. Each tool contributes partial data (search results, profiles, evaluations, outreach drafts), which the Agent combines into a full pipeline. Agent combines into a full pipeline.

Implementation with AdalFlow

We implemented the LinkedInAgent by encapsulating:

Agent → Planner + Tools (search, extract, evaluate, outreach)

Runner → Execution loop with error handling and logging

class LinkedInAgent:
    """
    LinkedIn recruitment agent powered by AdalFlow.
    - Encapsulates Agent + Runner
    - Provides default recruitment tools
    - Supports both sync call() and async acall()
    """

    def __init__(
        self,
        model_client: Optional[OpenAIClient] = None,
        model_kwargs: Optional[Dict[str, Any]] = None,
        max_steps: Optional[int] = None,
        role_desc: Optional[str] = None,
        **kwargs,
    ):
        # Defaults
        model_client = model_client or OpenAIClient()
        model_kwargs = model_kwargs or {
           "model": "gpt-4o", “temperature”: 0.3
        }
        max_steps = max_steps or 6

        # Recruitment workflow tools
        self.tools = [
            # 1. Search LinkedIn via CDP
            SmartCandidateSearchTool,
            # 2. Extract structured profile data     
            ExtractCandidateProfilesTool, 
            # 3. Score candidates   
            CandidateEvaluationTool,
            # 4. Draft personalized outreach         
            CandidateOutreachGenerationTool,
            # 5. Persist results 
            SaveOutreachResultsTool,         
        ]

        # Agent role description (personality / instructions)
        role_desc = role_desc or "You are a recruitment assistant that sources and evaluates LinkedIn candidates."

        # Initialize Agent + Runner
        self.agent = Agent(
            name="LinkedInRecruiter",
            tools=self.tools,
            model_client=model_client,
            model_kwargs=model_kwargs,
            max_steps=max_steps,
            role_desc=role_desc,
            **kwargs,
        )
        self.runner = Runner(agent=self.agent, max_steps=max_steps)

    def call(
        self, query: str, context: Optional[Dict[str, Any]] = None
    ):
        return self.runner.call(prompt_kwargs={"input_str": query})

    async def acall(
        self, query: str, context: Optional[Dict[str, Any]] = None
    ):
        return await self.runner.acall(
            prompt_kwargs={"input_str": query}
        )

Full Workflow Execution

Here’s how we stitch the agent into a production workflow:

def execute_search_workflow(
    self, progress_tracker=None) -> List[Dict[str, Any]]:
    logger = get_logger()
    logger.set_workflow_context("workflow_main", "initialization")

    log_phase_start("WORKFLOW_START", f"Target: {self.limit} candidates for {self.query} in {self.location}")

    candidates = []
    try:
        log_info("🤖 Initializing LinkedIn agent...")
        agent, user_query = self.initialize_agent()

        if progress_tracker:
            progress_tracker.start_workflow()

        # Run full pipeline
        result = agent.call(query=user_query)
        self._print_agent_execution_steps(result)

        # Collect data from global state
        from ..core.workflow_state import get_complete_workflow_data
        workflow_data = get_complete_workflow_data()

        candidates = self._build_complete_candidate_data(workflow_data)

        log_info(f"✅ Found {len(candidates)} candidates")
        return candidates

    except Exception as e:
        log_error(f"❌ Workflow failed: {e}")
        return candidates

Example Output

After running:

linkedin-agent --query “Product Manager San Francisco” --limit 10

We get structured results like:

[
  {
    “name”: “Alex Chen”,
    “title”: “Senior Product Manager @ Stripe”,
    “location”: “San Francisco Bay Area”,
    “profile_url”: “https://linkedin.com/in/alexchen”,
    “score”: 0.92,
    “outreach_message”: “Hi Alex, I came across your experience at Stripe...”
  },
  {
    “name”: “Maria Lopez”,
    “title”: “PM, Growth @ Airbnb”,
    “location”: “San Francisco Bay Area”,
    “profile_url”: “https://linkedin.com/in/marialopez”,
    “score”: 0.88,
    “outreach_message”: “Hi Maria, your background in growth product design really stood out...”
  }
]

============================================================

WORKFLOW COMPLETION SUMMARY

============================================================

Success: ✅ Yes

Total Candidates: 2

Duration: 68.8 seconds

Session: 20250909_220617

Log Files:

• Main: logs/workflow_20250909_220617.log

• Debug: logs/debug_20250909_220617.log

• Agent Steps: logs/agent_steps_20250909_220617.log

• Errors: logs/errors_20250909_220617.log

============================================================

🏁 MAIN ✅ COMPLETED - Processed 2 candidates

[RESULTS] ✅ Recruitment workflow completed!

[RESULTS] 📊 Final result: Successfully processed 2 candidates

Looking Ahead

As the LLM landscape evolves, frameworks like AdalFlow will become the backbone of application development. Just as PyTorch accelerated deep learning, AdalFlow has the potential to democratize LLM app building—from chatbots to agents to beyond.

If you’re excited about shaping the future of AI workflows, the project is open-source and community-driven. Whether you’re an AI researcher, product engineer, or just curious about building smarter applications, now’s the time to get involved.

🚀 AdalFlow isn’t just another library. It’s a paradigm shift in how we think about programming with language models.

Signup!

What we are going to build

We are going to build a fun web application where we can demonstrate how to orchestrate a robust RAG application using LangGraph, FastAPI, and React. Here is what we are going to build:

1. A user can pass a GitHub repository URL

2. The files of the related repository are scraped and indexed in a vector database

3. Now the code is available for the user to ask questions about.

On the frontend, we will need two main functionalities:

• A page where we can input the repository URL and start the crawling and indexing processes:

• And a chatbot interface to ask questions about the code in the repository:

On the backend, we will need the related endpoints:

• The indexing endpoint will respond to the provided GitHub repository URL and the “crawl“ action to start the crawling and indexing processes.

• The chat endpoint that will respond to messages sent by the user from the chatbot interface.

We are going to use the following tools:

• React for the frontend

• FastAPI for the backend

• LangGraph for the agentic orchestration

• Pinecone for the vector database

• Langsmith for observability

• Deploy everything on Google Cloud!

Project-based course

We will focus on building the project from the ground up, as we would on the job. Here is how we are going to structure the project development:

• Introduction

  • What we want to build

  • Setting up the environment

• The RAG Application

  • The Data Parsing Pipeline

  • The Indexing Pipeline

  • The Basic RAG Pipeline

  • Adding Observability to the Pipeline with Langsmith

  • Going Agentic

• The Backend Application

  • The Indexing API Endpoint

  • Adding Memory

  • Administering the Database Data

• The Frontend Application

  • The Indexing Page

  • The Chatbot Page

• Deploying to GCP

Each session will be a live, hands-on coding session where we are going to implement every component from scratch

Going Agentic

“Agentic” means that we are going to use an LLM as a decision engine to enhance the quality of our pipeline. We will focus on improving the accuracy of the pipeline at the cost of latency and cost, and discuss the opportunities to reduce those induced negative points with small language models and fine-tuning. In the RAG pipeline, we are going to build a subagent for each of the main components:

• Intent router: the entry point of the pipeline that will decide if a RAG pipeline is required.

  

• The retriever: The sub-agent that will extract the right data

  

• The generator: The sub-agent that will generate the response to the user

  

Scaling up

With this course, I want to focus on what we would need to do to deploy the application to 1M users. We will make sure to design every endpoint to be asynchronous, queue the indexing requests, and deploy the application with elastic load balancing to scale the application horizontally.

This is going to be a fun ride! Make sure to join us!

Signup!

The Real-World AI Engineering Roadblocks You Face Today

👋 Prototype → Production Gap — Moving from a notebook demo to a secure, observable, multi-tenant service requires orchestration, evals, guardrails, and ops most teams lack.

👋 “Easy RAG” vs “Reliable RAG” — Anyone can retrieve-then-generate; making answers faithful, fresh, fast, and cost-controlled under real traffic is the hard part.

👋 Framework Overload — The ecosystem is noisy; you need clear criteria (maturity, extensibility, latency, cost) and reference patterns to choose confidently.

👋 It’s Software Engineering First — Success hinges on clean interfaces, tests, typed configs, tracing, CI/CD, and change management—not just prompts and models.

👋 From Laptop to 1M Users — Scaling demands streaming, batching, caching, autoscaling, and SLOs, or your p95 explodes and costs spiral.

How this course will help you

✅ Ship a real Agentic RAG app, not a demo — Stand up an end-to-end stack—LangGraph → FastAPI → React, that runs locally today and deploys via a clean, fork-and-ship monorepo.

✅ Make retrieval dependable, not lucky — Adopt schema-aware chunking, strong dense embeddings with sensible metadata filters, and context packing with citations so answers stay faithful, fresh, and concise.

✅ Harden agentic workflows — Design a typed LangGraph state and build nodes for rewrite → retrieve → rerank → synthesize → cite → safety-check, with retries and timeouts so plans don’t loop or stall.

✅ Scale the experience, not the headaches — Enable server-streaming in FastAPI, cap top-k, trim context budgets, and add early-exit rules; deploy with autoscaling so you can serve real traffic without infra fuss.

✅ See enough to fix things fast — Bake in structured logs (no vendor tracing), per-step timing counters, and UI breadcrumbs/citations to follow query → context → answer and spot common failure patterns quickly.

✅ Choose frameworks with confidence — Follow an opinionated reference architecture plus a simple choice rubric (maturity, extensibility, latency, cost, swap effort) so you know when to stick—and how to swap components without rewrites.

✅ Write maintainable RAG code — Use clean module boundaries (ingest / retrieve / rerank / synthesize), typed configs (Pydantic Settings), and sensible secrets/env management so your team can extend it safely.

You’ll walk away with

✨ A running Agentic RAG app (LangGraph + FastAPI + React) in a fork-and-ship monorepo.

✨ An ingestion/indexing pipeline with metadata, hybrid retrieval, and optional re-ranking.

✨ A chat UI with citations, source previews, and conversation memory that behaves.

✨ Deploy scripts and env templates to go live right after class.

✨ A framework choice memo + adapters to swap models/vector stores without starting over.

Bottom line: this isn’t a vitamin, it’s a blueprint you can put in production.

What you’ll get out of this course

• Orchestrate complex RAG pipelines with LangGraph and OpenAI API: Build a typed LangGraph that routes rewrite → retrieve → rerank → synthesize → cite → self-check with retries, timeouts, early-exit rules, and real tool calls, exposed as a clean HTTP API.

• Build scalable asynchronous applications with FastAPI: Ship async FastAPI endpoints, well-typed request/response models, input validation, and sensible timeouts, ready to run locally and deploy to production.

• Implement chatbot interfaces with React: Create a chat UI that shows citations and source previews, lets users scope queries, preserves safe chat history, and handles transient API errors gracefully.

• Mitigate hallucinations with LLM judges, structured output, and context engineering: Cut errors via schema-aware chunking, dedupe and budgeted context packing, plus lightweight LLM checks and schema-constrained outputs to verify claims and enforce citations before responding.

• Design effective LLM prompts for high-level control on generation output: Write prompts that steer behavior: system prompts, task decomposition, Pydantic/JSON-schema constraints, and clear rules for tone, citations, and safe refusals.

• Develop end-to-end RAG applications using the software engineering best practices: Produce a maintainable codebase: clean module boundaries (ingest/retrieve/rerank/synthesize), typed configs, secrets/env management, reproducible local dev, and deploy that mirrors local.

Sign Up!

Signup!

What we are going to build

We are going to build a fun web application where we can demonstrate how to orchestrate a robust RAG application using LangGraph, FastAPI, and React. Here is what we are going to build:

1. A user can pass a GitHub repository URL

2. The files of the related repository are scraped and indexed in a vector database

3. Now the code is available for the user to ask questions about.

On the frontend, we will need two main functionalities:

• A page where we can input the repository URL and start the crawling and indexing processes:

• And a chatbot interface to ask questions about the code in the repository:

On the backend, we will need the related endpoints:

• The indexing endpoint will respond to the provided GitHub repository URL and the “crawl“ action to start the crawling and indexing processes.

• The chat endpoint that will respond to messages sent by the user from the chatbot interface.

We are going to use the following tools:

• React for the frontend

• FastAPI for the backend

• LangGraph for the agentic orchestration

• Pinecone for the vector database

• Langsmith for observability

• Deploy everything on Google Cloud!

Project-based course

We will focus on building the project from the ground up, as we would on the job. Here is how we are going to structure the project development:

• Introduction

  • What we want to build

  • Setting up the environment

• The RAG Application

  • The Data Parsing Pipeline

  • The Indexing Pipeline

  • The Basic RAG Pipeline

  • Adding Observability to the Pipeline with Langsmith

  • Going Agentic

• The Backend Application

  • The Indexing API Endpoint

  • Adding Memory

  • Administering the Database Data

• The Frontend Application

  • The Indexing Page

  • The Chatbot Page

• Deploying to GCP

Each session will be a live, hands-on coding session where we are going to implement every component from scratch

Going Agentic

“Agentic” means that we are going to use an LLM as a decision engine to enhance the quality of our pipeline. We will focus on improving the accuracy of the pipeline at the cost of latency and cost, and discuss the opportunities to reduce those induced negative points with small language models and fine-tuning. In the RAG pipeline, we are going to build a subagent for each of the main components:

• Intent router: the entry point of the pipeline that will decide if a RAG pipeline is required.

• The retriever: The sub-agent that will extract the right data

• The generator: The sub-agent that will generate the response to the user

Scaling up

With this course, I want to focus on what we would need to do to deploy the application to 1M users. We will make sure to design every endpoint to be asynchronous, queue the indexing requests, and deploy the application with elastic load balancing to scale the application horizontally.

This is going to be a fun ride! Make sure to join us!

Signup!

Sign Up!

From Prototype to Production: Ship Reliable and Scalable RAG Pipelines

The Real-World AI Engineering Roadblocks You Face Today

👋 Prototype → Production Gap — Moving from a notebook demo to a secure, observable, multi-tenant service requires orchestration, evals, guardrails, and ops most teams lack.

👋 “Easy RAG” vs “Reliable RAG” — Anyone can retrieve-then-generate; making answers faithful, fresh, fast, and cost-controlled under real traffic is the hard part.

👋 Framework Overload — The ecosystem is noisy; you need clear criteria (maturity, extensibility, latency, cost) and reference patterns to choose confidently.

👋 It’s Software Engineering First — Success hinges on clean interfaces, tests, typed configs, tracing, CI/CD, and change management—not just prompts and models.

👋 From Laptop to 1M Users — Scaling demands streaming, batching, caching, autoscaling, and SLOs, or your p95 explodes and costs spiral.

How this course will help you

✅ Ship a real Agentic RAG app, not a demo — Stand up an end-to-end stack—LangGraph → FastAPI → React, that runs locally today and deploys via a clean, fork-and-ship monorepo.

✅ Make retrieval dependable, not lucky — Adopt schema-aware chunking, strong dense embeddings with sensible metadata filters, and context packing with citations so answers stay faithful, fresh, and concise.

✅ Harden agentic workflows — Design a typed LangGraph state and build nodes for rewrite → retrieve → rerank → synthesize → cite → safety-check, with retries and timeouts so plans don’t loop or stall.

✅ Scale the experience, not the headaches — Enable server-streaming in FastAPI, cap top-k, trim context budgets, and add early-exit rules; deploy with autoscaling so you can serve real traffic without infra fuss.

✅ See enough to fix things fast — Bake in structured logs (no vendor tracing), per-step timing counters, and UI breadcrumbs/citations to follow query → context → answer and spot common failure patterns quickly.

✅ Choose frameworks with confidence — Follow an opinionated reference architecture plus a simple choice rubric (maturity, extensibility, latency, cost, swap effort) so you know when to stick—and how to swap components without rewrites.

✅ Write maintainable RAG code — Use clean module boundaries (ingest / retrieve / rerank / synthesize), typed configs (Pydantic Settings), and sensible secrets/env management so your team can extend it safely.

You’ll walk away with

✨ A running Agentic RAG app (LangGraph + FastAPI + React) in a fork-and-ship monorepo.

✨ An ingestion/indexing pipeline with metadata, hybrid retrieval, and optional re-ranking.

✨ A chat UI with citations, source previews, and conversation memory that behaves.

✨ Deploy scripts and env templates to go live right after class.

✨ A framework choice memo + adapters to swap models/vector stores without starting over.

Bottom line: this isn’t a vitamin, it’s a blueprint you can put in production.

What you’ll get out of this course

• Orchestrate complex RAG pipelines with LangGraph and OpenAI API: Build a typed LangGraph that routes rewrite → retrieve → rerank → synthesize → cite → self-check with retries, timeouts, early-exit rules, and real tool calls, exposed as a clean HTTP API.

• Build scalable asynchronous applications with FastAPI: Ship async FastAPI endpoints, well-typed request/response models, input validation, and sensible timeouts, ready to run locally and deploy to production.

• Implement chatbot interfaces with React: Create a chat UI that shows citations and source previews, lets users scope queries, preserves safe chat history, and handles transient API errors gracefully.

• Mitigate hallucinations with LLM judges, structured output, and context engineering: Cut errors via schema-aware chunking, dedupe and budgeted context packing, plus lightweight LLM checks and schema-constrained outputs to verify claims and enforce citations before responding.

• Design effective LLM prompts for high-level control on generation output: Write prompts that steer behavior: system prompts, task decomposition, Pydantic/JSON-schema constraints, and clear rules for tone, citations, and safe refusals.

• Develop end-to-end RAG applications using the software engineering best practices: Produce a maintainable codebase: clean module boundaries (ingest/retrieve/rerank/synthesize), typed configs, secrets/env management, reproducible local dev, and deploy that mirrors local.

  Sign Up!