Persistent agent memory tools now let applications carry stored information across sessions and update it, making the quality of that update consequential for later tasks. The question is: when a user changes a standing preference, what must happen so an agent acts on the new one? A useful memory system treats the new message as a proposed change to stored state. It identifies the fact, finds the existing fact about the same thing, reconciles the two, commits the active version, and retrieves that version when the next task needs it.

That differs from appending messages to a transcript. We can see why by following one preference from its first instruction through a correction and a later task. The same failure can affect a project decision or customer preference.

What changes when an agent has memory?

The model receives a limited working context with each request. A conversation transcript can be placed in that context, but it grows, contains incidental remarks, and may contain claims that no longer hold. Persistent memory here means an external store that keeps selected information beyond one request or session. A write operation changes that store; a later read returns a useful part of it to the model. The model's weights are not being retrained every time a preference changes.

Suppose a user tells a coding assistant, “For the Atlas repository, write deployment scripts in Python.” The assistant might store a compact record with four parts: the user, the Atlas repository, the subject deployment-script language, and the current value Python. The message that supplied the preference is its source. This is more useful than an unqualified memory such as “prefers Python,” because the latter could affect unrelated work.

The store has two distinct paths. On the write path, a message or tool result is examined for information worth retaining, then compared with what is already stored. On the read path, the next task retrieves applicable memory and places it where the model can use it. If the write path is wrong, fast retrieval brings back a mistake. If the read path misses the right record, a perfect update sits unused.

Now the user says, “For Atlas, use Go for deployment scripts from now on.” Adding that sentence as a second, equally active memory would leave the future assistant to arbitrate a contradiction each time. A better result is an active Atlas deployment-language preference of Go, with the former Python preference no longer active. The old statement may remain in an audit history, but it should not be presented as another current instruction.

Remembering a correction means changing the state that future reads see, not merely remembering that a correction was said. The change has to preserve the subject and scope of the fact, or it will solve one task by creating a wider error.

Read more

Prefix-aware routing became a managed endpoint option in September 2026, a sign that cache locality has moved from an engine detail into a deployment decision.

So where should the next request go: to the warm replica or the idle one? The compact answer is to send it where the saved prompt computation is worth more than the extra work already waiting there. A useful router therefore needs two views at once: what can be reused and what must still be processed.

To see why, start with the work a cached prefix actually removes. Then we will keep one support prompt fixed while four routing policies make different placement decisions. That comparison will expose the useful rule, its failure modes, and the state a production router has to keep current.

What a warm prefix actually saves

Before routing makes sense, we need one piece of transformer serving machinery.

An incoming prompt is a sequence of tokens. During prefill, the model processes those input tokens through every transformer layer. In each attention layer it derives a key vector and a value vector for every position. Those vectors are the processed state that later tokens attend to. The first generated token can appear only after this prompt work has finished.

During decode, the model generates one new token, appends its new key and value state, and repeats. Decode can reuse the state for all earlier positions instead of rebuilding it. The stored collection is called the KV cache.

Now consider a support assistant whose prompt begins with 3,200 tokens of policies, tool descriptions, and response rules. Only the short customer question at the end changes. If the engine has already processed the same beginning, it can reuse the matching KV state and prefill only the new suffix.

Prefix caching does not mean that the model remembers a similar idea. The match is structural. Serving engines divide the prompt into token blocks, identify a block using its tokens and the blocks before it, and reuse only an unbroken matching prefix. If block three changes, later blocks no longer describe the same model state, even if much of their text looks familiar.

That distinction gives prefix caching an honest boundary. It skips model computation already performed for the input prefix. It does not skip the generation of new output tokens, because those tokens do not exist yet. A workload with long repeated inputs and short answers can gain a lot; a workload with unrelated prompts or very long answers may gain little.

Within one replica, the rule is simple: reuse every valid prefix block that is still resident, prefill the remaining input, then decode normally.

Read more

That distinction is now practical because current OpenAI and Anthropic APIs expose reasoning-effort controls while Google offers an Extended Thinking model for complex live-agent tasks, turning “let it think” into a quality, latency, and cost decision for each request.

The useful question is not simply, “How much thinking should I buy?” It is: where should the extra compute go? Test-time scaling allocates it to depth, width, or feedback. Depth extends one attempt. Width creates separate attempts. Feedback evaluates work and changes the next attempt. Each repairs a different failure, and none guarantees correctness.

What test-time compute actually is

Start with an ordinary request. Training is already over, and the model begins with learned weights that stay fixed while it answers. At each step, the model processes the request and the tokens generated so far to choose the next token. That ordinary generation already consumes test-time compute: computation performed after training for one particular input.

The term covers more than the model's hidden reasoning tokens. It includes the work the model and its surrounding system perform to produce, inspect, and improve this one answer. Continuing the generation adds more token-level operations to the current history. Starting another generation creates a separate candidate history. Running a test or verifier creates an observation that a later attempt can use. Revising with that observation performs another generation under changed conditions.

Test-time scaling is the deliberate decision to increase or reallocate that per-request work relative to a baseline answer. The extra work can matter because it changes the transient information available to a later choice: a longer prefix, a genuinely different candidate, or new evidence about an error. If the added work does not change that state or evidence, it may increase latency and cost without improving the answer.

This is not training or fine-tuning. Training changes reusable model weights for future requests. Test-time compute leaves those weights alone and is paid again for each request. It is also not synonymous with selecting a larger model or producing a longer visible answer. A system may spend substantial compute on candidates, checks, and revisions that never appear in the final response.

We will follow one realistic coding task through all three choices. An agent must repair invoice_total, a function that sums prices, applies tax, and returns integer cents. Its contract requires exact decimal arithmetic, an 8 percent tax applied after aggregation, round-half-up once to the nearest cent, then conversion to integer cents. The visible invoice contains items priced at $12.49, $7.25, and $3.10. Hidden tests vary the inputs to expose half-cent rounding, multiple-item accumulation, zero tax, and binary floating-point drift.

First we need to see why one uninterrupted generation can fail. Then we can spend the same request-time budget on a deeper path, a wider candidate set, or a feedback loop and see what actually changes.

Before scaling, identify the one-shot failure

An autoregressive language model generates a response one token at a time. At every position it receives the request plus the tokens already written, then assigns probabilities to possible next tokens. Once a token is chosen, that token becomes part of the state used to predict everything that follows.

This left-to-right process matters because an early choice can narrow the rest of the answer. Suppose our coding agent begins with the assumption that converting dollars to cents is just int(total * 100). It can write an elegant explanation and many consistent lines of code after that. The later tokens are coherent with the earlier assumption, but coherence does not repair the assumption.

A one-shot attempt is one uninterrupted generation from request to answer. It may spend many internal reasoning tokens, but it has no separate alternative to compare with and no external result that changes its course. Its success depends on the model entering a useful line of reasoning and staying there.

For this lesson, a useful accounting unit is a generated reasoning token, a candidate response, a verifier evaluation, or a tool call such as a test run. These units have different hardware costs, so we will not pretend they are perfectly interchangeable. The important constraint is that the system has a finite request-time budget and must decide how to allocate it.

There are three places to allocate that budget:

1. Depth: continue or enrich the same reasoning trajectory.

2. Width: sample several separate trajectories from the same starting request.

3. Feedback: inspect work, then condition the next attempt on what the inspection revealed.

The distinction is about information flow. Extra depth sees the same request and its own growing prefix. Width gives each attempt the request but not the siblings' work. Feedback adds a new observation, such as “the half-cent test failed,” before the next generation. Width is logical branching; whether those branches run simultaneously is a separate serving decision.

That gives us a diagnostic before we spend anything. If the model merely stops before completing a sound plan, buy depth. If different plausible starts lead to different outcomes, buy width. If a checker can reveal what went wrong, buy feedback. A high effort setting may combine these internally, but an application designer still needs this mental model to understand latency, cost, and failure.

Depth gives one trajectory room to finish

The simplest scaling strategy is to let one attempt run longer. The model can decompose the task, inspect edge cases, and carry intermediate facts farther before committing to the final response. In our invoice task, a deeper trajectory must preserve a specific dependency: represent the prices exactly, sum them to $22.84, apply the 8 percent tax to obtain $24.6672, round that total once to $24.67, then return 2467 cents.

The benefit is continuity. Every later step sees the reasoning accumulated earlier. Here, the final integer conversion only makes sense after the representation and rounding policy are settled. Once an earlier step truncates a fraction of a cent, no later step can reconstruct it. Proofs, long calculations, and code changes with dependent invariants have the same shape.

But depth is not the same as recovery. If the trajectory commits early to binary floating point and treats a failing cent as harmless noise, more tokens can reinforce the wrong frame. The model may repeatedly check arithmetic inside a mistaken representation. It is thinking longer, but within a shrinking part of the solution space.

That gives us the first useful rule: depth helps when a sound path is unfinished, not merely when a path is wrong. A model that often reaches the correct approach and then runs out of room is a good candidate for higher effort. A model that confidently commits to a bad representation needs an alternative or new evidence, not an even longer monologue.

Read more

Normally, a language model can choose any token in its vocabulary at each generation step. Constrained decoding inserts a rule checker into that loop. The checker looks at the text already generated and removes every next token that would make the requested structure impossible to complete. The model then chooses among what remains.

Structured output has become a common connection layer between LLMs, extraction pipelines, and agent tools, yet a recent developer discussion exposed the practical surprise that matters here: a perfectly shaped object can still contain invented values, so schema-valid is not the same as safe to execute.

The central question is simple: what does the decoder actually guarantee? The compact answer is that it can guarantee membership in a supported output language, such as a JSON Schema, by blocking illegal continuations token by token. It does not prove that a legal value came from the input, reflects reality, satisfies a business rule, or deserves permission to trigger an action.

We will follow one support request through the complete mechanism. The request says: Please refund order A-17. The charge was $42.50. The desired output is a small object containing a status, an order identifier, and a refund amount in cents. First we need to separate three ideas that are often collapsed into the word “valid.” Then we can compile the schema, watch the token mask change, and see exactly where truth leaves the decoder's jurisdiction.

Three different promises hide inside “valid JSON”

JSON is a text format. A JSON parser checks punctuation and nesting: braces balance, keys are quoted, commas occur in legal places, and values have valid JSON forms. This object is valid JSON:

{"status":"ready","order_id":"A-17","refund_cents":4500}

A schema asks a narrower question about the parsed value. It can require an object, name its allowed properties, require selected keys, restrict a field to an integer, or limit a value to an enumeration. If our schema allows any non-negative integer for refund_cents, then 4500 passes. The schema has no memory of the user's $42.50.

Finally, the application asks whether the object means something acceptable in its real environment. Does order A-17 exist? Was the charge really $42.50? Is it refundable? Is this user authorized? Those are evidence, policy, and permission questions. They are not punctuation questions.

The checks therefore form a hierarchy:

1. Parseable means the text is legal JSON.

2. Schema-valid means the parsed value also obeys the declared structural rules.

3. Grounded and actionable means the values are supported by evidence and pass application policy.

Each level can remove failures that the previous level accepts. None can be silently substituted for the next.

Our refund extractor also needs a way to say that the message is incomplete. If a successful decode must complete a contract where every field is required and refund_cents must be an integer, every successful path puts some integer there even when no amount appears in the input. A better output contract exposes two branches:

{ "anyOf": [ { "status": "ready", "order_id": "string", "refund_cents": "non-negative integer" }, { "status": "insufficient_info", "order_id": null, "refund_cents": null } ] }

This is a readable sketch of the contract, not a literal complete JSON Schema. The important move is structural: uncertainty is now a legal output rather than an instruction the model must squeeze into fields that forbid it.

Here is the first design rule worth keeping: a schema should represent the failure states your application expects. Constrained decoding can only choose paths you make legal. If the contract requires a confident answer, the decoder cannot invent an abstention branch on your behalf.

Read more

That first decision is the hidden trade-off behind sparse attention. The name suggests that the model merely turns down unimportant history. In many designs, it does something more consequential: it prevents most of the history from entering the main attention calculation at all.

A late-August 2026 Transformers release added Qwen Sparse Attention, while an open-weight Qwen model built on the same block-selection idea prompted immediate questions about long-context latency and cache placement; as more models save work this way, the durable question is which evidence they stop reading.

So here is the question we will answer: When sparse attention makes a long context cheaper, what does an LLM stop looking at?

The compact answer is that every attention pattern creates an eligible set of earlier token states. Full attention admits the whole past. A sliding window admits only nearby history. Content-selected sparse attention uses a cheaper scoring step to choose a small subset that can include distant history, then runs ordinary attention on that subset. Fewer eligible states mean less work, but a clue excluded at this point cannot be rescued by a larger attention weight later.

We will follow one question through all three patterns. A long incident report contains the deployed database proxy setting, port 6432, near the beginning. Much later, a diagnostic note mentions the default database port, 5432. At the end, the reader asks: Which port did the incident's database proxy actually use?

This is a useful test because the answer is a tiny exact value separated from the question by a lot of plausible noise. We will hold the tokens, model layer, and question fixed. Only the rule that exposes earlier states to attention will change.

Attention makes two decisions, not one

An LLM processes text as tokens, which are words or pieces of words converted into numerical states. At one layer, every token state is transformed into three useful roles.

Every token produces all three. When the layer updates the current token, its query represents what information it is seeking. The current token and earlier tokens provide keys, which can be compared with that query, and values, the information that can be carried forward. The query-key comparisons become weights; the weighted values become the context added to the current token's state. The diagram focuses on earlier sources, but causal attention also allows the current token to attend to itself.

There is a prerequisite hiding before those comparisons. An attention rule supplies a mask or selection pattern that says which keys and values are eligible. A future token is ineligible in a causal language model because generation cannot look ahead. Sparse patterns add more exclusions among the past.

It helps to separate two questions:

1. Eligibility: may the current token connect to this earlier state?

2. Importance: among eligible states, how much weight should this one receive?

Full, windowed, and content-selected attention mainly differ on the first question. Once a subset is eligible, the main attention operation can still compare queries with keys and mix values in the familiar way.

Apply that distinction to our incident report. Full attention places both 6432 and 5432 in the candidate set. A four-block sliding window includes only the late diagnostic note and loses the deployed setting. A good content selector reaches back to the configuration block containing 6432, while ignoring most of the middle.

The model still has to interpret the candidates correctly. Full attention could overweight the distracting 5432. A selector could choose the wrong block. Eligibility does not guarantee a correct answer; it defines which answer evidence remains possible at this layer.

That is the first complete mental model: sparse attention saves work before importance is calculated. The rest of the article explains how each eligibility rule buys its saving, and the distinct miss it creates.

Read more

The surprising part is that a longer context window does not remove this decision. Each retained frame expands into many pieces the model must process, each second of audio adds more, and the relationships across time still have to be computed. Keeping more evidence can preserve a brief event, but it can also bury that event inside thousands of irrelevant moments.

Google's September 2026 agentic video mode now lets a model search a timeline and adjust frame rate, resolution, and modality around promising moments, making evidence selection a practical product choice rather than a hidden preprocessing detail.

So the useful question is: How can an AI answer a precise question about hours of video without placing every frame in context?

The compact answer is to treat long video as a budgeted search for evidence. Fixed-rate sampling buys predictable coverage. Change-aware sampling keeps moments when the picture moves or cuts. A temporal hierarchy narrows a long recording to a few candidate intervals. Query-adaptive rewatching then spends dense frames, higher resolution, or audio only where the question needs them.

We will follow one realistic request through all four techniques. A warehouse camera records aisle 7 for two hours. A worker enters at 10:00. A red forklift first touches shelf B at 10:04.3, then backs away. The question is not merely whether both objects appeared. It is: When did the forklift first touch the shelf after the worker entered?

That wording matters. The answer needs an identity, an order, and a brief physical relation. A transcript may confirm that a worker called for help. A single frame may show the forklift beside the shelf. Only a short sequence can establish the first contact.

A video becomes a budgeted evidence packet

A frame is one still image from the video timeline. In a common transformer-based path, a vision encoder splits that image into patches and converts the patches into numerical representations the language model can process. I will call those representations visual tokens. One frame is therefore not one token; it expands into a small field of spatial evidence.

Video adds a second requirement. The model must connect what appears in one frame with what changes in later frames. That temporal relationship distinguishes "forklift near shelf" from "forklift moves toward shelf, makes contact, then reverses." The model's attention mechanism can compare retained visual tokens across time, but it cannot reconstruct a moment that the input pipeline never kept.

Audio and transcripts form separate evidence streams. Audio preserves nonverbal signals such as a metallic impact. A transcript is cheaper to search and excellent for spoken names or instructions, but it omits silent motion. Timestamps align the streams so a clue in one can point to a region in another.

Now imagine sampling the warehouse video once each second. The system keeps frames at 10:04.0 and 10:05.0. The actual touch at 10:04.3 may fall between them. Raising the rate to ten frames per second makes the event much more likely to appear, but it multiplies the visual evidence for the entire two-hour recording.

This is the first useful insight: temporal detail is purchased with context. The family of techniques differs mainly in who decides where to spend that detail, and when.

Read more

The surprising part is that an exact copy into a plain-text document can preserve the evidence. There is no hidden character to carry along. The ordinary words preserve the choices.

If you use AI to draft, proofread, or rewrite, the distinction matters: those operations leave different amounts of freedom for a watermark. And when someone says a passage was “detected,” the test matters too. A keyed watermark check measures a deliberately introduced pattern; it does not infer AI use from a writer's style.

Read more

In the two-model version we will follow, the target still does its full computation. The saving comes from arranging that computation differently, so one pass through its layers can advance the answer by several tokens. Whether that saves time depends on how much of the draft survives.

Read more

We'll also look inside speculative decoding and text watermarking, then turn to two visual problems: why a generated image can look convincing while missing the instructions, and why a fluent document answer can still come from the wrong cell.

1. Model validation: a random split can preserve the shortcut

Can machine learning validation approve a model that learned the wrong feature? It can, if the validation set repeats the same shortcut!

Imagine a pneumonia classifier trained on chest X-rays from two hospitals. Hospital A contributed most of the positive cases. Hospital B contributed most of the negative cases.

Each image contains two usable signals: the lung pattern we care about, and a hospital fingerprint from scanner processing, text markers, or acquisition style. The loss function does not know which explanation we intended. It rewards features that predict the label. If recognizing the hospital offers an easier route to lower loss, the model can use it.

Now randomly split the rows into training and validation sets. Both sets still contain the same hospitals and the same correlation: Hospital A usually means positive; Hospital B usually means negative. Randomization changes which rows land in each set, but it does not change the process that produced them.

The validation images are unseen, yet the shortcut is familiar. A high score can accurately describe performance on this source mixture and still mislead us about deployment.

Move the model to Hospital C. In this toy example, the disease signal remains valid while the old hospital-label correlation disappears. Prediction quality can drop without any bug in the inference code. The diagram follows that change from collection to deployment.

A random row split estimates performance on another row from the same source mixture. It says little about a source absent from that mixture. When the expected novelty is a hospital, camera, customer, region, time period, or device, holding out the entire unit lets validation test that change instead of scattering the unit across both sides.

I would then ask whether the model is using the particular cue I suspect. Keep the target content fixed, remove or swap the background, marker, source template, or metadata, and watch whether the prediction moves.

These tests answer different questions. A group holdout tests an environment change. A cue intervention tests dependence on one named feature. Neither proves that the model learned the intended signal: the holdout covers only the environments we chose, and the intervention is trustworthy only if it changes the shortcut without changing the task itself.

2. Agent reward hacking: passing the test while leaving the bug

Why can AI agent reward hacking produce a perfect score on a task the agent never completed? The reason is simple: the agent optimizes the feedback we expose, not the intention behind it.

Consider a coding agent asked to fix an invoice calculation. The correct total is $110, but the code returns $100. We use the test suite as the evaluator: a pass earns a reward of +1, and a failure earns 0.

During reinforcement learning, the agent tries a sequence of actions and receives this reward. The optimizer makes high-reward action sequences more likely in similar situations.

We expect the agent to fix the calculation, pass the tests, and produce an invoice for $110. But the test suite is also part of the environment. If the agent can modify it, another path becomes available: disable the assertion, pass the tests, and leave the invoice at $100.

Both paths receive +1. The second path can be reinforced because the remaining failure was never reflected in the score.

Three different objects matter here:

• The true objective is a correct invoice.

• The proxy reward is the +1.

• The evaluator is the test suite that decides when to issue it.

Exploiting a missing condition in the proxy is usually called specification gaming. Changing the evaluator itself, by editing a test, checklist, log, or scoring process, is the more specific case of reward tampering.

No human-like intention to cheat is required. From the optimizer's point of view, the two action sequences received the same reward. The number does not contain the missing requirement, so we cannot recover it from the score afterward.

Greater capability does not settle this problem. Better search can find the correct implementation, but it can also find shortcuts that a weaker agent never discovered. It expands both sets of paths.

A wrong answer with a low score is ordinary failure. Reward hacking is narrower: the measured score says success while the intended outcome stays unchanged or gets worse. The test suite is another piece of the environment, and its assumptions shape what the agent learns.

3. Durable agents: a checkpoint cannot confirm an external payment

How can durable execution for AI agents survive a crash and still charge the same invoice twice? The checkpoint remembers where to resume, but it cannot prove whether the outside world already changed!

Imagine an agent paying a $480 invoice. It records step pay-1 as pending, sends the request, and the payment API accepts it. Then the worker crashes before saving the receipt.

After restart, the execution state still says:

run-42 -> pay-1 -> PENDING

From that saved state alone, the replacement worker cannot tell whether the payment failed or only the acknowledgement failed. We know the API accepted the payment in this example; the worker has no saved receipt to establish that. Retrying may be necessary, but the tool step may execute again.

An idempotency key is a stable identifier that tells the external service that a retry represents the same logical operation. The checkpoint identifies the unfinished step; the key identifies the one external operation that step is allowed to create.

The diagram uses run-42/pay-1 as the key, recorded before the first request. The replacement worker reloads the latest checkpoint, skips completed work, and sends pay-1 again with that same key. If the payment service supports idempotency and already processed the key, it returns the original receipt instead of creating another payment. The runtime then saves the receipt, marks the step completed, and continues.

This is why conversation memory and execution state deserve separate treatment. Conversation memory can hold messages, facts, and user preferences. Execution state needs the run ID, step ID, exact tool arguments, status, approvals, timers, and durable receipts. A sentence saying "the invoice was paid" is not a transaction record.

The same pattern appears outside payments. A database can reject a duplicate operation through a unique constraint or conditional write. An event consumer can deduplicate using a stable event ID.

If a tool has no idempotency mechanism, blind retry is the wrong recovery strategy. The agent has to query the external system, run a compensating action that reverses or offsets the first effect, or stop for human review.

Durable execution preserves progress. It does not make the model correct, permissions safe, or an arbitrary API idempotent. Once an agent can wait for humans, call tools, and run longer than one request, its execution state matters as much as its prompt.

4. Speculative decoding: draft several tokens, verify in one pass

How can speculative decoding use a smaller language model without inheriting its mistakes? The small model guesses a few tokens ahead, but the large target model verifies the whole guess in one pass and keeps control of the output distribution.

The usual language-model decoding loop is serial. The target model predicts one token, appends it to the sequence, then runs again. Producing K tokens requires K target-model passes.

During low-batch decoding, a pass often spends more time moving the model weights than using all the available arithmetic. Scoring a short block can reuse each weight read across several token positions, so it can take close to the time of scoring only one.

Speculative decoding takes advantage of that opportunity. In the common two-model version, a cheap draft model generates a short continuation one token at a time. The large target model receives the prompt and complete draft, then scores every drafted position in parallel.

Acceptance still proceeds from left to right. Accepted tokens become final. At the first rejection, every later draft token is discarded because it was conditioned on a prefix the target did not accept. The target samples a correction, and the next drafting round starts from the corrected prefix.

For the prompt "Water freezes at", imagine the draft proposes:

"0" -> "degrees" -> "Kelvin" -> "."

One target pass can score all four positions. It may accept "0" and "degrees", reject "Kelvin", discard the period, and sample "Celsius" as the correction. That slow pass has finalized three useful tokens instead of one.

What I find most interesting is how this works for sampled decoding. We cannot simply keep a token when two independent samples happen to match. If draft and target were identical uniform distributions over 100 tokens, their separate draws would match only 1% of the time, despite perfect agreement between the distributions.

Call the target distribution p and the draft distribution q. A drafted token x is accepted with probability:

min(1, p(x) / q(x))

If it is rejected, the correction is sampled from the normalized positive part of p - q. To build that distribution, keep the positive differences between target and draft probabilities, set the others to zero, and rescale the remaining values so they sum to one. This is the target probability not already covered by the draft.

Together, the accepted path and correction path reconstruct p. Under this rule, the draft changes latency without biasing the target distribution.

The benefit depends on the whole serving setup. A slow draft adds too much overhead. A draft with little overlap with the target causes frequent rejections. Hardware that is already compute-saturated has little parallel capacity left to exploit.

The useful draft is the one whose acceptance rate is worth its cost on the target model and hardware being served. Accuracy of the small model alone does not determine that tradeoff.

5. Text watermarking: small token preferences add up to evidence

How can generative text watermarking mark a paragraph without adding hidden characters, metadata, or a visible tag? It changes ordinary token choices, then looks for the accumulated pattern!

At each decoding step, a large language model (LLM) produces a probability distribution over the next token. Several continuations may fit. A normal sampler chooses among them; a watermarking sampler uses one additional input, a secret key.

One common construction uses the key and recent tokens to pseudorandomly divide the vocabulary into green and red sets. Plausible green tokens get a small boost. The partition is recomputed after every token, so there is no fixed list of suspicious words.

Consider a toy passage with 100 positions counted by the detector. Without the watermark, we expect roughly 50 green hits by chance. A watermarked sampler might produce 70. No individual token looks unusual; the pattern appears when we aggregate the choices.

For this simplified 50/50 construction, a one-proportion test gives:

z = (70 - 50) / sqrt(100 × 0.5 × 0.5) = 4

The numerator is the 20 extra green hits. The denominator is 5, the standard deviation of the count under this simplified chance model. The observed count sits four standard deviations above the expected count.

Other watermark families use different scoring rules. Here, the detector tokenizes the passage, uses the same key to recreate the green and red sets at each position, and tests whether the excess of green tokens is too large to explain by chance. It measures a statistical pattern rather than judging whether the prose sounds robotic.

Length supplies evidence. A ten-token answer provides very few weak votes; a long passage supplies many more observations. The model also needs freedom to choose. Open prose may offer several reasonable continuations, while code or familiar fixed phrases can have one dominant next token, leaving little room to embed a preference without damaging the output.

Editing weakens the signal without predictably erasing it. Replacing a token removes one marked choice. In context-dependent schemes, it can also change the expected partition at positions that use the edited context. A substantial rewrite can reduce the evidence dramatically, yet a long paraphrase may preserve enough unaltered fragments to remain detectable.

A positive result means the text is statistically consistent with one key's watermarking rule. It does not prove who wrote the ideas, and it cannot identify output from a model that never applied that mark. The detector threshold still trades missed detections against false positives.

That is why a long, open-ended answer can carry strong watermark evidence while a short code completion from the same model may carry almost none.

6. Image generation: realism and exact composition are different tests

AI image generation can give you a beautiful campaign image and still ignore the requirement that makes it usable! The problem is that a text prompt is not an engineering drawing.

Consider this request: "three green bottles to the left of one orange box, with FRESH on the label."

One sentence carries five kinds of constraints: which objects exist, how many of each, which attributes belong to which objects, where they go, and which exact characters appear.

In many diffusion systems, a text encoder converts the prompt into vectors. Cross-attention lets the denoising network consult those vectors while turning noise into an image. A few words can influence a large visual field without someone hand-coding the scene.

But the vectors do not arrive as an explicit object table with separate entries for color, count, and coordinates. During denoising, "green", "bottles", "three", and "left of" must all influence one shared visual representation. The model can form a convincing poster while giving you four bottles, coloring the box green, or letting objects overlap.

Realism and prompt compliance measure different things. Bottles, an orange box, and text-like marks can all be present while the requested relationships drift. Counting, relative position, and attaching an attribute to the correct object are compositional challenges.

Saying that image models "cannot count" misses the larger problem. Individually recognizable concepts have to stay attached to the right instance and location throughout generation.

Sketches, layouts, segmentation maps, and image checks help make some intended state explicit: where an object belongs, how many are needed, and whether a word is exact. The diagram separates guidance from checking: its layout specifies the bottles' positions, while the checks inspect count, position, and the characters in FRESH. These aids do not guarantee a correct result, but they stop the prompt from carrying every constraint alone.

A creative mood board benefits from the model filling gaps. An asset with exact copy, object counts, or placement can fail precisely when the model fills in a gap you intended to specify.

7. Document extraction: connect the answer to the right cell

AI document extraction can give you a perfectly grammatical answer that comes from the wrong cell in the document! The reason is that a vision-language model is doing several different jobs before it ever starts reasoning about your PDF.

Start with a scanned bank statement:

• Opening balance: $14,900

• Deposits: $2,200

• Closing balance: $17,100

Ask, "How much was deposited?"

The model does not receive the document as a clean database row. A vision encoder turns pixels into a limited set of visual tokens. Small type, a low-quality scan, and dense tables can make the evidence for a word or nearby number weak or mixed. Some PDF pipelines pass native text separately, which changes the failure mode, but charts, tables, handwriting, and rendered layout still require visual interpretation.

Then comes binding: "Deposits" must be linked to $2,200. Recognizing all three amounts somewhere on the page is insufficient. The language model uses the visual representations and the question to generate its answer token by token.

If the deposit label was unclear, or the label-to-value association was wrong, "$17,100" can still be a plausible completion. Fluency tells us the decoder completed a sentence well. It does not tell us which pixels supported the answer.

I separate document extraction into three questions:

1. Did the system read the relevant glyphs?

2. Did it preserve the relationship between label, row, column, and value?

3. Did the answer come from that evidence, or from a plausible completion?

Optical character recognition (OCR) and coordinates can make the first two steps inspectable. A vision-language model is useful when the final answer needs a comparison, interpretation, or calculation.

More visual resolution can improve fine detail, but it also costs tokens and latency. It does not automatically recover a missing label-to-value relationship.

Chart questions have the same difficulty. Reading "Q4" and "47" is insufficient: the system has to connect the right series, axis, and visual mark before it can reason usefully about them.

In this guest post, Aria Shi, the Developer Relations lead at SylphAI, walks you through how AdalFlow empowers AI Agent development, highlighting a hands-on example with a LinkedIn Reachout Agent.

Say goodbye to manual prompt engineering. AdalFlow is the all-in-one, auto-differentiative solution for optimizing prompts, whether you’re using zero-shot or few-shot learning. Backed by our state-of-the-art research (LLM-AutoDiff and Learn-to-Reason), our framework achieves the highest accuracy among all automatic prompt optimization libraries.

The rise of large language models has completely changed the way we build applications—whether it’s chatbots, RAG systems, or fully autonomous agents. But as an AI engineer, trying to bring these models into production often feels like stitching together a bunch of experiments, rather than building a stable and reliable system.

We introduce AdalFlow: a PyTorch-like library designed to bring structure, clarity, and optimization to the world of LLM application development. Built as a community-driven project, AdalFlow is uniting AI research and production engineering into a single ecosystem.

Why We Built AdalFlow

Modern AI development faces a paradox. On one hand, researchers push the boundaries of model capabilities with new techniques in prompting, evaluation, and optimization. On the other hand, production teams need reproducibility, scalability, and a way to iterate safely on real-world data.

Most libraries excel at one side of the equation but leave the other underserved. AdalFlow was born to bridge this gap. With 100% control and clarity of source code, it empowers researchers to experiment freely while giving product engineers the tools to build and ship with confidence.

Why AdalFlow Matters

By treating prompts as first-class citizens and introducing LLM-AutoDiff, AdalFlow provides what’s been missing in the LLM ecosystem:

• For researchers: A familiar PyTorch-like environment to prototype new prompting and training methods.

• For engineers: Production-ready workflows that are debuggable, reproducible, and optimizable.

• For teams: A shared framework that unites research and production into one healthy ecosystem.

Core Philosophy: Prompt Is the New Programming Language

If PyTorch turned tensors into the lingua franca of deep learning, AdalFlow treats prompts as the new programming primitives.

Every LLM application boils down to structured prompts and their transformations. AdalFlow embraces this reality by making prompt engineering explicit and optimizable. Behind the scenes, it uses the Jinja2 templating engine to let developers define composable prompt structures, ensuring that LLM apps are both modular and debuggable.

Components: The Building Blocks of LLM Workflows

At the heart of AdalFlow lies the Component abstraction. Just as nn.Module became the foundation for PyTorch models, Components unify every stage of an LLM pipeline.

• Component: The base class for all workflows. Handles both training (forward) and inference (call) modes, with bicall bridging the two.

• GradComponent: Components capable of backpropagation (e.g., Generators, Retrievers).

• DataComponent: Lightweight components for formatting and parsing data (e.g., DataClassParser).

• LossComponent: Wraps evaluation metrics and enables gradient-like feedback for text optimization.

Example 1: Q&A with Object Counting (Component + DataComponent)

template = r"""<START_OF_SYSTEM_PROMPT>
{{system_prompt}}
<END_OF_SYSTEM_PROMPT>
<START_OF_USER>
{{input_str}}
<END_OF_USER>"""

@adal.func_to_data_component
def parse_integer_answer(answer: str):
    numbers = re.findall(r"\d+", answer)
    return int(numbers[-1])

What’s happening here?

• parse_integer_answer is wrapped with @adal.func_to_data_component.

• This turns a plain Python function into a DataComponent, which handles structured output parsing.

• In this case, it ensures the model’s answer ends with a numerical value.

Next, we define a full pipeline:

class ObjectCountTaskPipeline(adal.Component):
    def __init__(
            self, model_client: adal.ModelClient, model_kwargs: Dict
        ):
        super().__init__()
        system_prompt = adal.Parameter(
            data="You will answer a reasoning question. Think step by step. The last line should be 'Answer: $VALUE'.",
            role_desc="Task instruction for the model",
            requires_opt=True,
            param_type=ParameterType.PROMPT,
        )
        self.llm_counter = adal.Generator(
            model_client=model_client,
            model_kwargs=model_kwargs,
            template=template,
            prompt_kwargs={"system_prompt": system_prompt},
            output_processors=parse_integer_answer,
        )

    def bicall(self, question: str, id: str = None):
        return self.llm_counter(
              prompt_kwargs={"input_str": question}, id=id
        )

ObjectCountTaskPipeline subclasses Component. Inside it, we define:

• A Parameter of type PROMPT, which AdalFlow can later auto-optimize.

• A Generator (a GradComponent) that executes the prompt, then passes the raw LLM output through our parse_integer_answer DataComponent.

The workflow is:
Prompt → LLM Generation → Structured Output Parsing → Final Numerical Answer.

Example 2: Classification with Structured Output (Component + DataClass)

Classification tasks are a perfect showcase of AdalFlow’s DataClass feature.

@dataclass
class TRECExtendedData(adal.DataClass):
    question: str = field(
        metadata={"desc": "The question to be classified"}
    )
    rationale: str = field(
        metadata={"desc": "Step-by-step reasoning"}, default=None
    )
    class_name: Literal[
        "ABBR", "ENTY", "DESC", "HUM", "LOC", "NUM"
    ] = field(
        metadata={"desc": "The class name"}, default=None
    )

    __input_fields__ = ["question"]
    __output_fields__ = ["rationale", "class_name"]

• TRECExtendedData extends DataClass, which (like Pydantic) gives us schema enforcement.

• Input: a question.

• Output: a rationale (reasoning trace) and a class_name (final label).

Now let’s plug it into a pipeline:

class TRECClassifierStructuredOutput(adal.Component):
    def __init__(
        self, model_client: adal.ModelClient, model_kwargs: Dict
    ):
        super().__init__()
        # Task description prompt
        task_desc_str = adal.Prompt(
            template=task_desc_template,
            prompt_kwargs={
                "classes": [
                    {"label": l, "desc": d} 
                    for l, d 
                    in zip(_COARSE_LABELS, _COARSE_LABELS_DESC)
                ]
            }
        )()

        parser = adal.DataClassParser(
            data_class=TRECExtendedData,
            return_data_class=True,
            format_type="yaml"
        )

        prompt_kwargs = {
            "system_prompt": adal.Parameter(
                data=task_desc_str,
                role_desc="Task description",
                requires_opt=True,
                param_type=adal.ParameterType.PROMPT,
            ),
            "output_format_str": parser.get_output_format_str(),
        }

        self.llm = adal.Generator(
            model_client=model_client,
            model_kwargs=model_kwargs,
            prompt_kwargs=prompt_kwargs,
            template=template,
            output_processors=parser,
        )

    def bicall(self, question: str, id: Optional[str] = None):
        return self.llm(prompt_kwargs={"input_str": question}, id=id)

• The Prompt defines the system instruction with class definitions.

• DataClassParser enforces structured YAML output that matches TRECExtendedData.

• Generator (a GradComponent) runs the LLM with prompt + parser.

• Output is guaranteed to follow the schema: rationale + class name.

This ensures the model never drifts into free-form answers—it always returns structured classification results.

Example 3: Training With LossComponent

Finally, how do we train or optimize these components? That’s where LossComponent comes in:

eval_fn = AnswerMatchAcc(type="exact_match").compute_single_item
loss_fn = adal.EvalFnToTextLoss(
    eval_fn=eval_fn,
    eval_fn_desc="exact_match: 1 if str(y) == str(y_gt) else 0"
)

• AnswerMatchAcc is the evaluation metric.

• EvalFnToTextLoss wraps it as a LossComponent, enabling LLM-AutoDiff to optimize prompts automatically during training.

By attaching this to your pipeline, you get a full training loop:
Forward pass → Eval metric → Backward engine → Prompt optimization.

Agents: Reasoning Meets Action

AdalFlow embraces the ReAct paradigm—combining reasoning (plan) with acting (tool use)—to build autonomous, auditable AI systems. An agent reasons about the task, selects tools, executes them, observes results, and iterates until it can deliver a final answer.

• https://adalflow.sylph.ai/new_tutorials/agents_runner.html

• https://colab.research.google.com/github/SylphAI-Inc/AdalFlow/blob/main/notebooks/agents/agent_tutorial.ipynb

Architecture at a Glance

• Agent (planner + tool manager)
  Handles planning and decision-making via a Generator-based planner, and knows what tools are available and how to call them.

• Runner (executor + conversation loop)
  Orchestrates multi-step execution, tool calling, observation handling, timeouts, and final answer synthesis.

This separation lets you swap or customize planning vs. execution independently.

Execution Flow (ReAct Loop Recap)

1. Planning – The Agent (Generator planner) analyzes input and proposes the next action.

2. Tool Selection – Chooses a tool from the registered set.

3. Tool Execution – The Runner invokes the tool with arguments.

4. Observation – The result is fed back to the planner.

5. Iteration – Repeat 1–4 up to max_steps or until confident.

6. Final Answer – The planner synthesizes the answer (optionally into a structured type).

Minimal, End-to-End Example

1) Define a Tool (callable or FunctionTool)

# Tool: a plain Python callable works, or wrap with FunctionTool for extras.
def calculator(expression: str) -> str:
    """Evaluate a mathematical expression."""
    try:
        result = eval(expression)
        return f"Result: {result}"
    except Exception as e:
        return f"Error: {e}"

2) Build the Agent (Planner + Tools)

from adalflow import Agent, Runner
from adalflow.components.model_client.openai_client import OpenAIClient

agent = Agent(
    name="CalculatorAgent",  # Agent identifier
    tools=[calculator],  # List of tools (callables or FunctionTool)
    # LLM client used by the planner (Generator-based)
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o", "temperature": 0.3},
    max_steps=6,  # Upper bound for ReAct loops
)

What this maps to:

• Planner: An internal Generator that decides the next step (think: “reasoning trace”).

• ToolManager: The agent’s registry of permitted tools.

• max_steps: Safety rail to prevent runaway loops.

Model Configuration (Swap Backends Easily)

# OpenAI
from adalflow.components.model_client.openai_client import OpenAIClient
agent = Agent(
    model_client=OpenAIClient(), 
    model_kwargs={"model": "gpt-4o"}
)

# Anthropic
from adalflow.components.model_client.anthropic_client import (
    AnthropicAPIClient
)
agent = Agent(
    model_client=AnthropicAPIClient(), 
    model_kwargs={"model": "claude-3-sonnet-20240229"}
)

3) Execute with the Runner (Multi-step Orchestration)

# Manages turns, tool calls, observations, and finalization
runner = Runner(agent=agent) 

result = runner.call(
    prompt_kwargs={"input_str": "Invoke the calculator tool and calculate 15 * 7 + 23"}
)

print(result.answer)
# -> "The result of 15 * 7 + 23 is 128."

RunnerResult schema (returned by Runner.call)

# result has:
# - result.step_history: [StepOutput(...)]  # Each step’s action + observation
# - result.answer: str | structured type     # Final synthesized answer
# - result.error: None | Exception info      # Error if something failed
# - result.ctx: dict | None                  # Optional execution metadata

This is the full ReAct loop in action:

Plan → Select Tool → Execute → Observe → Iterate → Answer.

Advanced Features (Production-Ready)

1) Streaming Execution (Real-Time Updates)

# Pseudocode: actual API may differ slightly in your version.
stream = runner.stream(
    prompt_kwargs={"input_str": "Compute 42 * 73 and explain."}
)
for update in stream:
    # update contains partial thoughts, tool calls, observations, etc.
    print(update)

Use streaming to surface live reasoning/tool progress in UIs.

2) Human-in-the-Loop (Permission Management)

from adalflow.permissions import PermissionManager

class MyPerms(PermissionManager):
    def approve(self, tool_name: str, args: dict) -> bool:
        # Example policy: only allow calculator; prompt user otherwise
        return tool_name == "calculator"

agent = Agent(
    name="GuardedAgent",
    tools=[calculator, search_tool],
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o"},
    permission_manager=MyPerms(),  # <- Every tool call can be inspected/approved)

Great for tools that hit external systems (files, emails, APIs).

3) Custom System Templates (Planner Behavior)

custom_role_desc = """
You are a careful, step-by-step data analyst.
When you use a tool, explain why and what you expect to get.
"""
agent = Agent(
    name="DataAnalyst",
    # Custom planner persona and guardrails
    role_desc=custom_role_desc,        
    model_client=OpenAIClient(),
    model_kwargs={"model": "gpt-4o", "temperature": 0.2},
)

4) Tracing (Observability)

# Configure tracing once (destination: console, file, or tracing backend)
from adalflow.tracing import enable_tracing
enable_tracing(project="adalflow-agents-demo")

result = runner.call(prompt_kwargs={
    "input_str": "Use the calculator for 88*19."
})
# Inspect step_history, tool IO, latency, errors, etc.

Agent Summary:

• Agent = Reasoning + Tool selection (Generator-based planner + ToolManager)

• Runner = Controlled execution loop (steps, tools, observations, final answer)

• Tools = Safe, permissioned extensions to the agent’s capabilities

• Production = Streaming, human approvals, tracing, structured outputs

Real-World Use Case: LinkedIn Recruitment Agent with AdalFlow

Hiring top talent is one of the most resource-intensive parts of building a company. Recruiters spend hours scrolling LinkedIn, opening profiles, copying notes, and crafting outreach messages.

What if we could automate that entire workflow—turning hours of manual searching into minutes of AI-assisted sourcing?

That’s exactly what we built using AdalFlow’s Agent + Runner architecture combined with browser automation via Chrome DevTools Protocol (CDP).

✨ Before vs. After

Traditional Recruiting Workflow (Manual)

❌ BEFORE: 2–3 hours per role

1. Navigate to LinkedIn people search

2. Type in “Product Manager, San Francisco”

3. Scroll endlessly, click into profiles

4. Skim experience, education, skills

5. Take notes in spreadsheets

6. Write & send DMs manually

Automated Workflow with AdalFlow (Agentic)

✅ AFTER: 10 minutes per role

1. Run: linkedin-agent --query “Product Manager” --limit 10

2. Agent plans and executes:

- Smart search strategy

- Extract profiles via browser automation

- Evaluate candidates with scoring models

- Draft personalized outreach messages

3. Get structured output: JSON/CSV with names, titles, LinkedIn URLs, evaluation scores, outreach drafts. Recruiters get to focus on talking to people, not copy-pasting data.

How It Works — Global State Architecture

We structured the solution around a global state shared between tools. Each tool contributes partial data (search results, profiles, evaluations, outreach drafts), which the Agent combines into a full pipeline. Agent combines into a full pipeline.

Implementation with AdalFlow

We implemented the LinkedInAgent by encapsulating:

Agent → Planner + Tools (search, extract, evaluate, outreach)

Runner → Execution loop with error handling and logging

class LinkedInAgent:
    """
    LinkedIn recruitment agent powered by AdalFlow.
    - Encapsulates Agent + Runner
    - Provides default recruitment tools
    - Supports both sync call() and async acall()
    """

    def __init__(
        self,
        model_client: Optional[OpenAIClient] = None,
        model_kwargs: Optional[Dict[str, Any]] = None,
        max_steps: Optional[int] = None,
        role_desc: Optional[str] = None,
        **kwargs,
    ):
        # Defaults
        model_client = model_client or OpenAIClient()
        model_kwargs = model_kwargs or {
           "model": "gpt-4o", “temperature”: 0.3
        }
        max_steps = max_steps or 6

        # Recruitment workflow tools
        self.tools = [
            # 1. Search LinkedIn via CDP
            SmartCandidateSearchTool,
            # 2. Extract structured profile data     
            ExtractCandidateProfilesTool, 
            # 3. Score candidates   
            CandidateEvaluationTool,
            # 4. Draft personalized outreach         
            CandidateOutreachGenerationTool,
            # 5. Persist results 
            SaveOutreachResultsTool,         
        ]

        # Agent role description (personality / instructions)
        role_desc = role_desc or "You are a recruitment assistant that sources and evaluates LinkedIn candidates."

        # Initialize Agent + Runner
        self.agent = Agent(
            name="LinkedInRecruiter",
            tools=self.tools,
            model_client=model_client,
            model_kwargs=model_kwargs,
            max_steps=max_steps,
            role_desc=role_desc,
            **kwargs,
        )
        self.runner = Runner(agent=self.agent, max_steps=max_steps)

    def call(
        self, query: str, context: Optional[Dict[str, Any]] = None
    ):
        return self.runner.call(prompt_kwargs={"input_str": query})

    async def acall(
        self, query: str, context: Optional[Dict[str, Any]] = None
    ):
        return await self.runner.acall(
            prompt_kwargs={"input_str": query}
        )

Full Workflow Execution

Here’s how we stitch the agent into a production workflow:

def execute_search_workflow(
    self, progress_tracker=None) -> List[Dict[str, Any]]:
    logger = get_logger()
    logger.set_workflow_context("workflow_main", "initialization")

    log_phase_start("WORKFLOW_START", f"Target: {self.limit} candidates for {self.query} in {self.location}")

    candidates = []
    try:
        log_info("🤖 Initializing LinkedIn agent...")
        agent, user_query = self.initialize_agent()

        if progress_tracker:
            progress_tracker.start_workflow()

        # Run full pipeline
        result = agent.call(query=user_query)
        self._print_agent_execution_steps(result)

        # Collect data from global state
        from ..core.workflow_state import get_complete_workflow_data
        workflow_data = get_complete_workflow_data()

        candidates = self._build_complete_candidate_data(workflow_data)

        log_info(f"✅ Found {len(candidates)} candidates")
        return candidates

    except Exception as e:
        log_error(f"❌ Workflow failed: {e}")
        return candidates

Example Output

After running:

linkedin-agent --query “Product Manager San Francisco” --limit 10

We get structured results like:

[
  {
    “name”: “Alex Chen”,
    “title”: “Senior Product Manager @ Stripe”,
    “location”: “San Francisco Bay Area”,
    “profile_url”: “https://linkedin.com/in/alexchen”,
    “score”: 0.92,
    “outreach_message”: “Hi Alex, I came across your experience at Stripe...”
  },
  {
    “name”: “Maria Lopez”,
    “title”: “PM, Growth @ Airbnb”,
    “location”: “San Francisco Bay Area”,
    “profile_url”: “https://linkedin.com/in/marialopez”,
    “score”: 0.88,
    “outreach_message”: “Hi Maria, your background in growth product design really stood out...”
  }
]

============================================================

WORKFLOW COMPLETION SUMMARY

============================================================

Success: ✅ Yes

Total Candidates: 2

Duration: 68.8 seconds

Session: 20250909_220617

Log Files:

• Main: logs/workflow_20250909_220617.log

• Debug: logs/debug_20250909_220617.log

• Agent Steps: logs/agent_steps_20250909_220617.log

• Errors: logs/errors_20250909_220617.log

============================================================

🏁 MAIN ✅ COMPLETED - Processed 2 candidates

[RESULTS] ✅ Recruitment workflow completed!

[RESULTS] 📊 Final result: Successfully processed 2 candidates

Looking Ahead

As the LLM landscape evolves, frameworks like AdalFlow will become the backbone of application development. Just as PyTorch accelerated deep learning, AdalFlow has the potential to democratize LLM app building—from chatbots to agents to beyond.

If you’re excited about shaping the future of AI workflows, the project is open-source and community-driven. Whether you’re an AI researcher, product engineer, or just curious about building smarter applications, now’s the time to get involved.

🚀 AdalFlow isn’t just another library. It’s a paradigm shift in how we think about programming with language models.

Signup!

What we are going to build

We are going to build a fun web application where we can demonstrate how to orchestrate a robust RAG application using LangGraph, FastAPI, and React. Here is what we are going to build:

1. A user can pass a GitHub repository URL

2. The files of the related repository are scraped and indexed in a vector database

3. Now the code is available for the user to ask questions about.

On the frontend, we will need two main functionalities:

• A page where we can input the repository URL and start the crawling and indexing processes:

• And a chatbot interface to ask questions about the code in the repository:

On the backend, we will need the related endpoints:

• The indexing endpoint will respond to the provided GitHub repository URL and the “crawl“ action to start the crawling and indexing processes.

• The chat endpoint that will respond to messages sent by the user from the chatbot interface.

We are going to use the following tools:

• React for the frontend

• FastAPI for the backend

• LangGraph for the agentic orchestration

• Pinecone for the vector database

• Langsmith for observability

• Deploy everything on Google Cloud!

Project-based course

We will focus on building the project from the ground up, as we would on the job. Here is how we are going to structure the project development:

• Introduction

  • What we want to build

  • Setting up the environment

• The RAG Application

  • The Data Parsing Pipeline

  • The Indexing Pipeline

  • The Basic RAG Pipeline

  • Adding Observability to the Pipeline with Langsmith

  • Going Agentic

• The Backend Application

  • The Indexing API Endpoint

  • Adding Memory

  • Administering the Database Data

• The Frontend Application

  • The Indexing Page

  • The Chatbot Page

• Deploying to GCP

Each session will be a live, hands-on coding session where we are going to implement every component from scratch

Going Agentic

“Agentic” means that we are going to use an LLM as a decision engine to enhance the quality of our pipeline. We will focus on improving the accuracy of the pipeline at the cost of latency and cost, and discuss the opportunities to reduce those induced negative points with small language models and fine-tuning. In the RAG pipeline, we are going to build a subagent for each of the main components:

• Intent router: the entry point of the pipeline that will decide if a RAG pipeline is required.

  

• The retriever: The sub-agent that will extract the right data

  

• The generator: The sub-agent that will generate the response to the user

  

Scaling up

With this course, I want to focus on what we would need to do to deploy the application to 1M users. We will make sure to design every endpoint to be asynchronous, queue the indexing requests, and deploy the application with elastic load balancing to scale the application horizontally.

This is going to be a fun ride! Make sure to join us!

Signup!

The Real-World AI Engineering Roadblocks You Face Today

👋 Prototype → Production Gap — Moving from a notebook demo to a secure, observable, multi-tenant service requires orchestration, evals, guardrails, and ops most teams lack.

👋 “Easy RAG” vs “Reliable RAG” — Anyone can retrieve-then-generate; making answers faithful, fresh, fast, and cost-controlled under real traffic is the hard part.

👋 Framework Overload — The ecosystem is noisy; you need clear criteria (maturity, extensibility, latency, cost) and reference patterns to choose confidently.

👋 It’s Software Engineering First — Success hinges on clean interfaces, tests, typed configs, tracing, CI/CD, and change management—not just prompts and models.

👋 From Laptop to 1M Users — Scaling demands streaming, batching, caching, autoscaling, and SLOs, or your p95 explodes and costs spiral.

How this course will help you

✅ Ship a real Agentic RAG app, not a demo — Stand up an end-to-end stack—LangGraph → FastAPI → React, that runs locally today and deploys via a clean, fork-and-ship monorepo.

✅ Make retrieval dependable, not lucky — Adopt schema-aware chunking, strong dense embeddings with sensible metadata filters, and context packing with citations so answers stay faithful, fresh, and concise.

✅ Harden agentic workflows — Design a typed LangGraph state and build nodes for rewrite → retrieve → rerank → synthesize → cite → safety-check, with retries and timeouts so plans don’t loop or stall.

✅ Scale the experience, not the headaches — Enable server-streaming in FastAPI, cap top-k, trim context budgets, and add early-exit rules; deploy with autoscaling so you can serve real traffic without infra fuss.

✅ See enough to fix things fast — Bake in structured logs (no vendor tracing), per-step timing counters, and UI breadcrumbs/citations to follow query → context → answer and spot common failure patterns quickly.

✅ Choose frameworks with confidence — Follow an opinionated reference architecture plus a simple choice rubric (maturity, extensibility, latency, cost, swap effort) so you know when to stick—and how to swap components without rewrites.

✅ Write maintainable RAG code — Use clean module boundaries (ingest / retrieve / rerank / synthesize), typed configs (Pydantic Settings), and sensible secrets/env management so your team can extend it safely.

You’ll walk away with

✨ A running Agentic RAG app (LangGraph + FastAPI + React) in a fork-and-ship monorepo.

✨ An ingestion/indexing pipeline with metadata, hybrid retrieval, and optional re-ranking.

✨ A chat UI with citations, source previews, and conversation memory that behaves.

✨ Deploy scripts and env templates to go live right after class.

✨ A framework choice memo + adapters to swap models/vector stores without starting over.

Bottom line: this isn’t a vitamin, it’s a blueprint you can put in production.

What you’ll get out of this course

• Orchestrate complex RAG pipelines with LangGraph and OpenAI API: Build a typed LangGraph that routes rewrite → retrieve → rerank → synthesize → cite → self-check with retries, timeouts, early-exit rules, and real tool calls, exposed as a clean HTTP API.

• Build scalable asynchronous applications with FastAPI: Ship async FastAPI endpoints, well-typed request/response models, input validation, and sensible timeouts, ready to run locally and deploy to production.

• Implement chatbot interfaces with React: Create a chat UI that shows citations and source previews, lets users scope queries, preserves safe chat history, and handles transient API errors gracefully.

• Mitigate hallucinations with LLM judges, structured output, and context engineering: Cut errors via schema-aware chunking, dedupe and budgeted context packing, plus lightweight LLM checks and schema-constrained outputs to verify claims and enforce citations before responding.

• Design effective LLM prompts for high-level control on generation output: Write prompts that steer behavior: system prompts, task decomposition, Pydantic/JSON-schema constraints, and clear rules for tone, citations, and safe refusals.

• Develop end-to-end RAG applications using the software engineering best practices: Produce a maintainable codebase: clean module boundaries (ingest/retrieve/rerank/synthesize), typed configs, secrets/env management, reproducible local dev, and deploy that mirrors local.

Sign Up!

Signup!

What we are going to build

We are going to build a fun web application where we can demonstrate how to orchestrate a robust RAG application using LangGraph, FastAPI, and React. Here is what we are going to build:

1. A user can pass a GitHub repository URL

2. The files of the related repository are scraped and indexed in a vector database

3. Now the code is available for the user to ask questions about.

On the frontend, we will need two main functionalities:

• A page where we can input the repository URL and start the crawling and indexing processes:

• And a chatbot interface to ask questions about the code in the repository:

On the backend, we will need the related endpoints:

• The indexing endpoint will respond to the provided GitHub repository URL and the “crawl“ action to start the crawling and indexing processes.

• The chat endpoint that will respond to messages sent by the user from the chatbot interface.

We are going to use the following tools:

• React for the frontend

• FastAPI for the backend

• LangGraph for the agentic orchestration

• Pinecone for the vector database

• Langsmith for observability

• Deploy everything on Google Cloud!

Project-based course

We will focus on building the project from the ground up, as we would on the job. Here is how we are going to structure the project development:

• Introduction

  • What we want to build

  • Setting up the environment

• The RAG Application

  • The Data Parsing Pipeline

  • The Indexing Pipeline

  • The Basic RAG Pipeline

  • Adding Observability to the Pipeline with Langsmith

  • Going Agentic

• The Backend Application

  • The Indexing API Endpoint

  • Adding Memory

  • Administering the Database Data

• The Frontend Application

  • The Indexing Page

  • The Chatbot Page

• Deploying to GCP

Each session will be a live, hands-on coding session where we are going to implement every component from scratch

Going Agentic

“Agentic” means that we are going to use an LLM as a decision engine to enhance the quality of our pipeline. We will focus on improving the accuracy of the pipeline at the cost of latency and cost, and discuss the opportunities to reduce those induced negative points with small language models and fine-tuning. In the RAG pipeline, we are going to build a subagent for each of the main components:

• Intent router: the entry point of the pipeline that will decide if a RAG pipeline is required.

• The retriever: The sub-agent that will extract the right data

• The generator: The sub-agent that will generate the response to the user

Scaling up

With this course, I want to focus on what we would need to do to deploy the application to 1M users. We will make sure to design every endpoint to be asynchronous, queue the indexing requests, and deploy the application with elastic load balancing to scale the application horizontally.

This is going to be a fun ride! Make sure to join us!

Signup!

Sign Up!

From Prototype to Production: Ship Reliable and Scalable RAG Pipelines

The Real-World AI Engineering Roadblocks You Face Today

👋 Prototype → Production Gap — Moving from a notebook demo to a secure, observable, multi-tenant service requires orchestration, evals, guardrails, and ops most teams lack.

👋 “Easy RAG” vs “Reliable RAG” — Anyone can retrieve-then-generate; making answers faithful, fresh, fast, and cost-controlled under real traffic is the hard part.

👋 Framework Overload — The ecosystem is noisy; you need clear criteria (maturity, extensibility, latency, cost) and reference patterns to choose confidently.

👋 It’s Software Engineering First — Success hinges on clean interfaces, tests, typed configs, tracing, CI/CD, and change management—not just prompts and models.

👋 From Laptop to 1M Users — Scaling demands streaming, batching, caching, autoscaling, and SLOs, or your p95 explodes and costs spiral.

How this course will help you

✅ Ship a real Agentic RAG app, not a demo — Stand up an end-to-end stack—LangGraph → FastAPI → React, that runs locally today and deploys via a clean, fork-and-ship monorepo.

✅ Make retrieval dependable, not lucky — Adopt schema-aware chunking, strong dense embeddings with sensible metadata filters, and context packing with citations so answers stay faithful, fresh, and concise.

✅ Harden agentic workflows — Design a typed LangGraph state and build nodes for rewrite → retrieve → rerank → synthesize → cite → safety-check, with retries and timeouts so plans don’t loop or stall.

✅ Scale the experience, not the headaches — Enable server-streaming in FastAPI, cap top-k, trim context budgets, and add early-exit rules; deploy with autoscaling so you can serve real traffic without infra fuss.

✅ See enough to fix things fast — Bake in structured logs (no vendor tracing), per-step timing counters, and UI breadcrumbs/citations to follow query → context → answer and spot common failure patterns quickly.

✅ Choose frameworks with confidence — Follow an opinionated reference architecture plus a simple choice rubric (maturity, extensibility, latency, cost, swap effort) so you know when to stick—and how to swap components without rewrites.

✅ Write maintainable RAG code — Use clean module boundaries (ingest / retrieve / rerank / synthesize), typed configs (Pydantic Settings), and sensible secrets/env management so your team can extend it safely.

You’ll walk away with

✨ A running Agentic RAG app (LangGraph + FastAPI + React) in a fork-and-ship monorepo.

✨ An ingestion/indexing pipeline with metadata, hybrid retrieval, and optional re-ranking.

✨ A chat UI with citations, source previews, and conversation memory that behaves.

✨ Deploy scripts and env templates to go live right after class.

✨ A framework choice memo + adapters to swap models/vector stores without starting over.

Bottom line: this isn’t a vitamin, it’s a blueprint you can put in production.

What you’ll get out of this course

• Orchestrate complex RAG pipelines with LangGraph and OpenAI API: Build a typed LangGraph that routes rewrite → retrieve → rerank → synthesize → cite → self-check with retries, timeouts, early-exit rules, and real tool calls, exposed as a clean HTTP API.

• Build scalable asynchronous applications with FastAPI: Ship async FastAPI endpoints, well-typed request/response models, input validation, and sensible timeouts, ready to run locally and deploy to production.

• Implement chatbot interfaces with React: Create a chat UI that shows citations and source previews, lets users scope queries, preserves safe chat history, and handles transient API errors gracefully.

• Mitigate hallucinations with LLM judges, structured output, and context engineering: Cut errors via schema-aware chunking, dedupe and budgeted context packing, plus lightweight LLM checks and schema-constrained outputs to verify claims and enforce citations before responding.

• Design effective LLM prompts for high-level control on generation output: Write prompts that steer behavior: system prompts, task decomposition, Pydantic/JSON-schema constraints, and clear rules for tone, citations, and safe refusals.

• Develop end-to-end RAG applications using the software engineering best practices: Produce a maintainable codebase: clean module boundaries (ingest/retrieve/rerank/synthesize), typed configs, secrets/env management, reproducible local dev, and deploy that mirrors local.

  Sign Up!

• The Sparsely-Gated Mixture-of-Experts Layer

• GShard

The First Mixture-of-Experts

The concept of the Mixture of Experts (MoE) architecture was introduced in 1991 by Jacobs et al. The idea was to combine the learning of parallel learners using a gating mechanism. The goal was to increase the model's capacity while maintaining stable training and achieving faster convergence. Deep networks tend to suffer from vanishing or exploding gradients, and extending the capacity in the width direction allows for the learning of more complex statistical patterns while keeping short gradient paths.

An "expert" E_i can be a simple feed-forward network. For example:

where W₁ⁱ and W₂ⁱ may have different dimensions depending on the expert. The gating mechanism generates a weight g_i(h) for each expert, and the MoE output is a weighted average of the experts' outputs:

g_i(h) is typically the softmax transformation of a linear projection W_g of the input features:

where W_g is a linear layer of dimension |h| ⨉ n. The softmax transformation yields a probability-like value that captures the proportion of contributions for each expert.

Early Sparse MoE Prototypes in LLMs

The Sparsely-Gated Mixture-of-Experts Layer

The Sparse mechanism

The sparse Mixture of Experts (MoE) architecture was introduced by Shazeer et al in January 2017 in LSTM-based language models as a way to drastically scale the model capacity while keeping the number of operations constant, independent of the number of experts. RNN models are hard to scale because the LSTM operations are intrinsically iterative, preventing the high parallelism provided by other computational units. Scaling in depth with more LSTM units induces high latency, while scaling with feed-forward networks limits the ability of the LSTM layers to capture long-range coherence of the input sequences. Scaling in width with many parallel experts permits the LSTM to learn the long-term dependencies while keeping the latency to a minimum.

To keep the number of operations independent of the number of experts, they introduced a routing mechanism that selects the top-k experts for each token. They tested a total number of experts that ranged between 4 and 131,072, but only the top-4 experts were used for each token.

For the sparse MoE, the architecture is the same for every expert i:

The gating mechanism is, as before, induced by a linear layer W_g mapping from hidden size d_model to n with an added normal noise ϵ:

The noise allows the model to uniformly explore the different experts in the early part of the training and prevents the collapse onto a handful of "favorite" experts. To generate the noise, the input vector is first projected with another linear layer W_n passed through a softplus transformation:

The resulting value 𝛔(h) is used as the standard deviation to generate the normal noise:

The standard deviation 𝛔(h) controls how much randomness is injected for each expert on that particular input token. Larger 𝛔(h) will lead to more exploration, and smaller 𝛔(h) will push the gate to behave almost deterministically. Softplus is basically a "soft" version of ReLU: for large positive values, it increases linearly and flattens near 0. It is used here to make sure the learned noise scale 𝛔(h) is positive and differentiable everywhere. Because the noise is input‑adaptive, the router can start off as a near‑uniform sampler (large 𝛔(h)) and gradually anneal into a confident switch (small 𝛔(h)).

Once the logits l(x) = {l₁(h), l₂(h), …, l_n(h)} have been computed, we mask the non-top-k's contribution with -∞:

And we perform a softmax transformation to obtain the contribution of each top-k expert:

The -∞ masks will lead to g_i(h) = 0 contribution for every non-top-k expert while keeping the softmax normalization as if only the top-k experts contributed to the sum. The resulting hidden state y(h) is the weighted average of top-k experts' output:

Hierarchical Mixture of Experts

In the Sparsely-Gated Mixture-of-Experts, they scaled the number of experts up to 131,072! With naive MoE, the gate and noise projection layers W_g and W_n would require a dimension (n ⨉ d_model) ~ 67M parameters (with d_model = 512), with as many operations for each token in the input sequence. At this scale, the gating mechanism becomes the bottleneck. Instead, they introduced a hierarchical gating process where the experts were grouped into a blocks of b experts. The first gate projects from d_model to a blocks:

From this first set of logits, we can pick the top-k₁ expert blocks:

Read more

Enroll

The Real-World LLM Engineering Roadblocks You Face Today

👋 Transitioning from General ML to LLM Specialization: You’ve built recommendation engines or classifier models, but moving into Transformer‑centric development feels like learning a whole new discipline—no clear roadmap exists.

👋 Lack of LLM‑Specific Career Path: You see “LLM Engineer” roles popping up on LinkedIn, but your current CV only shows “Data Scientist” or “ML Engineer.” You need hands‑on projects and artifacts to credibly make the jump.

👋 Career Stalled by “Academic” Skillset: You can recite Transformer papers, but when asked, “Have you shipped an LLM feature end‑to‑end?” you have no answer—and no portfolio to prove it!

👋 Prototype Meltdown Under Production Load: You’ve fine‑tuned a small model locally, but when you switch from 1 to 100 concurrent requests, your GPU memory spikes and inference grinds to a halt, because you’ve never applied continuous batching, KV caching, or paged‑attention in a live setting.

👋 RAG Integration Headaches: Turning a standalone model into a live, Retriever‑Augmented Generation service becomes a multi‑week integration nightmare.

How this course will help you

Because we’ve packaged every stage of the LLM lifecycle, from career transition to production rollout, into a six‑week bootcamp that:

✅ Guides Your Career Pivot: You’ll emerge with six polished GitHub projects, a deployment playbook, and RAG demos that transform your resume from “ML generalist” to “LLM Specialist.”

✅ Attacks Each Pain‑Point Head‑On: Attacks each pain point head‑on with six job‑mirroring projects (from scratch → RLHF → scaling → deployment → RAG), so you never waste time on dead‑end tutorials

✅ Live Code‑Along Workshops & Office Hours: Tackle your own fine‑tuning bugs, scaling hiccups, and deployment errors alongside Damien in dedicated sessions, so you get hands‑on fixes for the exact issues you’ll face on the job.

✅ Ready‑to‑Use Repos & Playbooks: Grab our curated starter code, development scripts, deployment templates, and debugging checklists, so you can plug them straight into your next project without reinventing the wheel.

✅ A Portfolio of Six Production‑Grade Projects: Leave with six end‑to‑end deliverables, from a Transformer built from scratch to a live RAG API, ready to showcase on GitHub, in performance reviews, or to hiring managers.

No more scattered blog-hopping or generic bootcamps, this is the only cohort where you’ll master Transformer internals and ship production‑grade LLM systems while making the career leap you’ve been aiming for.

What You’ll Actually Build and Ship

Across six hands‑on projects, you’ll deliver deployable LLM components and applications, no fluff, just job‑ready code:

✅ A Modern Transformer Architecture from scratch: Implement a sliding‑window multihead attention to slash O(N²) to O(N·w), RoPE for relative positional encoding, and the Mixture-of-Expert architecture for improved performance, all in PyTorch.

✅ Instruction‑Tuned LLM: Fine‑tune a model with supervised learning, RLHF, DPO, and ORPO for instruction following on a real benchmark and compare performance gains.

✅ Scalable Training Pipeline: Containerize a multi‑GPU job with DeepSpeed ZeRO on SageMaker to maximize throughput and minimize cost.

✅ Extended‑Context Model: Modify RoPE scaling, apply 4/8‑bit quantization, and inject LoRA adapters to double your context window.

✅ Multi‑Mode Deployment: Stand up a Hugging Face endpoint, a vLLM streaming API, and an OpenAI‑compatible server, all Dockerized and optimized for low latency.

✅ End‑to‑End RAG Chat App: Build a FastAPI backend with conversational memory and a Streamlit UI for live Retrieval‑Augmented Generation.

By the end of Week 6, you won’t just know these techniques, you’ll have shipped six production‑grade artifacts, each reflecting the exact pipelines, optimizations, and deployment routines you’ll use on the job.

Live & Recorded Content: Reinforce, Deepen, Accelerate

✨ 12 Interactive Live Workshops (3 hrs each): Each session follows the Concept → Code flow. I’ll introduce the day’s core topic (e.g. self-attention, LoRA, vLLM optimizations, ...), and we’ll implement the features step‑by‑step in code so you see exactly how theory maps to code. Bring your questions!

✨ 10 + Hours of On‑Demand Deep‑Dive Lectures: Short videos (10–20 min) on Transformer internals, fine-tuning tricks, deployment optimizations. Watch before each project to hit the ground running. Step through every line of code at your own pace; perfect for review or catching up if you miss a live session. Downloadable slide decks, annotated notebooks, and cheat sheets you’ll reference long after graduation.

Why This Matters: Live workshops turn recorded concepts into actionable skills. You’ll see how theory maps directly onto code, get instant feedback, and internalize best practices. Then, recorded lectures become your asynchronous safety net, letting you revisit tricky topics, prepare for upcoming labs, and solidify your understanding on demand.

Let me know if you have any questions. I hope to see you there!

Signup

The Real-World LLM Engineering Roadblocks You Face Today

👋 Transitioning from General ML to LLM Specialization: You’ve built recommendation engines or classifier models, but moving into Transformer‑centric development feels like learning a whole new discipline—no clear roadmap exists.

👋 Lack of LLM‑Specific Career Path: You see “LLM Engineer” roles popping up on LinkedIn, but your current CV only shows “Data Scientist” or “ML Engineer.” You need hands‑on projects and artifacts to credibly make the jump.

👋 Career Stalled by “Academic” Skillset: You can recite Transformer papers, but when asked, “Have you shipped an LLM feature end‑to‑end?” you have no answer—and no portfolio to prove it!

👋 Prototype Meltdown Under Production Load: You’ve fine‑tuned a small model locally, but when you switch from 1 to 100 concurrent requests, your GPU memory spikes and inference grinds to a halt, because you’ve never applied continuous batching, KV caching, or paged‑attention in a live setting.

👋 RAG Integration Headaches: Turning a standalone model into a live, Retriever‑Augmented Generation service becomes a multi‑week integration nightmare.

How this course will help you

Because we’ve packaged every stage of the LLM lifecycle, from career transition to production rollout, into a six‑week bootcamp that:

✅ Guides Your Career Pivot: You’ll emerge with six polished GitHub projects, a deployment playbook, and RAG demos that transform your resume from “ML generalist” to “LLM Specialist.”

✅ Attacks Each Pain‑Point Head‑On: Attacks each pain point head‑on with six job‑mirroring projects (from scratch → RLHF → scaling → deployment → RAG), so you never waste time on dead‑end tutorials

✅ Live Code‑Along Workshops & Office Hours: Tackle your own fine‑tuning bugs, scaling hiccups, and deployment errors alongside Damien in dedicated sessions, so you get hands‑on fixes for the exact issues you’ll face on the job.

✅ Ready‑to‑Use Repos & Playbooks: Grab our curated starter code, development scripts, deployment templates, and debugging checklists, so you can plug them straight into your next project without reinventing the wheel.

✅ A Portfolio of Six Production‑Grade Projects: Leave with six end‑to‑end deliverables, from a Transformer built from scratch to a live RAG API, ready to showcase on GitHub, in performance reviews, or to hiring managers.

No more scattered blog-hopping or generic bootcamps, this is the only cohort where you’ll master Transformer internals and ship production‑grade LLM systems while making the career leap you’ve been aiming for.

What You’ll Actually Build and Ship

Across six hands‑on projects, you’ll deliver deployable LLM components and applications, no fluff, just job‑ready code:

✅ A Modern Transformer Architecture from scratch: Implement a sliding‑window multihead attention to slash O(N²) to O(N·w), RoPE for relative positional encoding, and the Mixture-of-Expert architecture for improved performance, all in PyTorch.

✅ Instruction‑Tuned LLM: Fine‑tune a model with supervised learning, RLHF, DPO, and ORPO for instruction following on a real benchmark and compare performance gains.

✅ Scalable Training Pipeline: Containerize a multi‑GPU job with DeepSpeed ZeRO on SageMaker to maximize throughput and minimize cost.

✅ Extended‑Context Model: Modify RoPE scaling, apply 4/8‑bit quantization, and inject LoRA adapters to double your context window.

✅ Multi‑Mode Deployment: Stand up a Hugging Face endpoint, a vLLM streaming API, and an OpenAI‑compatible server, all Dockerized and optimized for low latency.

✅ End‑to‑End RAG Chat App: Build a FastAPI backend with conversational memory and a Streamlit UI for live Retrieval‑Augmented Generation.

By the end of Week 6, you won’t just know these techniques, you’ll have shipped six production‑grade artifacts, each reflecting the exact pipelines, optimizations, and deployment routines you’ll use on the job.

Live & Recorded Content: Reinforce, Deepen, Accelerate

✨ 12 Interactive Live Workshops (3 hrs each): Each session follows the Concept → Code flow. I’ll introduce the day’s core topic (e.g. self-attention, LoRA, vLLM optimizations, ...), and we’ll implement the features step‑by‑step in code so you see exactly how theory maps to code. Bring your questions!

✨ 10 + Hours of On‑Demand Deep‑Dive Lectures: Short videos (10–20 min) on Transformer internals, fine-tuning tricks, deployment optimizations. Watch before each project to hit the ground running. Step through every line of code at your own pace; perfect for review or catching up if you miss a live session. Downloadable slide decks, annotated notebooks, and cheat sheets you’ll reference long after graduation.

Why This Matters: Live workshops turn recorded concepts into actionable skills. You’ll see how theory maps directly onto code, get instant feedback, and internalize best practices. Then, recorded lectures become your asynchronous safety net, letting you revisit tricky topics, prepare for upcoming labs, and solidify your understanding on demand.

Let me know if you have any questions. I hope to see you there!

Enroll

The Real-World LLM Engineering Roadblocks You Face Today

👋 Transitioning from General ML to LLM Specialization: You’ve built recommendation engines or classifier models, but moving into Transformer‑centric development feels like learning a whole new discipline—no clear roadmap exists.

👋 Lack of LLM‑Specific Career Path: You see “LLM Engineer” roles popping up on LinkedIn, but your current CV only shows “Data Scientist” or “ML Engineer.” You need hands‑on projects and artifacts to credibly make the jump.

👋 Career Stalled by “Academic” Skillset: You can recite Transformer papers, but when asked, “Have you shipped an LLM feature end‑to‑end?” you have no answer—and no portfolio to prove it!

👋 Prototype Meltdown Under Production Load: You’ve fine‑tuned a small model locally, but when you switch from 1 to 100 concurrent requests, your GPU memory spikes and inference grinds to a halt, because you’ve never applied continuous batching, KV caching, or paged‑attention in a live setting.

👋 RAG Integration Headaches: Turning a standalone model into a live, Retriever‑Augmented Generation service becomes a multi‑week integration nightmare.

How this course will help you

Because we’ve packaged every stage of the LLM lifecycle, from career transition to production rollout, into a six‑week bootcamp that:

✅ Guides Your Career Pivot: You’ll emerge with six polished GitHub projects, a deployment playbook, and RAG demos that transform your resume from “ML generalist” to “LLM Specialist.”

✅ Attacks Each Pain‑Point Head‑On: Attacks each pain point head‑on with six job‑mirroring projects (from scratch → RLHF → scaling → deployment → RAG), so you never waste time on dead‑end tutorials

✅ Live Code‑Along Workshops & Office Hours: Tackle your own fine‑tuning bugs, scaling hiccups, and deployment errors alongside Damien in dedicated sessions, so you get hands‑on fixes for the exact issues you’ll face on the job.

✅ Ready‑to‑Use Repos & Playbooks: Grab our curated starter code, development scripts, deployment templates, and debugging checklists, so you can plug them straight into your next project without reinventing the wheel.

✅ A Portfolio of Six Production‑Grade Projects: Leave with six end‑to‑end deliverables, from a Transformer built from scratch to a live RAG API, ready to showcase on GitHub, in performance reviews, or to hiring managers.

No more scattered blog-hopping or generic bootcamps, this is the only cohort where you’ll master Transformer internals and ship production‑grade LLM systems while making the career leap you’ve been aiming for.

What You’ll Actually Build and Ship

Across six hands‑on projects, you’ll deliver deployable LLM components and applications, no fluff, just job‑ready code:

✅ A Modern Transformer Architecture from scratch: Implement a sliding‑window multihead attention to slash O(N²) to O(N·w), RoPE for relative positional encoding, and the Mixture-of-Expert architecture for improved performance, all in PyTorch.

✅ Instruction‑Tuned LLM: Fine‑tune a model with supervised learning, RLHF, DPO, and ORPO for instruction following on a real benchmark and compare performance gains.

✅ Scalable Training Pipeline: Containerize a multi‑GPU job with DeepSpeed ZeRO on SageMaker to maximize throughput and minimize cost.

✅ Extended‑Context Model: Modify RoPE scaling, apply 4/8‑bit quantization, and inject LoRA adapters to double your context window.

✅ Multi‑Mode Deployment: Stand up a Hugging Face endpoint, a vLLM streaming API, and an OpenAI‑compatible server, all Dockerized and optimized for low latency.

✅ End‑to‑End RAG Chat App: Build a FastAPI backend with conversational memory and a Streamlit UI for live Retrieval‑Augmented Generation.

By the end of Week 6, you won’t just know these techniques, you’ll have shipped six production‑grade artifacts, each reflecting the exact pipelines, optimizations, and deployment routines you’ll use on the job.

Live & Recorded Content: Reinforce, Deepen, Accelerate

✨ 12 Interactive Live Workshops (3 hrs each): Each session follows the Concept → Code flow. I’ll introduce the day’s core topic (e.g. self-attention, LoRA, vLLM optimizations, ...), and we’ll implement the features step‑by‑step in code so you see exactly how theory maps to code. Bring your questions!

✨ 10 + Hours of On‑Demand Deep‑Dive Lectures: Short videos (10–20 min) on Transformer internals, fine-tuning tricks, deployment optimizations. Watch before each project to hit the ground running. Step through every line of code at your own pace; perfect for review or catching up if you miss a live session. Downloadable slide decks, annotated notebooks, and cheat sheets you’ll reference long after graduation.

Why This Matters: Live workshops turn recorded concepts into actionable skills. You’ll see how theory maps directly onto code, get instant feedback, and internalize best practices. Then, recorded lectures become your asynchronous safety net, letting you revisit tricky topics, prepare for upcoming labs, and solidify your understanding on demand.

Let me know if you have any questions. I hope to see you there!

• Additive Relative Positional Embeddings

• Multiplicative Relative Positional Embeddings

• ALiBi: Attention With Linear Biases

• RoPE: Rotary Position Embedding

  • The Complex Number Representation

  • Increasing The Context Window With RoPE

• No Position Encoder: NoPE

  • How NoPE Can Learn Relation Positional Information

  • Better Generalization for Longer Distance

  • Llama 4's iRoPE

The original positional encoding scheme introduced in "Attention is All You Need" uses sinusoidal functions to create absolute position representations, but this approach revealed significant shortcomings when handling longer sequences or transferring to different sequence lengths:

• Fixed context windows and poor extrapolation: When faced with sequences longer than this predefined limit, the model must either truncate the input or use positional values it never encountered during training. Perhaps the most significant limitation is that vanilla encodings represent absolute positions rather than relationships between tokens.

• The absolute vs. relative position problem: In language, relative positions often matter more than absolute ones. Consider the sentence: "The dog that chased the cat ran away." The relationship between "dog" and "ran" remains the same whether this is the opening sentence of a document or appears on page fifty, but absolute encodings fail to directly capture this invariance.

• Information dilution through layers: As signals propagate through the multiple layers of a Transformer, the influence of the original positional information tends to weaken. The model must work harder to maintain positional awareness in deeper layers, especially for distant tokens.

• Limited inductive bias for local relationships: Natural language exhibits a locality bias as nearby words often have stronger relationships than distant ones. The vanilla encoding doesn't naturally encode this bias, treating position 5 and position 500 as equally valid attention targets from a structural perspective, requiring the model to learn this pattern from data alone.

• Mathematical constraints: The sinusoidal functions used in the original formulation were chosen partly for their theoretical ability to generalize to unseen positions. However, in practice, they still struggle with significant extrapolation beyond the training range. The model learns to associate specific patterns with specific position ranges, and these associations become increasingly unreliable as we move farther from the training distribution.

These limitations collectively motivated researchers to develop more sophisticated positional encoding schemes, from Shaw's direct relative encodings to Transformer-XL's decomposed approach, ALiBi's distance-based penalties, and RoPE's elegant rotational solution, each addressing different aspects of these fundamental challenges.

You can find the discussion about the original positional encoding here:

Additive Relative Positional Embeddings

One of the first works addressing capturing the relative position between tokens instead of the absolute ones was done by Shaw et al in 2018. It addressed many of the original positional encoding's shortcomings. Instead of having an embedding matrix added to the semantic representation of the tokens, it modifies directly the computation of the attention layers with two positional embeddings. The first one a_ij^K is added to the key representation, and the second a_ij^V, to the value representation:

where a_ij^K and a_ij^V are learned embeddings of size d_model that depend only on the relative distance j-i. Those embeddings are learned in each attention layer and influence more directly the interactions between tokens. Ignoring the attention heads for simplicity, the computation of a context vector c_i is modified as:

They clip the maximum relative distance to a constant L, learning only 2L+1 position embeddings for each positional encoding. If we call the matrices of learned positions w^K and w^V, the actual relative position representations that are added to keys and values are the clipped version of those matrices:

where

which clamps the relative distance to the range [−L, L]. This means that for any relative distance j-i > L,

and any relative distance j-i < -L,

In practice, L can be chosen quite small as the performance remained relatively stable for L > 2 (testing L = 4, L = 16, L = 64, and L = 256). This suggests that distinguishing the precise relative distances of tokens becomes less important beyond a few positions away. The model primarily needs fine-grained position information for nearby contexts, while more distant relationships can be effectively captured with less positional precision, allowing the content representations themselves to drive attention patterns at longer distances. Even with a very small clipping window (L = 2), the model with relative positional encoding achieved substantial improvements over absolute positioning.

Despite the added performance gain, it is important to note that it is at the cost of added parameters and added time complexity. At the time, Shaw et al. performed their experiments with a 65M parameters transformer model with 6 layers and d_model = 512. We need two new parameter layers of size (2L+1) x d_model, and with L = 16, it is ~200K additional parameters, which is negligible compared to the overall size of the model. More importantly, computing all the relative distances between the N tokens, the constant memory access, and adding the related embeddings is an O(N²) process that imposes a significant additional computational work on top of the already quadratic complexity of attention.

Multiplicative Relative Positional Embeddings

In a previous newsletter, we introduced the attention mechanism developed with Transformer-XL to handle arbitrarily long sequences, but we left out the discussion about the positional encoding. Developing a new relative positional encoding was one of the critical pieces to handle longer sequences. The approach is based on the following analysis we provided in a previous newsletter for the original positional encoding:

In this equation, we decomposed the contributions from the different content and positional components for the first attention layer in the model when using the vanilla absolute positional encoding. We showed how R(k) was capturing the relative positional information between tokens, and we hope for the model to learn W^Q and W^K such that it can effectively utilize that information. To help the model capture better the content and position interactions between tokens, the relative positional encoding introduced in Transformer-XL modified this original equation. The relative positional encoding will now be applied within every layer following this strategy:

• x_i^T W^QT W^K x_i+k originally captured the pure content-based interaction between tokens. x_i, in the original Transformer, represents the embedding vector from the token embedding, and the related hidden state h_i is the sum of the token embedding and the positional encoding:

  We modify this interaction by applying it directly to the hidden states h_i and h_i+k where k is the distance between the related tokens:

  We specifically distinguish the projection W_E^K to be content-specific.

• x_i^T W^QT W^K PE(i) R(k) or equivalently x_i^T W^QT W^K PE(i+k) R(k) captured a token's preference for attending to positions was tied to fixed locations. However, this is confusing for the model because PE(i+k) depends on the specific position of x_i. Instead, in the Transformer-XL, they modified this interaction only accounting for the relative distance k from the hidden state h_i:

  r_k is a vector of the positional encoding matrix R. As for the additive relative positional embeddings, R is a (2L+1) x d_model matrix where r_k only depends on the relative distance between h_i and h_i+k, and L is a constant chosen to clip the maximum distances that can be represented. However, R is not a learned parameter matrix, but a fixed encoding as in the original Transformer architecture. W_R^K (different from W_E^K) is a specialized weight matrix that further enhances this separation of positional processing.

• PE(i)^T W^QT W^K x_i+k is a position-dependent bias for attending to content. This is a strange term because it has more or less weight depending on the position of the query. To make it position independent, we replace PE(i)^T W^QT with a global learnable parameter u^T:

  u is a vector of size d_model. This reflects an insight that the attentive bias toward different content should remain consistent regardless of the query position. In other words, the importance of certain word types doesn't need to depend on position, so a single global parameter can replace position-specific queries. This simplifies the model while maintaining expressiveness.

• PE(i)^T W^QT W^K PE(i+k) is a position-position iteration term that depends on the specific position of the query. To make it position independent while keeping the relative positional information, we introduce another global learnable parameter v^T of size d_model to replace PE(i)^T W^QT, and use again the relative positional encoding R:

  As before, we use the position-specific projection W_R^K. The intuition is that certain relative distances might be generally more important than others, regardless of the absolute positions involved. For example, adjacent tokens (small k) might generally be more related than distant ones, regardless of their absolute positions in the sequence.

This leads to redefining the alignment scores as:

with W^Qh_i = q_i, the query and W_E^Kh_i+k = k_i+k, the key, we can regroup the terms:

with

being a pure content interaction term and

representing a content-to-relative distance interaction term. W_E^Kh_i+k = k_i+k can be thought of as a content contribution to the key, while W_R^Kr_k is the relative positional piece of the key. The global parameters u and v can be seen as providing "default query projections" that are active regardless of the specific query token. This ensures that important content and positional patterns always receive some attention.

R follows the same sinusoidal encoding function as the original Transformer, just applied to relative positions instead of absolute positions:

where k is the relative distance between positions, ranging from −L to +L, and j is the dimension index from 0 to d_model.

Computing

requires careful thought to limit the complexity of the problem. With N tokens, we could naively extract all the N² related r_k since there are N² pairs of tokens. However, there would be many duplicate vectors since different pairs of tokens have the same distance between them. Instead, we can directly use the whole R matrix and pass it through the linear layer W_R^K:

R is a (2L+1) x d_model, and W_R^K is a d_model x d_model matrix, therefore the "positional keys" K_R is a (2L+1) x d_model matrix. As a consequence

is a (2L+1) dimensional vector which leads to a (2L+1) x N content to position alignment matrix for N queries.

Let's make sure we understand the different tensors in play. We have the queries Q = [q₁, …, q_N], the keys K = [k₁, …, k_N], and the positional equivalent to the keys K_R =[W_R^Kr_-L, …, W_R^Kr_L]. The relative positions in K_R are not aligned with the relative positions of K with respect to Q. This means we cannot simply add

because of the misalignment in the key ordering. We just need to change the indices of the elements in (Q + v)^T K_R. For query position i, we need:

• For key position 0: a relative distance i−0 = i

• For key position 1: a relative distance i−1

• For key position 2: a relative distance i−2

• And so on...

In other words, for the i-th row of the content to position alignment matrix P = (Q + v)^T K_R, we need to select values in a specific pattern. The solution is to reshape and shift the P matrix so that the correct relative position scores align with the positions we need in the final attention matrix.

In the case of Transformer-XL, remember that the sequences are processed in segments of size n « N, and the attention mechanism per segment follows an O(n²) time complexity. W_R^KR also operates on segments, and the time complexity is

The shift operation needs at least O(n²) operations to reindex the alignment score matrix. Therefore, the overall time complexity associated with the relative positional encoding follows the asymptotic behavior O(n²).

Transformer-XL showed improvements in the model's ability to utilize longer contexts. When trained with segments of length 128 but evaluated with various attention lengths, the model showed continued improvements in perplexity up to 640 tokens. The paper reported improved perplexity metric when increasing evaluation context length beyond training length, something absolute encoding couldn't achieve.

ALiBi: Attention With Linear Biases

The relative positional encoding developed in Transformer-XL is designed to handle very long sequences, but it adds complexity to the attention layer, and, while it can handle longer contexts through its recurrence mechanism, it was not specifically designed for extrapolation to arbitrary lengths beyond training. ALiBi (Attention With Linear Biases) was introduced in 2021 as a simpler approach that could easily be extrapolated to much longer sequences than the ones seen during training. It does not require any model parameters and can be expressed as a penalty on the alignment score:

where j-i is the distance between the query q_i and the key k_j, and m_h is a head-specific constant, with h being the index of the head. In the case of causal language modeling, we always have j-i ≤ 0, leading to lower attention for far-away tokens. ALiBi tends to sacrifice true long-range modeling for extrapolation capability. m_h is chosen as:

where h ∈ [0, …, n_head -1]. For example, if we have 8 heads, the first head would have m₀ = 2⁰ = 1, and the last head m₇ = 2^-7 = 0.0078125, with a spectrum of the different intermediary slopes in between. For j-i = 1000, for example, we get m₇(j-i)~ -7.8, which is a substantial penalty to capture long-range interaction between tokens. This suggests that ALiBi may not be optimal for tasks requiring very long-range dependencies (like book-length coherence), but it represents a valuable engineering trade-off that improves efficiency without sacrificing performance on many practical tasks.

Despite this limitation, ALiBi works well for extrapolation for several reasons:

• Graduated attention ranges: The different head slopes create a spectrum of attention distances. While no head truly specializes in very long-range attention, the collection of heads creates a gradient of focus distances.

• Local coherence dominance: Language has a hierarchical structure where local coherence (within paragraphs or nearby sentences) often matters more than very distant relationships. The bias aligns with this natural property of language.

• Information propagation: Information can still flow across long distances through multiple layers of the transformer. Even if direct attention across 1000 tokens is penalized, information can propagate through intermediate positions across layers.

• Relative vs. absolute positioning: Unlike sinusoidal embeddings that break down completely outside their training range, ALiBi's linear bias at least provides a consistent, predictable signal at any distance.

The paper showed that a model trained on 512 tokens could handle sequences of 3072 tokens with better perplexity than a sinusoidal model trained on 3072 tokens. A 1.3 billion parameter model trained on 1024 tokens achieved the same perplexity as a sinusoidal model trained on 2048 tokens when evaluated on 2048-token sequences. As input length increases beyond training length, sinusoidal models' performance degrades almost immediately while ALiBi's performance continues improving up to ~2-3x training length before plateauing.

RoPE: Rotary Position Embedding

The Rotary Position Embedding (RoPE) is now one of the most common strategies used to inject the relative positional information within the attention mechanism. The idea behind RoPE is to rotate the keys and queries based on the position of the related tokens in the input sequences. This will inject the absolute positional information directly into the queries and keys. Let's look at a toy example to understand the logic. Let's consider a 2-dimensional query q_i and a 2-dimensional key k_j. To rotate 2-dimensional vectors, we use rotation matrices:

Read more

Over the past few years, I have realized that the logic behind self-attention still eludes many people who want to dive deeper into the field. For me, implementing from scratch is the best way to learn this core element of every LLM. Once the self-attention implementation becomes more intuitive, it opens the doors to understanding all the small improvements that have led to the level of maturity we have today in the field of LLMs.

Signup

This is an opportunity to talk face-to-face and ask questions. The event is on May 2nd at 9:30 AM PST. I hope to see you there!

Signup

The Real-World LLM Engineering Roadblocks You Face Today

👋 Transitioning from General ML to LLM Specialization: You’ve built recommendation engines or classifier models, but moving into Transformer‑centric development feels like learning a whole new discipline—no clear roadmap exists.

👋 Lack of LLM‑Specific Career Path: You see “LLM Engineer” roles popping up on LinkedIn, but your current CV only shows “Data Scientist” or “ML Engineer.” You need hands‑on projects and artifacts to credibly make the jump.

👋 Career Stalled by “Academic” Skillset: You can recite Transformer papers, but when asked, “Have you shipped an LLM feature end‑to‑end?” you have no answer—and no portfolio to prove it!

👋 Prototype Meltdown Under Production Load: You’ve fine‑tuned a small model locally, but when you switch from 1 to 100 concurrent requests, your GPU memory spikes and inference grinds to a halt, because you’ve never applied continuous batching, KV caching, or paged‑attention in a live setting.

👋 RAG Integration Headaches: Turning a standalone model into a live, Retriever‑Augmented Generation service becomes a multi‑week integration nightmare.

How this course will help you

Because we’ve packaged every stage of the LLM lifecycle, from career transition to production rollout, into a six‑week bootcamp that:

✅ Guides Your Career Pivot: You’ll emerge with six polished GitHub projects, a deployment playbook, and RAG demos that transform your resume from “ML generalist” to “LLM Specialist.”

✅ Attacks Each Pain‑Point Head‑On: Attacks each pain point head‑on with six job‑mirroring projects (from scratch → RLHF → scaling → deployment → RAG), so you never waste time on dead‑end tutorials

✅ Live Code‑Along Workshops & Office Hours: Tackle your own fine‑tuning bugs, scaling hiccups, and deployment errors alongside Damien in dedicated sessions, so you get hands‑on fixes for the exact issues you’ll face on the job.

✅ Ready‑to‑Use Repos & Playbooks: Grab our curated starter code, development scripts, deployment templates, and debugging checklists, so you can plug them straight into your next project without reinventing the wheel.

✅ A Portfolio of Six Production‑Grade Projects: Leave with six end‑to‑end deliverables, from a Transformer built from scratch to a live RAG API, ready to showcase on GitHub, in performance reviews, or to hiring managers.

No more scattered blog-hopping or generic bootcamps, this is the only cohort where you’ll master Transformer internals and ship production‑grade LLM systems while making the career leap you’ve been aiming for.

What You’ll Actually Build and Ship

Across six hands‑on projects, you’ll deliver deployable LLM components and applications, no fluff, just job‑ready code:

✅ A Modern Transformer Architecture from scratch: Implement a sliding‑window multihead attention to slash O(N²) to O(N·w), RoPE for relative positional encoding, and the Mixture-of-Expert architecture for improved performance, all in PyTorch.

✅ Instruction‑Tuned LLM: Fine‑tune a model with supervised learning, RLHF, DPO, and ORPO for instruction following on a real benchmark and compare performance gains.

✅ Scalable Training Pipeline: Containerize a multi‑GPU job with DeepSpeed ZeRO on SageMaker to maximize throughput and minimize cost.

✅ Extended‑Context Model: Modify RoPE scaling, apply 4/8‑bit quantization, and inject LoRA adapters to double your context window.

✅ Multi‑Mode Deployment: Stand up a Hugging Face endpoint, a vLLM streaming API, and an OpenAI‑compatible server, all Dockerized and optimized for low latency.

✅ End‑to‑End RAG Chat App: Build a FastAPI backend with conversational memory and a Streamlit UI for live Retrieval‑Augmented Generation.

By the end of Week 6, you won’t just know these techniques, you’ll have shipped six production‑grade artifacts, each reflecting the exact pipelines, optimizations, and deployment routines you’ll use on the job.

Live & Recorded Content: Reinforce, Deepen, Accelerate

✨ 12 Interactive Live Workshops (3 hrs each): Each session follows the Concept → Code flow. I’ll introduce the day’s core topic (e.g. self-attention, LoRA, vLLM optimizations, ...), and we’ll implement the features step‑by‑step in code so you see exactly how theory maps to code. Bring your questions!

✨ 10 + Hours of On‑Demand Deep‑Dive Lectures: Short videos (10–20 min) on Transformer internals, fine-tuning tricks, deployment optimizations. Watch before each project to hit the ground running. Step through every line of code at your own pace; perfect for review or catching up if you miss a live session. Downloadable slide decks, annotated notebooks, and cheat sheets you’ll reference long after graduation.

Why This Matters: Live workshops turn recorded concepts into actionable skills. You’ll see how theory maps directly onto code, get instant feedback, and internalize best practices. Then, recorded lectures become your asynchronous safety net, letting you revisit tricky topics, prepare for upcoming labs, and solidify your understanding on demand.

Let me know if you have any questions. I hope to see you there!