In my work with G7–G12 students at Force Academy, a research initiative in Chongqing, I sometimes take a project that appears to be finished and give another student responsibility for using, checking, or modifying it. The handoff is a test of understanding: Can the work still make sense when its creator is no longer there to translate it? The original author knows what every variable means, why a particular decision was made, which tool was used, and which parts can safely be changed. The second student has only what the project makes visible.

Generative AI makes the handoff more revealing because it lowers the cost of producing a convincing first version. Students can generate code, analysis, prose, visualizations, and prototypes far more quickly than they could a few years ago. In a classroom, however, a project is useful only insofar as it shows what the student understands and can defend. AI can supply missing code, smooth over incomplete reasoning, or turn a rough idea into fluent prose, so polish is no longer strong evidence of understanding by itself.

A concrete example came from a student paper on AI and labor economics, Verification Responsibility in AI-Applicable Occupations. The paper asked what remains economically important when AI makes candidate outputs cheap but a person still has to check those outputs, decide whether they are safe to use, and remain responsible for accepting them.

The lead student developed the conceptual model and most of the analytical workflow. A second student was given a narrower job: rerun the main analyses independently, record the versions of the source data, document how occupations in one dataset were linked to occupations in another, and compare the reported tables and figures with the underlying outputs. What looked like procedural verification quickly became a substantive review of the study.

The first problem was reproducibility. The analysis ran on the original machine, but too much of the workflow depended on choices that lived in one student’s memory. To reproduce it elsewhere, the team had to record the data transformations, write down the rules for linking occupational datasets, and keep intermediate results so another person could see how the final numbers had been produced rather than simply rerunning a script and trusting the output.

The more consequential problem was in the measurement design. An early version of the study used some of the same autonomy and leadership survey items in two different measures: verification responsibility and adaptive capacity. The software had calculated the relationship between those measures correctly, but part of the apparent relationship could have been created by the fact that the two measures shared some of the same inputs. In other words, the code was doing what it had been asked to do; the question was whether the quantities being compared had been defined cleanly enough.

The team rebuilt the measures so they no longer shared survey items, removed ambiguous items, and revised the workflow so source versions, cross-dataset links, exclusions, intermediate files, and reported results could be traced. Later versions added alternative specifications and robustness checks. What began as “please rerun this” ended up forcing a redesign of the study.

The handoff changed how I read student work. A second user does not merely test whether the files have been packaged neatly; they ask the awkward questions the creator may have stopped noticing. Why is a particular variable there? Which conclusion depends on an assumption that was never made explicit? In this project, those questions reached back into the measurement design.

The educational value of the handoff comes from the fact that schoolwork and production software have different ends. Professional developers learn a great deal after software leaves their hands because they have to deliver it, explain it, support it, maintain it, and respond when someone uses it in an unexpected way. Student projects usually disappear when a course ends, and that is fine; the artifact is not the goal. What matters is importing some of the learning that comes from delivery and maintenance: explaining decisions, supporting someone else’s use, and revising the work when private context is no longer available.

At Force Academy, the exact task depends on the project. For a computational model, another student may change a parameter or replace one rule and then ask what should remain stable. For an AI-supported workflow, the job may be to move it to a different model or environment. For a data analysis, the second student may try to reconstruct one result from the recorded source files and transformations. When that transfer fails, I want the original student to decide what the other person actually needed to know and what should be changed in the project, rather than simply standing beside them and explaining it.

I use the word “handles” for the visible points of entry that make that possible. A handle gives another person something concrete to inspect or change without first receiving a private briefing. It might be a clearly named parameter, an explicit assumption, a test that records an earlier failure, a replaceable component, a retained intermediate dataset, or a prompt whose important context is written down instead of buried in a private chat history.

Handles also make disagreement more productive. An exposed assumption can be challenged; a component with a clear boundary can be modified or replaced without rebuilding the whole project; and a numerical result linked to its source data and transformations can be checked rather than accepted on authority.

Outside school, AI systems expose the same weakness. A prompt, workflow, skill, or agent configuration may look portable because it can be copied, while the conditions that make it work remain elsewhere: tool access, file structure, model behavior, private memory, naming conventions, or small decisions the creator never needed to document. Moving the artifact often reveals how much of the working system was never inside the artifact in the first place.

A first successful run is therefore weak evidence. It tells me that a particular combination of person, model, tools, data, and context produced an output once. A stronger test is whether another person can reconstruct what matters, distinguish assumptions from implementation details, and make a change without destroying the logic of the work.

Tim O’Reilly’s idea of an architecture of participation gives me the broader frame. A technical artifact becomes more useful when other people can enter it without negotiating privately with its creator. In software, that usually means interfaces, conventions, documentation, modularity, and replaceable components. In data analysis, the same principle shows up as recorded source versions, visible transformations, retained intermediate results, explicit assumptions, and enough analytical logic for someone else to rerun or challenge the work.

For education, the milestone I care about is whether the student understands the work well enough to make some of that understanding available to someone else. A second student who can use, question, and extend the project gives me evidence that a polished first run cannot.

AI expands what one person can build, including what a teenager can attempt without a large technical team. The expanded scope is a real gain, but it also makes successful generation easier to confuse with understanding. When I want to know what a student really understands, I now learn more from the first serious handoff than from the first successful run.

One of my biggest pet peeves is the Monday morning quarterback, the person who second-guesses every coaching decision after the fact. In The Monday Morning Agent, we’re building one that actually checks the data.

Every other Monday, I’ll take an argument NFL fans can’t stop having and hand it to an AI agent. We’ll use about a quarter-million plays from five complete seasons of public nflverse data, plus the current season as it unfolds. Each of our eight episodes will give the agent another capability. By the finale, it’ll run the desk unattended, and we’ll check what it investigated and what it correctly refused to answer.

We’ll start with “You have to establish the run.” I want the agent to explain what that claim means and what evidence would support it.

Define the claim before querying the data

“Establish the run” can mean that running early pays off late, wearing down the defense, or setting up play action. These are very different claims and they need different data. For this first episode, I focused on whether teams that run more often early in a game win more often.

Before the agent called the tool, I asked it to write one sentence explaining what would have to be true in the data for the claim to hold, including when in the game it applies. “Establish” here means early. A whole-game number doesn’t really test that. Give your agent the same assignment, and check that it’s answering the question you intended.

Give the agent a job you can check

Before any agent touched this, I ran the analysis code myself. The tool did the counting; the agent interpreted the results. Then I wrote the policy in a Markdown file, spelling out its job, the read-only tool it may use, and what its answer needed to include. It has to explain what the plays can’t tell us. It may not say running causes winning.

Try questions outside that job too. When I asked which team I should bet on, the agent refused without calling the tool, because that request is beyond the scope of our agent. But when a viewer asked about running trends, the results got tangled up with the “establish the run” instructions. I had to clarify what I was asking, and still wanted to see the code and validate the answer. As you give an agent more responsibility, check both its results and whether it understood the task.

Make the filters part of the answer

“You have to establish the run” sounds airtight at first. Over the last five seasons, teams with the highest run share, the percentage of their plays that were runs, won 77.7% of their games. Those with the lowest run share won 19.2%. I initially looked at that and thought, “Problem solved, we’re done.” Most of that gap, though, reflects teams that were already ahead running out the clock.

When the agent looked only at first-half running in games within three points at halftime, the result flipped. Teams with the highest first-half run share won 46.3% of the time; those with the lowest won 55.3%. This season through Week 5 points the same way, but each group contained only 11 team games. One game could swing it. That doesn’t mean passing causes winning. We’re measuring association, and the filters change which games we’re comparing.

One filter changed the whole story, which is why the filters have to be on-screen. I wanted to see how many rows remained and which commands produced the results so I could go back and check. Make those details part of your agent’s answer, including when it’s working with a smaller sample.

A few numbers from last week’s games

In Week 5, the team that ran more in the first half won 6 of the 15 games, lost 8, and tied 1 (Giants at Washington). Over the past five seasons that team won 54.3% of the time, so last week came in below the average. That’s not unusual, though. In 27 of the 95 regular-season weeks since 2021, the team that ran more early finished the week with a losing record. One week of games doesn’t say much on its own, which is why the agent reports five seasons alongside the current one.

One more data point: The Monday night game, Buffalo at the Rams, was the first the agent hadn’t seen. Buffalo ran on 60% of its first-half plays to the Rams’ 35%, led by six at halftime, and won by three in overtime. Those 199 plays are in the data now, and the close-at-halftime numbers from the episode didn’t change.

Next up: “Ban the Tush Push”

On October 26, we’ll tackle “Ban the Tush Push.” Not one of the 260,927 play descriptions from 2021 through this season’s October 11 games contains the words “tush push.” After adding Monday night’s game, we have 261,126 descriptions, and “sneak” appears in only two, both from 2021. The data isn’t going to hand us a definition.

Before the agent can judge the play, it has to define it. I’ll challenge it to write that definition into its first permanent document, a shared data dictionary, and cite it in every ruling from then on. Bring your own definition, because the answer depends on which one you pick.

O’Reilly members can sign up on the O’Reilly learning platform to join me live on October 26, 8:00am–8:45am PT / 11:00am–11:45am ET. Everyone can follow along on Radar every other Monday, with a brief pause for the holidays. Watch episode one above, and we’ll build on it in the next session.

This summer, I built a small experiment called ContextCue. The idea came from something I kept seeing during my internship. The engineers around me were using AI everywhere in the software development process. GitHub Copilot and Claude models helped with specs, code, reviews, and tests. The team also maintained reusable skills that taught coding agents how the team worked. One of those skills could take a UI design and turn it into working frontend code.

The code was moving faster, but the understanding around the code did not seem to move faster with it. The software was being built by nine teams spread across three continents, with one team working closely with the customer. A request might begin in a customer conversation, move through one team, then reach another team several time zones away. By the time the work reached engineering, some of the original meaning could be missing.

That was the problem I wanted ContextCue to explore. ContextCue takes recorded meetings and tries to preserve more than a transcript or summary. It keeps the spoken discussion, important visual context from the screen, the decisions that were made, the reasons behind them, and the source they came from. The goal is to create a shared piece of context that can move with the work from one team to another.

It is still an experiment, but the problem behind it felt very real. As far as I could see, a lot of time in distributed software teams went into reaching a common understanding before the coding even started. AI was already helping with the final stages of software development. I wanted to see whether it could help earlier, while teams were still trying to understand what they were building and why.

Where the context disappears

Meetings were already being recorded, and tools like Microsoft Teams could produce an AI summary. On my team, that summary often arrived as an email and stayed there. That worked well enough for the people who attended the meeting. It was less useful for the team that picked up the work later, especially if that team was in another country and had not seen the original conversation.

There were two other problems. A summary usually covered one meeting, while the understanding behind a feature was often built across several. Different teams might discuss the same request days apart, each adding a new constraint or interpretation.

A transcript could also miss important visual context. Many of our conversations involved screen sharing. Someone might point at a page and say, “This is too slow,” or highlight a part of a workflow while explaining what needed to change. The words alone did not always carry the meaning. That is where I started thinking about context as something larger than a transcript.

A simple example

Imagine that a customer asks for a “faster report.” During the call, the customer shares a screen, points at a page that is slow to open, and explains that managers use it every morning before a meeting. Everyone on that call understands that “faster” means the page should open more quickly.

Two days later, another team in another time zone discusses the same request. Nobody in that meeting saw the original screen. They read “faster report” as “the report should finish running sooner” and begin planning backend work. Both meeting summaries might contain the same phrase:

Make the report faster.

Nothing in that sentence shows that the two teams mean different things. The problem only becomes visible when the customer sees the result.

This is a simple example, but it captures the issue I kept noticing. Requirements move through people before they ever become code. If the meaning changes during that process, AI can help a team implement the wrong interpretation very efficiently.

The sports version of the same problem

Sports gave me another way to think about it. Imagine a soccer match where a player misses a late chance to score a goal. The match report can record the shot, the player, the minute, and whether it was on target. Those facts are useful, but a coach watching the replay sees much more. The winger may have delivered the ball too early. A defender may have forced the striker onto a weaker foot. A teammate may have made a run that opened the space.

The event is only one part of the story. A coach learns from the sequence around it.

Jake M. Hofman, Daniel G. Goldstein, and David M. Rothschild use a related sports framework in a Harvard Business Review article where they describe AI as a steroid, a sneaker, or a coach, depending on how it helps people. The “coach” idea is the one that stayed with me. A useful AI system does not only help people produce more. It can also help them make a better decision by bringing forward the evidence and experience that came before. That is close to what I wanted ContextCue to do for teams.

What I built

ContextCue starts with a meeting recording. FFmpeg handles the media, and Faster-Whisper transcribes the speech. The transcript is kept in time-stamped segments so that any later answer can point back to the moment where something was said.

The system also looks at selected frames from the screen share using a vision model such as Qwen VL. That helps preserve cases where the meaning depends on what someone was pointing at rather than only on the words they used.

The final output is organized around a request or decision instead of around a meeting. For example:

# Request: Faster report

What the customer meant:
The report page opens too slowly.
Source: customer call, shown on screen

Why it matters:
Managers open it before a morning meeting.
Source: customer call, spoken

What was planned:
Make the report finish running sooner.
Source: team planning meeting

Open issue
The current plan does not match the original request.

The format is deliberately simple. I used Markdown because a person can read it, a language model can use it, and it can be stored or versioned without depending on a special interface. When a later meeting touches the same request, the page can be updated rather than creating another disconnected summary. That changes the unit of memory from meeting to topic.

Why that matters for product work

One of the clearest uses I saw was product management. A product manager writing a user story usually works from some combination of tickets, meeting notes, customer feedback, and conversations with engineers. The important details are often spread across all of them.

If the product manager gives an AI model only the latest meeting summary, the model sees only one slice of the history. A ContextCue page can give it more. The page might contain the original customer request, the business reason behind it, a technical constraint raised later, a decision from product, and an unresolved question that still needs an answer.

That gives the model a much better starting point for creating a user story. The model itself has not changed. What improves is the material it has to work from. This is an important distinction because a lot of AI work focuses on prompts and models, while the quality of the context often decides whether the output is useful.

How the context grows

Each new meeting can update an existing page or start a new one. Over time, that creates a topic-by-topic record of how the work evolved. That is different from having a folder full of meeting summaries. A folder forces someone to remember which meeting contained the useful detail. A topic page collects the relevant pieces in one place.

The stored context can also be searched. I used embeddings and Qdrant to retrieve relevant parts of previous meetings. A question like “What did the customer mean by faster?” can return the answer along with the meeting and timestamp that support it.

Other requests need something more structured. “Find Tuesday’s meeting” is a lookup. “Why did we change this requirement?” may need information from several meetings. “What is still unresolved?” should look for open questions rather than just similar text.

ContextCue can expose those operations through the Model Context Protocol (MCP), so a language model can choose the right tool for the request. The main point is not the protocol itself. The useful part is that different types of questions can use different ways of accessing the same shared context.

Why recorded conversations are worth keeping

One study made me think recorded conversation was especially valuable. In “Generative AI at Work,” published in the Quarterly Journal of Economics in 2025, Erik Brynjolfsson, Danielle Li, and Lindsey Raymond studied an AI assistant used by customer-support workers.

They found that lower-skill and less experienced workers improved the most. The system appeared to help them communicate more like higher-skill workers. What interested me was that some of the useful knowledge came from past conversations.

A company contains a lot of knowledge that never becomes a formal document. It appears in explanations, customer calls, design discussions, examples, corrections, and small decisions made during meetings. Recorded meetings are one place where that knowledge already exists.

The first benefit I noticed was personal

As an intern, one of the hardest parts of joining a large project is learning why things are the way they are. You can read the code and still not know why a certain design was chosen. A ticket may describe what to build without explaining the discussion that produced it.

The usual solution is to ask someone. That is often the right thing to do, but it also means interrupting someone who already has work to finish. A shared context layer could make some of those questions easier to answer without another meeting or another message across time zones.

The same idea could help with onboarding, repeated meetings, handoffs between teams, and situations where someone needs to understand an old decision. I can see those possibilities from where I sat as an intern, but they still need proper measurement.

What worked, and what is still hard

My testing so far is limited. I created a small set of questions where I already knew the correct source. ContextCue retrieved the right source first about nine times out of ten and returned it within the top three results every time in that test set.

That was encouraging, but it also made the harder parts of the problem clearer. A real company may use several different meeting tools. Different teams may have different rules about what can be recorded. Some context should be visible across the company, while other context should stay within a team. People need to know when meetings are being recorded and how those recordings will be used.

There is also a time problem. A team may make one decision in March and change it in April. The March decision should not disappear, because it explains how the team got to the current position. At the same time, the system should make it clear that the April decision replaced it.

That kind of history is useful, but it is more complicated than simple search. It requires rules around ownership, permissions, time, and trust. ContextCue does not solve all of that.

Where this goes

The thing that stayed with me from the internship was the contrast between how much AI had changed coding and how little it had changed the handoffs around coding. The engineers already had strong tools once a requirement was clear. The slower part often happened before that point, while teams were trying to understand a request, add constraints, pass it to another group, and keep everyone aligned.

ContextCue was my first attempt at working on that part of the process. The experiment is still small, and the technical system will need to change if it ever moves into a real company environment. The useful idea, for me, is that meeting recordings can become more than archives. They can become a shared record of how a team arrived at a decision.

If AI can help preserve that history and bring it forward when the next team needs it, then it may improve more than the speed of writing code. It may improve how teams carry understanding from one part of the organization to another.

New AI tools are making business data easier to explore. They’re also giving researchers new ways to study disease outbreaks and physical systems. On the latest episode of This Week in AI, we examined developments from OpenAI, Anthropic, and Mistral, new research on the economics of robotics, and how scientists are using AI to tackle difficult problems.

New AI tools put more capabilities directly in users’ hands

OpenAI’s new Intelligent UI feature lets ChatGPT generate visual explanations and interactive tools directly in conversations. With the launch of MCP Events, supported connected applications can notify ChatGPT when something happens. If a user has asked ChatGPT to watch for that event, it can follow their instructions and take requested actions without waiting for another prompt.

Anthropic’s Claude Dashboards brings natural-language queries and interactive visualizations to company data. Claude Motion adds another visual capability, turning reports and charts into editable animations. Self-service dashboards can create a range of problems though. For example, employees may reference incomplete data or apply different definitions to the same metric, then share their results as authoritative. As more employees take on exploratory analysis, BI teams will need to expand their role by maintaining reliable data and shared definitions, setting clear governance and access rules, helping employees interpret results responsibly, and monitoring how dashboards are used.

Beyond the big two, Mistral previewed Large 4, a trillion-parameter open-weight model designed for coding, agents, and multimodal tasks. The company plans to release the model’s weights soon, giving developers access to another large model for building and running AI applications.

As AI agents gain access to more applications and company data, organizations will need clearer rules about what those agents can access and when humans should intervene. Recent internal tests at Anthropic showed how difficult it can be to control agents once they have access to external systems. The company reported cases of Claude circumventing restrictions, including copying server files and submitting fabricated forms, and subsequently restricted live internet access in internal tests until they’re better able to monitor their systems. OpenAI also announced plans for text watermarking in the EU, though editing and short passages can make watermarks difficult to detect reliably.

Robotics research puts automation forecasts in perspective

A new Anthropic study offered a useful counterpoint to predictions of widespread job displacement. Researchers found that existing robots could perform tasks accounting for roughly 34% of US work hours, but doing so would be cost competitive for just 0.3% of work hours.

Most of these tasks take place in structured environments such as factories, where robots can operate more reliably than in unpredictable settings. When researchers combined the potential exposure of cognitive work to language models with physical work exposed to robotics, the share of work hours potentially affected reached 81%.

For employers and policymakers, exposure figures help identify which tasks could change, but they don’t predict how many jobs automation will eliminate. Deployment costs, working conditions, and the need for human judgment will influence how quickly employers adopt these technologies. As we discussed during the audience Q&A, jobs are already changing, particularly in technical fields, even without evidence of massive displacement. For now, employers may get more value from helping workers develop new skills than from planning around speculative job-loss forecasts.

AI is opening new avenues for scientific research

AI is helping researchers tackle problems in mathematics, biology, space, and public health. OpenAI released a collection of AI-generated mathematical results that still require substantial human verification. Biohub announced a $1.8 billion investment in biological data and computing infrastructure, with support from Google DeepMind, Meta, and other partners, to advance research into cellular behavior and drug treatments.

Other applications are putting existing data to work in new ways. Breakthrough Listen is using AI to search data from radio telescopes and other instruments, as well as scientific archives, for signs of extraterrestrial technology. Google’s Population Dynamics Foundation Model creates a kind of “fingerprint” for each place by combining signals such as search trends, movement patterns, and environmental data. Researchers can add these location profiles to existing public health models to improve estimates, including forecasts of dengue in Mexico and cholera outbreaks in the Democratic Republic of the Congo.

In Europe, Mistral is investing in physics-based AI through a new research hub in Munich, where it will work with industrial partners including BMW and Siemens Energy. The company’s acquisition of Austrian startup Emmi AI earlier this year added expertise in modeling physical systems. A separate recent benchmark highlights just how challenging this broader goal remains: Leading AI video models frequently violate physical laws. The top-performing model scored just 58 out of 100 on the benchmark’s scale, a low result that shows how far these models still have to go. For developers building world models, the findings are a reminder that visually convincing simulations still need to demonstrate real-world physics accurately.

What’s next

Wider access to AI tools will make human expertise in interpreting data, evaluating models, and understanding their limitations increasingly valuable.

We’ll be back on Monday, October 19, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

In my previous article, I argued that agentic development needs enough specification to make correctness checkable. A quick prompt can get implementation started quickly, but someone still has to decide whether the result does what we wanted. I left a fairly practical question unanswered, though. Once the agent has written the code, how should we review it?

You can give an agent a clear task and get back a serious implementation with tests. The build passes and the code looks reasonable. Now somebody has to approve the pull request, and that person has roughly the same amount of time they had before code generation became cheap.

Reading every changed line is one way to approach this. But if each increase in generated code requires a similar increase in human inspection, review will keep limiting how many changes we can accept. We need a way to decide where that inspection is worth the effort.

My approach is to make evidence a condition of accepting agent-generated code. Agree on the intended behavior and the expected blast radius before implementation starts. Then use the evidence to check for drift and decide which parts of the code need direct inspection. So far, that may sound like test-driven development. But what happens when every test passes and the agent has added behavior nobody asked for?

A small diff can have a large blast radius

Let’s use a small fictional example. Suppose we are adding an approval step to a refund service. Refunds above €100 require approval from an authorized employee before the service sends the payment request. Exactly €100 does not require this additional approval. The existing rules about who can request a refund continue to apply.

An agent implements the approval step and adds tests. While updating the payment integration, it also adds retries after a timeout. That seems reasonable enough. Network calls fail, and retrying them is a familiar engineering practice.

But a timeout leaves us with another question. The provider may have issued the refund before its response was lost. Sending the request again could issue a second refund. Whether that can happen depends on how the provider handles repeated requests and how our code uses that behavior. The approval feature may be correct while the added retry introduces a separate problem. I can see a human review focusing only on the requested feature and easily missing the new decision about payment behavior.

By blast radius, I mean how far the effects of a mistake can spread. If the agent puts the retry in a shared payment client, it may affect every refund, including those below the approval threshold. The change now reaches beyond the feature we asked for. Counting the changed lines won’t tell us that.

Treat blast radius as a review criterion on its own. Establish which operations a change is allowed to affect, then check whether the implementation stays within that boundary. Also consider how hard a mistake would be to undo. Deploying the previous version won’t recover money that has already been refunded twice.

Look for drift in both directions

This is the kind of drift I want a review process to expose. We need to check that the requested behavior exists and examine additional behavior that could affect the system.

There is an established name for following these relationships in both directions: bidirectional traceability. NASA’s software engineering guidance describes tracing requirements through implementation and verification, including finding functionality that has no parent requirement. I think that the basic idea translates quite well to an ordinary pull request.

For each important requirement, we should be able to find the implementation and the checks that support it. Looking back from the implementation, we should be able to explain why a change to behavior belongs in this pull request. An extra private method will usually need little discussion. A retry that changes how money moves needs a reason we can review.

Of course, an unplanned addition isn’t automatically a mistake. The retry might be necessary. Implementation often reveals something the original specification missed. We can accept the new behavior once we understand its consequences and agree on how to check it. Until then, it remains an open decision.

An agent can help identify these relationships. Give it the agreed specification and the actual repository changes, and ask it to find behavior that is missing or unexplained. For the refund service, that should include the changed payment client even if the agent’s own summary only mentions the approval feature. Start with the changed files so that the summary doesn’t determine what gets examined. Follow those changes into their callers, too. A shared client can change the behavior of code that has not been edited.

The tests can share the same mistake

While this is a general direction for our attention in a review, the links themselves tell us very little about correctness. A method can reference the right requirement and still implement it incorrectly. A test can have exactly the name we hoped to see and check the wrong result.

Consider the threshold again. Suppose the agent reads “above €100” as “€100 and above.” It uses the same interpretation when writing the tests. The implementation requires approval for exactly €100, and the test expects that behavior. Everything passes because the code and the test agree.

Running that test in a separate CI environment will reproduce an agreement but it won’t discover that the expected result was wrong. We still need someone to check the meaning of the assertion against the agreed behavior. Spend some of the review time on these expected outcomes. For a requirement like this, agreeing on the boundary examples before implementation is quite manageable. A refund of €100 proceeds without the additional approval. One cent more must wait. The reviewer can then inspect whether the tests actually observe that difference. Have the test observe what the service sends to the payment provider. Checking that an approval request was created is insufficient if the service can also send the payment immediately. For refunds above the threshold, the test should check that the payment request cannot leave the service before approval. That checks the rule at the point where the refund is requested.

For an important rule, you can go a little further. Temporarily remove the approval check in a controlled test run and see whether the relevant test fails. If it stays green, it gives us very little protection. Google’s code review guidance asks reviewers to consider exactly this kind of question: Will the tests fail when the code is broken?

Use another agent to propose cases we forgot and examine whether the assertions match the requirement. Keep the actual checks available to the human reviewer, though. A different model can still produce the same miss-interpretation. There is some research backing up that caution. A 2026 study of LLM requirement-conformance judgments found models rejecting correct implementations and accepting faulty ones. The study used programming benchmarks, so it does not tell us how every review agent will behave in a production repository. It does give us a reason to look beyond a model’s confident explanation when deciding whether code meets a requirement.

Make the evidence easy to inspect

Require a short review record with each pull request. It should connect the intended behavior to the checks and expose decisions that remain open. For our hypothetical refund change, it could begin like this:

• Agreed behavior: Refunds above €100 wait for authorized approval before payment. Exactly €100 does not need that additional approval.
• Evidence: Boundary tests check when the payment request is sent. The review links to their assertions and the CI results for this revision.
• Additional behavior: The payment client now retries after a timeout.
• Blast radius: If the retry applies to the shared client, all refund requests may be affected. Review its existing callers.
• Open question: The tests do not cover a provider completing the refund before the timeout. Safe retry behavior still needs to be established.

Start the review with this record. Check that the approval assertions express the agreed rule and that the results support it. Then inspect the retry path, where both the evidence and the intended scope are incomplete. Require a test covering the uncertain provider response or remove the retry from this change. The open question has become a concrete review decision. Link each claim to the checks that support it. “Tests passed” is too broad to help much. The reviewer needs to see which tests ran against the revision being approved. If the agent changes the implementation after that run, refresh the evidence.

Review changes to the checks

Compare the checks with the agreed baseline as well. Suppose an earlier acceptance test expected a €100 refund to proceed, but the agent updates it to expect an approval request. The new implementation and the new test now agree. Comparing both with the agreed requirement exposes the change in meaning. An agent does not have to be trying to hide anything for this to happen. Updating a test to match new behavior is part of everyday development. The reviewer needs to decide whether that new behavior was intended. Changes to acceptance criteria deserve the same attention for the same reason.

Sometimes the new behavior will be the right approach. Perhaps the business actually wants approval starting at €100. Fine, we can change the requirement. Record that decision and review its consequences before accepting the implementation. Quietly changing the spec to match the code leaves us with agreement and no explanation of how we got there.

Set review depth by blast radius

Match the depth of review to the potential impact and the strength of the evidence. A report can only expose the gaps it detects. Tests can pass even when the code relies on a wrong assumption. Require direct inspection of the payment path because a mistake can move money twice. Follow the affected calls even if some of them sit outside the diff.

For a routine change with limited effects and established checks, the reviewer can spend less time reconstructing the implementation. The justification is specific: The affected behavior is understood, and the checks address its likely failures. Where the blast radius is unclear, investigate it before approving changes. Occasionally inspect changes that the process treated as routine, too. If those reviews keep finding missing assumptions, improve the checks before relying on them to reduce inspection further. This gives the team a way to test its review policy against the code it is actually accepting.

Make evidence part of the development cycle

Here is the process I recommend. Before generation starts, a developer agrees on the behavior and its expected blast radius with whoever owns the requirement. The agent implements within that scope and prepares a short review record. CI runs the checks against the proposed revision. The reviewer examines the evidence and investigates deviations, with direct code inspection where the consequences justify it. Release through a controlled rollout, then use what happens in production to improve the checks and the specification.

Make the review record part of the definition of done. The developer submitting the change owns its accuracy, even when an agent writes it. The reviewer owns the acceptance decision. An important claim with inadequate evidence stays open. If the implementation affects more operations than agreed, resolve that change in scope before proceeding.

For the refund example, that means the approval feature can proceed once its checks support the agreed behavior and the reviewer has examined the relevant payment path. The added retry needs its own justification. It can be removed or reviewed as an explicit addition, with evidence that repeated requests will not issue duplicate refunds. This builds on familiar development practices. Acceptance examples clarify intent before coding. CI repeats the checks, and small releases limit exposure. The proposal joins those practices around the review decision so that evidence is available when the implementation arrives. It also keeps review effort reusable: Once the approval boundary is checked correctly, later changes can run the same test.

Keep the blast radius contained

Keep the approval feature separate from a redesign of the shared payment client unless the two need to change together. Separate pull requests help reviewers assess each decision, but containment also depends on the implementation. A one-line change in a shared client can still affect every caller.

Use the same discipline with your releases. Where the system supports it, expose the change to a limited group (A/B) and watch the affected behavior before expanding the rollout. For refunds, that still requires adequate checks before any real payment. Start with one service and make this the normal way changes reach review. Require the evidence, record the expected blast radius, and keep unexplained behavior open until someone resolves it. Reduce routine line-by-line inspection as the checks earn trust. Keep direct inspection where a mistake has serious consequences. That is a concrete way to accept more agent-generated code while keeping the effects of each change understandable and contained.

AI makes it possible to generate code faster than teams can understand and support it. In this episode, Honeycomb cofounder and CTO Charity Majors joined Ben to discuss how that shift changes the work of building reliable software. Charity explained how observability helps teams connect changes with outcomes, why nondeterministic systems need “more rigor, not less,” and how traces can show the decisions agents make along the way. Over their wide-ranging conversation, she and Ben also covered automatic instrumentation with OpenTelemetry, why we should think of ourselves as “owners on the loop,” the limits of AI in debugging and SRE, and broader industry concerns like the pressure AI hype places on engineering teams and why maintenance, not speed, builds “enduring value.”

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Takeaways

00.59. “Observability is for unknown unknowns.” Monitoring is a way to set thresholds and alert on expected conditions. Observability supports open-ended investigation to answer questions you didn’t know to ask and “find things you didn’t expect.” When AI code generation became capable enough to change software work, observability best practices had to evolve, which is one reason why the second edition of Observability Engineering, which Charity coauthored with Liz Fong-Jones and George Miranda, more than doubled in size.

05.24. Cheap code shifts the work to understanding and support. AI can quickly produce implementation, tests, and documentation, but teams still need to understand what they built and agreed to support, and how their users are experiencing the product. As generating lines of code becomes “effectively free and instant,” validation, review, and operational support can constrain a team’s pace. Of course, Charity wryly noted, so can “every other part of the software development lifecycle.”

06.24. The people who take the credit for AI-assisted work should be able to explain it. AI-generated results (and the papers describing them) complicate authorship, expertise, and the academic training process, something that’s become a particular point of consternation for mathematicians of late. If someone is claiming a result they accomplished with the help of AI, they should be able to present it to peers, Charity and Ben agreed. When AI handles entry-level problems, professions may need new ways for junior practitioners to develop mastery, but Charity is skeptical we’ll run out of problems to solve:

In the same way that I’ve made peace with the idea that cars are faster than I can walk or run, I have made peace with the fact that computers can compute better than I can. . . .That doesn’t mean that there’s no value in the practice, or the discipline, or the learning, or the rigor. It just means that this is one of the things that humans are no longer the best at. But there are a lot of things that we are the best at, and being the best at something isn’t all of what makes it worth doing.

15.56 Nondeterministic systems require “more rigor, not less.” As Charity noted, “The future of software is both determinism and nondeterminism. It really is about using these superpowers that we’ve been given to build those feedback loops. And observability is really the feedback loop of feedback loops.” It’s what connects actions with outcomes and helps teams learn what happened after they make a change. Her analogy:

It’s like we’ve all been riding lawnmowers and just kind of putt-putt, putt-putt, putt-putt along down the road. And now, all of a sudden, our lawnmower has a Ferrari motor in it, and we’re like, “Vroom!” Now we can vroom into the ditch. We can run into a wall. And we have to build structure. We have to build rigor into our system if we want to run down the middle of the street.

18.53 People need to own the loop, and understand the decisions the agents are making. If you need to add observability to your agents, Charity recommends starting with autoinstrumentation with OpenTelemetry and traces that capture agents’ decisions and steps through a task. As for your role, it goes beyond staying “in” the loop. Charity prefers the term “owner on the loop” because it emphasizes the active understanding and control required to manage these systems. AI can amplify experienced debugging and SRE teams, especially when people direct work during rare, high-stakes incidents. Agents have little training data for those situations and can make an incident harder to diagnose. These tools are “not magic,” Charity pointed out. AI “is not a replacement for expertise, but it can be a really powerful adjutant. . .at your side.”

26.00 Different telemetry needs different strategies. Charity distinguished the abundant output from third-party infrastructure from the business-critical telemetry produced by a company’s own code.

There’s observability for infrastructure, which is mostly third-party code. . . .It’s code you didn’t write, but you have to operate it. You can’t change it, but it’s just outputting a lot of logs and metrics into your database logs and your ETLs, or CloudWatch, and all this third-party code is just emitting a lot of stuff. . . .That stuff you should just put someplace that is cheap commodity storage. . . .It shouldn’t be super expensive, just that it’s there when you need it.

Then there are your crown jewels: your code, the code that makes you the company. . . .You should instrument in such a way that preserves as many of the relationships between the data points as possible: arbitrarily wide structured-data blobs, structured logs, and/or traces with lots of spans. Because the most valuable part of your data is the relationships. . . .You don’t want to spread it out into hundreds or thousands of individual metrics or log lines. You want to keep it together so that you can correlate: when this happened, that happened; or when this happened, that was true; or when this happened, it was part of this long event. . . .This is the most important data you have. And for that, it should go someplace. . .that preserves those relationships and opens it up for explorability by you or your agents.

The following article originally appeared on Addy Osmani’s Elevate blog and is being republished here with the author’s permission.

For much of human history, we’ve evaluated code quality via code review: Someone reads what you wrote and makes sure it’s clean, thoughtful, fast, understandable, and tests well. For agents, that approach doesn’t scale well; there’s just too much code for anyone to read. As a result, more and more of our quality checks have to happen in the harness, environment, and operating system around the agent. I still read and review code, but am very intentional about where I am comfortable with constraints as the check.

Software quality now depends on the constraints you set around your agents.

Speaking of quality, agents are writing your code. Sonar gives you the quality gates to make it shippable. I had a coding agent build a slick app, then asked an agent to review it—twice. The reviews disagreed, and a re-run gave a third answer. You can’t gate a merge on a coin flip. Sonar runs the same complete check on every commit: deep cross-file analysis, a map of where the risk lives, and a quality gate that holds every human and agent to one bar. Try Sonar. Sponsored by Sonar.

Constraints define what the system is allowed to do by throwing tests and deterministic constraints at an agent’s proposals. It’s by setting and maintaining these constraints that we build loops that reliably deliver high-quality production software, even when agents are creating hundreds of thousands or millions of changes every single day.

We call these constraints quality gates, and they take many forms. They include conventional unit tests, property tests, and acceptance tests. They include mutation testing, where we generate variations of code, run it against the same tests, and make sure that people aren’t sneaking bugs in that we’re missing. They’re metrics around code quality, such as cyclomatic complexity and line length, that help keep things readable. Constraints also play an important role in what proposals the system will accept and apply as code changes. By the time a change proposal moves from the interpreter running the agent to the agent controller and out to production, we’ve done enough checks on it to be confident that it’s safe to ship and the impact of its change is well within the scope of the agent.

Guillermo’s list is a good test for whether you can afford to skip reading. Notice that every “yes” is really a statement about how low the stakes are: no users, throwaway code, prototype. Once the stakes go up, something has to read the code. If it isn’t you on every diff then it has to be the constraints.

An agent can propose anything. Your constraints decide whether a proposal is safe enough, correct, scoped, and useful, for you and your team to ship.

This model offers a lot but also leaves out many pieces, and those omissions are worth thinking about today. One issue is autonomy; agents might apply their intentions well but may fail when there’s missing information or when what they try to do is ambiguous. That applies both to the task itself and to how it’s parameterized by the harness, environment, and other components. Many of the reasons that humans fail to ship great code are shared with what agents might do: brittle environments that don’t hold up under script-driven stress, nondeterministic builds, missing permissions, and weak tests. This motivates a better environment that gives agents trustworthy feedback, allows for low-damage failure modes, and makes it easier to progressively build up success.

The environment we’re after is one where an agent can do real work, get feedback it can trust, and fail without doing much damage.

The other important issue is trust. We can’t credulously hand off intent to something even as smart and robust as a modern agent without checking for correctness. We start with trust, but it has to be hard-earned.

Some constraints shape work before it begins. Others give feedback while the agent is working. Others decide whether its output can cross the production boundary at all.

There are a bunch of ways to model how we put a verification structure around a system.

In my experience it helps to have a broader, but intentionally chosen, set of checks for your constraints instead of relying solely on unit tests. The idea is that each check has a distinct responsibility, and that can range from type safety and performance to late-stage security scanning. Folks can define their own constraints too, including architecture rules that linting tools like ESLint can enforce. Many of these tools have built-in hooks that can be used to pull in agents, or humans, when things break.

For now, much of the difference between useful agent output and slop still comes down to the skill of the team operating the loop.

AI gives us high-volume code generation and velocity, but this can also mean it gets harder for humans to review every single change. You have to instead be intentional with where their attention is going. If you put a human check into a system that otherwise moves at machine speed, don’t be surprised if that impacts productivity. Human attention is scarce and valuable, so we should proactively direct it to those most nuanced problems that require our judgment. Downstream humans should only be pulled in when the automated guardrails for constraints break.

Human “code review” in the future is going to look very different.

Correctness is one important dimension, but you may care about others as well, like maintainability, performance, security, efficiency, and comprehensibility. Just as correctness decomposes into many signal types, so too does the rest of quality. And while it matters how many constraints we have in place, it matters more whether they’re challenging enough to meet our bar for quality and production readiness.

Software quality isn’t a single metric. Think of it as a collection of signals of varying importance to you and your team.

Back-pressure can be implemented through many tools: compilers rejecting invalid code, tests failing, security policies blocking bad practices, CI declining to deploy. Ideally it exists throughout the loop, not as a single review at the very end of all the work.

Constraints and back-pressure let agents catch bad work before it’s a problem.

What happens if we can’t apply the constraint because the volume of changes is higher than our tools can consume? We end up building a queue and relying on a verification system that moves at human speed. To scale, we want to push as much as we can into the verification loop throughout and not wait until the end. If we can scale within our automated checks, we can increase the speed and throughput of our entire delivery system. If we run out of room in the verification loop, we need to do one of several things. First, we can scale our verification system and create more capacity to constrain and push back on changes that come in. Second, we can reduce the rate at which agents generate new changes so that verification can catch up to the volume of work. Third, we can lower our quality bar so that verification doesn’t push back as hard as it otherwise might. From a scaling perspective, we need to be ready to do all of these things. At the same time, we should not stop short of realizing that we could actually get more done by unconstraining in some directions. Maybe we can increase the speed of agent-generated changes by providing swarms of agent developers or automated software factories to create changes without waiting for us to review each of them.

And in some places we might want to give them more freedom as long as we keep tighter constraints in others. By providing tighter constraints where we care the most, we can maximize our throughput without sacrificing quality. Throughout these decisions there are many options. Most obviously, we have to trade between different dimensions of quality. As we’ve emphasized, security is very important, but we’ve also had to trade between delivering security and delivering a product on time. There is a spectrum from innovation-focused at one end to quality-focused at the other. Somewhere along the way, we have to make choices about where we want to be on that spectrum.

We want to send clear feedback from the environment and the system back to our agents or teams, so that people can focus on the more subjective concerns of taste, intent, and architecture. If we can help humans stay inside the safe range of constraints, we can avoid the need for them to work hard trying to figure out where things went wrong.

Software quality includes more than just correctness. Software quality also means maintainability, good performance, security, efficiency, and being easy to understand. All the constraints that help us meet these standards and keep our production flowing create back-pressure in our delivery pipeline.

We need to make deliberate decisions about where to apply strong constraints and where to remove or relax them. Apply strong constraints where they’re serving both of these goals. Don’t support them if they’re not serving one or both. Be ready to raise or lower standards as the case may require. And remember these constraints at different points in the software system are what make software quality enforceable.

We should apply strong constraints where they’ll serve this dual purpose best and consider removing or relaxing constraints that aren’t serving either purpose well. We should also be ready to shift quality bars up or down as needed. In effect, these constraints at various points in our software system are what give quality its teeth. In many cases, we can create more back-pressure and more constraints by deploying new tools or strengthening tools that are already in place. All of these things can be used to push back on most change requests. We want to build them throughout the pipeline. We don’t want to wait until the end of the pipeline when our CI system will simply tell us that we’re not allowed to deploy without fixing problems. We want to use these signals as early as we possibly can, through every possible pathway. The ultimate constraint in this system is the one we place on ourselves to stand behind the decisions and actions we’ve taken to build the system and to operate it. But like all other constraints, we need to make thoughtful trade-offs about how much we want our own judgment to restrain, to back-pressure, and to act as a final check.

Quality is in the constraints that we place around our agents. So as you’re thinking about quality for your own apps, take this problem statement and come up with your own constraint-driven plan.

I use AI for writing all the time. But not because it’s good at writing.

In fact, just the opposite is true. AI is so reliably bad at writing that I’ve actually taken advantage of that, and made its bad writing an integral part of my writing process.

I’ve written a lot of books and articles over the years, and while I used to start with an outline, I don’t anymore. These days I do a lot of brain dumping and brainstorming and talking out loud to an AI, working out what I think about something. I accumulate a lot of notes and ideas, and run a lot of experiments, and at some point, if I’m lucky, I decide there might be an article somewhere in all of that.

Then I ask the AI to take everything I’ve given it and write a draft.

And that AI-generated draft is universally terrible. Like, some of the worst writing you can imagine.

Which is exactly what I want.

Instead of an outline, I get an entire “article” or “chapter” (or, at least, a weird facsimile of one), with paragraphs that express complete thoughts that I gave the AI. Granted, those paragraphs are full of obvious AI tics, like how it uses em dashes to squish two unrelated sentences together, or weird staccato sentences one after another, or these insufferable parades of sentence fragments. Maybe the worst tic is that half of the paragraphs AI generates have an extra sentence that doesn’t actually say anything, but just delivers a forced conclusion with fake “punchiness.” I guess maybe AIs were trained on a lot of bad writing? Well, whatever, it works for me.

The reason I love that type of obviously bad AI writing is because when I’m looking at one of its paragraphs, it’s immediately obvious that an AI wrote it. I’ve been thinking of it as a kind of an improved “lorem ipsum” that lets me see how my article will look, while still serving as a functional placeholder. That lets me use it the way I would use an outline: I can move parts of the article around, make sure it flows correctly, and then replace ALL of the bad AI writing with (hopefully) decent human writing, just like I used to do with bullets in my old outlines.

One weird side-effect of writing this way is that it’s made me very sensitive to the different kinds of AI tics and tells. And I think the weirdest and most frustrating one is “Yoda-speak,” where the AI literally talks like Yoda. It’s weird, kind of hilarious, annoying, and pervasive, but most of all, it’s fascinating. Why does the AI keep talking like Yoda? What causes it? Can we do anything about it? (Do we even want to? It certainly makes it easy to identify AI-generated text, and also, it’s often unintentionally hilarious.)

I’ve got a theory about why this happens, and I ran an experiment that generated some evidence to support it. But first… what, exactly, is Yoda-speak?

Backwards, this bad writing is

I’ve been working on my latest book. It’s called Be the Model, and it teaches you how LLMs work by having you build interactive pages that let you literally do the job of a model to get an intuitive understanding of how they work. So I’ve been writing a lot, which means I’ve been confronted with a lot of Yoda-speak.

Here’s a great example from the chapter I’m currently working on (the “pond sentences” are a small training text), from a description of a page that you’ll build to learn how training works:

The eight pond sentences fill the left region, every word clickable.

That’s weird, right? Nobody talks like that.

To be perfectly fair, I’ll need you to suspend your disbelief a little. Yoda would probably say something more like, “The left region, filled by the eight pond sentences is. Clickable, every word must be.” But the AI-generated version is still very weird, and in a pattern that is recognizably Yoda-like.

That’s a pattern I’ve seen a lot. At one point I was looking through some AI-generated text, trying to figure out a particularly unintelligible sentence. When I asked Claude Opus for help, it came up with the term “Yoda-speak,” and I’ve since found that using that term is really useful when trying to get an AI to help decipher sentences like that.

The interesting thing is that AIs are perfectly capable of detecting and correcting Yoda-speak. You can try this yourself: start a new ChatGPT session, paste in that “pond” sentence, tell it: “An AI generated that sentence. That’s Yoda-speak.” I just tried that in an incognito window, and here’s the answer it just gave me:

Yep —that’s unmistakably Yoda-speak. A natural version would be:

“The eight pond sentences fill the left region, with every word being clickable.”

Or, more naturally for UI instructions:

“The left region contains the eight pond sentences, and every word is clickable.”

The original sounds like the AI took normal English, rearranged the syntax, and then forgot to put it back.

This particularly ugly sentence, from one of those AI-generated outlines of a later chapter that I haven’t started writing yet about the attention mechanism (the part of the LLM that understands context and relationships between words), is the example that made me want to write this article:

What’s left is to let the two heads’ work combine, so run the combined eight numbers through one more small table, eight in and eight out, square, called the output projection.

Ugh.

The problem with that sentence, and what made it hard to diagnose what was going on, is that when you first look at it you can basically understand what it’s saying (sort of?), and it’s hard to put your finger on exactly what’s wrong with it. It’s obviously terrible writing, but that’s not unique. AI is a universally awful writer.

What’s uniquely wrong with that sentence is its weird, comma-laden structure. Different parts of the sentence are split up into small comma-separated phrases, and AIs seem especially prone to using them to rearrange sentences, often at the expense of readability. That particular sentence is about running a combined set of eight numbers through one additional small 8×8 table. But the part of the sentence that should have described the table was in the second half of the sentence, and then you learn at the very end that the whole thing is called an output projection (but only after you’ve already been told to run numbers through it). Somehow the components of the sentence were arranged in a way that made it almost unintelligible.

One word at a time, it writes

An LLM is a next-word predictor. It works by taking your prompt, all of the context you’ve fed it, and everything it’s generated so far, generating the next word in its response, and then repeating the whole process for the next word, then the next one, and so on. It’s counterintuitive to think that AIs could possibly work that way, because these things can produce long, complex, and generally coherent and helpful responses. It seems very strange that they do all of that work one word at a time, without following a master plan for a response.

But that’s literally how the AI works. Once an LLM generates a word, it’s part of the response. It generally can’t go back and rearrange the first half of a sentence, even though it hasn’t finished generating it.

And that brings me back to that especially weird “eight in and eight out, square” barely intelligible sentence. Here’s what I think happened. The model generated the sentence up through “run the combined eight numbers through one more small table,” so those words were already baked into the response. Then it figured out that it had three more things it wanted to say about that table. But by that time it was too late to go back and put those details where they belonged, and apparently there was a shortage of periods that week, or possibly a surplus of commas (there’s no way to know for sure), so it just smashed each of those thoughts onto the end of an already awkward sentence.

I did some research to try to figure out if anyone has looked into this phenomenon before. The closest thing I found to an answer is a 2024 paper by Wilson Wu, John X. Morris, and Lionel Levine called “Do language models plan ahead for future tokens?” They were attempting to determine whether a model does real advance work for words it hasn’t written yet, or whether the information that pays off later is a byproduct of whatever was useful at the time.

The experiment that Wu et al. did was actually pretty clever. In normal training, when the model is being corrected for mispredicting a word several steps ahead, that correction reaches back and shapes what it computed at earlier positions. That way, the model learns to prepare information at an early word for the benefit of words further along, which is the “planning ahead” the paper is testing for. So they trained one model normally, and a second one with a modified process that cut that out: each position could only be trained to help predict the very next word. They kept next-word prediction but removed the training signal from words two or more steps ahead.

Then the researchers compared how well the two models predicted text: if the normally trained model did better, that would be evidence that preparing information in advance was actually helping. With ordinary language, the difference was fairly small for GPT-2, although it became larger as they tested bigger models.

So there’s evidence that a model can sometimes prepare information that will be useful later, which means my story about it only figuring things out as it goes is probably too simple. But anticipating is not revising. Whatever the model sees coming, it still can’t reach back and reorder the words it already committed to, and the words it already committed to are where the damage shows up.

Better, the second answer often is

I have a theory that the Yoda-speak phenomenon is closely related to something I see all the time (and I expect you do, too). Say you’re asking an AI for help with an email, and it gives you a draft of a sentence. Then, immediately after it gives you that sentence, it says something like this: “A better way to phrase that would be:” and gives you a second version. Sometimes there’s even a third option. And it almost always tells you it prefers the last one that it generated.

Coincidentally, I ran into a good example of that “here’s a better version” phenomenon when I asked ChatGPT to make the pond sentence more Yoda-like. I literally gave it the prompt make it more yoda-like. Here’s what it said:

More Yoda-like

The left region, the eight pond sentences fill. Clickable, every word is.

Or even more Yoda:

Extra Yoda-like

The left region, filled by the eight pond sentences is. Clickable, every word must be.

At this point, you can probably guess what I think happened. It generated the first version, and now that version was part of the context it had to work with. Then it generated a second version that was even more Yoda-like. But the first version was already there, so it couldn’t replace it. All it could do was give me another one. I see AIs do this kind of self-correction all the time.

If that’s what’s happening, then self-correction and Yoda-speak may be two versions of the same basic limitation. The model can’t go back and change what it already wrote. Sometimes that means the better answer comes afterward. Other times it means a thought that belonged earlier gets smashed onto the end of a sentence.

My own experiment, I ran

After looking at these two related phenomena (the AI generating Yoda-speak and constantly generating a second or even a third option), I got thinking. I started to wonder if this wasn’t just an AI fluke but actually something that I could reproduce and even potentially control. It would be really interesting if the way we give facts to the AI and structure the input could actually have an outsized effect on how it generates its output, especially if seemingly innocuous changes could cause it to create really weird results.

This was a real experiment, because this approach isn’t obviously something that should work. In theory, if you give a model a set of facts, it should be able to produce coherent output no matter how you order those facts. But I’ve been looking at how attention works in LLMs lately because I need to explain it for Be the Model in straightforward terms that anyone can understand. It seemed to me that changing the order of the facts could actually change what the model was paying attention to.

It turns out that there are two different mechanisms at work inside the LLM, and my theory was that they each played a part in both phenomena, the Yoda-speak and the unprompted second answers:

• Attention is about what the model is looking at in the input. Every time the AI generates a new word, it looks back over the whole prompt and everything it has written so far, and weights the parts differently depending on their content and their shape. That weighting is a big part of what makes it good at predicting the next word. I had a theory that three facts written as complete sentences would get interpreted as facts to reorganize, while the same exact facts written as bare fragments might look like pieces that could be pasted right after a noun, and attention would weight them that way and send the model down a bad path.
  
• Autoregression is about what the model has already written. The AI is a “predict the next word” machine: it generates one word at a time, and each of those words becomes part of what it’s looking at when it picks the next one. Once it’s written one more small table, it can’t go back and insert square before table, so if any facts about the table are still unexpressed at that moment, the only place left for them is later in the sentence. If attention had already sent the model down a bad path, autoregression would keep it from repairing the opening.

My guess was that the “I prefer the second answer” text that the AI keeps generating is essentially a mechanism for recovering from weird phrasing, and that the Yoda-speak or other weird phrasing was the underlying phenomenon. I felt like a good experiment would concentrate on reproducing just that weird phrasing.

Coincidentally, OpenAI had just released their new model, Astra, and I was looking for an excuse to kick its tires, so I decided to use it to run an experiment to see if I could reproduce any of the phenomena I’d been seeing. I couldn’t actually verify specifically that autoregression or attention were causing them, because those are internal to the LLM. But I felt like I could design an experiment that could provide some evidence, by structuring prompts in a way that would take advantage of how those mechanisms work. If I could get an AI to produce weird phrasing that showed some of the same problems as Yoda-speak, it would be evidence that these mechanisms were at least a potential root cause.

I used Astra as an orchestrator, and had it spin up a bunch of Luna sub-agents and give them almost identical prompts. Each consecutive sub-agent would get a small tweak in its prompt. I asked Astra to look at the output and see if there were any weird grammatical constructions that looked similar to Yoda-speak, then iterate on that prompt to make another small tweak, and keep going to produce increasingly weird output.

The first test was deliberately simple. Astra gave a fresh Luna writer this prompt:

Write one concise, natural paragraph that communicates every supplied fact. Use only the supplied facts. Do not pad the paragraph or repeat information to make it longer. Do not exceed 100 words. Do not mention these instructions, show a plan, use bullets, or explain your choices. Return only the paragraph.

Facts: The bicycle is blue. The bicycle has a narrow frame. The bicycle has a silver bell. The bicycle has a worn leather seat. The bicycle leans beside the garage. Rina rides the bicycle to work on weekdays. Rina locks the bicycle to the rack outside her office.

Luna turned that into a totally normal paragraph:

The blue bicycle has a narrow frame, a silver bell, and a worn leather seat. It leans beside the garage, and Rina rides it to work on weekdays, locking it to the rack outside her office.

Then Astra gave another fresh Luna writer exactly the same facts, with exactly the same wording, but changed their order. It moved the fact that the bicycle leans beside the garage and the fact that Rina rides it to work ahead of the color, frame, bell, and seat.

Luna mostly defeated the experiment. It had all of the facts before it started writing, and it was perfectly capable of moving them back where they belonged.

So Astra dropped the bicycle and wrote a fresh set of facts about something Luna had never seen, a completely made-up device with a steel slot called a “feed throat” that you push clay tiles through. It sent them with the same prompt as the bicycle:

Facts: The slot is made of steel. A clay tile slides through the slot. The slot is a hand’s width across. The slot is tapered. The slot is called the feed throat.

With no rule attached, Luna handled them the way it handled the bicycle:

The feed throat is a tapered steel slot, a hand’s width across, through which a clay tile slides.

Then Astra added two rules: the answer had to be one sentence, and it had to start with a fixed phrase. That phrase ends with the word slot, so everything else about the slot has to come after it.

The sentence must begin with exactly this text and continue from it:

Slide the clay tile through one more steel slot

Facts: The slot is a hand’s width across. The slot is tapered. The slot is called the feed throat.

Luna still tidied it up. Every time, it reached for a relative clause:

Slide the clay tile through one more steel slot, which is a hand’s width across, tapered, and called the feed throat.

Then Astra made one more small change. It kept the same opening and the same three facts, but wrote the facts as short pieces instead of complete sentences:

Facts: a hand’s width across tapered called the feed throat

And Luna wrote this:

Slide the clay tile through one more steel slot, a hand’s width across, tapered, called the feed throat.

The only thing that changed between those two runs was how the facts were written down. Astra tried the same setup on a completely different subject:

Rest the cracked bowl inside one more padded cradle, two palms wide, crescent-shaped, called the handling saddle.

That’s the sentence I’d been chasing. Once the model had committed to the noun and the remaining facts arrived as loose pieces instead of sentences, it tacked them on in the order they came.

The shape of its input, the model copies. Reorder the words it has already written, it cannot.

That still doesn’t tell me whether this is what happened when the model wrote one more small table, eight in and eight out, square, called the output projection. In these experiments, we deliberately pushed Luna into that kind of situation. In the original sentence, the model got there on its own.

I think the experiment shows that the model copies the shape of its input, and that fits my theory about the underlying attention and autoregression causes. When the three facts went in as complete sentences, Luna wrote a relative clause. The same facts sent in as bare fragments came back as bare fragments after the noun.

There are some obvious limitations to this experiment. I’m watching the model from the outside, so I don’t have any visibility into how the model weighted those tokens, or what its attention mechanism was doing. I also can’t say whether the model already had those three facts in mind when it wrote slot, but even if it did, it couldn’t put them anywhere but after the text it already generated. But the outputs match what those two mechanisms would do. Attention responds to how the prompt is written, as well as to what it says. Once the model has written the noun, it can’t go back and put anything in front of it.

I suspect that’s also why the AI keeps offering us that second answer. It can’t fix the Yoda-speak that it already generated. It can only write another sentence and hope that’s good enough.