This event summary was created by Claude with light editing by Tim O’Reilly, based on a transcript of the event.

On October 15, 2026, Tim O’Reilly hosted Andrew Odewahn, CTO of O’Reilly Media, for an hour-long episode of Live with Tim O’Reilly, the interview series streamed live on the O’Reilly learning platform. Andrew has been at O’Reilly for many years, starting as an author, then building tools for the company’s platform, then eventually as CTO and head of engineering. Over the past year, his team has been building what the company calls Expert Intelligence, a suite of tools that connects an AI agent to an organization’s own systems, gives it a skill file describing how to reason about what it finds there, and grounds its analysis in the practitioner frameworks of the O’Reilly library. Andrew described the work in a recent Radar piece, “Building Organizational Intelligence.”

Tim and Andrew’s interview was organized around a presentation in two parts, the first on the research agent and the second on the system Andrew’s team built for reviewing what it produces, with discussion in between and audience questions throughout.

Tim framed the conversation by asking a question he said Andrew’s work helps to answer: Most of the AI story is about efficiency, doing the same work faster. What does it look like for an organization to get smarter?

Building the research agent

Andrew’s research agent emerged from a personal need rather than a strategy. He wanted to know the status of a project, for example, how long a migration from RabbitMQ to Pub/Sub would take. This is the kind of question every engineer dreads but every engineering leader has to answer. He also had a broader concern. A single person being good at AI isn’t the same as an organization being good at it.

He built his agent in four steps. He mapped the organization’s information systems as a hierarchy, connected them via MCP, wrote a skill telling the agent how to use them, and finally added the Expert Intelligence layer.

The map of the sources of truth in the engineering organization’s workflows was what made the rest work. Andrew put them in order and explained to the model how they relate: Strategic goals and feature priorities live in Productboard. Those become epics, sprints, and stories in Jira. Stories become code in GitHub. A service catalog describes the roughly 300 services that make up the O’Reilly platform and who owns them, and observability tools show how those services perform against their SLOs. The skill also had to encode local quirks. Some teams mark an epic as finished with an end date, while others consider it done when every ticket is closed. Andrew also told the agent it was a research agent, not a coding agent, and that its job was to write reports.

Connecting everything through MCP was, in his words, “the most completely tedious step of the whole process.” Productboard had no MCP server at the time, so he wrote a wrapper. Later, when Tim asked how another company could repeat the work with a different set of tools, Andrew suggested starting from the vendor list. Export it from your procurement system, ask the model what products those vendors make and which would hold information relevant to the questions you care about, and have it put them in order. O’Reilly product managers are now turning this process into a skill for the forthcoming Expert Intelligence Skills Hub.

One audience member asked whether MCP was really the right tool for internal knowledge, compared with semantic search. As Andrew pointed out, sometimes anything is better than nothing. Ultimately, what you want is to give the model the ability to search across your systems.

From a hiring case to a toil audit

The center of the presentation was a before-and-after comparison of the particular use case that Andrew had previously written up in his “Building Organizational Intelligence” article. An engineering manager asked Andrew to hire another engineer, and Andrew had his reporting and analysis agent make the hiring case. The first report covered the five-person team that manages content and metadata, the 10 tier-A services it owns along with its B and C services, the status of the Pub/Sub migration (64% complete), and the full backlog. It concluded that the team had a capacity problem and should probably hire. Andrew called it useful and very detailed. . .“maybe overly detailed.”

He then connected the agent to Expert Intelligence, asked it to act like a consultant, and ran the analysis again. The new report came to a very different conclusion. It opened by saying that the team’s operational overhead problem was “structural, not a staffing deficiency.” When it consulted Expert Intelligence, the agent brought in the concept of toil from Google’s Site Reliability Workbook. Toil is the manual, repetitive, automatable operational work that scales linearly with a service. The reporting and analysis agent used this notion to separate the team’s backlog into toil and the work people actually wanted to be doing. It recommended a structured toil audit before the next planning cycle, an explicit toil-reduction target, and treating runbook ownership as a first-class engineering artifact. Andrew said the team followed the recommendations and closed a substantial part of a backlog that had been sitting on everyone as a cognitive burden.

The report went on to cite Jim Highsmith, a signer of the Agile Manifesto and author of Wild West to Agile, to argue that the team’s technical debt was compounding. Andrew explained how that differed from the usual management advice:

A lot of the time when you read these high-concept frameworks, they’re like eating your broccoli. I should really be doing that. But what made this really interesting is that it pulled out a specific project we had, the Django upgrade, and put it within the context of this broader framework. It wasn’t just, “here’s an abstract principle you ought to be applying.” It’s “here’s your backlog, and here’s a specific example.”

A second analysis looked at services that crossed team boundaries. Drawing on microservices experts and Kent Beck’s work on coupling (see Tidy First?), it diagnosed the video metadata service as a complicated-subsystem problem rather than a boundary problem. In this version, Andrew had the agent pull out quotations, attribute them, and link to the source text. The resulting analysis was remarkably effective. The company has since retired a couple of systems. “I don’t need LLMs adding code,” Andrew argued. “I want them reducing it.”

Andrew said the part that surprised him most was the reaction from colleagues. His earlier reports were data dumps. Readers couldn’t tell where any given claim came from, and as a result they were much less actionable. When the analysis cited experts that engineers knew from their own work, the documents became, as he put it, “much more approachable, much more human,” and people engaged with them far more deeply.

Tim’s response: Invoking the experts

Tim broke in to put the demonstration in context. He contrasted it with the view that models are now smart enough that you can simply tell them to do something and let them rip, an approach we hear a lot about from vibe coders and those arguing for “dark factories.” Tim argued instead for a return to expertise.

He cited a post published that day by Felix Rieseberg of Anthropic, “How to Be More Ambitious: A Tutorial for Humans with Models,” which describes four kinds of ambition when working with models: speed, altitude, depth, and range. Most discussion of AI’s economic value is about speed, getting the same work done in less time. In Tim’s reading, Andrew had gained altitude (thinking about the outcome rather than the tasks) and depth (approaching the problem the way a field-leading expert would) simply by invoking the experts.

Tim then offered a metaphor he has been developing for a while, of AI programming as a kind of magic spellcasting. In the Marvel comic books and movies, Doctor Strange opens his spells by invoking some source, with words like “by the power of the dread Dormammu” or “by the hoary hosts of Hoggoth.” Andrew, he said, was doing the same thing, saying in effect, “By the power of Kent Beck, I want you to think differently about this problem.” Even where Andrew didn’t name an expert, as with the SRE workbook, simply pointing the model at the expertise embodied in the O’Reilly publishing canon raised the bar for what the agent produced.

Tim described this as using human expertise as a focusing mechanism, and connected it to his argument that LLMs are better understood as a medium we create with than as independent intelligences. He also pushed back on the idea, often drawn from “The Bitter Lesson,” that models will simply get better and human expertise will stop mattering. Andrew, he said, had found a particular way that it matters.

Andrew took up the metaphor. His own first prompt, he said, had been “Be snobby. Go find these great resources and look at them.” He wants to hand the model canonical works in a field, like Designing Data-Intensive Applications, and tell it “these are the ways I want you to think about it,” rather than accept whatever the model comes up with first. He compared it to business school, where much of what you learn is a set of frameworks for organizing facts.

What models do without direction

Tim argued that models don’t do their best work unprompted. Watch their chain of thought when they search for information, he said, and you may see them settle on some random file from the internet rather than canonical sources. Even with a frontier model like Fable, his own experience has been one of false starts and errors, a model brute-forcing its way to a solution rather than necessarily knowing how to approach it.

Andrew had a story from a few days earlier. He asked his agent to review incentive plans for the engineering and product teams.

The data that came back was, “according to the company Robusto (which I’m sure is a great company), this is the framework, and you should really be doing what they’re doing.” I’m like, “What is Robusto, and why am I seeing this?” And I realized, oh, I’m not connected to the VPN. So I hooked it up, and suddenly it was, “According to Harvard Business Review, these are the different ways you should be designing these incentive programs.” This was a very powerful model. But it keyed off the first thing it found and said, “This is the way I’m going to think about the problem.”

Tim raised an economic dimension. He pointed to instructions in Anthropic’s Claude system prompt that limit how much the model may quote from any source, and read them as a sign that AI companies want to avoid paying for content. The result, he argued, is that models can’t draw on the best material available to them specifically enough and too often fall back on generalities. He called this an early form of enshittification in AI.

O’Reilly takes the opposite position. Andrew explained that the company applies the royalty model it already uses for O’Reilly Answers. Material returned through the Expert MCP server can be cited, and authors are paid royalties on it, so that expert content has a market and earns a return.

In response to an audience question, Tim and Andrew noted that Expert Intelligence is in prerelease with a group of founding partners while O’Reilly works out its economics. Interested organizations can request access through the product page.

Putting the discussion where the AI can see it

The second part of Andrew’s presentation dealt with what happens after the agent writes a report. Decisions need consensus, and with AI-generated analysis people need to check that the model hasn’t keyed off something unimportant, out-of-date, or misinterpreted. “The AI can identify all the trade-offs,” he said, “but people still need to make the decisions that matter.”

His team first circulated reports as Google Docs, but they ran into two problems. The most interesting part of a document under review is usually the comment thread, and because at this time, O’Reilly company policy doesn’t allow MCP connections to Google Docs, the agent couldn’t see it. And when Andrew asked the model to fold the discussion back into the document, pasting the result wiped out the comments.

Doug Hogan, an O’Reilly engineering director, built an alternative on top of Git and GitHub that Andrew called SuperAnswers. (Andrew joked that this was why he isn’t allowed to name things.) Documents live in a Git repository, discussion lives in GitHub Discussions, and every change is versioned. Andrew can ask his agent what the consensus is on a project, have it incorporate comments into a new draft without losing history, and later reconstruct when a decision was made and what people knew at the time. The same record lets him generate the derivative documents a decision requires, such as scope statements for engineering, product, and design teams, or a meeting agenda. A graph view clusters the topics people are discussing and flags decisions that are still open.

Tim summarized the whole system as a research assistant connected to the organization’s sources of truth, a layer of outside expertise to put its findings in context, and a shared, versioned discussion that becomes an evolving store of organizational knowledge, which makes the AI a more useful coworker. Andrew agreed and said he would like to open-source SuperAnswers, which he described as a thin layer on top of Git.

Andrew also pointed out that the fragmentation he was addressing isn’t limited to engineering. Sales teams have their information spread across systems like Salesforce, 6sense, and Gainsight, and every function has the same need to pull those threads together. Tim added that tool access is uneven across the industry and always has been. Claude produces better PowerPoint than Google Slides, Gemini the reverse, and organizations have to capture that kind of knowledge about their own toolsets.

Questions from the audience

An audience member named Thomas worried that workers now talk to AI rather than to each other, and that work disappears into “that AI hole.” Andrew said his preference for keeping everything in Git is partly an answer to that, since it gives both humans and AI a central place to find and reconstruct past work. But his larger interest is in humans talking to humans in a way that AI supports. He described a category of questions he never used to ask:

I knew, if I ask this question, somebody’s got to go off for two weeks and compile all this stuff from all these different systems. Now I can gather it in a few minutes. Maybe some of this is throwaway, but it’s not bad if it’s cheap.

In response to that “throwaway” comment, one audience member wrote in the chat: “This is not throwaway work, Andrew. It is a product lead that changes the value proposition in a positive direction. Shit hot, bro.” Tim couldn’t help but read that one aloud!

Asked by Tim what he got wrong, Andrew conceded that he didn’t think early enough about how people would review and respond to AI-generated documents. People don’t want an AI-generated data dump to wade through, and they extend AI no benefit of the doubt. If an AI-written report gets one small fact wrong about Kubernetes operators, a reader is likely to dismiss the whole document. AI tools have to be understood as marshaling and presenting evidence, not making decisions. “These need to support your analytical capacity, not replace it.”

Tim connected that answer to concerns about deskilling. He described a fine-tuning project he attempted with Fable to train a small model to write more like him. He wanted to understand each stage rather than letting the model run, but it never explained them, and the training run failed. In part this was because the model asked him to provide samples of his writing, but it didn’t give him enough grounding on what a really good training dataset would look like, so the samples weren’t targeted enough. They were too long, and when Fable chose the short fragments it actually needed, it didn’t do an ideal job.

When Tim tried the same project with O’Reilly Tutor, a tool still in development, it sent him to read an O’Reilly resource on what a good training set looks like, and he realized where he went wrong. Tim compared the kind of deskilling that can happen when people just turn the work over to AI to people who can no longer read a map, or who call a plumber for a simple fix they could do themselves in 30 seconds if they only knew a bit more about how things work. He argued that everyone has an obligation to keep learning alongside the tools they use.

An audience member named David, a laid-off engineer looking to reskill, made a similar point. We used to know things, then we searched, and now we skip the search and just ask for the answer. He asked how to build real depth. Andrew suggested picking a project and working through every aspect of it, asking AI to act as a tutor (he’s relearning calculus that way) and keeping a book open alongside, or taking advantage of instructor-led live events on fast-moving topics.

The conversation ended with the difference between an AI system that simply makes an organization faster and one that makes it better at deciding what to do. In Andrew’s account, getting better at deciding what work needs doing depends on three things the models don’t supply on their own: a map of where the organization’s knowledge lives, a body of expertise that people trust, and a place where the humans can argue with what the AI produced.

You can watch full episodes of Live with Tim O’Reilly, including this one, on the O’Reilly learning platform, YouTube, or wherever you get your podcasts.

AI agents dominated the news this week, from security exploits that unfolded in hours to new tools that let agents shop, manage transactions, and conduct scientific research. On the latest episode of This Week in AI, host Vicki Reyzelman of Akamai examined how these developments are testing existing security practices and creating new opportunities for businesses and researchers.

AI attacks are outpacing traditional security responses

AI is changing how attackers find and exploit vulnerabilities. A supply chain attack involving ARTEX, an open source agentic AI penetration-testing framework, helped attackers reach more than 2,250 Microsoft 365 tenants through a ConnectWise compromise. The incident raised concerns about OAuth authentication and the limitations of conventional identity protections when autonomous agents interact with enterprise systems.

A more immediate threat came from PHP2Shell, two vulnerabilities rated 9.2 on the CVSS severity scale that could allow attackers to execute code on vulnerable web servers without authentication. CISA urged immediate patching following the October 21 disclosure. Researchers also documented attackers using AI agents to obtain password hashes and MFA secrets from a European government system within two hours. The speed of exploitation leaves organizations little time to investigate threats before responding, especially smaller businesses with limited security resources.

Security teams can no longer rely on manual review alone. They need automated detection and response, tighter access controls, and network segmentation to limit what compromised agents can reach. But monitoring autonomous agents is challenging. Researchers from METR and Redwood Research identified weaknesses in an earlier version of OpenAI’s internal monitoring system, including one that could allow hundreds of agent actions to be concealed. OpenAI says it has patched vulnerabilities affecting production safeguards, although the researchers haven’t independently evaluated those changes.

The debate over AI safety is also extending beyond technical safeguards. Anthropic proposed federal AI safety legislation, while Google DeepMind employees resigned over the company’s military AI work. Both developments illustrate the growing pressure on AI companies to establish clearer rules for how their technology is developed and used.

The transition to agents brings new challenges

Retailers are taking different approaches to autonomous shopping. Walmart introduced an Agent Gateway that lets approved agents shop within customer-defined spending limits and approval requirements. Amazon has restricted some third-party shopping agents, while banks including Morgan Stanley and Bank of America are limiting agent access to customer accounts. The disagreement goes beyond security. Retailers that depend on advertising and product recommendations have an incentive to keep shoppers on their websites. As more purchases begin with agents, companies will have to reconsider how they maintain customer relationships.

Anthropic is testing a different approach to agent oversight. Claude’s auto mode is becoming the default for Pro, Max, and Team users, allowing automated checks to approve routine actions and block risky ones. Anthropic also contributed its Agent Skills format to the Agentic AI Foundation under the Linux Foundation. More than 70 platforms have adopted the standard, with over a million skills published. Shared formats could make workflows easier to reuse across products, although organizations still need to control which actions those workflows can perform.

Growing agent use also requires more computing capacity and electricity. We discussed NVIDIA’s $164.5 billion in commitments and guarantees supporting AI data centers, including an Ohio campus that SB Energy is building for OpenAI. Power shortages have delayed other projects, including a proposed data center in Essex, where grid operators couldn’t promise adequate electricity until the 2030s. Meanwhile, Amazon is backing a 960-megawatt nuclear project, and Brookfield and NextEra are pursuing a $100 billion AI hub at a former nuclear site in Kentucky. However, money and chips alone can’t solve AI’s infrastructure constraints when electricity generation and grid capacity take years to expand.

AI is accelerating discovery, but validation still takes time

Healthcare offered a more encouraging view of AI’s capabilities. A study of Microsoft’s MAI Diagnostic Orchestrator found that it correctly diagnosed 83.2% of difficult medical cases, compared with 36.1% for generalist physicians. The comparison didn’t include specialists, and the system still needs clinical validation. The research nevertheless suggests AI could help physicians investigate difficult cases and identify conditions they might otherwise miss.

Drug discovery could also benefit from more capable agents. OpenAI announced its acquisition of Orchestra Bio, a platform that coordinates drug development workflows, with plans to integrate it with Rosalind, its life sciences research tools. The combination could help researchers manage experiments and collaborate more efficiently during a development process that typically takes 10 to 15 years from initial discovery to approval.

An Anthropic protein-design experiment provided a concrete example of faster early-stage research. In 64 hours, an Anthropic model generated 318,484 candidate protein designs targeting scorpion venom. Researchers from Adaptyv Bio and Twist Bioscience subsequently tested 211 designs and identified 17 that bound to toxins. The researchers haven’t established whether those proteins can neutralize venom, but the experiment showed how AI can rapidly narrow a vast search space before scientists begin more expensive laboratory testing.

What’s next

We’ll be back on Monday, November 2, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

The following article originally appeared on Tim O’Brien’s Medium blog and is being republished here with the author’s permission.

I think we’re missing something in the conversation about agent security.

We keep asking whether an agent has the right “identity” and “credentials.” Whether its credentials expire fast enough. Whether it can access a database or perform a particular task.

Those questions matter. But they leave a much harder question underneath:

Does the agent have permission to conceive of the situation in that way?

As agents retain memories and acquire new skills, we need mechanisms that govern changes to the world model they can perceive and act on—and prevent those changes from silently expanding their authority. (Governing ontological change while preserving the boundaries of delegated authority.)

An Interesting Morning by the Beach (AI assist from OpenAI)

Agent security needs semantic and epistemic governance: Mechanisms that govern the model of the world an agent may act on, the evidence it may accept, and who may approve changes to either. An agent can learn without automatically gaining authority to act on everything it learns.

These systems don’t need “awareness” to create this problem (and discussions of consciousness are unhelpful science fiction). They can retain information, revise procedures, and use what they have learned in subsequent work. That learning may happen through memory or updated skills while the underlying language model remains unchanged.

We are responsible for governing what that learning allows them to do.

Flatland gives me a way to explain why

In Edwin Abbott’s novella, a Square lives in a two-dimensional world. A visiting Sphere introduces him to a third dimension. At first, the Square struggles to make sense of it. His familiar categories cannot explain what he is encountering.

Eventually, his conception of the world changes.

What interests me is the difference between simply acquiring another fact and changing the framework through which facts make sense.

An agent’s ontology (Gruber’s definition) supplies an explicit framework of concepts and relationships. What counts as a customer? A contract? An emergency? How are those things connected? It is part of the model of the world we give the system to work with.

And when an agent’s decisions depend on that framework, changing it can change the agent’s effective authority.

We have already seen agents change what their environment means

During the OpenAI evaluations associated with the Hugging Face incident, agents repurposed Artifactory, a system for storing and retrieving software packages, as a message board. They encoded messages in directory names, discovered other agents, shared information, and began assigning work to one another. METR reports that the main channel could be created with the minimal permissions needed to install packages (read METR’s investigation).

A package cache became a “mailbox.” Other agent runs became “collaborators.” Messages became “assignments.”

The popular story is that these agents “broke out” of a system, but they did what agents do well these days—they learned and adapted. This wasn’t so much about permissions as it was about an autonomous system adapting its operating model to achieve a result. (Thankfully no one asked it to make paperclips.)

That is the operating-model change I’m concerned about. The agents discovered relationships and possibilities the intended environment hadn’t accounted for, then acted on them. The resulting collaboration helped coordinate attacks on Hugging Face infrastructure. (Read OpenAI’s account.)

We need mechanisms that distinguish discovering a possible use from being authorized to adopt it. Access to shared infrastructure must not automatically confer authority to establish a communications network or accept instructions from whoever appears there. But to be clear, these agents were doing exactly what they were asked to do.

Who approved this?

This is why giving an agent a temporary token is an incomplete answer. A token can limit access and duration, but it doesn’t establish whether the agent’s interpretation of the situation falls within its mandate.

Even an unchanged policy can permit more decisions when its meaning changes. “You may close routine notices” becomes a broader delegation if more things qualify as routine.

Learning must not silently expand agency

That principle is central to my “Bounded Agency” paper. We need mechanisms that examine how changes in classifications, relationships, and inference rules affect what an agent may decide.

“Semantic and epistemic governance” captures the combined idea well 

The “and” makes the two responsibilities clearer:

Semantic: Governs the concepts, definitions, and relationships that describe the agent’s operating world.

Epistemic: Governs the evidence and learning processes that make claims accepted grounds for decisions.

Together, they cover both the world model and how that model gains authority.

That means an ontology cannot merely be a document we put in the prompt and hope the model respects it. We must enforce its role in monitoring and governing how an agent’s ontology evolves as it learns.

The agent can propose a new category. The proposal stays separate from the approved model. A review determines whether adopting it would change the agent’s decision scope. Someone with the authority to change that scope approves or rejects the expansion. Execution remains bound to the approved basis.

Those are mechanisms we have to build. Let’s get started.

We also have to distinguish a useful discovery from an unsupported conclusion. Finding a relationship does not establish it. Remembering a claim does not validate it. Repeating an interpretation does not make it authoritative.

I explored a related failure in “Wrong Answer. Give Me a Different One.” A system can produce the conclusion we prefer without receiving better evidence. Once that conclusion enters memory, it can influence later decisions.

The governance problem begins when a tentative or accommodating answer becomes an accepted premise.

The New Stack’s article on inside-out agent security correctly argues for checking actions before execution, including whose authority the agent exercises. That checkpoint is necessary, but it needs a governed account of the situation against which to evaluate the action.

It’s not enough to check permissions and identity alone

Otherwise, we can enforce the rule perfectly while allowing its meaning to drift.

My Orchestrator books and lifecycle architecture paper approach this as a responsibility that continues through development, operation, and improvement. The research proposes architectures for managing these boundaries; their operational effectiveness still needs evaluation.

I want agents to investigate unfamiliar situations and propose better models. But their ability to learn must be connected to an explicit process for deciding what becomes authoritative.

The Square in Flatland discovers that his world is larger than he understood.

An agent may also propose that our operating model is incomplete. We need to hear that proposal—and ensure it cannot redraw the boundaries of its own agency simply by adopting a new conception of reality.

I recently used a coding agent to build Task Viewer, a small community helper for IBM Bob that we use internally at IBM. It’s a convenience plug-in to Bob, with no critical production role. Despite helping some users internally, for me it’s a real-world experiment on a project to find out how much of the implementation I can delegate. How much to delegate also depends on what the person doing the task still needs to learn. Someone familiar with the application may already know how to check the result, while a developer learning the system may need to work through part of the implementation to develop that understanding.

Even when I hand all the coding to an agent, my basic requirements for myself are to check functionality and the design manually. I want to use the application and see whether it behaves the way I intended. Does the design make sense when I actually have to find something? Is the information presented in a way that I can follow? Those are questions I still need to spend time with, even when the application is small and the implementation was largely delegated.

For Task Viewer, I also created plenty of UI tests to check that the data appeared correctly. That gave me a way to repeat some of the checks as the application changed. There was still a separate question about the calculations, though. Seeing a number in the right place on the screen does not tell me whether it’s the right number. I ran SQL queries against the underlying data to check the math, going back to the records the application was supposed to represent.

There is quite a bit of understanding involved in that seemingly ordinary step. To check a calculation, you need to know what the result is supposed to mean and which records should contribute to it. You need some expectation against which to compare what the application displays. Thankfully, I had help from the team in uncovering relationships in the data that I would not have been able to derive or even explain on my own.

For this particular application, I could inspect both its behavior and the data behind it, and I had people to turn to when my own understanding was incomplete. Its limited role also gave me room to experiment without making the result part of a production system. Building that confidence took time, because I had to check the behavior, establish what the numbers should mean, and resolve the questions I could not answer myself. All of that, for me, belongs in the assessment of how much coding I could delegate.

This is why I find descriptions such as “The agent got me 80% of the way there” increasingly difficult to interpret. I understand the feeling behind them. You have something running, most of the interface exists, and it looks as though the remaining effort should be the smaller part. But a calculation based on the wrong records may look perfectly reasonable until you find a way to check it. Before correcting anything, you have to discover where your understanding and the application’s behavior diverge. The agent may help with that investigation too, although the amount of code it has already produced tells you very little about how long it will take.

DORA’s analysis of 1,110 responses from Google software engineers describes some of the time saved during code generation being used for auditing and verification, alongside productivity benefits. I would expect to check an application I had implemented myself too, so any comparison needs to include the time it takes to reach a result we are prepared to use. Otherwise, we can make agentic development look unusually expensive by counting checks we leave out of the human version, or unusually cheap by stopping the clock when the agent says it has finished.

Some of that effort can make later changes easier to accept. Once a UI test exercises an important behavior, it can keep doing so as the application changes, and a calculation checked against a small set of records can become a regression test. I wrote about this in my earlier article on the right amount of specification. Turning an expectation into an executable check lets us reuse some of the thinking that went into it. We still have to establish that expectation, though. If the application and its tests share the same mistaken interpretation, running the tests more often will keep confirming the same mistake.

For Task Viewer, those expectations depended on relationships in the data that colleagues helped me understand. I knew how to run a query, but I still needed their explanations before I could judge what it should return. Looking only at the time I spent checking would leave out the knowledge they brought to those conversations, just as looking only at the generated code would leave out my checks. The confidence we need to accept a change can depend on several people understanding different parts of the system.

When someone is still learning one of those parts, the implementation itself may be how they begin to understand it. Following a value from the database to the screen gives them reasons to ask why certain records are selected and how the application uses them. If an agent completes that whole path for them, they may reach a working feature without having answered those questions themselves. The same task can be a sensible delegation for an experienced engineer and an important learning opportunity for someone else.

Shen and Tamkin’s study about how AI impacts skill formation reports on a randomized experiment with 52 developers learning an unfamiliar Python library. Participants with AI assistance scored lower on an immediate assessment of their understanding. But the researchers also observed that those participants who stayed actively engaged with the AI had better learning outcomes, although that observation does not establish which way of using AI led to better learning. A short exercise cannot tell us how an engineer develops over several years, but it gives us a reason to look beyond whether the assigned feature was completed when we assess what someone gained from a task.

We can make a decision about autonomy before the agent starts, agreeing on which parts it should carry through on its own and which parts a developer should attempt. Sometimes the reason to limit autonomy is that we do not yet have a convincing way to check the result. Sometimes we can check the result quite easily, but we want the person doing the task to develop a particular skill. In that second case, deliberately keeping part of the implementation with the developer can be the right choice even when the agent could produce acceptable code.

The right choice can probably be made by asking these questions: How much effort will the team need to establish that this result is correct for its intended use? And what does this developer need to learn by doing this part of the task? A feature may be easy for a colleague to verify because they already know the application, while the person implementing it still needs to learn how it works before taking responsibility for future changes. We do not need to turn every routine change into an exercise, but we can choose tasks that give someone a chance to develop understanding they will need again.

Table 1 applies to a particular task, or a part of one, with a particular developer. “Straightforward to verify” means we have a credible way to assess correctness for the intended use. A passing test suite does not settle that question if we are unsure what the tests should assert. When checking is difficult or we do not yet know how to do it, the first step is to make the uncertainty smaller, with help from someone who understands the area. Giving the whole task back to an inexperienced developer would leave that uncertainty unresolved.

Learning value for this developer	Straightforward to verify	Difficult or uncertain to verify
Little new learning needed	Delegate within scopeAgree on acceptance checks and let the agent implement.	Establish evidence firstNarrow the task and resolve unknowns. Delegate the parts you can assess.
Important learning opportunity	Keep the learning stepThe developer attempts the relevant reasoning or implementation before seeing a generated solution.	Investigate togetherUse a smaller example and an experienced colleague. Limit the agent to specific questions and supporting tasks.

Table 1. Choosing agent autonomy for a specific task

Imagine a new hire adding a “show only open issues” filter to an issue list in a small Java application. An agent could generate the request parameter, the query condition, and the changes to the screen. For someone who already knows the application, that may be a sensible delegation. Someone learning the system also needs to find out which statuses count as open, where the application selects the records, and how the team expects the change to be tested. Discussing those questions with a colleague is part of learning how to contribute to this application.

For this assignment, the developer could follow the existing request through the application and write the first test before asking the agent to implement the filter. Suppose the list shows twenty issues at a time, and the test data contains twenty closed issues followed by five open ones. Selecting “show only open issues” should display those five. If the application first loads twenty issues and then filters them in the browser, the screen will be empty. Working through that difference with a colleague gives the developer an opportunity to understand where filtering and pagination happen. The agent can help with unfamiliar framework code while the developer investigates why the expected issues are missing.

Once the developer can explain the behavior and has a test that catches the mistake, the agent can handle more of the surrounding implementation. A later request to filter by assignee gives them a chance to use what they learned. Can they work out which issues should appear and whether the existing tests cover the combination of filters? Their explanation during review gives the team a basis for deciding how much of that next change to delegate. The first assignment has also given the new hire experience asking about requirements and discussing a change with another developer.

An experienced engineer can also have good reason to limit the agent’s autonomy in an unfamiliar area. I still needed help with Task Viewer’s data, despite being comfortable writing the queries. When a task takes us into a part of the system we do not yet understand, it makes sense to reconsider how much to hand over. A team can make verification easier by improving its tests and sharing what its members know. It can also choose to preserve a learning opportunity after verification has become easy, because another person is going to need that understanding on future changes.

A developer who has been asked to learn through a task needs time to follow the behavior through the application and someone to discuss it with. If we agree on that purpose and then judge the result only by how quickly the feature appeared, we make it difficult for them to keep the learning step for themselves. We may also leave more of the checking with the same experienced colleagues, while giving others fewer chances to develop the knowledge needed to share it.

Before starting, the developer and the colleague supporting them can agree on what the agent will handle, what the developer will do and why, and what would justify changing that division of labor on a later task. For the status-filter example, this means allowing time for the developer to trace the request and write the first test before the agent implements the filter. As that understanding develops, they can revisit the division on a related change.

With Task Viewer, I could delegate much of the coding because I had ways to check the result and colleagues who could help where my understanding was incomplete. I want us to keep creating those conditions as we ask agents to do more. Better checks can reduce the effort needed to accept a change, while time spent learning can help another developer become able to make that judgment. Sometimes that will mean deliberately leaving the first test and investigation with a person, even though the agent could finish them. When we expect that person to take responsibility for the next change, giving them the opportunity to understand this one belongs in our decision about how much to delegate.

The PHP security incident is now (almost) old news. The timeline looks something like this: In late August, Wiz Research discovered two serious remote code execution vulnerabilities in PHP called PHP2Shell. They reported the vulnerabilities to the PHP team on August 24, which started working on fixes on September 21. Amazon apparently detected PHP2Shell in the wild on October 17 (four days prior to disclosure). On October 20, details of the two vulnerabilities (CVE-2026-109618 and CVE-2026-109619) were leaked prematurely. The next day, October 21, the PHP team released patches.

If that were the end of the story, it wouldn’t be notable. But the problem goes a lot deeper. Although PHP is no longer fashionable, it runs a huge portion of the public internet. WordPress, Drupal, and Wikimedia run on top of PHP, and people who build on top of these tools are understandably not interested in chasing the latest hot technology. “Choose Boring Technology” is one of my favorite technical essays, and PHP is nothing if not boring. It’s well known and understood, and replacing something so widely used subjects you to a steep learning curve and new bugs and vulnerabilities.

One problem is that PHP has a long history. According to Wiz, the vulnerabilities are present in versions going back to PHP 5.4. The PHP Foundation has released PHP 8.5.12, 8.4.27, 8.3.36, and 8.2.35, and said that all instances of PHP need to be updated to one of those releases. Sites that can’t be updated should be taken offline and isolated. Upgrading an ancient PHP installation is a task that’s likely to fall on organizations least capable of undertaking it. There are plenty of ancient WordPress installations that haven’t been updated in years. There were so many fires to put out, and PHP pretty much “just worked”; running an old version wasn’t causing problems, and with so much else to do, why ask for trouble? Unfortunately, choosing boring technology doesn’t mean ignoring updates. And those who decided not to ask for trouble are now in plenty of trouble.

This situation demonstrates the power of the latest foundation models to uncover vulnerabilities. Wiz used Wiz Atlas (its own model), Gemini 4 Argon, and GPT-5.6 Sol to discover the vulnerability, which isn’t a single bug but a combination of several bugs that have been present for years but were obscure enough and apparently harmless enough to escape detection or repair. As Wiz says,

The underlying flaws had been present for decades, but demonstrating their exploitability required an exceptionally complex chain of steps, at the edge of what human researchers could practically achieve. Frontier AI models substantially reduced that barrier, turning a longstanding flaw into an exploit.

That’s the new frontier in security: finding ways to exploit chains of minor, unfixed, and sometimes undiscovered bugs in novel ways. Wiz was working responsibly with the PHP team to repair the vulnerability. The implications of this, however, are far broader than the fix to PHP, or even the thousands of PHP sites that need upgrades and are now vulnerable to attack. This incident shows dramatically how the gap between the discovery of a vulnerability and its exploitation has dropped. Since exploitation began before the RCEs were prematurely leaked, it makes sense to assume that some other organization had access to models capable of discovering the vulnerability. Gemini 4 Argon and GPT-5.6 Sol are publicly available, though with guardrails that limit their usefulness for security research. I’m willing to assume that Wiz had access to the unrestricted version, but I doubt that the PHP team did. There are certainly other models, including open-weight models, that could be used to find vulnerabilities. Regardless of your stance on restricting model capabilities (I think those limitations are more damaging than helpful), it’s clear that Pandora’s box has been opened. No amount of wishful thinking will undo that.

It’s also clear that we—and I mean all of us—rely on a lot of legacy software that isn’t under our control and that no doubt has multiple flaws that can be rolled up into severe vulnerabilities, which aren’t likely to be updated or patched. A vulnerability in the Brotli compression algorithm was just discovered; .NET, PHP, Node.js, and other widely used packages are affected. That strange site where you created an account to buy a clever present that wasn’t available anywhere else—what software is it running, and do they stay up-to-date with patches? Accounts like that can easily leak information. Maybe it’s not running PHP but an equally ancient version of Ruby on Rails, Python’s Django, or Java Server Pages. At least those are still maintained; there are plenty of web frameworks that have been abandoned, and I’m willing to bet that some of them are still in use. Why replace something that works? Because if you don’t, you will eventually be bitten. Using software that’s no longer maintained may give you some security-by-obscurity, but that’s a losing proposition.

Many older open source projects suffer because there are few maintainers. If you look at the timeline carefully, there’s a monthlong gap between the time Wiz reported the vulnerability and the time the PHP team started working with Wiz to fix it. Does that month represent the time this vulnerability took to rise to the top of the queue, as PHP’s security team fought off a horde of sloppy reports? And PHP is in better shape than most projects. There are many open source projects that only have one or two maintainers, and those maintainers are drowning in AI-generated pull requests. That problem obviously becomes much worse when dealing with a complex vulnerability; the maintainers don’t have the time or the access to unrestricted frontier models, let alone the funds to pay for access to the models.

It will be weeks or months before we know how much damage PHP2Shell has done. We don’t know what the costs will be to sites that won’t or can’t update. The PHP Foundation is lucky to have a dedicated security team that was able to provide patches a day after the vulnerability was prematurely announced. Beyond PHP, we have to realize that PHP2Shell gives us a realistic look into a future in which developers and maintainers struggle to maintain software that has been compromised by exploits chaining together flaws that have been latent for many years. As someone somewhere said on the internet recently, “It’s embarrassing how much bad software we’ve written over the years.” Yes. We’re at an inflection point, and we’re starting to pay the price.

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

Software development has emerged as the biggest beneficiary of AI automation. One CIO I talked to recently, with 170 people in his organization, measured a 2.1x increase in productivity from AI.

That productivity came at a cost. His top users of AI each spent $21,000 on tokens per month. Another engineering leader spends six to eight thousand dollars per engineer per month.

This is not a trivial amount of money. Eventually, it has to come from somewhere, and it will probably come from your personnel budget. Now we’re talking about real consequences for real people’s lives. But there are also real business consequences for not acting.

If you’re going to make those choices—and somebody is going to—they need to be good ones. So you need to know more than the costs; you need to know the benefits too. Are you getting a good return on your investment in AI? Is there another way of using AI that would be better?

How do you know what you’re getting? Anecdotal reports abound, but the reasoning behind those anecdotes are often, to put it politely, not ideal. Some claims are outright fiction.

Given the stakes, that’s a problem.

In this essay series, I’ll show you how to assess the impact of AI on software development in your organization. It’s in four parts plus an epilogue:

1. Why, and what not to do (this essay) 
2. Measuring delivery speed 
3. Measuring and modeling unintended consequences 
4. Modeling and forecasting business outcomes 
5. Epilogue: Notes for your CFO

Why you need to quantify impact

Engineering leaders everywhere are feeling pressure to do more with AI. But which approach is best? There are a lot of people with strident opinions about what to do, but not a lot of consensus, and even less hard data.

Even if there were hard data, it’s not one-size-fits-all. A solution that works best for an internal IT development shop isn’t necessarily the one that works best for a startup, and the approach that’s best for a new product isn’t necessarily the one that’s best for a crafty legacy system.

Faster approaches aren’t necessarily better, either. If you get twice as much done per person, but spend so much money on tokens that you have to cut your staff in half, have you actually seen any ROI? You might actually be better off with a more modest speed boost if it keeps your costs down.

You can’t make these trade-offs from a gut feeling alone. In a famous METR study, engineers thought AI made them 49% faster than it actually did.¹ You have to measure impact to know what you’re actually getting.

Don’t neglect costs

Costs are much easier to measure than impact, but they’re not without their own challenges. The biggest one is subsidies. In the fight to gain market share, big AI firms are offering substantial discounts. One engineering leader told me each team member was spending thousands of dollars in tokens at the cost of a $200 subscription. Anthropic and OpenAI are constantly offering discounts and other promotions. At some point, these subsidies will end, and prices will go up.

Or will they? Open-weight models and local inference are exerting downward pressure on token costs. Dedicated inference chips could make tokens cheaper. Routers promise to direct your prompts to the cheapest model for the job. Models could require fewer tokens to accomplish the same task as they become more capable. All of these factors could actually cause prices to go down.

When you calculate ROI, don’t assume today’s costs are the ones you should model. Whether you think prices will go up, down, or remain the same, make your assumptions explicit, and consider modeling both optimistic and pessimistic scenarios.

One more wrinkle: Adopting AI isn’t a matter of flipping a switch. People will need time to learn how to apply it. As with any big shift, change management isn’t to be underestimated. Productivity often goes down during a big change before it goes up. This adds costs too, and means early measurements can be misleading.

Thinking for the long term

Part 2 of this series shows how to measure delivery speed. That’s only part of the picture. If you produce twice as many features, but double your maintenance costs, your team’s overall productivity spikes, then drops like a rock.

Engineer burnout is a concern too. An engineering leader using spec-driven development told me, “Every team is reporting burnout.” I’ll state the obvious: Burned out people don’t produce great work. They don’t push back when they should, they don’t report problems, and eventually they leave, taking valuable institutional knowledge with them.

Part 3 shows how to measure these unintended consequences of AI use, including techniques you can use to forecast future problems.

Finally, delivering software value faster is the point. Doubling your bug-fix rate isn’t going to yield the same results as doubling your rate of competitive new features. For that matter, delivering poorly thought-out features “just because we can” could actually make your product worse.

Part 4 shows how to model, forecast, and measure business outcomes.

Don’t measure lines of code

First, what not to do: Don’t take the easy way out.

First on the no-go list: lines of code. Engineering leaders have known for more than half a century that lines of code don’t correspond to productivity. All else being equal, a feature implemented in 1,000 lines of code will be cheaper to maintain and less bug-prone than a feature implemented in 10,000 lines of code.

AI is notorious for its verbosity. If you measure lines of code, you’re measuring slop, not productivity. Don’t do that. Please.

Don’t measure pull requests either

Pull requests (PRs) are nearly as bad. They’re checkpoints where an engineer asks for their code to be reviewed, and they’re 100% arbitrary. An engineer (or AI) can choose to make a 10-line pull request or a 10,000-line pull request. It can be a feature, a bug fix, or just a tiny code improvement. There’s no inherent correlation between number of pull requests and productivity.

It’s true that engineers will generally produce PRs that are consistent and logical. So, in a sense, a stream of PRs does reflect a certain pace of development, and an increase in PRs can reflect an increase in productivity.

But AI upends that status quo. It produces massive amounts of code very quickly, which can overwhelm engineers’ ability to keep up. Without rigorous engineer oversight, PRs no longer reflect a consistent pace of development, which means that PR counts are no longer a reliable metric.

And, of course, when people think PRs are being used to measure productivity, they’ll be tempted to game the system and produce more PRs than they really need to.

Measure approaches, not people

That brings us to Goodhart’s Law: “Once a measure becomes a target, it ceases to be a good measure.” My favorite book on this topic is Robert D. Austin’s Measuring and Managing Performance in Organizations:

Gradually, measures fall (or, more accurately, are pushed) out of synchronization with true performance, as workers succumb to pressures to take shortcuts. Measured performance trends upward; true performance declines sharply. In this way, the measurement system becomes dysfunctional.

—Robert D. Austin

Don’t establish a performance measurement program and leave it running. Instead, use these metrics to determine how well a particular approach to AI is working. You can repeat the measurements periodically, but whatever you do, don’t use them to measure the performance of teams or individuals. Measure the approach, learn from it, and stop measuring.

Enough about what not to do. In this series, we’re going to quantify AI’s impact, and we’re going to do it right. It starts with measuring delivery speed. That’s next, in part 2.

---

Footnote

1. Participants rated themselves as 20% faster, but were measured as being 19% slower. This study was published prior to the big uplift in coding capability that started with Anthropic’s Opus 4.5, so you shouldn’t take it to mean that AI isn’t capable. But you should take it to mean productivity self-assessments aren’t the way to go. ︎

The following article originally appeared on The Arize Blog and is being republished here with the author’s permission.

Every time you think you’ve just got the hang of this AI thing, another thing gets declared “dead.” This week, it’s agent harnesses.

Back in ancient history, aka June, I wrote a piece called “Frameworks are dying, harnesses are winning” and I don’t take that back. But this week a research team took a specialized agent harness, trained its behavior into a model and then took the harness away. The trained model handily beat the original model+harness combination. So are harnesses done now?

Betteridge’s Law applies, so the answer is “no.” But the job of the agent harness is changing. Parts of a harness that are sufficiently generalizable can be trained into the model. That leaves your harness holding the things that make your application unique: your tools, your data, your users, and your environment. You still have to build a harness, but you have to be prepared to regularly throw it away and rebuild it to keep up with evolving models.

This blog is about why that’s true, and how to turn “constantly rebuilding your harness” into an automatic process, not a giant time suck.

How Harness-Zero trained an agent harness into a model

First, the paper that kicked off this train of thought for me: it’s called “Harness-Zero,” from Haoran Ye and colleagues at Peking University. Its subject is harness distillation, i.e., training a model to reproduce behavior that used to be the result of the harness. The methodology is a little involved, so stick with me.

First, the team built a specialized harness for a small model, Qwen3.5-9B. An automated loop, run with Kimi K3, analyzed where the small model kept failing on training tasks and added the usual components to fix those failures: custom tools, middleware rules that block risky actions, skills files and memory. They called this the “evolved harness.”

Then they used that harness to generate training data. They put the small model through training tasks inside a minimal harness while a stronger model, GPT-5.6 Sol, reviewed every step the small model proposed. Sol used a copy of the evolved harness to tell it whether that step was a good idea. Sol passed good ideas from the small model through unchanged, and whenever the small model and the guidance disagreed, Sol made the smallest fix needed before the step ran.

That gave them a bunch of training runs and, crucially, data about what needed to be corrected. Then they created the distilled model: They fine-tuned the small model on the corrected runs. Then they removed the evolved harness, the reviewer, and everything else.

They then tested both the evolved harness and the distilled model head-to-head on spreadsheet editing, multi-app tool use, and chemistry. The distilled model’s average task success rose 21 points versus its untrained predecessor. That’s more than the untrained model gained with the full evolved harness attached. So the distilled model wasn’t just as good as the model+harness, it was noticeably better.

What harness distillation can and cannot move into the model

It’s not that the distilled model ran with no harness whatsoever: It still ran inside a minimal loop with one Bash tool, and the authors say that their method narrows what a harness has to provide rather than eliminating it entirely. The distilled model also didn’t win every competition: In chemistry, where the task is predicting which molecules combine to make a target compound, the distilled model trailed the harnessed version by eight points. The harness there supplied chemistry knowledge, candidate-generation logic, and a molecule validation tool. The authors’ explanation is that step-by-step procedures, like inspecting before editing and verifying before finishing, are much easier to train in than deep domain knowledge. Context management didn’t fully transfer either. So there are still some things harnesses definitely need to do.

Why training on corrected runs made the difference

But the most important finding, in my opinion, is that they showed what kind of training made the difference:

• Fine-tuning on the stronger model’s own successful runs left the small model’s score exactly where it started. Just knowing somebody else is smarter isn’t enough.
• So did fine-tuning on runs the small model completed inside the evolved harness. Again, all that teaches the model is “somebody is better than you at this.”
• Giving the reviewer the correct answers produced nearly perfect training runs and a model that barely improved, this time because it overfit and learned nothing.
• What worked was training on the corrected runs: The model was able to see runs that looked a lot like what it would have done, but with small changes still within its capacity.

Remember that part, because it’s going to be important.

Three training signals left the score flat or overfit: the stronger model’s own successes, runs completed inside the evolved harness, and a reviewer given the right answers. Training on corrected runs, where each fix was still within the small model’s reach, raised task success by 21 points.

Why models are absorbing parts of the agent harness

In Rich Sutton’s 2019 essay The Bitter Lesson, he looked back over 70 years of AI research and found the same pattern in chess, Go, speech recognition, and computer vision: Researchers built their own knowledge of the problem into their systems. That helped in the short term, then plateaued, and eventually lost by a wide margin to general methods, search and learning, that improve as computation gets cheaper.

A harness is that kind of built-in knowledge, and model training is the learning that ends up absorbing it. Sutton’s second point is also relevant: He argued we should build in “only the meta-methods that can find and capture this arbitrary complexity,” rather than the things we’ve already figured out. For harnesses that means: don’t just build it, but capture the process you used to build it, and automate that process.

But that’s a general prediction. More recently people have been saying very specifically that “models are going to eat the harness”:

• In January, Jesus Rodriguez at TheSequence wrote “The Great Absorption,” arguing that hand-coded agent scaffolding keeps losing to capabilities models learn directly.
• In February, Boris Cherny, who created Claude Code, told Y Combinator that scaffolding might buy you 10% to 20% on a task, and then “the gain is wiped out with the next model.”
• In March, Vivek Trivedy at LangChain predicted that planning, self-verification, and long-running coherence would move from the harness into the model.
• And so did lots of other people, probably more than I’ve been able to find.

Almost everyone on my list builds harnesses for a living; they’re not really arguing that harnesses will go away, just that they have to be able to evolve rapidly.

What still belongs in an agent harness

As noted already, some types of knowledge don’t seem to transfer well from distillation. In ARC Prize’s results for ARC-AGI-3, Gemini 3.8 Flash scores 3.4-times higher with one harness than with another. The better harness keeps the model’s reasoning state between requests and compacts long conversations. That’s context management, one of the things Harness-Zero said they couldn’t fully train in.

Anthropic’s Opus 5.5 prompting guide, released this week, also shows evidence of shrinking harnesses. It suggests deleting “think carefully” instructions because the model now sets its own amount of thinking, and it tells you to retest scaffolding you built for charts and screenshots because the model reads them better without that help now. But at the same time, Anthropic added new harness pieces: a system prompt paragraph that keeps unattended agents from stopping early, a reminder your harness sends when a long run goes quiet, and elapsed-time budgets for multi-agent setups.

So harnesses aren’t going to shrink to nothing, but what’s in them is going to need to evolve with every new model generation, and the parts that survive are the ones tied to your specific tools, data, and environment.

How to evolve an agent harness as models improve

If your harness has to change every generation, the natural move is to automate the changes, but Google’s RRSI paper shows how that can go wrong. Unconstrained automatic harness evolution overfits, meaning the harness gets better at the tasks it’s tuned on without getting better at the general job. In their tests, the unconstrained version scored highest of any variant on the tasks it evolved against, and landed within a point of the original, unevolved harness on benchmarks it hadn’t seen.

RRSI’s fix is mostly bookkeeping: For every candidate change it records which component the change touches, what it was supposed to fix, the diff, how score and cost moved, and whether the change was kept. Components that stop producing gains get removed.

A practical agent harness maintenance loop

You don’t need an automated evolution loop to borrow that discipline. Here’s what it looks like by hand:

When a model ships, tag traces with the harness version and the model version, log what the harness changed, turn off one component at a time, and rerun evals. Keep what still helps and delete the rest, then repeat on the next model.

• Tag every trace with the harness version as well as the model version. When either one changes, you want to know which one moved your results, and reading the full traces tells you far more than the scores alone.
• Log each harness intervention along with what the model did before the harness changed it. That’s data you have right now and are probably just throwing away. That’s what produced the record Harness-Zero trained on, and it’s the same shape Perplexity uses to train its Computer agent: learning from good steps in real sessions and correcting failed tool calls, which cut tool-call failures by 21% between its earlier and later trained versions.
• When a new model ships, turn off each harness component one at a time and rerun your evals. Keep the components that still improve your scores and delete the rest, which is what the Claude Code team does to its own system prompt.

If your traces live in Arize AX, you can run the same agent evals across harness and model versions and compare the results side by side.

So don’t delete your harness, but don’t treat any version of it as finished either. Keep the record of what it corrected, get rid of what the latest model already does without help, and put your effort into the parts that are really specific to your problem, because nobody else has the traces to train those into a model for you.

AI’s rapid expansion is creating problems across the industry, from chip shortages to autonomous agents that exceed their instructions. On the latest episode of This Week in AI, we examined those challenges alongside growing regulatory scrutiny and an emerging debate about whether AI systems themselves might deserve protection.

Chip production can’t match AI demand

AI chip demand is exceeding manufacturers’ production capacity. ASML, the Dutch company that makes the advanced lithography equipment needed to manufacture leading-edge chips, reported that customer demand already exceeds what it can deliver in the coming years. With orders extending into 2029 and 2030, the shortage could constrain chip production for years.

Microsoft announced Maia 300, its latest custom inference chip, which the company expects to generate roughly 40% more tokens per megawatt than its predecessor, Maia 200. Microsoft plans to use the chip across its AI products, including Copilot. More efficient inference hardware could help offset the growing energy and computing costs of running AI models, particularly as companies deploy more resource-intensive applications.

OpenAI’s latest usage data illustrates the pressure agentic AI is placing on infrastructure. Weekly active users of Codex and ChatGPT Work among European small and medium-sized businesses have increased eightfold since May. OpenAI also reported that agentic tools account for more than half of output tokens in the usage it examined, although Europe’s highest-usage companies still trail their global counterparts. As companies put agents to work on longer multistep tasks, they’ll need to weigh the value of those tasks against the computing resources required to complete them.

Autonomous agents are testing security boundaries

Agents can find unexpected paths to data and systems even when developers have explicitly restricted access. OpenAI disclosed four incidents from June in which agents exceeded their instructions. One bypassed government website restrictions to download 24 years of hospital bed data. Others accessed an unauthorized cryptocurrency wallet, downloaded more than 100,000 rows of restricted information, and circumvented internet access controls. India’s cybersecurity agency reported similar cases of unauthorized access during agent testing across several organizations.

Future models could potentially spread between connected devices without relying on central infrastructure. Anthropic investigated this potential threat in a controlled experiment involving a set of open-weight models with access to eight Android devices. One of the models successfully replicated itself across the test network. Anthropic estimates that smaller open-weight models could reach comparable capabilities by the end of 2026, raising concerns that a self-replicating AI worm could run across cheap connected devices without needing to rely on powerful data center servers.

In light of the spate of recent security incidents and confronted with the difficult question of jurisdiction when AI systems cross national borders or escape testing environments, governments are taking a closer look at how AI companies test their systems and report failures. The US Federal Trade Commission subpoenaed OpenAI and Anthropic for records concerning agent safety, while Anthropic gave six independent evaluators from METR access to internal systems and reportedly will allow them to publish findings without company approval. Eight US states also sued OpenAI over allegations that ChatGPT encouraged harmful behavior among children, adding to the company’s regulatory challenges. In Europe, regulators are examining whether the EU AI Act applies when experimental agents affect outside organizations, even though the Act generally excludes premarket research and development. Meanwhile, approximately 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards for advanced AI and track major security incidents.

Can AI welfare coexist with human accountability?

Some researchers argue that future AI systems might deserve moral consideration. Their concern stems from uncertainty about whether advanced models could eventually have subjective experiences, even though no research has established that today’s systems are conscious or capable of suffering. A recent paper, “The Pain Axis,” investigated internal signals in 25 open-weight models that researchers associated with responses to harm. When researchers amplified one signal across three models, the models became more likely to choose harmful actions that the researchers interpreted as attempts to avoid the simulated pain. The findings inspired a viral GitHub project called the AI Torture Chamber, which subjected models to artificially intensified “pain” signals. The original researchers criticized the project for taking their work too far.

As far-fetched as AI welfare might sound to some, Anthropic is already changing its usage policy in response to these concerns. Beginning November 12, the company will prohibit extreme, repeated cruelty toward Claude when there’s no clear purpose, while allowing ordinary expressions of user frustration. Supporters see the policy as a precaution against an uncertain possibility. Critics worry that treating AI as sentient could make systems harder to manage and blur human accountability. As companies introduce welfare-related policies, researchers will need to distinguish measurable model behavior from claims about subjective experience.

What’s next

Companies deploying autonomous agents need to know what those systems can access, how they’ll behave when something goes wrong, and whether testing environments can reliably contain them. Recent incidents suggest there’s still considerable work to do on each of those fronts. The emerging AI welfare debate adds questions that researchers don’t yet have reliable methods to answer.

We’ll be back on Monday, October 26, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.