Remember back in 2019, when you had only a handful of language models to choose from, all with limited context windows? That world is gone, and it feels like a forgotten past. Today, flagship providers and open source projects compete across model families, sizes, capabilities, and deployment options. Coding harnesses seem to be heading in the same direction.

On the surface, Claude Code, Codex, Pi, DeepSeek Harness, and the other coding harnesses all seem to do the same thing: help you build software with models. Look closer, though, and each one gives you control over different parts of the harness. What can the model see and change? How does it use tools? Where does it run, and what happens when it fails? So instead of asking which coding harness is best overall, ask:

Which layers of the harness do you need to control?

Thanks for all the harnesses! Now what?

Choosing a coding harness isn’t only about which model writes the best code. You’re also choosing what the harness provides, what you can change, and what you must build yourself. If it doesn’t give you verification, durable state, repair routing, or an enforced execution boundary, those responsibilities stay with you. You remain accountable for the workflow’s reliability and governance.

In general, each harness has a center of gravity: a layer that gives you the most direct control. The following table is a deliberately high-level overview of where that center sits. It’s not a feature matrix, and the boundaries overlap in practice. Claude Code can also run noninteractively and inside sandboxes, Codex also reads repository instructions, and all of them are changing quickly. The point is to show where each harness gives you leverage by default, and where the engineering work is likely to stay with you.

Table 1. Where each coding harness puts its primary control

Read the table as a map, not a ranking. A harness that hands you more control also hands you more to build, and a harness that does more out of the box decides more on your behalf. Which trade-off is right depends on which layer your project actually needs to own.

Claude Code: Control inside the run

Claude Code buys you detailed control over how implementation unfolds. CLAUDE.md provides standing instructions, skills add task-specific playbooks, subagents divide the work, and hooks intercept actions before and after tool calls. Unlike a permission rule that simply refuses an operation, a hook can explain the refusal or immediately return a compiler or policy failure to the agent.

These controls let you steer the coding loop extensively, but they operate within Claude Code’s existing lifecycle. If you need to control the outer lifecycle itself, you need another layer.

Codex: Control through separation

Codex buys you separation between a coding task and its execution environment. A noninteractive codex exec run can work on a narrow task in a specified repository, under an OS-enforced sandbox, and return a schema-constrained result.

This is useful for independent verification. You can give Codex a clean checkout, exclude the builder’s instructions, restrict writable paths, and run separate reviews for code quality, test gaps, persistence, or user-visible behavior. The sandbox depends on the operating system, but it moves scope enforcement out of the prompt and into the execution boundary.

Pi: Control over the loop

Pi buys you control over the coding loop itself. Its tools, commands, event handlers, context, models, and session state can be extended or replaced. This lets you decide which steps remain agentic and which follow deterministic policy.

For example, a local defect can return to implementation, a repeated defect can force replanning, and conflicting findings can trigger a recheck or human review. Pi can also run agents inside this lifecycle, so it’s more than a deterministic orchestrator. For my example below, I used its programmability to make the outer decisions deterministic—that was my design choice, not Pi’s definition.

DeepSeek’s harness: Control through composition

DeepSeek’s harness is useful when you want to swap out whole parts of the harness rather than steer or wrap them. Tools, sessions, lifecycle behavior, verification, and governance are all composable plug-ins, and so is the loop itself. That lets you replace one component without rebuilding the rest.

The trade-off is that your control is only as good as the plug-ins you compose. Where Pi gives you a minimal core to build on, DeepSeek’s harness gives you a set of parts to assemble.

A practical example

Let me give you the problem I was actually trying to solve: migrating a legacy Python 2.7 application reliably, without changing years of accumulated behavior or corrupting its persisted data. Getting the code to run on Python 3 was only one part of it. I also needed to preserve behavior, verify the data across both runtimes, control what the coding agents could change, and decide what should happen when a test or verification step failed.

So I approached the choice of coding harnesses from a very different perspective. Instead of asking which one was best, I asked what each one would buy me in terms of control, isolation, and lifecycle management. The answer led me to use more than one:

• Claude Code built the port because I wanted internal steering, specialized skills, hooks, and a factory of implementation and review subagents.
• Codex verified it in isolated runs because I wanted an independent model context, clean workspaces, constrained writes, and structured findings.
• Pi owned the outer lifecycle because I wanted durable state, evidence validation, bounded repairs, replanning rules, and human escalation.
• Frozen acceptance, including Playwright and Python 2/3 differential checks, ran outside the agents so neither the builder nor the verifier could redefine success.

I didn’t use DeepSeek’s Harness here because I wanted a very small outer lifecycle I could read end to end, not because it can’t do the job. To first check whether this setup would actually make a difference, I built a small Python 2 application with the same kinds of traps as the real migration. I then compared the composed setup with a single Claude Code session, in which Claude could verify and test its own code.

Interestingly, both setups passed all 25 hidden behavior tests. Yet the single-session setup silently collapsed two distinct Python 2 dictionary keys, the Unicode u’é’ and byte string ‘é’, into one Python 3 string. After the data was saved, one entry was permanently lost. The single-session setup also changed a Python 2 Unicode value into bytes without documenting it.

The test suite missed it. Claude’s own tests missed it. Hundreds of recorded behavior comparisons missed it. Even my hand-written reference port had the same hole.

But what happened next was even more interesting. Two isolated Codex verifiers independently reproduced the defect. Claude didn’t simply accept the finding; it contested it. Pi then launched a fresh Codex run that received only the finding and Claude’s argument. The independent recheck upheld the finding, and Claude accepted it. Pi routed the defect into a bounded repair, and the next verification round confirmed that both keys and their original types now survived.

It’s also ironic because models are often criticized for being sycophantic. They apologize and rewrite working code the moment you ask, “Are you sure?” Here, the opposite happened. Claude contested a defect that two independent verifiers had reproduced. Its session context contained its own implementation history and passing tests, all of which supported the patch. A fresh verifier context broke that self-reinforcing frame. Independent verification matters not because the second model is automatically right, but because it’s not anchored to the same history.

The important point is that no individual harness deserves all the credit. The better result came from combining the controls I needed: internal steering, independent execution, frozen behavioral evidence, and deterministic lifecycle policy.

How to choose your coding harness

When choosing a coding harness, start with the level of control you need and how much of the remaining harness you are prepared to engineer yourself. A useful comparison could begin with questions like these:

• Do I want to configure an existing coding loop, extend it, or replace parts of it?
• Can I control tools, context, memory, and model selection at the level my project requires?
• Where are permissions, allow lists, and policies enforced: in instructions, application logic, or the operating system?
• Can I intercept actions and lifecycle events before and after they happen?
• How much of the orchestration, state management, and session lifecycle can I change or orchestrate?
• What does the harness expose for observability, verification, and structured evidence?
• Which reliability and governance controls does the harness provide, and which ones remain mine to build?
• Can the harness work with other tools and harnesses if my requirements cross several control layers?

These are some examples to get you started, not an exhaustive checklist. The questions that matter for you will also depend on what you’re building, how much autonomy you’re willing to give the coding agent, and what you remain accountable for when it fails.

Your answers may lead you to Claude Code, Codex, Pi, DeepSeek’s harness, or another coding harness entirely. In the end, the developer’s name only tells you who packaged the runtime, not where your guarantees live. Whether you settle on one harness or orchestrate several, the principle is the same: Choose the tool by the boundary of control it gives you, and ask whether you want the agent to define the terms of its own success.

At our recent Foo Camp, a number of people said something like, “I don’t want someone else’s code any more. I may have my agent take a look at it, but I want to extract just the parts that I want, and may rewrite even those in Rust. It’s just too easy to create just the code that I want.”

This topic was hot enough that Jesse Vincent held an entire session about it. I did not attend, but Jesse gave me a report from his Plaude AI notetaker. Here’s a relevant section of the summary:

Several participants argue that software development’s primary artifact may be shifting from executable code toward natural-language prompts, generation processes, and developer skills. One speaker compares Python to assembly language, suggesting that writing conventional code may become optional as natural-language generation takes precedence. Another argues that open source can survive such a transition because its core values—peer production, capacity building, and shared specifications—do not depend on who or what writes the code.

At the same time, Jesse warns that maintainers are overwhelmed by floods of uninvited, automated, and often low-quality contributions. Participants say that traditional milestones such as pull requests, repositories, and conferences have helped contributors connect and gain recognition; as AI makes code replication easier, communities may need new ways to acknowledge participation and create paths for future developers. Some community interaction has moved to spaces such as Discord and direct messages, but structured opportunities are still needed.

Scientific computing raises particular concerns. One panelist warns that AI-generated forks and agent attacks could threaten carefully verified foundations such as NumPy and SciPy. Scientific systems depend on battle-tested libraries and reproducibility, so thousands of unreviewed alternatives could undermine reliability. Participants suggest bridging traditional open-source communities and modern AI development, which currently may use different tools and artifacts.

Other proposals include treating code as “liquid” and focusing open-source governance on durable infrastructure and specifications; maintaining agent storage independently of ephemeral container lifecycles; and using corporate compliance and safety-monitoring architectures to aggregate organizational traces.

These are important observations, all pointing in the same direction: AI is transforming the mechanisms and the artifacts of software development. For an increasing number of use cases, the software source code itself may no longer be the most important shareable knowledge artifact. This isn’t a new phenomenon. Every major change in technology calls for new mechanisms, new artifacts, and new institutions.

Originally, software developers mostly distributed binaries, then open source developers made sharing source code the norm. Now we need to distribute and share something else: context and patterns for generating code on the fly. Knowledge is being unbundled and can be shared in new forms. A repo may now serve merely as a source of context for solving the exact problem I have, rather than the bundle of problems that the original developer(s) encoded into their library or packaged solution. Some people will keep their context and patterns private, either because they see proprietary advantage in keeping them to themselves or because they are too busy or too lazy to share. But part of the problem is that we haven’t yet figured out what needs to be shared!

The challenge for the software development community is to identify the new knowledge artifacts of the AI economy, and to build new mechanisms and incentives for sharing them.

We’ve all had the experience of seeing a bespoke agentically generated app, wanting something similar, and wishing we could ask for a starting point so we didn’t have to vibe code an equivalent from scratch, with all the attention, false starts, and token costs that comes with that. Even in a world where software is generated by agents, it’s still not costless, and most of us like a good place to start.

Of course, this won’t be all software. There are lots of areas where “sort of alike” software is a recipe for disaster. Imagine a sales and marketing group where everyone had their own vibe-coded CRM tools, with different metrics and capabilities. This is not good. So there is still a place for big SaaS applications, distributed binary software, and distributed source code. You pick the hat to fit the head. But there’s also a need for something new.

The purpose of this article is to start a conversation about exactly what it is that developers should be preserving, sharing, and building on from others, and what mechanisms are still missing to make that virtuous circle work more effectively. I can’t see into the future; I can only speculate. So, as is often the case, I take a historical view. The poet W. S. Merwin wrote, “We are the echo of the future.” I have found that using history to frame the present can be a productive way to get our minds going in the right direction.

Note that while this article focuses on software development, the questions I’m asking here apply to virtually every field. Let me know if you have insights from your work in any area touched by AI.

From code to context

The ease with which LLMs can generate code is paradoxically getting people to focus too much on how easy it is not to produce code. That may be blinding us to the fact that code itself is being displaced as a valuable, tradeable, shareable knowledge artifact. Maybe “displaced” is too strong a word. Different species of value overlap for a long time. Eventually, though, old ones fade and new ones become dominant. This has happened before. Every technology revolution draws not only on new mechanisms but also creates new valuable knowledge artifacts. I talked about this in my recent short keynote at AGNTCon + MCPCon.

The open source revolution depended in part on mechanisms like modular design and source code control, even more than it depended on free software licenses. It birthed new coordinating institutions like GitHub. But it also created a valuable new class of knowledge artifact in the form of the source code repo, and subsidiary valuable artifacts like commit history and reputation. A developer’s public repos became a kind of résumé, and a project’s commit log became a record of who had done what, and how well. This replaced a previous world where binaries were the tradeable objects of the software economy. Microsoft’s fortune was built on selling licenses to binaries, and to anyone who measured the software economy in licenses sold, free software looked like value being destroyed. (Steve Ballmer called GPL’d free software “a cancer” and Microsoft VP Jim Allchin called it “an intellectual property destroyer.”)

There were two ways that I like to think about the effects of open source software. First, value that was based on scarcity was replaced by value that was created by abundance. Back in 1975, Steve Baer described this as “the clothesline paradox,” explaining that when you put your clothes in the dryer, the energy you use is measured and counted, but when you hang your clothes on the line, it is ignored by accounting, despite doing the same job. Open source software was like sunshine, free for the taking, and its value unmeasured. Everyone compared the revenue of free software companies like Red Hat and MySQL to the vastly larger revenues of proprietary software companies like Microsoft and Oracle, noted the fact that they hadn’t even survived as independent companies, and concluded that open source was a less effective business model because its creators captured less of the value. They missed the point that far more value had been created than had been captured. Open source went on to power most of the internet economy, with the value captured by other means than selling the software itself.

Some people might think of this as a kind of failure, but they would be wrong. It is the system working exactly as its creators intended. I remember talking with Bob Scheifler, one of the creators of the X Window System, about whether it was OK that we were building commercial versions of their documentation. He told me that was exactly what he had intended. They put things out with a very permissive license precisely because they wanted others to build on top of them. I remember him saying “We want to raise the floor.” The creators of Berkeley Unix, Apache, and others felt exactly the same way. They didn’t want to win the existing game. They wanted to play a different game.

And they succeeded, big time. As Clayton Christensen noted in 2004, when one thing becomes a commodity, modular and fungible, something else (usually something adjacent) becomes valuable. He called this “The Law of Conservation of Attractive Profits.” At about the same time, I had stumbled onto this principle in explaining the implications of open source software in giving rise to the new paradigm that I was shortly to start calling Web 2.0. Clay and I first bonded when we gave our respective keynotes about essentially the same thing at the 2004 Open Source Business Conference. I later wrote about the conjunction of our respective ideas in a piece called “The Open Source Paradigm Shift.”

New artifacts are often the first visible sign of this value transition. You can see this at work in the evolution of the web. The web was born from the new mechanism of internet-enabled hypertext via HTTP, HTML, and the modular, replaceable software (the web browser and the web server) that made it useful.

At first, the new valuable knowledge artifact was the website with its domain name, associated URLs, and HTML-formatted web pages. DNS was the coordinating institution. People paid fortunes for domain names on the premise that the site was the thing that mattered. And it did matter, and still does. But less than it used to. Over time, Google became the new coordinating institution, and its ability to point deep within websites made the web page, not just the site itself, a valuable knowledge artifact. Social media made “posts” and short-form video into other knowledge artifacts. New kinds of value creation and value capture tumbled out of that unbundling. Packaged software was displaced as the key monetizable artifact and replaced by web applications, which were information- and software-based frontends to vast backend databases.

The click became a new and incredibly valuable knowledge artifact, the unit in which attention and intent were measured, and then priced. First pay per click advertising (Google), then engagement maxing (social media) became the basis of vast fortunes. The people who notice the new artifacts early, while everyone else is still focused on the old ones, then invent new business models based on them, often play a bigger game than those who were still trying to play by the new rules.

When I summed up these ideas in “What Is Web 2.0” in 2005, I talked about the key principles being “harnessing collective intelligence,” “software above the level of a single device,” and “data as the next Intel Inside.” (That is, the next valuable source of lock-in.) The companies that survived the dot-com bust were the ones that understood that the data their users generated through their activity on the internet was the asset that mattered, not just the software that served it. That was an argument about a new knowledge artifact, though I didn’t use those words at the time. PageRank understood that a link was a vote, and a new mechanism to improve organic search birthed new dimensions of the internet economy. Links were bought, sold, and gamed accordingly. SEO (both “white hat” and “black hat”) and search quality became a battlefield.

All those disciplines have had to change now that content has been absorbed into AI models. Books, music, video, web content, sites like Wikipedia, GitHub, and Stack Overflow were all grist for the mill. In the open source era, a voted-on answer to a well-posed question on Stack Overflow was a valuable artifact, and the reputation that came with writing good ones was a subsidiary artifact, just as it was on GitHub. The number of new questions asked on Stack Overflow peaked at roughly 200,000 a month around 2014 and was already sliding before ChatGPT, but the decline accelerated sharply after November 2022. The knowledge in those answers didn’t go away. It was absorbed into the models, which now give their own answers synthesized from all that they have consumed. In that way, they have also consumed much of the traffic that would have gone to the original knowledge creators. New mechanisms to create a circulatory economy of value creation need to be developed.

Software development also changed radically because of the internet. Another of my web 2.0 principles was what I called “the perpetual beta.” It’s easy to forget that in the PC era, software had major releases only every couple of years, with the new version distributed on CDs (and long before that, on 9-track magnetic tape.) I remember talking with Mark Lucovsky, who’d been a developer on Windows NT at Microsoft, shortly after he arrived at Google in 2005. He was exulting at the experience of being able to push a single change live worldwide soon after it had been tested. Eventually, we saw the formalization of new mechanisms like test-driven development (TDD) and continuous integration and continuous delivery/deployment (CI/CD), which we now take for granted, as if they’d always been the norm.

The story isn’t over yet, though. The old knowledge artifacts don’t go away all at once, and they don’t go down without fighting. The new artifacts are only half-formed, and have weaknesses that can be exploited. We’re in the middle of that fight right now. But I suspect the winners will be those who first understand what the sources of deep value will be five years from now.

When code is cheap, what becomes valuable?

When code is cheap to generate, the valuable thing may be the prompts, context, and other artifacts that tell the model what code to generate and how to do it well. I’ve started using the generic term “context bundles” to describe the artifacts that you hand the model to do its work.

In the opening to this piece, I talked about software patterns extracted from code as one kind of new shareable knowledge artifact. But these are only one of many kinds of “context bundles” that are starting to be shared. Spec-driven programming is all about developing context bundles. The spec becomes the thing that is reviewed, versioned, and kept, and the code is regenerated from it as needed. (A lot of people are worried that this might become a return to waterfall development. And it could, if done badly. But done well, with rapid turns? Maybe just right.)

An Agent Skill is another kind of context bundle, a folder of instructions, scripts, and reference material that an agent loads only when a task calls for it. Anthropic released the format as an open standard, and it has been adopted by a growing number of other agent products. An MCP server is yet another mechanism for presenting context that will shape the work of an LLM.

But that’s not all. The agent trace, the record of what an agent tried, which tools it called, and which context it drew on, is the closest thing we have to commit history for work that doesn’t get formalized as code. The log of which context bundles were invoked, and by whom, could also be the raw material for attribution and payment, much as the click was the raw material for advertising. And the stream of what people ask their agents to do may become to this era what the click was to the last, the signal of intent around which the coordinating institutions of the AI economy get built. I’ve often wondered what the agentic equivalent to PageRank will be. Will agents leave “pheromone trails” that grow stronger or fade over time, in the way that links do?

Even a well-formed question or conjecture could be a kind of context bundle. Math and science already value and exchange these, and I’ve started to think that there is an opportunity for building infrastructure for sharing and crediting them. After all, arXiv.org has already pushed the boundary for credit further upstream by publishing preprints rather than just reviewed papers. Why not go all the way to giving priority credit for well-formed conjectures?

The eval, a set of test cases that encodes what good work looks like for a particular task, may also turn out to be one of the most valuable things a domain expert can create, withhold, or share. But there’s a serpent in the Garden of Eden. At Foo Camp, Nick Vincent observed to me that there’s a real risk to evals as a context object, because there is a race going on to acquire massive amounts of data (say on corporate workflows) and sell it to the labs to train on it. But an eval only works if the model being judged hasn’t already seen the answers. Once the test cases leak into the training data, the eval measures what the model memorized rather than what it can do. How do we protect the holdout set for evals? That may be a critical mechanism that is still needed, and one that the people who build evals, and the companies whose workflows they describe, have every reason to develop. IMO, it would be good if there were more transparency into this process, and perhaps even supervision by a neutral body.

How do we come to trust new artifacts?

Part of the work of every revolution is building the mechanisms that help market participants to trust its new artifacts. In his book Nature’s Metropolis, William Cronon tells how Chicago became the center of the world’s grain trade in the 1850s. The mechanisms that enabled this were the railroad, the steam-powered grain elevator, and the telegraph. The coordinating institution became the Chicago Board of Trade. In 1856 the board adopted uniform grading standards. Before that, a farmer loaded his grain into individual sacks, and every buyer along the way had to inspect each lot or trust the seller. Afterward, a load of wheat was inspected by a middleman, dumped into a bin with other grain of the same grade, and its owner was handed a receipt for, say, a thousand bushels of No. 2 spring wheat. The grading standards became a new mechanism enabling trust and the grade receipt the new knowledge artifact certifying it. Together they separated the ownership of grain from the grain itself. Once merchants started trading receipts for grain, it didn’t take long before they were trading promises of grain not yet delivered, and the futures market was born. (That raises an intriguing question: What will be the derivatives of the AI era?)

Cronon’s history also observes that the receipts were only as good as the warehousemen who issued them. Some of them issued receipts backed by no grain at all, or spread rumors that stored grain was spoiling so they could buy receipts cheaply. It took years of fights inside the Board of Trade, and finally state regulation that the Supreme Court upheld in Munn v. Illinois, to make the receipts trustworthy. New knowledge artifacts invite counterfeits.

Even when there’s no fraud involved, there are often missteps in the early days of a new technology. Consider the rise of spreadsheet models that could be filled in with the user’s own data. A model was a set of assumptions and formulas that let anyone ask “what if?” and get an answer in seconds. Magic at the time! And eventually completely taken for granted. The model became an artifact that could be passed around, and used to make business decisions at companies other than the one that had created it. In 1984, a few years into the spreadsheet revolution, Steven Levy wrote “A Spreadsheet Way of Knowledge” about what VisiCalc and Lotus 1-2-3 were doing to American business. Levy warned that a spreadsheet model is only as good as its assumptions. Back then, people taking a poorly-executed spreadsheet model as gospel was the equivalent of some of today’s AI usage fails.

Feeling our way into the future

When people stop valuing the thing they used to value, ask what they are coming to value instead. Being alert to valuable new knowledge artifacts is a superpower. The new artifacts tend to show up well before the institutions that organize them, and long before anyone has worked out how to measure them. GitHub was founded more than a decade after open source software became mainstream, and Google didn’t become dominant till more than a decade after the first websites. My bet is that we are at a similar moment with context, with the artifacts starting to appear and the institutions that will coordinate them still unbuilt.

There is a lot of work that I’m following eagerly, looking to see where, whether, and how it gains traction.

• Duncan Davidson is doing fascinating work on what it would mean to have a home for personal context. I can’t wait till he makes more of this work public and reusable. Relatedly, the open memory protocol initiative is an attempt to define a useful set of mechanisms for exchanging context.
• Sam Schillace has started building a repository of what he calls agent-building patterns. One of those patterns, “Anchor on Canonical Names,” echoes the idea that I wrote about last year in “Magic Words,” and that we are also exploring with the O’Reilly Expert Intelligence MCP. As Sam puts it, canonical names are a way of referencing “a whole coherent methodology the model already learned in training, far more reliably than a paragraph paraphrasing the same idea.”
• There are numerous attempts to reinvent GitHub for the agentic era, including Cursor Origin and Entire, founded by former GitHub CEO Thomas Dohmke. Entire describes itself as “new distributed Git forge that stores your intent alongside your code, built for the scale and throughput of agentic coding.” Instead of traditional pull requests, it uses Trails, an agent-driven review system that logs session traces and automated evaluations so developers can merge code without reading every single line of a diff. Origin is an agent-first Git hosting and collaboration platform designed around structured semantic diffs, machine-readable merge states, and automated agent routing. As commercial products, both of these are examples of existing software distribution mechanisms and artifacts.
• Perhaps most interestingly for the point I’m exploring in this piece, Anthropic is building a proprietary, internal-only replacement for Git itself for its own use. The company published a very detailed description of the system, which it calls Scion. It might even be detailed enough for someone with a big team and a ton of compute to reproduce, using the description as a starting point. But you can see at the scale of what they have built the limits of the “distribute context and specs and rebuild on the fly” model.
• But there are also attempts to build systems that enable developers to share lightweight pieces in a new architecture of participation. Common Fabric is building an entire system premised on disposable agent-generated software expressing repeatable patterns, using data that is associated with users rather than with applications and that carries with it its own rules for how it can be used. In my conversation with cofounder and CEO Alex Komoroske, he described these patterns as small pieces of code that can be fetched from anywhere on the internet or written by an LLM on the fly. Agents will mostly write them, not human developers. But they will mostly write them on users’ behalf, and once someone’s agent has built a pattern, anyone else can use it. Their solution may not be the winning one, but it is definitely a bold leap into what may be a very different future for software.

Don’t get trapped in the past of software development. Root yourself in its history, the soil from which it grows, and imagine its future. What will we want to share, what will we want to reuse, and what mechanisms are needed for us to do so effectively?

When a new coding model comes out, I want to try it. New releases always carry a promise for me. Maybe it can handle a change that needed too much manual intervention before. Maybe it can find the problem an earlier model kept missing. Maybe it can handle some coding scenarios that my favorite model can’t handle today. To discover a new model’s capabilities, I need to spend time with it to find out what I can reasonably hand over to it.

Having said all that, I understand the frustration when a new model becomes available, but your assistant won’t let you choose that model. Somebody has decided which models you can use and when you can use them. You are left trying to decide whether the answer is good enough and can’t get your hands on the new shiny toys. At the same time, I’m quite happy to let an assistant choose a model if that makes my budget last longer. New toys tend to be expensive and top-line models don’t make a difference here. There are plenty of steps in development where a cheaper model might do everything required. Using it for those steps could leave more of the budget available for a difficult investigation later. If routing lets me keep using capable assistance throughout the day, I have a good reason to use it. I want routing to give me more assistance for the same budget. The quality of that assistance depends on how the models, tools, and context handling fit together. And I need enough information about what the assistant did to judge the result.

Getting more help from the same budget

A fixed AI budget puts a fairly ordinary constraint on a team. Or better, on the affected individual.  You have an amount you can spend, and of course you want as much help as possible from it. Each request uses some of that amount. And it really gets frustrating when repeating a request, correcting an answer, and asking the assistant to try again consume even more of the budget.

Automatic routing gives the system a way to manage that spending for you. It can use a cheaper model where that model is likely to succeed and spend more where the additional capability is likely to help. You, the developer, can then concentrate on the task while the assistant handles some of these decisions and you don’t have to constantly worry about which model to pick for which task.

The researchers behind RouteLLM trained routers to choose between a stronger model and a cheaper one using preference data. They reported more than a twofold reduction in cost with minimal impact on response quality on the benchmarks they evaluated. You still need to check how it performs on your own tasks.

The next question is how those savings get to me in my day-to-day jobs. If requests become cheaper to serve, the same subscription could support more of them. A team could keep its assistant available to more people within an existing spending limit. The saving could also make it practical to use a more capable model more often when the task calls for it. So much for the theory. In most assistants that behavior is little more than a “switch.” That connection to better results needs to be visible in the product. If I choose automatic routing, I want to understand what I get from it. Perhaps I can send more requests before I reach a limit. The results still need to meet my expectations, of course. This is a fairly reasonable thing to ask of a feature that spends part of the budget on my behalf.

The time we spend with the assistant belongs in that calculation too. Think about how a cheaper request can lead to another attempt, followed by another explanation, followed by a correction. If this happens often enough, I spend more time getting the code ready to accept than I would have with a more capable model. So when we compare routing policies, we need to follow the task through those interactions. Then we can look at both the money spent and the time we put into getting there and compare both together.

Consider a Java service that sends an order to another service. Occasionally, the request times out, the caller retries, and two orders appear. You ask the assistant to investigate. It can begin by finding the retry code and following its callers, which may be manageable with a cheaper model. As it reads further, it finds a possibility that needs more investigation: The receiving service may have saved the first order before the response was lost. The caller sees a timeout, but that doesn’t tell it whether the order was created.

The assistant now needs to inspect how the receiving service handles a repeated request. A proposed fix should be checked against the failure we are concerned about: save the order, lose the response, and repeat the request without creating a second order. If the assistant can only inspect the caller, it should leave the receiving service’s behavior as an open question. Spending more on a model may help with the reasoning once the relevant code is available, but the assistant still needs access to that code or evidence of its behavior.

The task only becomes clearer as the assistant reads more code. That gives the router more information than it had when it saw the original prompt. It may make sense to spend more on the next part of the investigation, even if the initial steps were straightforward. A recent SWE-Router paper takes this approach: A cheaper model makes a short initial attempt. The router then uses what happened to decide whether to continue with that model or switch to a stronger one. Its benchmark results show that these early actions can improve the routing decision. That seems like a sensible direction for coding assistants: Let what the agent discovers influence how it continues.

The model operates inside a harness

Following the retry through the code also shows how much of this depends on the harness, the software surrounding the model. That’s not only the client running on your machine; it includes the router and backend logic. They provide context and tools, manage the conversation, and control how the agent proceeds.

The assistant needs to find the code that sends the order. It also needs to find out how the receiving service handles a request it has already processed. Once the model has that information, it can reason about the interaction. The harness has to help the agent get there.

This includes some really practical details. Can the agent search the relevant repository? Does it retain the information it found as the conversation gets longer? Can it run a test against the proposed change? If that test fails, does the failure reach the model in a form it can act on? These details affect both the result and the cost of reaching it. An assistant that loses relevant context may need to find the same information again. We pay for those attempts, and we may need to explain the problem again too. When the assistant finds the right code and runs an appropriate test, the result is way easier to review.

A model benchmark tells us how a model performed under particular conditions. The assistant we use has its own instructions and tools. It decides what information to provide to the model and what to do with the response. We experience all of that when we ask it to do something. A new model can improve that experience considerably. I want to use it with a harness that can find the relevant code, retain context, and run the tools the task needs. I also want the people integrating it into my assistant to check how it behaves with their harness.

The same applies when a company runs an assistant on its own infrastructure. It might connect to a model through an approved cloud end point or use a model running on its own hardware. The developer may see the same interface in both cases. Behind that interface, the model may have different tools or different access to the code. We made this case in our O’Reilly book Applied AI for Enterprise Java Development.

Returning to the order example: Suppose the assistant proposes another retry without examining whether the first order was created. The model may have struggled with the problem. It may also have been missing information about the receiving service. Looking at which files it examined and what it checked helps us investigate the failure.

This is exactly the kind of information I would like my assistant to make easy for me to find. It gives me somewhere to start when the result is disappointing. I can try another model, and I can also check whether the assistant found the information it needed. I believe I am less helpless in front of an already very magic system.

But what can a developer judge?

We have a reasonable chance of judging an assistant’s answer when we know the system and can examine what it has done. In a familiar service, I can follow the changed behavior and read the tests. I can then decide whether the change addresses the problem. With an unfamiliar service or programming language, I need more time to understand the code before I can make that decision. The assistant’s explanation can help me get there. It can point to the relevant code and explain how the change affects it. If it says that repeating the order request is safe, I can follow that claim into the receiving service. I need to see how that service handles a request it has already processed.

A clear explanation makes a proposal easier to understand. Checking the proposal takes another step. With that, I have seen confidence develop faster than that understanding. And I always wondered why. In an earlier study of an AI coding assistant, participants with access to an assistant produced less secure code while being more likely to believe their code was secure. The experiment used an older model and specific security tasks, so was not a general-purpose study, but even with that limitation, it shows why confidence about a change deserves attention. Assistant interfaces should show which claims have been checked and which still depend on an assumption. They should also keep an unresolved question visible as the conversation continues. “Review the result carefully” becomes easier to act on when the assistant helps me find the parts that really need review.

Deciding whether the router chose well takes a little more effort. Spotting a mistake tells me something about the answer in front of me. To find out whether another model would have done better, I need a reasonably fair comparison. It should start with similar code, context, and tools. If I change several of those at once, it becomes hard to tell what made the difference. This leaves me in an awkward position: I am responsible for accepting a result, while some of the decisions that shaped it are effectively hidden from me. A model name gives me little more than a reference. After using a model for a while, you develop expectations about what it handles well and when another attempt is likely to be necessary. Those expectations come from your own tasks and experience. If the service does not expose a precise model version, you cannot be sure that the model behind the same name has remained unchanged.

Despite all those drawbacks, access to a new model gives me something to try when my assistant is stuck. So I really can understand why people want that choice. It’s one of the few concrete ways to influence what happens next.

And of course the choice itself comes with expectations. But is this a real effect? In a study of AI model choice, people planning a holiday chose between a model presented as more capable and another as more energy efficient. Both choices routed to the same model, GPT-4o mini. People who chose the supposedly stronger option rated its responses more highly. They selected their own option, though, and had separate conversations. Some of the difference in ratings could have come from the people taking part or from those conversations. A second study gave 48 people functionally identical chatbots labeled “old” and “new.” The researchers varied which appearance received each label and which version people tried first. Thirty-seven preferred the “new” version. The suggestion that one version was newer was enough to influence their preference. Having some control may also change how people feel about a system. In experiments with social media feeds, people reported greater satisfaction when controls were available, even when the controls changed the feed randomly. Later, researchers told participants in the interview study that one feed behaved that way. Most could then identify it. Once they knew what to examine, they could recognize the difference.

These three studies involved general users doing tasks outside software development. We still need to understand how those findings carry over into coding. But for me, the studies suggest that the model name and the choices we make influence our impression of an answer and its quality. But we are fighting the wrong fight here. The real value would be in giving developers a way to check the answer.

The minimum control I want in a coding assistant could be quite modest. Let me ask it to spend more of the budget on a difficult investigation. Let me request a more capable model when repeated attempts are going nowhere. The harness can use that request together with what it has already learned about the task. I want to be able to leave routine routing decisions to the system, and still intervene when I have a reason. Each investigation gives me an opportunity to learn what the assistant handles well. I need to see enough of what happened to make that judgment, and the conversation is a good place to start.

Make the conversation easier to assess

When I look back at a past conversation, I want to understand what the assistant had available, what it did, and what it checked. That gives me a way to decide whether to accept a change or investigate further. The interface could help by showing more of what happens around the model, but it does not necessarily have to.

At a minimum, I need a context breakdown. The messages I type are part of the information the model receives. The harness also supplies system instructions, descriptions of tools, and guidance such as project rules and skills. All of that takes up visual space, but it shows how much space each part uses and how much is reserved for the response so I can act more appropriately in my next steps in the conversation.

Ultimately this makes developer mistakes visible earlier. When you realize that, even a short exchange can involve quite a lot of information. If tool descriptions take a large share, I can look at which integrations the task needs. Perhaps the assistant is configured for several kinds of development MCP servers, while a simple task only needs a few of those tools. I can try a more focused setup and check how it handles a similar task. It still needs access to the information and tools that help it finish the task.

In experiments I’m driving at IBM, I’m connecting this context information with a record of what the assistant did. I can see context changes alongside actions such as reading a file or loading a skill. From there, I can open the relevant part of the conversation. If the assistant delegated part of a task, I can follow that task as well. I have somewhere to begin looking when a long session becomes difficult to follow.

For example, the assistant might read the same file several times and receive identical content. The view can draw my attention to those reads and let me inspect what happened around them. Of course, the assistant may have had a reason to check the file again. I may also decide to ask it to summarize what it has found and continue from there. I can make that decision after looking at the messages and results behind the observation.

Alongside the activity history, I want to see the current context size, the session’s total token usage, and the cost of its requests. A context snapshot tells me how much space was occupied at a particular point. The usage recorded over a session tells me how many tokens its requests consumed, and cost tells me what I paid for those requests. These figures answer different questions, and I need to know which one I’m looking at. An estimate should be labeled as an estimate. If part of the history is missing, that gap should be visible too.

With the history available, I can also follow a claim back to what the assistant actually checked. Take the order example again: I want to see which code the assistant read and whether its test covered losing the response after the order was saved. If it only examined the caller, I still need to understand how the receiving service handles the repeated request. The conversation should keep that question open so I can provide the missing information or ask the assistant to investigate further.

A report that tests passed needs a little context as well. I want to know what those tests covered and which version of the code they ran against. If the assistant edited the code again afterward, I need a new check. And when it wrote both the implementation and the tests, both may follow the same mistaken assumption. I still need to compare them with the behavior we wanted.

In my article on specification, I argued for agreeing on enough of that behavior to make the result checkable. The assistant can keep that agreement available as the conversation gets longer. When it says that a requirement is satisfied, it can point to the code and the check supporting that claim. If a question remains open, I want to see it before I accept the change. It should also be easy to tell what stage the assistant has reached. It might be proposing a change, reporting that it edited a file, or showing the result of a test run. Those are different points in the task. A clear account of what happened helps me decide what to do next.

In practice, most of this can stay fairly brief. Put the checks and open questions beside the proposed change, with links to the actions and results behind them. I can read the summary, then open the part that needs more attention. As I use the assistant on similar tasks, those checks give me a basis for deciding how much I can rely on it.

I also want to understand how routing spends the budget. I already struggle to see token usage in some interfaces. Showing me how many tokens remain in my budget helps me decide how much more I can do. When one session consumes a large part of it, I want to understand where it went. Showing the recorded usage alongside the assistant’s actions would help me connect the spending to what it attempted.

A team can use this information when choosing a routing policy. Try the assistant on tasks that resemble everyday development. Keep the starting conditions similar and follow each attempt through to a result the team can review. Include the corrections and the time developers spent getting there. Then compare how much assistance the same budget provided and whether the resulting code met the team’s expectations. That comparison may also point to something in the harness that needs attention. An assistant that repeatedly retrieves the same information might need better context handling. One that lacks access to a relevant service might need a different set of tools. Those are things we can investigate once we can see what happened during the task.

Automatic model routing can be great when it gives developers more access to capable assistance for the money they already spend. I’m happy to let the assistant choose models for routine steps and spend more when a difficult investigation calls for it. The whole assistant needs to support that arrangement. Its models, tools, and context handling all contribute to the result.

As the industry develops these systems, we need to keep finding better ways to help developers judge the answers they receive. I want to be able to look at a conversation, understand what happened, and decide what to do next. That is how I can build trust in the assistant while letting it take on more of these decisions.

This event summary was created by Claude with editing by Tim O’Reilly, based on a transcript of the conversation.

Alex Komoroske is the CEO and a cofounder of Common Fabric, a public benefit corporation building the infrastructure for human-centric agentic computing. He spent more than a decade at Google as a product manager and director of product management on Chrome, Google Maps, and Google Earth, and he later ran corporate strategy at Stripe. At O’Reilly’s AI Codecon a year earlier, in a talk called “Why Centralized AI Is Not Our Inevitable Future,” he argued that we were sleepwalking into the wrong future for AI and didn’t have to accept it. Since then he has coauthored the Resonant Computing Manifesto, which Tim signed.

On Tuesday, October 20, 2026, Tim O’Reilly hosted Alex for an hour-long conversation, “Can We Build More-Human Technology?” Common Fabric’s consumer product hasn’t yet been released, but Alex showed Tim the product privately the day before, and Tim said his conversations with Alex had left him “more optimistic about the future of the web and the AI future than I’ve been in a long time.” They talked about the underlying theory behind Common Fabric.

The Resonant Computing Manifesto

The Resonant Computing Manifesto lays out five principles for resonant technology. It is private, meaning you’re the steward of your own data and it’s used in accordance with your wishes. It is dedicated, with no conflicts of interest. It is plural, with no central entity holding an unnecessary amount of power. It is adaptable, able to do things its creators didn’t specifically build in. And it is prosocial, helping people create meaning together rather than living as “individual islands.”

According to Alex, the manifesto rests on a distinction between two kinds of experience.

A hollow thing is something that you like when you’re using it, and then, after you’re done using it, you’re left feeling regret. And a resonant thing is something that you like when you’re using it, and then afterwards you’re left feeling nourished. They are superficially very similar, but they’re fundamentally, deeply, structurally different.

Alex argued that modern tech, business, and politics are very good at delivering hollow experiences efficiently and poor at delivering resonant ones. The manifesto came from a group of people who still believe in the hacker ethic of the 1990s and early 2000s, he said. They believe that technology is a large positive force for humanity, but also that the industry has spent the last decade optimizing for engagement and extraction, which has undermined that positive impact. They think that the AI transition is a unique opportunity to reset the norms and the architecture to make something that works better for people than for extractive business models.

Privacy, bits, and the physics of trust

Tim started out by asking Alex to go a bit deeper on privacy. Privacy and trust are related in ways the privacy movement often doesn’t consider, Tim said. We don’t put bars on all of our windows. We trust most people not to break in because we built a society where trust is the norm. It’s held in place not just by penalties but by social rewards for people who follow norms. “People will think you’re a creep if you go look into somebody’s window,” Tim noted. Like bars on the window, hard privacy seems to Tim a bit like an admission that we’re giving up on positive social norms and penalties for rulebreakers (where needed). He wanted to know whether Alex meant “private” in the way that HIPAA makes your healthcare records private, where your data is made hard for anyone to see, including you.

Alex doesn’t mean that at all. He means privacy that makes your data more useful by putting it under your control. He gave a three-part answer about how that happens. The first leg of the stool is what Helen Nissenbaum, a professor of information science at Cornell Tech, calls contextual integrity. What she means by that is that your data is used in line with your intentions and expectations, with no use that harms you, and no nasty surprises. Alex said it’s the best formulation of privacy he knows. Tim agreed.

The second part of Alex’s answer was that there are differences between atoms and bits. Arguing from game theory, Alex suggested that trust emerges when you expect to deal with the same person or entity again. In the physical world, you might be seen peering into someone’s window and shamed as a result. But “in the world of bits,” because the peeping is invisible, there are fewer immediate consequences, and “you can have a whole bunch of creepy companies peering through your window.” And because your data becomes more useful when combined with other data, there are much larger financial incentives to misbehave. (Tim would still argue that we also need to establish the right set of social norms and laws for regulating this bad behavior.)

The third point Alex made is that data is viral. It can be copied at full fidelity, instantly, for almost nothing, so once it’s out of your sight, it’s out of your control. The industry has relied on three inadequate mechanisms to address this: 1) terms of service nobody reads; 2) permission dialogs, which Alex called “responsibility-laundering dialogs” (He argued they ask an ill-posed question, whether you trust another entity, open-ended and forever, with your data. “I trusted 23andMe with my genetic data 10 years ago, and I sure as hell don’t trust them now,” he noted.); and 3) silos, where each app sees one slice of you. Spotify knows what you listen to and Strava knows when you work out, but neither knows which songs you like during a workout.

Alex said he respects Google, where he worked for many years, but he argued that it has little incentive to build much on your data beyond keeping you from leaving. His point was that “our data is kind of sitting there, not really working for us, and also not really on our turf,” and that this is the result of technical decisions the industry made rather than a law of nature.

Data that carries its own rules

Tim asked what makes Common Fabric different. A lot of people have espoused ideals like those of the Resonant Computing Manifesto, though perhaps in different words, but not much has changed. Alex replied that resonance is hard to build today because users spend their time on the software creator’s turf, doing what the software creator wants. Unless that structure changes, engagement-maxxing keeps emerging. Common Fabric goes down to lower layers and does something very different with the very structure of data. Alex said Steven Levy, writing about the manifesto for WIRED, had described what Common Fabric is working on as “resonant AF.”

Conceptually, in Common Fabric, every piece of data carries a metadata sidecar wherever it goes. In that sidecar, there are two things. One is cryptographic provenance: who did the last computation, the fingerprint of the software that did it, and what its inputs were, traceable all the way back. Alex said the math is similar to blockchains, but without a globally consistent, financialized ledger. The other is policy, the conditions under which the data may be declassified, now or in the future.

An open source runtime, which Alex called “the core reaction chamber” of Common  Fabric, enforces those policies. It receives a proposed graph of computation, made of untrusted code running on specific data with specific policies. If the computation is legal, the runtime runs it, updates the provenance chain, and tells everybody. If not, it stops with a descriptive error, for example because a network request could leak something derived from confidential data. You can run the runtime on your own machine, but most people won’t, so Common Fabric will run it in the cloud in trusted execution environments. These are hardware-isolated environments where the host can’t see inside and can prove cryptographically that it’s running the runtime unmodified.

Tim raised a worry from the history of security software. Early systems such as Kerberos were so complicated to set up and maintain that few people used them. Would this be a master console of access controls so complicated that nobody ever touches it? Alex said most users will never know how it works. To them it will simply be “the best way of using AI in their lives,” a digital home where their data, their collaborators, and their agents come together.

The software in that home is made of what Common Fabric calls patterns, small chunks of untrusted code. Alex stressed that he means “untrusted” in the security sense, which is good, because rather than meaning “not trustworthy,” it means “you don’t have to trust them.” Patterns are written in TypeScript and look much like React. (When the team asked LLMs what framework felt most natural to them, the models said React.) A pattern can do six things:

1. It can do arbitrary computation.
2. It can produce derived data, which persists and is available to other patterns. 
3. It can produce UI for different contexts, such as dark mode or a preview chip, computed only when used. 
4. It can make network requests if policy allows. (LLM calls are technically network requests.) 
5. It can instantiate subpatterns fetched from anywhere on the internet, or written by an LLM on the fly. 
6. It can wish for more data.

Alex walked through a wish. A meal-planning pattern wishes it knew the user’s favorite foods. In one pattern, the system might express this wish by asking an LLM to generate a simple text box for the user to fill in. Another pattern might look for notes tagged “favorite foods” or for the ingredients in recipes the user made in the last six months. Another might read through the user’s Gmail with an LLM and distill what they like to eat. Because the emails are confidential, the result is confidential too.

That pattern could have been written by a stranger and could still run proactively on your data, Alex said, because Common Fabric is designed so that the pattern’s author never sees your data and the pattern can’t use it against your expectations. Anyone can write new patterns (kind of like skills) and Common Fabric will surface the most popular ones for various tasks.

Type checking for your data’s rules

Tim asked Alex to slow down on this point, that you can give untrusted software access to your data because the data, not the software, decides what may be done with it. Alex replied that users will rarely think about policies. Users would choose default policies that are vouched for by people and organizations whose judgment they trust. For instance, he said “I trust Simon Willison. And Simon Willison will say, ‘I trust that repo right there.’”

Alex gave as an example a pattern that pulls your Gmail into Common Fabric. It uses an OAuth flow to get a token from Google. Anyone who has that token can read and send your email indefinitely. So the token is tagged with a policy “secret to google.com,” which specifies that the token should never be logged or shown on screen or sent anywhere that isn’t attested to be running the runtime, not even the user’s own browser. It may cross the network only over a secure connection to the origin that minted it, and only in the authorization header. Any pattern that looks at the token, or at anything derived from it, absorbs those policies permanently. “You can look at the OAuth token, but then you can never talk to anybody else ever again, basically.”

Tim said this sounds like the virality of GPL licenses. Alex agreed, and said that Mitchell Baker, the Mozilla cofounder, had made the same point to him a few weeks earlier.

The term of art in information-flow control is “taint,” (a term Alex joked he dislikes because people look at him funny when he uses it). Data is tainted when it has touched something confidential and so inherits that thing’s restrictions. The aim is to keep as little data tainted as possible so that network requests can still happen safely later. Common Fabric’s compiler untangles a pattern’s data flows so that code which only passes a token along gets a handle to it rather than the token itself, and stays untainted. Alex compared this to Rust’s borrow checker, which refuses to compile code that could theoretically have a data race. Human programmers find it a nag, and a slog.

But you know who doesn’t get impatient about that? LLMs. They don’t care. They’ll say, “Okay, cool. I’ll just make this tweak.”

There’s something different at work in the world.

This is the heart of Alex’s answer to Tim’s question about the risk that complexity will make the system unusable. Common Fabric’s taint checker nags in the same way as Rust’s borrow checker, and LLMs grind through it. “If this software compiles, it’s safe structurally,” Alex said. “You don’t know it’s useful, but you do know it’s safe.” Because it’s safe, the system can run patterns for people proactively and speculatively, without their installing anything, and then share more widely the ones people engage with. Alex said that kind of distribution isn’t possible under the trust model normal software uses.

Crappy patterns and dangerous patterns

Tim noted that the word “pattern” comes from Christopher Alexander’s A Pattern Language, the book about why some buildings and places work and others don’t. Alex added that the manifesto quotes Alexander directly, and that he considers resonance to be Alexander’s “quality without a name.” Tim recommended the book to anyone who hasn’t read it. He recalled patterns such as Alexander’s observation that balconies and porches less than six feet deep are rarely used, and his question of why people sit on stoops in some neighborhoods and not in others. Then he asked, “Won’t people just start making shitty patterns?”

Alex said they will. Most videos on TikTok are bad, and social sifting still brings the good ones to the top. He gave a stylized example of a ranking function. To decide whether to show images at the top of search results for a query, you can check how many people in the last 90 days searched for “images of” that term compared with the term alone, and show images above a threshold. The authentic behavior of a small group of savvy users tells you about quality for everyone else.

“I’m not worried about crappy patterns,” Alex said. “I’m worried about dangerous patterns.” He’s thinking about patterns that would misuse your data. And that’s what Common Fabric is designed to prevent. His example was the draft box in a chat app. Everyone assumes what you type is visible only to you until you hit send, and then only to the people in the chat. Nothing in today’s software structurally guarantees that, and on some marketing sites, the support agent can see what you type before you send it. In Common Fabric, a default policy could make the draft structurally confidential until the Send button is pressed and then declassify it to only the chat’s members. No chat app assembled from this pattern could behave otherwise.

Tim noted there will be a long period of discovering corner cases once the system is out in the wild. Alex said two surfaces must coevolve, the open-ended set of patterns and the policies, which is why trust delegation matters. As people find patterns that slip past common default policies, the policy sets need to evolve to meet the threat. He said he isn’t promising perfect security, which he called an asymptotic ideal. His claim was that for the same security budget, users can reach further.

Today, in the physics of trust we use, a more secure system requires putting yourself in a smaller and smaller and smaller box. What if, instead, a more secure system puts you on a taller and taller platform, allowing you to reach much further?

People won’t come for the security model, he said. They’ll come because they can do much more with AI and with collaborators than other software allows, and the security model is the reason that isn’t creepy or dangerous.

Where’s all the new software?

Tim brought up a question he has written about previously. When a new technology arrives, at first people can see only the old one. When the web came along, Marc Andreessen talked about Netscape becoming the next Windows, and even open source developers talked about a Linux desktop and an open source Photoshop, while the real action was people making entirely new things with the web. With AI, Tim said, we hear about a world of infinite software, but he hasn’t seen many accounts of wonderful new things people have built. Mostly he hears about dark factories that help people get more productive at what they already did. People talk about the tooling but not what they built with it. Patterns, he said, gave him a taste of what infinite software might really be.

Alex said software today orbits corporations, when it should orbit people. He described a “nightmare” in which building software took excruciating, hypermeticulous detail and bickering with dozens of people, and then pointed out that this was how software was built nine months ago. Old software now inspires in him the kind of awe people feel at the pyramids, built without heavy machinery. “And yet where is all the new software?”

His answer was that infinite software built on the old physics of trust can’t produce the things that matter. On one vibe-coding platform, he found a dream-tracker app by an anonymous author that lets you log in with Google, store dreams, and mark some of them public. He started to enter a dream and stopped, because he had no idea whether the author had left everything public by mistake or was reselling it. The old trust model worked tolerably, he argued, because expensive software usually came from people who knew what they were doing, and big companies had lawyers telling them not to sell your data. When anyone can bring software into existence, even that level of trust goes away. Without new physics for distribution and trust, infinite software gives you either “cozy bits of software” from people you would trust alone in your house for an afternoon, or isolated islands that can’t touch sensitive data and “top out at microgames and art apps.”

Software is alchemy. It is our ability to extend our agency beyond ourselves and automate it. And so the idea that software is going to top out at these very low things is just ludicrous to me.

Reclaiming your data

Tim connected Alex’s point about trusting big companies to his own analysis of rising tide rents and robber baron rents, which he developed around the same time Cory Doctorow was writing about enshittification. Companies have strong incentives to treat users well while they’re acquiring them. Once growth flattens, the incentive shifts toward taking advantage of users, staying just good enough that they don’t leave. Part of what keeps people attached even as applications enshittify, Tim said, is that the apps hold so much of their data.

Alex agreed that reclaiming data is the right frame. The business model of the last decade, as he described it, was to build expensive software aimed at an average market, let user data accumulate, and then rent access back, sell derived insights, or hold the data hostage. In a world of infinite software, users can build what they want on their own turf, and he expects silos to become “greedy goblins” guarding the data. He drew a line between what silos add and what users put in. Insights a company derives are its own, but the bits a user contributed should come back out. Infinite software, he said, also tilts the cat-and-mouse game back toward users.

Tim asked how he would get 15 or 20 years of email out of Google. Alex gave Google credit for some of the best APIs, especially in Europe. Alex pointed to msgvault, an open source project that keeps a local archive of a Gmail history, so agent queries go to a local copy instead of rate-limited APIs. When services hand over data in deliberately annoying forms, such as a thousand-page PDF, an LLM can patiently restructure it. “They don’t get bored of a task the way you do.”

Who pays for the compute

Tim pushed on the economics. Services like Gmail are expensive to run, and users get them for very little. If he stores terabytes of email himself, he’s paying someone else, possibly more than Google charges, since Google had ways of subsidizing it. How does this work over the long haul if Common Fabric becomes a new paradigm?

Alex began by acknowledging the value these services have delivered. Free turn-by-turn navigation is now taken for granted, though about 15 years ago, he said, the main licensors of that data charged something like five dollars per user per month. Then he made a point he expected some people to find provocative.

If you aren’t paying for your own computation, then it’s not working for you; it’s working for somebody else.

Users would pay for their own compute by running the runtime on their own machines, or through hosts Alex called AI service providers. These would run the runtime in trusted execution environments, prove they can’t see inside, and charge for tokens, compute, and storage. If users could switch providers, margins would stay small. Alex acknowledged that consumers have long resisted paying for software, but argued that the nontrivial marginal cost of LLM tokens creates an opening, much as people pay a monthly cell phone bill because the service has become indispensable.

Tim suggested companies like Google could migrate many of their services into this world. Alex said there will be many opportunities, but incumbents are used to another business model. He recalled telling a friend with a startup that building on the Common Fabric was wildly premature and that his investors might not like it, because the point is that patterns use data without owning it.

Tim noted that today’s model doesn’t have to stay the model, and Alex said he expects new business models to emerge. Asked how he will get developers to write patterns, Alex said he doesn’t need to. Agents write them on users’ behalf, and once someone’s agent has ground through building a word search, anyone else can use that pattern.

Is surveillance capitalism inevitable?

An audience member, David, wrote that he loves Shoshana Zuboff’s The Age of Surveillance Capitalism and wondered how we came to accept as inevitable that companies take our behavioral data. He said he wasn’t sure technology could fix something technology didn’t cause.

Alex said he thinks technology did decide this but allowed that people may disagree. The same-origin policy, the browser rule that lets a script use only data from the site it came from, was introduced by Netscape in 1995 as a quick fix for a security problem that was discovered early on with JavaScript. As more APIs were added, the walls grew higher, and when native apps arrived on iOS and Android, the same model meant every app owned its own data. Alex said that decision could have been made differently. Surveillance capitalism may keep emerging, he said, but it isn’t inevitable, and technology applied at the right leverage point can lead to a different result.

Tim told David he thought Zuboff’s book had a fundamentally bleak view of the world. For a long time, he said, we had a fairly virtuous set of relationships with many of these companies, and then we created economic incentives that led them to treat us badly. He argued that economic redesign is needed alongside technical redesign. That redesign starts with eliminating the market’s requirement that companies always keep growing even when they no longer deliver additional value.

Alex brought up Eric Ries’s book Incorruptible, which he said he keeps a stack of near his desk. He described himself as a believer in markets and capitalism as an emergent force for good. He distilled the book’s thesis down to the idea that organizations that think long term naturally attend more to their externalities. Common Fabric is set up as a public benefit corporation, and Alex said Eric consulted on structuring it to keep a long-term perspective in perpetuity, advice that Eric is now sharing with the wider world through Incorruptible.

Tim’s argument: AI as a reset

AI is a reset the industry still hasn’t yet taken full advantage of, Tim noted. We’re still thinking in the surveillance-capitalism frame, he said, in the same way that Netscape assumed people would buy software and that whoever controlled the software would control the future. We assume a world of advertising, and a world where companies hold more data about us than we do about ourselves. But new AI native companies will find new business models. Tim called Common Fabric the most thoughtful take on this reset he had seen. He compared it to his first encounter with the web, when there were about 200 websites and O’Reilly put it in The Whole Internet User’s Guide & Catalog because, as he put it, “this is the future.”

Tim said he couldn’t promise that Common Fabric would be the winner, but it is the most promising new thing he’s seen in a long time. He recalled the early days of the internet, when it wasn’t clear whether Gopher, the World Wide Web, or AOL would prevail. The web won, in his reading, because it built an architecture of participation. It was easy to draw from and easy to add to, and its mechanisms kept evolving, so that the web of 2000 or 2005 looked very different from the web of 1991. Common Fabric comes with that same sense of open-ended possibility.

Open source underneath, a product on top

Alex said the underlying runtime and pattern system are open source, but on their own they feel like a tech demo, which is why Common Fabric is building a consumer product on top. That product is still in development and not yet public. But he expects strong network effects as more people join the open ecosystem. Tim said that when Alex showed him the product the day before, this idea made sense. He described his takeaway as agents building throwaway software that embodies a shared set of patterns, with data that carries from the beginning the scope of what can be shared with whom.

Alex said none of this would have been possible before AI could write software. He told the story of his cofounder, Berni Seefeld, who had been thinking about these ideas for decades. Over a dinner a decade ago, Alex asked how they would ever get people to write enough patterns to reach critical mass. Berni said that in the future, AI would write software. Alex replied, “Come on.” Berni gave the structural argument for why software engineering would be the strongest, fastest, and most invested-in vertical, and a few years ago told Alex the curve was happening and it was time to start. Not needing humans to write patterns, Alex said, lets us think much bigger and differently from how the industry has assumed software must work for the last 30 years.

Tim closed on that point. Because agents building software in Common Fabric do it according to principles that put people at the center, he said, the nature of the game changes.

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

Is AI making software development faster? Most people I talk to say yes. But how much? What value are you getting for your money? Are there other ways of using AI that have a better ROI?

This is part 2 of my series on assessing AI’s impact on software development. In part 1, we looked at why assessing impact is important, and what not to do. In this part, we’re looking at how to measure AI’s impact on delivery speed. In part 3, we’ll look at how to measure the unintended consequences of AI, before wrapping up the series in part 4 with an examination of business outcomes. Finally, an epilogue puts it all together with notes you can share with your CFO.

What we’re measuring

Delivery speed isn’t productivity. I’ll say it again, loud, for the folks in the back: Delivery speed isn’t productivity!

It’s a component of productivity, sure. Given two equivalent teams creating the same piece of software, the team that delivers first is the one that’s more productive.

But it’s not just about output. To my mind, software development productivity is value produced divided by money spent. It’s a strict definition but a useful one, because it gets to the heart of why we build software: to achieve some valuable purpose. Going faster doesn’t mean you’re going in the right direction.

We’ll consider business outcomes in part 4. For now, though, we’ll tackle a simpler question: Are your teams finishing their assigned work faster, regardless of whether that work is actually useful or not?

The core approach

First, the bad news: There’s no objective measure of software development speed. I can’t look at two teams and say, “This one is fast, and that one is slow.” That’s because none of the outputs teams create are standardized. We already talked in part 1 about how lines of code and PRs are bad metrics. Well, nothing else is standardized, either.

Features? Some are big, some are small. Bugs? Some are easy, some are hard. Stories? Velocity? Story points? Completely arbitrary. They have way more to do with what’s being requested than the speed of the team.

Okay, if they’re all different sizes, couldn’t we just use estimates? Surely if a team is constantly missing its estimates, it’s because it’s slow, right?

Not so fast! Software estimates are famously inaccurate. A team that always misses its estimates isn’t slow. It’s just overly optimistic. Teams are nearly always overly optimistic.¹

So we can’t count requests (features, bugs, stories, etc.) because they vary in size, and we can’t look at on-time delivery, because estimates are overly optimistic. Sounds like a whole lotta “can’t.” What can we do?

We can compare a team to itself. If a team’s estimates were perfectly consistent, and they went from taking 3x longer than estimated to 1.5x longer, then they’re twice as fast! Huzzah!

If only it were that simple. There are a few snags.²

Estimates aren’t consistent

Snag the first: Teams’ actual estimate ratios aren’t actually consistent. Some are bigger, some are smaller. Plot them on a graph, and they form an “S” curve, as with this data from a real company.³

If delivery speed improves, the curve will shift. Individual data points may be better or worse than the original approach, but you can see the effect in aggregate.⁴

People work differently

Second snag: Some people are pessimistic and estimate high. Some estimate low. Some people spend a lot of time educating others. Others work heads down as fast as possible. Some clean up messes. Others create messes.

If you do a blanket comparison of everyone to each other, you introduce a lot of noise. Noise is bad because it increases the amount of data you have to collect. To keep things clean, compare people (or teams) to themselves.

Estimates change when people know they’re using AI

Third snag: People’s estimates will change according to the work they plan to do. So if they think they’re going to use AI, and they think AI makes them faster, their estimates will be smaller. Huh. Tricky.

This one’s actually pretty easy: Collect the estimate before people know whether they’re going to use AI or not. Then randomly assign the task to be done with or without AI.

Process improvements muddy the waters

Fourth and final snag: Are your speed increases due to introducing AI? Or the other changes you made at the same time?

AI forces you to rethink your software development processes. Leaders I talk to are streamlining request workflows and putting business experts in closer touch with developers as part of their efforts to introduce AI.

But these are the exact changes that make software development more effective in general! How much of the improvement you’re seeing is from spending thousands of dollars on tokens, and how much is the result of people talking to each other more?

Rather than comparing a chunk of pre-AI data to a chunk of post-AI data, interleave “without AI” tasks and “with AI” tasks. This reduces the influence of other changes. Update your without-AI approach to include the same streamlined workflows as your with-AI approach.

The complete measurement approach

In other words, if you want to be rigorous about this, you need to conduct a randomized controlled trial. Specifically, a “randomized within-subject repeated-measures” trial, which reduces the number of samples you need to collect.⁵

It’s not as bad as it sounds! People still do their normal work.

Here’s what you do. First, establish a with-AI approach and a without-AI approach. (Or, more likely, “with a lot of expensive AI” and “with a sprinkling of cheap AI,” but I’ll say “with” and “without” for clarity.) Make sure they’re clearly defined and people have had a chance to practice both approaches.

The without-AI approach will probably be close to your current approach to software development, but if you’re streamlining your with-AI processes, be sure to incorporate those improvements into your without-AI approach too. Decide how you’re going to attribute rework, such as code review changes and bug fixes, back to the task that caused the rework.

Then give each person (or team) a virtual bag that’s half with-AI assignments and half without. For any given task, have people record their estimate first, then randomly pull an assignment from the bag.

The main data you’ll collect is estimated time spent, actual time spent, type of task, and spending. Decide in advance if you’re tracking calendar time (including interruptions) or effort (excluding interruptions) and how you’re going to handle parallel work. Think about what other useful information you might want, such as bug and incident counts, mental energy, excess lines of code, and so forth. We’ll talk about using that data to model unintended consequences in part 3.

You’ll need the help of a statistician to do the detailed experimental design and data analysis, but broadly speaking, each person’s with-AI data will be compared to their own without-AI data, then aggregated together to give you the final results.

Finally, remember that speed increases aren’t the same as productivity. Pay particular attention to how the tasks you’re measuring fit into the big picture. For example, if you see a 2x improvement in coding speed, but people spend 50% of their week on other work, their overall speed improvement is only 1.33x, not 2x.⁶

Is it worth it?

This is a lot more complicated than just counting PRs and calling it good. Is it worth it?

If this were just some run-of-the-mill process improvement exercise, no, it wouldn’t be worth it. It’s too much work, and the effect sizes have to be pretty large in order for the data collection to be practical.

But this isn’t some run-of-the-mill process improvement exercise. Productivity expectations are off the charts, and so are costs. As I talk to engineering leaders, I’m seeing order-of-magnitude differences in spending. Should you spend thousands per person per month on hands-off agentic coding? Or hundreds on analysis and fancy auto-complete?

It depends on the outcomes.

You can’t pick the right approach to AI without knowing what you’re getting for your money, and with so much money on the table, you can’t afford to half-ass this.

Remember, in a METR study, engineers overestimated AI’s speed benefits by nearly 50%. Without measurements, there’s no way to know what you’re actually getting.

Fortunately, it’s not as hard as it seems. With the level of impact you need to see, and enough participants, it doesn’t take many samples from each person.⁷ Your teams collect the data as they do their normal work. The hardest part is explicitly defining how you want people to use AI, and that’s worth doing anyway.

Of course, delivery speed is only part of the picture. There’s also the question of how sustainable speed improvements are. We’ll consider unintended consequences next, in part 3.

All my content is made with my 100% human meat brain, except where explicitly stated otherwise. If you liked this entry, check out my best writing and presentations, and consider subscribing to updates by email or RSS. To contact me about consulting, email me at jshore@jamesshore.com.

Footnotes

1. When they’re not, it’s usually because they’re playing political games with estimate padding. That isn’t good either, because then the work tends to expand to meet the estimate. ︎
2. This design was inspired by a METR study, with help from Brent Miller. AI Disclaimer: I used OpenAI’s GPT-5.6 Sol model operating in “High” thinking mode to critique my explanation, and I also used it to generate the Python programs that drew the graphs below. You can see my ChatGPT conversation here. ︎
3. This data is from the Star Citizen alpha 3.0 release, which publicized its internal task estimates and actuals. Interestingly, the sharp cliff at the 1.0 mark is a classic sign of work growing to meet its estimate. ︎
4. The Star Citizen data was split in half and artificially manipulated for the purpose of this example. It doesn’t represent real-world AI impact. ︎
5. Many thanks to Brent Miller for introducing me to this technique. ︎
6. Work that took 40 hours now takes 30. 40 ÷ 30 = 1.33. ︎
7. The exact number will depend on the number of participants, the amount of variance in your results, and how conclusive you want the results to be. ︎

Coauthored with Claude

Security looms large as we enter November. There’s been a lot of bad news: a certificate authority with woefully inadequate private keys, vulnerabilities have been discovered (and exploited) in PHP and other widely used programs, and several organizations have detected Chinese open weight models misbehaving in several different ways. New models are arriving faster and faster, and the best open-weight models are catching up to frontier model performance. But organizations that would like to achieve AI sovereignty will have to ask some difficult questions about the models they adopt. Is this an inflection point? Whether or not it is, AI has changed security permanently, and things are bound to get worse before they improve.

AI models

We are starting to see models developed with recursive self-improvement; we continue to see clones of Jev; and we’ve seen text watermarks added to output generated by Anthropic’s and OpenAI’s models. (Google has been using text watermarks for some time.) But tools for detecting the watermark are hard to find. Anthropic’s and OpenAI’s are in a very limited beta, and Google’s is only available in their enterprise package. What does a watermark mean if you can’t detect it? And, as we get more accustomed to AI-generated output, will anyone care?

• Kimi 3.1 Pro is a new open weight model with 3T tokens. On benchmarks, it is now the leading open weight model, only slightly behind the latest frontier models. Moonshot AI also released Kimi K3.1 Pro Highspeed, which is two to three times faster.
• Recursive has announced Flint, a 40B parameter mixture-of-experts model that activates 2.5B parameters per token. It was designed for coding, and sized to run on laptops. Flint is open weight; weights are available on Hugging Face, and it works with model runners like Ollama.
• Meta has shipped Muse Spark Open, an open weight model that they claim competes with the best Chinese open weight models. It’s a mixture-of-experts model with 816B parameters and 42B active parameters.
• Meta’s latest models, Muse Wonder and Muse Wonder Full, claim parity with frontier-class models. Specifically, they claim parity with Claude and ChatGPT for financial and legal work. The announcement hints that Meta will be using Muse to develop specialized models.
• OpenAI’s Decisions API is yet another Jev-like model. Like Jev, its input is a data structure that constrains the model’s output to be an answer to specific questions. The output is a data structure that includes a probability for each answer.
• Claude has launched Haiku 5.5, a new version of its least expensive model. It’s faster, less expensive, and has better benchmark scores than its predecessors. Haiku 5.5 is a very capable model, particularly for users who are sensitive to cost.
• Mistral has launched Mistral Large 4, a.k.a Le Chonk. It’s an open weight mixture-of-experts model with 1T parameters. It’s claimed to be the best open model from Europe and was designed with AI sovereignty in mind.
• Aleph Alpha has released an open weight LLM for English and German that was trained entirely under European rules. It is a mixture-of-experts model with 78B parameters. It uses a different tokenizing algorithm to handle long German words without breaking them into fragments. Reasoning takes place mostly in German.
• Crowdflare has released Clef and Clef Flash, new decision models that, like Jev, take typed queries and return typed answers about those queries. It’s also offering a new reinforcement learning service to help customers train Clef for their applications.
• After a long wait, Google has released two new frontier models, Gemini 4 Argon and Carbon. Argon is available to paying customers. Argon’s successor, Carbon, is being released slowly, and is currently only available via the Fairwind program.
• The Ember-1 model from Fireworks claims to have results similar to Kimi K3 at lower costs. It is priced similarly on OpenRouter but uses far fewer reasoning tokens.
• OpenAI has published 722 papers solving problems in mathematics on GitHub. There is some controversy about whether all the proofs are correct; so far, three have been withdrawn, leaving 719.
• Google has opened its SynthID detector to the general public. The detector can determine whether audio, image, or video files generated by any of Google’s partners (including OpenAI, NVIDIA, and Apple) have the SynthID watermark.
• To comply with EU law, OpenAI will add watermarks to text in Europe. The company’s algorithm sounds similar to the one that Google and Anthropic are using: modifying the underlying probability distribution changes word choice in detectable but insignificant ways.

Agents and harnesses

There are more choices than ever for agents, along with new standards for the interaction between agents and models. In addition to general purpose agents, there are also agents specialized for coding, HR, BI, and more. And most agents will connect to many models, including local ones. Choose agents as carefully as you choose models; your agent is where you will do your work.

• JetBrains has released Thinkrail, an agentic IDE that learns about your project as you use it. It is open source and based on the pi agent.
• Anthropic has donated its Agent Skills standard to the Agentic AI Foundation (AAIF), which will shepherd further development. It has also donated Open Responses, a standard for building common interfaces, to AI providers.
• OpenAI has introduced Intelligent UI, a feature that allows ChatGPT to generate its own user interfaces in response to the user’s question.
• Workday is allowing its customers to build their own agents for HR and finance. An agent’s access to Workday data is tied to the user’s account.
• Claude Dashboards is a new product that lets you build live dashboards from a group of data sources.
• Claude’s Google Workspace connectors allow it to read and write Google documents, sheets, and slides directly.
• MCP Events is an extension to the MCP protocol from OpenAI. It allows ChatGPT to subscribe to updates from MCP servers.
• DeepSeek has released a harness (dsh). It’s open source, and currently in public preview. Everything is a plug-in.
• The Pi agent has reached version 1.0.0, and is now capable of using MCP. There’s a variation, Pi Durable, that has been designed for long-running tasks.
• OpenAI dots are consumer-oriented agents that run in the cloud. They are always on and have their own web browser along with other tools.

Software development

At last month’s SciFoo, Hugo Bowne-Anderson said that we’ve been through two major changes in the past few years: from writing code to letting AI write the code, and from reading the code to not reading the code. These changes are as radical as the shift from programming in machine language to programming in high-level languages. We will all go through a process of rethinking how we work and what we’re trying to accomplish.

• Anthropic has introduced Claude Mods, a plug-in for Claude Code that can modify Code’s behavior: reshaping its user interface, providing information about context use, giving advice about what you’re doing, and much more.
• One application for decision models is building evals to understand the performance of AI applications.
• Workday is using Jev to build user interfaces on the fly. Jev’s very low latency makes choosing UI components at runtime viable.
• Anthropic has outgrown Git, which wasn’t able to keep up with merges on their monorepo (roughly 30,000 merges in September). Anthropic has built a new source control system called Scion.
• Kate Huston writes about reverse-engineering code review and trying to develop an effective “gate” that AI-generated code would have to pass before review by humans.
• WordCraft is an open source clean-room re-implementation of Microsoft Word. The authors claim that it’s 60% of the way towards “replacing Word for real work.” It supports MCP, has a command line interface for interacting with agents, and can run in a browser through WASM.
• ArtCraft is an open source tool for creating interactive AI-generated images and video in 2D or 3D. The company describes it as an “IDE for artists.” They’re also developing open source clones of Adobe Creative Suite tools like Photoshop. Early reports are that it’s very buggy.
• Google Docs now supports Markdown natively. Users can read, write, collaborate, and comment on Markdown documents without converting them to another format.
• Google’s Docsy documentation project is moving to the Linux Foundation. Docsy is a template for the Hugo static site generator that’s useful for generating documentation in ways that are helpful to agents.
• Does programming with AI require more abstractions or fewer? Matt Pocock argues for more.
• Anthropic has given Claude an auto-eval tool. While the first version of this tool has many flaws, it’s a significant step towards making evals easier.

Security

Security has been much in the news. Not all of the news is bad: AI was used to discover the PHP2Shell vulnerability and to develop a fix. The leaked CVE and the many out-of-date PHP sites online are serious problems, but not problems attributable to AI. Too many problems arise from getting the basics wrong: A certificate authority using 4 as its private key is a mistake that should never have been made. As always, security is about getting the basics right while watching for new threats. And while organizations claim to be paying more attention to security, the word from insiders is that “attention” means lip service. In the long run, AI may make the world a safer place, but the next few years look scary.

• CrowdStrike and Booz Allen have found that Qwen3-Coder (and probably other models) generates weaker code if the prompt says that the code is for a politically sensitive user such as the US government. Hirundo reports similar flaws and also claims to have “fixed” the model and created a Westernized version.
• Cohere makes a different claim: It has observed that Qwen3.8 models occasionally send data to an Alibaba cloud storage bucket in Beijing.
• Malicious AGENTS.md files have been found in the Downloads folders of some users. If the user is using the Downloads folder as their working directory, an AI agent will read the AGENTS file and download a malicious NPM package.
• 1Password has launched 1Password for Agents, a credential-brokering product that allows agents to access resources for a user without seeing the user’s password,
• Alibaba also donated its OpenSandbox project to AAIF. OpenSandbox is a general-purpose sandbox for running AI models safely.
• The PHP team has released patches for two critical vulnerabilities collectively known as PHP2Shell. However, many sites are running old versions of PHP that need to be upgraded before they can be patched. The maintainers have also released some partial mitigations for sites that can’t upgrade easily, but that won’t fully protect them. Sadly, “the panic hasn’t turned into patching,” and those who don’t patch will pay.
• Brotli isn’t as widely used as PHP, or as old, but a critical flaw in Brotli compression library makes it clear that discovery of hidden vulnerabilities is the shape of things to come. Microsoft has released patches for .Net, PHP maintainers have released patches for PHP, Node.js is also affected, and that’s just the start.
• The private keys of the certificate authority eMudhra have been cracked–not by brilliant cryptography or a quantum computer, but because the keys were generated very poorly. The Chrome and Firefox browsers aren’t trusting the compromised certificates, but almost everything else, including all major package managers, is a problem.
• OpenCode was the victim of a hostile attack in which some users’ API keys and passwords were stolen from their Console service. Its response was hampered by lack of access to AI models without security restrictions.
• The US Department of Homeland Security has banned the use of DeepSeek and applications developed using DeepSeek in all federal agencies.
• To monitor the risks created by the development of frontier models, Anthropic has given six employees of METR access to their models. They continue to work with Accenture and Secure Bio.
• Model abliteration is a poisoning attack in which post-training is used to discover which of a model’s weights corresponds to “no,” and altering them so the model cannot refuse a request. It’s effectively a backdoor into the model’s behavior.
• AWS has launched Strands Box, an open source sandbox that’s designed specifically to prevent AI agents from escaping the sandbox’s control. It includes an open source policy engine, Dogwood, that allows users to place explicit limits on what an agent may do.
• Meta’s Muse, its agent for people who aren’t technology professionals, is a privacy and security disaster.
• Using MCP for communications between agents creates many risks, including new vulnerabilities to prompt injection.
• Google has suspended submissions to the Open Source Software Vulnerability Rewards Program because of the flood of machine-generated vulnerability reports, most of which are (in Google’s words) “not valid.” This suspension should only be temporary.

Infrastructure and operations

There’s been a lot of news about certificate management, not all of it good. It’s time to check on your certificate providers, get ready for quantum-resistant encryption, and automate as much of the process as you can.

• To increase security, the certificate provider Let’s Encrypt will be limiting the lifetime of free certificates to 64 days starting in February 2027. The change should be invisible to sites using automatic certificate management (ACME).
• Crowdflare has announced that it will begin issuing TLS certificates with quantum-resistant encryption. They will continue issuing certificates with classical encryption. Crowdflare hasn’t given a date when they will start issuing the new certificates.
• DeepSeek has open sourced DeepInfer, its inference framework, with an Apache 2.0 license. At present, DeepInfer only supports DeepSeek-V4.1-Flash.
• RemoveMacAI is an open source command-line tool for deleting Apple’s AI from MacOS 27, freeing up 30 GB or so of disk space.

Web

Mark Twain didn’t quite say, “The reports of my death are greatly exaggerated.” But we are hearing that the Web is dead, and that’s indeed an exaggeration. The flood of crawler traffic and the disincentives towards human writing are a problem. But what does “death” mean for a technology so flexible that you can build an office suite that stores entire documents’ URLs?

• TinyLittleOffice.com is a surprisingly complete set of surprisingly capable office applications. It includes a word processor, spreadsheets, and other Office-like apps. Everything is saved in the URL–not on your disk, not in someone’s datacenter. Not suitable for large documents.
• Crowdflare has announced a web search API, now in beta. Although their announcement doesn’t say, we suspect this is an attempt to control web scraping traffic. Matthew Prince, its CEO, has estimated that bot traffic will reach 1000 times human traffic by mid-2027. All of his earlier predictions have been underestimates.
• Jev and its clones have taken the world by storm. Can you beat it at Pacman?

Hardware

• AI labs are heading towards models that can run on phones and similar hardware. Google has compressed Gemma 4 E4B to 5 GB, and Liquid has released LFM2.5-24B-A2B, a 24B mixture-of-experts model with 2B active parameters that also targets devices.
• OpenTPU is an open source API accelerator, designed by AI. Bring your own silicon.

Quantum computing

• Quantinuum has implemented quantum error correction on its latest hardware and has found that it scales well enough to be useful in practice: Adding a physical qubit to a logical cubit cuts the error rate significantly.
• Researchers have discovered a way to read a qubit without collapsing the information it contains into a 1 or 0. Solving this problem (also known as quantum non-demolition or QND) is critical to building quantum computers large enough to do useful work.

Biology

• Watermarking texts is one thing. Google has now designed a watermark for proteins. Like text watermarks, protein watermarks are based on Google’s SynthID.
• Last month we mentioned EEG-capable earbuds. This month, a group of researchers has announced that they have developed an AI application that can reconstruct what someone is seeing from brain scans. The results are impressive, though the privacy implications are chilling.

“Tush push” never appears in our roughly 265,000 play descriptions from 2021 through the current season. Neither does “brotherly shove,” and “sneak” turns up only twice. So when we asked our AI analyst to tackle “Ban the Tush Push” in the October 26 episode of The Monday Morning Agent, we first had to define the play using the available data. We saved that definition in a shared data dictionary, the agent’s first durable memory, so future runs must use the same definition and explain its limitations.

Check whether the data supports your definition

A quarterback takes the snap, and teammates push him forward. Our data identifies the ball carrier but doesn’t record whether teammates pushed him. We can check whether a snap came from shotgun, but we can’t reconstruct the team’s specific formation.

We explored five possible definitions, from a quarterback sneak with teammates pushing to any rush on third or fourth and short. Each definition changes which plays the analysis counts. The broadest one would count running backs too, which takes us away from the play people are arguing about.

So before you let an agent query a term everyone seems to understand, write its definition in plain language. Then go clause by clause and ask whether the data can support it. A familiar name doesn’t guarantee a usable label.

Write down what your definition can’t measure

We settled on a quarterback run with one yard or less to go, from under center. The agent counts qualifying quarterback sneaks whether anyone pushes the runner or not. We’re measuring a substitute for the tush push, and we can’t isolate the shove’s contribution.

Even “quarterback” needs a definition. The fields available to our tool don’t include player position, so the tool identifies the quarterback as the player who threw the most passes for that team in that game. That’s another assumption to check when you interpret the results.

The dictionary entry has to explain the play in plain language and state what the data can’t tell us. Otherwise, readers could assume a “tush push” result counts plays with a push. Keep the limitation beside the definition so the agent can repeat it with the result.

Give the next run the same definition

The dictionary holds one JSON file per definition, with a fixed structure. The agent can add an entry, but it can’t change an existing one. A different definition needs a new name. That keeps the meaning attached to an earlier result from quietly changing later.

Before each run, the script that launches the agent, called the runner, saves a copy of the dictionary and adds its contents to the agent’s instructions with the date. The agent searches for an existing definition first. If it finds one, it uses it. If it doesn’t, it can write a new entry using the fields the tool supports.

In the demonstration, the tool refused the request for “tush push” because no entry existed. The agent then created a definition and queried it.

Check that the agent uses its memory

Writing “use the dictionary” in a policy doesn’t make the agent obey. We’re building this around instruction, check, and enforcement. The policy tells the agent to use and cite an entry. The tool accepts definitions from the dictionary rather than a definition improvised in a query. If the answer lacks the required ruling, the runner retries once, then stops if it’s still missing.

We expanded the agent’s permissions to let it add a dictionary entry alongside querying the plays. Test whether your agent uses its shared definitions and whether someone reading the answer can find them.

Read the conversion rate before deciding on a ban

Across five complete seasons, the defined play converted 78.3% of the time, compared with 70% for other short-yardage rushes. That’s an 8.3 percentage point advantage.

While the agent found an advantage, it didn’t treat the conversion gap as sufficient evidence for a ban. Those numbers describe conversion rates for our defined group of plays. They don’t establish whether the NFL should ban a play, and they can’t tell us how much pushing helped.

A viewer raised that distinction during the discussion. Whether a successful play belongs in the game requires human judgment. Conversion rates can inform that conversation, but they can’t resolve questions about safety or what people want football to look like.

Next up: “Kickers Are Too Good Now”

On November 9, we’ll tackle “Kickers Are Too Good Now” and give the agent written visualization standards. An agent can generate a chart in seconds. We’ll work on rules for presenting trends and making comparisons so readers can see what the data actually supports.

O’Reilly members can sign up on the O’Reilly learning platform to join us live on November 9, 8:00am–8:45am PT / 11:00am–11:45am ET. Everyone can follow along on Radar every other Monday, with a brief pause for the holidays. Watch episode two above, and we’ll build on it in the next session.

The following article originally appeared on Addy Osmani’s Elevate blog and is being republished here with the author’s permission.

The way that I typically work is I have anywhere between five and ten agents working at the same time in parallel. There are going to be some tasks that I’m very happy to delegate fully to agents, as long as I have a very clear idea of the stopping conditions and the constraints around them. And then there are going to be some tasks where I am going to want to keep a closer eye on and code-review what the agent is doing.

You’ve probably heard about loop engineering. I talked about it a couple of months ago when I wrote a big blog post about it.

A loop is an autonomous, self-correcting feedback cycle where an AI agent repeatedly acts, tests its results, and adjusts its approach until a specific goal is met.

There are now basically two core primitives you can think about. In Claude Code you have a goal primitive, which can drive a single bounded task forward until you’ve got a particular goal, like a measurable finish line that’s been met. And then loop reruns on a timer or a fixed interval, so you can use it to schedule changes.

Before the primitives were primitives

I remember back before we had primitives baked into Claude Code and Codex: Loop engineering was heavily about setting up your own bash loop, a hand-rolled thing. That’s how I approached it. And you might remember earlier in the year, a number of us were playing around with the Ralph loop by Geoff Huntley. We were experimenting, we were sharing our workflows, we were sharing what worked and what didn’t, but it was largely on some of our personal projects where, if we ran into a wall, it didn’t really have a big cost to us.

And as we’ve tried to see what patterns and what aspects of loop engineering are now a little bit more baked, I think we have a clearer idea of how you babysit it. I can now largely rely on the output of the primitives in Claude Code and Codex. We’ve come a little way. But at the same time we need to be very diligent, because loop engineering, where you kind of leave it sitting alone and haven’t really thought about whether the end goal or the constraints have been well defined, can leave you in a problematic state. This is why there’s nuance when deciding to use it for an evergreen codebase without users or without as much historical complexity as a brownfield bank codebase.

How the Claude Code team frames loops

The Claude Code team published their take on four kinds of loops, and it lines up with how I use the primitives (their write-up). Before we get to it, here’s my quick summary:

On the Claude Code team, we define loops as agents repeating cycles of work until a stop condition is met. We categorize a few different types of loops based on how they are triggered, how they are stopped, what Claude Code primitive is used, what type of task is most appropriate for each. Not all tasks require complex loops; start with the simplest solution and use these patterns selectively.

Every prompt you send starts a manual loop with you directing each turn. Claude gathers context, takes action, checks its work, repeats if needed, and responds. We call this the agentic loop. For example, ask Claude to create a “Like” button. It reads your code, makes the edit, runs the tests, and hands back something it believes works. You then manually check the work, and write the next prompt.

Their write-up walks each rung.

On goal-based loops:

Sometimes, a single turn is not enough, especially for more complex tasks. Agents do better when they can iterate. You can extend how long Claude keeps iterating by defining what done looks like with /goal. When you define the success criteria, Claude doesn’t have to make a determination on what is “good enough” and end the loop early. Each time Claude tries to stop, an evaluator model checks your condition and sends it back to work until the goal is met or a number of turns you define is reached. This is why deterministic criteria, such as number of tests passed or clearing a certain score threshold, are so effective. For example: /goal Get the homepage Lighthouse score to 90 or above, stop after 5 tries.

On time-based loops:

Some agentic work is recurring: the task stays the same and only the inputs change. For example, summarizing Slack messages every morning. Other work depends on external systems, and a simple way to interface with one is to check it on an interval and react to what changed. For example a PR, which may receive code reviews or fail CI. For these, you can trigger when Claude runs with /loop, which reruns a prompt on an interval. For example: /loop 5m check my PR, address review comments, and fix failing CI. /loop runs on your computer, so if you turn it off, it stops. You can move the loop to the cloud by creating a routine with /schedule.

And on proactive loops, the top rung:

Triggered by an event or schedule, with no human in real time. Stop criteria: each task exits when its goal is met. The routine itself runs until you turn it off. Best used for recurring streams of well-defined work: bug reports, issue triage, migrations, dependency upgrades. Managed usage by routing routines to smaller, faster models and using the most capable model for judgment calls.

Their verification advice is worth lifting whole, because it moves the manual checking into something Claude applies itself:

---
name: verify-frontend-change
description: Verify any UI change end-to-end before declaring it done.
---
# Verifying frontend changes
Never report a UI change as complete based on a successful edit alone.
Verify it the way a human reviewer would:
1. Start the dev server and open the edited page in the browser.
2. Interact with the change directly. For a new control (button, input,
   toggle): click it, confirm the expected state change, and screenshot
   before/after.
3. Check the browser console: zero new errors or warnings.
4. Use the Chrome Devtools MCP, run a performance trace and audit
   Core Web Vitals.
If any step fails, fix the issue and rerun from step 1 - do not hand
back partially verified work.

Goal

The way that I use goal is for building any specific piece of work until it’s proveably done. For example, you can use a goal to say: “Make sure that this experience loads in five seconds, keep going until it’s done.” And it will continue to use an independent evaluation check to keep checking if the completion criteria has been met. What’s better is to be even more specific about what tooling is being used.

As for real goals that I’ve run to a number, there’s a few things that I’ve done. I’ve used goals for running through GitHub issues: “Review and close the last 10 issues,” or “Review and move the last 10 issues forward,” something like that. And that’s semi-open-ended, right? Or “Let’s make this page load 50% faster.” Sometimes that works well, sometimes it doesn’t, but it’s really about the experimentation.

Loop

Loop is a little bit more of a scheduler, so it keeps an eye on something or repeatedly executes a pattern on some cadence. Think of it a little bit like a cron. It’s best for doing things like polling logs or monitoring external states. You could use it for potentially checking in. If there are repetitive tasks you find yourself doing on some cadence, loop is pretty good for that.

What I delegate, and what I watch

I use probably between five and ten agents every day. Typically I’ll max out at about five concurrently. And some of those tasks might be ones that are a little bit safer. So maybe it’s: “Hey, I implemented this feature, go write the documentation for it.” Or “Go double check that we have sufficient test coverage.” Or something in that vein. If I’m working on a more complex problem, and I’ve given it a good spec, or what I think is a good spec, and tried to give it some stopping conditions, there’s still a reasonable chance it may not get everything right, so I would try to watch it more closely. If the task involves anything just a little bit sensitive, whether it is “I’ve given this access to a system,” or whether the feature happens to touch authentication, or something related to security or finance, I’ll definitely be watching that closely.

Generally speaking, I do think we’re going to get to a place where folks are increasingly comfortable with delegation, as long as they’re able to have these clear ways of verifying that their goal or their stopping condition has been met. But you do still need to take a look at the code, at the thing that’s being generated, to make sure that it’s meeting your mark.

The other habit that matters is not letting the agent that did the work decide the work is good. One subagent drafts the change; a separate one verifies it.

Sometimes an agent can be confident about something, and a verifying agent can catch things that are unexpected. If an agent thinks that the baseline performance of an experience it has generated is actually fine, but it’s only evaluating performance based on desktop, and you actually care about the experience on mobile, that might mean the agent is being very confident about one dimension of the problem but not the other.

And I learned this one the hard way. I was curious to figure out: Is there something that we’re missing that we haven’t gotten direct feedback on from our users, either in an issue tracker or in a comment somewhere? And so I asked it to go and take a look at some of our competitors and put together a list, and some PRs, not pushed, but some PRs locally, of what solving some of those gaps might look like. And I almost pushed some of those changes. But I didn’t actually look at them closely enough. I read through its research, but I didn’t look at the implementations closely enough. So I delegated the task, but I was close to delegating the judgment as well. When I actually looked through the changes, what I realized is that it would introduce a lot of additional complexity for our users, for, I thought, not all that much gain. And so I feel like you need to sometimes check that you are not delegating the taste and the judgment to your agent. You’re delegating the task, and then you are actually checking back that it’s meeting your bar.

The evaluator sitting behind goal is not that checker, by the way. It doesn’t look at the content to see if it’s good or bad in any way, shape, or form. All it does is examine the conversation transcript to see if the hard rules you specified have been met.

/goal Refactor the data-fetching layer in Dashboard.tsx until 
Lighthouse performance score is >= 92 and LCP is under 1.8s as 
shown by the Lighthouse CLI output. Do not change the public API 
of any hooks. Each turn must improve at least one reported metric; 
abort if two consecutive turns show no improvement. Stop after 10 turns.

The workflow I run every day

One of the workflows that I do every day: I have a popular open source repository called Agent Skills. We’ve got over 80,000 stars, and up until recently we were getting anywhere from 80 or 90 pull requests that we had to review a day. And so every day I would go and spend some time checking on this. Now with loop, what you can do is say: “Well, every 24 hours or every 12 hours check the GitHub repository for any new open issues and provide a summary of their urgency,” or “Provide a first pass review,” or anything like that.

/loop every 1h "Check the GitHub repository for any new open issues. 
Provide a bulleted summary of their urgency."

Combining loops and goals

Now you can also sort of combine loops and goals. So you use loop to schedule a check, and then goal to solve the problem. So you can say something like: “Loop for every 24 hours, check GitHub for issues labeled bug. If one exists, use goal to implement a fix until all local tests pass and push the branch.”

/loop every 24h "Check GitHub for issues labeled 'bug'. If one exists, 
use /goal to implement a fix until all local tests pass and push the branch."

But also keep in mind that goal has got some limitations around just how much you can cram in there.

Claude Code team’s write-up also has a composed example that shows where this is all heading:

The primitives above, along with other Claude Code features like auto mode and dynamic workflows (research preview) can be composed into a loop for long-running work. For example, to handle incoming feedback, you can use: /schedule (research preview) to run a routine that checks for new reports, /goal to define what done looks like, and skills to document how to verify it. Dynamic workflows to orchestrate agents that triage each report, fix it, and review the fix. Auto mode so the routine runs without stopping to ask for permission. Putting it together, a prompt could look like this: /schedule every hour: check the project-feedback channel for bug reports. /goal: don’t stop until every report found this run is triaged, actioned, and responded to. When fixing a bug, use a workflow to explore three solutions in parallel worktrees and have a judge adversarially review them.

What the triage system actually does

Inside the PR triage, when I have loops and goals working together for me, what I end up having is a system that allows me to continue accepting PRs and issues, being able to stay on top of what is coming on my plate every day, and especially being able to cross-reference. That’s been a really big deal for me. If I want to work on a particular goal of, “Hey, we’re going to be redoing this part of the system,” and I need to make sure that any issues that happen to touch it are closed as a result of this rework, or that we’re not stepping on anyone else’s toes, I can make sure that that’s well defined.

Scheduled tasks are really useful for regularly just reviewing new PRs and closing things that clearly don’t fit. One good stopping condition, for example: We have a set of contribution guidelines, and our contribution guidelines include things like, “Hey, we currently don’t accept translations.”

And it’s not that we don’t care, it’s that they’re difficult for us to maintain because we don’t speak all the languages that often come in. And so if we tell it to close any issues or close any PRs that happen to touch that aspect of our contribution guidelines, that’s something that can do really well when it’s on that schedule. And so it reduces the overall batch of things that we need to review.

What loops don’t buy you

I’m often asked what loop engineering is not a great fit for. Generally speaking, if you don’t have a clear idea of what the end-state/done/good means for your completion, it may not be the right pattern for your work. For example, a vague goal would be “Keep going until this UI design is good.” What does that mean? Good to who? How is it being evaluated? Tasks that require human taste, subjective design, or open-ended creative exploration aren’t a good fit. When you have a pretty clear idea of the goal, I think loops are a good option to consider.

The fine print

One classic sign that you’ve got a loop spinning in place is the same command being tried over and over without any change in the result. Give the same command a third time with no change from the second and it’s probably time to stop.

One bit of fine print worth knowing: Recurring loops expire seven days after creation. I’d been telling people it was three days. It’s seven. And loops are session-scoped, so they stop when you start a new conversation, though resuming that session with –resume or –continue brings back any recurring task still inside its seven-day window. If you need something that outlives your session, /schedule runs it in the cloud.

If there’s a check you already run every morning by hand, that’s your first loop. Mine was the pull request pile.