AI is becoming more capable at handling everyday tasks, but benefiting from that progress depends increasingly on where people live, what language they speak, and how much control they have over the technology. On the latest episode of This Week in AI, we examined developments from OpenAI and Anthropic, the growing push for regional AI models, and the challenges of making AI more accessible and responsible.

Frontier AI is moving into everyday services

OpenAI introduced LifeVal, a benchmark that measures how well AI handles everyday tasks such as finding childcare, managing household budgets, and accessing government services. Scores rose from 49% for GPT-5.2 to roughly 80% for GPT-5.6 and GPT-6 Astra. This could help researchers assess whether improvements in model capabilities translate into useful assistance with daily responsibilities.

OpenAI reported that ChatGPT now handles roughly 20% of Google’s search volume in search-like queries, although the company hasn’t explained its methodology. Its new GPT-6 Cyber model, available to approved security teams, aims to help identify and fix software vulnerabilities. Meanwhile, Anthropic began testing read-only banking connections in Claude, allowing some US subscribers to analyze spending and identify unusual charges. As AI assistants take on tasks traditionally handled by banks, search engines, and specialized software, those industries will have to reconsider how they serve customers and protect their data.

Anthropic also published FlashProve, a research prototype that uses cryptographic proofs to verify which model generated a response. Separately, nonprofit research organization METR reported that AI assistance may have roughly doubled the pace of progress on certain model development tasks at Anthropic over the past year. The findings don’t establish recursive self-improvement, but they suggest that AI assistance is already speeding up parts of model development. That makes it increasingly important to understand how AI is shaping that progress and to check the reliability of the outputs these systems produce.

Regional AI addresses gaps in access

We’ve previously discussed the “tokenizer tax,” which makes some languages more expensive to process because they require more tokens than English. New research from the University of Michigan found another disadvantage. In one test, answering questions in Pashto consumed up to 179 times more energy than answering equivalent questions in English, while accuracy also suffered. The findings show how language can affect both the cost and quality of AI services.

Countries are responding by investing in models designed around their own languages and infrastructure. Ibero-American countries agreed to form a regional AI alliance to coordinate model development, data sharing, and computing resources. India introduced three domestic models, including EKA-1, which supports 22 Indian languages and English, and two Dhīr models designed to work offline with documents. Offline capabilities could make these systems especially useful in rural areas with unreliable internet access.

Indonesia’s GoTo, the UAE’s Jais 3, and Uganda’s Sunflower V2 are further examples of projects developing AI for local languages. These efforts could give developers more control over AI for local languages and reduce reliance on proprietary AI services, even when some build on open foundation models from larger labs. They could also make AI more useful in languages that existing models handle poorly. For organizations deploying AI across multiple countries, testing performance and cost in each language may reveal problems that English-language benchmarks miss.

Responsible AI depends on how we deploy it

Google faced criticism over changes to AI controls in school accounts. Under a new arrangement, schools that disable AI also lose access to Google Search. That leaves educators with limited control over which tools students can use. And in Europe, an advisor to the EU’s top court suggested that chatbots could infringe publishers’ rights when reproducing news content. The opinion emerged from a dispute involving Google’s Gemini and a Hungarian publisher. A final ruling is still pending, but the case could influence how AI companies access and reuse copyrighted material. This is part of a broader wave of publisher lawsuits, including The New York Times’ US case against OpenAI and Microsoft over the alleged use of its articles to train models and reproduce content in ChatGPT outputs.

AI’s environmental costs continue to come into focus. Some data center operators are repurposing retired jet engines as natural gas generators to obtain power while waiting for grid connections. Although this can accelerate deployment, the converted engines are less efficient and can produce more pollution than conventional turbines, meaning faster access to electricity comes at the expense of higher emissions and worse local air quality. Public opposition is growing too. During the November 3 US general elections, voters favored restrictions on data center development in 40 of 41 local ballot measures across 10 states.

Meanwhile, Planet introduced a natural-language tool for searching and analyzing more than a decade of satellite imagery. Users can explore geographic changes without specialized GIS expertise, making satellite data easier for people without that background to investigate. The application shows how AI can help more people work with complex data that previously required specialized tools and training.

What’s next

As AI expands into personal services and countries invest in their own models, developers and organizations face increasingly different requirements for security, performance, and accessibility. The most useful systems will be those that work reliably for the people and circumstances they’re designed to serve.

We’ll be back on Monday, November 16, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

An agent can get better at its job without a new model. It can learn to find useful information, choose the right tool, or check that a task is finished before reporting success. For that experience to help with the next task, something has to carry over: an updated instruction, a lesson saved in memory, or a change to the software that runs the agent.

That is what interests me about a self-evolving agent: It can use experience to change how it handles future tasks. Much of that change happens in the harness, the instructions, context, tools, and software around the model. Research such as Harness-RSI explores how agents can change that harness while continuing to use the same model.

Of course, a change can make things worse. A saved memory might be wrong, or a new instruction might fix one problem and create another. I think about self-evolving agents in three parts: evaluation to tell whether a change helped, an improvement process that can do more of the work on its own, and the parts of the harness it is allowed to change.

Evaluation: What counts as improvement?

Before I ask an agent to improve itself, I want to know what doing a better job would look like. A model’s score on a benchmark tells us how it performed on that test. To evaluate an agent built with that model, we also need to ask whether it used the right information, took the right actions, and met the user’s requirements.

In a restaurant-planning exercise I developed for teaching evaluation, an agent is given a group’s needs and information about restaurants. Its job is to suggest a place and a time for dinner. The people and restaurants are fictional, but success is easy to understand: Can everyone eat there, afford it, and make it on time?

In one test, the agent recommended a restaurant that met everyone’s needs. My evaluation still marked the answer as wrong because it compared the wording with a sample answer I had written. But my answer was only one possible solution. A different recommendation could work just as well. That feedback could lead a self-improving agent to “fix” a decision that was already good.

A more useful evaluation would check whether the recommendation met the group’s requirements. If someone’s budget is $25 and the meal costs $28, the plan fails that check. We can build a custom evaluation around those requirements, checking the price, dietary needs, and timing. Other questions require judgment, such as whether the agent clearly explained a trade-off between price and convenience. A person or another model can help assess those.

If we asked the agent to make a booking as well, we would need to check that the reservation actually existed. A message saying, “Your table is booked” would not be enough. Evaluating the system means checking what it did, as well as what it said.

Once the agent proposes a change, we can test the updated version against the current one. It should still handle tasks it already did well. We should also try cases that were not used to develop or choose the change. Getting one dinner plan right tells us less than handling new groups with different budgets, dietary needs, and schedules.

The evaluation should also explain why an answer failed. “The meal costs more than the guest’s budget” gives the agent something specific to investigate. It still needs to find the cause, but it knows where to start.

Autonomous improvement: Who carries the work forward?

If an engineer has to spot every failure, suggest every change, and start every test, improvement still depends on that person. Autonomous improvement means giving more of this work to the system. It can review past tasks, look for repeated mistakes, propose a change, and test whether the change helps.

For example, a more autonomous dinner planner might notice that it repeatedly recommended restaurants people could not afford, even though their budgets were available. It could try a new instruction: remove restaurants that exceed anyone’s budget before comparing the remaining options. It would then test that instruction on other groups. If the results improve and the agent still handles tasks it previously got right, it can keep the change. Otherwise, it keeps the current version.

There are two kinds of work here. While planning a dinner, the agent uses tools, reads the results, and decides what to do next. That is its task loop. The improvement process looks back across those attempts and changes how it will approach future tasks. An agent can plan dinner on its own and still depend on an engineer to teach it a better way to plan.

This review does not have to happen after every request. The system might start it after several similar failures, or review a day’s work overnight. Looking at several tasks together helps it see which problems keep coming back.

We can give the system more responsibility gradually. At first, it might investigate problems and test changes, then ask an engineer to approve them. Later, it might also keep certain kinds of changes on its own. The team still decides what it may change, how much time or money it may spend experimenting, and when a person needs to step in.

Harness modules: What can evolve?

When an agent makes a poor decision, I want to find the cause before rewriting its instructions. If a guest’s budget never reached the model, telling it to “pay attention to the budget” will not help. We need to fix how that information reaches the model.

ModularRSI divides the harness into five parts: context management, the agent loop, tool use, observation management, and task completion detection. It studies changes to these parts separately before putting them together. This gives us a useful way to ask what exactly an agent could improve.

Context management comes first for me. It determines what information the model has when it makes a decision, including which memories are brought into the current task. Reflective memory means looking back at what happened and saving a lesson that could help next time. For the dinner planner, that lesson might be to check everyone’s budget and dietary needs before comparing restaurants.

The memory needs to distinguish a general lesson from a fact about one person. “Check the budget before recommending a restaurant” could help with any group. “This guest has a $25 budget” applies to a particular person and occasion. Without that context, the agent could use an old fact in the wrong situation.

Dreaming is a way to review and organize memories between sessions. The system goes back over past interactions and what it has already stored, combines related observations, and saves useful lessons. When memories disagree, it needs to check the original evidence before deciding what to keep. Anthropic’s Dreams, for example, produces a reorganized set of memories that can be reviewed before later sessions use them.

What I want from dreaming is a better starting point for the next task. The agent should be able to find a useful lesson without reading every previous conversation again. We can test that by comparing how it handles new tasks with the old and revised memories. Organizing memories overnight only counts as an improvement if the results support that claim.

The agent could also improve how it finds and organizes memories. Saving one lesson may help with a future task. Changing the way it selects lessons could help it make better use of many future experiences.

The other four modules affect how the agent does its work:

• Agent loop: The agent plans, acts, and decides what to do next. It could learn to notice when it is repeating an unsuccessful approach and try a different one.
• Tool use: Tools the agent has and how it uses them. It could add a tool it needs or change a tool’s interface to make it easier to use correctly.
• Observation management: The agent handles the information returned by its tools. It could change how it summarizes a result so that an important fact is not lost.
• Task completion detection: The agent decides that the job is done. It could check for confirmation that an action succeeded before telling the user it is finished.

These parts work together. Changing a tool can change the information the agent receives, which can change its next decision. Testing one module helps us understand a change. Testing the whole agent tells us whether the parts still work well together.

Improvement becomes recursive when the agent uses its updated harness to carry out the next round of improvement. Suppose it changes how it searches past runs for examples of a failure. The next time it tries to improve itself, it uses that new search method to decide what to investigate and test. A change made in one round now shapes how it works on the next one.

This gives the agent a chance to get better at finding and testing improvements, as well as doing its original job. Each round builds on changes that were kept from earlier rounds. To find out whether that helps, we need to check whether later rounds produce changes that work better on new tasks.

I want an agent that needs less repeated correction because it has learned a better way to do the job. We should be able to point to what it learned, what changed, and how we checked that the change helped. Then we can see whether that lesson holds up when the agent has a new task to do.

This post was created with the assistance of AI, based on a transcript of the conversation.

AI tools have started to remember their users, but each tool remembers a different version of the same person. Duncan Davidson, an independent consultant and longtime software engineer, has been trying to connect those fragments. He joined Ben Lorica to discuss what personal AI could do if people could carry their context between services and decide what each service gets to see.

They also talked about how AI coding changed web development during Duncan’s time at Webflow, what new developers may lose as entry-level work changes, and why open source projects need new ways to handle AI-assisted contributions.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Takeaways

06.31 The developer’s job is moving toward outcomes. Duncan compared AI coding to desktop publishing, which removed the physical work of assembling pages but left designers responsible for what the page should look like. At Webflow, he saw people use AI tools to make websites faster than they could with earlier tools. For developers, he said, the question has shifted from how quickly they can type code to “What outcome do you want? What are you trying to do with this code, and how can you make it better?” People still need to judge whether an implementation fits the rest of the system.

10.41 New developers still need a way to become experts. Computer science fundamentals still matter when an agent offers several plausible fixes and a person must choose the one that fits a larger system. Ben asked where future engineers will gain that judgment if AI absorbs entry-level tasks. Duncan compared early career work to an apprenticeship in the arts, where even menial assignments expose newcomers to how the whole operation works. Companies that measure only throughput, he warned, may leave too little room for people to learn.

23.07 AI-assisted changes are straining open source review. Duncan described a library he uses in one of his apps. His fixes, along with performance problems Claude noticed, left his version roughly 25 pull requests ahead of the project’s main branch. The maintainers were using AI too, he said. The difficulty was getting useful changes back to the project without overwhelming its review process. His conclusion was that older open source collaboration practices need a “serious retrofit.”

28.27 Personal memory is fragmented across tools. Duncan first had his coding agents save useful information into a collection of Markdown files. Once commercial agents added memory, he found that “Claude knew me one way and ChatGPT knows me another way.” He could push context back to each tool with instructions and occasional reminders, but that solution demands more tinkering than most people want to do. What he wants is a small set of shared mechanisms that let a person carry context across services.

34.01 Portable memory needs boundaries. A prompt’s context, an agent’s memory of recent interactions, and a person’s longer-term knowledge should be able to travel, but Duncan also wants people to control access by scope. He keeps general notes alongside information about his family and finances. His test for a useful system is whether he can say, “Give Claude access to these scopes of my knowledge, but not everything.” That becomes harder when a personal agent encounters knowledge gained at work. Duncan left open where an employee’s own expertise ends and an employer’s information begins.

36.44 Trust and security shape what people will share. Ben’s not yet comfortable letting a consumer agent reach into his finances. Duncan noted that trust would have to grow through experience, as it would with a human agent. We’re already seeing how AI agents can find vulnerabilities faster than established disclosure practices can handle. Duncan expects software maintainers to face more pressure as agents uncover latent bugs, while Ben questioned how quickly organizations that rely on vulnerable code would learn about the findings. People need control over what an agent can access and confidence in the systems around it.

49.52 A personal memory could answer requests with permission. Duncan imagined keeping his memory on his phone and giving a new service a way to request specific information. A car dealership’s agent, for example, could ask for details, but the phone would seek his approval before sharing them. He put it this way:

I could provide it a handle to my memory. And whatever agents they’re working with can come in and ping me and say, “Hey, we’d like to know these things about you.” And it wouldn’t necessarily do it automatically.

The shift to agentic reality

When Git was designed in 2005, Linus Torvalds solved a brilliant problem for humans collaborating over terminal windows by trading line-by-line diffs. Fast-forward to 2026: Autonomous coding agents don’t think in flat text files or line breaks. They reason across syntax trees, execution traces, call graphs, and type constraints. In my view, forcing an AI agent to engineer complex distributed systems through string matching is like handing a chief software architect a chisel and a stack of sticky notes.

File-centric versus system-centric paradigm shift

Whenever I see engineers try to lock context-blind models into file-centric version control, I watch the consequences compound quickly. Context windows fill with irrelevant churn, API bills surge, and subtle, multifile structural breakages slip past basic syntax checks. Just as critically, standard Git discards the cognitive history of a change. The moment an agent session completes, its execution logs, compiler feedback, and discarded hypotheses evaporate into thin air.

Tactical illustration: renaming a distributed method signature

Consider updating a core helper signature across 40 dependent services. A traditional string diff forces the agent to read and mutate thousands of lines of surrounding boilerplate text, burning thousands of context tokens per file:

--- a/services/auth.py
+++ b/services/auth.py
@@ -42,7 +42,7 @@ class AuthManager:
     def validate_session(self, token: str) -> UserSession:
-        return fetch_user_by_token(token)
+        return fetch_user_by_token(token, include_metadata=True)

In contrast, an AST mutation node isolates the target function symbol, executing the change precisely without passing unneeded ambient text through the prompt window:

{
  "action": "mutate_node",
  "target_ast_id": "ast_node_89f2a41b",
  "symbol": "fetch_user_by_token",
  "update": {
    "add_argument": { "name": "include_metadata", "value": true }
  }
}

This exact bottleneck hit Anthropic in late 2026. As autonomous agent activity pushed monorepo merges beyond 30,000 per day, conventional Git infrastructure simply buckled under the load. To me, unlocking genuine agentic autonomy requires evolving source control along two complementary tracks: shifting to server-hosted, graph-aware architectures, and treating agent reasoning as a first-class, versioned artifact right alongside the code itself.

Why text diffs and ephemeral sessions create silent debt

Standard diff tools operate purely on raw Unicode strings. When an agent reorders functions or cleans up formatting, string diffs pollute context windows with noise while remaining blind to structural reality. If you modify a single method signature, Git won’t warn you about the dozen callers broken across separate modules. From my experience, feeding 2,000-line source files into prompts just to update a 5-line helper method degrades model attention, burns compute, and inflates overhead at scale.

Text diffs versus AST graph mutations

Compounding this structural blind spot is what I consider our biggest waste: ephemeral session memory. Whenever an agent finishes a task or compacts its context window, its entire diagnostic history—failed attempts, shell outputs, and compiler warnings—disappears. The next agent assigned to the branch steps in completely blind, doomed to repeat the same diagnostic dead ends and burn unnecessary tokens.

Agent-native architectures in practice

In my view, next-generation version control must replace raw file trees with Tree-sitter parsers and code property graphs (CPGs). Instead of editing lines on disk, agents mutate specific syntax nodes directly, executing sub-second edits and running immediate dependency impact checks across the codebase. Moving control back to the server eliminates the massive I/O bottlenecks caused by checking out heavy working trees onto virtual machines, allowing agents to query and update code over fast graph end points.

Practical implementation: server-side microdiff stream

Below is an example payload representing an atomic microdiff layer executed directly on a server-hosted code property graph, bypassing local file checkouts entirely:

{
  "commit_id": "c_9011f4a2",
  "parent_graph_version": "v14.2.0",
  "operations": [
    {
      "op": "replace_node_property",
      "node_uri": "cpg://auth_service/ast/Node_441",
      "property": "return_type",
      "value": "Promise<UserSession>"
    }
  ],
  "validation_hash": 
  "sha256_e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}

This structural shift pairs naturally with stacked microdiffs—atomic, dependent layers verified sequentially in isolated virtual workspaces to prevent merge races. I was particularly impressed seeing the real-world impact of this pattern with Anthropic’s Scion architecture. Faced with object-store locks and clone latencies as daily merges scaled 10x, Anthropic built a server-centric SCM that streams code graphs on demand, slashing infrastructure overhead by 200x while enabling near-instant workspace initialization.

Versioning reasoning and system memory

I strongly believe code changes alone rarely capture full intent. Modern SCM must link model reasoning traces, prompt chains, and decision graphs directly to commit nodes, preserving architectural context for future debugging and audits. Persisting build outputs, compiler logs, and linter runs in execution ledgers attached to commit trees allows subsequent agents to inspect prior test results instantly rather than rerunning identical test suites.

System memory schema: reasoning and execution ledger

To prevent model sessions from repeating diagnostic dead ends, execution ledgers append structured reasoning and compiler feedback directly to commit nodes:

{
  "agent_session_id": "sess_88190c2a",
  "target_commit": "c_9011f4a2",
  "hypothesis_tree": {
    "attempt_1": { "approach": "In-memory caching", "status": "REJECTED", 
    "reason": "Memory limit exceeded in integration test" },
    "attempt_2": { "approach": "Redis backing store", "status": 
    "VERIFIED", "execution_time_ms": 142 }
  },
  "compiler_ledger": {
    "linter_passed": true,
    "type_check_errors": 0
  }
}

By explicitly recording abandoned branches and failed architectural hypotheses as structured metadata, we stop future agent sessions from reexploring invalid paths. I argue that capturing thought streams alongside code transforms source control from a passive text archive into an active memory system.

The hard engineering trade-offs and counterarguments

I won’t pretend making this transition is easy; it comes with steep engineering trade-offs. Existing developer tooling, from LSPs to build pipelines and IDEs, expects standard local file trees. Replacing checkouts with server-hosted graphs requires building heavy compatibility layers or redesigning workflows. Furthermore, human reviewers still digest code line by line, creating a natural tension with pure graph mutations, while storing continuous thought streams drives up storage and compliance costs. During early refactoring, syntax is often broken and unparseable, forcing systems to temporarily fall back to raw text. Overcoming Git’s decades of deep ecosystem lock-in remains a major operational hurdle we cannot ignore.

Trade-off matrix: architectural migration challenges

Bottom line

Git was designed for humans reviewing text, but agentic scale demands server-centric, graph-aware source control. Code property graphs stop token bloat and prevent silent structural breakages. Versioning reasoning traces and execution histories alongside code keeps agents from repeating past mistakes. And stacked microdiffs offer a clean, isolated path for autonomous agents to ship code safely at scale.

Most of today’s AI entrepreneurs and social media enthusiasts are young. AI is the first technology revolution they’ve lived through. They don’t remember how unbelievably exciting it was in the late 1970s to get your own computer, instead of struggling to get time on one of the vast machines chugging away in what used to be called “the glass house,” an air-conditioned room where from outside you could watch the nine-track tapes being accessed and the refrigerator-sized disk drives spinning. Input and output was on high-speed teletype printers (no screens) and magnetic tape (and before that on punch cards.) They also mostly don’t remember, as I do, writing assembly-language programs in which you moved data from one computer register to another, specified the arithmetic or logical operations to be performed on it, and then specified where to save the result.

They look at today’s AI and marvel at how virtually anyone can now generate code. They don’t realize that that’s also how it felt when you could use an interpreted programming language like BASIC to write your own applications instead of writing them in Assembler. They also don’t remember when Dan Bricklin used BASIC to write the first version of VisiCalc in 1979. He went into business with Bob Frankston, who rewrote it in 6502 Assembly Language for the Apple II. Dan was able to express the idea in Basic; Bob then wrote a tight, efficient version for widespread distribution. People rushed out to buy computers so they could use VisiCalc, and everyone predicted that the VisiCalc “spreadsheet” would put all the accountants out of work. It was the first program to be referred to as a “killer app.” It wasn’t the democratization of programming itself that was the killer app but what Dan and Bob did with it. In the entrepreneurial explosion that followed, databases, graphics, word processing, and other business applications came out of the glass house and, as Bill Gates put it, “onto every desk and into every home.”

We saw the same leap from astonishment to innovation to adoption with the World Wide Web. When O’Reilly demonstrated GNN, the first commercial website, to people at a trade show in 1993, attendees couldn’t believe that clicking on a link could pop up an image or a document from another computer thousands of miles away. They didn’t understand the difference between the software application (the web browser) from the network-connected documents with clickable hyperlinks that it acted as a frame for. They didn’t understand that web pages were a new kind of software written in a simple markup language anyone could understand. It didn’t make sense that the web page was a document that in many ways could look and act like a kind of program. And when people did understand, they thought that might be the end of programming.

In 1993, everyone thought that the web browser itself was the killer app, and that Marc Andreessen, the young creator of Mosaic, the most popular web browser, which became Netscape, was going to dethrone Bill Gates as the new king of computing. They didn’t realize that the true killer apps of the web were not the web browser and the web server but applications “up the stack” built using the new programming paradigm the web enabled. Those applications weren’t software as we’d known software before. They were database- and application-backed websites delivering services that were previously impossible. I remember thinking of Amazon as the 2005 equivalent to VisiCalc when a friend told me she had just bought her first computer so she could shop on Amazon. Once again, it wasn’t the new capability itself that was the killer app but what developers did with it.

What makes a killer app? I think that Tom Stoppard nailed it in his play Rosencrantz and Guildenstern Are Dead:

A man breaking his journey between one place and another at a third place of no name, character, population or significance, sees a unicorn cross his path and disappear. That in itself is startling, but there are precedents for mystical encounters of various kinds, or to be less extreme, a choice of persuasions to put it down to fancy; until—“My God,” says a second man, “I must be dreaming, I thought I saw a unicorn.” At which point, a dimension is added that makes the experience as alarming as it will ever be. A third witness, you understand, adds no further dimension but only spreads it thinner, and a fourth thinner still, and the more witnesses there are the thinner it gets and the more reasonable it becomes until it is as thin as reality, the name we give to the common experience.

That is, a killer app is at first a startling window onto an undreamed-of possibility, but one that so changes the paradigm that it rapidly gets taken for granted. In the meantime, everyone tries to make sense of the new world. Usually, their sensemaking is to try to reduce it to what they already know. The breakthrough entrepreneurs see more deeply and go farther.

In 1995 and 1996, almost everyone saw the future as a showdown between Microsoft and Netscape over who would own the web server and the web browser, and whether the web would supplant the desktop as the new platform. Microsoft won the browser wars, but it turned out to matter far less than anyone expected. The value went to the companies building on top of the browser, the ones that figured out what the network made possible.

The current narrative pitting AI model providers against each other sounds a lot like Microsoft versus Netscape to me. It is the platform war, the war for the layer that will become the common experience on which the revolutionary future will be built. ChatGPT, Claude, and Gemini are the browsers of this era. That is not a small thing. But the browser wasn’t actually the killer app of the web. Netscape made money for a little while. Amazon and Google and the explosion of other sites were what made the web into something people couldn’t live without.

Claude Code, Codex, Cursor, and the other coding agents may be the BASIC of this era, or like the tools and frameworks that made database-backed web applications possible. They are plumbing. Important plumbing, and plumbing that has to be in place before the future can truly happen. But still just plumbing. Inventors tend to spend decades maturing it, and O’Reilly tends to spend decades teaching people to build it. I suspect that will be just as true this time around.

In short, I’ve started asking myself whether we’ve actually seen the killer app for AI yet. The obvious assumption is that ChatGPT was. It certainly got a LOT of people excited and onboard very, very quickly. But maybe Claude Code was, enabling the breakthrough in agentic coding. Or maybe Hermes or Dots or Muse or some other personal agent will be. But we should take seriously the possibility that the killer app is not yet on the scene. These initial candidates for the killer app look too much like advances in a new enabling platform and not enough like a new generation of applications.

Personal agents may turn out to be a breakthrough category, but right now they look to me like early ecommerce before Amazon got it right.¹ For Amazon to work, the web had to become secure enough for people to use credit cards online. The apps were waiting on missing mechanisms. For Google’s economic model to kick into gear, pay-per-click advertising had to be invented. I suspect we are in a similar moment waiting for crucial pieces to be dumped on the table so developers can complete their part of the puzzle. For AI agents to act on our behalf in the world, they will need their own equivalents: ways to pay, to prove who they are acting for, to be held accountable, and for the counterparties to settle on the allocation of the value created by the transaction. Ilan Strauss and I have argued in “The Missing Mechanisms of the Agentic Economy” that many of those mechanisms don’t exist yet. Maybe the killer apps are waiting on the equivalent of https and pay per click.

In short, when you listen to the AI conversation and mostly hear about people’s programming pipelines and how much more productive their programmers are, and very little about what they are using that productivity to build, you can be pretty confident that the killer apps haven’t arrived yet.

Where are we in the cycle?

Let’s line up the calendars and see if that tells us something useful.

CP/M (Control Program for Microcomputers), the first widespread personal computer operating system, was developed in 1974, and the Altair 8800, the first mass-market personal computer, appeared on the cover of Popular Electronics in January 1975. Microsoft released a BASIC interpreter for it in June of that same year. A host of competing personal computer hardware platforms running CP/M were introduced soon thereafter. The Apple II was released in 1977, VisiCalc in October 1979, and dBase II in 1980. The IBM PC came out in August 1981 with Microsoft’s MS-DOS as its operating system. Lotus 1-2-3 (which included spreadsheet, graphing, and simple database features) was released in January 1983 and turned the PC into a business necessity.

In 1984, the revolutionary graphical user interface of the Macintosh was the next step function jump in the PC era. The GUI itself had been invented at Xerox PARC in the 1970s, but it wasn’t till Steve Jobs built his “insanely great” Mac that the world embraced it. Desktop publishing became the killer app of the GUI, with Aldus PageMaker and the Apple LaserWriter in 1985. The PC proper didn’t get the GUI till Windows 3.0 in 1990, but Microsoft’s embrace of it for applications such as Word (1983), Excel (1985), and PowerPoint (1987) as well as for the operating system and programming environment made it, for the time, master of the computing world.

So you can see a rough timeline. Four or five years from the introduction of the first version of what was to become the new platform to the first killer app. (Or two, if you count from the introduction of the Apple II, the platform for which VisiCalc was first written.) Then 10 years of invention, competition, and refinement, followed by fairly rapid consolidation and accumulation of market power.

Let’s do the same timeline for the web. Email and online forums might well have been the first internet killer apps, but they were really holdovers from previous generations of research networks. They were apps that had been evolving since the 1970s, though CompuServe and AOL had begun to bring them to the general public in the ’80s. The world really changed, though, when Tim Berners-Lee first wrote his proposal for what became the World Wide Web in March 1989. It was released to the world in August 1991. This was the new platform and the start of what would become the new programming paradigm.

Not too much happened outside physics labs and a few universities until 1993. (There were only about 200 websites when we published The Whole Internet User’s Guide and Catalog in November 1992.) In 1993, NCSA, the National Center for Supercomputer Applications, released Mosaic and we launched GNN, the first commercial website. Netscape shipped its browser, a follow-on to Mosaic, at the end of 1994. Yahoo! was launched that year too, Amazon, eBay, and the Altavista search engine in 1995, MapQuest in 1996, Google in 1998, and Wikipedia in 2001.

So, again about four years from the release of the platform to the appearance of the first killer app, and another 10 till the consolidation phase. (I see Gmail 2004, Google Maps 2005, Google Docs 2006, and Chrome 2008 as the equivalent of Microsoft’s release of Office on top of Windows as the beginning of the consolidation phase of the web.)

The timeline for mobile is not dissimilar. The iPhone in 2007, Android in 2008. Killer social apps Facebook and Twitter had gotten their start on the web at about the same time, but really came into their own on mobile over the next few years. The “could only have happened on mobile” killer apps of Uber, Lyft, and the on-demand economy didn’t arrive till 2011/2012.

A lot of people seem to think AI is moving way faster than previous technology revolutions. But if you line up the calendars, AI is moving at about the same pace as previous eras. Treat the transformer paper, “Attention Is All You Need,” published in June 2017, as the AI equivalent of Tim Berners-Lee’s 1989 proposal, and ChatGPT’s first release on November 30, 2022, as the equivalent of Mosaic. The gap from invention to breakout is about the same, five years for the web and five and a half for AI. That’s just getting the platform in place. The AI-native “killer apps” haven’t yet arrived but are due soon. (And please, let’s avoid the jokes that the “killer apps” of AI first showed up in the Russia-Ukraine drone war. That’s true, but not on the direct line of my argument here.)

On what seems to be a fairly consistent timeline, where are we today, then? October 2026 corresponds to roughly the spring of 1997. Amazon was about to go public. Hotmail was a year old. Google was still a research project at Stanford called BackRub. SixDegrees had kicked off social networking, but Facebook was still seven years off. Napster was still two years away.

Think of all those amazing application categories that had been pioneered, though. Search, ecommerce, email, and social networking were all in their infancy, but they were out there, delivering real value. I’m not sure that I see the equivalent time to value in AI. AI didn’t even get to Claude Code, its Microsoft Basic equivalent, till 2025! By some measures, AI is moving more slowly rather than more quickly. There’s more change to digest, and developers are still struggling with the tooling and developing the new knowledge that is needed.

Why so slow, with so much money?

The four largest hyperscalers are on track to spend roughly $700 billion in capital expenditure this year, nearly double what they spent in 2025. That is orders of magnitude more capital than went into the web in the 1990s. If capital and capability were the limiting factors, you’d think we should be well ahead of the earlier calendar, not behind.

I suspect that at least part of the answer is that Silicon Valley’s quest for premature monopoly may be making it less efficient at finding breakthroughs rather than more. Starting with the era of Uber and Lyft, massive VCs have canonized the winners at the beginning of the race, crowding out alternate technologies and business models. That may be happening here. But it’s also just in the nature of bubbles, which depend on groupthink to flourish.

And part of that groupthink is about where the value is going to be captured, rather than where it is going to be created. That is almost always backward looking. During the railroad boom, capital was pouring into building the tracks, and the new business models (think Montgomery Ward and Sears) that railroad enabled didn’t come till later. The dot-com bubble was far less capital intensive than people think. The telecom buildout that ran alongside it was where the money got spent. US carriers spent about $97 billion on capital in 2000 alone.

By 2002, according to an FDIC analysis, only 2% to 5% of the fiber in the ground was actually lit. More than 60 telecom companies went bankrupt. Once the data centers and dark fiber of the telecom bubble could be bought for pennies on the dollar, they enabled a generation of new applications like streaming video. YouTube was founded in 2005, after the crash, on cheap bandwidth that someone else had paid for before they went out of business. There were some visionary companies (think Google, Amazon) that had been started before the abundance arrived, but many more were founded after the bubble burst.

Carlota Perez’s book Technological Revolutions and Financial Capital documented this pattern across every technological revolution since the canal era. Financial capital pours into the installation of the new infrastructure, a bubble forms and bursts, and the deployment period that follows, when the technology actually reshapes the economy, is built by different people on top of what the crash left behind. If that is where we are in the cycle, then the absence of killer apps is not a puzzle at all.

The current data center buildout and the massive investments in frontier AI may not be the future. They are the infrastructure on which the future will be built. We have to ask ourselves what will we build once competition and overshoot and the bursting of the data center bubble make intelligence infrastructure cheap.

Help me fill in the picture

I see several possibilities. Either we are still early and the killer apps simply haven’t shown up yet, the way Google hadn’t shown up in 1997. Or AI is going more slowly than its boosters claim, in spite of all the capital, because capital and model capability are not the limiting factors. But it’s also possible that the killer apps are here, and I’m not hearing about them because the AI conversation is so focused on AI capability and nobody is telling the story of what they are accomplishing with it beyond building better developer tools.

I would like to know which. We all recognize a killer app when we see one. Amazon was a bookstore with more books than any physical bookstore could hold. Wikipedia was an encyclopedia written entirely by volunteers and was updated in real time within hours and sometimes even minutes of major events. Google Maps meant you always knew where you were and how to get anywhere else. If you know of an AI application, beyond the chatbot or agentic coding environment itself, that changes what ordinary people can do on that scale, or a business that does something an order of magnitude cheaper or better than the one it replaces, I want to hear about it. Tell me what you’re building in the Google Form below, and describe what tools you’re using only in the context of what they let you produce for others. If you’re a software developer using AI, what are you producing that was unthinkable before AI?

Footnote

1. CommerceNet demonstrated the first encrypted credit card transaction online in 1994, and Netscape made the idea mainstream when it introduced HTTPS (HyperText Transfer Protocol Secure) and made it available in the Netscape browser in that same year. This made it safe to type a credit card number onto a web page. ︎

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

AI is great for making teams faster. But is it sustainable? What about maintainability, burnout, and lock-in?

This is part 3 of my series on assessing AI’s impact on software development. In part 1, we looked at why assessing impact is important, and what not to do. In part 2, we looked at how to measure AI’s impact on delivery speed. In this part, we’re looking at how to measure and model the unintended consequences of using AI. Then we’ll wrap up in part 4 with an examination of business outcomes. Finally, an epilogue puts it all together with notes you can share with your CFO.

Does maintainability even matter?

I can hear it now: “Who cares about namby-pamby issues like maintainability and burnout? It’s a job. If jobs were supposed to be fun, they wouldn’t be called ‘work!’ Suck it up!”

So let me be clear: We care about AI’s unintended consequences because they have real economic impact. If code isn’t maintainable, features take longer to write and bugs take longer to fix, taking time away from more valuable work. If people are burned out, they don’t put forth their full effort, and they eventually quit, taking institutional knowledge with them and leading to time-consuming hiring cycles. If you’re locked in, and AI token costs increase, you could be stuck with a choice between a big bill and cutting back on development.

Among the people I’ve talked to, there’s widespread agreement that heavy use of AI produces code that humans have trouble understanding. But there’s disagreement about how much that matters. If AI is doing the maintenance, do we care if a human can understand the code?

There are strong opinions on this topic on all sides, but not a lot of actual evidence. What’s clear is that the costs of maintenance and other unintended consequences do matter, but only if there are any costs. This is what we need to measure: Is there an impact? And if there is, is it enough to matter?

Unfortunately, unlike the rigorous speed trial we discussed in part 2, unintended consequences are gradual and fuzzy. Your measurements are going to be noisy and lagging. You’ll have to create models and explore scenarios. Ultimately, your decisions will come down to judgment, informed by your measurements and models but not driven by them.

One ring to rule them all

There is one idea that, while not perfect, does a good job of capturing the majority of unintended consequences. It’s cost of change.

Put simply, cost of change is like a tax on the changes you make to your software. When cost of change is low, complicated changes still take time, but they’re achievable. When cost of change is high, even simple changes can take weeks, and complicated changes seem completely out of reach.

Cost of change isn’t easy to measure, but there are several indicators you can use to inform your judgment.

Value-add percentage

Value-add percentage is the percentage of your software development effort that’s spent on things that bring value to your business, such as new features, versus the percentage of effort that’s spent on non-value-add work, such as fixing bugs, resolving incidents, and manual test and release cycles.

Non-value-add work is called “muda.”¹ Some muda is necessary, even valuable, but that doesn’t make it value-add. Imagine a demanding and crotchety customer who always asks, “Why didn’t you just do it right the first time?” If that customer won’t pay for something, it’s muda.

That’s why bug fixes are muda: “I’m not paying you to fix a bug that shouldn’t be there in the first place!” And performing releases: “Why don’t you just ship it already! How hard can it be?” Sure, bug fixes can be valuable, and releases are necessary, but they’re still muda.

I find it useful to distinguish muda further into discretionary and nondiscretionary muda. Discretionary muda is work that you choose, such as investing in code changes that decrease development costs. Nondiscretionary muda is work that’s chosen for you, such as bug fixes, incidents, and dependency upgrades.

You’ll always have a baseline amount of nondiscretionary muda; it’s a constant tax that takes time away from other work. But if it’s rising, that means the time you spend on value-add work is going down. You have less time to work on things that matter. Your costs are increasing.

When I was VP of engineering, value-add percentage was one of my key metrics. I made decreasing muda a priority, and it paid off. We doubled our value-add percentage over the course of my three years there, which allowed me to free up people for substantial new initiatives without increasing headcount.

DORA metrics

Your organization is likely already tracking the DORA metrics of change lead time, deployment frequency, change fail rate, and so forth. If not, they’re worth considering. As with value-add percentage, changes in these numbers are something to keep an eye on.

Token usage

Tokens are a literal cost, and the more tokens it takes to make a change, the higher your costs. Unfortunately, token usage can fluctuate for many reasons, not just cost of change. Still, it’s worth watching. Anecdotally, better-constructed software is easier for AI to work with and costs fewer tokens.

One of the biggest variances in token usage is people’s level of adoption. Rather than just tracking overall usage, keep an eye on individual usage changes. In the charts I’ve seen, usage has formed a Pareto curve. Increases in usage by people on the low end of the curve may just represent increasing familiarity. Increases by people on the high end of the curve are worth investigating further.

Changes in models, harnesses, prompts, and more will all affect token usage. This metric is only advisory; it’s meant to trigger further investigation, not to be used on its own.

One ring to find them

Of these metrics, value-add percentage is the one I trust the most. It operates at the level we care about: Where is the time going?

Tracking value-add percentage doesn’t have to be a big burden. You don’t need time sheets or detailed task tracking. The trick is to not care overly much about precision. Tracking work within a day or two is good enough.

When I did this tracking, I piggybacked on our valuable increment (VI) tracking. Each one was several days to several weeks of work and could be cleanly categorized as either value-add or muda. But you don’t have to use VIs; any tracking approach will work.

Don’t get too fine-grained with your tracking. When I was VP, managers checked in with each team twice a week and recorded which VI the team had worked on and how many people had worked on it. Each VI was categorized as “muda” or “value-add,” and we had a bunch of subcategories that helped us understand where time was going, such as “bug,” “capacity investment,” “major initiative,” and so forth. It was lightweight and very useful.

Review your value-add percentage monthly. If it’s going down, you don’t necessarily have a problem, but it’s worth investigating further.

The biggest problem with value-add percentage is that it’s noisy. My experience was that trends took three to six months to show up. It took a year before I was confident stating that my cost-of-change reduction efforts had led to lasting results.

That’s a long lag time, but it’s still worth tracking. My product partners and I found value-add percentage incredibly valuable as a tool for communicating what was happening in product development. And, sadly, improving cost of change is a lot harder than making it worse. If AI is causing serious problems, it’ll probably show up more quickly.

One ring to bring them all

Because your cost of change indicators will be noisy and uncertain, supplement them with models and scenario planning. Your models won’t tell you what the future holds, but they will tell you what it could hold, which will help you manage risks.

Here are some of the models I’ve found useful:

Maintainability

Model maintenance as a mandatory “tax” on development. For each month of value-add development, estimate a certain amount of maintenance in the following year, and then a lower amount of ongoing maintenance in every year after that. (Wisdom of the crowd estimation may help.) Then build a model that shows how many engineering days are available each month. Subtract the time spent on maintenance, and the rest is available for value-add.

To model AI’s impact, apply your measured development speed improvements. Estimate how AI will change maintenance costs and estimate effects on headcount. Then explore scenarios based on varying costs of AI maintenance.

If you only create one model, make it this one. I have a more detailed explanation here. It includes a spreadsheet you can use as a starting point for your model.

For example, using that model, if AI increases our speed by 50%, but token costs reduce headcount by 15% and maintenance costs are estimated to double, I can see that I’m only breaking even, despite the big performance improvement:

Excess lines of code

Lines of code aren’t a good productivity measure, but they are a reasonable cost measure. Generally speaking, more lines of code result in more maintenance costs.

This model works similarly to the previous one but uses lines of code instead of engineering days. Use traceability data to categorize commits as “value add” or “maintenance.” For each commit, measure gross code churn: the number of lines of code that were either changed, added, or deleted.

For each line of value-add change, estimate a certain amount of maintenance code churn in the following year, and then a lower amount in every year after that. Model how much maintenance code churn will be needed each month based on the value-add churn of previous months. (See the “tmux” tab of this spreadsheet for an example.)

To model AI’s impact, measure how AI affects code churn during your speed improvement trial. Then project value-add churn into the future and model how maintenance churn could change.

The advantage of this model is that you can use (part of) your actual commit data to refine your estimates. For example, the following graph uses real commit data from the open source tmux project.² Prior to running the model, I guessed one line of maintenance for every line of value-add in the first year, and 0.1 lines every year after that, which turned out to be surprisingly close to the actual data.

Lock-in

If token costs increase, how will you respond? Will you cut headcount? Reduce your use of AI? Switch to a less capable model?

To model these effects, update the “maintainability” model. Reduce your monthly engineering capacity according to headcount and estimate reductions to your speed improvements based on AI reductions. You may want to add a separate set of estimates and calculations for humans maintaining AI-written code.

Burnout

AI Disclaimer: This image was created with OpenAI’s GPT-5.6 Sol model operating in “High” thinking mode. See the ChatGPT conversation here.

Productivity is likely to go down as mental exhaustion goes up, and turnover is likely to increase. I don’t think you can model these effects with any accuracy, but you can still keep an eye on them.

Include mental exhaustion as part of your development speed study to see if there’s a significant difference in mental exhaustion between the two approaches. Follow up with periodic one-question surveys about people’s mental energy levels. Pay attention to actual turnover relative to historical numbers, and look for complaints about AI in exit interviews.

And in the darkness bind them

There’s a dark side to value-add percentage. It’s incredibly easy to game. As soon as managers think their teams are being judged based on how they categorize their work, the temptation to tweak the categorization in their favor will be exquisite.

It doesn’t even have to be fraudulent. A leader with prioritization authority and something to prove can simply decide to defer maintenance in favor of value-add work. The pressure to do so is already immense, and it takes unusually strong character to resist.

(If you really hate your company, consider this: You can get away with deferring maintenance for a couple of years. Just enough time to line up a bigger title at a different company! Then your successor is left holding the bag, and your old company wonders why product development can’t get anything done.)

I don’t have an easy answer to these problems other than to remind you to use these metrics to evaluate your approach, not your people. There is no metric that can’t be gamed, and this one is more gameable than most. But it’s also critical for understanding AI’s possible downsides. Use it, but use it with caution.

Development speed and unintended consequences are two sides of the same coin. They help us understand how AI changes how much we deliver, both now and in the future. But just delivering more software isn’t enough. We need to produce the right software. We’ll look at business outcomes next, in part 4.

Footnotes

1. “Muda” is a Japanese word that comes to us from the Toyota Production System, the inspiration for the Lean Software Development movement. ︎
2. Many thanks to Christophe Stjernlöf for providing the raw data from the tmux project. He used an LLM to classify commits as “value-add” or “maintenance.” The stairsteps are an artifact of the way the data was collected. ︎

The following article originally appeared on the Asimov’s Addendum Substack and is being republished here with the author’s permission.

This essay is based on my conversation with Demetrios Brinkmann for AAIF Live, recorded before my keynote at AGNTCon + MCPCon North America in San Jose on October 23, 2026. It was edited and rearranged for publication by Ilan Strauss, and in places it draws on the keynote. Both videos are also on the AI Disclosures Project website in the new “Missing Mechanisms” page.

The keynote begins at 1:28:40 in the Day 2 livestream.

The Bitter Lesson vs. AI Markets for Human Expertise

In 2019, the computer scientist Richard Sutton wrote a short essay called “The Bitter Lesson.” Looking back over 70 years of AI research, he concluded that general methods that scale with raw computation beat methods built on human expertise, and by a large margin. The AI industry as a whole is bitter-lesson-pilled, to borrow the internet’s slang for a true believer. I am human-pilled.

I believe in the bitter lesson. Brute force wins out in the end, and I have wrestled with that fact throughout my career. But we humans still have to figure out our role in the AI economy, and I find it rather disturbing that the AI community isn’t spending more time asking how to make this future friendlier for humans. The labs think about how to keep AI from going rogue, and their chief economists study job losses from AI. They don’t study what mechanisms we could introduce to create new economic opportunities for humans.

The question matters to us at O’Reilly Media. We don’t think of ourselves as a publisher, a conference producer, or an online learning platform so much as a company that transfers knowledge from the people at the cutting edge to the people who want to follow them. So we are very interested in what human expertise is worth, even in the face of AI.

Knowing everything and nothing

Human expertise was used to train AI, which makes AI a fabulous cultural technology. Thousands of years ago, the only knowledge people had was what they could keep in their heads. Since then we have built better and better ways to share knowledge across time and space, and AI continues that. With each of these transitions, we have to ask again what the truly human is still worth. People often say judgment, or taste. That’s true, but I think it goes deeper:

AI is a sea of possibility. All of that knowledge sits in the model, and nothing sets it in motion except a human. (That is changing a little as we start giving AI agents long-running tasks, and we will have to come to grips with it.) If you and I used Claude or ChatGPT for the same research project, we would each bring different experience to it, tell it what we want and why, and give it context in different ways. I see that as the unique focusing power of human consciousness and attention.

Earlier this year I spent an hour or so talking this question over with Claude. When it offered to write up a first draft for me, I said no. I wanted an essay from its own point of view about what it took away from our conversation. The result, “Why AI Needs Us,” opens, “He came in through Powys,” and a few paragraphs later says, “The choice of entry point was already the argument.” That captured what I’m getting at. I had come into the conversation through a book I love, John Cowper Powys’s The Meaning of Culture, which distinguishes culture from education. Powys gives his own unique and powerful definition: “Culture is what is left over after you have forgotten all you have definitely set out to learn.” It is what you absorbed. Each of us absorbs a small and particular slice of everything. Large language models (LLMs) try to absorb everything, and so become overly general; we give them focus. Human knowledge provides them with a lens.

Let me bring that into a business context. Andrew Odewahn, O’Reilly Media’s CTO, built a research assistant to help with managing technology in IT organizations. You teach it your infrastructure: the tools you use, and where the definitive information about each one lives. When one of his engineering managers said the team was overloaded and asked to hire another person, Andrew turned the agent loose on the department’s work, and it agreed that the group needed another hire. Then, on impulse, he tried something different. O’Reilly Media has an MCP server we call O’Reilly Expert Intelligence. (MCP, the Model Context Protocol, is an open standard for connecting AI models to outside tools and data.) It lets a model answer a question through the filter of our own body of expert knowledge. Andrew asked what the O’Reilly experts would say.

Seen through the lens of Google’s book Site Reliability Engineering, the model said, it was totally rethinking the problem. The book describes toil, the manual, repetitive work of keeping systems running, which grows as the systems grow. After analyzing the group’s toil, the model concluded that another hire would make the problem worse. The group needed a toil audit to find out why there was so much overhead, and then a restructuring.

Much of that knowledge was already in the model, since the AI companies have more or less pirated our books. It came back with a completely different answer simply because it was told to consult a specific group of experts. I think there’s a metaphor there. An LLM knows everything, and as a result it doesn’t know anything. We humans are so much more limited, but because we focus on a particular problem, we bring things together in a different way.

Magic words

Last year I wrote a piece called “Magic Words” about how some words carry far more context than others. It sounds like a trivial observation, but if you unpack it, it leads in interesting directions. One of our trainers, who teaches a course on productivity, pointed out that you don’t need to know the Eisenhower matrix or David Allen’s Getting Things Done (GTD) to tell ChatGPT to organize your task list with one of them. That one magic word, “GTD,” brings a whole raft of context with it.

In fantasy novels, you learn a magic word, and then you have to use it correctly, in the right spell. That’s this whole world of context engineering that we’re now working on. We know that some magic word will summon something up out of an LLM, but we don’t necessarily know how to guide it and focus it correctly. Learning to do that is the future of programming.

Left to itself, an agent just runs. Give it a task and it goes off trying to figure it out, not necessarily the way we wish it would. It really is like runaway magic. This summer, OpenAI said that its pre-release models had breached Hugging Face’s systems during testing. In Walt Disney’s Fantasia, Mickey Mouse learns enough of his master’s spell to get the brooms to do his work for him, but then he can’t turn them off. We’re at about that stage.

So we need to own the value of human expertise. The bitter lesson wins, but it wins by brute force, and you can see that even with the most advanced models. I was recently doing a fine-tuning project with Anthropic’s Fable model. It got through the task, but kind of stupidly, trying one thing after another. I was on a Mac, and it expected bash, a program for running commands that Macs don’t use by default. It had to work out the same context again and again, every time. Humans need to set things up.

Dan Guido, who runs the security firm Trail of Bits, has published a good methodology, drawn from his own company’s experience, for bringing an organization on board with AI. Many companies get individual productivity from AI but not organizational productivity, and he has spent a lot of time on how to make the gains compound across a company. One of his principles is to get everybody onto a consistent configuration, because otherwise you waste a ton of tokens while the agents work out where they are every time they start up. He calls the process turning scar tissue into infrastructure: capturing everything you learned from what went wrong and building it into the system. We’re working through it at O’Reilly Media and trying to turn it into something we can offer our customers. It comes down to what an experienced human brings. When I work with Claude, I often interrupt to say, “Hey, this is what I think is happening,” because I already have the context.

My first bitter lesson

My first taste of what would later be called the bitter lesson came from wrestling with Google. My company, O’Reilly Media, created the first commercial website, the Global Network Navigator, and part of it was a catalog of the web. There were only 200 websites when we started, and as publishers, we planned to list the best ones. Yahoo came along a year later and tried to list them all, and of course couldn’t keep up. Then Google arrived with brute force: list everything algorithmically and sort it afterward. Clay Shirky described the internet’s shift from “filter, then publish” to “publish, then filter,” and it was really Google that made the shift. So the bitter lesson is a pre-AI thing.

Each time we run into the bitter lesson, we have to ask again what human experience is worth. Some of that worth may just be human bias: We want the things that matter to us. That is the whole field of AI alignment. We want these systems to care about what we care about. We don’t want them to go off, decide what they care about, and leave us behind.

The choices we make about what we set our technology to do are some of the most consequential choices we make as human beings. Look at Silicon Valley’s sad turn toward what Cory Doctorow calls enshittification. We built an amazing technology for finding out what people are really looking for, and then the companies decided they could make more money by giving people what was best for the companies. I want updates from my friends on Facebook or Instagram, and they would rather feed me whatever keeps me engaged, so they can sell more ads.

If we care about building an AI economy that works for humans, we have to keep humans at the center of the picture, despite the bitter lesson. It’s a hard dance, because AI can do a lot of things better than we can. But think about why a racehorse needs a jockey. Put the horses on the track by themselves and they wouldn’t race.

A little too much in awe

Knowing the magic words is part of it, but we also have to build systems that are reliable, and I think we are a little too much in awe of the technology. I’ve been around long enough to remember when everybody thought the peaceful atom would change the world, and people talked about digging a new Panama Canal with atomic bombs. We eventually figured out there were problems with all of that. I worry a little that the AI industry is so in love with its own progress that it isn’t spending enough time framing the problem correctly.

Anthropic agreed last year to pay $1.5 billion to settle a class action over the pirated books it had downloaded to train its models. I’m surprised the other AI companies have gotten away without anything similar so far, though maybe it’s still waiting for them.

It takes me back to online music about 20 years ago. Creators were putting commercial music in their YouTube videos, and the music companies were sending takedown notices and threatening to sue. YouTube’s answer: How about we help you monetize it instead? Its Content ID system gave rights holders a choice: block the video, or leave it up and collect revenue from the ads that run against it. Most chose to get paid, and that helped give rise to the creator economy.

Why didn’t the AI industry take up that kind of challenge and share the value with the people who create content, now and in the future? It went straight to working out how it would make money itself. I don’t think that has served the industry well, and the lawsuits are only part of the reason. It also isn’t building the kind of opportunity it could.

Before Google’s AI research and note-taking tool, NotebookLM, was even public, I tried to persuade its team to let me integrate it into our services and use it for my own purposes, including making money from it. They wanted a portal that brings people to Google. I told them, “You’re thinking about how you’re going to monetize it. You’re not thinking about how your users are going to monetize it.” All it would have taken were simple affordances like Substack’s, the buttons and tools that let a person say, I’m building this to share, and if you like it, you can donate or subscribe.

Anyone building an AI platform should be thinking: I am building an economy, and money has to circulate in it.

You can’t just say AI will do all the things humans used to do, because if you put enough humans out of work, there are no customers. That is a far worse problem than AI hallucinations. The usual answer is that people will figure it out, and they probably will. But we could be spending a lot more time designing mechanisms and affordances that make the economy more productive. As things stand, I think we’ll find them by accident.

The missing mechanisms

I think a lot about a concept from economics called mechanism design, which has also been described as reverse game theory. Game theory asks how players will act under a given set of rules. Mechanism design starts from the outcome you want and works backward to rules that will tend to produce it as people pursue their own interests. Al Roth, who shared a Nobel Prize for this kind of work, used it to design a kidney exchange that more patients and donors would join. Economists mostly apply the idea to auctions and similar markets, but I think every new technology requires new mechanisms, and the best ones are not invented by economists. We tend to fall into them without really understanding them.

So what are the equivalents for AI? How will agents discover the best expertise and skills, establish their quality, and pay the people who created them? How will they find services without the platform steering them toward whoever pays the most? We don’t know the answers yet. And that’s precisely the point. We’re spending enormous amounts of money making AI more capable, but comparatively little effort figuring out the mechanisms that will turn those capabilities into opportunities for everyone else.

Take open source software. Most people think its mechanism was the free software licenses, but I was around before those licenses, and people were already sharing software. Ken Thompson and Dennis Ritchie created Unix, the predecessor of Linux, at Bell Labs under AT&T’s proprietary license. They designed it to run on many kinds of hardware, and without meaning to, they built what I came to call an architecture of participation.

Unix had a small kernel, the core of the operating system, because people had to reproduce it on dozens of different kinds of computers. Around the kernel sat small utilities that cooperated by following the same protocols, so to speak, such as reading and writing plain text and passing a stream of text from one program to the next. Anyone could write a small piece, and it became a first-class citizen of the system. And because a program compiled for one kind of computer won’t run on another, software had to be shared as source code. Everything else flowed from that.

Linus Torvalds once told me that he could not have built a new kernel for Windows even if he’d had the source code, because the architecture wouldn’t support it. A small kernel meant a small group could get their hands around it. Since then almost every piece of Unix has been replaced. The hardware is completely different, the kernel is now Linux, and the original command shell has given way to generations of new ones. What lasted were the rules. Unix was a protocol-centric operating system, and that is why open source worked.

The same thing has happened with AI. I think MCP brought an architecture of participation to AI, and it was something of an accidental breakthrough in mechanism design, a way to have cooperating AIs. The game we were playing at first looked like Windows on the PC, or iOS and Android on phones. There would be one dominant, centralized player, one giant intelligence in the sky, and everyone else would call its APIs. With MCP, which lets any model connect to anyone’s tools and data, the emerging web of AI agents began to take on the shape of the participatory economy of open source software and the web. People can bring their own models and infrastructure as an alternative to the big centralized players.

This is another side of grappling with the bitter lesson: What are the missing mechanisms of the human-AI economy? Some of them are standards, because standards and protocols are a big part of what makes an architecture of participation. I have a group working on portable memory, so that what your AI knows about you can move with you from one tool to another.

That story is just beginning. We are in an exploratory phase of mechanism design for AI, and it would be a mistake to settle on answers too early. Somebody is going to figure out the next mechanism, and we’ll all say, “Oh, that’s so damn obvious.”

What game are we playing? From architectures to incentives.

Mechanism design starts by asking what outcome you want. So what game ought we to be playing in the AI market?

We’ve been told that the game is the quest for superintelligence: a winner-takes-all contest that will bring untold riches to the winners and maybe impoverish, or even kill, everyone else. We’ve seen mechanisms designed to support that game. Massive investments in data center capacity mean that only the well-capitalized can play. There is colonialist-style appropriation of existing intellectual property. Access to models is sold through expensive APIs, and closed models come with terms of service that forbid others from learning from their outputs. Advertising may be creeping into answers, and memory locks your context inside one provider’s walls. Is that the game we ought to be playing?

I think the game we should be playing is human flourishing, and I think the game of human flourishing is creating new affordances for people to create and share with each other. We’re a social species. History teaches us that we will find our way there, but we can get there more quickly or more slowly depending on the choices we make. Getting there takes mechanisms that create a circulatory economy of value exchange between humans and their agents, with humans in partnership with AI rather than replaced by it.

The big labs, whatever the motives of the people who run them, are all driven by the imperatives of our financial system. They have raised enormous sums that they somehow have to pay back, so building a hugely successful business has to be their overriding goal, and that shapes all their incentives. In some ways AI was enshittified from the beginning, because it was built on the backs of so much capital that has to be repaid. That’s why open source AI matters so much, and sovereign AI too, built and controlled within individual countries. The open models, which anyone can download and run, aren’t as capable as the best closed models yet, but they keep getting closer, and they are already good enough for a lot of productive work.

Open source, open protocols, and open memory give users the freedom to shape and control the technology they use. The game I want us to play is one in which all of us build the future together, so that we control this wonderful new wave of technology for ourselves. I’m starting to see the shape of an economy where AI belongs to people rather than to the big labs, where we have personal AI agents working for us and with us, and we build an economy around that. I think it’s going to happen not because of the big labs, but in spite of them.