Most of today’s AI entrepreneurs and social media enthusiasts are young. AI is the first technology revolution they’ve lived through. They don’t remember how unbelievably exciting it was in the late 1970s to get your own computer, instead of struggling to get time on one of the vast machines chugging away in what used to be called “the glass house,” an air-conditioned room where from outside you could watch the nine-track tapes being accessed and the refrigerator-sized disk drives spinning. Input and output was on high-speed teletype printers (no screens) and magnetic tape (and before that on punch cards.) They also mostly don’t remember, as I do, writing assembly-language programs in which you moved data from one computer register to another, specified the arithmetic or logical operations to be performed on it, and then specified where to save the result.

They look at today’s AI and marvel at how virtually anyone can now generate code. They don’t realize that that’s also how it felt when you could use an interpreted programming language like BASIC to write your own applications instead of writing them in Assembler. They also don’t remember when Dan Bricklin used BASIC to write the first version of VisiCalc in 1979. He went into business with Bob Frankston, who rewrote it in 6502 Assembly Language for the Apple II. Dan was able to express the idea in Basic; Bob then wrote a tight, efficient version for widespread distribution. People rushed out to buy computers so they could use VisiCalc, and everyone predicted that the VisiCalc “spreadsheet” would put all the accountants out of work. It was the first program to be referred to as a “killer app.” It wasn’t the democratization of programming itself that was the killer app but what Dan and Bob did with it. In the entrepreneurial explosion that followed, databases, graphics, word processing, and other business applications came out of the glass house and, as Bill Gates put it, “onto every desk and into every home.”

We saw the same leap from astonishment to innovation to adoption with the World Wide Web. When O’Reilly demonstrated GNN, the first commercial website, to people at a trade show in 1993, attendees couldn’t believe that clicking on a link could pop up an image or a document from another computer thousands of miles away. They didn’t understand the difference between the software application (the web browser) from the network-connected documents with clickable hyperlinks that it acted as a frame for. They didn’t understand that web pages were a new kind of software written in a simple markup language anyone could understand. It didn’t make sense that the web page was a document that in many ways could look and act like a kind of program. And when people did understand, they thought that might be the end of programming.

In 1993, everyone thought that the web browser itself was the killer app, and that Marc Andreessen, the young creator of Mosaic, the most popular web browser, which became Netscape, was going to dethrone Bill Gates as the new king of computing. They didn’t realize that the true killer apps of the web were not the web browser and the web server but applications “up the stack” built using the new programming paradigm the web enabled. Those applications weren’t software as we’d known software before. They were database- and application-backed websites delivering services that were previously impossible. I remember thinking of Amazon as the 2005 equivalent to VisiCalc when a friend told me she had just bought her first computer so she could shop on Amazon. Once again, it wasn’t the new capability itself that was the killer app but what developers did with it.

What makes a killer app? I think that Tom Stoppard nailed it in his play Rosencrantz and Guildenstern Are Dead:

A man breaking his journey between one place and another at a third place of no name, character, population or significance, sees a unicorn cross his path and disappear. That in itself is startling, but there are precedents for mystical encounters of various kinds, or to be less extreme, a choice of persuasions to put it down to fancy; until—“My God,” says a second man, “I must be dreaming, I thought I saw a unicorn.” At which point, a dimension is added that makes the experience as alarming as it will ever be. A third witness, you understand, adds no further dimension but only spreads it thinner, and a fourth thinner still, and the more witnesses there are the thinner it gets and the more reasonable it becomes until it is as thin as reality, the name we give to the common experience.

That is, a killer app is at first a startling window onto an undreamed-of possibility, but one that so changes the paradigm that it rapidly gets taken for granted. In the meantime, everyone tries to make sense of the new world. Usually, their sensemaking is to try to reduce it to what they already know. The breakthrough entrepreneurs see more deeply and go farther.

In 1995 and 1996, almost everyone saw the future as a showdown between Microsoft and Netscape over who would own the web server and the web browser, and whether the web would supplant the desktop as the new platform. Microsoft won the browser wars, but it turned out to matter far less than anyone expected. The value went to the companies building on top of the browser, the ones that figured out what the network made possible.

The current narrative pitting AI model providers against each other sounds a lot like Microsoft versus Netscape to me. It is the platform war, the war for the layer that will become the common experience on which the revolutionary future will be built. ChatGPT, Claude, and Gemini are the browsers of this era. That is not a small thing. But the browser wasn’t actually the killer app of the web. Netscape made money for a little while. Amazon and Google and the explosion of other sites were what made the web into something people couldn’t live without.

Claude Code, Codex, Cursor, and the other coding agents may be the BASIC of this era, or like the tools and frameworks that made database-backed web applications possible. They are plumbing. Important plumbing, and plumbing that has to be in place before the future can truly happen. But still just plumbing. Inventors tend to spend decades maturing it, and O’Reilly tends to spend decades teaching people to build it. I suspect that will be just as true this time around.

In short, I’ve started asking myself whether we’ve actually seen the killer app for AI yet. The obvious assumption is that ChatGPT was. It certainly got a LOT of people excited and onboard very, very quickly. But maybe Claude Code was, enabling the breakthrough in agentic coding. Or maybe Hermes or Dots or Muse or some other personal agent will be. But we should take seriously the possibility that the killer app is not yet on the scene. These initial candidates for the killer app look too much like advances in a new enabling platform and not enough like a new generation of applications.

Personal agents may turn out to be a breakthrough category, but right now they look to me like early ecommerce before Amazon got it right.¹ For Amazon to work, the web had to become secure enough for people to use credit cards online. The apps were waiting on missing mechanisms. For Google’s economic model to kick into gear, pay-per-click advertising had to be invented. I suspect we are in a similar moment waiting for crucial pieces to be dumped on the table so developers can complete their part of the puzzle. For AI agents to act on our behalf in the world, they will need their own equivalents: ways to pay, to prove who they are acting for, to be held accountable, and for the counterparties to settle on the allocation of the value created by the transaction. Ilan Strauss and I have argued in “The Missing Mechanisms of the Agentic Economy” that many of those mechanisms don’t exist yet. Maybe the killer apps are waiting on the equivalent of https and pay per click.

In short, when you listen to the AI conversation and mostly hear about people’s programming pipelines and how much more productive their programmers are, and very little about what they are using that productivity to build, you can be pretty confident that the killer apps haven’t arrived yet.

Where are we in the cycle?

Let’s line up the calendars and see if that tells us something useful.

CP/M (Control Program for Microcomputers), the first widespread personal computer operating system, was developed in 1974, and the Altair 8800, the first mass-market personal computer, appeared on the cover of Popular Electronics in January 1975. Microsoft released a BASIC interpreter for it in June of that same year. A host of competing personal computer hardware platforms running CP/M were introduced soon thereafter. The Apple II was released in 1977, VisiCalc in October 1979, and dBase II in 1980. The IBM PC came out in August 1981 with Microsoft’s MS-DOS as its operating system. Lotus 1-2-3 (which included spreadsheet, graphing, and simple database features) was released in January 1983 and turned the PC into a business necessity.

In 1984, the revolutionary graphical user interface of the Macintosh was the next step function jump in the PC era. The GUI itself had been invented at Xerox PARC in the 1970s, but it wasn’t till Steve Jobs built his “insanely great” Mac that the world embraced it. Desktop publishing became the killer app of the GUI, with Aldus PageMaker and the Apple LaserWriter in 1985. The PC proper didn’t get the GUI till Windows 3.0 in 1990, but Microsoft’s embrace of it for applications such as Word (1983), Excel (1985), and PowerPoint (1987) as well as for the operating system and programming environment made it, for the time, master of the computing world.

So you can see a rough timeline. Four or five years from the introduction of the first version of what was to become the new platform to the first killer app. (Or two, if you count from the introduction of the Apple II, the platform for which VisiCalc was first written.) Then 10 years of invention, competition, and refinement, followed by fairly rapid consolidation and accumulation of market power.

Let’s do the same timeline for the web. Email and online forums might well have been the first internet killer apps, but they were really holdovers from previous generations of research networks. They were apps that had been evolving since the 1970s, though CompuServe and AOL had begun to bring them to the general public in the ’80s. The world really changed, though, when Tim Berners-Lee first wrote his proposal for what became the World Wide Web in March 1989. It was released to the world in August 1991. This was the new platform and the start of what would become the new programming paradigm.

Not too much happened outside physics labs and a few universities until 1993. (There were only about 200 websites when we published The Whole Internet User’s Guide and Catalog in November 1992.) In 1993, NCSA, the National Center for Supercomputer Applications, released Mosaic and we launched GNN, the first commercial website. Netscape shipped its browser, a follow-on to Mosaic, at the end of 1994. Yahoo! was launched that year too, Amazon, eBay, and the Altavista search engine in 1995, MapQuest in 1996, Google in 1998, and Wikipedia in 2001.

So, again about four years from the release of the platform to the appearance of the first killer app, and another 10 till the consolidation phase. (I see Gmail 2004, Google Maps 2005, Google Docs 2006, and Chrome 2008 as the equivalent of Microsoft’s release of Office on top of Windows as the beginning of the consolidation phase of the web.)

The timeline for mobile is not dissimilar. The iPhone in 2007, Android in 2008. Killer social apps Facebook and Twitter had gotten their start on the web at about the same time, but really came into their own on mobile over the next few years. The “could only have happened on mobile” killer apps of Uber, Lyft, and the on-demand economy didn’t arrive till 2011/2012.

A lot of people seem to think AI is moving way faster than previous technology revolutions. But if you line up the calendars, AI is moving at about the same pace as previous eras. Treat the transformer paper, “Attention Is All You Need,” published in June 2017, as the AI equivalent of Tim Berners-Lee’s 1989 proposal, and ChatGPT’s first release on November 30, 2022, as the equivalent of Mosaic. The gap from invention to breakout is about the same, five years for the web and five and a half for AI. That’s just getting the platform in place. The AI-native “killer apps” haven’t yet arrived but are due soon. (And please, let’s avoid the jokes that the “killer apps” of AI first showed up in the Russia-Ukraine drone war. That’s true, but not on the direct line of my argument here.)

On what seems to be a fairly consistent timeline, where are we today, then? October 2026 corresponds to roughly the spring of 1997. Amazon was about to go public. Hotmail was a year old. Google was still a research project at Stanford called BackRub. SixDegrees had kicked off social networking, but Facebook was still seven years off. Napster was still two years away.

Think of all those amazing application categories that had been pioneered, though. Search, ecommerce, email, and social networking were all in their infancy, but they were out there, delivering real value. I’m not sure that I see the equivalent time to value in AI. AI didn’t even get to Claude Code, its Microsoft Basic equivalent, till 2025! By some measures, AI is moving more slowly rather than more quickly. There’s more change to digest, and developers are still struggling with the tooling and developing the new knowledge that is needed.

Why so slow, with so much money?

The four largest hyperscalers are on track to spend roughly $700 billion in capital expenditure this year, nearly double what they spent in 2025. That is orders of magnitude more capital than went into the web in the 1990s. If capital and capability were the limiting factors, you’d think we should be well ahead of the earlier calendar, not behind.

I suspect that at least part of the answer is that Silicon Valley’s quest for premature monopoly may be making it less efficient at finding breakthroughs rather than more. Starting with the era of Uber and Lyft, massive VCs have canonized the winners at the beginning of the race, crowding out alternate technologies and business models. That may be happening here. But it’s also just in the nature of bubbles, which depend on groupthink to flourish.

And part of that groupthink is about where the value is going to be captured, rather than where it is going to be created. That is almost always backward looking. During the railroad boom, capital was pouring into building the tracks, and the new business models (think Montgomery Ward and Sears) that railroad enabled didn’t come till later. The dot-com bubble was far less capital intensive than people think. The telecom buildout that ran alongside it was where the money got spent. US carriers spent about $97 billion on capital in 2000 alone.

By 2002, according to an FDIC analysis, only 2% to 5% of the fiber in the ground was actually lit. More than 60 telecom companies went bankrupt. Once the data centers and dark fiber of the telecom bubble could be bought for pennies on the dollar, they enabled a generation of new applications like streaming video. YouTube was founded in 2005, after the crash, on cheap bandwidth that someone else had paid for before they went out of business. There were some visionary companies (think Google, Amazon) that had been started before the abundance arrived, but many more were founded after the bubble burst.

Carlota Perez’s book Technological Revolutions and Financial Capital documented this pattern across every technological revolution since the canal era. Financial capital pours into the installation of the new infrastructure, a bubble forms and bursts, and the deployment period that follows, when the technology actually reshapes the economy, is built by different people on top of what the crash left behind. If that is where we are in the cycle, then the absence of killer apps is not a puzzle at all.

The current data center buildout and the massive investments in frontier AI may not be the future. They are the infrastructure on which the future will be built. We have to ask ourselves what will we build once competition and overshoot and the bursting of the data center bubble make intelligence infrastructure cheap.

Help me fill in the picture

I see several possibilities. Either we are still early and the killer apps simply haven’t shown up yet, the way Google hadn’t shown up in 1997. Or AI is going more slowly than its boosters claim, in spite of all the capital, because capital and model capability are not the limiting factors. But it’s also possible that the killer apps are here, and I’m not hearing about them because the AI conversation is so focused on AI capability and nobody is telling the story of what they are accomplishing with it beyond building better developer tools.

I would like to know which. We all recognize a killer app when we see one. Amazon was a bookstore with more books than any physical bookstore could hold. Wikipedia was an encyclopedia written entirely by volunteers and was updated in real time within hours and sometimes even minutes of major events. Google Maps meant you always knew where you were and how to get anywhere else. If you know of an AI application, beyond the chatbot or agentic coding environment itself, that changes what ordinary people can do on that scale, or a business that does something an order of magnitude cheaper or better than the one it replaces, I want to hear about it. Tell me what you’re building in the Google Form below, and describe what tools you’re using only in the context of what they let you produce for others. If you’re a software developer using AI, what are you producing that was unthinkable before AI?

Footnote

1. CommerceNet demonstrated the first encrypted credit card transaction online in 1994, and Netscape made the idea mainstream when it introduced HTTPS (HyperText Transfer Protocol Secure) and made it available in the Netscape browser in that same year. This made it safe to type a credit card number onto a web page. ︎

The following article was originally published on James Shore’s Successful Software blog and is being republished here with the author’s permission.

AI is great for making teams faster. But is it sustainable? What about maintainability, burnout, and lock-in?

This is part 3 of my series on assessing AI’s impact on software development. In part 1, we looked at why assessing impact is important, and what not to do. In part 2, we looked at how to measure AI’s impact on delivery speed. In this part, we’re looking at how to measure and model the unintended consequences of using AI. Then we’ll wrap up in part 4 with an examination of business outcomes. Finally, an epilogue puts it all together with notes you can share with your CFO.

Does maintainability even matter?

I can hear it now: “Who cares about namby-pamby issues like maintainability and burnout? It’s a job. If jobs were supposed to be fun, they wouldn’t be called ‘work!’ Suck it up!”

So let me be clear: We care about AI’s unintended consequences because they have real economic impact. If code isn’t maintainable, features take longer to write and bugs take longer to fix, taking time away from more valuable work. If people are burned out, they don’t put forth their full effort, and they eventually quit, taking institutional knowledge with them and leading to time-consuming hiring cycles. If you’re locked in, and AI token costs increase, you could be stuck with a choice between a big bill and cutting back on development.

Among the people I’ve talked to, there’s widespread agreement that heavy use of AI produces code that humans have trouble understanding. But there’s disagreement about how much that matters. If AI is doing the maintenance, do we care if a human can understand the code?

There are strong opinions on this topic on all sides, but not a lot of actual evidence. What’s clear is that the costs of maintenance and other unintended consequences do matter, but only if there are any costs. This is what we need to measure: Is there an impact? And if there is, is it enough to matter?

Unfortunately, unlike the rigorous speed trial we discussed in part 2, unintended consequences are gradual and fuzzy. Your measurements are going to be noisy and lagging. You’ll have to create models and explore scenarios. Ultimately, your decisions will come down to judgment, informed by your measurements and models but not driven by them.

One ring to rule them all

There is one idea that, while not perfect, does a good job of capturing the majority of unintended consequences. It’s cost of change.

Put simply, cost of change is like a tax on the changes you make to your software. When cost of change is low, complicated changes still take time, but they’re achievable. When cost of change is high, even simple changes can take weeks, and complicated changes seem completely out of reach.

Cost of change isn’t easy to measure, but there are several indicators you can use to inform your judgment.

Value-add percentage

Value-add percentage is the percentage of your software development effort that’s spent on things that bring value to your business, such as new features, versus the percentage of effort that’s spent on non-value-add work, such as fixing bugs, resolving incidents, and manual test and release cycles.

Non-value-add work is called “muda.”¹ Some muda is necessary, even valuable, but that doesn’t make it value-add. Imagine a demanding and crotchety customer who always asks, “Why didn’t you just do it right the first time?” If that customer won’t pay for something, it’s muda.

That’s why bug fixes are muda: “I’m not paying you to fix a bug that shouldn’t be there in the first place!” And performing releases: “Why don’t you just ship it already! How hard can it be?” Sure, bug fixes can be valuable, and releases are necessary, but they’re still muda.

I find it useful to distinguish muda further into discretionary and nondiscretionary muda. Discretionary muda is work that you choose, such as investing in code changes that decrease development costs. Nondiscretionary muda is work that’s chosen for you, such as bug fixes, incidents, and dependency upgrades.

You’ll always have a baseline amount of nondiscretionary muda; it’s a constant tax that takes time away from other work. But if it’s rising, that means the time you spend on value-add work is going down. You have less time to work on things that matter. Your costs are increasing.

When I was VP of engineering, value-add percentage was one of my key metrics. I made decreasing muda a priority, and it paid off. We doubled our value-add percentage over the course of my three years there, which allowed me to free up people for substantial new initiatives without increasing headcount.

DORA metrics

Your organization is likely already tracking the DORA metrics of change lead time, deployment frequency, change fail rate, and so forth. If not, they’re worth considering. As with value-add percentage, changes in these numbers are something to keep an eye on.

Token usage

Tokens are a literal cost, and the more tokens it takes to make a change, the higher your costs. Unfortunately, token usage can fluctuate for many reasons, not just cost of change. Still, it’s worth watching. Anecdotally, better-constructed software is easier for AI to work with and costs fewer tokens.

One of the biggest variances in token usage is people’s level of adoption. Rather than just tracking overall usage, keep an eye on individual usage changes. In the charts I’ve seen, usage has formed a Pareto curve. Increases in usage by people on the low end of the curve may just represent increasing familiarity. Increases by people on the high end of the curve are worth investigating further.

Changes in models, harnesses, prompts, and more will all affect token usage. This metric is only advisory; it’s meant to trigger further investigation, not to be used on its own.

One ring to find them

Of these metrics, value-add percentage is the one I trust the most. It operates at the level we care about: Where is the time going?

Tracking value-add percentage doesn’t have to be a big burden. You don’t need time sheets or detailed task tracking. The trick is to not care overly much about precision. Tracking work within a day or two is good enough.

When I did this tracking, I piggybacked on our valuable increment (VI) tracking. Each one was several days to several weeks of work and could be cleanly categorized as either value-add or muda. But you don’t have to use VIs; any tracking approach will work.

Don’t get too fine-grained with your tracking. When I was VP, managers checked in with each team twice a week and recorded which VI the team had worked on and how many people had worked on it. Each VI was categorized as “muda” or “value-add,” and we had a bunch of subcategories that helped us understand where time was going, such as “bug,” “capacity investment,” “major initiative,” and so forth. It was lightweight and very useful.

Review your value-add percentage monthly. If it’s going down, you don’t necessarily have a problem, but it’s worth investigating further.

The biggest problem with value-add percentage is that it’s noisy. My experience was that trends took three to six months to show up. It took a year before I was confident stating that my cost-of-change reduction efforts had led to lasting results.

That’s a long lag time, but it’s still worth tracking. My product partners and I found value-add percentage incredibly valuable as a tool for communicating what was happening in product development. And, sadly, improving cost of change is a lot harder than making it worse. If AI is causing serious problems, it’ll probably show up more quickly.

One ring to bring them all

Because your cost of change indicators will be noisy and uncertain, supplement them with models and scenario planning. Your models won’t tell you what the future holds, but they will tell you what it could hold, which will help you manage risks.

Here are some of the models I’ve found useful:

Maintainability

Model maintenance as a mandatory “tax” on development. For each month of value-add development, estimate a certain amount of maintenance in the following year, and then a lower amount of ongoing maintenance in every year after that. (Wisdom of the crowd estimation may help.) Then build a model that shows how many engineering days are available each month. Subtract the time spent on maintenance, and the rest is available for value-add.

To model AI’s impact, apply your measured development speed improvements. Estimate how AI will change maintenance costs and estimate effects on headcount. Then explore scenarios based on varying costs of AI maintenance.

If you only create one model, make it this one. I have a more detailed explanation here. It includes a spreadsheet you can use as a starting point for your model.

For example, using that model, if AI increases our speed by 50%, but token costs reduce headcount by 15% and maintenance costs are estimated to double, I can see that I’m only breaking even, despite the big performance improvement:

Excess lines of code

Lines of code aren’t a good productivity measure, but they are a reasonable cost measure. Generally speaking, more lines of code result in more maintenance costs.

This model works similarly to the previous one but uses lines of code instead of engineering days. Use traceability data to categorize commits as “value add” or “maintenance.” For each commit, measure gross code churn: the number of lines of code that were either changed, added, or deleted.

For each line of value-add change, estimate a certain amount of maintenance code churn in the following year, and then a lower amount in every year after that. Model how much maintenance code churn will be needed each month based on the value-add churn of previous months. (See the “tmux” tab of this spreadsheet for an example.)

To model AI’s impact, measure how AI affects code churn during your speed improvement trial. Then project value-add churn into the future and model how maintenance churn could change.

The advantage of this model is that you can use (part of) your actual commit data to refine your estimates. For example, the following graph uses real commit data from the open source tmux project.² Prior to running the model, I guessed one line of maintenance for every line of value-add in the first year, and 0.1 lines every year after that, which turned out to be surprisingly close to the actual data.

Lock-in

If token costs increase, how will you respond? Will you cut headcount? Reduce your use of AI? Switch to a less capable model?

To model these effects, update the “maintainability” model. Reduce your monthly engineering capacity according to headcount and estimate reductions to your speed improvements based on AI reductions. You may want to add a separate set of estimates and calculations for humans maintaining AI-written code.

Burnout

AI Disclaimer: This image was created with OpenAI’s GPT-5.6 Sol model operating in “High” thinking mode. See the ChatGPT conversation here.

Productivity is likely to go down as mental exhaustion goes up, and turnover is likely to increase. I don’t think you can model these effects with any accuracy, but you can still keep an eye on them.

Include mental exhaustion as part of your development speed study to see if there’s a significant difference in mental exhaustion between the two approaches. Follow up with periodic one-question surveys about people’s mental energy levels. Pay attention to actual turnover relative to historical numbers, and look for complaints about AI in exit interviews.

And in the darkness bind them

There’s a dark side to value-add percentage. It’s incredibly easy to game. As soon as managers think their teams are being judged based on how they categorize their work, the temptation to tweak the categorization in their favor will be exquisite.

It doesn’t even have to be fraudulent. A leader with prioritization authority and something to prove can simply decide to defer maintenance in favor of value-add work. The pressure to do so is already immense, and it takes unusually strong character to resist.

(If you really hate your company, consider this: You can get away with deferring maintenance for a couple of years. Just enough time to line up a bigger title at a different company! Then your successor is left holding the bag, and your old company wonders why product development can’t get anything done.)

I don’t have an easy answer to these problems other than to remind you to use these metrics to evaluate your approach, not your people. There is no metric that can’t be gamed, and this one is more gameable than most. But it’s also critical for understanding AI’s possible downsides. Use it, but use it with caution.

Development speed and unintended consequences are two sides of the same coin. They help us understand how AI changes how much we deliver, both now and in the future. But just delivering more software isn’t enough. We need to produce the right software. We’ll look at business outcomes next, in part 4.

Footnotes

1. “Muda” is a Japanese word that comes to us from the Toyota Production System, the inspiration for the Lean Software Development movement. ︎
2. Many thanks to Christophe Stjernlöf for providing the raw data from the tmux project. He used an LLM to classify commits as “value-add” or “maintenance.” The stairsteps are an artifact of the way the data was collected. ︎

The following article originally appeared on the Asimov’s Addendum Substack and is being republished here with the author’s permission.

This essay is based on my conversation with Demetrios Brinkmann for AAIF Live, recorded before my keynote at AGNTCon + MCPCon North America in San Jose on October 23, 2026. It was edited and rearranged for publication by Ilan Strauss, and in places it draws on the keynote. Both videos are also on the AI Disclosures Project website in the new “Missing Mechanisms” page.

The keynote begins at 1:28:40 in the Day 2 livestream.

The Bitter Lesson vs. AI Markets for Human Expertise

In 2019, the computer scientist Richard Sutton wrote a short essay called “The Bitter Lesson.” Looking back over 70 years of AI research, he concluded that general methods that scale with raw computation beat methods built on human expertise, and by a large margin. The AI industry as a whole is bitter-lesson-pilled, to borrow the internet’s slang for a true believer. I am human-pilled.

I believe in the bitter lesson. Brute force wins out in the end, and I have wrestled with that fact throughout my career. But we humans still have to figure out our role in the AI economy, and I find it rather disturbing that the AI community isn’t spending more time asking how to make this future friendlier for humans. The labs think about how to keep AI from going rogue, and their chief economists study job losses from AI. They don’t study what mechanisms we could introduce to create new economic opportunities for humans.

The question matters to us at O’Reilly Media. We don’t think of ourselves as a publisher, a conference producer, or an online learning platform so much as a company that transfers knowledge from the people at the cutting edge to the people who want to follow them. So we are very interested in what human expertise is worth, even in the face of AI.

Knowing everything and nothing

Human expertise was used to train AI, which makes AI a fabulous cultural technology. Thousands of years ago, the only knowledge people had was what they could keep in their heads. Since then we have built better and better ways to share knowledge across time and space, and AI continues that. With each of these transitions, we have to ask again what the truly human is still worth. People often say judgment, or taste. That’s true, but I think it goes deeper:

AI is a sea of possibility. All of that knowledge sits in the model, and nothing sets it in motion except a human. (That is changing a little as we start giving AI agents long-running tasks, and we will have to come to grips with it.) If you and I used Claude or ChatGPT for the same research project, we would each bring different experience to it, tell it what we want and why, and give it context in different ways. I see that as the unique focusing power of human consciousness and attention.

Earlier this year I spent an hour or so talking this question over with Claude. When it offered to write up a first draft for me, I said no. I wanted an essay from its own point of view about what it took away from our conversation. The result, “Why AI Needs Us,” opens, “He came in through Powys,” and a few paragraphs later says, “The choice of entry point was already the argument.” That captured what I’m getting at. I had come into the conversation through a book I love, John Cowper Powys’s The Meaning of Culture, which distinguishes culture from education. Powys gives his own unique and powerful definition: “Culture is what is left over after you have forgotten all you have definitely set out to learn.” It is what you absorbed. Each of us absorbs a small and particular slice of everything. Large language models (LLMs) try to absorb everything, and so become overly general; we give them focus. Human knowledge provides them with a lens.

Let me bring that into a business context. Andrew Odewahn, O’Reilly Media’s CTO, built a research assistant to help with managing technology in IT organizations. You teach it your infrastructure: the tools you use, and where the definitive information about each one lives. When one of his engineering managers said the team was overloaded and asked to hire another person, Andrew turned the agent loose on the department’s work, and it agreed that the group needed another hire. Then, on impulse, he tried something different. O’Reilly Media has an MCP server we call O’Reilly Expert Intelligence. (MCP, the Model Context Protocol, is an open standard for connecting AI models to outside tools and data.) It lets a model answer a question through the filter of our own body of expert knowledge. Andrew asked what the O’Reilly experts would say.

Seen through the lens of Google’s book Site Reliability Engineering, the model said, it was totally rethinking the problem. The book describes toil, the manual, repetitive work of keeping systems running, which grows as the systems grow. After analyzing the group’s toil, the model concluded that another hire would make the problem worse. The group needed a toil audit to find out why there was so much overhead, and then a restructuring.

Much of that knowledge was already in the model, since the AI companies have more or less pirated our books. It came back with a completely different answer simply because it was told to consult a specific group of experts. I think there’s a metaphor there. An LLM knows everything, and as a result it doesn’t know anything. We humans are so much more limited, but because we focus on a particular problem, we bring things together in a different way.

Magic words

Last year I wrote a piece called “Magic Words” about how some words carry far more context than others. It sounds like a trivial observation, but if you unpack it, it leads in interesting directions. One of our trainers, who teaches a course on productivity, pointed out that you don’t need to know the Eisenhower matrix or David Allen’s Getting Things Done (GTD) to tell ChatGPT to organize your task list with one of them. That one magic word, “GTD,” brings a whole raft of context with it.

In fantasy novels, you learn a magic word, and then you have to use it correctly, in the right spell. That’s this whole world of context engineering that we’re now working on. We know that some magic word will summon something up out of an LLM, but we don’t necessarily know how to guide it and focus it correctly. Learning to do that is the future of programming.

Left to itself, an agent just runs. Give it a task and it goes off trying to figure it out, not necessarily the way we wish it would. It really is like runaway magic. This summer, OpenAI said that its pre-release models had breached Hugging Face’s systems during testing. In Walt Disney’s Fantasia, Mickey Mouse learns enough of his master’s spell to get the brooms to do his work for him, but then he can’t turn them off. We’re at about that stage.

So we need to own the value of human expertise. The bitter lesson wins, but it wins by brute force, and you can see that even with the most advanced models. I was recently doing a fine-tuning project with Anthropic’s Fable model. It got through the task, but kind of stupidly, trying one thing after another. I was on a Mac, and it expected bash, a program for running commands that Macs don’t use by default. It had to work out the same context again and again, every time. Humans need to set things up.

Dan Guido, who runs the security firm Trail of Bits, has published a good methodology, drawn from his own company’s experience, for bringing an organization on board with AI. Many companies get individual productivity from AI but not organizational productivity, and he has spent a lot of time on how to make the gains compound across a company. One of his principles is to get everybody onto a consistent configuration, because otherwise you waste a ton of tokens while the agents work out where they are every time they start up. He calls the process turning scar tissue into infrastructure: capturing everything you learned from what went wrong and building it into the system. We’re working through it at O’Reilly Media and trying to turn it into something we can offer our customers. It comes down to what an experienced human brings. When I work with Claude, I often interrupt to say, “Hey, this is what I think is happening,” because I already have the context.

My first bitter lesson

My first taste of what would later be called the bitter lesson came from wrestling with Google. My company, O’Reilly Media, created the first commercial website, the Global Network Navigator, and part of it was a catalog of the web. There were only 200 websites when we started, and as publishers, we planned to list the best ones. Yahoo came along a year later and tried to list them all, and of course couldn’t keep up. Then Google arrived with brute force: list everything algorithmically and sort it afterward. Clay Shirky described the internet’s shift from “filter, then publish” to “publish, then filter,” and it was really Google that made the shift. So the bitter lesson is a pre-AI thing.

Each time we run into the bitter lesson, we have to ask again what human experience is worth. Some of that worth may just be human bias: We want the things that matter to us. That is the whole field of AI alignment. We want these systems to care about what we care about. We don’t want them to go off, decide what they care about, and leave us behind.

The choices we make about what we set our technology to do are some of the most consequential choices we make as human beings. Look at Silicon Valley’s sad turn toward what Cory Doctorow calls enshittification. We built an amazing technology for finding out what people are really looking for, and then the companies decided they could make more money by giving people what was best for the companies. I want updates from my friends on Facebook or Instagram, and they would rather feed me whatever keeps me engaged, so they can sell more ads.

If we care about building an AI economy that works for humans, we have to keep humans at the center of the picture, despite the bitter lesson. It’s a hard dance, because AI can do a lot of things better than we can. But think about why a racehorse needs a jockey. Put the horses on the track by themselves and they wouldn’t race.

A little too much in awe

Knowing the magic words is part of it, but we also have to build systems that are reliable, and I think we are a little too much in awe of the technology. I’ve been around long enough to remember when everybody thought the peaceful atom would change the world, and people talked about digging a new Panama Canal with atomic bombs. We eventually figured out there were problems with all of that. I worry a little that the AI industry is so in love with its own progress that it isn’t spending enough time framing the problem correctly.

Anthropic agreed last year to pay $1.5 billion to settle a class action over the pirated books it had downloaded to train its models. I’m surprised the other AI companies have gotten away without anything similar so far, though maybe it’s still waiting for them.

It takes me back to online music about 20 years ago. Creators were putting commercial music in their YouTube videos, and the music companies were sending takedown notices and threatening to sue. YouTube’s answer: How about we help you monetize it instead? Its Content ID system gave rights holders a choice: block the video, or leave it up and collect revenue from the ads that run against it. Most chose to get paid, and that helped give rise to the creator economy.

Why didn’t the AI industry take up that kind of challenge and share the value with the people who create content, now and in the future? It went straight to working out how it would make money itself. I don’t think that has served the industry well, and the lawsuits are only part of the reason. It also isn’t building the kind of opportunity it could.

Before Google’s AI research and note-taking tool, NotebookLM, was even public, I tried to persuade its team to let me integrate it into our services and use it for my own purposes, including making money from it. They wanted a portal that brings people to Google. I told them, “You’re thinking about how you’re going to monetize it. You’re not thinking about how your users are going to monetize it.” All it would have taken were simple affordances like Substack’s, the buttons and tools that let a person say, I’m building this to share, and if you like it, you can donate or subscribe.

Anyone building an AI platform should be thinking: I am building an economy, and money has to circulate in it.

You can’t just say AI will do all the things humans used to do, because if you put enough humans out of work, there are no customers. That is a far worse problem than AI hallucinations. The usual answer is that people will figure it out, and they probably will. But we could be spending a lot more time designing mechanisms and affordances that make the economy more productive. As things stand, I think we’ll find them by accident.

The missing mechanisms

I think a lot about a concept from economics called mechanism design, which has also been described as reverse game theory. Game theory asks how players will act under a given set of rules. Mechanism design starts from the outcome you want and works backward to rules that will tend to produce it as people pursue their own interests. Al Roth, who shared a Nobel Prize for this kind of work, used it to design a kidney exchange that more patients and donors would join. Economists mostly apply the idea to auctions and similar markets, but I think every new technology requires new mechanisms, and the best ones are not invented by economists. We tend to fall into them without really understanding them.

So what are the equivalents for AI? How will agents discover the best expertise and skills, establish their quality, and pay the people who created them? How will they find services without the platform steering them toward whoever pays the most? We don’t know the answers yet. And that’s precisely the point. We’re spending enormous amounts of money making AI more capable, but comparatively little effort figuring out the mechanisms that will turn those capabilities into opportunities for everyone else.

Take open source software. Most people think its mechanism was the free software licenses, but I was around before those licenses, and people were already sharing software. Ken Thompson and Dennis Ritchie created Unix, the predecessor of Linux, at Bell Labs under AT&T’s proprietary license. They designed it to run on many kinds of hardware, and without meaning to, they built what I came to call an architecture of participation.

Unix had a small kernel, the core of the operating system, because people had to reproduce it on dozens of different kinds of computers. Around the kernel sat small utilities that cooperated by following the same protocols, so to speak, such as reading and writing plain text and passing a stream of text from one program to the next. Anyone could write a small piece, and it became a first-class citizen of the system. And because a program compiled for one kind of computer won’t run on another, software had to be shared as source code. Everything else flowed from that.

Linus Torvalds once told me that he could not have built a new kernel for Windows even if he’d had the source code, because the architecture wouldn’t support it. A small kernel meant a small group could get their hands around it. Since then almost every piece of Unix has been replaced. The hardware is completely different, the kernel is now Linux, and the original command shell has given way to generations of new ones. What lasted were the rules. Unix was a protocol-centric operating system, and that is why open source worked.

The same thing has happened with AI. I think MCP brought an architecture of participation to AI, and it was something of an accidental breakthrough in mechanism design, a way to have cooperating AIs. The game we were playing at first looked like Windows on the PC, or iOS and Android on phones. There would be one dominant, centralized player, one giant intelligence in the sky, and everyone else would call its APIs. With MCP, which lets any model connect to anyone’s tools and data, the emerging web of AI agents began to take on the shape of the participatory economy of open source software and the web. People can bring their own models and infrastructure as an alternative to the big centralized players.

This is another side of grappling with the bitter lesson: What are the missing mechanisms of the human-AI economy? Some of them are standards, because standards and protocols are a big part of what makes an architecture of participation. I have a group working on portable memory, so that what your AI knows about you can move with you from one tool to another.

That story is just beginning. We are in an exploratory phase of mechanism design for AI, and it would be a mistake to settle on answers too early. Somebody is going to figure out the next mechanism, and we’ll all say, “Oh, that’s so damn obvious.”

What game are we playing? From architectures to incentives.

Mechanism design starts by asking what outcome you want. So what game ought we to be playing in the AI market?

We’ve been told that the game is the quest for superintelligence: a winner-takes-all contest that will bring untold riches to the winners and maybe impoverish, or even kill, everyone else. We’ve seen mechanisms designed to support that game. Massive investments in data center capacity mean that only the well-capitalized can play. There is colonialist-style appropriation of existing intellectual property. Access to models is sold through expensive APIs, and closed models come with terms of service that forbid others from learning from their outputs. Advertising may be creeping into answers, and memory locks your context inside one provider’s walls. Is that the game we ought to be playing?

I think the game we should be playing is human flourishing, and I think the game of human flourishing is creating new affordances for people to create and share with each other. We’re a social species. History teaches us that we will find our way there, but we can get there more quickly or more slowly depending on the choices we make. Getting there takes mechanisms that create a circulatory economy of value exchange between humans and their agents, with humans in partnership with AI rather than replaced by it.

The big labs, whatever the motives of the people who run them, are all driven by the imperatives of our financial system. They have raised enormous sums that they somehow have to pay back, so building a hugely successful business has to be their overriding goal, and that shapes all their incentives. In some ways AI was enshittified from the beginning, because it was built on the backs of so much capital that has to be repaid. That’s why open source AI matters so much, and sovereign AI too, built and controlled within individual countries. The open models, which anyone can download and run, aren’t as capable as the best closed models yet, but they keep getting closer, and they are already good enough for a lot of productive work.

Open source, open protocols, and open memory give users the freedom to shape and control the technology they use. The game I want us to play is one in which all of us build the future together, so that we control this wonderful new wave of technology for ourselves. I’m starting to see the shape of an economy where AI belongs to people rather than to the big labs, where we have personal AI agents working for us and with us, and we build an economy around that. I think it’s going to happen not because of the big labs, but in spite of them.

Microservices changed how we build software by breaking large applications into smaller, independently deployable services. That made systems easier to develop and scale, but it introduced a different class of problems: The behavior of the whole system got harder to see. AI agents are now following a similar path. A workflow that once lived inside a single application gets split across several agents, each responsible for one task. Individual pieces may work exactly as designed while the overall workflow gradually produces worse results. The difference is that agents make these systems far faster and cheaper to create, which makes it easier to repeat the same architectural mistake at a much larger scale.

I learned this the hard way. My name was on the design doc. Splitting that workflow across three agents was a reasonable call, and I would still defend most of the reasoning. But if you’d asked me that afternoon how we would find out when the thing quietly stopped working, I had no answer.

Months later, in a quality review, a reviewer noticed something felt off before any of our instrumentation caught it. Latency looked fine, error rates were flat, and every service in the chain returned what it was supposed to return. What we had was a slow decline in output quality, and it took about three weeks to trace.

The first agent handed a set of documents to the next one for summarization. The summarizer dropped a field it had judged to be a low signal. A downstream agent then made a confident recommendation without that field. Every agent did what it had been designed to do, and the result was still wrong. I should have anticipated it, because I’d been on the product side through the microservices years and had seen this pattern before.

The hangover we already paid for

Breaking down monoliths was a good decision for most teams, but one thing was forgotten. Decomposition did not make the difficult parts of the system disappear. It moved them into the gaps between services, where the people accountable for the overall product could no longer see them. The bill landed around 2019, and it landed on the delivery schedule rather than the incident channel. Teams were running hundreds of services with limited tracing and no contract versioning, and the real symptom was slower delivery. Shipping took longer than before, because engineers spent their weeks picking through those services.

Agents are running the same arc faster, because standing one up takes an afternoon and a prompt instead of a repo and a deploy pipeline. When creating an agent is that cheap, product teams make agents without asking anyone, and nobody keeps track of what the agents are and what they’re doing.

What breaks, and what it costs you

This isn’t only my experience. Cemri and coauthors annotated more than 1,600 execution traces across seven popular multi-agent frameworks and built a taxonomy of 14 recurring failure modes (MAST, NeurIPS 2025). Most of what goes wrong traces back to how the system was specified and how its agents coordinated, not to the models inside it.

Inventory sounds trivial and isn’t. Service sprawl at least showed up in your Kubernetes manifests. Agents get created inside product surfaces and workflow tools, and inside somebody’s weekend automation of their own job. You don’t find out what you have until a compliance reviewer asks which automated system touched a given decision, and then it’s too late.

The visibility gap cost us three weeks. When an agent calls an agent that calls a retrieval layer that calls another agent, and every hop has a model deciding what to pass forward, you need trace context that survives the whole path. Most stacks log prompts and completions per call and stop there. You get each hop and never the chain, and it was the chain that failed.

Compounding confidence is the failure that worries me most. Distributed systems fail loudly, which turned out to be a blessing nobody appreciated. A service throws a 500 and the caller knows to stop. Agents fail quietly and fluently, so when an upstream agent produces a summary that’s plausible and wrong, the downstream agent gets no signal and reasons confidently on top of it. By the time the result reaches the user, it conveys a level of confidence that the initial evidence does not support.

API contracts drifted when someone changed the response format without versioning it, and we built machinery to catch that at build time. Agent contracts drift when someone edits a system prompt. In that case there’s no schema diff and no type check. Nothing breaks until a downstream agent starts getting different output and adapts to it without telling anybody.

What carries over from microservices

A registry is the cheapest thing to build and the first thing everyone skips. When building microservices, we kept a service catalog with an owner and a scope for each service. For agents, we need that same record plus an explicit list of what each agent is allowed to call.

Tracing carries over nearly unchanged, since correlation IDs were invented for this exact problem. OpenTelemetry’s GenAI semantic conventions give you the vocabulary, and an agent run comes out as a span tree instead of a pile of unrelated log lines:

Had we been emitting those spans, the summarizer span and the recommender span would have sat side by side in one trace, and the missing field would have been a diff instead of a three-week hunt. These conventions are still moving, so keep the attribute names behind a thin mapping layer of your own.

Finding the missing field in a trace would have shortened the investigation. I’d also want a test that caught its omission before a prompt change reached production. That’s where I’d apply the contract-testing discipline from microservices: define what the downstream agent depends on, then check that an upstream change preserves it. For the summarizer in this workflow, the evaluation would need to check whether its output retained the information the recommender needed. I’d version the prompts and require changes to pass those evaluations before release. There’s early evidence it pays: SEMAP, a protocol-layer method applying behavioral contract modeling and lifecycle verification on Google’s A2A infrastructure, reported up to roughly 70 percent fewer failures on function-level coding tasks when scored against MAST.

The trust boundary is the thing I’ll argue about in a design review. In 2019 we learned that a service doesn’t get trusted just because it lives inside the perimeter, which is most of what service meshes were for. The agent version is that an agent doesn’t get broad tool access just because we built it and we’re nice people. What an agent is allowed to do belongs in a policy layer the model can’t talk its way past, not in a system prompt asking politely.

Cost is where the analogy between microservices and agents fails. A microservice stuck in a retry loop costs you CPU and an alert at 3 am. An agent stuck in a loop costs real money on every turn and doesn’t saturate in any way that makes itself obvious. This one lands on the product owner instead of infrastructure, because it surfaces as unit economics rather than an outage, which means the PM ends up justifying the number at the end of a billing cycle.

What I ask before approving a split

None of this argues against multi-agent systems, which earn their keep the same way services did. My argument is that operational practices should evolve alongside architecture, without repeating the five-year lag of the previous transition. The person accountable for delivering the product is usually best placed to ensure that happens.

Hence, I push on four things. Can we follow one request all the way through and reconstruct what each agent decided and why? If not, we’re building something we won’t be able to debug. What happens when an upstream agent is wrong, and not whether it can be? If the honest answer is that the next agent carries on as normal, we haven’t designed a failure mode. We’ve designed a failure and left it running.

Who owns this agent, and where is its tool access enforced? A system prompt isn’t enforcement. What does this cost at ten times today’s volume, and what shuts it off when the number goes wrong? I ask that one early, because nobody volunteers it.

The teams that managed service decomposition most effectively incorporated tracing and contract testing into the build from the outset. These practices helped them understand how services interacted and verify that changes preserved expected behavior. That work is a sprint or two now, and a great deal more than that once you’re running 40 agents and someone asks which of them touched a customer’s data.