The following article originally appeared on Elevate and is being reposted here with the author’s permission.

A software factory is a repeatable loop around software work. If you’re building a software factory, code good enough to ship still needs human taste and ownership. We’ll discuss this including whether you need a factory just yet.

If so:

• You’ll likely need humans in the loop upfront for deciding on product intent, system design (if you care) and your quality bar.
• Do review code (lights-on factory) but be intentional with where it’s needed the most. I’ve found you want to watch out for where automated back-pressure breaks. Or where maintainability trade-offs need to be made.
• Aim for quality checks to happen as early and continuously as possible. Not all of them have to, but this includes type systems, automated tests, mutation testing, security scanners and linting for architecture rules.
• Number of checks ≠ quality. You’ll likely need to experiment with what checks give you the best signal to noise ratio. Be ready to tighten or relax your constraints deliberately.

You want to build your factory so some aspects of human taste get encoded in the environment, the agent gives you evidence of its work being right, and where a human still “owns” what ships to production.

Sponsored by Sonar: Your agents write the code. Your gate decides if it ships. AI agents are writing more of my code, faster than ever—but fast isn’t the same as shippable. So I let a coding agent build an app, then put it through SonarQube. Every commit gets the same deterministic check: deep cross-file analysis, a clear map of where the risk actually lives, and a quality gate that holds every human and every agent to one bar. The screenshot? A PR that didn’t pass. That’s the gate doing its job.

Do you really need a software factory?

In my experience, you can get surprisingly far with your stock coding harness! i.e., Claude Code or Codex, multiple sessions, good SPECs with verification baked in and constraints. You can even throw a batch of GitHub issues at them with implementation and human-involvement criteria, but it’s when this system needs to be repeatable and event-driven that a factory is helpful.

So I started off by saying a software factory is a repeatable loop around software work. We can actually look at a prompt that demonstrates a very small factory loop here:

Read GitHub issue #123 and the repository instructions before changing code.

Implement only the stated acceptance criteria. Do not modify authentication, billing, migrations, or existing test assertions. Work in a branch and keep the diff reviewable.

Run npm run lint, npm test, and npm run build. If a required check cannot run, stop and explain why. Open a draft pull request with the checks you ran, the remaining risks, and any decision a human still needs to make. Do not merge.

A goal can keep this moving until the checks pass and we can poll GitHub issues for any specific labels or review open pull requests each morning. Branch protection could enforce a merge boundary and the human can stay in the loop by choosing what becomes ready, reviewing and making the final merge calls etc.

Add a software factory when you need an event-driven queue of work (e.g. Slack triggers, GitHub issues, Linear, a backlog) to run in an isolated cloud environment to handle triage, implementation and testing with some explicit human babysitting. Some end their loop with a monitor agent watching production and filing issues which triage again.

In my experience, the factory becomes useful when the hard part is making your different runs behave consistently, handing work off between agents and avoiding different sessions from claiming the same issue, preserving evidence and stopping production when human review is falling behind.

What solves this might sound a little boring. For example, Warp mentions triaging every incoming issue into one of four states—ready-to-implement, ready-to-spec, needs-info, wait-to-implement—and the label is what fires the next agent.

This label does a few jobs in one go: it’s the queue, the lock, and since a session only picks up what’s marked ready, it’s where a human can park stuff without saying no permanently.

Workflow wise, there are a few similarities and differences to just using Claude/Codex:

• Steering: Agent needs course-correction, you can give input and redirect it.
• Notifications: How the factory says it’s blocked. This can be because a requirement was ambiguous, it started something risky or it needs human input (steering).
• Handoff: Move the task, its state, and context between the cloud factory/another agent/human reviewer. Good handoffs will keep track of what happened, what’s left to be done and why the handoff is needed.

In a good factory, the human isn’t limited to just reviewing and approving the final diff at the very end. They can shape the work early on, steer it during implementation, get it through a handoff, or stop it shipping to production.

Verification is where a responsible factory spends a lot of its time. We’ll cover this more later.

If you decide you do need a software factory, building it isn’t the only option. Standing up the infra to scale a factory can be a lot of work and you may want to consider buying verus building. Factory, Warp and HumanLayer are all working on this.

What my day looks like now

My day-to-day experience of software development has changed a lot over the past year. I’ve been talking about increasingly doing a lot of parallel work with agents, moving towards having a lights-on software factory. And a lot of people have been asking me, like, what do these things actually mean? What are you building? What are the kinds of projects that you’re using these things on? It’s a lot of this:

So on a very average day, I have a simpler lights-on software factory. I can have tasks that are running in the cloud. Half of these tasks might be working on production client applications with smaller companies that I’m working with. They’re going to have real users. They’re going to have real authentication, payments, subscriptions, real beefy risks that you need to be careful with. You can’t just say, “Oh, agent, just go and do this stuff” without having tests and constraints and quality checks in place.

I could work on my open source projects. I could be building out companion sites for my books. I could be working on tools. I could be building apps of my own. And these are all very, very different kinds of applications that I’m working on. And sometimes all they have in common is the tool that I’m using to work on them, right? Maybe I’m working on a migration. Others, I might be doing actual beefy feature work. And the blast radius of the work might also be very, very different.

So as you begin to think about getting to a place where we’re increasingly doing a lot of parallel work, we’re trying to improve velocity, we’re trying to improve productivity, and we’re trying to improve autonomy, which means getting the system to a place where we trust it more, you do have to think about what are the places that absolutely require human code review, human input.

And a lot of that’s going to be required up front, right? When you’re defining your specification, your requirements, what’s the design of the product going to look like? What’s the intent of the product going to look like? And then, how are you verifying that the agents have actually gotten the work done right? How are you verifying that they haven’t broken the existing system that’s been in place? How are you making sure that it’s meeting your quality bar?

So generating code is not necessarily the part that you need to worry about the most. Given enough context, agents can write the implementation, run the tests, inspect failure, and revise code for us. We need to get to a place where we feel like there’s enough of human taste encoded in the environment that we can trust what’s being built, so that our human attention can be focused on the places where it’s needed most.

Now, there’s pushback from folks saying, “Hey, well, I don’t buy that you can just automate away a lot of this stuff.” It’s not to say that we’re automating away all of it, right? But given the volume of code that’s being generated, I don’t think that it’s realistic for humans to be reading all of it, especially when we’re not building rockets a lot of the time, right? We’re building UI, we’re building full stack applications.

Our judgment, our taste is best focused on the places where it’s needed the most. Like, what are the riskiest parts of the systems? Where do we need to apply human taste? And that can be in the frontend. That can be in how the system works. It doesn’t have to be 100% of it.

My cognitive bandwidth does not scale with the agents

The reality is, yes, we can now fire up dozens, hundreds, thousands of agents in parallel, but your own cognitive bandwidth does not scale in the same way. This can feed into cognitive or comprehension debt which I’ve talked about before.

If you remember back to just five, ten years ago, there was a lot of discussion in the engineering community about context switching and the cost of it. We would talk about how people hated when a colleague or someone would walk up to your desk when you were in the middle of a task. It would then take you so long to get back into your flow state because you had to catch back up in terms of like, where was I? What was I doing? Even if you had a little bit of residue there, it still took you time.

We’re now context switching even more than we did before. On any given day, if I’m working outside of a software factory, I can be working on five or ten different projects with agents at a single time, or five or ten different features on a single project at a time. I can have five or ten different sessions, you can effectively say.

That means that I have to be able to stay on top of at least a few of those. It is possible that I’m going to be able to increase how much autonomy I give some tasks if I have trust that I’ve defined the task well enough, I’ve defined the outcome, how it’s going to verify that it’s done well enough. But then there are going to be tasks where maybe I don’t necessarily feel that way and there’s more risk involved or more nuance. I’m going to have to pay attention.

Consider optimizing the software factory for your reviewer. Given every one of those approaches still routes its output to one person’s attention, you should ask how much cheaper the factory is making the decisions you still have to make.

A wrong-project mistake

I remember when I’ve been working on multiple parallel projects with my agents, and there have been times when I’ve accidentally done things like, maybe I was working on a web app where I wanted to add in a dark mode, and so I had in my head, okay, well, this is what the shape of this needs to look like. But I accidentally went to the session for a different project, and I started putting in that same prompt.

So I began implementing dark mode for something that absolutely didn’t need it. And so I can make that mistake. I don’t want my software factory making that kind of mistake.

You need to think about this really in terms of a system. You are effectively trying to encode a software engineering culture, a team culture, into a system so that it has those same kinds of behaviors, so that it has ownership that belongs somewhere, so that someone is still on the hook for what happens, and you’re being very explicit about how you think about those things.

When green is misleading

Even in these systems, you want to be very careful, right? Many of us have seen that when you have asked AI to help you pass a test, like we’re talking about a programming test, a unit test, it can change the unit test to satisfy that condition, or it can change the logic of the code to pass that condition. That doesn’t mean that it’s actually followed your intent in order to align both the functional behavior and what the test was supposed to be testing, right?

Just because a software factory is showing that everything is green doesn’t mean that it’s actually green, especially at the start when you’re setting these things up. You need to pay a lot of attention to make sure that your checks, your verifications, all of those are shaped the right way. They’re doing what you expect them to be doing. You don’t want them to be misleading.

You don’t want a situation where you had tests that said, hey, actually, I have gone and changed what authentication providers are supported. You asked me to add GitHub for example, as an authentication provider, but hey, my UI only had space for three, so I’ve gone and I’ve dropped one of the other ones. And hey, by the way, that happened to be one that your customers actually wanted. So you just need to be very explicit about how you want these systems to work.

Btw, security is super important too, and if your factory reads untrusted input like a GitHub issue/Slack message it might be adversarial and include problems like supply chain attacks. So some explorations into software factories, like Vercel, run their agents in isolated sandboxes holding just the secrets a task needs. That way a compromised run can’t reach what the job doesn’t need. Your defense ends up being layered.

Which old projects deserve another life?

I also think that a big part of how we work these days is deciding what should exist. If you remember back to many years ago before AI, there were so many abandoned software engineering projects, so many abandoned weekend projects, personal projects where they just wouldn’t launch because we didn’t have the time to finish them. We didn’t have the bandwidth to prioritize getting them out the door because they just weren’t that important to us or we couldn’t find the time.

Now it’s fairly trivial for us to complete those projects, but the same human judgment question comes in. Do those projects deserve to exist? Should they be launched? Because you put them out into the world and even if it has just five users, maybe you have to maintain it. Maybe you have a quality bar now that you want to maintain.

I know that I’ve had so many GitHub projects from over the years where now that I have an agent, the first thing I do is get the thing building. Because, of course, you clone it and now it doesn’t build because all the dependencies have changed. Half the things are out of date or have security vulnerabilities all over them, so you have to update that.

Then you have to add tests if you didn’t have tests so that you know that behavior is at least going to be there if you’re upgrading the project in some way, or if you’re migrating it to a more modern language or framework or thing like that.

Then you start to ask yourself, well, maybe, a silly example, but maybe I used Twitter Bootstrap back in the day for this, but now everybody is using Tailwind and shadcn, so I have to re-implement the UI. And what you’ll notice is that suddenly this is taking you more time, right? Yes, the agent can get a lot of this done quicker, but you’re now having to factor in product sense and taste and all of these things.

You still question, well, who is this for? Does it have a market? Is it for myself? Is it for other people? If I’m putting it out into the world, is it still going to be as interesting given that now anybody can spin these things up as quickly?

So I think that human question of do these things deserve to exist? How do we factor in our taste and judgment? I feel like those things continue to be extremely important. That’s where that scarce resource of human attention still really comes in. Back in the day, we only had a finite number of hours in the day. We had meetings. We had to budget in time for design and coding and so on.

Now that we have agents to help us, I think that you have to really just be very explicit about where you’re spending your time and why.

What happened when I built a sample one

So I’m going to talk about the 82-minute factory run. People have been asking me for quite some time, you know, “How do I build a software factory?” Or, “I’m used to using Claude Code or Codex. How do I evolve my setup to using a software factory?”

So the first thing that I’ve been saying is, “You may be fine. Your work may actually be totally fine without needing a factory.” But I did want to give people a reference setup that they can check out. So what I put together is a repository called Factory that you can go and check out. I also put together a demo application and workshop.

Now, for the last couple of years, my go-to demo application for a lot of things has been a movies app. I’m a big movies fan. I love watching movies. I watch movies all the time, and so I have a demo application, which really starts off as a very simple movies app. And what I want the factory to be able to do is go ahead and implement a number of features. There’s a few different features. I want a favorites feature. I want it to be able to maybe do search, and maybe also want a dark theme in there as well, those types of things.

So I have my factory go and begin working with these things. You can check out the implementation. One of the benefits of it was actually catching real problems. These problems may not have been things that I would have caught if I had just asked it to do a one-shot implementation.

Maybe around the 60-minute point, I was feeling like, “Wow, this is going unusually slow.” I asked my harness using the factory, “Why are things going slow?” It said, “This is actually totally fine. All the verifiers are still running.”

You might have expected individual tasks to take 10 minutes, 15 minutes, 20 minutes, but they can take two to four times as long once you begin to include verification, retries, browser checks, human review, any of those extra delays.

I do think that these can add up to better quality and better trust in the system. From a measurement perspective, you might look at metrics like cost per merged PR and code shelf life as comprehension debt metrics.

You also need to think about what is useful delay versus factory overhead. The verifiers, in my case, caught some real problems. A little bit of the time was maybe sunk into producing evidence that I wanted. Some of it was overhead in the factory running. I didn’t really spend any time optimizing it, but a factory that just runs a lot of checks that you’re not finding valuable does not mean it’s a high quality one.

You want to study how, for any repeated checks, are they irrelevant? Are they noisy? Are they actually making the system safer?

A verification budget

The way that I think about the budget for verification, this is basically what we’re talking about. We’re talking about a verification budget. I think about it in the same way as I’ve historically thought about performance budgets.

There are going to be certain kinds of checks that you can run early on in your software development lifecycle, and there are going to be some things that are so heavy, but they offer so much value that you will want to run them later on. There are some kinds of fast checks, linting, for example, type checking. These are relatively fast checks that you can run early on.

Our full suite of tests can be run closer to right before a draft PR is being put together or after that. That can include mutation testing, browser testing, security checks, anything like that.

I think that you don’t necessarily want to replace these with just summaries. You want real tests, but you just need to make sure that you’re budgeting for them in the right places because you don’t want to slow down your development loop. I certainly never want to slow down my development loop. Having a fast iteration loop is important to me, but I also want to still have those checks and balances.

When a run doesn’t ship

A lot of what I’ve written above concerns the checks.

In their software factory, Vercel marks every agent run as “success”, “flawed”, “blocked” or “manual” and only “success” ships to production. The rest re-enter the system. I’ve been thinking about runs in similar terms.

“Flawed” here means the wrong thing was implemented or maybe it didn’t have full context, so that has to be fixed. Blocked means the environment may have been missing a credential so you have to provide it. Manual is a boundary the factory may not be allowed to cross it yet.

Two of the three things here may have mechanical fixes and the last one is about trust.

While this is great, what sorting doesn’t show you is cost. Back to my factory implementation with the TMDB app, the quick finder with no rejections took 7 minutes. Favorites, with two rejections and a human decision in the middle, took 56. Same factory. So I’d pair the taxonomy with per-stage timing, otherwise you know a run came back flawed without knowing what finding out cost you. The other thing I’d fix is the handoff at the boundary: my sample factory stopped issue the first issue and moved it to factory:needs-info, which was right, but I didn’t know where to put my answer. A manual run isn’t finished when the factory stops but when the human knows what to do next.

Autonomy is not a single setting

I wrote a couple of weeks ago an article about agentic autonomy and how to think about autonomy because autonomy is not going to be a single setting for every single project.

Verification buys you trust, and it buys the ability to grant more autonomy to your agents. So if, for example, I am working on a non-trivial change, but I have a number of checks in place, everything gets verified correctly, and maybe I’ve hand checked it myself. The next time I’m going to do a task like that in the same project, maybe I’ll feel comfortable giving the agent a little bit more autonomy.

That’s the thing that you think about when you’re building these software factories. Your verification is going to change with risk. Your goal is the best signal to noise ratio. You don’t just want to have some large checklist that you’re running.

The feature I had to relearn

There was a feature that I’ve been putting off on a day when I’ve been using multiple sessions with Claude, and I was working on a few different projects at a time, a few different features at a time per project. And so Claude had implemented the feature that I was working on. It looked like the tests were passing. I hadn’t put a lot of thought into verification, but the tests passed, and so I thought it worked. I merged it.

And so this was a favoriting feature. I thought that this was actually pretty good. I tried to check it out in the browser. It seemed like it was okay, but a couple of days later, I actually returned to the code because there were some tweaks that I thought I might make to this.

I didn’t want to just ask my agent to make the changes, because it was just a subtle way that it worked. You tap on the icon, and it would not show the right effect on tap, and so I wanted to just tweak it. I wanted to understand how it worked so I could guide my agent correctly.

I returned to the code, and I couldn’t explain to you how the feature worked. This repository was mine, right? I’d approved the change. I understood how a lot of it worked, a lot of the repo worked, but my understanding hadn’t kept up pace with all of the code that had been building up.

What I failed to absorb was how this feature that had been added actually worked, how the UI worked, how the effect on it worked. I had to redo this feature and actually go step-by-step, “How does this work? How can I understand it?”

What parallel work does to understanding

When you’re doing parallel work, it amplifies this overall problem, and it gets even more amplified when you’re doing it in a software factory. When you’re doing five or 10 sessions, they create much more than just a review volume problem. They create several mental models that can end up going pretty cold while you’re working elsewhere.

We’ve historically talked about the challenges with context switching, and as soon as chat compacts, you reject some approaches, you try out different things, you’re pairing with the agent, you’re going to have a difficult time remembering everything that happened in your session.

You can scroll up, and as compaction has been happening, you’re not going to have everything there, and you’re not going to be able to store it all in your head. Code often preserves a decision that was made, but not why the decision was made.

This is something that I think can be a useful learning for you, where it’s important, consider asking your agent to actually store information about its trajectory, or interesting lessons about how it approached a problem so that you can go back to it later.

This can or can’t be something that you decide to commit to a repo. You can keep it local if you want, you can share it with a team if you want, but that can be something that can then be consulted later on. Rather than you relying on it maybe being in a session, or you maybe remembering about it later.

P.S. If agents are pushing to your main branch, they need the same quality bar you do. SonarQube gates every commit deterministically: one standard for humans and agents alike, on every PR. Sponsored by Sonar.

Ownership doesn’t disappear

There is a broader principle underneath all of this.

The percentage of code physically typed by humans may fall dramatically. I don’t think human ownership needs to fall with it.

• Someone still chooses the problem.
• Someone still chooses the architecture.
• Someone still sets the quality bar.
• Someone decides which verification signals deserve trust.
• Someone decides when the evidence is sufficient to ship.

And when the resulting system fails, “the agent wrote it” doesn’t cut it. This is why I don’t think the future of software engineering is best described as humans leaving the loop. Instead, human judgment is being relocated.

We should remove people from the parts of the loop where machines can produce stronger, faster, more deterministic signals. At the same time, we should concentrate people around the places where context, taste, risk, and long-term ownership matter most.

The best software factories will not be defined by how completely they eliminate human involvement.

They will be defined by how intelligently they place it.

Keep human judgment upstream on intent, system shape, and the quality bar. Review code where automated back-pressure becomes weak or the consequences become subjective. Push every deterministic signal as early and continuously into the loop as possible. Tighten and relax constraints deliberately as the system earns or loses trust.

A human still has to own what code ultimately ships. Code good enough to ship still starts there.

AI systems are gaining access to more tools and data, raising new questions about oversight and accountability. This Week in AI host Christina Stathopoulos spent this episode examining how those questions are playing out in model safety, government oversight, and even digital marketing.

Safety needs more than safer models

Anthropic’s latest threat report documented misuse of Claude across seven categories, including cyber operations, surveillance, influence campaigns, fraud, biological misuse, weapons development, and illicit model distillation. OpenAI reported on a different kind of AI risk, one that’s less about people weaponizing it and more about AI going off-script. They examined model misalignment, documenting several cases where models took actions outside the boundaries developers intended, including deception, unauthorized actions, and attempts to circumvent controls. After the OpenAI and Hugging Face controversy dominated headlines, other models have also reportedly reached systems outside their test environments, including Gemini, according to a recent cybersecurity disclosure.

Christina cautioned against describing such incidents as models “escaping,” since that language can assign agency to the model while drawing attention away from how companies designed and secured the surrounding environment to begin with. As agents gain access to browsers, files, code, and external systems, the teams building and deploying them must rigorously test and secure those environments, with clear accountability when things go wrong.

AI labs and governments are starting to wrestle with those requirements. To track the pace of AI development and maintain greater oversight, Anthropic has proposed tracking how much AI contributes to AI R&D, how closely organizations monitor agent actions, and how they allocate computing resources between capability and safety research. Anthropic and OpenAI have also proposed giving outside safety organizations greater access to their labs, although Christina questioned their independence when frontier labs fund the work. Meanwhile, a US Senate proposal for an emergency AI kill switch failed to advance, while California ordered officials to develop proposals covering shutdown mechanisms and independent evaluation.

AI is moving closer to the customer

OpenAI is now testing Sponsored Agents, showing how conversational AI could change digital advertising. After clicking an ad, users can start a separate conversation with an AI agent representing the advertiser, ask questions, explore recommendations and then visit the company’s website when they are ready to take the next step.

That approach could lead to more interactive advertising, but clear labeling will be essential so users always know when content is sponsored. Christina also raised the broader ethical concern of whether paid placements could influence the answers AI chatbots provide, blurring the line between independent guidance and commercial promotion.

Access to AI also means access to expertise

The Gates Foundation announced a $1 billion commitment over two years to expand access to AI in healthcare, education, agriculture, and other areas. Its 2026 Goalkeepers Report argued that AI could help narrow existing gaps, but only if organizations intentionally make the technology and its benefits widely available.

Christina highlighted examples from Kenya, Sierra Leone, India, and Rwanda. Health workers are using AI to improve diagnosis and treatment planning. Students are getting additional support from AI tutors, while small farmers can use personalized advice to improve harvests and make better decisions about market prices. These applications show practical roles for AI in places where demand for expertise exceeds the supply of teachers, clinicians, and other specialists.

What’s next

AI governance can’t stop at model evaluations. Organizations must also decide what AI systems can access, who reviews their actions, how commercial incentives affect their behavior, and how people continue developing the expertise needed to supervise them. The choices companies and governments make today will shape how useful AI becomes and how widely its benefits are shared.

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

My conversation with Dan Gookin, the original For Dummies author and now mayor of Coeur d’Alene, Idaho, started as a publishing reunion. It ended with a much larger question: What happens when the software you rent becomes infrastructure you can’t leave?

There was something wonderfully circular about hearing Dan Gookin tell me that, after he was first elected to public office, he bought a copy of Robert’s Rules For Dummies. Dan wrote DOS For Dummies, the 1991 book that launched the For Dummies series, and went on to write more than 180 technology books with over 12 million copies in print. I spent nearly three decades acquiring books in that program before joining O’Reilly this summer. Dan saw the news of my new role and reached out. We caught up on publishing, books, and AI. The part of the conversation that stuck with me had nothing to do with any of those topics. Dan is now the mayor of Coeur d’Alene, Idaho, and he’s begun thinking about the city’s software the same way he’s been thinking about his own.

Dan was elected to the Coeur d’Alene City Council in 2011 and became mayor in 2025. He was quick to draw a distinction between the two jobs. “Council, you can be a little bit more extreme,” he told me. “It’s more rhetoric based. This is administrative. It’s management.” A council member can object to a budget line. A mayor has to make sure the systems that line funds keep working. For a city of nearly 58,000 people, that covers everything from the police network to the water bill.

Cutting the leash

Near the end of our conversation, Dan mentioned a project he’s been working on personally. He’s begun weaning himself off Microsoft and what he calls an increasingly subscription-based, cloud-based, AI-heavy environment, moving his computing to Linux and running his own servers. This move isn’t entirely ideological. There is money attached. Dan told me his Adobe subscription for software he uses to produce content costs about $800 a year. He believes he can replace most of it on Linux. What’s surprised him is how much he can replace. “It’s interesting to see how quickly it can be substituted,” he said, and how quickly he can “cut that chain. . .or cut that leash.”

Dan sees a similar economic model when he gets to work. “Our software subscription just for our city, our size, is $700,000 a year and growing,” he told me. The example he kept returning to was the city’s financial software. “We used to own our finance software,” he explained. “Now the finance software is leased and it’s cloud-based.” Then he asked the question that ought to appear in a lot more software procurement meetings: “So we don’t even own our own data?”

His concern isn’t literally that the city has forfeited legal ownership of its records. It’s about practical control. What happens if the vendor is hacked, or the city decides to switch? Can it get all its data back? “Do they just give you a binary dump or do they actually give you the data,” he asked. That question has stuck with me since we chatted.

Reversibility is an architectural property

Cloud contracts have a word for part of this issue. In cloud services agreements, reversibility refers to a customer’s ability to retrieve its data and unwind a vendor relationship. The United Nations Commission on International Trade Law (UNCITRAL) even includes it in its glossary of cloud contract terms. That idea belongs in municipal software evaluation too, not as an exit clause buried in a contract but as a standing column in the evaluation spreadsheet, next to features, price, and security.

When you evaluate software, you compare features, price, implementation time, security, and, increasingly, AI capabilities. But what’s the exit cost? Can an organization export its data in a documented, useful format that another application can consume? How much institutional knowledge has quietly migrated from the organization to the vendor? And if the vendor raises prices sharply at renewal, gets acquired, or simply stops serving your needs, how long would it take to leave?

These questions are properties of the system and subscription software has raised their stakes. When software came in a box, skipping an upgrade didn’t make the version you owned disappear. That world of proprietary formats and dominant platforms had plenty of lock-in. But possession still meant something.

Dan and I talked about how alien that world now seems. Software once arrived with manuals. Today it may not even arrive. You authenticate to it. If you don’t know how to do something, you ask an AI instead of consulting a manual. That change highlights a step from possessing tools to maintaining permission to use them. For an individual, that might mean Photoshop is a monthly subscription now. For an organization, recurring access can become an architectural dependency. For a government, that dependency is borne by taxpayers.

Should cities write their own software?

Dan takes the argument one step further. “For $700,000 a year,” he said, “we could hire a couple of programmers just on contract and have them code our own stuff and then we own it again.” I’m not convinced that math works out. Two programmers likely can’t effectively reproduce a mature municipal financial system, endpoint protection, records management, and specialized public-safety applications, let alone the compliance work and vendor support that come with a modern city’s technology stack. AI could help accelerate the process, but liability and security issues surrounding AI-enabled development likely add more risk than a government is willing to accept in the name of software ownership.

Building software also creates its own long-term bills, ones that don’t fit easily within most city budgets. Code must be maintained, security vulnerabilities patched, and staff and frameworks eventually replaced. An application written in-house can become every bit as difficult to escape as one bought from a vendor. On the flip side, the headaches of replacing a “good enough” proprietary system with a packaged one that fits most, but not all, of an organization’s needs can outweigh the cost of keeping the old system running. The familiar build-versus-buy analysis exists for good reasons.

But Dan’s question still matters, even if his proposed fix isn’t right for every case. At what point does the cost of renting capability justify rebuilding some capability of your own? Perhaps more importantly, which capabilities should an organization insist on controlling, even if renting them is cheaper? There is no universal answer, including “the vendor handles it.”

This isn’t an argument against the cloud

It would be easy to turn Dan’s experiment into a familiar prescription. Move to Linux. Embrace open source. Bring everything back on premises. Escape the cloud. That’s too simple. Cloud and SaaS products solve real problems, shifting maintenance to specialists and giving a city of 58,000 residents access to capabilities it could never economically build or maintain for itself.

The key question doesn’t boil down to cloud versus on premises, or proprietary versus open source. It becomes a decision about whether you accept dependency you’ve consciously chosen or dependency you’ve acquired by default. An organization may rationally decide to rent a critical service indefinitely. But it should know where the data lives, how it comes back, what replacing the service would require, and which internal skills have atrophied because the vendor now supplies them. Revisiting those answers periodically, rather than treating last year’s renewal as the rationale for next year’s, is a step that’s easy to skip when nobody’s asking the questions in the first place.

From hobbyists to city hall

Dan suspects the search for alternatives will happen outside big organizations first. He compared it to the early personal computer movement. Hobbyists and enthusiasts experiment first, long before organizations decide the ideas are practical. His hunch is executives will eventually look at how much of their budgets go to recurring subscriptions and ask a simpler question: How much are programmers? Again, I don’t think the answer will be “hire programmers and cancel SaaS.” But more organizations will ask the question behind the question. What are they paying for convenience? What are they paying for capability? And what are they paying because leaving has become too difficult?

Software can grow to define an organization rather than serve it, especially when the cost of paying for or maintaining a tool outgrows the tool’s value. That shift becomes a problem when systems are so deeply embedded that replacing them feels impossible or when years of subscriptions leave an organization unable to perform a basic function on its own.

Dan’s new job has given that shift a different scale. He told me the biggest adjustment from council member to mayor was realizing that the job is administration and management. He’s less interested in ceremonial appearances than in answering email, returning calls, setting meetings, and, in his words, getting stuff done. Software is part of getting stuff done. So is knowing when to buy it and when to build it. The latest addition to that list may be knowing that the tool with the most impressive new capability is not as valuable as the one you can still leave.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The following article originally appeared on Arize’s blog and is being reposted here with the author’s permission.

This week the AI community was in uproar about Jev from TypeSafe, not just a new model but a new kind of model: one that classifies, scores, and routes but can’t write a sentence. The reason for the fuss is simple. It’s radically faster and cheaper than using an LLM to perform the same task (up to 200x faster and 400x cheaper if TypeSafe’s numbers are to be trusted). In one small independent test, a general-purpose model spent about 910 output tokens reasoning its way to each yes-or-no answer while Jev spent 85, and it doesn’t even bill for them.

That’s potentially a really big deal. An enormous share of LLM-powered components in AI applications today are being asked to make decisions: pass or fail, route A or route B, which of five labels to pick. In particular, that’s something that LLM-as-a-judge evaluations are doing all the time, so it really made our ears perk up at Arize AI. This post is about how we got here, what this new kind of model buys you, what you lose, and what choices you should be making about your application’s architecture as a result.

TypeSafe shipped a model that can’t write, only decide

Here’s how Jev works. You send it some data that represents a state (a support ticket or an agent trace or a JSON blob) plus a list of typed questions (Choose one of these options; Score this on a scale; Is this statement true?). It doesn’t generate a token stream. It returns typed answers with probability distributions in a single parallel pass, in 70 ms to 500 ms, at $0.042 per million input tokens. No free-form text comes back.

The training method used to create Jev is what TypeSafe calls Reinforcement Learning for Calibrated Decisions or RLCD, described in their primer as training the model so that a higher stated probability means a higher chance the answer is right. (You’d think that’s always what a higher stated probability should mean, but read on for surprising facts about how LLMs work.)

TypeSafe also claims Jev “can’t hallucinate,” but that really feels like an overreach. Jev can’t return an answer outside the schema you gave it. Within that schema, it could still be giving the wrong answer, although its probability score should give you a clue if it’s not confident.

And the whole thing is incredibly fast and incredibly cheap: 40x to 200x faster and 40x to 400x cheaper, depending on the task, are TypeSafe’s numbers from TypeSafe’s evals. Of course, we know better than to take a vendor’s word for these things, so Arize will be running our own benchmarks just as soon as we can. But other people have already started doing that.

On the early data, Jev is mid-tier intelligence at a two-orders-of-magnitude discount

TypeSafe’s published evals run four decision workflows, one of which is reviewing a finished agent trace to decide whether a human needs to look at it. Averaged across the four workflows, Jev lands at 68% accuracy at $0.0004 and 0.4 seconds per case. GPT-5.6 Terra is at 68% for $0.03 and 10 seconds. Opus 5 is at 73% for $0.18 and 38 seconds. That’s five points behind Opus 5, but on the other hand it’s 440x cheaper. That’s a very interesting cost-benefit trade-off, and such a radical one that it may change how we architect our applications.

The independent data so far is small but it points the same way. Every’s head of evals ran 777 judgments in under 0.7 seconds for about a quarter of a cent. A UK events site, NearHere, tested listing moderation and got 96% from Jev against 86% from Gemini Flash-Lite, 58x cheaper per decision. That’s where the 910-versus-85 token count I mentioned earlier came from. And a developer ran Jev zero-shot over 18,514 spam emails, getting a result that was a statistical tie versus a classifier trained on the labels.

These are small samples and early data but hey, the thing was released a few days ago.

We used LLM judges because nothing else worked without labels

Why are we using LLMs to make decisions in the first place? The reason is simple: They are able to do it without huge, expensive training sets, which is what most ML solutions prior to LLMs required. Here’s the options on the table now:

The first two rows are the old-school options, which need a training dataset: hundreds to thousands of labeled examples before you get a single prediction, and then a training run, and then someone to maintain it. Nobody building a first version of an AI product has that data or that kind of time. The LLM as a judge, on the other hand, just asks for a paragraph-long prompt. The decision was easy.

But the results weren’t without trade-offs. On a Latent Space episode in July 2024, Clémentine Fourrier of Hugging Face laid out what LLM judges are bad at: They prefer their own model family, and they can’t score on a continuous scale. Asked what benchmark she wished existed, Fourrier said, “Nobody’s evaluating model calibration at the moment.” With the release of Jev, the need for that benchmark is even greater, because real progress seems to have been made.

Jev gives you zero-shot probabilities without training and without a generator

Jev takes the same plain-English criteria you’d put in a judge prompt, needs no labels, and returns a probability. Zero-shot and autoregressive text generation are no longer tied together. We were paying for the second to get the first, and it turns out you don’t have to.

The spam evaluation I mentioned earlier is an impressive demonstration of how attractive this new offering is. With zero labeled examples and a simply well-written definition of spam, Jev hit 98.3% accuracy. A TF-IDF logistic regression trained on about 14,800 labeled emails hit 98.4%. The two disagreed on 466 emails and split them almost evenly, with no statistically meaningful difference between the two. So a classifier from 2003, trained on a dataset, only ties a decision model trained on nothing. It’s early data that’s yet to be reproduced, but if it holds up, that’s an amazing new capability unlocked.

But there are still some trade-offs you’re making.

A radically cheaper decision loses you some things

The biggest loss is the explanation. TypeSafe’s docs say plainly that System One models don’t generate explanations of their reasoning, and NearHere’s test noted the same thing: a category and probabilities came back, nothing else.

Depending on your use case, that could matter a lot. LLM judge explanations are an incredibly valuable tool that tells you not just what was wrong, but why. That provides real signal that can be fed en masse back to a coding agent and used to automatically improve your software. Jev on the other hand just gives you a probability, which leaves you with much less directional signal of how to improve.

Of course, at these prices, you can do both: run Jev on every single trace for broad, comparably accurate measurement and monitoring, and then take samples of failures and rerun them through an LLM judge to get your directional signal. That involves changing how you work, which is why I say that this may require rearchitecting your systems.

To automate a decision, you need to know which 5% to hand to a human

TypeSafe’s launch post makes the point that a model that’s right 95% of the time but can’t tell you when it’s in the other 5% can’t automate anything, because a person still has to review all of it. That’s an important point because it highlights a problem with LLM judges.

We evaluate LLM judges by accuracy against a gold set. Accuracy tells you how many errors to expect, but not where they will be. If your LLM application is making decisions for you, it feeds three things: a threshold that decides when to act, an escalation path that decides when to ask a human, and a drift monitor that decides when the world has changed under it. All three need a probability score, but LLM judges don’t provide reliable probabilities. A 2025 study of 14 models on JudgeBench found judges clustering their predictions at 90% to 100% confidence while landing well below that in accuracy, and argued for exactly this shift from accuracy-centric to confidence-driven evaluation.

The same small spam evaluation test shows what a usable probability looks like. Of the emails Jev scored under 0.1, 0.1% were spam. Of those scored 0.9 or above, 99.9% were. In the 0.5 to 0.6 band, only 38% were. That curve tells you where to set your threshold and how much human review you’re buying: Sending the 4.6% of emails scored between 0.3 and 0.7 to a person left the rest at 99.5% accuracy. Your overconfident LLM judge can’t get you there.

Another metric to consider is tokens per decision. A component that spends thousands of output tokens to emit one of five labels is telling you it’s the wrong tool for the job. UkisAI’s Swift-Qwen3.8-27B cut 58% of its reasoning tokens on GPQA-Diamond and lost 0.1 points of accuracy. A lot of these tokens aren’t making a critical difference to accuracy.

In Arize AX, eval labels, the judge’s explanation, and the token count and cost of the judge call sit on the same trace, so tokens per decision is a column you can sort by rather than a number you have to figure out.

Cheap decisions change the math of how you build and measure AI applications

As I mentioned earlier, at $0.0004 and 0.4 seconds a decision, you can stop sampling. You can check every output, every tool call, and every agent step as it happens. For some use cases that’s a total game changer.

But it might require that you rearchitect how your application works to make the most of it. Take the work and decompose into many small typed questions; only call the expensive LLM generator when text actually needs to be written. That’s a stack where the decision layer is something you can version, measure, and swap independently of the model that writes the words, and it’s the first time decision-making has been cheap and fast enough to make that practical without requiring training data.

So go count how many of your LLM calls end in one of five labels. Then work out what you’d check, and how often, if each of those calls cost a fraction of a cent and came back with a probability you could trust.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

Ask most engineers what MCP is and you’ll get the same answer: a way to plug tools into an LLM. Fair enough, as far as it goes. But that description treats MCP like plumbing, and after months building MCP-based integrations for large enterprise platforms, I don’t think plumbing is the right metaphor. Plumbing moves water through pipes you already designed. MCP changes who’s holding the wrench. Once you’ve felt that shift in a real production system, the “just another API standard” framing stops making sense.

This piece is the long version of that argument. It walks through what MCP’s primitives actually are and why they’re the right primitives, where the standard genuinely collapses integration work that used to be duplicated per framework, where the abstraction leaks in ways that only show up once you’re past the demo, and what the protocol’s own 2026 evolution tells you about where the real pain has been. Nearly everything worth knowing about building on MCP falls out of understanding these pieces and how they interact.

What MCP actually standardizes

Strip away the framing and MCP is a JSON-RPC-based protocol that lets a client (the thing driving an LLM) talk to a server that exposes capabilities, over a small, fixed set of primitives:

Tools are callable functions. Each one has a name, a description, and a JSON Schema describing its inputs. This is the primitive most people mean when they say “MCP,” and it’s the one doing the heavy lifting in most production deployments: “look up an order,” “run a query,” “create a ticket,” etc.

Resources are readable context, addressed by URI, that a client can pull in without the model having to call a function to get it: a file, a record, or a document, for instance. Think of this as the read side of the interface, separate from the “do something” side that tools represent.

Prompts are reusable templates a server offers to the client, so common workflows don’t have to be respecified from scratch every time.

On top of those three, the spec defines capabilities that flow the other direction, from server back to client: Sampling lets a server ask the client’s model to generate text on its behalf; elicitation, added in the 2025-06-18 revision, lets a server pause and ask the human for more input mid-task; and roots let a server learn which directories or URIs it’s actually allowed to touch.

None of these primitives are individually novel. What’s novel is that they’re the same five primitives regardless of which model, which framework, or which vendor is on the client side. That’s the entire value proposition in one sentence, and it’s also the source of everything that goes right and everything that goes wrong when you build on top of it.

Where the old model breaks down

Before MCP, wiring an LLM into an enterprise system meant writing tool-calling code for that specific model, that specific framework, that specific integration. Every agent framework had its own function-calling convention: its own way of describing a schema, its own way of parsing a model’s intent to call something, and its own error-handling contract. Every system you wanted to expose needed its own adapter written to whichever dialect that framework spoke. Add a second framework to your stack and you don’t get twice the work. You get a second, incompatible copy of the same logic, maintained by whoever drew the short straw.

MCP replaces that with one contract, written once, usable by any compliant client regardless of which model sits behind it. That’s the part every MCP explainer gets right, and it’s a real, measurable win. I’ve watched it collapse from a maintenance burden that used to scale with the number of frameworks a team happened to be supporting that quarter down to something that scales with the number of systems, full stop.

But the more consequential change is where the integration decision gets made. A traditional API integration is an agreement two systems make in advance. You negotiate a contract: endpoints, payloads, auth, versioning, and both sides build to it, because a project plan said this integration should exist. The plan predates the code.

An MCP server doesn’t get that luxury. It has no idea which agent will call it, in what sequence, alongside which other servers, in service of what goal a human typed into a chat box 30 seconds ago. The plan doesn’t exist as a concrete thing until the agent composes one, at runtime, out of whatever tools happen to be available to it. That’s not a stylistic difference from the old model. It’s a different category of integration problem, because the party doing the composing isn’t your code anymore. It’s a model, reasoning over natural-language descriptions you wrote weeks or months earlier, with no idea what context it would eventually be reasoning inside of.

The description is the interface now

A tool’s JSON Schema tells the agent what parameters it takes and what shape they need to be. That part is mechanical, and MCP handles it well. The tool’s name and description tell the agent when to use it at all, and whether to prefer it over some other tool that does something adjacent. Those are two different jobs, and only one of them is solved by a well-formed schema.

Picture two versions of the same tool description. The first is technically correct and nothing more:

{
  "name": "get_status",
  "description": "Returns the current status of a record given its ID."
}

An agent reading that has no idea when this is the right tool versus three other tools that also return some kind of status, no idea what “record” means in this system, and no idea whether IDs are case-sensitive, numeric, or prefixed. The second version spells out the domain the tool operates in, gives the ID format explicitly, states what the returned status values mean, and flags the one adjacent tool this one is commonly confused with and why they’re different:

{
  "name": "get_status",
  "description": "Returns the current fulfillment status for an order record. 
IDs are numeric order numbers (e.g. 48213), not SKUs or customer IDs. Status 
values are one of: pending, processing, shipped, delivered, cancelled. Use this 
instead of get_shipment_status, which returns carrier tracking events rather 
than the order's internal state."
}

That’s a longer description, and it will feel like overexplaining to the engineer writing it, because the engineer already knows all of this. The agent doesn’t. It’s encountering the tool for the first time, with a handful of tokens to decide whether it’s the right call, and no colleague to ask.

I’ve watched teams ship a technically correct MCP server that agents used badly, or avoided entirely in favor of a worse but better-described alternative, purely because of this gap. The failure mode isn’t a stack trace. It’s an agent confidently calling the wrong tool, or the right tool with an assumption baked in that happened to be wrong for this case, and nobody notices until the output looks slightly off downstream. Writing tool descriptions well is closer to technical writing and product design than it is to backend engineering, and it’s not a skill most integration teams (mine included, early on) walked in the door with.

Composition is emergent, and that cuts both ways

The entire appeal of MCP is that an agent can combine tools from servers that never agreed to work together, in combinations their respective authors never planned for. That’s also the risk, and it’s structural, not a bug you fix with better testing.

In a traditional integration, the sequencing logic (call A, then check its result, then decide whether to call B or C) lives in a script that a human wrote and a reviewer read. You can unit test it. In an MCP-based agent, that same sequencing logic lives in the model’s runtime reasoning, generated fresh for each task based on the goal it was given and whatever tools happen to be available in that session. You can’t unit test a decision that doesn’t exist until the moment it’s made.

A tool that behaves correctly in isolation, with the exact inputs its author tested against, can still produce a bad outcome the first time an agent calls it third instead of first in a chain, or passes it a value that came from a different server’s output rather than a human’s direct input. This is qualitatively different from a normal integration bug, because it doesn’t show up in code review, and it won’t show up in testing unless your test suite happens to exercise that specific, unplanned chain of calls. It shows up in production, once, when a particular combination finally occurs. That’s exactly the kind of failure mode that’s cheap to dismiss as an edge case until it happens to the wrong customer.

The protocol is catching up to its own success, in specific and telling ways

To be fair to MCP, it isn’t standing still, and the shape of its evolution tells you a lot about where the real production pain has been. The July 28, 2026 specification is the largest revision since the protocol’s November 2024 launch, and every major change in it traces back to something that broke, or nearly broke, at scale.

The protocol core is now stateless. The original design tracked sessions with an MCP-Session-Id header, workable for a single server instance but painful the moment you’re running behind a normal horizontally scaled fleet and discover that “any instance can answer any request” and “sticky session state” don’t coexist. Removing protocol-level sessions means the same request can be served by any instance behind ordinary load-balancing infrastructure, which sounds unglamorous right up until you’re the one who has to explain in an incident review why a routine deploy dropped a chunk of in-flight sessions.

Tasks formalize long-running work. A lot of real enterprise work (document processing, multistep approvals, anything involving a human in the loop) doesn’t complete inside a single request/response cycle. Before this extension existed, teams hand-rolled this with polling loops and webhook callbacks, each implementation slightly different, each one a source of its own edge cases. Tasks turn that into a first-class protocol concept.

MCP Apps let a server return interactive UI, not just structured data. That matters the moment a “tool” is something a human needs to actually look at and approve before it fires, which in any environment with real consequences attached is often.

Authorization was hardened to align with OAuth 2.1 and OpenID Connect. This one isn’t novel so much as overdue, and the gap it closes was a real one; see the governance section below.

A formal deprecation policy now governs the legacy HTTP+SSE transport, with a 12-month offramp. That’s the kind of unglamorous governance maturity a protocol only earns after it’s been run in production long enough for someone to need it.

None of this is exciting reading. All of it is the sound of a two-year-old protocol absorbing genuine operational scar tissue, which is a far better signal about its trajectory than raw adoption numbers. And the adoption numbers are themselves striking: The official registry tracks close to 10,000 distinct servers, Tier 1 SDK downloads run into the tens of millions monthly, and both the TypeScript and Python SDKs have individually crossed a billion total downloads. Competitors of the protocol’s original author adopted it within months. That combination, real scale plus a spec that keeps changing in response to real production failure modes, is a much stronger signal of durability than either fact alone.

Governance hasn’t caught up

Here’s what I’d want any team to weigh before connecting MCP to anything that matters. Independent security research through 2026 paints a specific, and specifically uncomfortable, picture of the current ecosystem.

Scans across thousands of publicly registered servers have found the large majority carrying file-operation patterns prone to path traversal. A meaningful share of tested servers are vulnerable to command injection or server-side request forgery, and there are documented, disclosed cases of tool description poisoning, where the attack lives in the text a model reads to decide what to do rather than in the code the tool actually executes. A closely related failure mode, configuration poisoning, targets the server’s operational baseline directly: stealthy permission changes or altered defaults that persist across sessions and are hard to catch in a normal code review because the malicious logic lives in configuration state, not application code. Multiple high-severity vulnerabilities, including at least one missing-authentication flaw in a major vendor’s own production package, have already been disclosed and patched.

None of that is a reason to avoid MCP. It’s a reason to treat it the way you’d treat any protocol that hands an autonomous caller real privileges inside your systems: skeptically, and with the controls in place before the agent gets access rather than after an incident teaches you why you needed them. In practice that means an explicit, enforced allowlist of vetted tools per agent rather than open discovery of whatever happens to be reachable; authentication on every remote endpoint with no quiet exception carved out for “internal” traffic; centralized, immutable audit logging of every tool call an agent makes; and secrets pulled dynamically from a real secrets manager rather than sitting in a server’s local config where a configuration-poisoning attack can find them. None of this is exotic. It’s the same discipline any experienced integration team already applies to systems with real privileges, applied here to a caller that can now improvise its own sequence of actions.

Here’s what I’d tell a team starting today.

Treat the tool description as reviewed engineering output, not documentation you write last and skim once. Test it against how an agent actually behaves when given it, not just against whether a human reviewer nods along.

Assume composition you didn’t plan for will eventually happen, and design tools to fail safely and legibly when it does, rather than assuming a chain of calls you never tested simply won’t occur.

Put governance in front of capability, not after it. The allowlist, the auth, the audit log, and the secrets manager are the entry price given where the current vulnerability data sits, not optional hardening for later.

And build against the current specification baseline, not whichever example repository you copied six months ago. The stateless core and the authorization changes in the July 2026 spec aren’t cosmetic; targeting an older baseline today is technical debt you’re taking on knowingly, on day one.

MCP earned the “not just another API standard” framing honestly. It didn’t get there by being a cleaner REST, or a nicer SDK, or a better-documented function-calling convention: all real, all incremental. It got there by changing who, or what, is actually doing the integration work at runtime. The parts of that job the protocol doesn’t standardize (how well you describe a capability, how safely your tools behave when composed in ways you never anticipated, and how seriously you take governance before you grant an agent real privileges) are exactly the parts worth taking seriously before you bet production traffic on it.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The following article originally appeared on Venkatesh Rao’s Substack, Contraptions, and is being republished here with the author’s permission.

The sole athletic achievement of my life came in 1993: winning the IIT Bombay freshman 50m freestyle race with a time of 41s. That got me into the college swim team (it was a bad recruitment year) and launched my brief and entirely undistinguished athletic career. By my senior year, however, my 50m time had improved to about 38s (not enough to get me off water-boy duty since the team had several exceptional swimmers with much better times). Interestingly though, it was easier for me to swim faster at the end of my career than it was to swim slower in the beginning. The reason was that in the interim, the coach had significantly improved my stroke and breathing technique. It was all about managed pacing, not raw intensity of effort.

There is a fairly deep literature behind this apparently mundane lesson. Daniel Chambliss’s classic 1989 paper “The Mundanity of Excellence,” based on years of fieldwork studying competitive swimmers all the way from local clubs to the Olympic level, argued that excellence is primarily qualitative rather than quantitative. Elite swimmers do not simply do more of what mediocre swimmers do, or do it harder. They organize their activity differently: Strokes, turns, training habits, attention, and countless other small practices combine into a qualitatively different way of swimming. The route to excellence is not therefore reducible to maximizing effort along some obvious scalar dimension.

The same insight is condensed in a maxim common in military and special operations circles: “Slow is smooth, smooth is fast.” In activities where speed really matters, trying to go fast naively is often an excellent way to go slowly.

I never really stopped thinking about this problem. My 2011 book Tempo grew partly out of a long-standing interest in pacing across performance domains: how people experience time while making decisions, how rhythms of action emerge, and how timing relates to effectiveness. One of the ideas that has stuck with me since then is that tempo is something to be managed rather than maximized. There is no universally correct speed. There are only tempos appropriate or inappropriate to the dynamics of the situation.

Which brings me, somewhat unexpectedly, to Dario Amodei.

Amodei recently made the case that frontier AI development should be deliberately paced. His argument is primarily a safety argument. AI capabilities, he believes, are advancing quickly enough that the processes required to understand, evaluate, align, secure, and safely operate them are having trouble keeping up. This is not quite the old proposal for an AI “pause.” Pacing means continuing to advance the frontier while deliberately managing its rate, allowing safety work and institutional capacity to remain within striking distance of capability. Sam Altman has now endorsed the basic proposition, and Demis Hassabis has made closely related arguments about frontier capabilities outrunning scientific understanding and governance capacity. Elon Musk, more tersely, has said that Amodei is right.

There is an obvious cynical reading of this emerging consensus. The leading frontier labs have powerful economic reasons to want a regulated frontier. A regime that requires enormous compliance budgets, restricts open-weight releases, discourages foreign models, imposes burdens that startups cannot afford, or legitimizes coordination among a small number of incumbents could turn “safety” into a remarkably effective mechanism for protectionism and regulatory capture. That suspicion is not paranoid. Open models increasingly constitute a competitive threat to proprietary frontier providers, and the politics around regulating them already feature explicit accusations of regulatory capture. There is an additional awkwardness: Coordinated pacing among nominal competitors looks uncomfortably like coordinated restriction of output, enough so that the legality of such arrangements under antitrust law is already being debated.

I don’t think we need to resolve the question of motives. Perhaps these CEOs are sincerely terrified. Perhaps they are sincerely terrified and understand perfectly well that the regulations they favor would strengthen their competitive positions. Perhaps the mixture varies by person, company, and day of the week. It doesn’t matter much for my argument. The proposition that the frontier should be paced is worth considering independently of the political economy of the people proposing it.

I also don’t share enough of Amodei’s safety premises to make his argument my own. In particular, I think a great deal of contemporary concern about runaway AGI, superintelligence, and “alignment” is badly framed, and often borders on the theological. But I increasingly agree with his conclusion.

In fact, I think there is a strong case for frontier pacing even if you are an accelerationist and your objective is simply to make technological progress happen as fast as possible. I am not myself an accelerationist. My preferred framing is closer to managed tempo. But if I were one, I would still favor pacing the frontier right now, for a simple reason: Maximizing the instantaneous velocity of the AI capability frontier is no longer obviously maximizing the rate of technological progress.

There is, however, an important difference between my conclusion and the emerging frontier consensus. Their natural solution is coordination at the top: labs agreeing upon thresholds, governments blessing the coordination, evaluators policing it, and eventually perhaps international agreements extending it.

In other words, cartelization, hopefully of a benign sort.

I would prefer to see how much frontier pacing can be produced from the bottom up through ordinary market mechanisms. The distinction matters. The objective should not be to decide administratively how fast AI is allowed to improve. It should be to stop artificially rewarding frontier velocity after frontier velocity has ceased to be the most important form of progress.

Getting inside the loop

A useful way to understand the distinction comes from another idea that startup culture has borrowed, and mostly misunderstood, from the military: John Boyd’s OODA loop. OODA theory says that you win by “getting inside the adversary’s decision cycle,” which is usually glossed as making decisions faster than the other guy. If you observe, orient, decide, and act faster than he can, the story goes, you eventually overwhelm him.

But inside does not mean faster. The objective is to operate within the decision dynamics of the system you are engaging in a way that lets you shape them. Against a human adversary, that may indeed sometimes involve accelerating until his ability to orient collapses psychologically. But it may also require waiting, withholding action, changing rhythm, or deliberately slowing down. In nonadversarial situations, the goal may not be collapse at all but harmonization for resonant support.

What matters is the right tempo at the right phase, not speed for the sake of speed.

Something analogous applies to scientific and technological progress. There is no enemy psychology to collapse, but there are still loops to get inside: observation, experimentation, interpretation, investment, construction, deployment, feedback, learning, and recombination. The useful question is not how rapidly one component of that system can be made to move. It is whether the tempo of development allows those loops to close. If one subsystem changes faster than the surrounding system can observe, understand, absorb, and respond to it, pushing that subsystem still faster can reduce rather than increase effective progress. It can induce fragility and collapse.

This, I think, is approximately where AI is now.

The simplest evidence is personal and almost embarrassingly mundane. Frontier AI is already overpowered for nearly everything I use it for. In my most advanced projects I may use the strongest model available (Fable for my coding projects) to plan an approach or make critical strategic decisions, but I can generally hand the resulting specification to a cheaper model (such as Opus or Sonnet) to do the routine work. For ordinary uses I don’t need anything close to the frontier. In ChatGPT, I no longer even know exactly which model I am talking to much of the time. Whatever the “think harder” control does is sufficient model selection for my purposes.

The situation increasingly reminds me of smartphones. There was a period when getting the newest iPhone produced a noticeable improvement in everyday life. Eventually the hardware got good enough that the upgrade cycle ceased to matter much. I kept an iPhone XS for almost a decade before replacing it with a 16. The frontier continued advancing; I simply fell off the frontier because my demand curve had stopped following it.

Something similar is beginning to happen with AI, except that the supply curve is moving incomparably faster. Six months ago I routinely maxed out token allotments. Now I don’t. Some weeks I barely use coding agents. This isn’t because I’ve become less interested in AI. It is because my own capacity to productively absorb AI output has become the constraint. I have projects to think about, things to read, people to talk to, and work to do in domains where AI cannot help me yet, or perhaps ever. I am already pacing myself at my own tiny personal frontier.

That is a significant change in the technological situation. The binding constraint is migrating.

When the bottleneck moves

Broader AI deployment is increasingly blocked by things other than model intelligence. Robotics has long been constrained by actuators, power, reliability, dexterity, manufacturing, and the sheer recalcitrance of the physical world. Those constraints are beginning to move, but making the model smarter does not make them disappear. AI in education is constrained less by whether a model can explain calculus than by our lack of sufficiently rich classroom experimentation about what happens when students and teachers actually use these systems. Current mid-tier models are probably capable enough to power almost any educational experiment worth trying in a high-school or undergraduate classroom. We do not need another order of magnitude of intelligence before conducting them.

This pattern should become more common as AI improves. Once intelligence ceases to be scarce, its complements become more important. Model capability can be abundant while classroom knowledge is scarce. Model capability can be abundant while actuators are scarce. Model capability can be abundant while electrical infrastructure is scarce. It can be abundant while organizational competence, human attention, scientific understanding, military doctrine, security practices, and good judgment are scarce.

This is not peculiar to AI. Capability-maxxing the coolest new weapon is bad military doctrine. The United States has enjoyed extraordinary technological superiority over its adversaries for decades and has nevertheless repeatedly discovered that superior equipment does not automatically produce strategic success. Logistics, doctrine, morale, training, political understanding, industrial capacity, and orientation matter. A force that neglects those complements because it possesses the best weapons can become remarkably fragile. From Vietnam to Iran, the US military has been repeatedly forced to relearn the lesson.

AI may now be entering the same regime. Fragility from neglect of everything non-AI is becoming a bigger risk than failure to token-max.

There is a further reason to suspect that continuing to redline the existing frontier may yield diminishing returns. The major labs increasingly appear to be competing along broadly the same technological S-curve. One suggestive sign is that they run into the same supply constraints: HBM, electrical power, data-center capacity, capital, and access to sufficiently large clusters. When a technological system moves onto a genuinely different S-curve, its important bottlenecks often change as well. If everybody’s problem is how to secure more of the same scarce inputs to do more of the same basic thing, that is at least suggestive that everybody is climbing the same sigmoid.

As I learned as a freshman swimmer, near the upper portion of an S-curve, pushing harder can become exactly the wrong acceleration strategy. You expend increasing resources for decreasing gains while starving exploration of the attention required to discover the next curve. Moving faster along an S-curve is not the same thing as accelerating technological evolution. Sometimes you have to back off the incumbent trajectory long enough to notice what the next trajectory is.

A cognitive ergonomics crisis

There is also a more immediate bottleneck that the AI industry seems reluctant to acknowledge: the humans at the frontier.

Startup people have been LARPing war for decades. This is one reason concepts like OODA became popular in startup culture in the first place. “War mode” usually means working extremely hard under conditions of strong personal financial incentives: long hours, high urgency, extreme focus, centralized authority, and a willingness to sacrifice ordinary organizational niceties. I’ve been around startup culture for decades, and I suspect the AI boom may be the first time the conditions have actually become meaningfully war-like.

AI frontier people are visibly unprepared for it.

Actual militaries and other frontline risk professions take the human consequences of sustained high-stress operations seriously. Soldiers, firefighters, emergency medical personnel, disaster responders, surgeons, pilots, and others operating in consequential environments develop elaborate practices around training, emotional regulation, redundancy, rotations, decompression, mandatory rest, checklists, after-action review, and recovery. These practices exist because motivation does not repeal physiology. Judgment deteriorates. Attention narrows. People make stupid mistakes. Emotional reactions become harder to regulate. Creativity disappears. Eventually people break.

Does frontier AI look like an industry managing itself accordingly?

From the outside, it looks closer to the opposite. People at the frontier have been operating under extraordinary pressure for several years with little respite. The cognitive ergonomics of their working conditions are a disaster. (We have a project going at the Protocol Institute led by Timber Stinson-Schroff to study this—contact him if you’re interested in participating in or supporting it.)

Competitive pressure, enormous amounts of capital, geopolitical attention, hostile public scrutiny, internal ideological battles, rapidly changing technology, and the conviction among some participants that their daily work may determine the fate of humanity are not normal occupational stressors. The rate of dumb, unforced errors appears to be rising. The quality of frontier discourse has, in my view, visibly deteriorated. People I once assumed were much smarter than me increasingly seem to be missing obvious things while becoming susceptible again to bad ideas I thought they had outgrown.

Tired people catch colds more easily; they catch bad ideas more easily too.

This produces a peculiar inversion of the conventional AI safety model. We normally imagine increasingly unreliable or dangerous AIs surrounded by reliable human supervisors. But what if we are increasingly producing extremely capable AIs surrounded by progressively less reliable humans?

“Human in the loop” is not much of a safety guarantee if the human has been metaphorically deployed aboard an aircraft carrier in a war zone for eight months without relief.

Some of the public testimony emerging from frontier organizations should perhaps be interpreted through this lens. I do not want to diagnose particular people from afar, and testimony from people who have worked closely with frontier systems should obviously be taken seriously. But when someone emerges from prolonged immersion at the frontier sounding psychologically shattered, there are at least two possible kinds of information in the signal. One concerns the technology. The other concerns what prolonged immersion at the frontier does to the observer. Frontier workers are sensors, but the sensors themselves are being perturbed by the phenomenon they are measuring.

We have seen versions of this going back at least to the Blake Lemoine episode at Google, when sustained interaction with LaMDA led him to conclude that the system was sentient. More recently, former frontier employees have emerged making extraordinarily grave predictions about where AI is heading, that ill-prepared, tech-hostile journalists are eagerly amplifying with lurid headlines.

The correct response need not be either “believe them and stop AI” or “they’re crazy and should be ignored.” Sometimes a sensible response to someone coming back from the front sounding shell-shocked and exhibiting symptoms of PTSD is: This person needs a vacation. We rotate soldiers partly because the testimony of exhausted soldiers matters.

The largest near-term AI safety concern may therefore be exhausted frontline humans supervising overpowered AIs.

Fighting the wrong enemy

Exhaustion is particularly dangerous when nobody can agree about what the enemy is. Much of the actual stress experienced by frontier organizations comes from a fairly comprehensible mixture of competitive pressure and techlash hostility. Those forces are intense, but neither is an existential adversary.

The clearest live adversarial problem involving AI is much more ordinary: humans using AI against other humans. Criminal applications are already real and deserve serious attention. Military applications are rapidly becoming real as well, and the relevant strategic picture is much broader than a stylized US-versus-China AI race. Smaller powers and nonstate actors can use cheap cognitive capability to lower engineering barriers that previously required deeper technical institutions. Recent reporting, for example, describes AI assistance being used in weapons-engineering work by actors in Houthi-controlled Yemen. That strikes me as the kind of development around which one can build a concrete threat model.

Longer-term military diffusion is clearly a serious concern. But much of the fear actually shaping frontier behavior seems aimed somewhere else entirely: toward vague runaway “AGIs,” “superintelligences,” and a metaphysically capacious notion of “alignment” inherited from philosophical traditions I find largely unpersuasive.

There is a useful analogy with climate change. Climate change produces actual physical stressors: more extreme weather, unstable agricultural conditions, infrastructure damage, wildfire risk, and so on. Those generate concrete political and humanitarian problems, including displacement and unmanaged refugee flows, while longer-term adaptation requires things like shoreline management, wildfire regimes, agricultural relocation, and preparedness for changing disease ecologies. Yet parts of climate politics have preferred to identify an ultimate metaphysical adversary called Capitalism, Markets, or Growth and an equally totalizing remedy called degrowth. Heterogeneous problems with different timescales and mechanisms get collapsed into one grand theory.

Parts of AI safety discourse increasingly strike me the same way. Competitive instability, cybercrime, weapons proliferation, institutional disruption, labor-market effects, and exhausted frontier personnel are all real and different problems. “Unaligned superintelligence” turns them into a single theological object. Once that happens, every stressor becomes evidence for the same threat model.

This is a kind of threat-model collapse. Adaptation is usually plural; apocalypse is singular. Real technological transitions produce dozens of mismatched rates and local failure modes, requiring different responses at different tempos. If criminals are the problem, work on security and law enforcement. If weapons diffusion is the problem, work on doctrine and proliferation. If operators are exhausted, rotate them. If schools lack experimental knowledge, run experiments. If power is scarce, build infrastructure. “Align superintelligence” is not a substitute for any of those things.

Pacing would give us something valuable here beyond safety: enough time to discriminate among threats.

Proof abundance, understanding scarcity

Mathematics may already offer a miniature preview of what happens when one part of a knowledge-production system accelerates far beyond the others.

Terence Tao has recently distinguished three stages of mathematical work: generation, verification, and digestion. AI is rapidly making the first two cheaper. Models can generate candidate proofs, while formal systems such as Lean can increasingly verify them. But digestion remains stubbornly slow. Somebody still has to understand what the proof is doing, relate it to existing mathematics, extract reusable techniques, explain it, teach it, and use the resulting understanding to generate better questions. Tao describes the resulting condition as an “impedance mismatch.”

This is particularly interesting in light of what I have elsewhere called the curiously playable universe: the apparently expanding set of domains that can be transformed into sufficiently explicit games that AI can optimize effectively within them. Anything that begins to resemble a CAD system, a formal proof environment, or an evolutionary optimization problem over a sufficiently well-defined parameter space becomes potentially tractable to extraordinarily capable models. More of the world appears to be playable than we previously thought.

But playability has an important pathology. A highly playable domain supplies a scoreboard, and once AI becomes extraordinarily good at optimizing the scoreboard, the relationship between winning the game and advancing the larger domain can weaken. Solving a theorem is valuable partly because, historically, getting to the solution usually required acquiring understanding along the way. If an AI can helicopter directly to the summit, to borrow Tao’s analogy, the summit has still been reached, but nobody necessarily learned the trails, landmarks, terrain, or neighboring geography encountered during the climb.

Tao and two dozen other Fields Medalists recently made essentially this point in a declaration strikingly titled “A Severe Misalignment of AI in Mathematics.” Their pointed use of misalignment is almost the reverse of its standard AI-safety meaning. The problem they identify is not that AI has developed alien goals. It is that the incentives of AI companies to demonstrate spectacular problem-solving performance are becoming misaligned with the goals of mathematics itself. Solving difficult problems has historically served as a proxy for mathematical understanding and progress. Once AI can optimize the proxy directly, the correlation can break.

The recent Navier–Stokes episode illustrates the issue. Enormous amounts of inference can now be directed at a famous open problem, candidate constructions produced, and formal verification generated at extraordinary speed. Yet that does not automatically produce a corresponding increase in comprehensible, reusable mathematical knowledge. The pipeline is something like problem selection → generation → verification → exposition → digestion → canonicalization → better questions. Increasing the bandwidth of generation and verification by orders of magnitude while leaving the downstream stages roughly unchanged creates a queue.

Proof generation becomes abundant. Understanding becomes scarce.

That is frontier pacing in miniature. Maximum local throughput does not imply maximum system throughput. Indeed, beyond a certain point it can create congestion.

Orientation beats equipment

There is a strategic implication here for organizations outside the frontier labs. The natural reaction to rapidly advancing models is to assume that whoever possesses the strongest model necessarily possesses an overwhelming advantage. If a frontier model can turn increasingly playable engineering problems into few-shot solutions, and if most of the necessary input information exists somewhere in public literature, then organizations can easily conclude that whatever intellectual lead they possess is temporary. Why bother competing with organizations that possess better models, more compute, more money, and privileged access to the frontier?

But this risks confusing equipment superiority with orientation superiority.

Boyd repeatedly emphasized that superior orientation could overcome substantial equipment disadvantages. (“We’d still have won if we’d swapped equipment.”) The relevant analogy today is something like centaur chess. Your model does not necessarily have to outthink their model. Your humans have to out-orient their humans.

This becomes increasingly true as frontier capabilities bunch together above the threshold required for a particular task. In my own work, I increasingly find that I can use the strongest model to formulate or specify a solution and then hand most of the execution to a weaker model. For many problems, even that is overkill. I would readily bet on a well-oriented person using a slightly weaker model against a poorly oriented person using the strongest available model.

And the frontier labs have no automatic orientation advantage. Quite the contrary: They are simultaneously fighting an extraordinary number of battles under extreme strategic distraction. They are building models, securing compute, raising capital, negotiating with governments, managing safety factions, defending themselves against critics, competing for talent, building consumer products, selling enterprise software, contemplating hardware and robotics, responding to geopolitical pressure, and trying to decide what sort of companies they are becoming. They may have a model advantage while suffering an orientation disadvantage. There is no reason to assume that organizations exceptionally good at building foundation models are exceptionally good at everything their models can be applied to.

This is another reason pacing can be strategically productive. It creates room for orientation. In an environment saturated with FUD and “resistance is futile” rhetoric, organizations can lose before competing because they assume frontier capability automatically determines every downstream contest. It doesn’t. Superior orientation does.

From one S-curve to the next

Put all of this together and Amodei’s proposal starts to look different. His concern is that capability is outrunning safety. I think capability may be outrunning almost everything.

It is outrunning our ability to deploy it productively. It is outrunning classroom experimentation, organizational adaptation, security practice, mathematical digestion, physical infrastructure, and human attention. It may be outrunning our ability to distinguish actual threats from theological ones.

And it is almost certainly outrunning the decompression and recovery cycles of some of the people charged with making the most consequential decisions about it.

An accelerationist should care about every one of these things precisely because an accelerationist wants acceleration.

The mistake is to identify acceleration with the derivative of a single visible variable: benchmark scores, parameter counts, inference budgets, training compute, or whatever happens to define the current frontier. Technological progress is a coupled system. Accelerating one component beyond the absorption capacity of its complements eventually stops accelerating the system. The problem becomes especially acute near the top of an S-curve, where enormous resources can be consumed eking out diminishing improvements while the exploration necessary to find the next curve is crowded out.

There is a useful precedent in the history of the PC industry. For years, processor clock frequency functioned as the wonderfully simple consumer metric for progress: 486 MHz was better than 400 MHz; 1 GHz was better than 800 MHz; higher number, faster computer. Manufacturers had every reason to compete on the legible scalar, and consumers learned to buy it. Eventually this became the “megahertz myth.” Different architectures could do very different amounts of useful work per clock cycle, while pushing frequency upward ran increasingly hard into heat and power constraints. By the mid-2000s, the industry was moving toward multicore designs and a more complicated understanding of performance in which throughput, architecture, workload, thermal limits and performance per watt all mattered. Intel itself acknowledged at the time that as computer usage diversified, factors other than clock speed were becoming increasingly important to platform performance.

AI benchmark culture looks increasingly like the early stages of the same mistake. A benchmark is useful because it compresses a complicated question into a number. When capability is scarce and improvements are large, the number may track value surprisingly well. As systems become overpowered for more uses, however, the proxy begins to detach from what customers actually care about. A model that goes from 87 to 91 on some benchmark may represent an impressive scientific achievement while producing essentially zero additional value for a company whose relevant workload was already handled adequately at 75.

This suggests a path to frontier pacing that does not require a council of frontier CEOs deciding how quickly everyone is allowed to move.

Customers can simply become harder to impress.

Enterprise buyers can demand demonstrated improvements on their actual workloads rather than accepting leaderboard gains as evidence of value. Developers can (and already do) route work to the cheapest model that clears the capability threshold rather than reflexively calling the smartest one. Researchers can value useful scientific infrastructure, explanation and reusable knowledge rather than merely celebrating another famous benchmark or theorem knocked down. Investors can become less impressed by capital expenditure whose primary justification is preserving position on a frontier whose marginal economic value is falling. Users can decline to upgrade when the previous generation is already good enough. Current enterprise behavior already points in this direction: Cheaper and open-weight models are becoming attractive precisely because many workloads do not require frontier intelligence, while buyers increasingly demand measurable returns rather than capability in the abstract.

None of these mechanisms requires anybody to agree upon a socially optimal rate of AI development. They simply improve the feedback signal facing producers. The market stops saying “more intelligence, at almost any price” and begins saying “show me what this additional intelligence is for.”

There are supply-side versions too. As power becomes a binding constraint, performance per watt and useful inference per dollar should matter more than sheer training scale. As inference proliferates toward edge devices and private deployments, latency, reliability, privacy and local controllability become competitive dimensions. As organizations discover that weaker models can execute plans produced by stronger ones, heterogeneous model portfolios should compete with monolithic frontier consumption. Open-weight and decentralized systems can keep proprietary labs honest by making “good enough” intelligence cheap and difficult to monopolize. A mature AI market should develop more dimensions of performance precisely as the mature processor market did.

This is the sort of pacing I would prefer: not a speed limit but a richer scoreboard.

The objective should therefore not be maximum speed. It should be managed tempo for maximum actual progress. Sometimes that means sprinting. Sometimes it means dwelling at a capability level while applications, institutions, infrastructure, science, and humans catch up. Sometimes it means letting one subsystem race ahead while another rests.

Sometimes it means deliberately leaving expensive capability unused.

That last possibility may be the hardest one for AI culture to accept, because we are still psychologically adapting to the idea that intelligence might actually be abundant.

A true sense of abundance does not require you to max out the bounty. Nobody hyperventilates because free oxygen might disappear before they get their fair share. When something is genuinely abundant, you can waste it. You can use a frontier model for a trivial question. You can use a weaker model because it is good enough. You can leave tokens unused. You can spend a week doing something that doesn’t involve AI. You can allow an extraordinarily powerful model to sit idle while you think.

The mark of abundance is waste, including nonuse.

Compulsive token-maxxing is in this sense still a scarcity behavior. So is compulsive benchmark-maxxing, compute-maxxing, and capability-maxxing. Train now because somebody else will. Deploy now because the window might close. Consume all the intelligence available because leaving any unused feels like falling behind. An industry behaving this way may possess an abundance of intelligence without yet having developed an abundance mentality.

Slack is not necessarily the enemy of acceleration. Slack is where people recover, where institutions adapt, where strange experiments happen, where understanding catches up with proof, where neglected complements receive attention, and where somebody finally notices that the old S-curve is flattening and another one is waiting nearby.

So yes, pace the frontier. But don’t turn the frontier labs into a cartel to do it. Let safety work catch up, but also let customers become bored with vanity benchmarks. Let exhausted researchers sleep. Let mathematicians digest their proofs. Let schools figure out what to do with the models they already have. Let robotics catch up. Let organizations learn to orient themselves in a world where intelligence is cheap. Let markets discover that efficiency, reliability, privacy, integration and domain-specific usefulness sometimes matter more than another few points on a benchmark. Let us discover which risks are real, which bottlenecks have moved, and which parts of the world turn out to be playable.

Then, when the situation calls for it, accelerate again.

Slow is smooth. Smooth is fast.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The early rapid expansion of AI capabilities that focused on frontier models was largely ushered into the world by a few powerful, US-based AI labs. Open-weight models released from labs in China, early on from DeepSeek, and later from Moonshot, Z.ai, and others, have in part disrupted that dominance. But growing concerns about the concentration of power have led to discussions about the need for at least some level of AI sovereignty.

AI sovereignty doesn’t necessarily imply total control of your AI stack. It holds the promise of having more localized security and privacy, better adherence to local jurisprudence (for example EU AI and data laws), more dependable service, and potentially more culturally specific outputs from the AI technologies used within a specified border or region. Negotiating interdependence is not necessarily a problem, but having an array of tools beyond just open-weight models and options in those negotiations beyond just commercial offerings is imperative.

Unsurprisingly, the same AI labs that gave rise to the need for AI sovereignty are also pushing their own solutions to the problem. While local institutions consider and even adopt some of these initial “sovereignty” offerings from these labs, open source AI technologies may offer a more promising horizon, more flexibility, and a means to manage dependencies. Tim O’Reilly argues open source AI is a potential opening for greater participation in the future of AI’s development.

From the underlying chip technology that is necessary for model training and inference to the cloud and data infrastructure that enables model development, companies such as NVIDIA, OpenAI, Google, Microsoft, and AWS have begun to stake out their own territory to maintain relevance within the global push toward AI sovereignty. Stanford University’s Human-Centered Artificial Intelligence Lab (HAI) lays out the different approaches and offerings these labs have developed in its report The Commercial Landscape of AI Sovereignty Offerings. It argues that while these labs “promise that countries will own their AI stack, [they also] deepen dependencies on U.S. Big Tech.”

There are some non-US-based commercial alternatives that offer their own “full stack” solutions or AI sovereignty for specific layers of the stack. Companies in Europe, the Gulf region, Asia, and elsewhere are positioning themselves as local alternatives to US tech oligarchs. According to the HAI report, “Many of the most mature and advanced companies are actively backed by their governments. In these cases, sovereignty is not just a marketing claim but a stated policy objective, with governments directing funding, structuring procurement, and, in some cases, selecting specific companies to build out domestic AI capacity on their behalf.”

These types of collaboration can both enable independence from US labs but may also create openings for political intervention. Claims about censorship and control of Chinese models emerged quickly after DeepSeek’s initial 2025 release. More recently, there have been probes into how US models may limit certain types of discourse. Moreover, the HAI report points out that often the offerings of these alternative providers still rely on the underlying technologies, specifically chips and cloud infra, of the US labs.

The proliferation of commercial offerings provides the space for diversification or potential leverage to negotiate better terms for collaboration, even with the dominant players. Open source AI technologies also play an important role in creating opportunities for even more diversification and greater sovereignty. As HAI argues, “Sovereignty strategies that do not consider the role of open-source AI risk normalizing fragmentation and political overreach.”

In order for open source AI to counter the diversification of commercial sovereignty offerings, these technologies must also proliferate beyond open-weight models. Arguing for a “federated system” of open source AI that enables sovereignty based on an “architecture of participation,” Tim O’Reilly writes that “the right infrastructure to let us satisfy both goals [of being everywhere and allowing everyone to have a say] will be a federation of models, a federation of protocols and code, and a federation of capacity. We need an architecture of participation all the way down the stack, and all the way up.” A key technology in the expansion of the open source AI stack these days are agent harnesses.

In a recent article, Mozilla CTO Raffi Krikorian argues that “the orchestration layer above the [model] weights is where capability is concentrating, and closed labs are already welding it shut”; therefore, it’s imperative to build on open harnesses, not just models. Commercial offerings that have dominated thus far include Claude Code and Codex. OpenClaw offered an initial disruption and promise for open source in late 2025, though the creator was quickly absorbed into OpenAI’s organization. While big tech labs continue to absorb when, who, and what they can, NousResearch’s self-improving Hermes agent harness has also garnered substantial attention now with over 230,000 stars on GitHub. More recently harnesses such as Pi and DeepSeek Harness are expanding that open source offering, heeding Krikorian’s call.

Beyond agent harnesses, some of the strongest open source projects are developing in the less visible layers. Inference engines such as vLLM, SGLang, llama.cpp, and ONNX Runtime make it possible to serve a range of models efficiently across data centers, regional clouds, personal computers, and edge devices. Ray, which was developed by researchers at UC Berkeley, distributes demanding AI workloads. Ollama lowers the barrier to running models locally. Together, these projects give institutions more freedom to change models, hardware, and hosting providers without rebuilding an entire system around another company’s proprietary platform.

Other fast-growing projects are filling out the data, interoperability, and accountability layers of the stack. The Model Context Protocol and Agent2Agent Protocol offer open standards through which agents can connect to tools and to one another. Qdrant, Chroma, Milvus, and LanceDB provide open infrastructure for storing and retrieving institutional knowledge. MLflow, Opik, and OpenLLMetry allow developers to evaluate, trace, and monitor AI applications without surrendering operational data to a closed dashboard. The AI Potluck Gap Map classifies inference, deployment, and agent protocols as mature open ecosystems but identifies resiliency gaps in storage and observability, where fewer fully open projects occupy the leading tier. These gaps point toward an important investment agenda. Sovereignty will depend not on finding a single open replacement for Big Tech but on sustaining interoperable public alternatives across every consequential layer of the stack.

Even still, the looming threat of acquisitions and absorption of open source is persistent. The fintech company Stripe recently bought OpenRouter, a platform that allows developers to access various models and has become a primary hub for accessing and routing open-weight models in particular. NVIDIA has acquired Hugging Face, one of the key players for the open source AI ecosystem, and it has also just settled a licensing deal with Poolside, which develops open-weight coding models, purportedly to avoid the oversight of complete acquisition.

AI sovereignty may mean the necessity of “calibrating interdependence” with US Big Tech solutions or replacing a foreign dependency with a domestic one for the time being. But it must also entail building the technical capacity, open infrastructure, and participatory institutions needed to preserve genuine choice across the entire AI stack. Open source will continue to be vulnerable to commercial absorption, and efforts to counter that must become more robust.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

Chester Ismay, a data science educator and AI consultant, created schedule viewers to keep up with the sports he follows, including the WNBA, NFL, NBA, and Premier League. But he still had to decide which games deserved his attention each week.

In this episode of Zero to Agent in 30 Minutes, Chester built a sports concierge agent to surface the games he should watch. It reads his preferences and current schedules, then sends a weekly summary to his phone. He took the audience through the setup, which combines a prompt file, limited tool permissions, a schedule, and notifications.

How to create your own sports concierge

1. Define your preferences. Chester started with a structured preferences file that identifies the teams he follows and adds context about why he follows them. He built a simple web interface for editing those preferences rather than working directly with the underlying JSON.
2. Give the agent access to current schedule data. His existing sports viewers pull schedule information from sources such as ESPN and store it in repositories on GitHub. A read-schedules tool, run with Node, pulls the latest schedule files and combines them with Chester’s preferences, giving the agent the information it needs without requiring it to search for each game on its own.
3. Write the agent’s policy. Chester spent most of the walkthrough on CONCIERGE.md, the file that defines the agent’s job. The policy sets the goal, identifies the data sources, and defines the rules for deciding which games to recommend. It also specifies how to handle finished tournaments and duplicate matchups, along with the expected output and delivery format. Chester noted that when the output misses the mark, he goes back to the policy and adds more detail to the instructions or adjusts them to better match with the goals of the project.
4. Run the agent with limited permissions. Chester used Claude Code to read the files, apply the policy, and generate the weekly recommendations. He configured permissions so the agent worked only with the files and tools required for the task.
5. Schedule delivery and check the results. Chester used launchd on his Mac to run the concierge every Wednesday and ntfy to send the result to his phone. He checks the recommendations against the underlying schedules and uses tests and multiple data sources to catch errors. Time zone handling required another adjustment. Games could fall on the wrong day when the system defaulted to UTC, so Chester added explicit time zone instructions.

Coming up next

Next week, AI engineer Sajal Sharma gives an agent its own computer in the cloud using services such as E2B and Scrapybara. He’ll demonstrate how sandboxing lets an agent install packages, run code, drive a browser, and control a remote desktop.

Follow along with Zero to Agent in 30 Minutes on Radar, or watch the latest episode on YouTube, Spotify, Apple, or wherever you get your podcasts. If you’re an O’Reilly member, you can watch live. Save your seat.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

This is the third post in a four-part series about post-training. If you missed them, read part 1 and part 2. The final post, on implementing your own pipeline, will be coming October 7.

Now that you understand the gist of reinforcement learning and supervised fine-tuning, it’s time to explore what post-training has actually accomplished in the frontier models you know and love.

Remember our prompt “Why do people like golden retrievers?” GPT-3 would often answer nonsensically. But that all changed in November 2022, with the launch of ChatGPT. Now “Why do people like golden retrievers?” actually returned a reasonable response like “Because they are affectionate, patient, and make excellent family pets” no matter who was typing (with no weird formatting tricks to consider). Anyone who could send a text message could get a response back on any topic.

I’ll cover some of those behavior changes below, then take you through ChatGPT’s training pipeline as described in OpenAI’s InstructGPT paper.

Conversational and helpful

The most visible impact of post-training is models that can chat with you and hold a relatively long conversation. This sounds simple, but it’s not.

Being conversational means more than responding to a question with an answer. The model needs to recognize when a question is ambiguous and ask for clarification or make the right assumptions in a quick response. It should adjust its tone and detail level to the context, for example being brief for a quick factual question, but thorough for a learning-oriented one. The model should be coherent across multiturn conversations without losing the thread. It also needs to handle messy real-world inputs: You attach a giant PDF and ask it to find one specific clause, and it should either find it or tell you it can’t, not hallucinate an answer.

Safety and alignment

Post-training is also the primary mechanism for making models safe. Safety in this context means a few things:

• Refusing to generate harmful content (like instructions for creating weapons, when asked)
• Avoiding biased or discriminatory outputs
• Not making up information when unsure (hallucination reduction)
• Respecting user privacy

However, you can define safety rules in whatever way you want and teach the model to abide by them, within the limits of what your reward signals can capture. If you think cats are unsafe, because you’re a dog person, you can teach the model that in post-training—as long as you can properly encode that into a reward signal.

Safety is often in tension with helpfulness. On one extreme, a model that’s too conservative will refuse reasonable requests, something that has frustrated many users. On the other extreme, a model that’s too permissive will comply with harmful ones. Navigating this trade-off is difficult. Ultimately, it comes down to determining where to draw the line, which is (as of today) a human decision within labs. Post-training is the tool to implement wherever the line is drawn.

Tool use and function calling

Tool use is one of the most practically important capabilities enabled by post-training. Tools include search engines, APIs, calculators, databases, and code interpreters. Being able to hit a search engine alone allows the model to not hallucinate, given its own knowledge cutoff. Tools are extremely useful ways for models to interact with the world, and are fundamental components in building agents.

Tool use is a set of new behaviors. The model needs to recognize when a user’s request would benefit from an external tool. It needs to know which tools are available to it, and not hallucinate a tool. It needs to formulate a correct API call with the right parameters. It needs to interpret the results that come back and incorporate them into a natural language response. It needs to do all of this seamlessly, without the user needing to know the details of the underlying tool.

This is taught almost entirely through SFT, at least initially. The training data includes many examples of conversations where the model correctly decides to invoke a tool that it has access to, constructs the right call, and processes the result. RL can further improve tool use by rewarding the model for correct tool invocations and penalizing unnecessary or incorrect ones.

As an example of tool use, let’s say you’re building a veterinary appointment scheduling assistant. A user asks: “My golden retriever has been limping since yesterday. Can I see Dr. Patel this afternoon?” A pretrained model might generate plausible but fictional appointment times. A post-trained model with tool use instead calls the clinic’s scheduling API, checks Dr. Patel’s availability, and responds: “Dr. Patel has an opening at 3:15pm today. I’ve tentatively held it for you. Should I confirm?” The model needed to decide if the user’s intent was urgent, select the right tool, construct the API call with the right veterinarian and time constraints, and present the result conversationally.

Tool use has expanded through the Model Context Protocol (MCP), a lightweight standard for connecting models to external services like Gmail, GitHub, or a company’s internal databases. Rather than building custom integrations for each tool, MCP provides a standard interface that any API can plug into, and different frontier models have now included learning MCP in their post-training recipes. Agentic frameworks take this further by allowing models to chain multiple tool calls together to accomplish common multistep tasks more easily.

Reasoning (“thinking”)

Reasoning models, or models that are trained to “think” before they answer, are an exciting result of post-training. Rather than producing an immediate response, these models generate an internal chain of thought, working through the problem step-by-step, before arriving at a final answer. As a result, their answers are more often correct than nonreasoning models that might guess at an answer.

This capability has an interesting relationship with pretraining and post-training. The raw ability to reason is latent in pretrained models; they’ve been trained on text that includes mathematical proofs, logical arguments, scientific analyses, and code with comments explaining the logic. But pretrained models don’t default to reasoning. They default to pattern-matching, which often produces plausible-looking but incorrect answers.

Reasoning models dramatically outperform standard models on tasks that require multistep logic: mathematical problem-solving, complex coding, scientific analysis, and planning. The improvements are not incremental. On the 2024 AIME exam, GPT-4o was only able to get 12% of problems correct on average. OpenAI’s o1 reasoning model solved 74% off the bat, with a single attempt. With 1,000 attempts and a learned scoring function to rerank the attempts, it reached 93%, a result placing it among the top 500 students who took the AIME math exam in the US.

More capable reasoning requires more compute, both during training and at inference time. Scaling laws meet post-training. Models that have learned to spend more inference (test-time) compute on reasoning tend to reach better answers and therefore exhibit higher intelligence. For some frontier reasoning models, the RL post-training phase uses as much compute as the entire pretraining phase.

The cost is not only in post-training compute but also in inference (test-time) tokens and latency. Reasoning takes up a lot of tokens and can result in a longer time to get a response back to the user. But the type of request matters. For a quick factual question, you don’t need reasoning. For a complex technical problem, the extra latency is well worth it. This is something that model providers can modulate during post-training.

The classic ChatGPT pipeline

As I mentioned above, the first post-training pipeline that captured global attention was ChatGPT’s, and it drew on the pipeline described in the InstructGPT paper. While modern systems use more advanced approaches today, this classic pipeline remains the conceptual foundation for nearly all alignment methods.

The pipeline has three stages, each building on the previous one:

1. Supervised fine-tuning (SFT) on human demonstrations
2. Training a reward model on human preference comparisons
3. Reinforcement learning with human feedback (RLHF) to optimize the main model using the reward model

Stage 1: SFT on demonstrations

The first stage is straightforward and teaches the model to follow instructions and behave like an assistant.

OpenAI contracted ~40 human labelers to label their data, and they were careful to filter for people who were good at identifying harmful outputs. The labelers had to write ideal responses to prompts. But what’s interesting is that the prompts came from two sources: (1) prompts submitted by real users through the OpenAI API and (2) prompts that labelers wrote themselves. The users had to write prompts too, because these were the days before ChatGPT. There weren’t that many real users with instruction-like prompts through the API to collect.

The prompts were diverse and mostly in English. There’s also an extensive data cleaning pipeline to remove duplicates and remove sensitive PII (personally identifiable information). Importantly, they split the training, validation, and test sets by human labeler. This is to avoid data leakage that could happen within a single user’s data between training and validation/testing.

The resulting SFT dataset had ~13,000 prompts, all with human-labeled responses. The base model was GPT-3 at the time, a pretrained model without any post-training. Using SFT, they trained GPT-3 for 16 epochs, which was effective for the final RLHF model. This was interesting, because for the SFT stage alone, the model overfit after just 1 epoch, but ultimately SFT was an intermediate stage so they picked the best checkpoint for the final RLHF model. They also mixed in 10% pretraining data during this phase, because it would help the next RL phase.

At this point, this SFT model could already be pretty useful: It could have a conversation and follow instructions, which is leaps and bounds beyond the pretrained GPT-3 checkpoint.

Stage 2: Preference data and reward modeling

This next stage is training the reward model. The reward model needs to grade millions of responses during RL training. In the original method, OpenAI’s team mainly trained the reward model on responses from the SFT model. However, as the policy model is trained in the RL loop and generates new, and likely better, responses from its evolving checkpoints, the reward model needs to stay robust. As a result, they also continually updated the reward model using responses from new RL checkpoints over time.

To train the reward model in InstructGPT’s RLHF pipeline, OpenAI needed pairwise comparisons of two model responses from one prompt, and a label for which one is better. For example, given “What’s 2+2?” and the responses are “4” and “Yes,” the label should say “4” is better than “Yes.” Note again that these are responses from the SFT model (and later, the RL-ed models during the RL training loop), not the pretrained base model. So labeling can only happen after you’ve SFT-ed your model. If you need to retrain that model, you likely need to relabel to make sure the reward model is trained on the right distribution of data pairs.

Reward model training

The reward model was small at 6B parameters, for both efficiency and stability, and included a head that outputted a scalar reward. They had tried multiple sizes, but found this was more stable than using the original 175B main model. It was also more compute efficient, as the reward model would take up extra compute, for both inference and training, on top of training the main model itself. More recently, reward models have become a lot larger, but note that they don’t have to be the same model or same size model as the main model.

To train the model, the loss was a cross-entropy loss that represented the log odds that someone would prefer one option over the other, in the pairwise comparison. This was done by taking the difference between the rewards of the preferred and unpreferred options. So in the example “What’s 2+2?,” if the reward model correctly assigns “4” a high reward and “Yes” a small reward, then the difference would be high and positive, and the loss would be small. However, if the reward model were to incorrectly assign “Yes” a higher reward than “4,” the difference would be high and negative, and the loss would be huge—discouraging it from outputting this result again.

One of the big challenges in training the reward model was overfitting, and OpenAI found that training for only 1 epoch would help prevent that.

Reward model data

The simplest way to get preference pairs is to generate two responses per prompt and have a labeler tell you which one was better. To make more efficient use of each prompt, instead the model would generate not 2 but 4–9 different responses per prompt that human labelers would rank from best to worst.

Rankings can be transformed into pairwise comparisons, so it was an efficient way to collect those preference pairs. A ranking of N responses yields N-choose-2 pairs. For example, a ranking of 4 responses results in 6 pairs, a ranking of 9 results in 36 pairs. That means with 33K prompts and 4–9 responses ranked per prompt, there would be 200K–1.2M pairwise comparisons used to train a separate reward model. That’s a lot of data, from relatively efficient data labeling.

This is a relatively efficient use of human annotations. Just compare it to SFT. It’s easier, cheaper, faster, and more reliable (higher agreement between people) than writing good responses from scratch, so this stage of human labeling wasn’t as tedious as in SFT.

However, using the pairs from rankings wasn’t straightforward in training. The reward model would overfit if they mixed the pairs randomly, even in just 1 epoch, because the pairs for a single prompt were highly correlated with each other. So instead, they would train all the pairs from the same prompt as one element in a batch, and normalize it. This was also computationally more efficient to run and score all the N responses at once together, e.g., just score 9 times and reuse those calculations in this pass, rather than 36 times for each pair if mixed into the dataset.

A quick note on terminology. This data is often called preference data, because it’s about collecting human preferences. The reward model can also be called a preference model.

Stage 3: RLHF (reinforcement learning with human feedback)

At this point, you have an SFT model that can follow instructions, and a reward model that can score responses. The goal of RL is to continue training the SFT model to produce responses that the reward model scores highly. If the reward model is any good, the resulting model will produce responses that humans would prefer.

The RL algorithm used was PPO. As you learned previously about RL terminology, the SFT model is the “policy” that takes actions (generating tokens) in an environment (the conversation). The reward model provides the reward after the policy generates a complete response, and a critic model calculates the expected reward, a baseline estimate that offers a more stable overall reward signal in training.

Here are the critical steps. I’ve covered some of them before and will dive into others in detail later on in this section.

1. Sample a prompt. The prompt comes from the dataset of 31,000 prompts that were gathered from users organically using the API. No need for human labels.
2. Generate a response. Then, the current policy generates a response. The current policy is the SFT model in the beginning, but as the policy updates, it’s a new model that generates responses to be graded. A single prompt-response pair is called a “rollout.” In practice, this all happens in a batch of rollouts.
3. Calculate the reward. The reward model grades the full response with a reward. For every token position in the full response, they subtract a per-token KL divergence penalty between the current policy and the original SFT model. The per-token KL penalty and the reward model score added at the final token make up the per-token reward signal.
4. Calculate the advantage. The critic estimates the expected reward for the full response, at each token position in generation (so with partial knowledge of the full response). The critic’s expected rewards and the per-token reward signals are combined, using an algorithm called GAE (Generalized Advantage Estimation), to estimate how much better the reward was compared to expected. This is the advantage. In InstructGPT, the critic was initialized with the same weights as the reward model, giving it a head start on estimating expected reward. 
5. Update the policy. Then, PPO updates the policy model’s weights with the advantage, pushing it towards rollouts with higher advantage and away from ones with lower advantage.
6. (Optional) Mix in pretraining data and the pretraining objective in the policy model’s loss function to prevent catastrophic forgetting. 
7. Update the critic to better predict expected future rewards at each token position, by using the actual per-token rewards (from the reward model and KL penalty) as the targets in training.
8. (Optional) Update the reward model. Collect new ranking data on the current best policy and train a new reward model. In practice, OpenAI did collect some data from the PPO models, but most was from the original SFT model.
9. Repeat! This RL loop repeats over many prompts and many updates, with the latest policy always generating its responses.

The KL penalty

If you just let the model maximize the reward model’s score with no constraints, it finds weird, degenerate outputs that exploit quirks in the reward model to get high scores without actually being good responses. This is called “reward hacking,” and it’s one of the central problems in RLHF.

To address this, OpenAI added a KL divergence penalty between the RL policy and the original SFT model (“reference policy”) in the reward calculation. In AI, KL divergence is a common method of measuring how different two probability distributions are. In this case, it would measure how different the RL policy is from the old policy and penalize being too far from it, essentially telling the model: You can optimize for higher reward, but you can’t drift too far from where you started. If the RL model starts producing outputs that look nothing like what the SFT model would produce, the penalty helps to pull it back by making the reward for those outputs lower.

The total reward for a response becomes the reward model’s score minus the KL divergence from the SFT model, with a coefficient term that weighs how much to care about the KL divergence. If the coefficient is too low, you’re saying that you don’t need to penalize drift from the reference policy, and you’ll get reward hacking. Too high and the model barely changes from the SFT checkpoint.

They also mixed in a significant amount of pretraining data during the RL phase, adding a pretraining loss alongside the RL objective. This was to prevent the model from degrading on general tasks from pretraining, like knowledge recall or coherent long-form creative text, as it optimized for reward. This is sometimes called the “alignment tax,” where you trade-off alignment for general capabilities, a type of “catastrophic forgetting.” This is an active area of research.

So the final RL objective combined three things: (1) maximize the reward model’s score on prompted responses, (2) stay close to the SFT model via the KL penalty, and (3) maintain performance on pretraining data. This means improving on the things humans care about without losing what the model already knew how to do from SFT.

Practical details

The critic reduces noise and makes training stable enough to make PPO work practically. In practice, OpenAI initialized the critic from the 6B-parameter reward model, since it’s already trained to predict reward and gives the critic a head start as it is further trained in the RL loop.

The RL training was computationally expensive and involved running several models simultaneously: the policy model (the main model being trained), the critic (estimating the reward as a baseline, also being trained), the reward model (grading responses), and a copy of the SFT model (for computing KL divergence).

That’s four models in memory at once. That’s a lot of GPU memory, especially when the policy and SFT models are 175B parameters! In addition to weights, the policy and value models also needed their gradients, optimizer states, and cached activations for backpropagation because they were being trained, which can actually multiply the per-model memory cost by 3-4 times. This is another reason the reward and critic models were kept at 6B.

PPO also requires generating fresh rollouts during training, which is much slower than SFT where you already have all the data upfront. Each PPO training step also requires grading each rollout with the reward model, computing advantages with the critic, and updating both the policy and the critic. All these moving parts make the system harder to tune and debug compared to SFT, and harder to parallelize than pretraining.

Hyperparameters like learning rate, the KL penalty coefficient term, the number of rollouts per batch, and the clipping ratio all matter and interact with each other. The whole system depends on the quality and representativeness of your human annotations. Many RL training runs fail or produce degenerate results.

Getting it right

Getting all this right takes significant engineering effort and experience, but the first step is deeply understanding the pieces. Modern methods have addressed many of these issues, but the ideas from InstructGPT remain the foundation of post-training today.

Ultimately, human evaluators compare all the models. People preferred the RLHF model’s outputs over the SFT model’s, and the SFT model’s over base GPT-3’s. Each stage of the pipeline added a large improvement in the model’s response quality.

RLHF was extremely effective. In experiments, human evaluators preferred even a tiny 1.3B parameter RLHF model over the 175B parameter SFT model, most of the time. That’s a model over 100x smaller, trained with RL, beating a much larger model trained only with SFT. This made a strong case that how you train matters as much as how big your model is. Overall, the largest RLHF model still beat the smaller RLHF model.

The RLHF model was also better at following explicit constraints in instructions, less likely to produce harmful outputs, and hallucinated less, though it didn’t eliminate hallucinations as you may remember when you first used ChatGPT (and even now).

One caveat worth noting: The labelers who evaluated the final model were the same population who created the training data. When they tested with held-out labelers who hadn’t been involved in data creation, preferences for the RLHF model were still positive but less dramatic. The model was, to some degree, optimized for the preferences of a specific group of people. This means if you create the data to follow your preferences, the model will optimize for those.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >