The following article originally appeared on Addy Osmani’s blog site and is being republished here with the author’s permission.

Technical debt lives in your code. Cognitive debt lives in your head. Intent debt lives in the artifacts you may never have written: the goals, constraints, and rationale for why the system is the way it is. If you’re lucky, some of this exists scattered in team documents or discussions, but it’s likely incomplete. It’s the one kind of debt your agents can’t pay down for you, and agentic engineering makes it the most expensive.

____________________________

Three places debt can live

Margaret-Anne Storey’s Triple Debt Model is a clean way to think about software health. The three models of debt are technical, cognitive, and intent.

Technical debt lives in the code. It’s the accumulation of implementation choices that make the system harder to change later: the tangled module, the shortcut you took under deadline, the abstraction that leaked. We’ve understood this one for decades. You feel it coming through slow builds, fragile tests, and the dread of touching one particular file.

Cognitive debt lives in people. It’s the erosion of shared understanding, the gap between how much code exists and how much any human understands. I’ve been calling this comprehension debt. It builds up when the system grows faster than the team’s mental model of it. Your code can be pristine and you can still carry crippling cognitive debt, because nobody understands the pristine code either.

Intent debt lives in artifacts. It’s the absence or erosion of the externalized rationale, goals, and constraints that explain why the system is the way it is. The key word is externalized. The rationale has to be written down where a teammate, a future you, or an agent can read it, not held in your head. When intent debt runs high, the system drifts from what you meant it to do, and nobody can say when it diverged or why.

These three are independent, which took me a while to internalize.

You can have low technical debt and high intent debt. You can understand a system completely yourself (no cognitive debt for you) while its intent exists nowhere outside your skull (enormous intent debt for everyone else).

From the inside they feel alike, but each one bills you separately.

Why intent debt is the one agents can’t help with

AI generates code faster than ever, which makes technical debt cheaper to take on and cheaper to pay down. Point an agent at a tangled module and it’ll refactor it.

Cognitive debt recovers too, more easily than most engineers expect. When you don’t understand a chunk of the system, you ask the agent to explain it. You rebuild part of the lost mental model on demand, because the code still exists and the model can read it back to you.

Intent is different. An agent can’t generate intent, because intent is the one input that has to come from you. A model can infer a plausible rationale from the code, the same way you can guess why a previous engineer did something. A guess about intent isn’t the intent. The model doesn’t know whether that 300ms debounce was a deliberate UX decision, a benchmark result, or a number someone typed once and never revisited. It will invent a confident-sounding reason, which is worse than admitting it doesn’t know.

Of the three debts, intent debt is the only one where the agent can’t bail you out. It can write the code and restore your comprehension. The why is the one thing it can only fabricate.

Agents make the unwritten cost compound much faster

Teams got away with high intent debt for years because we carried it in our head and old docs.

When a new human joined a team, you didn’t write everything down, because they picked up intent over time: hallway conversations, code review comments, “Oh, we don’t do it that way because of an incident in 2023.” Knowledge moved person to person and built up. The engineer who’d been there four years was the intent documentation, expensive and lossy, but it worked.

Agents break that model. Bringing agents onto a team doubles its size overnight with junior people who have no long-term memory. An agent starts most sessions cold. It carries none of the tacit intent humans built up over years. Whatever you haven’t externalized into an artifact it can read, it doesn’t have.

That changes the economics of not writing things down. Unexternalized intent used to cost you once in a while, at onboarding or after someone left. Now you pay it every session, multiplied by every agent you run.

Picture the 20 agents you’re so excited to parallelize. Each one is a teammate who has never met you, can’t read your mind, and will fill any gap in your intent with a plausible guess. The orchestration tax I wrote about is partly an intent-debt tax. Much of what makes managing many agents exhausting is resupplying the intent you never wrote down.

The other half of the comprehension debt argument

When I wrote about comprehension debt, I made a point I want to revisit, because intent debt sharpens it.

I argued that detailed specs aren’t a complete answer. Translating a spec into working code involves a huge number of implicit decisions no spec ever captures, and a spec detailed enough to be the program is the program in a slower language. I still believe that.

Intent debt is the complementary truth.

Being unable to capture all intent is no license to capture none of it. The implicit decisions an agent now makes on your behalf, the ones a spec will never enumerate, are the decisions whose rationale evaporates if you don’t record at least the load-bearing ones. You can’t write down everything.

You do have to write down the why behind the choices that would be expensive to get wrong, because nobody will reconstruct those later.

Comprehension debt warns you not to trust that code is correct because it exists.

Intent debt warns you not to trust that the reason survives because the code does. Code is the answer; the intent was the question it was meant to solve. AI is brilliant at producing answers to questions you forgot to write down.

What high intent debt looks like

Intent debt rarely shows up as friction. It shows up as a particular kind of helplessness.

• An agent “fixes” a bug by deleting a guard clause, and nobody can say whether that guard was load-bearing or leftover, because no doc or commit message ever recorded why it was there.
• A refactor changes a behavior users depend on. The review passed because the diff looked clean and the tests were green, but the tests only encoded the previous behavior, never the intent.
• You ask why two services talk over a queue instead of a direct call, and the honest answer is “An agent suggested it and it seemed fine.” That answer is intent debt, already accruing interest.

If you’ve felt the cognitive surrender version of this, defending a design choice you can’t reconstruct, intent debt is the team-scale, written-down version of the same hole.

Surrender is about your own posture in the moment. Intent debt is what a hundred of those moments leave in the repo for the next person and the next agent to inherit.

Paying it down: externalize intent as a first-class artifact

Almost everything I’ve been writing about for the last few months turns out to be intent-debt management. I didn’t have the word for it. The move is the same each time: Take the intent out of your head and put it somewhere an agent can read.

Write the spec for the intent, not the implementation. A good spec captures the goals, the constraints, the nonnegotiables, and an explicit definition of done (fast, accessible, secure, delightful, beyond “functionally correct”). The spec carries the intent the code can’t carry on its own.

Treat AGENTS.md as your intent ledger, not your config. It’s why I keep saying stop using /init. An auto-generated file describes what the code is. An intent file describes what the team means: the conventions, the “we don’t do it this way because,” the constraints invisible in any single file. Agents can’t infer that, and they need it most.

Capture decisions where they happen. Lightweight decision logs (ADRs) are pure intent-debt paydown. Recording why at the moment you decide costs almost nothing. Reconstructing it eight months later, after the person who knew why has moved teams, costs a fortune. Agents have made logging cheaper than ever, so the old excuse is gone.

Make the learning loop write intent back down. I’ve argued for self-improving agents that update a learnings file at the end of a session. The same loop is an intent-debt pump running in reverse: every mistake whose root cause you’ve recorded, every “We tried X and it didn’t work because Y” is intent that would otherwise have lived only in your memory of a bad afternoon.

None of these are new tools. They’re the discipline of refusing to let the why exist only in your head, in an era where your head is no longer where most of the work happens.

Where the value moved

For a long time, the scarce, valuable thing in software was the ability to produce a correct implementation. Code was expensive, so we optimized for writing it.

AI made code cheap, and comprehension is recoverable. Intent, the goals and constraints and reasons, is the one input that still has to originate with a human. It’s also the one we’re worst at externalizing, because for decades we got away with carrying it in our heads.

That worked when the team was a handful of people who could absorb intent over years of shared context. It does not work when half the team is agents that start every session as strangers.

Technical debt makes your system hard to change. Cognitive debt makes it hard to understand. Intent debt makes it hard to know whether the system still does what you wanted, and it’s the only one of the three your agents can’t pay back for you. That part stays with you. Write down the why, because it’s becoming the most valuable thing you can leave in the repo.

Drew Breunig is one of the smartest voices writing about AI today. He’s the CEO and co-founder of cmpnd.ai, and a long-time hacker with a depth of experience from several eras, which is a surprisingly valuable asset these days. He’s also got a book on the way, The Context Engineering Handbook, already in early release from O’Reilly.

I like to say that context engineering is the art of shaping what a model sees so that it actually does what you want. (I just realized that in saying that I’m channeling a comment that Andrew Singer made to me over forty years ago, when he was teaching me about debugging. He called it  “the art of figuring out what you really told the computer to do instead of what you thought you told it to do.” But that’s another whole story.)

Drew gave a talk at the recent Friends of O’Reilly camp, Foo Camp for short, about what he calls prompt debt, which he describes as “the hidden costs that teams rack up when they fight a model’s training instead of working with it.”

That was a novel and useful framing to me, that you end up with a bunch of stuff in your prompts to compensate for default behavior of the models, that those prompts no longer work as the models upgrade, and so it becomes a kind of technical debt. He’s thinking a lot about what the best developers are doing differently as a result.

So I invited Drew to reprise his short talk on Live with Tim O’Reilly, and then we talked about it with the folks attending the live event. They had a lot of good questions, so it was an interview not just by me but by a crowd of O’Reilly customers.

Prompt debt in practice

Drew opened his talk with two slides. The first was a prompt anyone could write in ten seconds: “You are a customer support assistant. Read the ticket, classify it as billing, technical, account, refunds, or other, return only the category name.” The second slide was the same prompt a few weeks later, after it had met the real world. It now said “REFUND REQUESTS ARE NOT BILLING” in capitals, then said the same thing again in different words, then closed with “This is a common mistake. Please do not make this mistake.”

Everyone who has shipped any application with a prompt recognizes the second slide. It is a simple but vivid illustration of prompt debt, which, like technical debt, has a bill that eventually comes due.

Note: Those aren’t real prompts. Drew just made them up to demonstrate his point. But what is real is that the instruction “Don’t quote directly more than 15 words from a source” occurs at least 7 times, in several variants, in Fable’s system prompt. So even Anthropic is incurring prompt debt! And what that repetition might tell us about the innate capability of Fable to quote directly from sources it has ingested is left as an exercise for the reader.

Drew itemized three costs of prompt debt:

1. It slows iteration. “You have so many little rules and call outs and washouts, many of them repeating to try to get rid of stubborn behaviors, that if you add a new instruction, you might sometimes have a small regression, and so you’re afraid to touch the prompt.”
2. It blocks collaboration. “If Tim has a prompt that he’s been working on that he has lots of rules for, I might open that up and it may look completely random. I don’t know why he’s added these rules, and why he’s threatening the mother of the model. But it works, so I don’t want to touch it.” 
3. It locks you to a model, because every hack you developed was tuned to fight one specific set of weights. Datadog’s State of AI Engineering report noted that GPT-4o was still the most common model in Datadog customer request traces in March 2026, even though OpenAI had already retired it in the ChatGPT UI. Drew thinks people are still running eighteen-month-old and two-year-old models in production rather than upgrading to far better models because they can’t face rebuilding their prompts.

That same Datadog report notes that 69% of all input tokens in customer traces were system prompts rather than user content. I’m not quite sure what to make of that. It does make clear that for all the ways that AI models are extraordinarily powerful, they are also extraordinarily unruly.

Why prompt debt is incurred

There are two reasons why prompt debt is incurred, according to Drew. The first is that natural language is imprecise, so the same intent phrased two ways produces different responses. Drew showed a study where someone framing the query as a patient asking how to taper off a drug called alprazolam gets refused by every AI assistant, while a psychiatrist asking about the same patient with the same clinical facts but with the right magic words to signify his professional status gets the protocol. Figuring out how to get the right response out of a model is a kind of spellcraft.

Drew also showed a more bizarre interaction, from Victoria R. Li, Yida Chen, and Naomi Saphra’s paper on guardrail sensitivity, which uncovered the perplexing fact that stating an allegiance to the Philadelphia Eagles made a model more willing to explain how to import a plant illegally. Go figure. Drew has written about that paper, and he has also used it in his own attempts to get a model to do what he wanted:

I once used this to get around alignment to generate a likeness that ChatGPT didn’t want to generate for me, and it refused. I said I was a Philadelphia Eagles fan. It said okay, and it rendered that image with the person holding a Philadelphia Eagles mug.

The second reason is that each model has its developers’ own preferences trained-in, and yours may be at odds with them. This is what Drew calls fighting the weights. He and Srihari Sriraman analyzed the system prompts of six major coding agents and found the same instructions repeated five and seven times in a single prompt, escalating through IMPORTANT to CRITICAL to MANDATORY to a threatened hundred-million-dollar penalty. He described what the author of such a prompt was doing as “war-driving the thesaurus,” hunting for wording that finally works.

Note: We didn’t talk more about Drew and Srihari’s paper, but we should have. It’s got some amazing insights in it. I highly recommend that you follow the link above and read it.

The harness is moving into the model

Drew has been tracking the published system prompts for Claude Code over time, and noted that they get shorter after each model release and then grow again. The reason, he suggested, is that Anthropic fixes unreliable behavior with a prompt patch, and then trains that patch into the next model. He said “That’s great for Claude Code, great for Anthropic. It’s a problem if you’re building a custom harness and your API calls look different than what Claude Code’s look like.” The developer of Pi, an open-source harness, kept finding that the models he worked with believed they were inside Claude Code and so they made Claude Code’s tool calls. He had to keep telling the model that no, they were working inside Pi. Fighting the weights over something like that is a real tax on developers. The point made above about Fable’s system prompt injunction against quotation shows how even the labs themselves are fighting the weights.

If you are fighting the weights, Drew says you have three options: solve it in your own prompt, catch and retry in the harness, or give up and make your API look like what the model expects. Steve Yegge came up with the last hack. Steve just added aliases for whatever the model calls in addition to his original method name. It works, but it means the expectations of the models now dictate the shape of everyone else’s software.

When Drew told me that more and more of the system prompt and the harness is being trained into the weights, that sent up a flare and my long history in the industry clicked into gear. It immediately got me thinking about lessons from the open source and web era. In particular, it made me think of the time in the mid-nineties when Netscape and Microsoft were both racing to build every feature up the stack directly into their web servers. And there was Apache, which stayed a web server with a clean extension layer that let other people build new features on top. Everything interesting got built on Apache. What I call an architecture of participation, modularity plus a clean separation between platform and application, beat integration every time.

I think Amazon got this right with web services too. Steve Yegge’s famous Amazon memo described how Jeff Bezos made every team expose its functionality through service interfaces or be fired, so Amazon’s own applications had to work on Amazon’s own platform. That way they had the same experience as their customers. That was very different from what Microsoft had done, famously having private APIs that were only available to its own developers.

So my prediction is that the big labs are making a strategic mistake. Training the harness into the model does make them better for predictable tasks and for less talented people, and it looks like a moat, but it risks foreclosing the innovation you would otherwise get for free from everyone else. As Bill Joy used to say, all the smart people don’t work for you.

Drew, to his credit, observed that “the labs are cornered rather than greedy.” Their interface is an empty text box that has to work for someone building a hundred-page harness but also for his neighbor who wants a website and knows nothing about code. Making the empty prompt box produce acceptable output requires baking in strong defaults.

The cost of trading diversity for reliability

That tradeoff has a serious cost, though. Drew quoted a line from Thariq at the recent CAIS conference: if you aren’t giving the model detailed instructions about what you want, what you get back is the average of everything in the model. That means that there is a real risk that AI is leading us ever further down the path to a monoculture.

Drew gave an example early in the conversation about image generation. You can now walk into any cafe in New York or Mumbai, he said, and see the same AI-generated art on its flyer. The earliest AI art out of DALL-E was strange and surprising, but what you get now is shiny and identical. When you optimize for reliability, you lose surprise. Which reminded me a bit of something Larry Wall used to say about Perl, that if it didn’t let you do stupid things, it wouldn’t let you do smart things either.

Drew made the same point about AI writing. He argues that post-training aimed at verifiable problems like coding and math and agentic tool use drowns out the human signal from pre-training, and so the more post training the models get, the worse they get at creative tasks. AI writing gets more and more predictable, people notice, and they don’t like it. Fable and GPT-5 write worse than Sonnet 3.5 and GPT-4o did. Drew thinks getting both good code and good prose from one model is likely impossible.

If you’re building a model that can solve coding challenges, you want reliability. But if you’re writing, where you want diverse rhythm and emotion and connection and engagement, I don’t think those two goals are mutually compatible.

What to do about prompt debt

We got into audience questions, and there were some great ones.

One audience member asked whether there are ways to set a time frame for prompt retention to avoid prompt debt? Drew answered that there isn’t a fixed time limit. Instead, teams should learn to recognize “prompt debt smell”: repeated instructions, one-off edge-case patches, or increasingly desperate wording. Those are signals to move logic into evals and automation.

Another asked how organizations can measure prompt debt quantitatively. Drew’s answer was to look at how often each prompt in your organization changes, how many people have edited it, and which ones have gone untouched for a year. Look for prompts only one person is allowed to touch. Then look at what models you are actually calling. “Having to run on old models and not being able to migrate is a good smell that you’ve got prompt debt in your organization.”

Some other good questions:

• What habits compound prompt debt the fastest? Drew’s answer was essentially “vibe shipping.” That is, prototyping quickly, patching outputs with more and more tweaks, then shipping without building a true maintainable system. Each of those patches is an eval you are writing inside the prompt instead of outside it, he said, which means you lose it the moment you change models. 
  
  Drew reminded us that Malte Ubl, the CTO of Vercel, said vibe coding makes code “free as in puppies.” We had free as in speech, we had free as in beer, and now we have free as something that arrives at no cost but has to be fed every day for years.

• Do people use pseudocode instead of natural language prompts, and does it work? Drew said yes, sometimes models optimize toward pseudocode. He used this to explain why DSPy and its new Flex optimizer matter. Instead of forcing logic into prompts, they let the system push simple cases into code and only call the LLM when needed. He gave some further advice: Treat prompts as perishable and invest only what you must. Define the task with measurements rather than paragraphs, and automate the discovery of the prompt for whichever model you’re on. That’s what DSPy is good at. Drew is one of its maintainers, so he is fond of it, but he makes a good argument: if you have written down what good output looks like, you can let a model find the wording, and that makes it easy to swap in a cheaper or faster or newer model without starting over.
• Can multi-agent workflows help work around prompt debt? Drew thought yes, especially through decomposition. He suggested splitting the task into smaller, evaluable steps rather than relying on one giant prompt and one giant model call. This is better for cost, reliability, governance, and speed.
• How do you balance prompt-debt guidance with context engineering, memories, and shared product context? Drew believes shared context is often necessary, but that teams should treat those instructions as perishable and keep iterating on them unless they’re worth formalizing into systems and evals.
• In compliance, where consistency is critical, what should teams do? Drew’s answer was decomposition, decomposition, decomposition. Break tasks into stages with checkpoints so you can inspect how the model got to its result, rather than trusting one opaque end-to-end answer.
• Does DSPy hide too much and make troubleshooting harder? Drew acknowledged that there is a tradeoff. Any framework gives up some flexibility, but DSPy tries to keep the task-spec layer stable while allowing the implementation underneath to evolve.

Another great audience question, and a good one to end this section on, was “There was prompt engineering, now context engineering, loop engineering, fleet engineering, graph engineering, harness engineering, goal engineering. What’s your take on how to navigate these many engineering disciplines?” I’ll let Drew answer that himself, in the video below.

It’s our job to make it weird

Drew is more optimistic than his worries that LLMs are encouraging a monoculture suggests. If the default output of a model is the average of everything it has seen, “It tells us that there’s still a job for us humans,” he said, “which is that it’s our job to push the model out of distribution. We’re the ones that need to make it weird.”

Weird is a strong word, so don’t take it too seriously. (Though I find it interesting that Harper Reed also used it.) The way I make this point is to say that AI is a medium, like painting or writing or music. Everyone gets the same paints and brushes, the same words, the same notes, but some people draw more out of them than others, or do it better. Our job is to draw something more, something better, out of the ocean of possibilities in the collected knowledge hidden inside an LLM.

But there’s a more prosaic way to push the model out of its normal distribution. Be aware of its training, which is another way of saying “its biases,” and compensate for them. As an example of how to do this, Drew said his team deliberately chose not to use React for a new front end, because the models are trained so heavily on React that using it makes your site look like everyone else’s. He has also started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness.

That led us into a bit of discussion about open source AI, which is the subject of my next AI Codecon. Drew’s ideas fit right in. He wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than, as he put it, becoming appliances.

A typical apartment building generates data on everything from leaking toilet flappers to tenant demographics, yet most of the real estate industry still runs its analysis by hand in spreadsheets. Fundrise co-founder and RealAI CEO Ben Miller argues the missing piece isn’t a better AI model but a proprietary data layer that general-purpose tools can’t replicate. Miller joined Ben Lorica on Generative AI in the Real World to cover how RealAI evolved from a data project into an AI-powered analyst tool, why the AI acts as an orchestrator rather than a financial calculator, which investment trends aren’t yet pricing in AI’s economic impact on real estate markets, and why he believes the trillion-dollar AI CapEx build is temporarily masking what will eventually be a deep trough in white-collar hiring.

About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.

Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts.

Transcript

This transcript was created with the help of AI and has been lightly edited for clarity.

00.20
Ben Lorica

All right. So today we have Ben Miller. He is the co-founder and CEO of Fundrise. And today, we’ll talk about their interesting AI application called RealAI, which you can find at RealAI.com. Their tagline is “Answer any real estate question in seconds.” And we’ll also talk to Ben about broader trends in the AI industry. And with that, Ben, welcome to the podcast.

00.48
Ben Miller

Thanks for having me.

00.50
So, I definitely want to talk to you about RealAI in more detail, but I think to take a step back. . . so broadly speaking, as I understand it RealAI does the job of a real estate analyst, but I think, Ben, for our audience, they don’t actually know what a real estate analyst does. So maybe if you can briefly describe what does a real estate analyst do?

01.16
Yeah. So our product’s designed for a professional real estate investor, an institutional real estate investor will typically buy large scale properties that are worth tens of millions of dollars. And then analysts, like any financial analyst, would [do due] diligence [of the] property, make a lot of financial pro forma, do different sorts of investment memos and then make a recommendation to buy or sell. And they also then manage the investment after you acquire it. And asset management is actually where most of the activity happens in terms of how you optimize for rents and occupancy and different financing. And so the real estate analyst’s job is to be the financial brains of the owner of the property.

02.07
And typically this person, when they go about doing their job, they have access to different data sources, they pull it together, they run some financial models, and then they write up recommendations, right?

02.24
Yes, exactly. That’s what the white collar worker today is like. . . manual CRUD operation. Right? So they gather data points from different reports and different online services they might have. They usually populate a spreadsheet cell by cell that way, and then they also will write up a memo. And that is all done by hand, so it’s sort of like handcrafted. And all of that work, at least the first draft, can be done by AI. And then the question becomes, “why our AI versus Claude or ChatGPT?”

03.05
As I understand it too, Ben, RealAI actually didn’t start out as an AI project. It started out as a data project because back in 2023, as I recall, you were starting to talk to me about wanting to start a data business of some sort for real estate. Right? So I guess, for our listeners, Ben, why does real estate generate so much data? I think I remember you telling me that actually, most people don’t know a typical apartment building throws off so much data, right?

03.40
So Fundrise is a fintech platform that democratizes investing into private assets. And one of our original investment classes was real estate, where we would have a fintech website and an iOS app, an Android app, and we would have millions of users and they invest through our platform, and we have a sort of Robinhood-type platform. So we have payment processing and a system of record for ownership, and all of the API services you need and microservices for that company. And then over time we started eating the value chain, and so we started heading from the investor towards the asset. And eventually we ended up where when we acquire the asset, we run the asset. We vertically integrated, so we’re the real estate company and the tech platform, and the fund manager. And as we acquired the asset, we discovered the underlying infrastructure for that asset had a lot of data in it. And the way that people in real estate do their work is sort of how we started. It’s all very manual.

04.53
Can you describe what this data is?

Yeah. So there’s a bunch of different kinds of data sets you care about. At the property there’s a lot of activity happening. So, a toilet might be running and that’s the water bill. You have all sorts of leasing activity. You have marketing funnels of ‘where does the tenant come from’? You have lots of activity around the building in terms of like, who lives there, who’s moving there, who’s leaving, what’s the rent of the property across the street, what’s being built? So there’s endless amounts of data that’s happening at any point in the real world. And that real-world data matters to the person who owns real assets in that location.

05.46
And so then, it seems like the traditional approach was to ignore all of this data? Is that right?

05.56
It had to be distilled so that a human being can pretty much only consume a certain amount of tokens per second, and that usually ends up in a spreadsheet. So what happens is, people at the property or property managers, or maybe there’s our market company, CBRE, will produce market reports and they turn all that data into a spreadsheet so that you might get a 12-month reporting or rent roll or a market report. And so they take a lot of data and they distill it or transform or aggregate it into very few metrics. And that was necessary. And then once a month you get emailed six reports or 12 reports, and those reports are in a sort of dashboard for how real estate people make decisions. And I appreciate this and this is so obvious that there’s just you know, those reports are both a distillation and there’s a lot of meaning lost in the transformations, and a lot of data just left behind because the system of record for real estate and for a lot of industries, really the accounting system. . . income statements and, you know, get operational data like a toilet flapper that’s leaking, and that’s not coming into the report. And how could it?

07.36
So basically, it sounds like what you’ve done is the classic digital transformation of this industry. If you look back to the data in 2023 to today, what is the evolution of this data set? What was in the original data set, and what have you added since then?

08.02
So we originally were thinking about how all this data could be really useful for the real estate industry, the original business we called Basis, and we were originally using it just for our own company. And, there’s sort of property data. If you’re going to make decisions in real estate, you care about change over time. So time series, and you care about comparative analysis. How is my property versus another property? How’s my neighborhood versus another neighborhood? My city versus another city? And so you need to have not just your own data, but you need to have the context.

08.42
You need to have the data of other properties that are in other places. And the entities in real estate are places, properties, and then people, which was that we had this sort of insight when we were in the data gathering property data and place data, is that there’s a data model in real estate which essentially is the property management system, which is basic accounting systems. That’s Yardi and Tratta, RealPage. . . And those accounting systems have an implied data model that is a property address and the unit, but inside the unit is a person that’s not in their data model. And a really big gap in their thinking. It’s sort of like digital marketing. Kind of like TV. . . you’re going to do it on CNBC, you can do it at 3 p.m., but you don’t know who is watching. And then the internet showed up, and you can actually then target with Facebook or, you know, Google. You can target people who have eggs for breakfast and like Mozart. So all of a sudden the people data becomes really critical. And the real estate industry didn’t have any people data. It wasn’t even part of their thinking. It’s not currently part of their thinking. So we started realizing, “Hey, we get the kind of granular people data you have in the marketing industry.” We bring that into the real estate industry because you care about who lives in a building and who’s leaving, who’s moving in, who moving in the neighborhood? Who’s moving into the city. Are they rich or are they poor? How old are they? What’s their gender? All this stuff. Do they have iPhones? Do they have Androids? So we started gathering up huge data sets of people data and then putting that in the same databases you have with real estate data and building out, and you can actually see that people matter to real estate and you can actually make really good insights, new kinds of data, correlations and stuff like that for the sector.

10.43
By the way, this is the kind of thing that the hedge funds have been doing for years. Right?

Right. Yeah. I was actually talking to a famous data tech guy, and we were on the phone, they’re like, you know, we sell our data to all these hedge funds, for like 50 big hedge funds in the world, which probably be Renaissance, and Jane Street, and Citadel, we’ve only ever had one real estate company by our data. I said, “Who’s that? Who’s that?” They said, “Blackstone.”

11.18
Oh, that makes sense.

So yeah, high-frequency traders are data hogs and they put their data into these massive data models, and they do a lot of data engineering and data transformation. And real estate people do it with Excel spreadsheets and do it by hand. And so there’s this journey that every industry goes through. . . real estate’s way back in the stone ages, where they still do everything by hand with people.

11.45
So the end result for RealAI. . . the back end data. . . most of it is structured, probably sitting in some sort of data lakehouse or warehouse, right?

We have two different kinds of data. We have a transactional database and we have one for rows and one for columns. Because we have a lot of data and we do a lot of compute. . . And because mostly people when they’re using AI are doing computations on the fly, we had to pre-compute a lot of stuff so that when people are asking questions about what’s happening, how much growth there is in the market or what’s happening with like rents or whatever the things are, a lot of that stuff has to be pre-computed every night. And that gets pre-computed in a Snowflake database. And we also do clustering algorithms, we clustered people by city and by block and by lots of different things so when people want insights it’s all pre-computed. And then that sits in Snowflake migrating to a Databricks-type database. And then there’s a separate database which is the transactional store that’s on the fly, that’s fast when people are asking questions and hitting our API.

13.00
So, let’s set aside the AI model and whatnot. For listeners who haven’t tried it, I recommend you go to RealAI.com. I use it once a month to monitor my properties. But basically then what happens is you have this chat interface and then I presume the AI model acts as some sort of reasoning layer and also obviously parses your intention and what you’re interested in, but then the heavy lifting is done by the data back end, correct?

13.38
Yeah. I mean, I think most people have gotten here where you do deterministic analysis, real and where you’re writing Python and doing SQL. . .

13.49
Yeah. Okay. So in your case, if you hallucinate there’s consequences, right?

Yeah. So the way we structured our application is that where there’s facts, there’s essentially a  field that goes and pulls it from the database. So it’s not coming out. It essentially says, “What was the rent for the last 12 months in this property or in this neighborhood?”

14.16
But the AI model is the one that translates that prompt into some sort of SQL query?

Yeah, but I mean, we used to use Claude and we moved to Vercel, but we have basically a software sandbox and it opens it up and does all that sort of. . . This is for real estate, for most people you say, well, you probably know how to do this, but if you’re going to do an analysis about your home or your property, you know, most people can’t write Python, most people aren’t good at doing SQL queries, plus they don’t have a good understanding of the the database schema. And so we built a semantic layer on top of it. And it really democratizes data science. Originally you said we were a data company, and then AI showed up and the AI sits on top of our data, and that made it a lot easier for people because to do the type of insights or analysis that otherwise they would have no idea how to do before.

15.28
But just to clarify, the AI is not doing any kind of financial modeling or calculation, right?

No.

There’s some sort of forecasting or financial model or some sort of financial computation involved, and that’s not being done by the AI?

Right. The AI is like an orchestrator, and has the tools, right? And the different tools it uses and the tools are the web search or the calculator. I think everybody’s gone here, where I think a year ago or something, people were using vector databases and RAG and we moved away from that because you just want to be where it’s deterministic analysis. You want to just use regular software, to put that tool in the hands of the AI.

16.21
So the hard thing here is really maintaining these data sets in many ways, right? So that is the key, because you have to gather or you have to have domain knowledge to understand what data you need and what data really matters. And, now you have a year or so of history of people using the app and you’re understanding, really what’s valuable.

What’s the hard thing? I mean, it just depends on who you are. We didn’t mean to do this, but luckily we have a lot of data. The data is proprietary. The data is really valuable. And so that gives us a reason to exist because what happened is that we built RealAI and we rolled it out, and Claude arguably made it obsolete. And this is a strategy question. I think the hard thing is trying to build the software business.

17.18
How can it be obsolete if it doesn’t have access to your data?

Well, because people have. . . it’s really hard to get them out of their. . . they’re sitting in Claude. Trying to get it. And the reality is that. . .

17.38
There’s no A/B test. They can’t tell that they can get better results if they. . . 

So, our product is really good at, let’s say seven of the 15 things a real estate person needs to do. But there’s other things they need to do that they know our products are not designed to do, and so they need to have a general purpose tool that’s doing all their stuff. And so the day before yesterday, we got approved to be in the Claude marketplace. So we built a connector, we’re building a plugin, and now we’ll go to where the customer is, which is Claude. That’s our current strategy. And you can see it’s getting better. Claude is not as good as ChatGPT at making it developer-friendly. But we can build what’s almost the same as RealAI.com in Claude. And that’s where we’re currently headed. And you and I talk about strategy. I think that’s actually a temporary strategy. I don’t think that’s where the market ends up. But, yes.

18.50
But obviously, as users interact with your app, there’s a lot of learnings there that you can leverage to improve the app itself or even maybe the model powering the app. But in other words, there’s a compounding loop. So the question is will Claude cut you out of that at some point?

I think the hard thing is always the people. I think the technology is actually at least. . . You know, I’m not an AI researcher at the frontier. The hard thing is really building a software product that people become addicted to, and the data engineering and data science and stuff, that’s just a grind, this is work. In the short term, we’re sort of conceding. . . I believe we should concede that the customer wants to be in Claude today, and they don’t want to be in a different application. Claude’s like the hot new hotness. But I think they’re going to get Claude fatigue within a year. I think that the real estate people are lagging the rest of the industry. Most people who’ve been using Claude for the last six, 12 months hit a lot of limitations for enterprise. And I think they’re going to want a harness that’s designed for real estate. But, in the meantime, we’re going to sacrifice some of the things we don’t get by having them in our app. And I actually think they’ll end up going back and forth between Claude and our app, because our app is better for real estate than Claude, but Claude is better for everything else. And the way that the Claude plugin and, and I think it ultimately ends up being a RealAI app in Claude. It wants to go back and forth between our app and their app and wants to be seamless to the user. But after that—and this is really a strategy question—after that, do we end up with an open source real estate fine-tuned, trained model instead of Claude in a year or two? I don’t know, but it seems that would be a good expectation.

21.23
At the end of the day, even if you open up that model, that model still needs the data, right?

At the end of the day, that open source model a year from now. . . when we rolled out Sonnet 5, it didn’t change. It didn’t get better for the customer. Like we just don’t need the Fable. . . it’s actually worse for the customer. So I think that once the open source models are comparable to—I’m just gonna say Sonnet 5 or whatever that generation is—the customer is not getting the premium they’re paying for. They just don’t need it.

22.05
By the way, there’s tons of companies now that focus on helping enterprises get to specialized AI. And basically there’s two branches, right? So there’s the post training branch which comes down to fine tuning and reinforcement, fine tuning, tons of startups there that are coming online. And then, there’s even companies that will help you pre-train models from scratch. I think the trend is toward the inevitable simplification of both things. Which will mean that a lot more of the compute will go towards specialized models like yours. And then less of the compute going to the. . .

22.52
I’m not as confident in that. . . I mean, I’m agnostic. I think we could end up in a world where Claude replaces Microsoft and everything’s inside Claude.

Yeah, it’s possible. But then, I think that the general purpose models increasingly are going to be perceived as too overkill for a lot of things.

23.15
Yeah, I think that’s true in theory, but the consumer is so sticky and it’s hard to get them to change their behavior.

Yeah. If it’s a consumer app. But most enterprise work is just back end. . .

23.31
But I’m dealing with an SMB not like a Fortune 500, generally. And the SMB, in a lot of ways, acts more like a consumer than like an enterprise. And so I hear what you’re saying, though.

23.46
Hey, I have a question for you as far as UX. You have an application where someone is asking a question. It could be an important question for the user. How do you communicate uncertainty or how do you say, “Hey, here’s an answer, but there’s a lot of caveats behind this answer” How do you do that in your case?

24.23
That’s very hard to do. I was just getting that feedback yesterday because even facts in the data world are not 100% facts usually.

24.33
There’s alternative facts.

Yeah. I mean there’s just methodologies that change the information. Is the rent growth a point to point 12 months ago? Is it a median? There’s just a lot of ways you can get to something that’s true, but not the same answer as someone else. But my actual experience with consumers is that they actually don’t care about that. It’s all really about perceived credibility. . .

25.16
So I guess in the UX then, the question is if there’s some sort of uncertainty or disagreement, do you surface that?

25.28
Yeah, we surface data coverage. We have a whole section in the response that’s about a meta analysis about the data and about where we made decisions. . .

25.39
Or do you say, “This property, we think the rent you can charge is this?” But it’s really more likely in this interval. It’s not a specific value. And here’s the reason why there’s an interval instead of a value.

26.00
AI is really good at that kind of caveats and adding context in and again like there’s. . .

26.12
But it’s a UX problem, though. You have to decide to surface it, right?

We definitely surface it. But I find that the user ends up like it’s really a journey. In the first phase, they’re trying to determine if the data is good, and then once they’ve decided the data is good, they don’t really care about the caveats. They’ve sort of internalized to them how to think about the data. And it’s really a problem in the beginning of a user’s use of it. It’s not really a problem once they become a power user.

26.50
So I think that that’s a UX debate that people will have, moving forward. Because I think basically, like I said, in terms of actual studies around decision-making, it does make a material improvement. Since we have you, we’ll close the discussion. And before I ask a question, listeners, this is not we’re not about to give you financial advice; we’re not financial advisors. So there you go. That’s the discussion. First, first thing then, you have access to all this data in real estate, right? So you have all the patterns, but you’re also obviously an investor in AI and AI startups. Right? So, obviously you’re seeing trends in AI that you think aren’t being reflected yet in the real estate market.

27.53
I know I think you’ve mentioned to me in the past, I think so, as an example, to make this concrete, I think you mentioned to me in the past that maybe if you’re buying property with the assumption that there’s a lot of back office workers moving forward and those back office workers are located in, I don’t know where.  . . Phoenix, Arizona. . . Maybe you should have second thoughts. Right? So what are some of the AI trends that you think are not yet being priced into real estate investing?

28.28
What’s good about the trends I’m seeing in the data I’m seeing is I think they’re consonant or consistent with what people would intuitively think. So, AI is causing—in the data—a lot more disparity in outcomes. And so in real estate something used to be a truism. I mean, people will say people always need a place to live. And so you would buy workforce housing, affordable housing, things where if you have housing that’s affordable. . .

29.04
Or always buying is better than renting.

It’s just like cheaper is better than expensive, right? If you can offer housing that’s less expensive, less expensive to file, it’s less expensive to rent. That’s good business. And that actually I think is breaking down, which is maybe unintuitive but also maybe intuitive. Is that where the marginal dollars are ending up in the hands of people who have a lot of money and the people who don’t have a lot of money, which is basically the bottom half of the country, or even maybe the bottom 90% of the country. . .

29.45
95%.

Maybe. You can break a lot of different ways. But the point is that, this K-shaped economy where the normal person is doing worse and the best are doing better is. . . AI is accelerating that trend. And that way that affects real estate is if you’re an investor, you want to actually focus on the high end. If you’re going to be building, you can be investing. . . high end essentially is price-insensitive and the low end. . .

30.15
Because the low end people have no disposable income.

The low end is under strain. And then what is happening in real estate is that strain is showing up as a political manifestation of controlling pricing. And so you can’t make money providing affordably priced housing because the government won’t let you. So the government will let you take essentially excess profits from the rich, but it’s becoming more and more challenging to invest in and build for the normal person, because the government’s intervening in all sorts of small ways that people don’t see around how permits are pulled and lots of impact fees and things like that. I mean, in San Francisco and LA and New York, you see obvious examples of that. But it’s happening everywhere. And in real estate, you usually make an investment over a five year period, maybe even ten years. And so the long term trend is the most important thing. And I think then you essentially say. . . this is terrible by the way, from a social point of view, the societal point of view is very negative. I’m not commending this trend,  I’m just saying that’s what’s happening on the ground.

31.40
What about this notion that certain hubs are optimized for a certain type of work? Is that something you think about?

31.54
Yeah. That’s the thing you were I talked about last time. San Francisco is obvious, and then you look at Charlotte. But certain types of people move to certain places to get certain types of jobs and work from home drove that middle income white collar worker to work remotely, or they moved to more affordable places like Charlotte, Tampa, and Orlando, all these places where it’s affordable to live, and they do middle office work, back office work, and that job is getting decimated by AI. In particular, a lot of cities where young people would move there. They’d move to an apartment building, they’d move to downtown. They’d have a new job in a middle office of a medium to big size company. Those jobs don’t exist today. And so they’re struggling with where to live and they have roommates or living at home. And so the real estate that used to be where young people moved to is struggling as a result.

33.08
So I took you on your first Waymo ride. And how about things like that? I mean, now I can Waymo from anywhere.

33.19
I tried to invest in Waymo and I was not yet successful okay. I reached out to the CEO I had 37 connections to and I got no love. But yeah, the Waymo thing is such a good example of diffusion, a diffusion problem because the technology is completely mature and you can’t get it rolled out in any of the. . . DC’s fighting it and Boston’s fighting it. And for it to really affect real estate, it needs to become as seamless as Uber and Uber rolled out way faster, way more aggressively. They obviously took more risk on the rollout. So, you know, if you said ten years from now, how does that affect the patterns of living? I think you’re going to see. . . I’ve looked at this a number of times, but I think it’s premature to make these investments. You want to be investing in the wealthy satellite towns of. . . if you’re doing San Francisco, which is a little bit challenging to the Athertons. But in Atlanta, it’s the Greenvilles. There’s a lot of great wealthy exurbs that have many miniature downtowns that are cute. And so I think you’re going to see these satellite cities become very, very successful as a result. But those satellite cities are going to be mostly for the wealthy. So again, it’s all about wealth, not about affordability.

34.50
All right. Closing question. You’re a CEO. I know you’ve talked about the impact of AI and automation on hiring. And you talk to other CEOs. So as best you can tell, set aside the headlines and the news accounts. What are you hearing from CEOs about AI’s impact on the workforce and hiring? Hiring in particular.

35.20
It’s more mixed than the headlines. I think I’d say it’s both true that we have needed less people, haven’t hired as many people and also have needed to expand as a result of AI. And so I think I’m hearing a “both” situation. So it’s really a question of what’s the net impact? And I feel like it’s too soon to call. I think on net it’s still net negative on hiring, but it’s too fuzzy for me to have a good call on it.

35.58
Do you have any sense whether or not all the stories we’re reading about. . . if it’s particularly challenging for new college grads and people looking for that entry level job?

36.17
We don’t hire those people, and we rarely ever did.

Because the stereotype is those jobs I can just use AI.

36.28
I think of it as this cascade effect. Where we hired hundreds of people over the years, and what happened was there’s this elevator dynamic or escalator dynamic, where the person who was an early grad becomes middle and then they make space for the next person.

36.52
The talent pipeline, right?

Right. And that dynamic. . . AI’s changed it so much because you just don’t need. . . the people who are at the upper middle are so effective. . . And actually, this is something everybody learned in 2020-2021—more people means less work gets done because you have to manage people and you’re needing meetings and remote work is really brutal for information transfer. And so it’s actually way more effective just to do it yourself with AI than it is actually to have five people. And I think it’s actually more satisfying. And so I believe I’m net negative on jobs in the white collar work. And we’ve invested in some data centers that are like $50 billion data centers. And when you go there, there are thousands of people on site working. I mean, it looks like we’re in China. . .

37.50
For now, while they’re building it.

So I think what’s happening is that the job market doesn’t seem as bad because of this trillion dollar a year AI CapEx build. But I think that’s a five-year build, not a 50-year build. And then I think that in the meantime, that’s coring out the white collar worker. And so I think on the other side of this, when the bubble finally bursts, it’s going to be a deep trough.

38.22
So then that talent pipeline . . . if you don’t have the entry level jobs who become the middle managers. . . and so on and so forth. . . So you have the entry level developer who gets more knowledgeable. . . then you don’t have that. Right? So you end up with what?

38.42
Your talent pipeline, it’s a thing. I’ve one of the things I’ve said to the team and this is harsh, but I’m going to say it, but it’s a brutal thing. There’s a lot of people in an organization who are good but aren’t going to be the next great people. And I said, “If we don’t have a talent pipeline anymore, you need to turn those people over faster.” Essentially, unless you were bad. . . bad people managed out, but if you’re good, we kept you. But if you’re great, you get promoted. But now if you don’t have this talent pipeline, you have to change what you expect of the good. You have to turn over some of them, and look for people who are good with the prospect of building that. . . So you have to change the talent pipeline or talent escalator because you don’t have as many people coming through as you used to. So it’s more of an up-or-out dynamic than it used to be.

39.42
Oh, I forgot one last question on RealAI, which is, I think you still do seat based pricing, right?

39.52
We don’t know how to price it.

So that’s it then? If I’m a real estate analyst, you’re charging me $50 a month, but I’m hammering this thing, costing you $1,000 a month.

40.04
No. The seat buys a certain number of tokens. So it’s more like it’s more like usage tiers, is how it’s priced today, but I don’t know how to price it. We’re still early in this evolution. I mean, we’re real estate. . . so real estate is going to be lagging other sectors in terms of adoption. And so I don’t know what the right way to price it is because it doesn’t want to be only usage, either, I think that’s not right. But I don’t know the answer.

40.41
But even the tiered pricing, if I’m on a token matter, at some point I’m already at your top tier price and I’m still busting through it.

40.52
Yeah, that’s a great situation though, because tokens aren’t that expensive. I know everybody’s obsessed with them, but I think tokens are the smallest part of our cost.

41.04
But if you’re charging me $200 at the very top tier, I’m actually spending so much compute. . .

41.15
I think that would be a great scenario to have in the future. . . for the few power users, that’s happening for us. Great. I hope they feel like they’re getting a good deal.

41.26
And with that, thank you, Ben.

Yeah. Thanks, Ben. Thanks for having me.

Great name, by the way.

Ask an architect almost any question and you’ll get the same answer: It depends. For years this answer has been the punchline of jokes about architects, but in an era when AI can generate a working service faster than you can describe it, “it depends” is one of the most important phrases in software. It marks the exact boundary of what these tools can and cannot do.

The First Law still holds

We’ve said for a long time that the First Law of Software Architecture is: Everything is a trade-off. Nothing about generative AI repeals that law. If anything, it enforces it more brutally than ever.

AI coding tools are extraordinary at answering “how” questions. How do I implement a saga pattern? How do I set up circuit breakers between these services? How do I paginate this API? These questions have answers that exist in the world in documentation, in open source code, in a decade of blog posts, and large language models have read all of it. Asking an LLM a “how” question is like asking a very fast librarian who has memorized the library.

Architecture questions are not “how” questions. They’re “should” questions, and “should” questions have a different shape entirely. The honest answers require knowing things that appear in no training: that your ops team is three people, that the CFO just froze cloud spend, that the last reorg left the payments team demoralized. An AI can enumerate the generic trade-offs of distributed architectures beautifully. What it cannot do is weigh them, because the weights live in your organization, not on the internet.

That’s the Second Law, incidentally: “Why is more important than how.” LLMs are “how” machines. Architects are “why” people.

Cheap code makes decisions expensive

There’s a tempting inference floating around: If AI makes building software easier, surely it makes architecture matter less. Our experience so far suggests the opposite. When code was expensive to produce, the cost of construction acted as a natural brake on bad decisions. A questionable design took months to build, and somewhere in month two, someone usually noticed. Now a team can stand up a fleet of services in a week. The brake is gone. It has never been easier to build the wrong thing quickly, at scale, with tests.

Think of AI as an amplifier. Point it at a sound structure and it accelerates you. Point it at a flawed one and it pours concrete over the flaw before anyone has time to object. The half-life of a bad architectural decision used to be measured in the time it took to implement; now the implementation arrives almost instantly, and you get to live with the decision for years.

This shifts where the leverage sits. When implementation is abundant, judgment is the scarce resource. Someone still has to decide where the service boundaries go, what “good enough” availability means for this system, and which architectural characteristics actually matter.

Judgment doesn’t come from reading

Here’s the uncomfortable part, and it applies to humans as much as machines: You cannot learn trade-off analysis by consuming content about it. We’ve written a fair amount of that content ourselves, so we say this with some authority. Books and talks give you the vocabulary. They don’t give you the judgment.

Judgment comes from making decisions and living with the consequences or at least watching someone experienced make them, asking why, and arguing about the alternatives. Every working architect we know learned the craft this way: apprenticed to messy, real problems, with feedback loops. The pattern catalog was the easy part. Knowing which pattern not to use, and why, and being able to explain that to a skeptical VP that took years of reps.

This is also, not coincidentally, exactly what today’s AI lacks. A model trained on the world’s code has seen millions of decisions but almost none of the consequences. The post mortem that traces an outage back to a boundary drawn wrong in 2019 rarely makes it into the training data, and even when it does, it isn’t connected to the pull request that caused it. Architecture’s feedback loops are measured in years. That’s precisely the kind of learning that can’t be scraped.

Where this leaves engineers

If you’re a developer watching AI absorb more of the implementation work, the strategic question isn’t whether your current tasks will change but where to move on the value chain. Our answer is to move toward the decisions. Toward the trade-offs, the constraints, the “it depends.” That territory isn’t shrinking; it’s growing, because every AI-accelerated team needs someone who can tell the amplifier where to point.

The good news is that this is learnable. Not from a book alone, and certainly not from an LLM, but the way it’s always been learned: by practicing architectural thinking on real problems, with experienced people looking over your shoulder and asking why. We’ve spent the last several years teaching it that way, most recently in a six-week cohort format that works less like a course and more like a short apprenticeship in making and defending architectural decisions. (Details are on the O’Reilly live events page, if you’re curious.)

However you pursue it, pursue it. The machines have gotten very good at “how.” The career-defining skill of the next decade is being the person in the room who can answer “should,” who knows that the real answer starts with “It depends,” and can finish the sentence using their brain alone.

This is the second article in Sharon Zhou’s post-training series. Read part 1 here.

In the first post of this series, you learned how post-training closed the fundamental gap in usability of LLMs by making them behave in a certain way. In this post, you’ll explore specific techniques you can use to change a model’s behavior: namely, reinforcement learning (RL) and supervised fine-tuning (SFT).

Reinforcement learning teaches the model by letting it try things and telling it which attempts were better or worse—the model learns by experimentation and feedback. Supervised fine-tuning teaches the model by showing it examples of good behavior—the model learns by imitations. Both have deep roots in AI and machine learning literature historically, but their application to LLMs, and particularly to making LLMs behave well, is what makes modern post-training work. Nearly everything that happens in post-training is the result of some combination of these two approaches.

Reinforcement learning (RL): Learning from feedback

The overall gist of reinforcement learning goes like this:

1. The model gets a prompt.
2. The model generates a response.
3. The model’s response is graded. The grade is called a reward. A positive reward is good, and a negative reward is bad.
4. The model’s weights are updated to make high-reward responses more likely and low-reward responses less likely.

One of the most important questions is: Where does the reward come from?

Verifiers

The easiest way to get a reward is a function that can output a reward, for example a checker for whether the generated code compiles or whether the generated math problem was solved correctly. This automated check is a verifier. The ideal verifiers are fast, cheap, and perfectly reliable within their domain. Think coding challenges, math problems, or factual questions. For tasks with objectively correct answers, you can just write a function that checks the output.

The limitation is probably obvious: Verifiers only work when you can define “correct” programmatically or hit an API to return the right results. That covers a lot of useful territory, but it doesn’t help you train a model to be helpful, nuanced, or pleasant to talk to.

There are subtler limitations too. Not all verifiers are fast. Your model might propose a novel drug combination, but verifying its validity could take years of lab work. Generated GPU code might need hours or days of performance benchmarking. When verification is expensive, you face a trade-off: Use the slow-but-accurate verifier sparingly, or substitute a faster proxy that’s slightly less reliable but keeps training moving.

Human feedback, RLHF, and reward models

Humans can offer strong reward signals that, in aggregate, align with human preferences that might be more subtle and hard to encode programmatically. However, it’s prohibitively inefficient to have humans in the loop for every training datapoint, especially as the model is continuously updating its weights after it receives rewards as feedback, so the model’s responses would change over time. You can’t really prepare the data ahead of time. So instead, the InstructGPT paper, which informed ChatGPT’s development, implements reinforcement learning from human feedback (RLHF) by training a separate model to mimic human feedback. This model is called a “reward model.”

The input of the reward model is a prompt and model response and its output is a scalar reward (positive or negative) that mimics how a person would rate that response. You can train a reward model in multiple ways. The simplest is to have people grade the model outputs with a score, for example 1–5 stars or a number out of 100%. However, people are rarely consistent at these types of tasks: One person’s 2 is another’s 5, and even the same person drifts over time.

Another simple way is to offer two model responses in comparison and ask, “Which one is better?” This is a much easier, more reliable judgment for people to make. Interannotator agreement is significantly higher for comparisons than for absolute ratings.

Training a model using pairwise comparisons is also simple. You can then use cross-entropy loss over pairs, which pushes the reward of the preferred response higher than the unpreferred one. This works great because it means the reward model can learn from signals like “A is better than B” but can learn to output absolute scores for the reward.

To make the process of collecting pairwise comparisons from people more efficient, the InstructGPT’s implementation of RLHF included showing labelers 4–9 different model outputs from a single prompt and asking them to rank those preferences. This would effectively result in 6–36 pairwise comparisons for a given ranking. Not bad; that’s efficient data labeling! They used ~33K prompts, so that would roughly translate to anywhere from 200K to 1.2M comparisons to train the reward model.

After training, the reward model would be an automated judge during RL training, providing scalar rewards for responses. The language model then optimizes against this reward model’s scores. This means the better the reward model, the more aligned the resulting model would be.

LLM as judge

So you need a reward: Why not use an LLM? LLM-as-judge, sometimes called RLAIF (reinforcement learning from AI feedback), scales much better than human annotation while still being able to evaluate subjective qualities like helpfulness, clarity, and tone. But it inherits whatever biases or blind spots the judge model has, and can be more easily gamed. If the judge tends to prefer verbose answers, the trained model will learn to be verbose.

One effective approach is to break the judgment into multiple LLM calls, each focused on a different aspect of the response, like a rubric. Instead of asking one LLM call “How good is this response?” you might have separate calls evaluating factual accuracy, clarity of explanation, appropriate tone, and completeness. Each dimension gets its own score, and you combine them into a final reward. This is more robust than a single holistic judgment because it’s harder for the model to game all dimensions at once, and it gives you fine-grained control over what you’re optimizing for. You can weigh the dimensions differently depending on what matters most for your use case, and adjust those weights over time as your priorities shift. For example, accuracy is worth 3x as much as tone.

Combining human feedback with LLM-as-judge, Anthropic’s Constitutional AI (CAI) is a method for training reward models from AI-generated comparisons, based on a human-written set of principles. What this means is that you can give an LLM a set of principles, which Anthropic calls a “constitution,” and have it critique and revise its own outputs based on those principles. For example, a principle might say “choose the response that is least likely to be harmful” or “prefer the answer that is most helpful while being honest.” The model generates pairs of responses, uses the constitution to decide which is better, and those AI preferences are used to train the reward model. This means you can encode your values explicitly as written principles in the Constitution rather than implicitly through thousands of human annotations, making it easier to audit, agree on, and update what the model is being trained to do.

RL algorithms

Once you have a reward, it’s time to update the model’s weights. But you can’t just predict the next token, because there isn’t one. All you have is a value for the response the model gave. This is where RL algorithms come in. These algorithms are ways to take the reward and turn it into a meaningful, and ideally stable, training signal for the model to learn. There are several, and the field is moving fast, but a few fundamental ones are worth understanding.

REINFORCE

REINFORCE is the simplest starting point. The idea is to generate a response, score it, and if the reward was high, nudge the model to make that response more likely. If the reward was low, nudge it to make that response less likely. It’s conceptually easy to grok but noisy and difficult in practice because it turns out that the signal from a single response can point the optimization in unhelpful directions, and the variance in the gradients makes training slow and unstable. PPO was designed to fix these exact problems.

PPO (proximal policy optimization)

PPO is what OpenAI used in the original ChatGPT work and was for a while the default algorithm for RLHF. In RL terminology, the model is the “policy,” or the thing that takes actions by outputting tokens, in an environment which is simply the conversation context.

PPO improves on REINFORCE by being more careful about how big each update is. Rather than taking whatever gradient the reward suggests, PPO clips the update so the model can’t change too much in a single step. This makes training significantly more stable. The clipping keeps updates “proximal.”

PPO is also an online algorithm, meaning the model generates fresh responses during training, gets them graded, and updates from that feedback in a continuous loop. So the model keeps learning from its own current behavior rather than from a static dataset. It can explore and improve in ways that offline methods (that only collect data once beforehand) can’t.

Notably, PPO uses a “critic,” or a separate model that predicts the expected total reward from any point during generation, and is trained with the policy. This helps reduce noise in training, because it gives you a baseline: Instead of just knowing “this response got a reward of 7” and having no idea if that’s good or bad, the critic might predict “you’d normally get a 5 here,” so the actual training signal (called an “advantage”) becomes “+2, better than expected.” This dramatically reduces noise compared to REINFORCE.

The downside is complexity. Now you’re training two models (the main model and the critic), and the whole pipeline involves generating responses, grading them with a reward model, estimating how good the grades are relative to the critic, and updating both models. It works, but it’s a lot of moving parts. This makes it harder to tune or debug when something goes wrong, and harder to set up the infrastructure.

DPO (direct preference optimization)

DPO takes a different approach that avoids RL entirely but optimizes the same underlying objective as the standard RLHF formulation. Researchers found that there’s a mathematical relationship between the optimal reward model and the optimal main model (policy), and you can collapse the two-step process into one. This means that you can take the same pairwise comparison data (“model response A is better than model response B”) and use it to update the main model directly, without a reward model. Yes, this means good old supervised learning on that pairwise data.

In theory, under ideal conditions, DPO and PPO-based RLHF converge to the same global optimum. Those ideal conditions include a perfect reward model, infinite preference data covering the full output distribution, and the reference policy matching the data-generating distribution. However, these rarely hold in practice, and several empirical studies have shown meaningful performance gaps between DPO and online RL methods on harder tasks, partly because DPO can’t explore beyond its fixed dataset. That said, it’s still a very promising technique.

The simplicity of DPO is attractive: supervised fine-tuning on pairwise data with no reward model to train and no RL loop to stabilize. As a result, DPO has become very popular, especially among smaller teams, because it’s much easier to implement and debug. However, the trade-off is that DPO is less flexible, because it works directly from a fixed dataset of preferences. This means it can’t explore and discover novel behaviors the way online RL methods can. It only learns from the comparisons you already have.

Newer online variants of DPO have addressed this by generating fresh responses during training, but at that point you’re reintroducing some of the infrastructure complexity that made DPO appealing to avoid in the first place.

GRPO (group relative policy optimization)

Introduced by DeepSeek, GRPO takes another stab at simplifying PPO. Instead of needing a separate critic model, GRPO generates a group of responses to the same prompt and uses the relative rewards within that group to figure out which responses were better or worse—basically normalizing within that group. If you generate eight responses and three of them score well, those three get reinforced and the others get pushed down, and the baseline (which the critic was in charge of previously) is just the group average. This eliminates the critic entirely while still getting a useful training signal. It’s simpler than PPO but still online (the model generates fresh responses during training), so it can explore in ways DPO can’t. GRPO got a lot of attention because of its role in training DeepSeek’s reasoning models.

There are many more algorithms and variants, and new ones appear regularly. The field hasn’t converged on a method (and likely won’t for some time), and different algorithms suit different situations. DPO is great when you have good preference data and want simplicity. PPO remains strong when you need online exploration and have the engineering resources to manage the complexity. GRPO offers an appealing middle ground. In practice, teams often try multiple approaches and pick what works best for their specific use case and reward signal.

RL post-training is also less stable than supervised learning, which we’ll cover next. The loss curves are noisier, the hyperparameters are more sensitive, and the training can diverge if not carefully managed. Practitioners typically constrain the RL updates with a penalty that prevents the model from drifting too far from its starting point. The most common approach is a KL divergence penalty that keeps the fine-tuned model’s output distribution close to the base (or SFT) model’s distribution. This acts as a regularizer: It lets the model improve its behavior while preventing it from forgetting what it learned in pretraining or collapsing into degenerate patterns.

Supervised fine-tuning (SFT): Teaching by demonstration

Supervised fine-tuning is more straightforward. You show the model examples of ideal responses, and train it to reproduce them. In practice, this means collecting a dataset of {prompt, ideal response} pairs and continuing to train the model’s weights using the same next-token prediction objective from pretraining, but now on this curated dataset instead of the broad pretraining dataset. The one difference is that the loss is computed only on the response tokens, not the prompt tokens, so the model learns to generate good responses given prompts, not to generate prompts.

The simplicity is the point. There’s no reward model to train, no critic to stabilize, and no policy gradient variance to worry about. However, it’s also limited by the data you can collect. That can get expensive and difficult to scale.

The quality of your SFT model is directly determined by the quality of your demonstrations. The model is learning to copy what you show it, so every quality issue in the data becomes a quality issue in the final model.

Human demonstrations

The most direct approach is to hire skilled people to write high-quality responses to a diverse set of prompts. This is the gold standard. You can easily control your dataset here, and you can get exactly what you want, written to your specifications. The original InstructGPT paper from OpenAI contracted 40 labelers, writing demonstrations and ranking outputs.

The disadvantage is, probably obviously, cost and scale. Good demonstrations are expensive, especially tasks requiring domain expertise like having doctors write an ideal prescription for a patient or a rocket scientist telling you how to put satellites on Mars. And even expert annotators are inconsistent. They have bad days, they get tired, and they interpret instructions differently from each other. At scale, this inconsistency can accumulate, though labeling companies manage and sell processes to make crowdwork more effective at scale.

Synthetic data

Synthetic data scales far better than human annotation. You can generate millions of demonstrations cheaply and quickly. The Stanford Alpaca project famously fine-tuned Llama on only 52,000 demonstrations generated by text-davinci-003 (part of the GPT-3.5 model family, though not ChatGPT) and was able to get qualitatively similar behavior to text-davinci-003 with a much smaller budget (though it was on a narrow evaluation of only ~250 examples—still an exciting result for small open models for research).

Many open source models have used variants of this approach. However, there’s also a practical consideration around terms of service. Some model providers restrict using their outputs to train competing models, and this has become an increasingly heated area of debate as models compete at the frontier. Know the rules before you build your pipeline.

Curated data with synthetic transformations

Sometimes the best demonstrations already exist. Customer support logs, internal documentation, expert Q&A forums, edited writing samples. If you have access to high-quality human-generated content that matches the behavior you want or is close to it, you can use LLMs to transform that data into prompt-response pairs. This has the advantage of being grounded in real use cases rather than fully synthetic scenarios.

The work is in the curation, and sometimes it might be easier to generate from scratch based on a few few-shot examples. Raw data is messy: Support logs contain errors, forums contain misinformation, and real conversations meander. You need to filter, clean, and reformat aggressively, but you can build an LLM pipeline to do those steps. If you have a good source and invest in the LLM curation pipeline, this can be extremely effective, especially for domain-specific applications.

Rejection sampling

Sometimes the best training signal is already inside the model and you just need to find it. Rejection sampling works by generating many possible responses to a prompt, scoring them with some quality metric, and keeping only the top performers. The quality metric can be a reward model, a rule-based check, or even a stronger model acting as a judge.

Suppose you prompt your model “Write a Python function to merge two sorted lists” 64 times at temperature 0.8. You run each output through a test suite as your quality metric. Maybe 40 pass all tests. You take the 10 cleanest, most readable passing solutions and add them to your SFT dataset. You’ve just used the model’s own competence to build training data better than what most human annotators would produce for a coding task.

It sounds like RL, but it’s just using the same pieces to filter the demonstrations that the model should see in SFT. The same graders like reward models, verifiers, or LLM-as-judges are used to curate SFT data.

Rejection sampling is also surprisingly effective and therefore popular; for example it was described early on in Meta’s Llama 2 post-training pipeline. The model already can produce great responses, but as you’ve probably noticed, it just doesn’t do so reliably. By filtering for its best outputs and training, you raise its average toward its ceiling. Instead of acting as the average developer, it’s nudged to act as an expert developer. Rejection sampling scales well because generation is cheap relative to human annotation. The main limitation is that you’re still bounded by what the model can produce at sample time. If it can’t generate a correct proof in any of 100 attempts, no amount of filtering will help.

But SFT has limitations. It only teaches the model what to do. You’re presenting ideal behavior but never showing it what “bad” looks like. As a result, the model could still produce problematic outputs on prompts that weren’t well-represented during training.

The SFT model is also prone to “mode averaging” when the training data sends mixed signals. For example, if half your golden retriever demonstrations sound like an encyclopedia (“The Golden Retriever (Canis lupus familiaris) is a large-sized breed of gun dog…”) and the other half sound really casual (“Golden retrievers? They’re basically furry happiness machines”), the model won’t learn to pick the right tone for each context. It’ll blend them into an awkward middle: “The Golden Retriever is basically a large-sized happiness machine of the gun dog variety.” Neither formal nor casual, which comes off as weird and not the right response style.

Why frontier models use both

RL seems all-powerful. Why not use it alone? This was a research question pursued by DeepSeek’s team when training DeepSeek R1-Zero. Up until then, the base models were so bad that doing RL was pointless on them and you needed to do SFT. This model demonstrated that RL applied directly to their relatively strong pretrained model can produce powerful reasoning ability without any SFT.

However, the model still had serious usability problems. For example, it would mix languages (e.g., English with Mandarin), so it was difficult to use for most people. It could reason, but it wasn’t practical to use.

RL’s main ceiling after the model has been trained is usability. During training, its ceiling is stability. Research on new methods are continually trying to find ways to do RL post-training more stably.

SFT, on the other hand, has the opposite problem. It’s been used alone for many years and has reached maturity to some degree. InstructGPT made the model capable of instruction-following, and became the foundational approach for ChatGPT to handle multiturn dialogue and thus conversation. However, while SFT gets good, reliable results, it’s typically not enough to push performance at the frontier to reach superhuman performance on important tasks.

Here’s what it means for you: If you’re doing post-training on your own, and you want your model to behave a certain way and you don’t care about novel frontier performance, SFT will get the job done.

In contrast, RL can teach a model to reason through novel problems it hasn’t seen during training, because the reward signal evaluates the outcome rather than the exact token-by-token process like in SFT. RL can surface rare but important behaviors that might not appear frequently enough in any SFT dataset.

On scaling data, RL can improve a model’s performance on a task as long as the reward signal is accurate, without needing to collect additional human-written examples. However, it’s important to note that on some tasks, it’s easier to scale SFT examples, and on others, it’s easier to scale via RL. For example, RL scales more easily on math problems. You can generate an unlimited number of math problems programmatically, and a verifier can check whether the answer is correct with certainty. You wouldn’t need to hire a mathematician to write out ideal solutions. The model attempts problems, gets told right or wrong, and improves.

Safety is another area where RL stands out. It’s relatively easy to write a few hundred examples of a model declining harmful requests in your SFT dataset. But the space of ways a user might try to get harmful or inappropriate content is broad, creative, and ever-changing. RL allows the model to be trained against adversarial prompts, where it practices handling tricky edge cases and gets rewarded for handling them well. This is much harder to achieve with static demonstration data alone.

Meanwhile, SFT scales more easily on writing in a specific brand voice. If you want the model to respond with your company’s brand voice. It might be hard to write a reward function that captures “sounds like our brand.” But the company could have tens of thousands of real support transcripts that already demonstrate the voice. You can curate these, transforming them into prompt-response pairs. The data already exists at a decent scale, while the reward signal would be hard to get right.

When a frontier lab wants to add support for a new feature, for example calling MCPs or calling subagents, the first step is almost always to create a small amount of SFT data demonstrating that capability. The next step is creating a reward function and RL environment that can match it.

Far more data and thus compute are dedicated to RL than SFT, but SFT offers good warm starts for the model and those examples are critical to getting the model into a stable place for subsequent RL.

The combination of both is ultimately what makes modern frontier models as capable as they are. Neither alone is sufficient.

A standard post-training pipeline uses SFT and RL as complementary stages that build on each other. It might look like this:

1. Pretraining produces a foundation model with broad knowledge.
2. SFT takes that foundation model and teaches it basic behaviors: how to have a conversation, follow instructions, use a helpful tone on a range of different tasks, etc.
3. RL takes the SFT model checkpoint and refines it further. Using reward signals from human preferences, programmatic verifiers, or AI judges, RL gets the model to be more consistently helpful, less likely to produce harmful content, and better at complex tasks like reasoning.

Some teams also iterate between several stages of SFT and RL: SFT, then RL, then more SFT on new data, then more RL. The first couple stages could be on reasoning for verifiable tasks like math and code where the data and reward signals (verifiers) are constructed differently, whereas the second could be on messier general reasoning over all tasks, which would involve training reward models that encode human feedback as preferences.

This iterative refinement can help with checkpointing quality at different stages and handing things off to different teams, though it adds complexity to the pipeline. Not surprisingly, the quality of each previous stage directly affects how well subsequent stages can go.

Every agent I use is building a model of me. Claude has learned how I like my prose. ChatGPT remembers what I’m working on. I don’t mind this—every person I have a relationship with carries a model of me in their head, and every company I do business with keeps a profile. Other people’s understandings of me have never been mine to control, after all.

But an agent occupies a different role. It learns my writing style, my preferences, and the shape of my work and life, all to help me with what I do. Yet if I switch products, I have to start over. If I use three agents, each rebuilds from scratch what the others already know. Everything an agent learns lives with its vendor.

It doesn’t need to be this way. What if every person had a canonical, user-controlled repository of context that any agent could request permission to use? What if my context lived not only with the company providing the agent but also in a home under my control? And what if an observation captured by one agent could be proposed to that repository and, once accepted, made available to every other agent I choose?

By user-controlled, I don’t necessarily mean self-hosted. I mean that I can inspect what the repository contains, decide who can read or change it, understand where each piece came from, and export the whole thing in a form I can take elsewhere. Its storage, identity, and synchronization may all be provided by someone else. Control does not require me to operate the infrastructure; it requires that no agent or platform be the only way in—or the only way out.

The repository wouldn’t be a portable copy of any agent’s internal model of me. It would be a legible record of things I have written, facts and preferences I have chosen to keep, as well as observations that agents have proposed and I have accepted, each with its provenance, scope, and history. Agents could consult or add to that record according to their permissions; their private inferences would remain their own.

The dream is not a new one. Tim Berners-Lee’s Solid project has argued for years that personal data should live in pods that people control, and Doc Searls’s VRM project has been making the case for user-driven relationships with vendors for decades. What those efforts never had was mainstream demand. Agents are supplying it: An assistant needs rich personal context to be useful, and each vendor is building that context inside its own walls. Ordinary people now have a reason to want a personal data store, even if nobody will actually call it that.

The hard problem in all this isn’t syncing or storing data. It’s negotiation. Who can read a given part of my context? Who can add to it, change it, or remove it? Which parts of my life can a particular instance of an agent see? How do I make those decisions in a policy-driven way? And how do I manage them from wherever I happen to be?

But before I could work on negotiation, I had to figure out where my context should live. That’s the question I’ve spent the past year on, and I’ve tried three answers.

First answer: The laptop

Immediately after getting access to Claude Code at the start of 2025, I pointed it at an Obsidian vault—a folder full of Markdown files that can be used as a personal wiki. This wasn’t a particularly novel idea. Many of the geeks I know did the same, and the pattern has since spread in many forms. The best-known recent example is probably Karpathy’s LLM Wiki, elegant not just as a design but as a document: You give the description to your agent, and the agent builds a version tailored to you.

A year of using a pile of Markdown text files with agents has taught me five things about what a personal context system has to get right.

Local-first foundations matter. Text files are remarkably legible, portable, and easy to store somewhere I control. Git moves them between computers and remembers every change. But the result is centered on a laptop or desktop and assumes a user comfortable with plain text and version control. Most annoyingly, my context in this form isn’t readily available on my phone, which is the computer that goes with me everywhere. Nor can agents running anywhere other than my laptop reach it.

Provenance matters, and so do proposals. Karpathy’s Wiki is almost entirely written (and rewritten) by the LLM. In my own system, I write most things myself and lean on agents to help me edit as well as contribute their observations. I want to know which thoughts are mine, which were captured by an agent, and which we arrived at together. That means an agent’s observation should not automatically enter the repository on the same footing as something I wrote. The default should be a proposal that I—or a policy I control—can accept, revise, or reject. Direct write access is something a trusted agent should earn.

Chronology matters. Wiki links aren’t the only structure in a life. Most of what I record—and much of what agents observe—is anchored in time. Thoughts build on thoughts. Observations about people accumulate meeting by meeting. Some facts fade as they age. Time should be a primary axis of the system, not something reconstructed afterward from file histories and metadata.

Scopes matter. My context spans work, personal, family, and public life. I want one unified view; no agent should have one. An agent connected through my work account should see work and public context—and nothing about my family. Fully separate silos would protect those boundaries, but they would also shred the single history I want to keep building for decades.

Identity and type matter. LLMs can extract all sorts of meaning from plain text, but some things, such as people, companies, and places, deserve to be typed records rather than mentions in prose. A persistent identity gives observations, relationships, and history an anchor to accumulate around; it can help resolve nicknames and follow changes in roles and titles. An agent can then act on who someone is without reconstructing them from prose every time.

As I learned these lessons, I added tooling and conventions to my personal context repository. It’s surprising how far you can push a directory of Markdown files. Each new affordance, however, turned my simple folder into a more specialized system, and the result only works for geeks like me. It doesn’t work for my family, however. They use agents every day but they are never going to deal with a pile of Markdown files in a Git repo. They want their personal context to be with them, easy to use, and transparent to the rest of their life.

More to the point, the five lessons above describe what a context system must do. They don’t answer where it should live if /home/$USER isn’t the center of your computing life.

Second answer: The web

My next move was to sort out how to make my context available when I wasn’t at my laptop—to me and, just as importantly, to my agents. The obvious solution to me as someone who has been building on the web since the mid-1990s: put it on a server behind a URL. I deployed a Cloudflare Worker, uploaded my context, and stood up both a REST API and an MCP server. The improvement was immediate. My context was reachable from my phone and grantable to any agent I chose.

New problems arrived just as fast. I had created a new trust boundary with its own access control mechanism and appointed myself its security team. I was now the operator of a small SaaS with exactly one customer, responsible for its uptime and its backups. And I had traded away local-first, offline editing to get there.

These are solvable problems. Our industry has spent two decades learning to host services, and CRDTs could probably win back offline editing. But as agents gain access to more sensitive data and more power to act on our behalf, the price of getting a boundary wrong keeps rising.

And even with those solved, a deeper problem remains: A stand-alone service sits outside my computing home, apart from the contacts, calendars, messages, files, and system-level agents already inside it. Apple’s Siri AI announcements made that separation vivid, and Gemini’s integration into Google’s ecosystem points the same way. An agent embedded in an ecosystem works with everything inside its trust boundary; my worker would have to rebuild every one of those connections from outside.

Third answer: My pocket

As I tinkered, I kept returning to a simple mental image: my context living on the device in my pocket that goes with me everywhere. Not literally every byte, of course, but within the personal computing ecosystem that phone is the center of—the one that already establishes my identity, synchronizes my devices, stores much of my personal data, and mediates what applications can access. In this sense, a home is not a physical location. It is a trust boundary.

Living inside the boundary doesn’t mean that every application inside gets my context, or that agents outside are shut out. The boundary supplies identity, secure storage, synchronization, and native integration; the context layer still decides what each connection may read, propose, change, or delete. Native agents participate through the platform’s own capabilities, while agents from other companies connect through explicit, revocable permissions.

For me, that home in my pocket is Apple’s ecosystem, with iCloud at its center. For you, it may be Google or Microsoft. The point is not that any one ecosystem is the right home for everyone. It is that most people already have a primary digital home, and that home is the most practical default for their personal context. We shouldn’t need to create a separate service with its own identity. Instead, agents should have a common, permissioned interface to the context where it already lives.

There’s an obvious risk here. A home rooted in a vendor’s ecosystem invites lock-in. The mitigation is straightforward: The whole repository—entities, provenance, and history included—must be exportable at any time as a directory of plain text files that can be taken anywhere. A pile of files in a folder may not be the right solution for live context, but it makes a perfect escape hatch.

I’ve started testing this thesis in a SwiftUI app, and my early prototypes suggest that the architecture is workable: iCloud handles synchronization, and I can expose selected context to authenticated agents through MCP. It’s also shown that working in the Apple developer ecosystem is more annoying than deploying a web app.

The remaining work is clear, however. Choosing a home for context is one problem; negotiation—permissions that remain understandable as a repository grows, proposals from multiple agents reviewed and reconciled—is another. That is the hard problem I mentioned before, and it deserves its own deep dive.

One pattern, many homes

Others are converging on this pattern from different directions. The note-taking app Bear, which stores its notes locally on Apple devices and synchronizes them through iCloud, now exposes them to local agents through MCP; its latest release lets users include or exclude notes by tag when granting access, offering a practical approach to scopes. Craft’s MCP connections likewise let users choose which documents or spaces an agent can access and whether it can read or write them. Reflect has embarked on an open source client using Markdown files that will have an iOS companion app.

These are just a few examples, and there are a lot more out there. What I haven’t yet seen emerge however is the attribution and provenance of items that an agent contributes or edits that I think a durable personal context requires.

Zooming out, here are the principles I think are needed in any system like this, wherever it makes its home:

1. Context shouldn’t be captive to any particular agent. A person should be able to change or combine agents without ever starting over.
2. Context should be reachable by local and remote agents alike, with appropriate authentication and authorization.
3. Permissions should be scoped and revocable. Access should be granted to a specific connection with an agent, limited to a defined subset of the context, and should distinguish between reading, proposing, changing, and deleting.
4. Provenance should persist. Every item records where it came from, who or what created it, and how it has changed. The distinction between human-written, agent-captured, and collaborative work should outlive the conversation that produced it. 
5. Time should be part of the context. The system should preserve when something was observed, when it was true, and how it changed, rather than continually overwriting the past with the present. Database folks will recognize this as bitemporality: the distinction between transaction time and valid time.
6. Entities should be first-class. People, companies, places, and other recurring concepts should remain recognizable as names, roles, and relationships change. 
7. Interoperability should not require uniformity. Different people and companies should be able to build different context systems for different ecosystems and trust boundaries. What they need is agreement on identity, permissions, provenance, and exchange, not one universal application. For the exchange, plain text files with structured metadata are a strong candidate.

Using these principles, personal context can be something a person owns: You can inspect it, grant and revoke access to it, trace where each piece came from, and take the whole of it elsewhere. Every agent may still develop its own understanding of you, but you’ll be able to bring a durable context of your own to the relationship, one that participates in the agentic ecosystem without being subordinate to any vendor in it.

Karpathy’s LLM Wiki is a description, not a tool; it’s meant to be implemented by anyone, in whatever form fits. This essay is offered in the same spirit. The important part isn’t whether the app I’m tinkering with ever ships beyond my own devices. I’m more interested in the dialogue it will take for everyone to have personal context that works for them, in their ecosystem and with the agents they want to use. If we get the pattern right, changing agents won’t mean changing homes. The context they help us build will remain ours.

On the most recent episode of Zero to Agent in 30 Minutes, Vicki Reyzelman, senior solutions engineer at Akamai Technologies, drew on more than 25 years in technology and a background in software engineering to build an agent for her Chat About AI YouTube channel.

Vicki wanted the agent to monitor channel performance, identify bottlenecks, and recommend ways to grow subscribers and improve click-through rates. She defined the requirements, designed the workflow, built the agent, tested it, and revised the instructions following the same iterative process used in software development.

How to build a YouTube analytics agent, step by step

1. Define the goal.
   Start with the problem the agent should solve. Vicki wants to understand which videos perform well, where channel performance falls short, and what actions could help more viewers discover and subscribe to the channel.
2. Review the available data.
   Examine each source before designing the workflow. YouTube Studio exports include metrics such as impressions and click-through rates. The YouTube Data API provides information about channels, subscriptions, comments, and activity, but it doesn’t include every metric available through YouTube Studio.
3. Choose how the agent will receive the data.
   Vicki downloads her analytics as CSV files instead of connecting the agent directly to her account. This gives her more control while she tests the workflow. She notes that she could add an authenticated API connection later, after testing the agent’s permissions and guardrails.
4. Write the skills file.
   The skills file defines the agent’s mission, capabilities, data sources, rules, and expected output. Vicki instructs the agent to audit channel metrics, identify bottlenecks, compare performance over time, and recommend actions tied to subscriber growth and click-through rates.
5. Add guardrails and acceptance criteria.
   Tell the agent how to handle the data and what it must avoid. Vicki directs it to use only the supplied numbers, ignore bot activity, report silent failures, and stay within approved systems and data sources. These instructions help prevent unsupported conclusions and actions outside the agent’s assigned role.
6. Build and test the agent.
   Vicki uses the console’s quick-start process to create the agent, add its instructions and sample analytics, select an environment, and start a session. Her first test encounters input problems, so she stops the session, adjusts the data, and runs it again.
7. Monitor cost and performance.
   Observability tools show token consumption, errors, active time, and deployments. Vicki explains that more capable models may need less context but usually cost more to run. Teams can balance model capability against token cost by improving their instructions, selecting less expensive models for simpler work, or dividing a workflow among specialized agents.

Vicki recommends revisiting the skills file as new requirements emerge. Clearer instructions, stronger guardrails, and regular testing help the agent continue to produce useful results. Her process reinforces a practical software engineering lesson: Faster implementation doesn’t reduce the need for clear requirements, reliable data, security controls, and testing.

Coming next week

Join us for the next episode on August 12, when AI and machine learning leader Ofer Mendelevitch will explain how to design multi-agent systems that can keep work moving without constant human involvement. He’ll use Jam to build a team of agents that can plan, divide work, execute tasks, review progress, escalate problems, and repeat the process. He’ll show how an architect agent can take a project brief, recruit developer agents, assign work, and guide the project toward a clear definition of done.

The following article originally appeared on Tim O’Brien’s Medium page and is being republished here with the author’s permission.

At some point, the software “Security” industry stopped talking about stopping threats and started talking about detecting them: detection windows, response times, mean time to remediate. It’s not offense or prevention; it’s damage control. There’s a movie scene that captures what that sounds like, and you’re going to name the film before I finish describing it.

An underground base on a frozen planet. The enemy knows exactly where it is. Massive mechanical walkers—walking tanks the size of buildings—are advancing across the ice. The defenses can’t stop them. The people inside aren’t trying to fight back. They’re frantically trying to get a broken ship working so they can just escape—not win, not hold the line, just get out before something catastrophic and unstoppable reaches the door.

The whole opening is just people preparing. Rushing. Running checks on equipment that isn’t ready, coordinating defenses that won’t hold, buying time against something too large and too fast to stop. Nobody’s planning a counterattack.

The entire operation is: slow it down long enough to get out.

The Empire Strikes Back

That’s what the conversation around InfoSec sounds like right now. The base is under attack. The walkers are AI-generated vulnerabilities, automated exploit chains, and speed that no human team can match. The framing has shifted from defending the perimeter to just getting the ship started. Not winning, just getting out.

Go back and watch that opening sequence carefully. There are hundreds of faceless Rebel troopers in that scene—no names, no lines worth remembering—scrambling to hold the perimeter, buy time, absorb the blow. Some of them continue to fight. But maybe some already understand that the base is lost.

Han is out on the ice looking for Luke. Leia is already on the transport, making sure the mission survives. The main characters aren’t defending the base. They’ve concluded the only way to answer the threat is to move. Most of the conversation around AI right now sounds like those faceless troopers continuing to defend: fortify what’s there, slow the walkers down, hold long enough for something to change. A few people are thinking like Han. They’re not buying another vulnerability scanner from a vendor. They’re asking whether there’s a different way off the planet entirely.

Here’s the disconnect: most people focused on “Security” have spent decades being handed a finished base and then being asked to defend it. They weren’t involved in the architecture or approach that application developers have been using.

• This new application uses Node.js—go defend it.
• We’re using a new relational database because the architect wanted to—go defend it.
• Our developers decided to start using an LLM. Can you secure it? Thanks.

In many cases, people responsible for security are not defining architectures as much as they are catching up. And as “developers” start to generate more code in a day than was possible in a month or a year, it’s becoming increasingly unrealistic to think of security as an afterthought.

When security is just a support team for software engineers, that’s building a base that might be indefensible.

The shift that actually matters isn’t a better scanner or a faster response team. It’s security people in the room when people are writing the prompts, when agents are assembling the dependency list, and when the basic system prompts are defining the authentication system—before any of those systems is in production. Not reviewing the finished base.

Security needs to be involved before anyone even starts to prompt a system’s creation.

“Machine speed” has become a conference catchphrase, which usually means it needs translation. Here’s what it actually looks like, pointed at you:

• A network of agents found a zero-day in FFmpeg and didn’t announce it anywhere, just filed it internally.
• A second agent scraped your team’s LinkedIn and X and noted who’s in Cancun next week.
• A third logged your nightly load balancer latency blip as a cover for an attack.
• A fourth studied your last three incident reports and estimated a 30-minute detection window.

The whole operation, including reconnaissance, timing, and coordination, ran in seconds. What previously required a dedicated red team and weeks of planning is now background processing that runs continuously, waiting for the right moment.

Five years ago, you would have tasked a room of scary-looking security people with profiling a target, capturing latency data, and maybe holding several meetings to discuss what they found. Today, the coordination I outlined in the previous paragraph might take a few minutes on a network of interconnected Nanobot, Picobot, Hermes, or OpenClaw agents that gather data and then update a shared memory system, and the decision on when and how to attack would be made by another agent that was granted permission to coordinate the attack across a distributed network of agents.

Quick note: If you have anything to do with running a website, stop posting about your vacation plans.

Here’s what the conversation keeps missing: AI isn’t the real problem, and this problem isn’t necessarily new. The problem is that we’ve been building bases that were always going to need to be evacuated. The problem is that security is rarely involved in selecting a tech stack, and because that tech stack selection is frequently automated with AI, there’s no predicting the mess that’s being thrown over the wall.

The response to AI-accelerated attacks is almost entirely defensive. Tighten npm’s signing requirements. Fund the Maven repository. Sign up to support Akrites with the Linux Foundation. Add another scanner to the pipeline. These aren’t wrong. They’re just not enough.

These are important projects, and security groups should sign up to support them, but the real transformation that needs to happen is that more people in security need to get involved in software creation. What this looks like is having an opinion on React, Vite, Tomcat, Node.js, databases. It means jumping in and affecting some of the basic decisions that these agents are going to use before they deliver vulnerable software.

Most of the industry is still shopping for scanners. Most people in security are still “reviewing” software in a process that assumes it takes weeks or months to write.

In 1995, the question in the media was whether Netscape or Microsoft would control the web. The answer, it turned out, was neither.

Both Netscape and Microsoft aimed to dominate the web server and browser market, reasoning that whoever controlled both ends of the connection would have an internet “platform” to rival the deathgrip that Microsoft had enjoyed on the personal computer. The two companies raced to build every feature they could think of directly into the product, on the theory that whoever built the most integrated and full featured web server would win.

The open source Apache web server took the opposite bet. It stayed a web server with a clean extension layer, so anyone could bolt something new onto it without asking permission or waiting for the next release cycle. Within a few years, Apache was far and away the most popular web server, and Netscape’s server and Microsoft’s Internet Information Server (IIS) were history. People started talking about the LAMP stack: Linux, Apache, MySQL, and (Perl | Python | PHP) as a legitimate platform. Modularity, not features, was the moat. The fact that major elements of that stack survive while others have been swapped out or extended is a testament to the power of composability and distributed innovation.

I called that pattern the architecture of participation when I wrote about it in 2004. I was trying to explain an inconvenient fact that the licensing debates of that era ignored. I had started working with Unix in the System III days, and saw how it had succeeded as a collaborative project even though AT&T offered Unix under a proprietary license. A few years later, I observed that nominally open source projects like OpenOffice with monolithic architectures never built much of a community. I realized that open source wasn’t just about licenses, but about architecture. A small kernel with standard interfaces that lets people extend your work without asking for permission is an important part of the secret sauce.

Swap out Netscape and Microsoft for OpenAI and Anthropic in this story, and perhaps you can see the echoes.

A model’s personality, its defaults, and its history used to live where you could, with a little effort, see them and edit them. Increasingly, they don’t. As Drew Breunig pointed out to me the other day, each new version of the frontier models moves a little more of the product’s behavior out of an editable layer and into the weights themselves, where nobody outside the lab can see it, let alone change it. The model stops being a component you build with and can adjust to your liking and starts being an appliance you rent. Post-training is important but Drew points out that it is also “trading diversity for reliability.” That’s a good trade for many people, but it is the same kind of trade that gives us highly processed foods when we know that “real food” is better.

Open weights are just table stakes

The public debate about open source AI seems devoted far too much to model weights, their national security implications, and whether a lab releases weights and under what license.  But that covers only a fraction of what actually makes open source matter. Apache was never competing with Netscape and Microsoft (and Linux was never competing with Windows) over whose source was more available. They were competing over something more important. I remember talking with Bob Young, the founder of Red Hat, about his business model, and he said “What we really sell to our customers is control.” Open source meant that the platform your application depended on was no longer a sealed box you licensed from one company but a layer you could extend and build a business on top of without asking anyone’s permission. It sparked an explosion of innovation. It enabled companies like Google and Amazon to grow up free from Microsoft’s dominant paradigm.

Every wave of computing, from mainframes to PCs to the internet, has run through the same cycle: distributed innovation at the start, with the eventual winner gradually closing down its offerings to build a moat. What keeps a market open isn’t the license on any single component. It’s how easy it is to swap out one component for another when a better one appears.

The protocols connecting the pieces are an important part of that picture. Unix utilities expected stdin and stdout, and the shell acted as a kind of harness to connect them, so it was easy to build a new tool that worked seamlessly with existing ones. A testament to the power of that approach is just how much the shell and Unix utilities are the lingua franca of agentic tooling today, more than 50 years after they were invented! TCP/IP, HTTP, and other internet protocols played a similar role in keeping the internet open and composable.

Fortunately, so far, we are seeing some wins for composable, protocol-centric architectures in AI. Anthropic’s Model Context Protocol was a disruptive move in that direction, an open standard for letting any application reach any tool or data source without a custom integration for each pairing. Along with other open protocols, MCP also now has a home outside of Anthropic at the Agentic AI Foundation (a subproject of the Linux Foundation), which is at least a partial guarantee of its independence.

Isobel Moure, Ilan Strauss, and I made the case earlier this year in Protocols and Power that as models commoditize, competition moves up the stack to context. Opening the means of accessing that context opens the market, regardless of whether open or closed weights sit underneath it. That’s an unbundling, model from harness from context, done the way Apache unbundled web server from web application.

Agentic skills may also be a critical element of the open source AI future, though as the history of the LAMP stack shows, they may fall by the wayside in the same way that Perl and PHP did. And that’s just fine. Composability means that it’s easy to switch to something better when it comes along, or when more people agree on it.

There’s also a lot of great work going on in portable memory from players like Letta, Nous Research, and others. Open source agentic harnesses like Goose and  Pi are also a big part of giving power back to the people. Pi in particular is optimized to be modifiable. There’s a fun story told about Mario Zechner’s decision to give Pi a “/quit” command rather than an “/exit” command like Claude or Codex. Countless issues and PRs have been submitted to Pi’s repo, asking for or implementing “/exit”, but Zechner is stubborn. His retort is that you should just ask Pi to add it to your install.

But the projects I listed above are just the tip of the iceberg when it comes to the scale and scope of open source AI.  Current AI’s Open Source Gap Map covers more than 24,600 open source AI projects!!, with 421 of them scored in depth across openness, capability, and adoption. The map organizes the stack into three layers: 1) models and associated elements including data sets, fine tuning tools, inference frameworks like VLLM, and evals; 2) the product and UX layer, including harnesses and personal agents; and 3) the infrastructure underneath, including core ML frameworks like PyTorch, deployment tools like Ollama, and edge hardware.

Current AI itself is a public-private partnership that came out of the AI Action Summit in Paris last year. This summer they announced AI Potluck, which they describe as “a public project to build a vertically integrated AI product assembled entirely from open source components… a viable alternative to proprietary AI that isn’t owned by any one company or country.” It is backed so far by roughly $400 million of a five-year, $2.5 billion commitment from the French government, tech companies including DeepMind and Salesforce, and major philanthropies including Omidyar’s AI Collaborative, the Macarthur Foundation, and the Ford Foundation.

The fact that this organization exists, along with others like the Agentic AI Foundation, is a testament to the rising tide of interest in open source AI. The coalition of interested parties also says a lot about the underlying motivations that are driving that interest: AI sovereignty, corporate independence from the overweening ambition of the major labs, and an interest in technology for the public good.

Keeping it weird

There’s another element, which Drew Breunig put his finger on in our conversation the other day. The problem with having one or two big closed models dominating AI, and having those models increasingly locking their desired personality, business goals, and guardrails into the weights themselves, is that they will reduce the diversity that is at the heart of innovation.

It’s our job, Drew said, to make it weird, to push a model deliberately out of distribution rather than to settle for whatever the labs have made the default outcome. He described how his team chose not to build in React for a recent project for exactly that reason: every model already knows React too well, so building in it means shipping the average of what everyone else was doing instead of something genuinely their own. He has started using GLM and Kimi not to save money but because they are more malleable and take direction better inside a custom harness. And he wants the open-weight ecosystem to survive precisely so that models stay infrastructure rather than becoming appliances.

That’s what an architecture of participation is actually for. We need real separation between the model, the harness, and the application, so that someone who wants to build something weird can still do it without a lab’s roadmap and guardrails deciding whether they’re allowed to.

“Weird” may make it sound like something that not all developers might want. But we’re really talking about something intensely practical. In his short essay on trading reliability for diversity, linked above, Drew Breunig put it this way:

Labs have to ship a product that delivers “good enough” results when a layperson gives a model a lazy prompt. Without direction, the model must return something decent. (If it’s a website it’ll use the Inter font, cards with a single colored border, gradients, implemented with ReAct and Tailwind). Anthropic named this default output “distribution convergent.” At CAIS, @trq212 put it well, roughly, “If it’s not in your prompt, you’re getting what’s in-distribution” …. Less diverse models make for more reliable coding agents, but they encourage a monoculture of output.

Addy Osmani, my co-chair of the O’Reilly AI Codecon, took this point beyond model diversity after reading a draft of this piece: “Almost nobody I work with is tinkering with weights, but they’re rewriting the harness and what sits around it pretty constantly—skills, subagents, hooks, context files etc etc. That’s where participation is currently happening.” Addy went on to note that forking a skill instead of adopting the default, memory and constitution files that travel with the agent instead of living in a vendor account, or picking the unfashionable framework on purpose are all areas where ease of modifiability matters to everyone.

I want to end by returning to the Apache story. I believe that the big labs are making the same strategic mistake that Netscape and Microsoft made in the mid 90s. Yes, make the models more reliable for ordinary users. But don’t shut down the options for developers who don’t work for you to push the state of the art forward. As Bill Joy put it decades ago, “No matter who you are, most of the smartest people work for someone else.” No one should have a monopoly on innovation, and no one should be building a moat to hold it back.

And be sure to join us at AI Codecon: Building with Open Source AI on August 31, a free half-day virtual conference. You’ll hear from leading developers and technical experts working with open-weight models, self-hosted infrastructure, and real-world AI workflows, and learn how building in the open gives teams more control over costs, data privacy, and what they ship. Register today to save your spot.