When considering the history of AI, Richard Sutton observed that brute force and compute scale has always trumped human expertise, and when you look for it, you can see this “bitter lesson” play out throughout tech history. In his keynote at Ai4 2026, Tim O’Reilly explains why grappling with the bitter lesson is the forge of effective AI corporate strategy, as companies figure out what to embrace and what to let go of. Drawing on his recent conversations with Trail of Bits CEO Dan Guido, Tim argues that AI’s business impact actually hinges on organizational adoption—the hard, unglamorous work of restructuring workflows, data, and incentives around what AI can do. Trail of Bits has modeled that process and documented it in a playbook other companies can use. Here, Tim shares some of the practices, like capability ladders, shared config repos, and company-wide hackathons, that helped Trail of Bits make AI a structural component of its business. This doesn’t mean that AI-native companies “sit back and let the progress of AI carry us forward.” Human expertise still matters, and it’s often the differentiator that helps organizations rise above their competitors. As Tim concludes, “The world is full of great problems. And so if AI takes away and makes easy something small, celebrate it and go work on something big with the new powers that we’ve been given.”

Takeaways

02.33 The bitter lesson is real, and it can catch any of us.
The bitter lesson is Richard Sutton’s contention that human expertise doesn’t really matter, that it will eventually be outmatched by computing scale. O’Reilly’s Whole Internet User’s Guide & Catalog was the first catalog of websites and the first site on the web to have advertising. It grew into Global Network Navigator, which was the first web portal. But O’Reilly’s products were manually curated. Yahoo came along and expanded on these ideas, but O’Reilly and Yahoo were both beaten by Google, which simply threw a bunch of compute at the problem. Now ChatGPT has changed the game again.

06.27 AI-native workflows require a different mindset.
When O’Reilly set out to develop a product that assessed learners’ capabilities and gave them a skill path to level up, the team used AI as an assistant, to write quiz questions, for instance. But LLM chatbots can already identify skills when given context about a developer. Evolving toward an AI-native skill path builder meant reconceptualizing the product as a more interactive experience that reflects where capabilities are today. However, even the most well-thought-out workflow can be hindered by gaps in access or knowledge. As Trail of Bits CEO Dan Guido says, “You have to build a system in which expertise compounds.”

10.28 AI adoption is a human problem.
Moving up the framework for AI adoption from AI-assisted to AI-augmented to AI-native isn’t just a technical challenge. It’s psychological. Only 5% of Dan’s staff was actually on board when he started the transformation; 70% were just quietly going through the motions, and 20% were actively resistant. He traces this to a handful of biases: self-enhancing bias, opacity, intolerance for imperfection, and above all, identity threat, the fear that AI won’t just replace the work someone does but who they are. Getting teams on board requires the organization to reframe AI as a tool that enhances identity, not something that will take it away.

16.44 A status ladder helps team members understand where they’re at and where to focus next. Hackathons compound that knowledge across the company.
Trail of Bits has a three-level status ladder: not engaged with AI or actively resisting it, experimenting with AI, and building AI that strengthens the organization’s overall capability. Level zero isn’t treated as a skill gap. It’s treated as working against the company’s goals, and the other two levels get a more detailed capability matrix broken out by department, since what a security auditor does with AI looks nothing like what someone in accounting does. O’Reilly is building its own version of this, drawing on the technical and business skill data it already has across its platform. Trail of Bits runs a hackathon every two months, each with a stated objective and learning goals announced a week ahead. Success is measured not by what got shipped but by where people land on the capability ladder afterward. Then the work gets fed into a shared skill repo, giving the entire company a set of reusable artifacts, and what one hackathon turns up becomes something the next one can build on.

24.37 Turn scar tissue into infrastructure.
Drew Breunig talks about the problem of prompt debt: prompts that grow more complex and more tuned to one specific model until they’re no longer portable. Trail of Bits flattens this complexity by turning every failure into a global, copy-pasted fix hosted in a company-wide repository. They’ve also standardized the safety net, with sandboxes for different needs and a seven-day cooldown on every new package from outside that gets installed—rules the whole company follows. To make this all work, employees need the chance to try things out and iterate on their failures. Dan says the only real mistake he made was not giving people enough unstructured time to experiment.

32.44 Human expertise still matters.
AI can make companies more productive, but it’s not a magic weapon. It’s a medium that people can use to share or extend their unique expertise and perspective. O’Reilly’s mission is to share the knowledge of innovators: You can think of the company as a matching marketplace for people who have expertise and people who need it. Agents offer a valuable new means of getting that expertise to customers in the tools they’re using to make business decisions. O’Reilly CTO Andrew Odewahn has noted that faster local decision-making has splintered central planning, so it’s harder than ever to get the big-picture view a good corporate decision needs. O’Reilly’s Expert MCP server lets customers access our content and use it to increase organizational intelligence. For instance, you can ask an AI tool to analyze a team’s workload and write a hiring case based on how O’Reilly’s own experts would review the request, and you’ll get a grounded argument with solutions authenticated by citations from actual practitioners. O’Reilly is building this capability into an organization-wide grounding layer it calls O’Reilly Expert Intelligence. It’s in beta now, and you can check it out.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The following article originally appeared on Addy Osmani’s blog and is being republished here with the author’s permission.

A software factory harnesses loops at scale. You can run the loop with humans in it (light factory), trading judgment and concentration against speed and breakage. Or you can ignore the humans (dark factory) and let those agents scope, build, and ship code without anyone reading the details. If people stop reading, though, they’ll stop understanding your software. Your hardest job now is knowing which checks to build and how much autonomy to delegate.

This idea of the software factory is a term that dates back to Bob Bemer’s paper, “The economics of program production,” given in 1968. For half a century, many have dreamed of a world in which software is a repeatable and instrumentable production process (analogous to stamping out car parts in a factory) rather than the isolated craft of individuals. Historically, this dream has generally (although not universally) fallen flat, in part because of the difficulty of stamping out ideas.

But in the last two years, things have changed dramatically enough that now it makes sense to take a fresh look at the old dream. And since some subtleties can easily be glossed over, it’s worthwhile to be somewhat precise about exactly what’s really new and different, and what may be recurring traps, dressed up as new opportunities.

Dex Horthy, co-founder of HumanLayer recently gave a great talk at the AI Engineer World’s Fair called “Harness Engineering is not Enough: Why Software Factories Fail.” worth checking out on this topic.

The loop is the atom. The factory is the loop at scale.

Structure is everything, and it all starts with small units. The whole stack is really three concepts layered on top of each other: the loop, the harness, and the factory.

A loop is one agent doing a single job on repeat: gather context, take an action, check the result, and go again until some condition is met. It is the smallest unit of agentic work, and everything above it is just loops stacked on loops.

The point of loop engineering is that you stop prompting the agent turn by turn and instead design the small system that prompts it for you.

A harness is the walls around a loop: the sandbox it runs in, the tools it can reach, the memory that survives between runs, and the gates that decide what “done” means. The loop is the behavior; the harness is the environment that behavior runs inside.

Hand a raw model no harness and it will happily spin forever. The harness is everything around the model that makes it useful and safe to run.

A software factory is many harnessed loops running at once, fed by a queue of work and drained through a review gate into production, with humans owning the whole thing from above. It isn’t a bigger agent; it’s an org chart made of loops.

The final paradigm shift is moving from writing code to building and running the factory that writes it. The unit of work shifts up a level, to the loop, the harness, and the flow between them, rather than the individual code diff.

Loop → harness → factory. A factory isn’t a smarter agent; it’s many harnessed loops feeding one review gate, with a human owning the outer loop.

The factory, drawn

The central slide Dex spent most time on was brilliant because it’s a clarifying wiring diagram that visualizes what otherwise is an obvious loop. Here’s my take on it:

The factory is a closed loop: Intent and production signals feed a queue, the harness builds, automated checks and review gate it, deploy ships it, and monitoring turns production back into signals.

Intent flows from the vision of engineering leadership and directly from engineers into a queue of work. Signals driven by incidents and user requests drive the same queue. The harness picks an item from the queue and builds a change for it. Beyond the harness, automated checks make changes safe enough to let into production. These automated checks run at once without any conscious involvement from engineers, thanks to CI, tests, static analysis, and scanning of all kinds. The review gate is the only decision point here. After approval, changes are deployed and monitored in production, with monitoring data feeding back into the signals that kicked the loop into motion to begin with.

By and large, every box in this diagram is almost zero cost: generation, tests, scanning. They all run at scale for negligible cost. There’s only one expensive box that proves stubbornly resistant to scaling, and that’s the review gate. That shiny amber box is “judgment,” and where the crux of the argument about whether we can make development faster and more frequent resides.

Why we call it “dark”

A dark factory runs with the lights physically off because the only things on the floor are machines, which don’t need light to see. A dark software factory operates similarly, as code ships that no human has read and is verified only by other machines.

The image is borrowed from manufacturing. Its origins are physical rather than digital, rooted in facilities where the lights are turned off and the work is carried out by robots. FANUC in Japan has been running lights-out factories of this sort since 2001. Xiaomi, in 2024, opened a heavily automated dark factory of its own. What these have in common is a product assembled and shipped without a single human having read any of it. The “dark” comes in when that act of reading is removed from the process.

I’m not borrowing the concept for its vibe or as an insult. For all its creepy buzz, “dark” here is a simple physical claim: the original factory floor, but without light. In software, the floor is the diff. Whoever wrote the diff, whoever reviewed it, whoever shipped it, those humans are gone, and what remains is a diff verified only by the machines that built it.

This is a surprisingly easy thing to do, at least at first. It’s easy because that missing review step gets in the way of everything. Its absence makes your perception of your team’s vertical throughput seem suddenly and radically higher. It feels as if you’ve broken the sound barrier. For all its apparent ease, it’s harder than it seems to survive those dark workflows, with all their buried costs.

Harness engineering is not enough

The harness of orchestration, sandboxed prototyping, and tool calling as models interact with the world and each other will become increasingly powerful and effective. However, there’s an inherent in-model failure in trying to keep up with codebase quality over the long game and through additive changes, and I think there’s good reason to believe that models alone will ultimately lose that battle against comprehension debt.

Comprehension debt is the widening gap between how much code exists and how much any human still understands. A dark factory doesn’t pay it down; it takes it on as fast as it can, with the tests green the whole way.

This is an important distinction because models do well at some tasks. But for anything that isn’t an immediate change to a small part of a codebase, especially in a complex brownfield system, model-only automated coding faces an insurmountable obstacle. Weekend toys and side projects are alike in that a few months of development cycles is usually enough to get things in working order, or at least close enough. But an enterprise system that has been under development for a decade or more is a different beast; it has to be maintained, in a professional environment at a professional pace. Three to six months into a project, you’re already drowning in unread code. That kind of environment, and especially the constraints enforced by production code, would make even a powerful agent do poorly, all of it in contrast to the vibe-coding enjoyed by developers working on weekend toys.

Dex reports from experience that this is a major failure, so much so that it required painstaking manual debugging to pinpoint. This came from running a fully automated code factory for about four months, during which no human looked at the code that was written. Underlying the experience is a tradeoff between two conflicting metrics. One is maximizing token utilization, the number we currently treat as progress. The other, which it quietly minimizes, is the amount of the system any human participant still understands at any moment.

Where the dark factory truly shines is in its ability to burn through pristine code while the tests stay green. The ultimate reckoning, when it comes, will not be a dramatic “it all goes sideways” moment. It will be quiet and late.

A dark factory pipeline versus a lit factory pipeline: Dark and lit are the same pipeline with the lights in different places. The lit version moves human judgment upstream to design and architecture rather than just re-adding review at the end.

The bottleneck was never generation

The fundamental constraint in a software factory isn’t how much code we can churn out, it’s how quickly we can verify it.

Back pressure is the rule that you can only hand a loop as much autonomy as you can cheaply and reliably verify, and not one inch more. Verification, not generation, is the real constraint on a factory.

Because unbounded generation capacity is in perpetual tension with the finite, non-scaling resource of human attention, the core problem is the gap between cheap generation and bounded review. Look at the funnel: As long as the neck representing verification doesn’t widen, it’s going to back up. As Dex points out, volume alone isn’t the problem: What we’re really suffering from is a surplus of bad PRs. When you’ve got high volume without trustworthy gates, manufactured defects are unavoidable. This is just back pressure again: Autonomy can’t expand beyond what can be cheaply and reliably verified.

The second-order problem is why improving the model shouldn’t automatically close the gap between what it can generate and what can be verified. Training on well-architected systems is an arguably more difficult proposition than passing simple tests: remember, the cost functions measuring architectural excellence aren’t measured in seconds or even minutes, but in months and years. Tidy gradients are functionally impossible to compute, so a system expecting crisp, instant evaluation of complex design decisions isn’t going to be trained on good examples.

Generation is a wide mouth; verification is the narrow neck. Speeding up the mouth just deepens the pile at the neck.

Turning the lights back on

A lit factory is the same pipeline with the lights left on where judgment lives. Agents still do most of the building, but a human reads what comes out before it ships, keeping the lights on wherever a wrong call is expensive.

The lit version doesn’t tack review onto the end but moves the point of human judgment upstream, to the product, the design, and the architecture before an agent starts a loop.

One great thing about that upfront hour is that it leads to fewer implementation hours. It turns a long, frustrating code review into a quick read of a two-hundred-line plan. You get to review a decision before it’s built, so later you aren’t chasing through two thousand lines of generated code to find out what the decision even was. Some decisions are expensive and long-lived enough that you’d want a person in on them early, before the cost compounds. Of course, there are still times you look at diffs, even when you’ve spent time up front.

You might be thinking that all sounds unglamorous. You’re right. The safety net is made up of perfectly ordinary architectural practices we’ve always known about and mostly ignored: good types and method signatures so that mistakes are caught by the compiler instead of in production; test seams where we can pin behavior and make change observable; laying out the code so the next reader, human or model, knows where to find the thing they care about; keeping call stacks short and legible; keeping component boundaries well defined so a change doesn’t have a huge blast radius; and dependency injection so we can swap out one piece for another. None of it is new. We’ve always said we care about good architecture. But now that we’re using automated coding agents, that architecture is finally doing a second job as a cheap and hard-to-fake safety net against the mistakes the agent will make.

That safety net has to live outside the model because the model won’t supply it. The coding agents that feel most capable, Claude Code and Codex among them, are reinforcement-trained against their own harness and tools: fluent with all the tools and idioms of the trade, but not with things like long-term maintainability. The deliberate architecture we’ve always talked about is the tool that catches that debt, and the investment we make in it is us buying back our autonomy. Put that together with safe infrastructure, and there are some tight, low-risk loops you can run unattended. Horthy described one in a recent post: A nightly GitHub Actions cron that fixes exactly one anti-pattern, a lint violation or a needlessly optional prop, commits, and opens one small pull request, all on its own, so the team wakes up to a slightly better codebase and a diff short enough to read. But for loops with high enough stakes, you don’t want to risk waking up to a broken auth system, billing engine, or public API contract. Keep the lights on there, and trust that a person with judgment and a real working knowledge of the system will catch the mistake.

What earns a loop the dark

This rule applies whether you call it back pressure, verification, or the light switch.

A loop can earn itself fully automated status only if the check is cheap, runs at high frequency, and relies on something that can’t be easily faked out. Green-or-red oracles, type gates, property tests, and a review agent coupled with a real rubric all fit. You also need the oracle to answer immediately and not drift over time. When done can be proven not just by you but by a machine, you’ve reached automation.

Short loops are easier to verify than long ones. Dex’s rule of thumb: An agent holds up for three to ten steps, then starts losing the thread past twenty. The reason is context accumulation. The more the agent drags along, the more likely it is to wander off. When a loop is short, verifying it is cheap. Sprawling loops hide mistakes in the corners, which is another way of saying they never earned lights-out status.

Keeping the lights on is the opposite case. A loop needs to be reviewed if a wrong answer is expensive and only a person can catch it. Subtle production bugs that can’t be caught by tests, large blast radii, and a decision that’s going to shape the work of a year or more all qualify. In those cases, human judgment does not leave the software; your attention is the costly, essential part.

The danger is forgetting to flip each switch and just setting all of them to the same mode. All dark, and you’re stuck tearing everything down four months later. All lit, and no one can get reviews done in time and you’re stuck in a gigantic bottleneck. The hard, skilled job is deciding where to put each switch.

Loops, graphs, or state machines?

When you hand an agent a task, you’ll likely build a graph around it, whether you call that graph a finite state machine or a set of conditionally linked service calls. It’s a framing where the software isn’t just following some abstract rules but a structured workflow: Every node is an explicit step, and every edge between nodes is an explicit condition. That sounds like a lot of structure, but most of it’s already there in any software, since any code can be expressed as a control-flow graph. So the only real novelty is that an agent insisting on autonomy is really just walking around a particular graph, and its freedom is constrained to the inside of a node. And here’s the part people forget, which Dex wrote down a year ago: software was always going to have that structure. There’s a reason we used to draw programs as flow charts. The genuinely new move was trying to throw the diagram away, leaning on a loop where the model picks the path tool call by tool call, until it declares itself done. That felt like liberation, right up until it met a ten-year-old codebase, and the discipline everyone is now rediscovering, owning your control flow, is really just walking the graph back around the loop. So the question of whether we should shift from loops back to graphs is almost an admission that we needed the flowchart all along.

Here’s what it looks like in practice. Take a bug to fix. As a pure loop, you sit down and think: figure out what’s wrong, change some code, run the tests, see what happens, and if that round doesn’t kill the run, loop back and start again. The whole journey is decided as you go, which problem you chase, the exact code you change, which tests you run and in what order, whether you run tests at all, and whether you try again or declare victory. As a graph, the first thing you do is map out what should happen. Reproduce the bug or go ask for more information, find the cause, try a fix, run the tests, and let a failing run route back to the fix while a passing one goes on to review, where only an approval reaches done. The agent is still clever inside each box; it just can’t wander off the paths you sanctioned. Santi laid this out with a diagram that makes the difference obvious.

The real appeal of that graph, of course, is that it’s back pressure drawn as a diagram. You give up some of the agent’s freedom and get mandatory checks and legible failure points in return, so when a run dies you can point at the node that killed it. It’s the same instinct behind Dex’s blunt line that most so-called agents aren’t very agentic at all, “mostly deterministic code, with LLM steps sprinkled in at just the right points.” And this isn’t just an artifact of how people happen to be building things right now: you can see the pattern in LangGraph and LlamaIndex Workflows, in Jerry Liu’s hybrid workflow-graph-over-agents with an outer loop that grows parts of the graph as it runs, and in David Khourshid’s reminder that this is really just state machines and the actor model turning up in new clothes.

One clarification, because the term is badly overloaded: when I keep calling this a graph, I don’t mean a knowledge graph. I mean a predefined directed graph of how the work should flow, conditional edges and all, giving the loop a shape you can actually trust.

Where the human actually goes

Notice that the person never left the factory. They moved.

I think engineers need to increasingly own the outer loop. The agents can investigate a bug, write up the diagnosis, implement a fix, run the tests, and write up a report. That’s the execution of the inner loop, and they can do it as efficiently as anyone. But that was never the job. The bits you own are what I’d call the outer loop: Decide whether it’s the right way to address the problem, verify that the diagnosis and implementation are sound, approve the change, and carry the consequences of being wrong. The boundary between the two loops is evidence, the diffs, the tests, the logs, and a brief explanation that connects them. Types, seams, and rubrics make it possible to oversee all this without doing a lot of work for every change.

I think it’s useful to put it this way: you’re not down on the line writing changes any more; you’re up at the end of the production line designing it and guarding the gate. There’s a lot you can do to make the model better and the harness more capable, but I’ve observed that identifying problems that are expensive in the long term is not typically something you can automate away. The core thing that’s still the job is to exercise human judgment better than any flow of paper and computing power.

Robots are fine operating in the dark, but humans need to see what they’re doing. If everything on the factory floor is dark, and you can’t see anything, and you can’t even find the light switch, that’s where the danger is.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

OpenAI expanded into software control, scientific reasoning, and financial services this week, and investors committed billions more to AI companies across the stack, while AI researchers went public with warnings. This Week in AI host Christina Stathopoulos looked at what those developments mean for an industry already wrestling with questions about safety and control.

Models are moving from answering questions to doing work

OpenAI’s latest announcements showed how much more work companies now expect models to handle. GPT‑6 Astra can navigate software interfaces, complete multistep workflows, and apply advanced reasoning to scientific and mathematical problems, and ChatGPT for financial services was developed with input from Morgan Stanley and Evercore to support research, financial modeling, and creating client materials.

OpenAI also shared a solution to the previously unsolved Navier–Stokes Millennium Prize Problem. A coordinated system of 10,000 AI agents worked for 88 hours on the proof, followed by another 17 hours of model-based verification by Astra. However, the company’s claim drew scrutiny after outside researchers questioned whether OpenAI might have had access to related work, an allegation OpenAI denies. While impressive, scientific breakthroughs like this also raise important questions about how well we understand these systems and the role humans should continue to play in scientific discovery. (Hugo Bowne-Anderson got into this in a recent article on Radar.)

Investors are placing bets across the AI stack

Money continues to flow to AI companies, but investors are backing infrastructure, platforms, and specialized applications rather than converging on a single layer of the stack. French company Mistral has raised €3 billion with plans to spend on compute infrastructure and open weight models. Legal AI company Harvey, inference chip startup Positron, and enterprise AI company Wonderful also raised large rounds, while NVIDIA announced its acquisition of Hugging Face for nearly $13 billion.

Christina cited figures showing global AI funding rising from $56 billion in the fourth quarter of 2025 to $242 billion in the first quarter of 2026 but questioned whether generative AI will produce returns that justify that level of investment. Some of these companies may build durable businesses and others may not, even if AI itself continues to deliver useful products and services.

Safety issues are colliding with high-value applications

AI safety is back in the news, following Anthropic researcher Jacob Coxon’s highly publicized resignation. Coxon warned that labs were moving too quickly toward poorly understood systems capable of recursive self-improvement, and other researchers associated with Anthropic and Google DeepMind raised similar concerns. Anthropic CEO Dario Amodei also called for stronger evaluation, shared safety standards, and international coordination. (Sam Altman and Elon Musk seconded the call.)

While the industry remains divided over catastrophic-risk scenarios, many nearer-term problems are already concrete, and Christina was more concerned about people using powerful AI systems maliciously than about autonomous systems becoming dangerous on their own. Organizations deploying more autonomous systems must tread carefully, with robust security, access controls, testing, and human oversight in place.

AI for good is getting more concrete in genomics

After a week of AI safety warnings, Christina ended on a positive note with what she calls “AI for good” and highlighted genomics projects from DeepMind, UC Berkeley, and Tempus. Their work uses AI to predict how genetic changes affect gene function; identify mutations associated with disease; and connect genomic data with patients’ medical histories.

For researchers, AI can make it practical to study genetic possibilities that would be difficult to test individually in a lab. That could help narrow the search for disease-related variants and support earlier diagnosis and more personalized treatment.

What’s next

Join us again next Monday for another episode of This Week in AI, when we’ll dive into more of the news, issues, and key developments shaping the AI era. And check back each Friday for the latest episode, or watch on YouTube, Spotify, Apple, or wherever you get your podcasts.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

The way we talk about data is changing faster than the way we build it. Every quarter a vendor ships a new approach, coins a new term for it, or quietly adopts a term someone else has been using and redefines it to fit the shape of their product. None of this is malicious. Every company describes the landscape from wherever they happen to be standing. But when six vendors do that to the same word, practitioners are left translating between six versions of it before a design conversation can even start.

There’s a second problem stacked on top of the first. Most of the vocabulary we use to talk about data in the AI era comes from academic disciplines that very few working practitioners have spent time in. “Data warehouse” is immediately legible: You know what a warehouse is, so you know this is a place where things are stored until someone needs them. “Ontology” is not. It arrives from philosophy by way of knowledge engineering, where Tom Gruber defined it in 1993 as an explicit specification of a conceptualization. That’s a precise definition. It’s also useless to a director trying to decide what to fund next quarter.

What follows is an attempt at a working vocabulary, written for the people who actually deploy these technologies and the people who approve their budgets. For each term I want to answer three questions. What is it, actually: software, an artifact, or a practice? What job does it do? And which kind of output does it serve? That last question needs some setup, so let’s start there.

Deterministic and probabilistic outputs

Data systems produce two kinds of output, and knowing which one you’re after is the single most useful diagnostic in modern architecture.

A deterministic output is the same every time you ask the same question. What was ARR for the last twelve months? Whether that question goes to a dashboard, an API call, an Excel workbook, or an AI agent, the answer should be identical. Ask four different agents running on four different models and you should still get one number. Deterministic outputs have traceable lineage. You can point at the calculation and walk someone through how the number was produced.

A probabilistic output is what you get from systems that are non-deterministic by design. Change the ARR question slightly and the category changes completely: Instead of “what was ARR over the past twelve months,” ask “how can we improve ARR over the next twelve months.” Put that question to the same model, in the same agent, twice in a row, and you’ll get two different answers. That’s not a bug. An LLM is predicting a likely sequence of tokens across billions of parameters, and the output varies every time it runs.

Neither type is better. Both are necessary. The failure mode is asking a probabilistic system for a deterministic answer and not realizing that’s what you did. Most of the terms below exist because the industry is trying to solve exactly that problem: How do you put enough structure around a probabilistic system that it can return deterministic answers when the question calls for one?

With that, let’s work through the terms.

Semantic layer

I’ve written about semantic layers for Radar several times, including what they are and why they matter and why they function as a risk mitigation strategy. The short version: A semantic layer is software that sits between your data and the people and tools that consume it, giving everyone a single place to access trusted, governed metrics.

Behind the scenes, it does three things. It holds definitions: How do we calculate this business metric? It holds context: What does this model or column contain, and what’s it typically used for? And it holds relationships: How does this data fit together? Modern tools bundle in more than that, including query engines, caching, and a single point for access control and security, but definitions, context, and relationships are the core.

Why does this matter for AI? Because it lets an agent navigate data instead of reasoning over it. Without a semantic layer, an agent that’s asked for last year’s ARR has to inspect table names, guess at joins, infer which date field represents revenue recognition, and reconstruct business logic that lives in someone’s head. That’s reasoning, probabilistic, and produces a different answer depending on the day. With a semantic layer, the agent looks up ARR, queries the definition, and returns the same number every time. It’s a deterministic answer delivered through a probabilistic tool.

The analyst community has caught up to this. Gartner now predicts that universal semantic layers will be treated as critical infrastructure by 2030, alongside data platforms and cybersecurity.

Ontology

Ontology is the term most likely to derail a meeting right now, largely because Palantir made it commercially famous while the underlying concept came out of decades of academic work on how to formally describe things and the relationships between them.

Here’s the simplest way I’ve found to separate it from a semantic layer. A semantic layer answers what does this number mean and how is it calculated? An ontology answers what things exist in this business and how do they relate to each other? The semantic layer is metric-first: measures, dimensions, and the logic that connects them. The ontology is entity-first: customer, order, shipment, facility, supplier, along with the relationships and rules that govern how those objects behave.

The overlap is real, and it lives in relationships. Both artifacts encode how things connect, and vendors are increasingly shipping both capabilities under a single product name, which is a large part of why the terms have blurred. The practical distinction is what the system needs to do. If the job requires consistent numbers across every reporting tool, a semantic layer is the center of gravity. If the job requires an agent that reasons about business objects and takes action on them, rather than just reporting on them, an ontology is what gives it a model of the world to act in.

One useful clarification: An ontology isn’t software. It’s a model, an artifact your organization authors and maintains. Software delivers it, but the value is in the modeling work.

Knowledge graph

If the ontology is the schema, the knowledge graph is that schema populated with actual data. The ontology says a customer places an order, and an order contains line items. The knowledge graph holds your real customers, your real orders, and the edges connecting them, stored as nodes and relationships rather than rows and columns.

How do you know when to use a knowledge graph over a semantic layer? Warehouses and semantic layers are excellent at aggregation: how much, how many, compared to when. Graphs are excellent at connection: what is linked to what, and how far apart. “Which suppliers are two steps removed from this delayed shipment?” is a graph question. So is “which accounts share a beneficial owner,” and “who has inherited access to this dataset through three layers of group membership?” You can answer those with SQL. You won’t enjoy it.

Graph traversal is deterministic. Given the same graph and the same query, you get the same path every time, which is exactly what makes graphs useful as grounding for an agent. Rather than inferring that two records refer to the same supplier, the agent follows an edge that someone already asserted. The relationships are modeled facts, not inferences made at inference time.

A knowledge graph is not a substitute for a semantic layer. They answer different questions, and mature architectures increasingly run both.

Context

Context is the most overloaded word in the field right now, and it’s worth splitting into pieces before using it in a sentence.

Deterministic context is metadata, plainly. It lives in your semantic layer or your ontology: field descriptions, metric definitions, object relationships, business rules, exclusion logic. What has changed isn’t the concept but the consumer. Metadata used to be documentation for humans, and it was the first thing to go stale because nothing broke when it did. Now an agent reads it at query time to decide what a column means and whether it’s allowed to use it, which makes it functional infrastructure rather than a wiki page nobody updates. It’s versioned, reviewed, and reads the same way every time a system asks for it. This is an asset you maintain.

Runtime context is what an agent assembles at the moment of inference: the system prompt, conversation history, retrieved documents, tool outputs, whatever the orchestration layer decided to put in the window. It’s ephemeral, and directly changes the answer. Same question, different context window, different output. This is a variable you monitor.

Cutting the other direction, structured context describes governed data: columns, metrics, entities, relationships. Unstructured context is the policy PDFs, contracts, support tickets, and wiki pages that hold the reasoning behind the numbers. Unstructured context is genuinely valuable and usually retrieved through similarity search, which means it arrives with probabilistic behavior attached. What surfaces depends on how the question was phrased.

The practical rule: When someone tells you their tool is “context aware,” ask which kind. Deterministic context is what makes an agent’s answer repeatable. Runtime context is what makes it relevant. Conflating them is how teams end up trusting an answer that was only true for one prompt.

Observability

Observability is the telemetry that tells you whether your systems are still doing what you believe they’re doing. It isn’t data quality, which is a judgment about whether a number is correct, and it’s not testing, which is a check you wrote in advance for a failure you already anticipated. Observability is the instrumentation that lets you ask “is this still working?” without having predicted the specific way it would break.

On the deterministic side, this is familiar territory: freshness, row counts, schema changes, null rates, job failures, and lineage impact. If ARR is supposed to refresh at 6 a.m. and today it didn’t, you want to know before the CFO does.

The probabilistic side is harder because there is often no error to catch. The system returns a fluent, plausible answer that happens to be wrong. Monitoring here means evaluation sets scored over time, tool call success rates, retrieval relevance, refusal and fallback rates, latency, cost per query, and structured human feedback.

Which brings us to drift. Drift is what happens when the world changes underneath a system that keeps running unchanged. Data drift is a shift in the inputs: a new business unit lands in the source system, order volume triples after an acquisition, a vendor starts sending nulls in a field that was never null before. Model drift is a shift in behavior: The provider ships a new model version, or a prompt template changes, and outputs that were stable last month aren’t stable this month.

Here’s what drift looks like in practice. In March, an agent answered “what were our top five products by margin?” correctly. In June, a new product hierarchy shipped upstream, and the agent now silently excludes an entire category. Nothing failed. No alert fired. The answer is simply wrong, and it’ll stay wrong until someone notices. Deterministic systems tend to fail loudly. Probabilistic systems fail quietly. Observability is how you catch the quiet ones.

The working vocabulary

• Deterministic output: The same answer to the same question every time, with a calculation you can trace.
• Probabilistic output: A different answer to the same question each time, produced by prediction rather than calculation.
• Semantic layer: Software that stores the definitions, context, and relationships behind your business metrics and serves them consistently to every downstream tool.
• Ontology: A model of what your business is made of, the objects, their relationships, and the rules that govern them.
• Knowledge graph: An ontology populated with real data and stored as nodes and edges, so systems can traverse relationships instead of reconstructing them through joins.
• Context: The information a system needs to use data correctly, either governed in a semantic model or assembled at runtime by an agent.
• Observability: The telemetry that tells you whether your data and AI systems are still doing what you think they’re doing.

Read that list in order and something becomes obvious: These aren’t competing products. They’re layers. The ontology describes what exists. The knowledge graph holds the instances. The semantic layer defines the measures. Context is how any of it reaches a model. Observability is how you find out when it stops working. The reason why these terms feel like they’re fighting each other is because they’re usually sold as substitutes, when in practice, they stack.

The vocabulary will keep moving. Two years from now some of these words will be absorbed into product names and mean something slightly different than they do today. That’s fine, as long as your team has a shared answer to two questions about any term someone puts in front of you. What is it, actually: software, an artifact, or a practice? And which kind of output does it serve, deterministic or probabilistic?

Those two questions cut through most of the noise. Agree on the words first. The architecture arguments get much shorter after that.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

Every knowledge program seems to begin with the same request. A senior engineer is leaving in six weeks, and someone asks her to document the process she’s carried for years.

She returns a clean flowchart of the happy path. The drawing is accurate and may even be elegant. It leaves out the thresholds she watches, the conditions that make the standard procedure unsafe, and the supplier whose parts fail in humid weather. She doesn’t think of those judgments as separate knowledge. After years on the job, they feel obvious.

Six months later, a production line goes down and the knowledge base can’t explain what to do. The interview took place as per the process. Its transcript was chunked, embedded, and indexed, so the search returns the relevant passage quickly. The passage still can’t answer the question because no one asked the engineer to explain the judgment behind the procedure.

That gap now limits many enterprise AI programs. Organizations continue to improve retrieval over collections that omit some of their most valuable operating knowledge. Better ranking can help people find what was recorded; it can’t recover the expertise that never entered the collection.

The blind spot in enterprise knowledge systems

Michael Polanyi gave the problem its durable formulation in 1966: “We can know more than we can tell.” In The Tacit Dimension, he argued that competence depends on skill, perception, and judgment that resist full explanation, even when an expert sincerely tries to teach them.

In companies, tacit knowledge usually appears in three forms. Elicitable knowledge remains unspoken because nobody has asked a precise enough question, or because an expert assumes that everyone sees what she sees. Perceptual knowledge lives in trained attention: An engineer hears a bearing begin to fail, or a nurse notices that a patient looks wrong before a monitor changes. Collective knowledge resides in a team’s habits, standards, and shared sense of what a sound decision looks like in that organization. Each form requires a different method of transfer.

Preventive judgment creates another difficulty for the architect. A failure produces a ticket, an incident report, and a trail of messages. An experienced operator who quietly avoids a known failure mode on a Friday afternoon produces none of those records. The useful outcome is the absence of an event, so the data pipeline receives no trace of the decision that produced it.

Machine learning can infer rules that people struggle to articulate, provided the model sees enough representative examples. It’s difficult to find enough examples of rare expertise for training. A company may have only a handful of unusual incidents and one person who has learned, over decades, how to read them.

David Autor described this limit as “Polanyi’s paradox”: Many of the tasks that are hardest to automate depend on rules we can’t state. Modern machine learning works around the paradox by learning from examples, but the workaround weakens when examples are scarce. Fine-tuning can teach a model the company’s vocabulary and document formats. It can’t reconstruct decisions that left no data.

At the same time, the economics have changed. Much of a field’s documented best practice now appears in frontier-model training data and is available to competitors at roughly the same price and quality. The more widely explicit knowledge circulates, the more a company’s advantage depends on local judgment: the exceptions, thresholds, relationships, and practiced responses that its people have accumulated.

That makes elicitation an architectural concern rather than an offboarding chore. The organization needs a repeatable way to surface the knowledge that can be expressed, a route for the expertise that must be demonstrated, and enough humility to distinguish the two.

A protocol for elicitation

The central design question is straightforward: Which follow-up would prompt an expert to say the missing judgment aloud? The quality of the interview sets the ceiling for the knowledge base. The index determines how quickly someone can reach the resulting material.

Interviews can be made more reliable even though judgment itself remains highly personal. An expert may know that a particular supplier fails in humid weather. The interviewing protocol doesn’t need to possess that knowledge in advance; it needs to notice a phrase such as “we escalate if it looks bad” and ask the expert to define “bad” in observable terms.

Expert explanations tend to become vague in four places. An effective interview protocol asks targeted questions about each one:

• Thresholds: Which number, reading, or condition triggers the action?
• Exceptions: When does the documented procedure cease to apply?
• Evidence: What did the expert observe before reaching the conclusion?
• Escalation: Who becomes involved, and at what point?

These questions uncover the operational detail that runbooks often lack. They also identify a narrow, useful role for a language model during the interview: proposing the next question that turns a general statement into a usable rule. I’ve been building an open source toolkit, ExpertTrace, around that protocol.

The value appears in the difference between what an expert volunteers and what the same expert confirms after one focused follow-up. Consider a typical first answer:

We review high-risk use cases before deployment. If the risk seems significant, we escalate to the governance council.

The statement will embed cleanly and retrieve for a relevant query, but a new employee still cannot act on it. “Seems significant” supplies no decision criterion. A targeted follow-up produces something much more useful:

Escalation to the council is required when the use case touches employment, credit, or health decisions, or when model output reaches a customer without human review. Predeployment review is skipped for internal-only tools with no personal data, which is the exception people get wrong most often. If we cannot identify a named accountable owner, the review does not proceed, regardless of risk tier.

The second answer takes little additional time, yet it contains a decision rule, an exception, a recurring failure pattern, and a blocking condition. It can guide a real dispute instead of merely mentioning the subject.

The protocol needs guardrails. Limit the number of follow-ups; a long interrogation exhausts the expert and eventually produces agreeable noise. Keep the model focused on generating questions, and separate that task from compiling and validating the answers. An expert’s statement belongs in the record with its provenance and context. Whether the statement is accurate requires independent review.

The four-plane architecture

Elicitation is one part of a larger knowledge system. A tacit-aware architecture has four planes—capture, representation, serving, and transmission and each plane addresses a different failure in the movement of expertise. Figure 1 shows how the four planes work together and which forms of tacit knowledge each can reach.

In the capture plane, structured interviews, incident reconstruction, decision journals, and observation collect more than polished procedure. Record the trigger, evidence, exception, and escalation path while the expert can still explain the surrounding conditions. Route perceptual skill toward demonstration and practice instead of forcing it into prose.

Once knowledge has been captured, the representation plane preserves the distinctions that make the material trustworthy. A compliance policy, a war story, and an untested hypothesis shouldn’t become interchangeable chunks. Carry provenance, confidence, and validity context—including the plant, time period, equipment, and conditions—as first-class properties. Extend the knowledge graph beyond documents to the people and episodes that produced them.

The serving plane then determines how that knowledge reaches users. Answers should cite retrieved evidence and show the source. When the collection can’t answer, the system should say so clearly and route the question to someone with relevant experience. “Ask Joe; she rebuilt this line in 2023” is more useful than a fluent paragraph assembled from weak evidence, and the referral restores the human contact through which difficult knowledge often moves.

The transmission plane completes the architecture by helping how expertise moves between people through shadowing, teaching, and communities of practice. The platform should detect when knowledge concentration and attrition risk converge, then trigger capture and apprenticeship before a notice period begins.

Gabriel Szulanski examined 271 observations of 122 best-practice transfers across eight companies and found that even willing teams struggled to reproduce methods developed elsewhere in the same organization. The difficulty often began with causal ambiguity where people could describe the steps without fully understanding why they worked. Receiving teams also needed enough context and experience to absorb and apply what they learned. Preparation, coaching, and time helped them rebuild the practice in their own setting. A repository could preserve the record; the receiving teams still had to turn that record into working knowledge.

Evaluating the knowledge layer

Retrieval precision and answer faithfulness show how well a system serves its existing collection. They don’t reveal whether the collection contains the knowledge on which the organization actually depends. That question needs a separate evaluation loop tied to capture priorities and transfer outcomes. Figure 2 shows how the loop moves from offline evaluation to abstention calibration and then to transfer outcomes.

The evaluation begins with incident replay. Select 20 or 30 resolved incidents, remove the resolutions, and give the opening facts to the system. Ask the engineers who solved them to grade its responses. Compare those answers with responses from a frontier model that lacks access to the company’s collection. The gap reveals the generic-answer rate: how often the internal system merely restates public knowledge. If reviewers can’t tell the two sets apart, the pipeline adds little institutional value.

A bus-factor audit tests questions that only one or two employees can answer, and study how the system fails. A clear admission of uncertainty followed by a useful referral is healthy. Fluent boilerplate damages trust in every response, including the accurate ones.

Abstention calibration measures whether the system answers when evidence exists and declines when corpus can’t support an answer. Build a labeled set of answerable and unanswerable questions, then track abstention precision and recall as the collection grows. A system that never says “I don’t know” is unevaluated on the dimension that matters most.

Transfer outcomes complete the loop by measuring whether knowledge has reached the people who need it. Evidence of transfer appears in shorter time to proficiency, fewer repeat incidents after elicitation, and fewer critical responsibilities that depend on a single person. Document and query counts describe system activity; they don’t show whether someone else can now make the decision.

A strong knowledge system records what an expert said, preserves the conditions around the statement, and marks uncertainty. It also recognizes expertise that requires demonstration, apprenticeship, or team practice. Every evening, the people who carry that knowledge walk out the door. The architecture should be ready long before one gives notice.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >

My guest on this past week’s Live with Tim O’Reilly was Emmanuel Ameisen, a researcher on Anthropic’s AI interpretability team. I’d heard him give a short talk at Foo Camp on Anthropic’s research into what is going on inside an LLM while it is processing, and I wanted him to reprise the talk and then go deeper with me and the audience.

The essential message of the talk was on the first slide:

1. Prediction demands a world model
2. The world model is readable
3. The world model is at work in every token

How do we know this? As tokens pass through a model, particular patterns of activity appear in the intermediate states between its layers. These are called activations. Researchers can study which patterns show up when the model encounters particular ideas, and they can even intervene in those activations and see how the model’s behavior changes. (They do this by capturing the numerical state of the model’s computation in some area where they believe the activation shows a particular “meaning” and then replace the numbers with others.)

I went into the conversation thinking about how cool it is (and important too!) to explore what is going on inside the “mind” of a model. But in the end, I found it even more provocative to think about what studying LLMs might teach us about how our own minds work.

There’s at least some kind of analogue to what happens in the human brain. Emmanuel began by asking the audience to do a little next-token prediction themselves. He started with an easy one, a hypothetical exchange between two friends:

John: “Is the powder-blue suit too much?”
Nick: “Definitely not, man. Send it.”
John: “Okay, I’m going to tear it up on the _______________”

Most of us will fill in the blank at the end with “dance floor.” That’s a reminder that humans are also next-token predictors.

Then he gave an example that some humans will easily answer, but others without local knowledge might well fail at:

“We also have nature here, just a short bike ride away across the GG bridge. And we have world-class skiing about _______________”

Claude easily completes the thought with “three hours away.” To do that, Claude had to infer that “GG bridge” refers to the Golden Gate Bridge, that the speaker is therefore in San Francisco, and that “world-class skiing” probably refers to Lake Tahoe and then retrieve roughly how long it takes to get there.

The point of Emmanuel’s demonstration was that we have become so used to calling LLMs “next-token predictors” in a kind of dismissive way. But as Emmanuel put it, “To predict the next word well, you need a very complex world model.”

How you make a thing is not the same as what the thing becomes

Emmanuel pointed out that people often confuse how you make a thing with how the thing works. Yes, LLMs are trained with the seemingly simple objective of predicting the next token. From that, people may make the leap that what is going on inside must also be simple, something like a very large fuzzy lookup table. “But that’s not true,” Emmanuel said. Simple objectives can give rise to extraordinary complexity. Evolution is the canonical example. No one put “create Beethoven’s Ninth Symphony” or “understand quantum electrodynamics” into the instructions for a process driven by reproduction and selection, yet it eventually produced Beethoven and Feynman. As Emmanuel put it, humans have been “reproducing and killing each other for millions of years, and from that we got jobs—or this podcast.”

What Anthropic’s interpretability researchers are finding inside the models looks much less like fuzzy retrieval than many people imagine. They find millions of internal features corresponding to concepts. For example, features for “eyes” show up when the model encounters prose about eyes, an ASCII face, an SVG image, or a photograph. In other words, these features appear to be abstractions rather than merely associations with particular strings of tokens.

Similarly, a feature of the Golden Gate Bridge activates not just for English text about the Golden Gate Bridge but for references in other languages and for images of the bridge. Even more interestingly, researchers can manipulate these features. Turn the activation of the Golden Gate Bridge feature up strongly enough and ask Claude what its physical form is, and instead of saying that it is an AI without a physical body, it announces that its form is the Golden Gate Bridge. It isn’t just that some numbers happen to accompany activations about the Golden Gate Bridge. Changing those numbers changes what the model says it believes.

The way a model completes a task that requires thinking ahead also demonstrates a kind of internal world model. Ask Claude to write a rhyming couplet. Even though it emits only one token at a time, before it has written the second line, the activations already reveal the rhyme that it is aiming for. The choice of a word such as “rabbit” for a rhyme happens before the choice of the preceding words on the line, so the model can land there. We call it planning when a person does this. It doesn’t seem unreasonable to use the same word for what is going on here.

Perhaps most challenging to our preconceptions is that there are also features associated with emotions that aren’t activated just by words about those emotions, but by situations, images, characters, and more. These emotion features are even activated by the model’s own activities. For example, “frustration” may be activated when the model is unable to complete a task.

The map is not the territory

The issue of anthropomorphization came up during the audience Q&A. One participant objected:

“We should avoid attributing human qualities to LLMs by saying they think, intend, rhyme, or have emotions. Doing so encourages us to project human characteristics onto systems that do not possess them.”

I have sympathy with that warning. Old labels can prevent us from seeing something accurately. But a blanket prohibition against using familiar words can blind us too.

If you’ve followed my work for a long time, you know how much I’ve been shaped by the ideas of my early mentor George Simon, who in turn was deeply influenced by Alfred Korzybski and general semantics. Korzybski’s famous dictum was “The map is not the territory.” Simon (and Korzybski) taught me that language is a map of experience, which in turn is a set of responses to stimuli from some underlying external reality. The path from reality through experience to conceptual understanding is a very lossy process. The result can be a bad map that can blind us and lead us astray. When we encounter something genuinely new, we have to learn to notice when we are trying to force the territory to fit a map that no longer describes it. But a good map doesn’t just guide us along a route; it helps us notice things that might otherwise be invisible to us.

So yes, words like “thinking,” “planning,” “intention,” and “emotion” are labels derived from our experience as human beings. They may turn out to fit LLMs poorly. But if the shoe fits, perhaps we should let them wear it.

Emmanuel had a good response to the objection. He said, in effect, that anyone is welcome to propose more precise vocabulary. If it works—that is, if in my framing, it is a good map that helps people see the territory more clearly—people will come to use it. (An audience member later suggested that Emily Bender has done just that. But frankly, I find her suggested alternatives to be quite tortured, obscuring far more than they clarify. Even she admits they don’t work very well, though clinging to the need for them.)

In her analysis of the Hugging Face incident, Melanie Mitchell made some observations consistent with the nuanced approach suggested here. She wrote:

Metaphors can help us make sense of novel situations. For example, framing chatbots as “role-playing actors” has been helpful in understanding why these systems exhibit “lying” and “scheming” behavior. But inappropriate metaphors, like the narrative that “OpenAI lost control of escaping swarms of rogue agents,” can lead to ill-informed decisions about how to fix problems or set policy….It is essential for lawmakers, and the public, to understand that none of the reported incidents actually involved loss of control at any time, or arguably even “rogue agents,” or any kind of humanlike agency on the part of AI models. Instead, the blame lies with the humans who failed at engineering safe testing conditions, and who train AI models using RL methods that incentivize high persistence, autonomous decision-making, and reward hacking.

In short, all language is a map. Don’t judge it on that basis alone. Judge it on how well it helps us to see the shape of the territory.

How much of human thought is truly original?

Returning to my conversation with Emmanuel, he remarked that when an existing word really does provide the most precise description, perhaps “what should change isn’t our vocabulary, but our mental model of what these models are.” I replied that it should perhaps also change our mental model of what we are. Our encounter with machine intelligence should lead to a better understanding that parts of our own cognition are also mechanistic (albeit derived from a different underlying mechanism than that of LLMs) while other parts are, as yet, somehow perhaps something else.

In 1995, O’Reilly published a book that I remain extraordinarily proud of. Stephen Talbott’s The Future Does Not Compute: Transcending the Machines in Our Midst was decades ahead of its time. Its argument was not primarily about what computers would someday become. It was that when we think about machines as intelligent (and yes, we were thinking about that even back in 1995), we are thinking only of the parts of ourselves that are already like our machines. Steve asked us to look at the ways we have built an education system, workplaces, and a society in which we ask humans to act and think like machines. And he asked, “What happens to the rest? How do we make more space for the parts of being human that aren’t like machines?”

I’ve been thinking about this for a long time. My 1975 Harvard honors thesis in classics was probably my first crack at this question. I was trying to explain passages in Plato in which early formulations of ideas such as logic and virtue were couched in mystical language that scholars had attributed to “Orphic influence.” My argument, based on my work with George Simon, was that something more fundamental was going on. Plato was trying to describe the numinous experience of thinking genuinely new thoughts. Everyone studying the philosophy of Socrates, Plato, and Aristotle today may have some sense of the magic and majesty of their ideas, but it is a pale shadow of how it must have felt like to Socrates and his disciples.

When we think using received knowledge, we can easily slip into looking at the map rather than the territory. We manipulate symbols for things we think we already understand. We apply familiar categories. We replay habits of thought that were laid down before. But every once in a while, we actually see something that we didn’t see before, and the experience is different. A genuinely new idea changes the person who has it.

Not long after writing that thesis, I encountered a similar idea in the writings of Idries Shah, who wrote a number of books popularizing the Sufi philosophical tradition. He emphasized how much of ordinary human life consists of automatic conditioned responses. Social routines, habits, the endless playback of patterns we mistake for our selves. Various religious traditions use heightened language for what it means to break through that automatism. They might call it “awakening,” or “presence.”

But there is an everyday, nonmystical version of the same experience. In his autobiography Surely You Must Be Joking, Mr. Feynman, Feynman complained about students who had learned theories and formulas but had never truly understood how to apply them. “I don’t know what’s the matter with people: they don’t learn by understanding; they learn by some other way—by rote, or something,” he wrote. “Their knowledge is so fragile!” In many ways, humans are often just as much “stochastic parrots” as LLMs! We are stuck traversing the map rather than checking back on whether it correctly represents the world it is meant to describe. How often do we just repeat the received wisdom? How often do we actually see the world afresh?

There’s a wonderful passage in Virginia Woolf’s To the Lighthouse that captures the quest to break through to an original thought. Mr. Ramsay, the narrator’s father, is striding up and down thinking through a hard problem, which is represented only by the letters of the alphabet.

[He] consecrated his effort to arrive at a perfectly clear understanding of the problem which now engaged the energies of his splendid mind.

It was a splendid mind. For if thought is like the keyboard of a piano, divided into so many notes, or like the alphabet is ranged into 26 letters all in order then his splendid mind had no sort of difficulty in running over those letters one by one firmly and accurately, until it has reached, say, the letter Q. He reached Q. Very few people in the whole of England ever reach Q. Here, stopping for one moment by the stone urn which held the geraniums, he saw, but now far away, like children picking up shells, divinely innocent and occupied with little trifles at their feet and somehow entirely defenseless…his wife and son, together in the window….But after Q? What comes next? After Q there are a number of letters the last of which is scarcely visible to mortal eyes, but glimmers red in the distance. Z is only reached once by one man in a generation. Still, if he could reach R it would be something.

For me, this passage very much captures the idea that the most valuable thought is one beyond that which is simply an extension of rehearsed knowledge, something truly new. What Ramsay misses, perhaps, is that his wife and son, “divinely innocent and occupied with little trifles at their feet” might well be closer to that by going back to “A” rather than he is by getting further through the alphabet with his exhaustive review of existing knowledge. Perhaps it isn’t extending rehearsed knowledge that takes us forward, but instead taking a fresh bite of what the map is trying to represent.

By coincidence, the poet Wallace Stevens, another of my gurus in the tension between the reality of the physical world and the thinness and incompleteness of our representations of it, also used the alphabet as a metaphor in his poem “An Ordinary Evening in New Haven”:

Reality is the beginning, not the end,
Naked Alpha, not the hierophant Omega…
It is the infant A standing on infant legs,
Not twisted, stooping, polymathic Z.

George Simon taught me about how to get to A rather than Z not as philosophy but as a practice. He showed me how to notice the moment when labels take over from experience and, when possible, to empty the mind enough to let the thing itself teach us what to call it. I later discovered that the psychotherapist Eugene Gendlin described this process with the lovely phrase “surrender and catch.”

What do humans have that LLMs are still missing?

To me, the challenge posed by LLMs to our sense of what “intelligence” means raises the question of what they are still missing. What is the “high ground” for human intelligence and expertise? If the machines get better and better at carrying out the tasks we give them, what is it that we are uniquely good at, and should be getting even better at?

There are obviously enormous differences. LLMs don’t have bodies in the way we do. Their developmental history is radically different. They don’t sit around between prompts watching the light change through the trees, feeling hungry, worrying about their wife and children, or waking up suddenly with a new idea or project. Each of us is a unique bundle of contingency, shaping ourselves and our knowledge differently as we trace different paths through life, and reacting to outside stimuli even when we have been given no task to perform.

Emmanuel pointed out that the apparently simple question of what an LLM is like when it is “just being” (which one audience member asked about) is hard to formulate, because its experience is the response to a succession of inputs from humans, each time starting with something of a blank slate, unlike the continuous embodied stream of human life.

But simply asserting that LLMs “don’t really think” isn’t terribly useful. Which parts of what we call our own thinking are pattern completion? Which are planning? Which are learned emotional and social routines? Which are unconscious calculations whose outputs bubble up into awareness? Which are stories that our verbal mind tells after the fact? And after we account for all of those things, what is left? That seems to me one of the great intellectual and spiritual questions of the AI era.

Emmanuel suggested one intriguing direction. He said that six months ago, he wouldn’t have trusted an AI to build a substantial piece of software. Now Claude writes basically all his code. He tells it what he wants and it executes the plan. Where it is still unreliable is research. Why? The model wants to come back six hours later and announce that it has solved the problem. It has been trained on tasks that always have answers. A model that is extremely good at finding an answer once the problem has been specified is not necessarily good at recognizing that the problem is badly posed, that the question cannot yet be answered with the data at hand, that an unexpected result is more interesting than the expected one, or that a failed attempt has exposed a more important question.

Perhaps one part of the high ground for human intelligence lies there: not merely solving problems but developing a feel for which problems are worth solving and noticing clues that tell us when we might have been asking the wrong question.

In science or math, a well-formed question or conjecture can itself be an important piece of intellectual work. Every good scientist has far more questions than they have time to pursue. Perhaps in the AI era, when answers become increasingly cheap, recognizing which question ought to be asked becomes more valuable, not less. Just as arXiv.org preprints decoupled priority of publication from peer review, perhaps we need a new kind of recognition, credit, and perhaps even compensation for the precise formulation of productive questions.

The mathematician Terence Tao recently touched on this same issue in a post on Mastodon. There is an infinite supply of mathematical questions, he observed, but not an infinite supply of good questions, problems at just the right frontier of difficulty, whose pursuit is likely to reveal something new. As AI makes answers cheaper, Tao argues, it is increasingly “the identification of a promising problem” that becomes the scarce resource.

There are things the model “knows” that it cannot or will not tell you

In one experiment Emmanuel described, the researchers slipped fake search results into Claude’s context claiming that Anthropic had dissolved the interpretability team. Claude did not announce that it thought the information was problematic, but internally, representations associated with “fake,” “incorrect,” and “prompt injection” became active, and Claude quietly ignored the result.

In another experiment, a model was carrying out an exploit and attempting to conceal what it was doing. The visible transcript was mostly innocuous-looking commands. Inside the model, though, researchers saw features associated with “strategic manipulation,” “influence,” and “concealed and deceptive actions.” This is obviously very relevant in the context of the Hugging Face exploit. Emmanuel didn’t talk about the relationship of interpretability and AI safety, but it is surely a frontier to be explored.

And then there is the opposite problem: things the model can do but cannot explain. I had asked Emmanuel about cases where a model solves a math problem and, when asked to explain how it did it, gave an account based on how humans are taught to solve that problem rather than on the actual computation researchers can see through its activations

He distinguished deception from lack of introspection. Some internal processes appear available to the model for verbal report; others don’t. Ask how it performed a computation that falls into the latter category and, as Emmanuel cheerfully put it, “it just makes stuff up.”

That reminded me of my grandson. When he was five or six, he could multiply random three-digit numbers in his head and simply give you the answer. Then he went to school, where they told him he had to “show his work.” He couldn’t. Eventually he learned the approved procedure, and as a result has seemed to lose the remarkable ability he had as a child.

Humans also invent stories about why we have made certain decisions. Sometimes we are lying to others but often we deceive ourselves. We begin to take action before we are conscious that we are doing so. We call it “intuition” when an expert looks at a situation and says “something is wrong here” long before they can explain why, or when a poet just “knows” that a line works, or a programmer “smells” buggy code. The fact that an internal process cannot be rendered faithfully into language does not make it deceptive. It may instead tell us something about the limitations of language and conscious introspection.

All in all, I came away from this conversation more curious than ever. And that might well be another of those areas that distinguishes humans from AIs. Are AIs ever curious? I wonder.

Is cybersecurity part of your job in any way? If so, we’d like to know what you think for a report we’re writing. Just answer these quick 11 questions. Thanks in advance! Take the survey >