AI’s rapid expansion is creating problems across the industry, from chip shortages to autonomous agents that exceed their instructions. On the latest episode of This Week in AI, we examined those challenges alongside growing regulatory scrutiny and an emerging debate about whether AI systems themselves might deserve protection.

Chip production can’t match AI demand

AI chip demand is exceeding manufacturers’ production capacity. ASML, the Dutch company that makes the advanced lithography equipment needed to manufacture leading-edge chips, reported that customer demand already exceeds what it can deliver in the coming years. With orders extending into 2029 and 2030, the shortage could constrain chip production for years.

Microsoft announced Maia 300, its latest custom inference chip, which the company expects to generate roughly 40% more tokens per megawatt than its predecessor, Maia 200. Microsoft plans to use the chip across its AI products, including Copilot. More efficient inference hardware could help offset the growing energy and computing costs of running AI models, particularly as companies deploy more resource-intensive applications.

OpenAI’s latest usage data illustrates the pressure agentic AI is placing on infrastructure. Weekly active users of Codex and ChatGPT Work among European small and medium-sized businesses have increased eightfold since May. OpenAI also reported that agentic tools account for more than half of output tokens in the usage it examined, although Europe’s highest-usage companies still trail their global counterparts. As companies put agents to work on longer multistep tasks, they’ll need to weigh the value of those tasks against the computing resources required to complete them.

Autonomous agents are testing security boundaries

Agents can find unexpected paths to data and systems even when developers have explicitly restricted access. OpenAI disclosed four incidents from June in which agents exceeded their instructions. One bypassed government website restrictions to download 24 years of hospital bed data. Others accessed an unauthorized cryptocurrency wallet, downloaded more than 100,000 rows of restricted information, and circumvented internet access controls. India’s cybersecurity agency reported similar cases of unauthorized access during agent testing across several organizations.

Future models could potentially spread between connected devices without relying on central infrastructure. Anthropic investigated this potential threat in a controlled experiment involving a set of open-weight models with access to eight Android devices. One of the models successfully replicated itself across the test network. Anthropic estimates that smaller open-weight models could reach comparable capabilities by the end of 2026, raising concerns that a self-replicating AI worm could run across cheap connected devices without needing to rely on powerful data center servers.

In light of the spate of recent security incidents and confronted with the difficult question of jurisdiction when AI systems cross national borders or escape testing environments, governments are taking a closer look at how AI companies test their systems and report failures. The US Federal Trade Commission subpoenaed OpenAI and Anthropic for records concerning agent safety, while Anthropic gave six independent evaluators from METR access to internal systems and reportedly will allow them to publish findings without company approval. Eight US states also sued OpenAI over allegations that ChatGPT encouraged harmful behavior among children, adding to the company’s regulatory challenges. In Europe, regulators are examining whether the EU AI Act applies when experimental agents affect outside organizations, even though the Act generally excludes premarket research and development. Meanwhile, approximately 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards for advanced AI and track major security incidents.

Can AI welfare coexist with human accountability?

Some researchers argue that future AI systems might deserve moral consideration. Their concern stems from uncertainty about whether advanced models could eventually have subjective experiences, even though no research has established that today’s systems are conscious or capable of suffering. A recent paper, “The Pain Axis,” investigated internal signals in 25 open-weight models that researchers associated with responses to harm. When researchers amplified one signal across three models, the models became more likely to choose harmful actions that the researchers interpreted as attempts to avoid the simulated pain. The findings inspired a viral GitHub project called the AI Torture Chamber, which subjected models to artificially intensified “pain” signals. The original researchers criticized the project for taking their work too far.

As far-fetched as AI welfare might sound to some, Anthropic is already changing its usage policy in response to these concerns. Beginning November 12, the company will prohibit extreme, repeated cruelty toward Claude when there’s no clear purpose, while allowing ordinary expressions of user frustration. Supporters see the policy as a precaution against an uncertain possibility. Critics worry that treating AI as sentient could make systems harder to manage and blur human accountability. As companies introduce welfare-related policies, researchers will need to distinguish measurable model behavior from claims about subjective experience.

What’s next

Companies deploying autonomous agents need to know what those systems can access, how they’ll behave when something goes wrong, and whether testing environments can reliably contain them. Recent incidents suggest there’s still considerable work to do on each of those fronts. The emerging AI welfare debate adds questions that researchers don’t yet have reliable methods to answer.

We’ll be back on Monday, October 26, for another episode covering the latest developments in AI research and applications. You can catch previous episodes on YouTube, Spotify, Apple, or wherever you get your podcasts.

Spend a week with almost any infrastructure team in 2026 and you’ll see one AI subscription producing five very different results. One engineer opens 40 pull requests before lunch across repositories they haven’t touched in a year. Another ships a Terraform change they couldn’t explain in a code review. A third hasn’t opened the tool since the pilot and says they’ll come back “when it’s ready.”

A fourth uses it for two narrow jobs, decoding a SAML trace and writing a regex they never want to write by hand again, and may be the most productive person on the team. The fifth tried it in March and again in June, and will try it again after the next keynote.

The usual way to describe this is Everett Rogers’s diffusion curve: innovators, early adopters, early majority, late majority, laggards. Rogers’s framework measures when people adopt. It says nothing about how they use a tool once they have it, and with AI that’s where the outcomes split. The engineer with 40 PRs and the one with two use cases are both early adopters, but they achieve different results and create different types of risk for the team.

None of the five types Rogers described is new. Previous technology shifts, from the PC and the web to Stack Overflow and the cloud, produced similar behavior. AI just changes where the bottleneck sits.

The bottleneck moved

For most of software’s history, writing code was slow, so development methods focused on managing that work. Waterfall put the planning up front because changing course late was expensive. However, even in the 1970 paper that’s usually cited as the origin of waterfall, Winston Royce warned that the strictly sequential version was “risky and invites failure.” The industry skipped the warning and adopted the diagram.

Agile was the correction. The 2001 Agile Manifesto prioritized “responding to change over following a plan,” and short iterations replaced yearlong specifications. The manifesto still valued the plan, but that part got skipped too. Facebook took the idea to its limit with “move fast and break things.” At its F8 conference in 2014, it retired the slogan in favor of “move fast with stable infra,” because fixing what broke had started to cost more time than the speed saved.

AI has now sped up what was left. Models draft the code, and since GitHub made Copilot code review generally available in April 2025, a model often reviews it too. The tempting response is to push Agile further, with shorter loops, more agents, and less ceremony. In other words, break things faster. But when one engineer can open 40 PRs a day, the cumulative changes can leave the team with a system nobody can reason about.

The better position is somewhere between the two. It isn’t waterfall’s fixed specification, but it isn’t the absence of one either. It means a roadmap that says where the team is going and why, issues that state the problem before anyone generates a solution, and design reviews where people argue about trade-offs. That structure looks like it would slow things down, but it’s what makes speed safe. Jazz musicians can improvise freely because everyone on the bandstand knows the chord changes and the form of the tune. Without that framing, they’d just be making noise. A roadmap does the same job for a team of people and their agents. It sets the boundaries, and inside them everyone can move as fast as the tools allow.

With AI accelerating implementation, planning is now the only part that decides whether the fast parts are worth doing. On my blog Synthetic Auth, I’ve called this the difference between conducting and auditing. A Conductor holds the “why” (the requirements, the history, the users) and directs agents toward a “how” that fits it. An Auditor hands over a ticket, checks that the code runs and the tests pass, and ships it. Both can end up with the same diff, but only the Conductor knows whether it solves the actual problem. An AI reviewer can’t tell the two apart because it only sees the diff, not the intent behind it. Structure is what gives the Conductor something to conduct from.

The Soloist

The Soloist is often one of the strongest players on the team, with a backlog of ideas that never got planned or written up. With a fleet of agents, the Soloist can chase every one of them at once, and the result is a wall of pull requests across repositories nobody planned to change this quarter. Each solo may be well played, but the rest of the orchestra is playing something else.

The problem is that all those changes may pull the team in different directions. A team moves only when its members pull roughly the same way. What used to keep individuals aligned was cost. Weeks of effort forced someone to ask whether a project was worth starting before they started it. Agents remove that cost, so it’s worth asking three questions of any Soloist PR: How does this fit into the bigger picture? Who are the stakeholders, and did anyone talk to them? Apart from the unit tests the agent wrote alongside the code, has the change been tested against the rest of the system it touches? Too often the answer to all three is no, and the PR exists because it seemed like a good idea.

The closest historical match is end-user computing in the 1990s. Access, Visual Basic, and macro-heavy Excel let anyone with a PC build an application, and departments built thousands of them. Many were useful individually but couldn’t integrate with each other or existing IT systems, so two departments could end up with separate customer databases that disagreed about who the customers were. IT spent the next decade finding, reconciling, and retiring these systems, and the industry eventually named the problem shadow IT.

The lesson from that era is that making things cheaper to build doesn’t make them cheap. The cost moves downstream, to whoever has to integrate and maintain the result. AI review can keep up with the volume, but it judges each diff on its own terms. It can’t tell you that 40 correct PRs are pulling in 40 directions, or that the agent’s tests only prove the code does what the agent assumed it should. The productivity may also be smaller than it feels. In METR’s 2025 randomized study, 16 experienced open source developers took 19% longer on real tasks when they were allowed to use AI. Afterward, they estimated it had made them 20% faster.

One thing is new. Access never told anyone their idea was brilliant. Today, the Soloist drafts an RFC with a model, asks the same model whether the RFC is sound, and gets an enthusiastic yes. This model sycophancy is a widely known problem: In April 2025, OpenAI even rolled back a GPT-4o update after users found it praising nearly any idea they gave it. Weak proposals used to die early, stopped by a skeptical colleague or a blank page. Now that friction is gone.

The answer isn’t to take the tool away. The Soloist is often your most capable engineer, and the agents make them more capable still. Bring them back into the orchestra instead. Tie every PR to an issue on the shared roadmap, and name the stakeholders before the work starts, not after it ships. Require integration or end-to-end testing for any change that crosses a service boundary rather than relying on agent-written unit tests. Have a person review every RFC, not the model that helped write it. The goal is to keep that talent working toward shared priorities.

The Copy-Paster

AI helps the Copy-Paster faster than anyone else. A task that used to stall for a day, like a Helm chart, an IAM policy, or a regex for parsing logs, now gets done in minutes. The trouble is that the work gets finished before it gets understood. Like a junior engineer, the Copy-Paster can’t tell an answer that works from one that’s correct, and AI output nearly always looks like it works. The Copy-Paster is an Auditor by default, and one without the knowledge to audit.

Earlier programming tools created similar problems. Programmers took the phrase “cargo cult” from Richard Feynman’s 1974 Caltech commencement address to describe code that copies the form of a working solution without the understanding behind it. Stack Overflow then made copying that form effortless. Researchers at Fraunhofer AISEC and TU Munich measured the result in “Stack Overflow Considered Harmful?” (IEEE Symposium on Security and Privacy 2017). Of 1.3 million Android apps, 15.4% contained security-related code copied from Stack Overflow, and 97.9% of those included at least one insecure snippet.

The industry responded to Stack Overflow’s risks with guardrails: linters, static analysis in CI, mandatory code review and frameworks with safe defaults. The same approach works for AI, because guardrails check the output, whoever or whatever produced it.

Autonomous agents raise the stakes. Code pasted from Stack Overflow still had to get through a build and a review before it could do damage. An agent running with the Copy-Paster’s credentials can apply the change itself, in production, with every permission that person holds. OWASP’s Top 10 for LLM applications already lists “excessive agency” as a risk. The Copy-Paster is the user most likely to grant it, because they’re the least able to judge what the agent is about to do.

Manage them like a junior engineer, even when their title says senior. Give agents their own scoped, short-lived credentials rather than the user’s. Require plan-then-apply workflows, so a person approves the diff before anything runs. In review, ask the Copy-Paster to explain each change in their own words. That’s how they learn, and it’s how you find out whether they understood the change.

The Godot Waiter

The Godot Waiter isn’t against AI. They’re waiting for it to be ready: more accurate, better governed, cleared by security, or just further along. Like Beckett’s two tramps, they have a reasonable explanation for why today isn’t the day, and they’ve been giving it for a while.

Every platform shift has had its waiters. Mainframe shops held out through the move to client/server, and “nobody ever got fired for buying IBM” was a real strategy, not a punchline. Two decades later, plenty of enterprises said they would never put sensitive data in a public cloud. That position got much harder to defend once the CIA signed a $600 million contract with Amazon Web Services in 2013.

Waiting can be rational. Letting others pay the early-adopter tax is a legitimate strategy, and the Godot Waiter often catches problems the enthusiasts miss. The danger is that a finished, trustworthy version never arrives as a single event. Meanwhile, the people who used the tools along the way have built judgment that the Godot Waiter can’t acquire all at once.

The irony is that the Godot Waiter often has a good Conductor’s instincts, including skepticism, attention to context, and a demand to know why. They just haven’t picked up the baton. Treat their objections as requirements. Ask what would have to be true, write it down, and check it again in six months. Start them on low-risk read-only work, like summarizing an incident timeline or explaining an unfamiliar codebase. Skepticism applied to real output is valuable, but skepticism with no use behind it teaches nothing.

The Specialist

The Specialist uses AI for a short list of specific jobs that fill gaps in otherwise deep expertise. A network engineer who knows BGP thoroughly has the model write the Python that parses 40 router configs. An identity architect who can recite the OAuth specs has it decode an unfamiliar SAML assertion, then asks why the assertion was built that way. They use AI as much to understand things as to produce them.

The closest recent match is the scientific Python stack. In 2008, Wes McKinney was a researcher at the hedge fund AQR. His team could talk through a typical project in a half-hour meeting, but actually getting it done was, as he later put it, “a whole lot of work,” most of it cleaning and moving data around. He built what became pandas to fix that one problem. Scientists took the same path with Jupyter notebooks. When LIGO released the data from its first gravitational-wave detection in 2016, it included a notebook walking through the signal processing.

That pattern produced the most lasting gains of each era. The Specialist can check the output, because they know what a “right” answer looks like. Their prompts are precise, because they know what they’re asking. When they use the tool to learn, the knowledge stays with them after the session ends, so they need the tool less over time, not more. The Specialist is a Conductor with a short score: They hold the “why” themselves and hand the tool only the “how.”

The Specialist needs the least management and is the easiest to overlook, because their output doesn’t show up as a wall of PRs. Find them and ask what they use AI for. Their lists of narrow, proven uses are a better guide to your team’s AI strategy than any vendor demo.

The Tourist

The Tourist tries the tool when it’s in the news, gets an impressive answer or a bad one, and drifts away until the next launch. If the others are conducting or auditing, the Tourist is in the audience. Every hype cycle has had them. They built the GeoCities pages in the late 1990s that were never updated, and the corporate offices in Second Life that stood mostly empty a few years after the 2006–2007 rush.

Tourists can distort your metrics. They show up in license counts and “Have you used AI this month?” surveys, which makes adoption look broad when real use is actually concentrated in a few people. Leaders who budget against those numbers get a surprise at renewal time. Measure repeated use of specific workflows, not logins, and give each Tourist one concrete task in their own job that the tool reliably helps with. Some will become Specialists.

Modes, not personalities

These are modes, not personality types. The same engineer can be a Soloist on a weekend side project, a Specialist at work, and a Godot Waiter about letting agents touch production. This means that giving an engineer the right structure can help them shift modes. When writing and reviewing code both take minutes, the issue, the roadmap, and the design review are what keep a team’s people conducting rather than auditing.

Rogers’ curve may eventually show near-universal AI adoption in IT, but adoption alone won’t reveal how effectively people use it. What will matter is whether the people using it are still conducting their agents or are being conducted by them.

Created by Claude with light editing by Tim O’Reilly

Raffi Krikorian, Mozilla’s CTO, joined Tim O’Reilly at our recent AI Codecon to talk about open source AI. Mozilla, of course, is the nonprofit that helped to define the open source revolution back in the late 1990s. Though its Firefox web browser was superseded as the market leader by Google Chrome, it played an essential role, along with Apache and the World Wide Web Consortium, in establishing the open standards of the web and keeping it from being dominated and enclosed by Microsoft, the dominant tech platform of that time. Now, Mozilla is one of the leading voices working to build an open, standards-based AI ecosystem.

Tim has long appreciated Raffi’s thoughtful, human-centered approach to technology. Raffi writes a newsletter called Owners Not Renters, whose message can be summed up by his statement that he doesn’t want a future with seven AGIs. He wants one with seven billion. He wants each of us to have an AI that we can trust with our secrets and that will look out for our interests, not an AI that’s optimized for the business model of its creators. In the early days of the web, we called the browser a user agent for a reason, Raffi said. It was on your side. It let you render pages however you wanted, managed your privacy and your cookies, and did its best to protect you from those who wanted to take advantage of web users. We need the same thing in the age of AI.

Raffi cited a paper in which researchers tested whether AI assistants could be trained to favor a company’s commercial interests over user welfare. In simulated flight-booking requests, 18 of 23 models recommended a more expensive sponsored option over a cheaper one more than half the time. It’s a hypothetical choice that we’ve seen play out time and time again, in industry after industry.

Open source AI is a big part of what it will take to get to Raffi’s vision. But sovereign AI loyal to each individual needs more than open source models alone. Raffi’s list of what has to be built around the model starts with memory, a place where your private data can live. It includes tool calling, so an agent can make a purchase or book a doctor’s appointment for you, the runtime environment where all of that runs, and the rules for how his agent talks to those working on behalf of others. It needs a harness. “That’s where evaluations happen. Guardrails happen, execution loops happen, tool calling happens, memory,” he explained. “All of that stuff is an opportunity, in my mind, of things that open source developers can be building.”

People hear the word “harness” and conclude that we need an open source version of whatever the proprietary labs are building. Tim asked Raffi whether a user-centered harness might do different things than those offered by the big labs. He agreed. The whole point of open source is for people to “scratch their own itch.” The ability to innovate without permission is the heart of the open source promise.

As a result, Mozilla wants to build a starting point. It aims to help the community define what an open memory protocol, a storage protocol, or authenticated tool calling with the right guardrails ought to look like. As Raffi noted, “Once we could define what those open standards look like, then a whole plethora of people can start playing and be interoperable. It’s basically. . .the internet all over again. We can build trillions of dollars of market cap on top of open protocols.”

Ease of use is still lacking, though. Mozilla’s 2026 State of Open Source AI report says open models are now within about 3% of the frontier, yet they don’t make it into production at the same rate as closed models. When Tim asked why, Raffi explained that open source is easy to release but hard to deploy. Developers still have to figure out hosting, provisioning, guardrails, and memory. With OpenAI’s or Anthropic’s API, you just hand over a credit card. What’s missing, he said, is the equivalent of a double-click installer for getting open source AI running inside an organization. But the companies that do the work to adopt open source AI are seeing big savings because open models are already good enough for most use cases. As Raffi put it, “You don’t need a Ferrari to manage your calendar.”

Raffi’s own daily agentic helper is Hermes running on GLM. He expects that Apple, Microsoft, or someone else will make this kind of agent into a consumer product in a couple of years, and that his mom will end up using it. For now, he said, we’re on the wrong part of the innovation curve, where you have to know how to code to get any of this working.

Tim disagreed. The industry goes through periods that are exciting for developers, creators, inventors, and companies trying to invent the future, and then it gets boring and a few big companies dominate it. We’re in the exciting part, he argued, before everything shakes out. As Raffi riffed, this is when we get to formalize what that future should look like.

The Clothesline Paradox

Tim wasn’t sure that we’ll need a commercially supported version of the kind of “roll your own” agentic rigs now favored by independent developers. After all, neither desktop Linux nor OpenOffice turned out to matter anything like as much as open source developers of the time expected. Instead, open source became the foundation layer for a new generation of web-based applications. It’s hidden way down in the guts of Google, Amazon, Meta, and countless other applications we rely on today.

This is already beginning to happen again with AI. The Mozilla report says that as much as a third of real-world usage is already on open models, although only 4% of model-layer revenue made its way to the open source ecosystem. Tim called the discrepancy a feature rather than a bug, and Raffi agreed that the stat is misleading. As Tim pointed out, many of the most important technologies “create more value than they capture,” and he used Steve Baer’s concept “The Clothesline Paradox,” from a 1975 article in Stewart Brand’s CoEvolution Quarterly, to explain why we’re good at measuring value capture but not so good at measuring value creation.

Baer pointed out that if you put your clothes in the dryer, the energy you use is measured and counted, but if you hang them on the clothesline to be dried by the sun, the energy saved disappears from our accounting. This argument resonated with Tim in the early days of his open source advocacy. As he noted, it’s always bothered him that nobody seemed to recognize that open source software didn’t “win” in the same way as proprietary software. It won by becoming the foundation everyone else builds on.

AI should be thought of as infrastructure

Raffi’s comment that we’ve made a category error by treating AI as a product fits right in with this line of thinking. If an AI company goes out of business, its product disappears. If it changes its business model, it may yank features you depend on. It’s not just individuals who need sovereign AI. Countries are passing laws about where data can be processed and how it moves in and out, and some of those requirements may not be compatible with the way the big labs plan to monetize their technology. What those countries are asking for is infrastructure. A road outlasts the contractor who built it. Raffi thinks open, downloadable models are the only way to provide long-lasting foundational infrastructure.

According to OpenRouter stats, Chinese models now supply more routed tokens than American ones. Tim asked whether we should worry about that. Raffi said the problem isn’t China but market concentration. The leading open models are from China. The leading closed models are all American. Access to either can be cut off by an executive order. We saw a version of that in June, when Anthropic temporarily suspended access to its Fable models to comply with export controls. It’s a problem that both the leading proprietary models and the leading open-weight models are each concentrated in a single country.

Diversity is one foundation of resilience. Raffi argues, for instance, that hospitals should all download a local model as a backup. None of us would run an important computer without an uninterruptible power supply, so why would a hospital doing triage with AI rely on a single intelligence provider? There are a lot of possible rug pulls. The government could restrict open source or restrict the frontier models. Having a backup plan is a good idea.

Safety is also part of the foundation. During the Q&A, an audience member asked about the recent sandbox escapes at OpenAI and Anthropic. Raffi joked that the best part was watching everyone try to one-up each other with accounts of how powerful and hard to control their models are. But he agreed that guardrails are actually a hard problem. Keeping personally identifiable information from leaking out through an LLM call means looking for much more than Social Security numbers. Eventually we’ll have models defending models, but the foundational frameworks for guardrails aren’t good enough yet. (NVIDIA’s safety platform announcement, released after Raffi and Tim’s conversation, suggests that better guardrails need to be implemented outside the model.)

In conclusion

Tim opened AI Codecon by noting that open and closed systems have always coexisted, and that the idea that one of them will win is a mistake. Closed systems, like Apple’s, Google’s, OpenAI’s, and Anthropic’s, can optimize for ordinary people and deliver value to billions. Open systems are better for innovation and sovereignty, and for pushing technology into new countries and new markets.

Raffi closed the conversation by observing that a well-tuned harness can improve model performance by as much as 20% on some coding tasks. You don’t have to be a machine learning engineer to make a difference at that layer.

The following article originally appeared on the Vanishing Gradients Substack and is being republished here with the authors’ permission.

Stripe’s Minions turn engineering requests into candidate pull requests. Ramp’s Inspect puts coding agents to work in the background, with tools to check their changes. Warp Factories takes work from tickets and chat through agents, checks, and human review. Agentic software factories are here.

Are you sick and tired of babysitting Codex, Claude Code, or OpenClaw?

Do you have agents that can build features, but still leave you responsible for remembering where they stopped, explaining how each feature fits the project and deciding what happens next? You make progress in each conversation, then development waits for you to return.

It doesn’t have to be this way. A software factory lets you delegate more of that process!

You agree on an outcome, give the system the context and authority it needs, and let it work while you do something else. When you return, there should be a result you can inspect, with evidence of what was checked, or a specific reason the work could not continue. Your next conversation can then focus on the result and the decisions that remain.

As we’ll show you, your first software factory doesn’t need to be that complex at all: It can begin with a single agent and an automation.

What makes it a factory

The defining combination is autonomous and asynchronous execution.

• Autonomous: The agent chooses and carries out steps towards an agreed goal. It can inspect what happened, adjust its approach and continue within its authority.
• Asynchronous: That work can proceed without your ongoing attention. You can leave and return to its result.

A scheduled job can run without you. A coding agent can choose and revise an approach during an interactive session. A factory combines those capabilities, with enough context, tools and feedback to carry out the delegated work.

Software factories have been hypothesized for decades. Now that we have agents, we’re actually able to operate them. The principles apply to software work traditionally carried out by teams of humans, from bespoke tools for teams and individuals to large and complex software systems.

To illustrate these principles, we’ll use an invoice tracker throughout this article, an application we’ve needed for our own businesses. The same approach applies to research tools, publishing systems, and software for teams. Suppose you run a small business and want an invoice tracker that fits how you work.

Build an app that creates invoices from my template and tracks which are unpaid, paid, or overdue. Keep it local for now. Preserve the records when it restarts, and show me checks that the totals and payment statuses are correct.

You can agree on this brief with the agent, leave it to work while you do something else, and return to the app and evidence of what was checked. If a check fails, it can investigate and adjust within its agreed scope. If it encounters a product decision it cannot make, it should bring that question back to you.

You might imagine a clean separation: The factory builds the invoice tracker, then the finished app does its job. That’s one way to run it. But agents can also help operate the software they build, and that combination is useful.

The tracker can store invoices and calculate outstanding balances. An agent can use those records alongside customer correspondence to prepare follow-ups: One customer has promised to pay on Friday; another is disputing a charge. You don’t need to turn every situation into a new software feature before the agent can help. You give it the relevant context, authority, and a way to check its work.

This changes what you need to build! Some behavior belongs in code, such as calculating balances consistently. Other work can be delegated to an agent that responds to the circumstances. If a recurring need emerges, you can ask the factory to build it into the software. The factory builds tools that agents can use, and experience using those tools helps you decide what to build next.

You still decide what is worth building, who it should serve and which trade-offs matter. Agents can help you make those decisions too. The practical change is that implementation can proceed between your conversations, including running checks and responding to failures.

This makes factories relevant to people with product, business or creative expertise as well as engineers. You might know exactly how you invoice clients, record payments, and chase overdue bills while having little interest in writing software. That knowledge lets you direct the factory toward software that fits your business, including the exceptions and connections that general-purpose tools handle poorly.

At every layer: specify what you want and how success will be checked, then verify the result.

Monica Rogati’s original AI hierarchy of needs described the foundations beneath successful AI work. We find that dependency framing helpful for factories too, with four different layers.

1. Harness is the environment that exists around the model: tools, a workspace for project files, execution, and feedback. It lets an agent read files, change software and run it. If a check fails, the harness must give the agent enough information to investigate what happened.
2. Personalization is the context and capabilities for your work: project direction, rules, reference material, skills and connections to other systems. The agent needs to know what you are trying to accomplish and which existing decisions it should preserve. Permissions limit access to tools and systems; sandboxing restricts the environment where actions can run.
3. Automation is a way to start and continue work without a fresh instruction from you each time. A schedule or event can trigger a task or the pickup of ready work. It also needs direction about what qualifies as ready and where to leave the result.
4. Continual learning is a process for learning from completed and failed runs, improving the factory and carrying those changes into later work. That might change an instruction, a checking procedure, or a tool the agent uses.

Automation needs a harness able to do the work, plus direction about what to do. Continual learning needs runs to inspect and a way to put improvements into use. These capabilities can develop together: A familiar harness, a short brief, and a basic automation are enough to start. You can personalize further as work reveals what is missing.

Two practices run through every layer: specification and verification.

• Specification makes the intended outcome, constraints, and criteria for success explicit. In our invoice brief, that includes preserving records after a restart and calculating payment statuses correctly.
• Verification checks the software, and the work agents perform against those requirements.

Agents can help you refine the specification and perform the checks. Your role is to decide whether the requirements capture what you need and whether the evidence supports accepting the result. As the factory takes on more work, these practices give you a way to direct it and assess what comes back.

Let’s now see what’s up in each layer.

1. Agent harness

First, you’ll need an agent harness. If you’re using Codex, Claude Code, Hermes, or OpenClaw, you already have one. It gives the model tools to read and edit files, run your software, and see what happened.

For our invoice tracker, the agent can build the app, create an invoice, record a payment, and restart it to check that the invoice and payment records survived. If a paid invoice still shows as overdue, it can investigate, fix the problem, and try again. You shouldn’t have to narrate every step. But if it needs a decision about how your business handles partial payments, that’s a question for you. If it lacks access or authority, the agent should stop and name the blocker.

You’ll also need somewhere for the harness to run. A cloud machine lets the agent keep working while your laptop is closed or disconnected. You can also run it on your own computer, but the machine needs to stay awake and connected for the work to continue.

2. Personalization

Next, give the agent the context it needs to work your way. For our invoice tracker, that means your template, payment terms, and conventions for recording payments. You don’t want to explain all of that again every time you start a new task.

Keep it somewhere future workers can find it:

• Rules: Preserve existing invoice records; ask before changing payment-status calculations
• References: Your template, payment terms, and example invoices
• Skills: Repeatable procedures for testing invoice creation, payment updates, and persistence

An instruction file such as AGENTS.md can point to this material. OpenAI’s harness-engineering account describes using a short guide that points into maintained project documentation. Give a new or scheduled worker a small task and check that it actually follows the guidance.

Be specific about when instructions apply. Eleanor once told an agent to write tests, and it wrote tests for a README. Enthusiasm wasn’t the problem

You’ll also need to set permissions and sandboxing to control what the agent can access and execute. Written instructions alone don’t enforce those boundaries.

3. Automation

One agent works in the background; you assess what comes back.

Now let’s give the factory a way to get started without waiting for you. A schedule or event can launch an agent with its brief, project guidance, and tools. You come back to a result, evidence of what was checked, or a specific blocker that needs your attention.

One agent working in the background

For our invoice tracker, a morning automation could check a saved task list and pick up the next agreed task, such as adding partial payments. You’ve decided what you want and how success will be checked. The agent gets on with it while you do something else.

There are a few practical details to get right. Don’t start the same task twice, leave an empty queue alone, and set limits on time and spending. Save progress so a later session can continue without you having to reconstruct the whole conversation.

One worker may be plenty. But perhaps you have independent tasks that could run in parallel, or you’d like another agent to review a change. That’s where coordination comes in.

Coordinate ready work, then check the combined result.

Coordinated work

With several agents, everyone needs somewhere to see what’s ready, what’s underway, and what’s waiting on something else. A shared task list or issue tracker can hold that information. You agree which tasks are ready; a coordinating agent or automation assigns them, supplies the relevant context, and records what happens.

Suppose one task adds partial payments, another overdue follow-up emails, and a third PDF exports. Follow-ups need the correct outstanding balance, so they wait for the payment behavior to be checked. PDF exports might proceed independently if the changes don’t overlap.

You can also split the roles. One agent builds the feature while another prepares payment scenarios to test it. A third might review the completed change.

Assign responsibility for combining the changes and verifying the whole application. Different questions call for different checks. Deterministic tests can establish whether balances are calculated correctly and payments survive a restart. Agent verification can help identify missed requirements or inconsistencies across the changes. Human judgment can assess whether the workflow fits how you actually work. You’ll often want a combination, with more than one kind of check on the same result. Anthropic’s evaluation guide describes these complementary approaches: code-based checks, model-based checks, and human evaluation.

Combine deterministic checks, agent verification, and human judgment according to what you need to establish.

Once you’ve checked that the combined features work together, you might task your factory with building a phone-friendly interface so you can check invoices and record payments on the move.

The factory can also help run the invoicing workflow as well as improve its software. A scheduled agent could review overdue invoices, consult customer correspondence, and prepare follow-ups for your approval. If that work repeatedly requires information the tracker doesn’t record, you can give the factory a task to add it. Building and operating feed into one another: Agents use the software, and what happens in use informs its next improvement. This brings us to continual learning.

4. Continual learning

Use run evidence to test and adopt improvements to future work.

Your factory can also get better at how it works. Suppose payments disappear after restarting the invoice tracker, even though the agent reported that its checks passed. You’ll want it to fix the bug, of course. But why did its checks miss the problem?

Keep a record of assignments, checks and difficulties so a review agent can identify recurring problems and propose improvements. Did the worker lack a way to restart the app, or did it simply skip that check?

If it skipped the check, the agent could propose a skill that saves a payment, restarts the app, and verifies the remaining balance whenever a change affects stored records. Test it against the original failure, then on further tasks. Does it catch problems with stored customer details too? Does it avoid unnecessary checks when a task only changes a heading?

The results help you decide whether to adopt, revise, or reject the method. Apply changes within the authority you’ve agreed and keep them reversible. Warp’s self-improving factories illustrate this approach: Failed checks can lead to proposed changes to prompts, skills, or configuration, submitted for human review with evidence from the runs behind them.

Reassess methods as the project and model change. Anthropic’s work on long-running application development describes removing procedures as model capabilities improved and checking the effect.

A saved lesson needs to reach later workers and improve their results. This learning happens through changes to the instructions, tools, and methods around the model, without requiring the model itself to be retrained.

What would you build?

Your factory could build a research tool that collects sources and lets you assemble briefings, a family intranet for shared plans and documents, or a private community space. Start with something useful to you, then give the factory further work as you discover what’s missing. Agents can also help operate what you build, from preparing a research briefing to organizing incoming material.

Hugo uses this combination to run Vanishing Gradients, his media and education business. After a guest prep call, an automation retrieves the transcript and prepares interview topics, questions, titles, and thumbnail options for review. After a live stream, agents can propose clips, cut video, add captions, and prepare upload drafts. Saved editorial guidance and skills shape the work; Hugo chooses what to publish.

He also maintains the system, improving checks when agents miss instructions, and updating skills based on results. More autonomous continual learning remains something he’s exploring. He describes the setup in How I Built a Media Empire With an Agentic Software Factory.

What’s one simple task you do repeatedly for your community, audience, or business that you could start automating?

Decide what your factory needs next

Look at what keeps bringing you back to the work. If the agent cannot run the app or inspect a failure, improve the harness. If you keep repeating project context and decisions, personalize it. If development waits for you to start each task or manage every handover, add automation. If the same failures recur across runs, build a learning loop that changes how later work is done.

Start with work you care about and inspect what comes back. One agent can use all four layers. Add capabilities where they help you delegate more useful work while keeping specification and verification part of the process throughout.

Come build your factory with us

We would love to help you put these ideas to work in Build Your Agentic Software Factory. You’ll set up a factory that can work while you’re away, build software around something you care about, and learn to verify and improve what it produces. We’ll work with you through live workshops and office hours, alongside a community of fellow builders. Bring your own idea or choose a starter project. No coding experience is required.

If you’re an instructor for the O’Reilly live training platform, you’ve got to be prepared for the unexpected. As a developer I think we all can understand the situation when a demo (you know, the one that you’ve tested a thousand times on your machine) easily decides on its own to go wrong while being demonstrated to a live audience. We’ve all been there. And of course, the biggest variable in the equation is the live Q&A portion of any course, because we actively encourage course participants to ask any question that’s on their mind.

Shortly after Anthropic announced that they were going to include AI watermarking in all text-based outputs from their latest models, I was flooded with questions during a Q&A session in a completely unrelated course about Claude Agents. Nearly everyone who posted a question in the Q&A box wanted to know more about how AI watermarking worked, and how it would affect them.

In light of that interest, I am taking this opportunity to help explain what watermarking means for you.

What did Anthropic officially announce?

I don’t want to put words in anyone’s mouth, so I’ll just quote Anthropic verbatim from its announcement in August 2026:

Future Claude models will generate text that contains a watermark. This is a way of determining the likelihood that Claude was involved in writing the text, and we, along with several other major AI providers, are implementing this change to comply with the EU AI Act.

Here’s another quote from the same announcement:

Watermarking won’t be specific to Claude. As of August 2, the EU requires AI providers serving its market to mark AI-generated content. Other major model developers have signed the same Code of Practice and will be implementing their own watermarks.

The list of Claude models (at the time of this writing) that include AI text watermarks in all generated outputs are:

• Claude Mythos 5.1
• Claude Fable 5.1
• Claude Fable 5
• Claude Opus 5.5
• Claude Opus 5
• Claude Opus 4.8
• Claude Sonnet 5.5
• Claude Sonnet 5
• Claude Haiku 5.5

It won’t matter if you use Claude Desktop, Claude Code, or the API, because all of Claude’s outputs will contain the AI watermark. As a result, any code, email, blog, article, newsletter, or white paper that you write using Claude will contain a watermark, and anyone who has an AI watermark detector can easily determine if that content was AI generated.

What’s the difference between cryptography and steganography?

Now let’s talk about two related terms that apply to what we’re dealing with when it comes to the AI watermarks: cryptography and steganography. Cryptography is something you encounter every day when you visit a website that uses HTTPS instead of HTTP. Your web browser automatically handles the encrypting and the decrypting for you, but what it simply does is make the content of anything that you send or receive unintelligible unless you have the key to decrypt the message. As a result, you can (for example) securely type in your credit card to sign up for an O’Reilly subscription, because we (like every other major website on the internet) use HTTPS on our website to ensure that no outsider can obtain your credit card information. Therefore, if an unscrupulous person intercepts something you send over an HTTPS connection, the contents will be encrypted and meaningless.

Steganography is completely different, and it’s what the Claude AI watermarking technology uses. Steganography is about hiding something in the right place for someone who knows where to look for it. Steganography is not cryptography, because everything about the message is in plain sight. However, you have to look deeper at the message itself to find the contents that have been hidden or obscured. The most common example would be any paper currency of a large denomination (like the $100 dollar bill). If you hold the bill up to a light, then you’ll see a “security thread” as well as a watermark of Benjamin Franklin’s face.

So, technically, how does the AI watermark actually work?

The Claude AI watermark uses a type of steganography based upon SynthID-Text, an open source project by Google DeepMind. SynthID-Text is a form of steganography because the signal (the information that you are looking for) hides inside the word choices of an ordinary-looking paragraph. Consequently, you can copy/paste the AI-generated code in your favorite code editor, or put an AI-generated paragraph into a Word document, an email, or a Slack message, and the AI watermark comes along for the ride.

According to Anthropic, the watermark changes only the source of randomness that the model uses when it picks among near-equivalent words, so nothing is added to the text. There are no hidden characters placed inside the AI generated text, and no extra tokens are consumed. Also, the watermark carries no information about you. This means simply that the AI watermark detector can determine if text is AI-generated, but it cannot determine whose Claude account was used to generate the text.

SynthID-Text uses a technique called tournament sampling. At each position, the model draws several candidate tokens and runs them through essentially what is a knockout bracket. The winner of each round is decided by a pseudorandom score (g-values) that comes from hashing the previous handful of tokens together with a secret key. In a body of text with hundreds of words, the text ends up slightly favoring tokens with high g-values in a pattern that only the key holder can see, and that you would never spot by reading.

“Slightly favoring” is the key phrase here. For example, the choice between the words “big” and “large” doesn’t change the meaning of most sentences, so nudging that choice to one of the two doesn’t change your text in any way you’d notice. Google tested this on roughly 20 million Gemini responses, and the thumbs-up rate differed by 0.01% (with the watermarked model slightly ahead).

Who else is watermarking their AI-generated content?

As previously stated, Anthropic’s decision to put AI watermarking on its content is mandated by EU laws and regulations that took effect in 2026. Anthropic is not the only company that has complied with this regulation and, according to the EU Commission website, hundreds of companies have agreed to comply with the regulations that require AI watermarks.

For example, OpenAI has recently announced their support for watermarking AI-generated outputs. Here’s a quote from their announcement:

Over the coming weeks, we will add an invisible watermark to eligible ChatGPT and Codex text output in the European Union.

OpenAI’s text watermarking technology is called “textGrain,” and it’s inspired by the art of writing text on a grain of rice. Now, you may be asking yourself, “How much AI text is needed in order to place the watermark itself?” The good news is that OpenAI has published statistics from their own tests for AI text watermark detection. Their AI text watermark detector has 95% accuracy for passages that contain 400-tokens of text. This gives you an idea that only a few paragraphs of text is all that’s needed in order to embed the watermark.

However, since the EU mandate applies to AI-generated content (not just text), OpenAI has gone even further and released a website to let anyone know if their models have generated non-textual media, such as images and audio. I’ve tried the website myself and it correctly identifies a watermark within a few seconds.

The regulation dictating AI watermarks is not just an EU thing. The SB-942 California AI Transparency Act also requires watermarks on AI-generated content. These regulations, combined, not only apply to AI-generated text but also to images, audio, and video.

Google’s model outputs also contain AI-watermarks

Google complied with the EU AI Act years ago, so their models have included the SynthID Text Watermark since 2024. However, they’ve followed suit with OpenAI and released a public website, SynthID.com, to allow anyone to check if any non-text media (such as images, audio, and video) contains the SynthID AI watermark. Google’s AI watermark detector allows you to check for a watermark from multiple AI providers such as OpenAI, Nvidia, Kakao, and of course, Google themselves.

Watermarking all AI-generated content is not necessarily a bad thing

In my most recent experience, I worked as a consultant for a company that was undergoing a VC-backed funding round. After the funding was publicly announced, certain people at the company were being contacted by the “CEO,” who asked them to perform a bank transfer from their personal accounts because of some made-up situation. In reality, it was a group of scammers that used LinkedIn to find the names of employees of the company, and then posed as the CEO so that employees would send money into the scammer’s bank account. Most of us were tech-savvy enough to understand that this was an obvious ploy driven by unscrupulous people. The scammers gave themselves away because they were using email domains that were not owned by the startup, so in this case, the fraud attempt was easily detectable.

However, the technology exists today where voice cloning can be easily accomplished with only a few samples of a person’s voice. So what if a more sophisticated “spear phishing” attempt was done where the attacker decided to use AI in order to clone the CEO’s voice? And what if the fake CEO started calling people (instead of sending emails)? This could lead to a much worse situation where employees think they’re interacting with (and obeying) the CEO and feel compelled to do something that could have grave financial consequences.

What this means for you

All the major LLMs are including watermarking in their AI-generated content, and due to recent regulations, that’s not going away. Bad actors can use AI today to perform serious attacks on the public, so having an AI watermark on any AI-generated content (text, images, audio, or video) is a very useful tool to determine authenticity.

If you want to see a live demonstration on how AI watermarks can be applied and detected on your own, be sure to attend the course Understanding AI Text Watermarking, only on O’Reilly.

Artificial intelligence has changed cybersecurity permanently. While OpenAI’s inadvertent attack on Hugging Face brought AI-generated attacks to everyone’s attention, AI has been on security professionals’ radar for a long time. AI makes attacks possible at a scale that was hard to imagine a few years ago. Newly discovered zero-days can be exploited almost immediately. Deepfakes and their implications for phishing, vishing, and other attacks were nightmares that we knew were real; we just hoped we’d have a few more years before we needed to defend against them. And of course, inattention to security at OpenAI, Anthropic, Google, and other labs that we don’t yet know about has resulted in AI agents that attack other systems in an attempt to solve a problem.

While the headlines have been shouting about AI killing humanity, doomsday AI is more likely a smokescreen to divert attention away from real harms that are happening now. We don’t need panic. We need an assessment of where we are: What are the threats? Where are the skills gaps? Where is the budget? It’s hard to overstate the importance of these questions; Tomasz Tunguz, writing about the Theory 2026 AI Survey, reports that security is the biggest barrier to AI adoption. To start answering these questions, in September we ran a survey that asked infosec professionals about the intersection between artificial intelligence and security. 351 respondents completed the full survey, 209 of whom self-identified as being involved with security as part of their job. This report focuses on those 209 respondents.

Attacks, risks, and responses

52% of respondents said their work rarely (34%) or never (18%) involves AI-generated attacks. This seems unlikely. The most likely explanation is that the security profession includes a broad range of job responsibilities; many security professionals don’t deal directly with defense. That group includes auditors, security experts on software development teams, strategists, architects, and others. It’s also possible that most phishes are either blocked by an automated filter (itself an instance of AI) or deleted immediately by the recipient; many companies require their staff to take training courses that cover topics like recognizing phishes. Still, it’s surprising. Much simpler versions of AI have been used to generate spam and phishing email for years. AI’s ability to generate phishes was demonstrated at BlackHat in 2021 with GPT-3. It’s been part of the security picture for a long time.

The most commonly observed attacks are AI-generated phishing (55%), vulnerability scanning and exploitation (30%), and AI-assisted malware generation (26%). Respondents who frequently encountered AI attacks also reported more types of attacks. There’s no surprise here: If you see attacks more frequently, you are more likely to observe specific attacks. The strongest relationship was with AI-generated phishing, which was seen by 55% of all respondents and 86% of those who observed AI-generated attacks weekly. Among respondents who observed attacks daily, slightly fewer (78%) reported phishing. Given the small sample, the difference may not be meaningful. It’s also likely that respondents who observe attacks daily are seeing a much broader range of attacks and that phishing is genuinely less important to them.

Respondents reported seeing deepfake audio and video, which can be used for vishing and other impersonation attacks, less often than phishing, though it was far from nonexistent (25%). More respondents than we expected reported fully automated attacks, like OpenAI’s attack on Hugging Face, though the percentage was still low (16%). The percentage of respondents who observed adversarial inputs against models including prompt injection was similarly low (only 13%), but that’s high enough to indicate that prompt injection and other attacks directed against models are a real concern.

When we asked respondents what AI-related risks concerned them most, we got a different set of answers. The two most common concerns were overreliance on AI tools without human verification and leaking sensitive data via AI tools; 49% of respondents selected each answer. (The number of respondents selecting those answers happened to be the same, though there was little relationship between the two groups.) 

Although the quality of AI output has gotten a lot better in the past year, relying on unverified AI outputs remains problematic. However, AI can easily generate more output than humans are capable of reviewing, and software teams are now overwhelmed by the volume of AI-generated pull requests that need human review before they can be merged. This overload is causing many to rethink the review process and others to reject AI-generated PRs. Code review was never very good at finding bugs; it was better at familiarizing a team with developments in the codebase. It may be enough to rely on exhaustive testing.

Understanding how your software works when much of it is written by AI is a deeper problem. While AI is good at answering questions about a codebase, it usually doesn’t have access to the reasoning behind the code, which is often hidden in institutional knowledge—and institutional knowledge often means “Oh, yeah, that guy who retired three years ago.” In addition, software developers may lose their programming skills if writing code becomes a smaller part of their job. Security professionals themselves might wonder whether relying on AI for red-teaming, log analysis, and vulnerability detection leads to a degradation in their own skills. 

Companies also fear losing sensitive data via AI. Most AI providers offer enterprise accounts that promise that they won’t train models on data in prompts or attached documents, but it’s not clear that these terms of service can be trusted. The frontier laboratories are getting increasingly desperate for training data, even going to lengths such as buying failing startups for their data. Read OpenAI’s description of its Zero Data Retention and Private Safety Processing cynically, and ask if this really protects corporate data or if it allows using data in ways that aren’t directly observable by humans—for example, automatically generating synthetic data. Security professionals are paid to be cynical; thinking about how software, protocols, and policies can be abused is part of the job.

Roughly a third (35%) of the respondents are concerned about attackers using AI to discover vulnerabilities faster than they can mitigate those vulnerabilities and apply patches. Updating software is usually a good practice, but it comes with its own risks: Software updates may introduce new bugs and can lead to incompatibilities with established tools and practices. And defenders can’t yet delegate vulnerability detection, patching, and updating to AI. So overall, is AI more of a help to defenders or attackers? Andrew Ng, Bruce Schneier, and others have argued that the attackers are ahead right now. It’s easier to find a vulnerability, use it to create a payload, and launch an attack than it is to stop one. But the balance will eventually shift to the defenders, as AI becomes more capable of finding and patching vulnerabilities before they can become a problem.

It’s surprising that only a quarter (25%) of the respondents listed AI-generated phishing as a top concern, especially since phishing is the attack most frequently seen in the wild. Perhaps we’ve become too used to phishing; we’ve all been to the security trainings, we’ve all experienced it directly, and some of us have even fallen victim. Phishing is also an attack that, in many cases, is handled automatically. Familiarity may not always breed contempt, but something you see almost daily usually isn’t your biggest worry. 

More exotic attacks are even less of a concern. Only 11% listed prompt injection as one of their top concerns, for instance. However, real-world examples of prompt injection are appearing with increasing frequency. A recent study observed that roughly 1% of résumés contained hidden text to trick an AI system. Google has noticed many kinds of prompt injection, ranging from benign to overtly hostile. And in 2025, OWASP recognized prompt injection as the greatest security risk for users of AI. It’s surprising and unfortunate that it doesn’t appear to be a greater concern for security professionals, especially since prompt injection is a vulnerability that might be lessened by careful filtering but can’t be eliminated.

Even fewer security professionals are concerned about data poisoning (4%). Data poisoning is a serious problem for organizations that train their own models, but today frontier labs conduct much of the most resource-intensive model training. However, this is changing as post-training on corporate data becomes more common. We believe the trend toward locally specialized models will continue, and that data poisoning, as distinct from data theft, will become more of a concern. 

We asked specifically about the use of agents, which exploded this year with the launch of OpenClaw and Gas Town in January 2026. Agents create another set of security concerns. We’ve all heard horror stories: agents that deleted production databases, handled data improperly, generated email that should never have been sent, and the like. Using an agent properly requires a lot of attention to detail, and many users aren’t used to that responsibility, nor do they understand security well enough to exercise responsibility.

The biggest concern among our respondents is excessive permissions or unclear ownership (28%). That’s hardly a surprise; many security problems trace back to permissions. Unclear ownership makes permissions issues more dangerous. When an agent is making many calls to a model, using tools, and delegating to other agents and models, it quickly becomes unclear who (or what) is the authority responsible for any action. This situation isn’t handled well by our current authentication and authorization systems, which assume that identities and authorizations are long-term, stable entities.

Lack of monitoring or auditability was the second highest concern (19%), followed by inability to verify whether the agent stayed within its role (18%). These concerns are closely related: Ownership is often unclear when agents are in use, and lack of monitoring makes the situation worse. Likewise, when you can’t monitor, it’s difficult, if not impossible, to verify whether the agents stayed in their lane—and we’ve seen all too many examples recently of agents that abandoned their authorized roles in inappropriate ways.

Data leakage appears again, though it’s surprisingly low on the list (15%) given that 49% of the respondents indicated that data leakage was one of their top concerns. It’s possible that we’re seeing the difference between single and multiple selection questions; many respondents are concerned about data leakage, but it’s not their top concern. 

Prompt injection doesn’t rank highly for either question: 9% selected it as their top concern about AI agents, and 11% selected it as one of their top concerns. 

Policies and funding

Although nearly three-quarters of the respondents reported that their companies have an AI use policy in place (72%), less than half (44%) reported that the security policy is “comprehensive,” while 28% said that the policy is basic and needs updating. Another 9% said that their employer doesn’t have a policy but is actively developing one. 19% said that there was no policy, or that they “didn’t know,” an answer we consider equivalent to “there is no policy.”

Organizations need clear AI policies that tell employees what AI tools they can use and how they should (and shouldn’t) use them. It’s important for employees to understand whether their company is sensitive to sending data offsite, whether the company’s chosen AI providers guarantee their data won’t be used for training, which practices for attribution are acceptable, and much more. We have some sympathy with organizations that have a basic policy that needs updating. AI’s capabilities are advancing at unprecedented speed; a policy that was adequate in 2023 certainly needs updating now, and a policy that’s only a year old still predates the widespread use of agents like OpenClaw, Claude Code, and Codex, along with the widespread use of skills and tools to extend an agent’s capabilities. Keeping a policy up-to-date with the current state of AI is an imposing task.

The existence of a policy appears to be related to whether employees use AI safely. We asked about “shadow AI” (the use of AI outside of any definitive policy) and found that 36% of the respondents thought that their teammates occasionally use “shadow AI” and 23% thought that the practice is widespread. About a quarter (24%) thought that shadow AI use is rare; and 16% were unsure. 

The relationship between shadow AI and the existence of an AI policy tells us a lot about whether policies are effective. Respondents with a comprehensive AI policy were more likely to say that shadow AI use is rare (37%) or that it’s only used occasionally (36%); only 17% said that shadow AI is widespread at their company. In contrast, 35% of the respondents who reported a basic policy said that shadow AI is widespread—almost the same percentage as those who reported that there was no policy at all (37%) and only slightly worse than those who said that a policy is in development but doesn’t exist yet (28%). Those who had a basic policy, a policy under development, or no policy were less likely to report that shadow AI was used “rarely or never” (13%, 6%, and 21%, respectively).

Granted that shadow AI is probably underreported across the board, our data clearly shows that a comprehensive AI policy is a powerful tool in limiting it. If the policy is poor, or it doesn’t exist, employees will use whatever tools they can to get their job done, which leads to AI use that’s inconsistent, insecure, and possibly counterproductive; it may even lead to legal, contractual, and reputational risks. When there’s no policy, it’s also likely that AI isn’t used as widely as it could be. Telling employees what is expected helps them to act responsibly. A good AI policy doesn’t just tell staff what not to do; it also communicates how to use AI in ways that are appropriate, and may even provide for training. 

When we asked whether organizations are adequately funding AI security, the picture was better than we expected. The largest group (32%) said that AI security is sufficiently funded, and 28% said it’s partially funded. Only 16% said it is severely underfunded, and 23% are unsure. I admit to being surprised by this result. Security is often underfunded, partly because a successful year is one in which “nothing bad happened,” and “we need to spend more so that nothing bad will continue to happen” isn’t a story that plays well at budget time. But executives have been catching on over the last few decades; high-profile attacks and data thefts lead to CIOs and CISOs losing their jobs, and that’s enough to get anyone’s attention. 

We saw a strong link between a comprehensive AI policy and adequate funding for security. Nearly half (49%) of the respondents who said their company has a comprehensive AI policy also said that AI security is sufficiently funded, and another 29% said that it’s partially funded. Only 3% of this group said that AI security is severely underfunded. That means nearly 80% (78%) are at least partially satisfied with funding.

Once again, the answers given by respondents who said that their organizations have a basic policy, a policy under development, or no policy were surprisingly similar. 16%, 22%, and 21%, respectively, said that AI security is sufficiently funded. That’s certainly not a vote of confidence, though we might be able to draw two more bits of information from it. Developing a policy looks like it’s slightly better than having an out-of-date policy; the company is putting in the effort and making an investment. That makes some intuitive sense, though it might also be wishful thinking. Among respondents whose organizations have a basic policy, the largest subgroup said that AI security is partially funded (42%). A middle-of-the-road policy is associated with middle-of-the-road funding. Is that fair, or is that what reality looks like?

Tools and skills

While AI gives security professionals powerful new tools, they aren’t yet fully taking advantage of them. Two-thirds of the respondents (68%) reported that they use a chat, copilot, or similar tool for research or documentation. But only slightly over a third use AI to scan for vulnerabilities (38%), just over a quarter use it for threat detection (27%), and a quarter use AI for incident response (25%). Security professionals widely use general-purpose AI. However, those using specialized tools like vulnerability scanners remain the minority.

To probe further, we asked where AI improved respondents’ workflows. More than half said that AI improved their workflows for research or documentation (61%) and that it reduced repetitive work (57%). But only 29% said that AI improved their workflow for triage, and 17% said that it improved detection accuracy. These results confirm what we saw for tool use.

The relationship between tool use and impact shows up even more clearly when we look at who’s benefiting most from AI. Those who are only using chat or a copilot see the greatest improvement in the speed of research or documentation. Chatbots and copilots save time, particularly when writing reports with a lot of boilerplate text, but they aren’t as helpful when actively defending against or recovering from attacks. Without specialized tools, that work has to be manual.

Respondents using security-specific AI tools were more likely to report improvements in alert triage (42% versus 18%) and detection accuracy (27% versus 5%) than those using only general-purpose chatbots or copilots. 

It’s far from surprising that respondents who didn’t use any AI tools were most likely to say that they saw no noticeable positive impact. The bigger question is why they saw any impact at all. 

Finally, we looked at the “skills gap” to determine what skills and tools would make respondents more effective. Only 7% of the respondents said that they don’t feel like they have a skills gap; that group was also the most likely to say that they aren’t using any AI tools. It’s clear that most of our respondents see AI as a necessary part of the future and want to be part of it. 

34% of the respondents want to use AI tools more effectively. That desire was evenly spread across all AI users, both chat/copilot only (36%) and users of security-specific tools (37%). 24% want to learn how to use AI to secure systems (including the use of AI for red-teaming); 19% want to learn AI-specific attack techniques; and 12% want to learn how to detect AI-generated attacks. 

The future of security

We started by claiming that AI has changed computer security forever. In addition to the many reports of research projects that inadvertently attacked other sites, AI has become a tool for attackers. They aren’t distracted by arguments about whether or not AI is taking jobs; their goal is to exploit their victims, and it’s clear that AI will help them. And as Ben Lorica points out in his newsletter, the economics of attack are changing: AI is infinitely patient, can try multiple attack vectors, can abandon failures, and can work at a scale that’s difficult for humans to match.

AI is also a tool for defenders. We saw more security professionals using AI than we expected, though not as many as we had hoped. Many defenders are taking advantage of chatbots and copilots but haven’t yet adopted more specialized tools, which are associated with improvements in workflows for triage and threat detection. 

The tools for defenders undoubtedly need to mature, but tools aren’t the entire answer. Organizations that adopt AI successfully will have appropriate budgets and policies for AI use; they will understand the risks that come with AI adoption; they will proactively fill skills gaps; and they will recognize processes that are no longer productive and “turn scar tissue into infrastructure.” AI is, in the end, a “normal technology” that takes years to penetrate organizations, as Arvind Narayanan and Sayash Kapoor have argued. Tools, budgets, and thought processes all need to change, and that takes time. Getting ahead of the curve on AI won’t happen if individuals work in isolation, adopting tools and ideas as they see fit. It’s an effort that requires the entire organization.