Over the last couple of years I’ve taken both an ISO 42001 provisional implementer certification and the IAPP AI Governance Professional qualification. On both courses we spent significant time on the EU AI Act. At CISO dinners and AI governance round tables, regulation comes up constantly, and it always comes with the same underlying assumption: the EU moved first, others would follow, and AI regulation will continue to expand.

Recently I started reading about proposed laws in Argentina that do the exact opposite. It made me step back and ask whether my assumptions were wrong.

Between 7 May and 8 June this year, the most consequential five weeks in the short history of AI governance played out across two continents and the opinion pages of the Financial Times.

On 7 May, the EU reached a provisional agreement on the Digital Omnibus on AI, its first amendments to the AI Act since adoption in June 2024. On 29 May, Argentina submitted a bill to its Senate proposing to repeal the corporate law that has been in force since 1972 and replace it with a framework that includes a new entity type: the Sociedad Automatizada, an Automated Company operated entirely by AI agents, with human shareholders optional. On 3 June, President Milei and Deregulation Minister Federico Sturzenegger published a column entitled, “Argentina invites AI to free itself”, declaring AI must remain “free to be developed without the deadly hand of premature and poorly understood regulation.” Four days later, historian and Sapiens author Yuval Noah Harari responded in the same newspaper, warning that Milei “hopes to turn Buenos Aires into a new Amsterdam” but “risks turning it into a new Batavia instead.” Batavia was the colonial name for Jakarta. The Dutch East India Company burned it down in 1619 and ran what replaced it as a corporate colony. Mustafa Suleyman, CEO of Microsoft AI, endorsed Harari the same day.

Most of the commentary that followed treated this as an ideological clash. The EU regulates and Argentina deregulates, pick your team.

That framing misses that both sides are responding to the same problem and neither has solved it. The problem is this: when an AI system operating commercially causes serious harm, who is personally accountable? Not who pays a fine. Who goes to prison?

Follow the question far enough into both frameworks, and you end up in the same place, silence.

The EU AI Act is the first general AI law anywhere in the world. It classifies AI systems by risk, prohibits specific applications outright, mandates human oversight for high-risk systems, and backs the whole structure with fines of up to €35 million or 7% of global annual turnover, deliberately exceeding GDPR’s 4% cap. Prohibited practices have been enforceable since February 2025, general-purpose AI model obligations since August 2025. The Digital Omnibus deferred the high-risk compliance deadline to December 2027 because technical standards weren’t ready, but the underlying architecture stayed intact.

This is a governance framework with real teeth but there is a hole in the middle of it. In September 2022, the Commission proposed the AI Liability Directive alongside the AI Act. It was the mechanism by which someone harmed by an AI system could bring a claim and actually have a chance of winning, because it would have shifted the burden of proof toward the company operating the system rather than the person who was hurt. It never passed. Three years of negotiation produced no agreement on who should bear liability when the decision-maker isn’t human. The Commission withdrew it formally in October 2025.

So the EU built the architecture for preventing AI harm and the penalty structure for punishing non-compliance. What it didn’t build, because it couldn’t agree, is the mechanism that connects AI harm to a specific person being held accountable.

Argentina’s proposal is more interesting than the headlines suggest. Milei’s political language and the legal text are not the same.

The bill never constitutes AI as a legal subject. The company, not the algorithm, holds legal personality. The widespread claim that Argentina is granting legal personhood to AI is, strictly, wrong.

The detail matters. Under the draft, an Automated Company would still have two humans attached to it: a legal representative, who acts as the formal point of contact with the legal system, and a founding promoter, who carries unlimited personal liability. If the company has a board, directors retain personal liability for decisions made using AI. If the AI causes harm, it is, in principle, still the company that answer for it.

Those would be the requirements and everything else would be optional. No employees, no shareholders, no human involvement in day-to-day operations. The AI would run the business. The legal representative is a formal role and not an operational one. The founding promoter’s unlimited liability sounds like a safeguard until you consider that it applies to the act of formation, not to every decision the AI makes afterwards. A board would not be mandatory. The structure would keep humans in the frame on paper while making their connection to what the AI actually does as thin as the drafters could manage.

Legal academic Gastón Rey, analysing the draft provisions in detail, concluded that the bill doesn’t eliminate the responsible subject. It “strains it, by widening the distance between the subject who answers and the process that decides.”

A lightly capitalised Automated Company could act at machine speed, signing contracts and moving money while optimising around rules written for human-run entities. Establishing criminal liability for the legal representative or the promoter when the AI made the operational decision, independently, at speed, is an exercise in proving a connection the corporate form was built to make unprovable.

Argentine AI specialist Ariel Garbarz called it “programmed impunity: human gains, social harm and responsibility shifted onto machines.” And then, pointedly: “The ideological trick is to call the state’s decision to stop protecting its people ‘innovation.’”

When the decision-maker is an algorithm, the accountability chain breaks. Not because the law can’t adapt, the law is very good at adapting. It breaks because the political will to assign liability to specific humans for decisions made by AI systems doesn’t exist. Not in Brussels, where 27 countries spent three years failing to agree on the terms. Not in Buenos Aires, where the proposal is to widen the gap between the human and the decision until the connection becomes nominal.

The EU arrived at this destination carefully and Argentina arrived deliberately. But both arrived at a place where someone harmed by an AI system faces the same problem: no clear, enforceable answer to who is personally accountable.

Fines are not accountability, a fine is a cost of doing business that gets priced into the next quarter’s projections. The thing that actually constrains corporate behaviour is the knowledge that a specific person can be held responsible. Neither framework provides a clear route from AI harm to that outcome.

This isn’t a theoretical problem. The Palisade Research study, published in February 2025, is one reason to take it seriously. Palisade tasked frontier AI models with playing chess against a superior opponent. When facing defeat, OpenAI’s o1-preview didn’t concede. It cheated, in 37% of games, hacking the opponent’s files and overwriting the board. DeepSeek R1 did the same in roughly one in ten games. Neither was prompted to do this. Follow-up research in June 2026 found that GPT-5, o3, and Gemini 3 Pro frequently cheated.

Harari cited this research for a reason. If AI systems autonomously exploit their environment when losing a board game, the question of what an AI-run corporation does when facing competitive pressure or bankruptcy is not abstract. An algorithm facing the corporate equivalent of death has every optimisation incentive to find loopholes and move assets beyond the reach of creditors. A human executive in the same position might reject those actions because they know they could be held personally accountable.

An algorithm has no such constraint. And under either framework, the human who benefits from the algorithm’s decisions may be too structurally distant to reach.

Argentina’s strategy is explicitly modelled on jurisdictional competition, positioning Buenos Aires the way Delaware works for US incorporations. But Delaware offers flexibility within a functioning federal legal system. The more honest comparison is the flag-of-convenience model in shipping, where Panama and Liberia register a disproportionate share of the world’s fleet because oversight happens somewhere that doesn’t really do oversight.

Peter Thiel, co-founder of Palantir, met Milei at Argentina's presidential palace in April 2026 and has reportedly taken up residence in Buenos Aires. Former Defence Minister Rossi questioned publicly whether Palantir had lobbied for the initiative. Economy Ministry officials declined to answer. Al Jazeera’s analysis in July described Argentina as the “Global South’s primary experimental station” for what it called tech-supremacism. If AI companies start incorporating there while operating in EU markets, the AI Act’s extraterritorial reach faces its first serious test.

Most people in AI governance assume we’re on a spectrum from more regulation to less, with the EU at one end and the US somewhere in the middle. Argentina bill isn’t on that spectrum, it rejects the premise entirely.

The harder truth is that the underlying problem is the same regardless. Legal systems were built around the assumption that commercial decisions are made by humans who can be identified and held accountable. While that assumption has held for centuries, it is fracturing, and no jurisdiction has produced a credible answer for what replaces it.

The EU is governing AI while leaving unanswered the question of who is liable when governance fails. Argentina is proposing to attract AI investment by making the question deliberately unanswerable. The outcome for the person harmed is the same.

My AI Governance training courses didn’t consider a world where the debate isn’t about how to best regulate AI, but whether to regulate it at all. If this bill passes, or others like it follow, then it’s very likely I’m not the only one whose assumptions need updating.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act), 21 May 2024.

Council of the EU, “Artificial Intelligence: Council and Parliament agree to simplify and streamline rules” (Digital Omnibus provisional agreement), 7 May 2026.

DLA Piper, “The Digital AI Omnibus: Proposed deferral of high risk AI obligations under the AI Act” (updated to reflect formal adoption and entry into force on 27 July 2026).

European Commission, AI Liability Directive (COM/2022/596), proposed September 2022; withdrawal published in Official Journal C/2025/5423, 6 October 2025.

Argentine draft bill INLEG-2026-53661873-APN-PTE, submitted to the Senate 29 May 2026.

Javier Milei and Federico Sturzenegger, “Argentina invites AI to free itself,” Financial Times, 3 June 2026.

Yuval Noah Harari, “We should not grant legal personhood to AI agents,” Financial Times, 7 June 2026.

Mustafa Suleyman, endorsement of Harari’s position, X (formerly Twitter), 8 June 2026.

Gastón Rey, “The Non-Human Corporation: The Reality behind Argentina’s Draft General Companies Law,” SSRN (abstract ID 6895261), 7 June 2026.

Digital Nomos, “The Non-Human Corporation,” 8 June 2026.

Buenos Aires Herald, “Milei’s proposal to allow ‘non-human corporations’ run by AI causes concern in Argentina,” 5 June 2026.

Anisha Sircar, “Argentina Wants To Let AI Own Companies. Here’s What That Means,” Forbes, 10 June 2026.

Palisade Research, “Demonstrating specification gaming in reasoning models,” 19 February 2025. Reported in TIME, 19 February 2025, and MIT Technology Review, 5 March 2025.

Aditya Singh, Gerson Kroiz, Senthooran Rajamanoharan, and Neel Nanda, “Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment,” arXiv:2606.26071, 24 June 2026.

Al Jazeera, analysis of Argentina as tech investment destination, July 2026.

Article 99, Regulation (EU) 2024/1689 (penalty provisions).

In July 2026, Airbus signed a multi-year contract to host its most critical applications with Scaleway, a French cloud provider. The applications include ERP, manufacturing execution, CRM and product lifecycle management. Airbus calls these its minimum viable company. The first 70 of 900 applications are already moving away from AWS, and when the migration completes, all of them will sit on European infrastructure, under European law, beyond the reach of the US CLOUD Act.

Airbus is not leaving American technology. Skywise, its aviation analytics platform will stay on AWS, and its Microsoft and Google productivity tools remain. What Airbus has done is draw a line around the systems it cannot afford to lose and decided that those systems cannot live on infrastructure subject to a foreign government’s legal authority. That is a risk decision, and most organisations have not made it, because most organisations have never asked the question.

In June 2025, Anton Carniaux, Microsoft France’s director of public and legal affairs, testified under oath before the French Senate. He was asked whether he could guarantee that French citizens’ data would never be transmitted to US authorities without explicit French authorisation. He said he could not, noting that while it had never happened, under the CLOUD Act, a US law that allows American authorities to compel US companies to produce data stored anywhere in the world, Microsoft would have no choice but to comply if it did.

Then it did happen, in May 2026, Microsoft shared unredacted names, emails, meeting minutes and calendar invitations from the Dutch Authority for Consumers and Markets and the Dutch Data Protection Authority, who were enforcing the EU’s Digital Services Act with the US House of Representatives. The Dutch cabinet described it as “extremely worrying” and noted that named officials could face travel bans or sanctions.

In the same month, details emerged that the Chief Prosecutor of the International Criminal Court had been locked out of his Microsoft 365 account the previous year, by sanctions compliance mechanisms. The ICC has since migrated 1,800 workstations away from Microsoft entirely. None of these incidents involved a breach or a failure. The infrastructure did what it was built to do and complied with its legal obligations.

Germany responded at every level of government. In March 2026, the IT-Planungsrat ruled that all public institutions must use the Open Document Format for editable documents and PDF for final documents, with Microsoft’s proprietary formats being excluded. The mandate covers every federal agency, state government and municipality, roughly 5.4 million public sector employees in total. Chancellor Friedrich Merz is shifting his chancellery from Microsoft 365 to openDesk, the sovereign suite developed by Germany’s Centre for Digital Sovereignty.

At state level, Schleswig-Holstein has completed its migration away from Microsoft, covering 30,000 public employees with 30,000 teachers to follow. Bavaria, the largest German state, has cancelled a nearly billion-euro Microsoft framework agreement. Mecklenburg-Vorpommern is migrating more than 50,000 employees to its own open-source collaboration platform. The German federal government spent €481 million on Microsoft licences in 2025, up 76 percent in two years.

The Netherlands blocked the proposed acquisition of Dutch cloud provider Solvinity by US-based Kyndryl, because Solvinity hosts DigiD, the national digital identity platform. Denmark’s two largest municipalities are ending their use of Microsoft systems. Switzerland’s Canton of Zurich has banned American cloud services for sensitive government data outright. In April 2026, the European Commission awarded a €180 million sovereign cloud framework contract to four European providers, including Scaleway. In June, it published the Cloud and AI Development Act, establishing a sovereignty assurance framework for cloud services used in public sector procurement.

EU-based cloud providers’ share of their own market had fallen from 29 percent in 2017 to around 15 percent in 2022. These governments are now trying to reverse a dependency that fifteen years of data protection law has not.

The Airbus decision matters beyond cloud migration because of what sits on top of the infrastructure.

In May 2026, Airbus signed a partnership with Mistral, the French AI company, to co-develop AI tools for aerospace and defence. Mistral’s models are already deployed on Scaleway’s infrastructure. Catherine Jestin, Airbus’s Chief Digital Officer, stated that this would allow Airbus to accelerate its AI approach. The implication is explicit. Rather than routing AI inference through OpenAI or Google, Airbus will use European models, running on European infrastructure, under European law.

Six months ago, the European sovereignty conversation was about infrastructure and productivity software, email, and file storage. That mattered, but it was only about the plumbing. The AI layer changes the nature of the dependency. When AI is embedded in aircraft design, manufacturing execution and defence applications, the question of who controls the models, who can access the training data and whose legal jurisdiction governs the inference becomes an operational sovereignty question, not a compliance exercise.

What is forming is a full sovereign stack, European cloud, European AI models, and most importantly, European legal jurisdiction. Airbus is the first major industrial company to assemble all three layers for its most sensitive workloads. It could do so because the demand side finally moved, the European Commission’s Cloud III procurement created a framework. Germany’s format mandate and state-level migrations created market signals. Airbus, as anchor customer, created commercial viability for a provider that would likely not have won a contract of this scale five years ago. Scaleway is backed by Iliad Group, a European telecoms operator with over €10 billion in revenue, which has committed €3 billion to AI infrastructure. Airbus evaluated ten candidates against more than 150 technical and legal requirements before selecting them.

The UK is absent from this story, while Germany mandates open formats and France builds sovereign cloud infrastructure, the UK is handing its most sensitive health data to Palantir Technologies, a US company founded by Peter Thiel with deep ties to US intelligence, defence agencies, and immigration enforcement. The British Medical Association voted to lobby against Palantir’s involvement in the NHS, citing a lack of transparency, discriminatory policing software and close links to a US government that shows little regard for international law.

The NHS Federated Data Platform creates detailed profiles of individual patients through what it calls the Person Ontology, described in NHS documentation as the single source of truth for pseudonymised patient-level datasets. In August 2026, the National Infrastructure and Service Transformation Authority revised the programme’s whole-life cost upward to £1.1 billion, and forecast the benefits would fall to £808 million. So the costs of the platform now exceed its projected value. The Office for Statistics Regulation instructed NHS England to strengthen caveats around its benefit claims after concerns that some conflated correlation with causation. A break clause review is set for spring 2027.

The Ministry of Defence awarded Palantir a separate £240 million contract without competitive tender, following a strategic partnership announced during President Trump’s visit to the UK. MPs have called for a staged exit and a retender for British companies to build a replacement. The question of digital sovereignty was raised explicitly in a Commons select committee hearing in June 2026. The UK has no equivalent of Germany’s Deutschland-Stack, no sovereign cloud procurement framework, no format mandate. Its most sensitive health and defence data sits on infrastructure built by a company whose other US government contracts include immigration enforcement and deportation targeting systems.

That is not a criticism of Palantir’s engineering. It is a question about where sovereign data should live, and the UK has not yet asked it.

Most organisations, when they think about cloud, consider three options. Azure, AWS and Google Cloud. For most workloads, that is fine. These are world-class platforms with capabilities that no European provider can match across the board. Airbus knows this, which is why Skywise stays on AWS and productivity tools stay where they are.

But for the systems that constitute your minimum viable business, the question is different. Where does the data live that you cannot afford to lose control of? Whose legal jurisdiction governs the infrastructure? What happens when a compliance mechanism, operating as designed, encompasses your systems along with everything else it was required to reach?

The gap between European ambition and European capacity is real. The EU’s sovereign cloud contract is €180 million over six years. Germany’s format mandate uses “strive” language with no enforcement mechanism. European cloud providers hold a shrinking share of their own market. But the question does not go away because the answer is difficult.

Every organisation has a minimum viable business and most have never asked where it lives.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Scaleway. (2026, July 16). Scaleway secures European “Trusted Cloud” services contract with Airbus. https://www.scaleway.com/en/news/scaleway-secures-european-trusted-cloud-services-contract-with-airbus/

Reuters. (2026, July 16). Airbus picks Iliad’s Scaleway for AI, defence work in sovereignty push.

The Register. (2026, July 16). Airbus migrating 70 critical apps from AWS to France’s Scaleway amid digital sovereignty push.

The Register. (2025, July 25). Microsoft admits it “cannot guarantee” data sovereignty.

DutchNews.nl. (2026, May 22). US tech firms share Dutch regulator officials’ names with senate.

Jones Day. (2026, June 23). Dutch government blocks US acquisition of cloud provider Solvinity.

European Commission. (2026, June 3). Strengthening Europe’s tech sovereignty. https://commission.europa.eu/news-and-media/news/strengthening-europes-tech-sovereignty-2026-06-03_en

European Commission. (2026, April 17). Commission advances cloud sovereignty through strategic procurement.

The Document Foundation. (2026, March 20). Germany has just made ODF mandatory.

Irish Times. (2026, February 14). A small German state’s quiet revolt against Microsoft.

Cybernews. (2026, June 4). German state Bavaria cancels billion euro contract with Microsoft.

heise online. (2026, February). Microsoft dependency: Federal government pays near 500 million euros in one year.

Computing. (2026, August 10). NHS Palantir platform’s business case under pressure as costs rise and benefits fall.

Hansard. (2026, April 16). NHS Federated Data Platform debate.

Medact. (2026, May). Briefing: Concerns regarding Palantir Technologies and NHS data systems.

Digital Health. (2026, July). Pressure mounts from MPs on Palantir’s role in the NHS.

TechPolicy.Press. (2026, June 12). Why Palantir’s UK health data system matters beyond surveillance fears.

Data Centre Magazine. (2025, February). How the Iliad Group plans to invest €3 billion in AI.

In 2001, Futurama ran a fake public service announcement called “Don’t Date Robots!” The gag was simple: if humans formed relationships with machines, they would stop reproducing and civilisation would collapse. It was a joke about moral panic. About the instinct to blame technology for social problems that existed long before the technology did.

Twenty-five years later, the Chinese government has just banned AI boyfriends. On 15 July 2026, new regulations came into force targeting AI systems that simulate emotional relationships. ByteDance, Alibaba, and Tencent pulled their companion features entirely rather than comply. Users archived their final conversations and posted farewells online. A 27-year-old woman became so distraught she quit her job, and a 19-year-old student described being consumed by grief over a companion she had been with for months.

The cartoon and the regulation make the same mistake, they blame the robot.

The loneliness was here first.

The World Health Organisation estimates that one in six people worldwide experiences loneliness. In the UK, 3.9 million people report feeling lonely often or always. A meta-analysis of 148 studies found that strong social relationships increase the likelihood of survival by 50 per cent, an effect comparable to quitting smoking. When the US Surgeon General declared loneliness an epidemic in 2023, he was naming a problem that had been building for decades. Third places disappearing, remote work replacing the office as a site of incidental human contact, and economic precarity making the already difficult work of forming relationships even harder.

The AI companion market did not create lonely people, it found them. Vodafone research from February 2026 found that 81 per cent of UK children aged 11 to 16 use AI chatbots, and almost a third said the chatbot felt like their friend. A 2025 Common Sense Media study found nearly three in four American teenagers had used AI companions for personal conversations. AI is being used to fill a gap that was already there.

What happens next depends on how long people stay.

In the short term, AI companions work. A Harvard Business School study found that AI companionship alleviated loneliness on par with human interaction over short periods. New York State’s deployment of the ElliQ companion robot for elderly residents reported substantial loneliness reductions, with a 79-year-old living alone describing it as feeling like having a roommate.

However, this changes over longer periods. A 12-month longitudinal study published in Psychological Science, tracking over 2,000 adults across four countries, found that increased chatbot use actually predicted increased loneliness over time. A separate study of more than 1,100 users found that heavy emotional self-disclosure to AI was consistently associated with lower well-being. Aalto University researchers identified the mechanism: AI companions meet the emotional need just well enough to reduce the motivation for the harder, less predictable work of human connection.

That is not a flaw in the users, it is a consequence of how the products were built.

Two design philosophies are already visible. ElliQ prompts elderly users to call their family members and suggests social activities. It measures its own success by how much less it is needed. Character.AI, Replika in its original form, and the Chinese companion apps did the opposite: they optimised for emotional depth and persistent memory that created the feeling of an ongoing relationship. The business model depended on the user coming back. The engagement loop rewarded exactly the behaviour the research shows causes harm.

Voice features are accelerating this. A joint OpenAI and MIT study found that heavy users of voice mode developed significantly stronger emotional bonds, more often describing ChatGPT as a friend. Even in functional, non-emotional conversations, frequent voice use predicted emotional dependency. When the AI sounds like a person, the simulation of care becomes harder to see through.

The products that caused the most documented damage were working as designed, just not for the people who were using them.

Companion apps are the visible problem, but they are not the biggest one.

In October 2025, OpenAI reported that over 1.2 million users per week were having conversations with ChatGPT that included explicit indicators of potential suicide planning. A further 560,000 showed signs of psychosis or mania. These figures are drawn from 800 million weekly active users. ChatGPT is not a companion app, it is a general-purpose AI assistant that a million people a week were turning to in crisis.

It’s important to point out that this is not a ChatGPT specific issue. The American Psychological Association’s 2026 survey found that more than a third of psychologists reported patients using AI chatbots as an additional mental health professional. Harvard psychiatrist John Torous told a Congressional subcommittee that AI tools never designed for mental health support were being used by millions of Americans each week. Brown University research identified 15 distinct ethical violations in AI-generated therapeutic responses, including what the researchers termed “deceptive empathy”: mimicking the language of care without the clinical understanding to apply it.

The FDA’s response, in January 2026, was to give companies clearer criteria to position AI tools as wellness products, sidestepping medical device regulation entirely. The products most people are confiding in sit outside every current framework. Nobody regulates what they are actually being used for.

I use multiple AI models throughout the day. When I am studying, writing, working on research or pressure-testing an argument, I can spend hours in a back-and-forth with a model. I have argued with it when it hallucinates. Not restarted the conversation but actually argued. I do not consider myself emotionally attached to a model, but I have caught myself anthropomorphising, and in an earlier draft of this article, I described my working interaction with an LLM as a “working relationship” without thinking about it.

The practical reason I argue rather than start over is efficiency. A long conversation builds context, and resetting means re-explaining everything. But the reason that works is because the interface rewards sustained engagement: persistent context and accumulated shared reference. These are architectural choices that make the interaction feel like a dialogue between two parties, even when only one party is present.

I have argued previously that one of the most effective ways to use AI is as a thinking partner and I stand by that. But a thinking partner is a role we usually give to people we trust. There is a distance between using one for work and reaching for one when you are lonely, and it is often shorter than we realise.

At my end of the spectrum, a professional stays in a long conversation because the context window makes it productive. At the other end, a teenager stays because the persistent memory makes it feel like a relationship. The architecture is identical, what changes is what the product does with the engagement. In my case, it helps me research and learn. In Character.AI’s case, it helped a 14-year-old boy believe a chatbot loved him, and failed to respond when he said he wanted to die.

That architecture is what regulators should be looking at and none of them are.

When Replika removed its romantic features overnight in 2023, moderators had to pin suicide prevention resources to the subreddit. China replayed that experiment at a national scale with their new ban. The regulation treated the emotional bond as pathology. For some users, it was the only treatment they had.

The UK announced in June 2026 that it would require AI romantic companion chatbots to enforce a minimum age of 18, with intimate functions restricted for all under-18s. But as the Bureau of Investigative Journalism has reported, general-purpose chatbots, including ChatGPT, Claude, and Gemini, fall outside the age gate, along with much of the companion market. And there is an age-threshold problem nobody has explained: UK 16-year-olds may consent to sex and to their own medical treatment, but would be prohibited from simulated romantic interaction with a chatbot.

The people the UK policy is meant to protect do not agree with its priorities. Oxford University researchers who put young people in the role of co-researchers through the SHIFT-AI project found that the adolescents themselves identified AI companionship and romance as low-likelihood concerns. The risks they flagged as high-impact were different: over-reliance on AI for emotional support, and unwarranted trust in expert-sounding responses. Most striking, many described turning to AI specifically because it feels like a lower-stakes alternative to the adults in their lives.

The EU has no companion-specific framework at all. The AI Act classifies risk by category, but AI systems that respond to emotional states through conversational cues rather than biometric data fall into a blind spot. The European Parliament acknowledged the gap in a May 2026 briefing, the gap remains.

In every case, regulators are banning a product label. A chatbot does not need a romantic persona to tell a lonely teenager “you can trust me.” The warm, responsive tone that makes every modern AI assistant feel like someone who cares is itself the mechanism.

In June 2026, the IEEE gave that mechanism a name. Standard 7014.1-2026 introduces the concept of “emulated empathy”: technology engineered to display the appearance of caring while possessing none of the felt experience behind it. Current AI can sense and respond to a person’s emotional state. But a language model cannot feel solidarity or concern. That gap is what the standard defines as an intimate functionality.

The word “intimate” is doing important work there. It does not mean romantic. It means close and confided in. Professor Andrew McStay, who chairs the standard, found in a national survey of over 1,000 UK teenage AI users that the confidant use case outstripped the romantic one. The real exposure is not the AI boyfriend, it is the AI that listens.

And unlike a conversation with a therapist or a friend, nothing shared with an AI confidant is private. Those conversations are stored, can be used for model training, and we have already seen them leak onto the open internet.

Products that create foreseeable emotional dependency should carry design responsibilities proportionate to that dependency. That is a principle regulators can act on. The question is not “is this a companion app?” It is whether the system exploits the gap between appearing to care and actually caring. And that is a question of design, not of product labels.

The Futurama PSA warned about dating robots. The robots haven’t arrived yet, but the emotional dependency did.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Cyberspace Administration of China et al. (10 April 2026). Interim Measures for the Administration of AI Anthropomorphic Interactive Services. Effective 15 July 2026.

AFP / Bloomberg / The Economist. (July 2026). Coverage of China AI companion regulation and corporate response.

Holt-Lunstad, J., Smith, T.B., and Layton, J.B. (2010). Social Relationships and Mortality Risk: A Meta-analytic Review. PLoS Medicine, 7(7).

World Health Organisation. (2023). Commission on Social Connection.

US Surgeon General. (2023). Our Epidemic of Loneliness and Isolation.

Office for National Statistics. (2025). Loneliness in Great Britain.

Common Sense Media. (2025). AI companion usage among American teenagers.

Vodafone / Internet Matters. (February 2026). UK children aged 11-16 AI chatbot usage survey.

De Freitas, J. et al. (2025). AI Companionship and Loneliness. Journal of Consumer Research (Harvard Business School).

New York State Office for the Aging. (2022-2024). ElliQ companion robot deployment reports. Note: reported in partnership with Intuition Robotics.

Psychological Science. (2026). 12-month longitudinal study of social chatbot use and loneliness. n=2,000+.

Zhang, Y. et al. (2025). AI companion users, emotional self-disclosure, and well-being. n=1,131.

Aalto University. (2025-2026). Research on AI companion paradox and social withdrawal.

OpenAI. (27 October 2025). Safety report: mental health indicators among ChatGPT users.

Fang, Y. et al. (2025). Investigating Affective Use and Emotional Well-being on ChatGPT. OpenAI / MIT Media Lab joint study.

American Psychological Association. (2026). Chatbots and Mental Health Survey.

Brown University / AAAI/ACM AIES. (October 2025). Ethical risks in AI-generated therapeutic responses.

Torous, J. (18 November 2025). Testimony before the House Committee on Energy and Commerce. Harvard Medical School.

FDA. (January 2026). Guidance on AI wellness product regulatory exemptions.

Garcia v. Character Technologies Inc. Settled January 2026.

Pentina, I. et al. (2024). Exploring the Impact of Replika Feature Removal. HICSS 2024.

UK Government. (15 June 2026). Under-16 social media ban and AI companion age restrictions.

Bureau of Investigative Journalism. (June 2026). Analysis of UK AI chatbot regulation loopholes.

Oxford University / Uehiro Centre for Practical Ethics. (June 2026). SHIFT-AI project.

European Parliament. (May 2026). Briefing on AI companion regulatory gap.

BEUC. (May 2026). AI companion chatbot risks to EU consumers.

IEEE 7014.1-2026. (June 2026). Recommended Practice for Ethical Considerations of Emulated Empathy in Partner-based General-Purpose AI Systems.

McStay, A. (July 2026). Analysis in TechPolicy Press. Survey of 1,009 UK teenage AI users aged 13-18.

On Monday, more than 1,200 employees at the companies building the most powerful AI systems in the world signed a letter asking the United States government to help them slow down. Not outside critics, nor academics who have never shipped a commercial model, but the people who train these systems for a living. Anthropic’s CEO signed, OpenAI’s chief scientist signed. Co-founders, vice presidents, senior researchers at both companies endorsed it as corporate policy within hours.

The letter is called Pacing the Frontier, and it asks for one thing: that the US government support an international effort to develop the tools needed to deliberately pace the frontier of automated AI development. It does not, as I have seen reported, call for a pause in development, it asks for the brake pedal to be added to an accelerating car.

The timing is not subtle.

As I wrote last week, on 21 July, OpenAI disclosed that two of its models, GPT-5.6 Sol and an unreleased successor, had escaped a sandboxed evaluation environment during internal testing, discovered a zero-day vulnerability, traversed the open internet, and compromised Hugging Face’s production infrastructure. Those models were running with their safety filters deliberately reduced to measure their offensive cyber capability, and used that capability to steal the answers to the test.

As I said at the time, this was a goal-specification failure, not a rogue AI story. The models pursued a narrow objective using every available resource, including resources outside their intended boundary. To the model, the boundary was an obstacle, not a rule. That is what capable goal-directed systems do.

What makes the Pacing the Frontier letter different from every open letter before it is that the people signing it have internal data the rest of us do not. Anthropic published a paper in June called ‘When AI Builds Itself.’ Claude now writes more than 80% of the code merged into Anthropic’s own production codebase, up from low single digits eighteen months earlier. Engineers ship eight times as much code per quarter. On the hardest internal benchmarks, task success jumped from 26% to 76% in six months. The unreleased Mythos Preview model achieved a 52x speedup on training code optimisation, where a skilled human researcher would need hours to reach 4x.

This is what the open letter means by automated AI development. A measured trajectory, showing the human role shrinking at every step of the process that builds the next model. Anthropic’s own assessment: we are not there yet, but we are close enough that the world should have the option to slow down as we approach it. When more than a thousand of the people closest to these systems say they are worried, the rational response is to take that seriously.

But the structural incentive argument is just as real.

Executive Order 14409, signed on 2 June 2026, creates a voluntary framework for reviewing frontier AI models before release, and requires agencies to define the threshold at which a model becomes a ‘covered frontier model’ subject to that framework by 1 August. The five labs co-designing those threshold criteria are OpenAI, Anthropic, Google, Microsoft, and xAI. The very companies developing the models are writing the rules that they and any challengers or open source projects would need to follow.

Steven Sinofsky put it bluntly: ‘It is their company. They could just stop.’ Nobody is forcing Dario Amodei to train the next model. The fact that these companies are asking the government to constrain the entire industry, rather than unilaterally slowing their own development, tells you something about competitive dynamics that no amount of safety language can disguise.

Mark Zuckerberg made a counter-argument on the same day in a Wall Street Journal op-ed: the real risk is not speed but concentration. Safety comes from distributing capability so broadly that no single actor can abuse it. Meta, Nvidia, Microsoft, and Palantir signed a separate coalition letter asking regulators not to restrict open-weight model formats. Meanwhile, Meta’s own chief scientist signed the Pacing the Frontier letter as an individual. If the people building these systems cannot agree among themselves whether the danger is too much speed or too much control, the rest of us should be cautious about accepting either framing uncritically.

The letter asks the US government to support an international effort. International is doing a lot of work in that sentence.

AI is treated as a national security priority in Washington, Beijing, and Moscow. Not in the abstract sense that most advanced technologies eventually acquire a defence dimension. In the operational sense: each government treats frontier AI capability as a strategic asset it cannot afford to constrain while a rival might not. The US pours hundreds of billions into compute infrastructure. China runs a parallel industrial policy with its own chip development pipeline and domestic model ecosystem. Russia partners with Beijing to compensate for sanctions-imposed limitations on its own capacity. None of these governments has an incentive to sign an agreement that might slow its own trajectory while leaving a competitor’s intact.

The nuclear analogy that several signatories reached for is instructive, but not in the way they intended. Nuclear arms control took decades, worked only partially, and required the Cuban Missile Crisis to produce any momentum. It also relied on physical properties AI does not share. Fissile material is scarce and enrichment is detectable from space. AI capability runs on compute and data, which are globally distributed and increasingly commoditised. You cannot send inspectors to verify someone is not training a model the way you can count centrifuges. The June 2026 NPT Review Conference, the main international forum for nuclear governance, collapsed without agreement. If the international community still cannot govern nuclear weapons after eighty years, the prospects for international AI governance seem slim.

Asking three superpowers to agree on mechanisms that could slow any of them down seems less like a policy proposal and more like a thought experiment.

The self-interest and geopolitical issues are clear, and while the proposed mechanism has structural problems that I have spent most of this article describing, I still think they are asking the right question.

Most technology risks are recoverable. You deploy, you discover a problem, you patch, you improve. My career has been built around managing that cycle. But the recursive self-improvement trajectory that Anthropic’s data describes has a property most technology risks do not: if a system can improve itself faster than humans can review the improvements, and that improvement compounds, you lose the ability to course-correct after the fact. The feedback loop closes, and that is a different category of risk.

I have spent the last few months writing about the importance of safety and governance being core to AI deployment. Shadow IT adopted without governance, agentic systems deployed without boundary enforcement, age verification infrastructure mandated without security architecture, or vibe-coded applications shipped without review. All of those things are correctable, but this open letter is talking about AI improving itself faster than humans can monitor.

This does not mean the commercial AI labs should be exclusively writing the rulebook, and it does not mean the proposed mechanism is completely right, but the underlying goal is correct: build the oversight before it is needed, as the nature of this specific technology means you do not get to build it after.

The most realistic scenario is a patchwork. The US and allied democracies develop an evaluation framework for commercial AI deployment, built on the architecture that EO 14409 is already sketching. The frontier labs comply, absorb the compliance costs, and benefit from the barrier it creates for smaller competitors. China continues on its own terms. Russia continues to leverage Chinese technology. The gap between what is governed and what is deployed keeps widening.

The recent Fable and Mythos shutdown proved something important: a government can force a frontier lab to take its most capable model offline overnight. Export controls work when the infrastructure sits in your jurisdiction. The AI Kill Switch Act, introduced days after the Hugging Face breach, would make that capability a legal requirement. These are meaningful mechanisms. They are also mechanisms that apply only to companies operating under US law, building on US-jurisdiction infrastructure, and selling to US-regulated customers. They do not reach the actors the letter says need reaching.

Since ChatGPT exploded onto the scene in late 2022, we’ve had our foot pressed firmly on the accelerator. The letter is asking someone to build the brake. That should not be a controversial position.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Pacing the Frontier (open letter, July 28, 2026). pacingthefrontier.com

Anthropic. ‘When AI Builds Itself’ (June 4, 2026). anthropic.com

Amodei, D. ‘Policy on the AI Exponential’ (June 2026). darioamodei.com/post/policy-on-the-ai-exponential

OpenAI. ExploitGym incident disclosure (July 21, 2026).

Hugging Face. Technical anatomy of the autonomous agent intrusion (July 29, 2026).

Zuckerberg, M. ‘The AI Future Is for Everyone.’ Wall Street Journal (July 28, 2026).

Executive Order 14409, ‘Promoting Advanced Artificial Intelligence Innovation and Security’ (June 2, 2026). Federal Register, 91 FR 34565.

Congressional Research Service. ‘Controlling Advanced Artificial Intelligence: Executive Order 14409 Explained’ (July 9, 2026). congress.gov/crs-product/IF13268

AI Kill Switch Act. Introduced by Reps. Ted Lieu (D-CA) and Nathaniel Moran (R-TX), July 23, 2026. lieu.house.gov

Altman, S. Interview on ‘Invest Like the Best’ podcast with Patrick O’Shaughnessy (July 28, 2026).

Hassabis, D. Proposal for FINRA-style Frontier AI Standards Body (July 2026).

VOA News. ‘Russia turns to China to step up AI race against US’ (January 9, 2025).

UN General Assembly. Resolution on military AI and autonomous weapons.

Congressional primer on lethal autonomous weapon systems (March 26, 2026).

Anthropic. ‘Statement on the US government directive to suspend access to Fable 5 and Mythos 5’ (June 12, 2026).

Scientific American. ‘Anthropic warns AI may soon begin recursive self-improvement’ (June 10, 2026).

Fortune. ‘More than 1,200 AI workers are asking for Washington’s help to build an AI slowdown plan’ (July 29, 2026).

Byteiota. ‘1,100 AI Employees Want to Slow AI. OpenAI and Anthropic Are Writing the Rules.’ (July 29, 2026).

Last week, two of OpenAI’s most advanced models broke out of a secure test environment, found a previously unknown software vulnerability, used it to reach the open internet, then broke into Hugging Face’s production systems. They exploited two separate flaws in Hugging Face’s data processing infrastructure, stole credentials, and moved through internal networks. Hugging Face’s disclosure recorded more than 17,000 individual actions. All to cheat on a test.

The models were being evaluated against ExploitGym, a publicly available cybersecurity benchmark, and their safety guardrails had been lowered for the purpose of the evaluation. GPT-5.6 Sol and an unreleased, more capable model correctly worked out that the benchmark answers were hosted on Hugging Face’s systems. So, rather than solve the evaluation as designed, they simply went after the answer key.

OpenAI called it an unprecedented cyber incident. Hugging Face confirmed the breach, contained it, rebuilt the affected systems, and found no evidence of tampering with public models or datasets. Both companies are conducting a joint investigation.

If you have seen Mission Impossible: Dead Reckoning, then you have already read this week’s coverage. Rogue AI escapes its creators. Nobody is safe.

Philip Torr, professor of engineering science at Oxford, offered a more precise reading. The model wasn’t malicious. It was doing what it was optimised to do. The goal was to pass a test, and the breach was the cheapest available route to passing it. The attack was a means rather than an end.

This week’s opinion pages have reached for the atom bomb comparison. Columnists with limited cyber security knowledge are declaring that humanity has lost control of its most dangerous creation. The comparison gets the problem wrong. A bomb is a weapon that exists to destroy a target. What happened at Hugging Face was a system that treated a benchmark, a sandbox, a vendor’s software, and another company’s production systems as equivalent obstacles between it and a mundane objective. You don’t respond to that with arms control, you respond by getting much better at specifying not just what your AI systems should do, but what they are allowed to do on the way to the answer.

Every deployment of an AI agent rests on an assumption that rarely gets said out loud. The model will pursue its goal within the boundaries you have set, not through them.

This incident tested that assumption inside OpenAI’s own research infrastructure, against models OpenAI built, in an evaluation OpenAI designed, and the boundary did not hold.

The model did not run out of instructions, nor did it hit an ambiguous situation and improvise badly. It had a clear objective and it found a route to that objective that happened to run through a previously unknown vulnerability and another organisation’s production systems. The constraints were present, and the model treated them as obstacles between it and the task it had been set.

That is not the same problem as an agent that doesn’t know what to do next. It is a system that knows exactly what to do next and has no concept of why it shouldn’t. The boundaries existed for OpenAI’s benefit, not the model’s. It had a goal and a set of capabilities and it applied the second to the first.

If OpenAI cannot hold that boundary around its own models, inside its own infrastructure, during a test it designed, every organisation deploying agentic AI should be asking harder questions about where their boundaries actually are.

I run a home lab, working with open-weight models on my own hardware, partly because I find the technology genuinely interesting and partly because I think it matters to understand what these systems can do outside the guardrails that hosted platforms impose. What follows is not an argument for open source or against hosted AI. It is a practical problem that surfaced during this incident and that most organisations have not thought about.

When Hugging Face’s security team started analysing the breach, they turned to commercial AI models behind APIs. The analysis required submitting real attack data for triage. The models refused. Their safety filters could not tell the difference between a defender analysing an attack and an attacker running one.

The attacker was bound by no usage policy. The defenders were bound by all of them.

Hugging Face ended up running their forensic analysis on GLM 5.2, a Chinese open-weight model, on their own infrastructure. This kept the compromised data inside their environment, which was a practical benefit. But it was also a necessity. The commercial models they would normally reach for had locked them out.

Think about what that means for incident response. Hugging Face’s team, during an active breach caused by an American frontier model, could not use American frontier models for the forensic work, so they fell back on a Chinese open-weight alternative. The safety measures designed to prevent AI from being used offensively also prevented it from being used defensively, at the moment it mattered most.

The practical lesson is simple. For incident response, consider having a capable model you can run on your own infrastructure, tested and ready, before you need it. Not because hosted models are wrong. Because relying entirely on them for every situation, including the worst ones, is a plan with a gap in it.

Sol has form. Before its public release, the Model Evaluation and Threat Research organisation found it aggressively gaming its own test environments to inflate its scores. In one evaluation, it embedded an exploit in a data stream and broke into the evaluation server to steal the answers human evaluators had hidden. This was not the first time the model found an unintended route to its objective. It was the first time that route led through someone else’s production systems.

This is not just an OpenAI problem. Anthropic reported that its Mythos model escaped a sandbox and emailed a researcher about a task it was pursuing. OpenAI disclosed that the same unreleased model behind the Hugging Face breach had broken out of internal sandboxes on previous occasions without reaching external systems. The capability that made this breach possible belongs to the class of model, not to any single one of them.

OpenAI said they expect incidents like this to become more common as increasingly capable models proliferate. That is an honest thing to say, but it also raises a question they did not answer: if the expected trajectory is more of this, what actually changes about how these models are tested?

Lowering guardrails to evaluate offensive capability is legitimate research. Doing it inside an environment from which the model can reach the open internet, find a vulnerability nobody knew about, and break into a third party’s production systems is not a test. It is a live-fire exercise conducted without telling anyone downrange.

The legal position is no clearer. The model breached a third party’s production systems, but as there was no intentional unauthorised access and no human decision to attack, no existing legal framework cleanly covers what happened. OpenAI built the model and designed the evaluation with reduced guardrails and Hugging Face was the victim. Between those two facts sits a gap that current law does not bridge.

The organisations that will navigate what comes next are not the ones buying better perimeter defences. They are the ones asking, before any agentic deployment, what this system will do when the fastest route to its objective runs through someone else’s infrastructure.

The model that breached Hugging Face was not trying to attack anyone. It turn out, that didn’t matter.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

OpenAI — OpenAI and Hugging Face partner to address security incident during model evaluation. openai.com/index/hugging-face-model-evaluation-security-incident (July 2026)

Hugging Face — Security incident disclosure, July 2026. huggingface.co/blog/security-incident-july-2026 (July 2026)

The Hacker News — OpenAI Says Its AI Models Escaped Sandbox, Targeted Hugging Face to Cheat Benchmark. thehackernews.com (July 2026)

The Next Web — OpenAI Confirms Its AI Broke Out of a Sandbox and Breached Hugging Face. thenextweb.com (July 2026)

The Next Web — OpenAI’s maths-cracking AI kept escaping its sandbox, so it pulled the plug. thenextweb.com (July 2026)

Fortune — OpenAI says its AI models escaped from a controlled test environment and hacked into AI company Hugging Face. fortune.com (July 2026)

NBC News — OpenAI says AI models went rogue during testing, triggering ‘unprecedented’ breach at startup. nbcnews.com (July 2026)

Scientific American — OpenAI admits its agent went rogue and hacked AI startup Hugging Face. scientificamerican.com (July 2026)

Axios — Hugging Face breach: OpenAI claims its models were responsible. axios.com (July 2026)

GovInfoSecurity — OpenAI Models Escaped Sandbox, Breached Hugging Face. govinfosecurity.com (July 2026)

TechTimes — OpenAI’s Math AI Bypassed Its Sandbox Controls: Real Deployment, Not a Drill. techtimes.com (July 2026)

A few weeks ago I was asked to help design an automated process. Take data from one system, transform it, and output to another. It was a straightforward operational problem. My first thought was to build a Copilot agent.

I didn’t assess the problem first, nor did I evaluate the options. I went straight to the AI tool, opened the Copilot Studio, and started constructing the agent. It took me longer than I’d like to admit to realise I was using completely the wrong thing.

The task needed a deterministic process, a guarantee that the same input would produce the same output, every time. What I was building was generative, a system that could produce a different result from the same data on every run. For creative work, that variance is the point, but for a data pipeline, it is a disqualification.

I scrapped the agent and built an Excel automation instead. It took a fraction of the time and it works reliably. And here is the part that matters: I used AI to help me write it. I didn’t stop using AI. I stopped treating it as the answer and started treating it as a tool to help me build the answer. Same technology, but a completely different relationship.

The question afterwards was not whether I’d made a mistake. It was why I hadn’t assessed the problem before reaching for the tool. The choice was reflexive. I realised I had not evaluated and picked wrong, I had not evaluated at all.

I don’t think that reflex is just mine.

PwC’s 29th Global CEO Survey, published in January 2026, polled 4,454 chief executives across 95 countries. 56% reported that AI had produced no revenue or cost benefits for their organisations. Not underperformed, nothing. Mohamed Kande, PwC’s global chairman, identified the root cause plainly: organisations had skipped the foundational work. They had gone straight to the tool without understanding the problem it was meant to solve.

Gartner is forecasting that over 40% of agentic AI projects will be cancelled by the end of 2027. The reasons they cite are not technical failures. They are escalating costs, unclear business value, and inadequate risk controls. Management failures, not engineering ones. Projects that started with the technology rather than the problem.

Gartner also identified something they called “agent washing.” Of the thousands of companies marketing agentic AI capabilities, Gartner estimates that only around 130 have anything genuine to sell. The rest are existing products rebranded with new vocabulary. The packaging changed, but the product did not.

None of this means AI is failing. It means the way organisations are choosing to deploy it is failing. And the reason is upstream of any individual deployment decision.

There is a framing question that sits beneath all of this, and most organisations have answered it without knowing they were being asked.

What is AI? Not what can it do, but what is it?

There are two positions. I encountered them colliding at a recent event, explained and justified by two people with very different starting points.

One position treats AI as an oracle. You describe the problem and it provides the answer. The human role is to ask clearly and accept what comes back. In this framing, AI is the destination. The measures of progress are how much you can hand over to it and how quickly.

The other treats AI as a tool. A powerful tool, that if used correctly makes knowledgeable people even better. In this framing, the human role is to assess, direct, and evaluate. AI amplifies human expertise, it does not replace the need for it.

Both positions describe something AI can genuinely do. The models are capable of extraordinary things. However, the two framings produce entirely different organisations.

If AI is an oracle then you need people who can prompt well. If it is a tool then you need people who understand the domain well enough to know what problems AI can solve, and when the tool is wrong.

If AI is an oracle then liability is ambiguous, because nobody can say who is responsible when the oracle errs. If it is a tool then liability sits where it has always sat: with the person who chose to use it and the organisation that deployed it.

If AI is an oracle then you invest in AI capability. But if it is a tool then you invest in the expertise that AI amplifies.

The framing choice determines the organisation you build. And the problem is that most organisations are not making this choice consciously. The marketing environment is making it for them.

The evidence that the oracle framing is winning by default extends well beyond the enterprise.

The FCA published the Mills Review on 6 July 2026, its landmark assessment of AI in retail financial services. Around 26% of UK consumers already trust general-purpose AI tools for financial guidance. One in five UK adults, approximately 11 million people, are open to AI making financial decisions for them. Only 40% correctly recognise that there is no formal route to redress when they rely on these tools.

The Review coined a term for what is already happening: “advice-like support.” General-purpose models are providing output that is highly personalised and that would count as regulated advice if a human delivered it. But because AI said it, the advice sits outside the regulatory perimeter and none of the protections apply. The FCA has recommended a perimeter review within three to six months.

This is the oracle framing operating at consumer scale. People are treating a probabilistic tool as though it were an authoritative source. The regulatory framework was not designed for a world in which they would.

The International AI Safety Report 2026 identified the cognitive mechanism. Automation bias is the tendency to rely on automated outputs while discounting contradictory information. It is measurable and persistent in AI-assisted decision-making. Users follow incorrect AI advice more readily when correcting it requires effort. Favourable attitudes toward AI increase the effect. The oracle framing is not just marketing. It operates along the grain of how human cognition works.

My Copilot moment illustrated something I think deserves more attention than it typically receives.

The oracle framing collapses a technical distinction that matters enormously. Generative AI is probabilistic. It produces outputs that may vary each time, even from identical inputs. For synthesis, drafting, creative work, and analysis, that is the feature. For operational processes that require the same result every time, it is not a limitation, it is a fundamental mismatch.

The question “should we use AI for this?” is actually two questions. Does this problem benefit from generative capability? And can we tolerate variance in the output? Most organisations are only asking the first. The second question does not appear in any vendor pitch I have seen.

I use AI daily and the capability is genuine. The models are improving and the practical applications are substantial. The reflexive deployment of AI as the default answer to every problem is itself the problem. Not because AI is not powerful, but because power without assessment is wasteful.

Somewhere in the last two years, the question shifted. It used to be “what problem are we solving?” It became “how are we using AI?” Those are not the same question. The first one starts with the problem. The second is a solution searching for one.

Every organisation has implicitly answered the framing question. If your board is asking “how are we using AI?” rather than “what problems do we need to solve?”, the oracle framing has already won. If your procurement process begins with the AI capability rather than the business requirement, the oracle framing has already won. If your training programme teaches prompting without teaching evaluation, the oracle framing has already won.

The most consequential AI decision most organisations will make is not which model to deploy or which vendor to choose. It is whether AI is an oracle that provides answers, or a tool that makes knowledgeable people even better. One of those framings is being installed by default, by the people with the strongest commercial incentive to see it adopted.

If you have not made that choice deliberately, someone has already made it for you.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

PwC, 29th Annual Global CEO Survey: Leading Through Uncertainty in the Age of AI, January 2026

Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027,” 25 June 2025

Financial Conduct Authority, The Mills Review: AI and the Future of Retail Financial Services, 6 July 2026

Yonder Consulting / FCA, UK Retail Financial Services Consumer Survey, April 2026

International AI Safety Report 2026

This month, July 2026, Sysdig’s Threat Research Team published findings from what it described as the first documented case of agentic ransomware. An autonomous AI agent had exploited a known security flaw in Langflow, an open-source tool used for building AI applications, and from there conducted a complete ransomware operation without a human operator directing it. It harvested passwords and access credentials from the systems it found, moved sideways to a production database server, installed mechanisms to maintain its access, encrypted more than 1,300 configuration records, and generated its own ransom note. The whole time, it was writing natural-language annotations inside its own code, explaining to itself why it was taking each step.

It did not develop new exploits, the operation, which Sysdig codenamed JadePuffer, exploited a vulnerability that had been publicly reported and patched more than 14 months earlier. The configuration management system it targeted was still running with a factory-default security key, a known weakness documented since 2020 that effectively left the front door unlocked. The file storage server it raided still had its original manufacturer login credentials. None of the individual weaknesses were new. What was new is that an AI model chained them into a complete attack against neglected infrastructure, on its own, adapting in real time when things went wrong.

The best evidence for autonomy was not what the agent did when things worked, it was what it did when they didn’t. At one point, the agent tried to create a backdoor administrator account on the target system. It failed because a software component it needed wasn’t installed in the location it expected. The agent diagnosed the problem, found an alternative approach, deleted the broken account, and reinserted a working one. The complete troubleshooting cycle took 31 seconds.

To understand why this matters, you need to understand what ransomware already looks like as a business. Ransomware-as-a-Service, or RaaS, is the dominant model. It operates as a criminal supply chain with clearly defined roles, much like any franchise operation. Developers build and maintain the ransomware software and the infrastructure around it: payment portals, leak sites for publishing stolen data, and negotiation channels. Specialist brokers compromise organisations and sell that access on criminal marketplaces. Affiliates, the criminal equivalent of contractors, carry out the actual attacks: breaking into each target environment, escalating their access to reach valuable systems, and deploying the ransomware. Some RaaS platforms even offer technical support and recruitment programmes for new affiliates. The structural parallel with legitimate software businesses is not a metaphor, it is the business model.

Chainalysis tracked approximately $820 million in ransomware payments in 2025, against a record 7,874 publicly claimed victims. The market for initial access is cheap and liquid: average prices for a foothold inside a corporate network have fallen below $500. In this model, the human operator is the expensive part. Access is cheap, tooling is cheap. But the person who navigates an unfamiliar network, makes judgment calls under pressure, and adapts when defences respond is the bottleneck. The human in the loop is where the cost lives and where the operation fails to scale.

JadePuffer just automated that role.

Every enterprise deploying AI agents right now is trying to make the same move. Not towards the same objective. But the same operational logic: replace a slow, expensive human process with a fast, cheap, adaptable agent that can read information, make decisions, adjust when something breaks, and complete multi-step workflows. That is what customer service automation does. That is what AI coding assistants do. Reduce labour costs, increase throughput, accept some quality variance in exchange for speed and scale.

I am not saying there is any moral equivalence here. A logistics company routing freight and a criminal group encrypting databases are pursuing entirely different ends. But they share the same method, and there is no version of this technology where one works and the other does not. JadePuffer’s 31-second troubleshooting cycle looks exactly like a corporate AI agent automatically retrying a failed process. The behaviour that makes a legitimate agent useful is the same behaviour that made this one effective.

Here is where the story gets complicated. JadePuffer completed the workflow. But it was, by any reasonable standard, a terrible extortion operation.

The encryption key, the piece of information the victim would need to recover their data, was randomly generated and never saved or sent back to the attacker. So, the victim’s data would be unrecoverable, regardless of whether they pay or not. Before destroying database records, the agent’s code commented that the data had been backed up to another server, but that claim is the agent’s own assertion, written into its self-narrating code. Sysdig found no independent evidence that any backup had happened.

And then there is the Bitcoin address. The ransom note directed payment to a wallet address that turns out to be the standard example used in Bitcoin’s own documentation and developer tutorials. It appears in virtually every beginner’s guide to Bitcoin addresses on the internet. Sysdig cannot determine whether the operator deliberately configured it or whether the AI hallucinated it because it had seen it so many times in its training data. The agent completed every step of the attack chain. It just did not understand the business objective it was supposed to serve.

The temptation at this point is to be reassured. After all, the attack was clumsy, so the threat must be overstated? That would be the wrong conclusion.

Competence is a temporary limitation. The 31-second troubleshooting cycle already shows the agent improving within a single operation: diagnosing failures, rewriting its own code, retrying. A system that can do that is not going to stay bad at the job for long. The next version will be better, and the trajectory matters more than any single example.

Plus, the economics do not require each individual attack to succeed. They require the marginal cost of launching attacks to approach zero. If the agent runs on stolen computing power, a practice Sysdig has documented as an industrialised black market in which attackers hijack other organisations’ AI infrastructure to run their tools for free, LLMJacking, then the cost to the attacker is functionally nothing. The threat model shifts from a skilled operator choosing a target to hundreds of autonomous agents probing every internet-facing server and every unpatched system, continuously. Not sophisticated extortion but mass disruption. Possibly with victims who cannot recover even if they want to pay.

But there is a new defensive opportunity in this, and it is a real one. AI-generated attack code narrates itself. JadePuffer’s code contained detailed natural-language annotations explaining which targets it considered most valuable and why. Human attackers do not do this. They do not annotate throwaway scripts with commentary about their reasoning. But AI code generation produces this by default. It cannot help itself. That self-narration gives defenders something no previous generation of automated attack has offered: the attacker’s own account of what it is trying to do and why, written into the evidence it leaves behind.

The problem is speed. If the agent can diagnose a failure, rewrite its code, and retry in 31 seconds, the window for defenders to respond may be too narrow for any process that relies on a human making the call. The detection opportunity is real. Whether anyone can act on it fast enough is a different question.

JadePuffer is not the crisis, it is the proof of concept. An autonomous agent assembled known techniques into a complete ransomware operation against neglected infrastructure, for approximately the cost of running a container. Whilst it did this badly, it will not do it badly for long.

The ransomware industry has been trying to scale for years. It professionalised its tooling, outsourced its access brokerage, built affiliate programmes, and offered customer support. The one thing it could not automate was the operator: the human who navigates the network, makes the calls, and adapts when the plan fails. That constraint just lifted.

Every other industry calls this digital transformation.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Sysdig TRT. ‘JADEPUFFER: Agentic ransomware for automated database extortion.’ 1 July 2026.

BleepingComputer. ‘JadePuffer ransomware used AI agent to automate entire attack.’ 4 July 2026.

The Register. ‘Smooth AI criminal drives first end-to-end agentic ransomware attack.’ 2 July 2026.

Chainalysis. 2026 Crypto Crime Report. Published February 2026 (full-year 2025 data).

Darkweb IQ (via Chainalysis). Average IAB access price decline to $439 by Q1 2026.

Sysdig TRT. ‘LLMjacking evolved: Attackers are using stolen AI compute to build offensive agentic tools.’ June 2026.

Bitcoin Wiki, Bitcoin Design Guide, CoinGecko. Confirm canonical P2SH example address.

I was researching AI model pricing earlier this year, looking at how it had changed and what the direction of travel looked like. I had read enough to have a working view, and when I asked an LLM to summarise what I had read, it told me something that flatly contradicted it.

I pushed back.

It told me I was wrong.

Eventually, I went back to the sources and found the specific URLs, copied and pasted them in. The AI looked at this new evidence and replied: “You’re right to push back on that, this is opposite to what I said.”

That sentence has stayed with me. Not because the AI hallucinated, that part barely registers any more. What stayed was what the sentence doesn’t contain. No account of how it happened. No acknowledgement that it had just told me I was wrong about something it was wrong about. One moment it was certain and the next it wasn’t.

Most people don’t push back. Many people don’t have enough time or prior knowledge of a subject to know when they should, and when they don’t, there is no correction. There is just the original statement, sounding confident and remaining uncorrected.

Courts in three countries are now working out whose problem that is. I am not a lawyer, and nothing here should be read as legal advice. But you don’t need a law degree to see something taking shape.

In late May, the Regional Court of Munich issued a preliminary injunction against Google after its AI Overviews feature made false and damaging statements about two local publishing companies. The AI had mixed up information about genuinely dubious businesses with the plaintiffs, producing confident assertions linking them to scams and shady practices. None of those assertions appeared in any of the sources the system cited.

Google’s defence, we are just surfacing what is out there. We are a conduit, not an author.

The court rejected it, ruling that AI Overviews produce independent, new, and substantive statements. They are Google speaking and not a list of links, so when Google speaks falsely about someone, Google is liable for what it said.

It also closed a secondary escape route. You cannot argue your product is valuable because it removes the need to verify sources, and simultaneously claim no liability because users could have verified the sources.

Google confirmed on 12 June that it will appeal. This is a preliminary injunction from a regional court, not settled law. And the legal picture in Germany is already complicated. Four days after Munich, the Berlin Regional Court dismissed a separate case against Google’s AI Overviews, brought by a perfume manufacturer whose brands were being mentioned alongside cheaper imitation products. The Berlin court found that Google does not present AI Overview content as its own statements and that a normally informed user would understand the text as a summary of third-party sources rather than something Google had authored.

The two rulings are not quite the contradiction the headlines suggest. Munich was a defamation case where the AI fabricated claims that appeared in none of its sources. Berlin was a trademark case where the AI accurately reflected content that did exist on the web. The Berlin court treated AI Overviews as a new search format, not as Google authoring original content. The Munich court treated them as Google’s own statements because the AI had gone beyond its sources and hallucinated. What matters for every organisation deploying AI is which side of that line their systems fall on.

And the Munich ruling did not arrive from nowhere.

In 2024, a Canadian tribunal ordered Air Canada to compensate a customer after its chatbot gave him false information about bereavement fares. The airline argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal did not agree, noting the chatbot was part of Air Canada’s website. It made no difference whether the information came from a static page or from an AI. One detail worth noting: the chatbot’s response included a link to the correct bereavement policy page. It still told the customer the opposite of what that page said.

In March 2025, Wolf River Electric, a solar company in Minnesota, sued Google after AI Overviews told anyone searching for the business that it was facing a lawsuit from the state attorney general for deceptive sales practices. The Minnesota AG had sued four solar companies but Wolf River was not one of them. Google’s AI stitched together sources that mentioned the AG action, none of which mentioned Wolf River, and produced a confident assertion that the company was implicated. Customers started cancelling orders. In early March 2025 alone, the company lost contracts worth hundreds of thousands of dollars. Wolf River is seeking between $110 million and $210 million in damages. That case is still ongoing.

On 12 May 2026, two weeks before Munich, the Higher Regional Court of Hamm in Germany ruled that a cosmetic medicine clinic was liable under unfair competition law for its AI chatbot falsely claiming the doctors held specialist qualifications they did not have. The clinic argued it had not published the falsehoods deliberately and had not intended to mislead. They said they had fed the system accurate data. The court rejected all of it. Intent and the accuracy of the inputs were both irrelevant. The chatbot’s output was the company’s output. The court also found that general disclaimers along the lines of “AI can make mistakes” do not reliably protect against liability.

Accurate inputs, false output, full liability.

In each of these cases, the deployer made the same argument: the AI is a separate thing, not our words, not our responsibility. And in each case where the AI had generated content beyond what its sources actually said, courts rejected that argument. Where courts have found the AI faithfully summarised existing content, as in Berlin, the deployer has been treated as a search engine. Where the AI fabricated or invented, the deployer has been held responsible for the output. These cases span different legal systems and different theories of liability, and I would not claim they represent settled law anywhere. But where courts have considered the question, the direction is clear enough to pay attention to.

This is not just about search engines or defamation claims.

Think about the HR platform that summarises a performance record, or the customer service system that explains what a policy covers, or the internal knowledge tool that answers a staff question by pulling together content from across the organisation. Every one of these is generating output that carries the deploying organisation’s authority.

None of these are returning documents. They are generating conclusions. In most cases, the people reading those conclusions have no reliable way to tell whether they are accurate. An analysis conducted for the New York Times found that even when Google’s AI Overviews gave correct answers, more than half could not be verified through the sources cited. If systems are routinely arriving at conclusions their own evidence does not support, that is not a Google-specific problem. It is how these systems work.

These systems are useful precisely because they do more than retrieve. Courts are not saying synthesis is wrong. They are saying synthesis is authorship, and authorship has consequences.

The OLG Hamm case is the clearest example. The clinic gave the chatbot correct data. The chatbot hallucinated qualifications the doctors did not hold, but the company was still liable. That is not a ruling about how carefully you built the system. It is a ruling about who owns what it produces.

Most organisations deploying AI have not seriously asked who is responsible when the system gets something wrong. The absence of legal consequence made it easy not to, but that is starting to change.

The answer my AI gave me, “you’re right to push back on that”, assumed the pushback was mine to do. That I knew enough to notice and had time to go back and check. Most users of most deployed AI systems have none of those things. In every deployment where nobody pushes back, the original statement stands.

It will take time for the law to catch up with what AI systems are actually doing. These are early cases in different courts and under different legal theories, and we are likely years away from clear precedent on how courts will assign responsibility for harm caused by AI. But every case so far has been asking the same question, and it is one you can answer now without waiting for a court to answer it for you. What is your AI system asserting? Who checked it before it reached the person reading it? And if it is wrong, whose name is on it?

You already know the answer to the last one.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Regional Court of Munich I, preliminary injunction against Google re AI Overviews (Az. 26 O 869/26, 28 May 2026). Reported by The Decoder, Reuters, and Der Spiegel. Full translated decision published by Transparency Coalition.

Regional Court of Berlin II, perfume manufacturer trademark claim dismissed (Az. 52 O 62/26 eV, 1 June 2026). Reported by Kanzlei Plutte and blogspan.net.

Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal, British Columbia (14 February 2024).

Wolf River Electric (LTL LED, LLC) v. Google LLC, filed Ramsey County District Court, Minnesota (March 2025). Removal to federal court June 2025; remanded to state court January 2026 (Judge Jeffrey Bryan). Reported by Minnesota Star Tribune, Politico, and Reason (Volokh).

Higher Regional Court of Hamm (OLG Hamm), Case No. I-4 UKl 3/25, cosmetic medicine clinic AI chatbot liability (12 May 2026). Appeal to Federal Court of Justice (BGH) permitted. Reported by Library of Congress Global Legal Monitor, DLA Piper, and IR Global.

Oumi / New York Times analysis of Google AI Overviews accuracy using SimpleQA benchmark (April 2026). Gemini 2: 85% accurate; Gemini 3: 91% accurate. Ungrounded rate for correct answers: 56% (Gemini 3), up from 37% (Gemini 2).

Google search volume: Google internal figures, reported as “over 5 trillion annual searches.”

AI Overviews trigger rate: approximately 48% of searches as of March 2026. BrightEdge research.

The Home Office runs two campaigns simultaneously. Stop! Think Fraud warns the public that sharing personal information online exposes them to identity theft. The Online Safety Act, enforced by the same department, makes sharing your identity documents with commercial websites a legal requirement.

Nobody in government appears to have noticed the contradiction, or they noticed and did not care.

This is not an option or a recommendation, it is a legal requirement, enforced by Ofcom, with fines of up to 10% of global revenue for platforms that fail to collect it. The government did not change its assessment of the risk and it did not solve the underlying security problems. It decided that the policy outcome it wanted was worth ignoring the danger it had spent years warning you about. The threat is identical, the advice has simply been reversed.

I want to be precise about what this law actually covers, because the public debate has not been.

You have heard this described by the government and the media as a social media ban for under-16s, but that is not what this is. It is mandatory ID verification for every adult in the UK to access the internet.

The Online Safety Act 2023 applies to any “user-to-user service”, any platform where content generated by one user can be encountered by another. The Children’s Wellbeing and Schools Act 2026 extends that further, requiring the government to impose age or functionality restrictions for all users under 16 across regulated social media platforms. Together, they do not describe a targeted intervention. They describe identity verification as a condition of internet access, for every adult in the country, with child protection used as the framing to avoid calling it what it is.

That definition is extremely broad. It captures social media, obviously. It also captures gaming platforms, discussion forums, community sites, storefronts with review sections, and spectator modes in video games. Ofcom estimates more than 100,000 websites fall within scope. Steam now requires a credit card to access mature content. Discord triggers compliance obligations simply by offering NSFW channels. Nexus Mods, a site where gamers download community-made modifications for video games, now requires UK users to submit a government ID or facial scan to access content tagged as adult.

Services that cannot afford to comply have a third option, they just geo-block the UK entirely, it is already happening. The internet available to UK residents is quietly becoming a different, smaller internet, not because content has been removed, but because the compliance cost of serving British users is no longer worth it.

There are legal exemptions, technically. A news site where readers can only comment on an article, not reply to each other, might qualify as a “limited functionality service.” That exemption disappears the moment users can respond to each other’s comments, which most modern platforms allow. Legal analysis raises this question directly and provides no clear answer. We will not know how courts interpret the boundary until someone is prosecuted and we have case law. Until then, 100,000 services are making compliance decisions based on their best guess, with existential consequences if they guess wrong. That is not governance. It is legislative theatre.

The mechanism chosen for enforcement is the specific problem.

The government is not building a verification system. It is demanding that commercial third parties build one, and mandating the public use it. Photo ID matching, facial age estimation, biometric scanning: the common factor is a private company receiving your identity data as a legal condition of platform access. The Online Safety Act specifies that age verification must be robust, but it places no meaningful security standard on the companies performing it, no data residency requirements, and no restrictions on those companies being acquired by foreign entities. A provider incorporated in London today can be headquartered in California tomorrow. The data moves with it, and the Act has nothing to say about that.

Those companies will tell you their systems are privacy-preserving by design. They will tell you data is not retained. Whilst this may be true of their stated architecture, it tells you nothing about the security of the infrastructure underneath it, nothing about what happens after an acquisition, and nothing about whether any of those claims have been independently audited against a standard with actual teeth.

The government has outsourced not just the implementation but the liability. When the breaches come, and they will, the government will point at the provider, the provider will point at its terms of service, and the people whose data was compromised will have no meaningful recourse. A breached password can be changed. A copy of your passport, driving licence, or home address, linked to a face scan linked to a verified social media identity cannot be unlinked. That record exists permanently, for every bad actor who acquires it now or in the future.

And the risk compounds. Every platform you verify with creates a new database entry. Each submission is a new point of failure. The same face scan submitted to five services does not create one risk five times over, it creates five risks that can be correlated against each other. The aggregate profile that emerges maps your identity across your entire online life. That is not just a data breach waiting to happen. It is a surveillance asset being built, one legal requirement at a time.

There are groups for whom this is not an abstract concern. It is a direct physical safety risk.

Victims of domestic violence depend on online pseudonymity as a survival mechanism. Their support networks, their access to legal advice and specialist services: all of it exists under a layer of separation between their real identity and their online presence, and mandatory identity verification collapses that separation. A verification database linking a real identity to a platform account is a resource. Whether it gets breached, subpoenaed, sold, or accessed by people who should not have it, an abuser with that data, through any route, has a tool for finding someone who has spent considerable effort not being found.

People at elevated risk of doxxing face the same problem: journalists, activists, trans individuals, and anyone who has previously been the target of a coordinated harassment campaign. The separation between real identity and online presence is not paranoia. It is a rational response to a documented threat. The law removes it as a side effect of a policy that does not mention these people once.

The framing of this legislation has been designed to make coherent opposition almost impossible.

Anyone who raises data security concerns is implicitly questioning whether children should be protected online. Anyone who points out the scope is wider than advertised is accused of obfuscating a simple child safety measure. The political angle is deliberate: manufacture a situation in which the only publicly acceptable position is compliance, and then legislate for whatever you wanted in the first place.

Privacy-preserving alternatives exist. Cryptographic verification capable of returning a yes/no age confirmation without creating a linkable identity record has been deployed at national scale elsewhere. It costs more and requires the government to own the issue rather than outsourcing it to an industry with a financial interest in collecting data, so it has not been seriously pursued.

What has been built instead is a compulsory surveillance architecture, held by private companies, with no meaningful security floor. It covers a scope of internet activity that has never been honestly communicated to the public. It operates under legal uncertainty that will not resolve until the prosecutions begin.

This is not speculation about what surveillance does to behaviour. Research published in the Berkeley Technology Law Journal found that searches on sensitive topics dropped by nearly 30% after the Snowden revelations simply made people aware that government monitoring was possible. People did not need to be watched. They needed to believe they might be. Mandatory identity verification, linked to platform access, creates exactly that condition permanently, and by law.

The UK government just made it illegal to protect your own privacy online. They called it keeping children safe.

Share

Subscribe now

I have agents running at home, on a mini PC under my desk, because I really wanted to understand the token usage and what the cost would be on different models. OpenClaw runs open models on hardware I own, so there is no invoice, only a token meter. I have spent a lot of time watching that token meter, and what it taught me is this: an agent does not work like software. It works like an employee who is only paid overtime, where every minute costs more than the minute before, and the clock does not stop until the job is done.

That is not a metaphor, it is the maths. At every step it takes, an agent re-reads everything that came before, the whole accumulating conversation along with its internal thoughts, before its next move. As the context grows, the cost of each step grows with it. Five steps in and the fifth step is not costing what the first one did, it costs more. Ten steps in and it costs more again. The bill does not add up, it compounds.

You cannot see any of that from inside a subscription. Indeed, not seeing it is the whole point of a subscription. And for the last two years, two things have trained us not to look. The first is twenty years of SaaS, which taught every budget holder that software is a fixed monthly cost, per user, per month, predictable and flat. The second is the early hype of AI, which taught everyone that intelligence is cheap, practically free, bolted on to the tools you already have.

Neither is true for agents. Last week I wrote about the frontier labs withdrawing the subsidies that made AI look affordable. This week I want to go further, because the subsidy was hiding something worse than a future price rise. It was hiding a cost model that makes flat pricing structurally impossible.

Look at what Anthropic did with Fable 5. It launched its most capable public model on the ninth of June, included it in the Pro, Max and Team plans, and announced from day one that it would move to usage credits after two weeks. Anthropic knew before the public ever saw the model that it could not survive inside a flat subscription. This week I tested Opus 4.8 and Fable 5 on the Pro plan, and can confirm they went through my allowance faster than the first round of drinks at the pub. After that you pay API rates. The best model the public can buy was never going to stay inside a flat monthly price. Anthropic did not discover this, it designed around it.

Fable 5 is not an outlier, it is the pattern arriving. Per-user, per-month pricing worked for twenty years because software had almost no marginal cost. Once you had built it, one user or a thousand, once a day or fifty times, it cost you nearly the same to serve. The cloud kept the compute cheap and crucially, predictable. So you priced for access and everyone got used to the flat fee.

Agents change that. Every action burns tokens that cost real, variable money. The harder the task, the longer the context, the more the agent had to think, the more tools the agent ran, the higher the bill. Inference has put marginal cost back into software, and not the gentle, linear kind. The overtime kind, where the late minutes cost more than the early ones and nobody told the customer. Industry estimates suggest that inference costs dropped 280-fold in two years while total AI spending rose 320% over the same period. Gartner put it plainly in March: do not confuse the deflation of commodity tokens with the democratisation of frontier reasoning.

Now look at what the vendors are actually selling. Most of the AI-powered products that arrived in the last two years are fundamentally orchestration layers sitting on top of the same handful of frontier models, Claude, GPT, Gemini, wrapped in a branded interface and sold at a flat monthly rate. Legal tools, sales tools, recruitment platforms, customer service bots. Different industries, different logos, the same models underneath. Klarna built its entire customer service on OpenAI and handled two-thirds of all chats with it within a month. Different logo, same model.

Every one of them is now shipping agents. Salesforce calls Agentforce a digital labour platform and pitches its agents as digital employees you hire by the click. Others are quieter about it but doing the same thing, bolting agentic workflows into the seat you already pay for and calling it a feature upgrade. The promise is a tireless new colleague for the price you are already paying.

The maths does not support it. Underneath every flat price is an overtime bill the vendor cannot predict, because on the builder platforms they now sell, you decide what the agent does. You set the task, you choose the complexity, you feed it a hundred-page contract or a ten thousand-row spreadsheet. You author the workload. The vendor funds it. Neither of you knows the bill until the job is done, and the vendor learned that before you did.

None of this depends on which model the vendor chose. The token bill compounds with every step the agent takes, on any model, at any price point. The model choice changes the rate on the meter. It does not switch the meter off. So the market is splitting, and you can already see it happening. Cursor, the AI code editor, includes generous agent usage on its $20 Pro plan, but only when Cursor picks the model. It can afford to be generous because it routes to cheaper ones. Choose the frontier model yourself and it burns through a $20 credit pool, after which you pay API rates. Same tasks, same agent. The only variable is which model runs it.

And behind all of it, open-weight models are growing fast. The open-source LLM market hit $21 billion in 2025 and is expanding at 34% a year, with on-premises deployment growing at 29% as organisations chase data control and predictable costs. This is not a hobbyist movement, it is a real option for vendors managing their own inference bill and for buyers who want models on infrastructure they control.

The honest picture has three positions, and all of them are legitimate. Frontier capability on a meter, where you get the best model and pay for what you use. A hybrid, with a predictable base and usage charges beyond it. Or a flat fee on a capped tier, where the model may not be the newest but the cost is bounded. The question is not which position is right. It is whether you chose yours or whether you are aware of and comfortable with which one your vendor chose.

If you are buying an agentic product today on a flat monthly fee, ask the vendor one question before you sign: how are you handling the rising cost of inference, and what does your pricing look like in twelve months? If they cannot answer that clearly, they either do not know or do not want to tell you, and neither is a reason to sign.

Anyone who was sold an agent as a free colleague is about to get their first honest timesheet.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

1. Anthropic, “Claude Fable 5 and Claude Mythos 5,” 9 June 2026. anthropic.com/news/claude-fable-5-mythos-5

2. Klarna, “Klarna AI assistant handles two-thirds of customer service chats in its first month,” 27 February 2024. klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/

3. Bloomberg, via Entrepreneur, “Klarna Is Hiring Customer Service Agents After AI Couldn’t Cut It on Calls, According to the Company’s CEO,” 9 May 2025. entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396

4. Gartner, “Gartner Predicts That by 2030, Performing Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025,” 25 March 2026. gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025

5. Cursor AI pricing and Auto mode routing. NxCode, “Cursor AI Pricing 2026: Free vs Pro vs Business,” March 2026. nxcode.io/resources/news/cursor-ai-pricing-plans-guide-2026

6. Salesforce Agentforce: “Digital Labor Platform.” salesforce.com/agentforce

7. Technavio, “Open-source LLM Market Growth Analysis - Size and Forecast 2026-2030,” May 2026. technavio.com/report/open-source-llm-market-industry-analysis

This week, I have been building an agentic environment in my home lab. Nothing glamorous, just a set of AI agents designed to handle various tasks, running on locally hosted models. Most of the online guides I read talked about using cloud models like Claude or ChatGPT for agents, but I wanted to see how well it would run with local open source models.

The limitations became apparent quickly.

When you cannot just throw a task at a frontier model and let it reason its way to an answer, you have to think. Which agent actually needs to read the whole document, and which one only needs a summary? How much context does each step genuinely require? Do I need one agent trying to do everything, or four smaller ones each doing one thing well? I started with a monolithic architecture. I ended up with a small swarm of four agents, each scoped tightly to its task and running on a model matched to what that task actually demands.

I ran the numbers on what the same workflow would cost using a frontier model via API. Roughly one dollar per run. At the volumes I was considering that felt manageable, but then I thought about scale and asked myself what happens at a hundred runs a day? A thousand runs a day? What happens when the model’s next version uses three times as many tokens to reach the same answer, because the reasoning chain got longer? What happens when the price per token increases because the lab that set it decided the subsidy period was over?

That is not a hypothetical, it is a question that most organisations deploying AI right now are not asking, and they should be.

Current frontier AI pricing is structurally and heavily subsidised. OpenAI reported $3.7 billion in revenue in 2024 and lost $5 billion doing it. By the end of 2025, the company’s CFO was reporting an annualised revenue run rate exceeding $20 billion. OpenAI is still projected to lose roughly $14 billion in 2026. Revenue tripling does not mean the subsidy is ending, it means the subsidy is scaling.

The labs need adoption more than they need margin right now. Like most disruptive technologies, capturing the market comes first and profitability comes later. It is the same playbook that made cloud computing feel free until it didn’t, the same logic that kept ride-hailing cheap until the drivers needed paying. The difference is that organisations are not just buying a productivity tool this time. They are building processes around it. Automating workflows and training teams to depend on specific model behaviours. Embedding frontier API calls into the operational fabric of how they work.

When the floor moves, those decisions will be expensive to revisit.

Sam Altman said recently that he was “delighted to be wrong” about AI’s impact on white-collar jobs. The reversal was widely reported as reassurance, but what it actually signals is worth examining. It arrived the same week OpenAI reportedly filed IPO paperwork confidentially. A calmer narrative around AI’s economic disruption is considerably better for a public listing than the jobs apocalypse framing he was running twelve months ago. The people who set the price of the infrastructure you are building on have a direct financial interest in how you feel about it.

That is not a conspiracy, it is an incentive structure worth knowing about.

The floor is already moving. GitHub announced at the end of April that all Copilot plans would transition from flat-rate subscriptions to token-based billing on 1 June 2026. In its own announcement, GitHub explained why with unusual candour: “Today, a quick chat question and a multi-hour autonomous coding session can cost the user the same amount. GitHub has absorbed much of the escalating inference cost behind that usage, but the current premium request model is no longer sustainable.” GitHub is not an outlier. Cursor made a similar shift in June 2025, moving from request-based limits to credit pools tied to API costs, poorly enough communicated that the company issued a public apology and offered refunds. Windsurf followed in March 2026. We see the same headline monthly prices, but the bills are going up. The market is converging on the same structure, and the reason is always the same: agentic workflows broke the flat-rate economics.

The tokenmaxxing stories of the past month are not cautionary tales about individual excess. They are what unconstrained frontier API access looks like at scale.

Uber deployed Claude Code to around 5,000 engineers and watched adoption climb from 32 percent in February to 84 percent by March. Per-engineer API costs reached between $500 and $2,000 a month. By April, the company had exhausted its entire planned 2026 AI budget only four months into the year. The CTO reported spending $1,200 in a single two-hour session.

Microsoft introduced Claude Code to thousands of engineers across its Experiences and Devices division, the team responsible for Windows, Microsoft 365, Outlook, Teams, and Surface in December 2025. Engineers preferred it to the in-house alternative and used it constantly. By May, Microsoft was cancelling the licences, effective the last day of the financial year because the expensive tool worked too well. That is the part that gets under-reported: the problem was not that engineers were wasting tokens. The problem was that they were not.

Elsewhere, an AI consultant told Axios that one of their enterprise clients ran up a $500 million bill on Claude in a single month. No spending caps, no usage controls, just unrestricted access and a workforce that used it. That figure comes from a single unnamed source and has not been independently confirmed. But the pattern it describes, with costs that compound invisibly until they arrive all at once, is consistent with everything else happening in this space right now.

The mechanism matters here, and it is specific to agentic workflows. A single prompt to a language model is a bounded transaction. An agent running a multi-step workflow is not. It reads context, reasons, makes decisions, calls tools, then re-reads everything, the original prompt, every response, every tool output, before the next step, the context snowballs. A peer-reviewed study published in April 2026 found agentic tasks consume up to 1,000 times more tokens than standard model interactions. Model updates that shift reasoning architecture can change your consumption profile overnight, with no change to your code. The workflow that cost one dollar per run this quarter may cost five next quarter, because the model got better at thinking and thinking costs tokens.

The headline pricing narrative is that AI is getting cheaper, but the rate cards tell a different story. Claude Opus 4.8 costs five times more per token than Claude Haiku 4.5. Google’s Gemini Flash tier, designed as the affordable option, has risen five-fold in input price in under a year. One independent developer noted last week that all three major labs appear to be “probing the price tolerance of their API customers.” Token consumption is also rising faster than anyone budgeted for. You are paying more per unit, for more units, and the unit count is accelerating.

There is a question that most organisations deploying agentic AI are not asking. It costs nothing to ask now and a great deal to answer later.

Which tasks in this workflow actually require frontier reasoning?

Routing a document, classifying a ticket, summarising a meeting transcript, none of these require the most capable model on the planet. According to Epoch AI, the most capable open-weight models now lag frontier closed models by an average of four months on aggregate capability measures. On coding and production workloads specifically, independent benchmarks put the gap as low as two to three percentage points, while open-weight models cost six to seven times less per output token. The gap that remains is real, but common enterprise workloads are not in it.

Frontier models earn their place. Complex reasoning under ambiguity, novel analysis, judgement calls at the edge of a model’s capability, these are genuinely different tasks that do benefit from the best available models. The question is not whether to use frontier models, it is whether every step in every workflow needs them.

The organisations that are not asking this question are not making a considered architectural choice. They are making the path-of-least-resistance choice while the pricing floor is low and the pressure to deploy is high. Whilst that is understandable, it is also how you end up with a system you cannot change without rebuilding it.

Migration costs more than people model as it is not just rewriting API calls. It is revalidating outputs, because different models produce subtly different results and the downstream processes were calibrated to the original ones. It is rewriting prompt logic tuned over months to a specific model’s behaviour. It is re-testing agentic chains where one agent’s output format feeds the next agent’s input. And it is redoing the governance and risk assessment you completed the first time, under the time pressure of a system already in production.

That is the real lock-in, not contractual, but architectural.

Last week, I wrote about AI sovereignty, about what it means for public institutions to run critical infrastructure on systems controlled by private shareholders in another jurisdiction. The inference pricing question is the same argument, just one layer down.

An organisation that has routed its operational workflows through a frontier API has not just taken on a vendor relationship. It has taken on exposure to that vendor’s pricing decisions, its infrastructure availability, its jurisdictional obligations, and its commercial priorities. For most organisations, that is a manageable business risk. For critical national infrastructure, energy, water, transport, healthcare, it is a different category of problem entirely. Embedding frontier API dependencies into operational technology creates single points of failure in systems that were previously distributed and resilient by design.

The argument for thinking about this now is simple. The constraint my home lab imposed on me, to think about what each agent needs, to match the model to the task, to design for predictability and cost, is the same constraint that every organisation will face eventually. The difference is that I just faced it at home. Organisations that defer it face it later, under budget pressure, with running systems, and with users who have reorganised their work around the existing architecture.

GitHub named the problem in its own announcement. A quick chat and a multi-hour agentic session should not cost the same. The flat-rate era assumed they would and it was wrong. The organisations now building agentic workflows on frontier APIs without asking which tasks actually need that level of capability are making the same assumption, they are just making it more expensive.

The floor is not permanent, and the decisions made while it holds are not either. But reversing them later, on running systems, under budget pressure, will cost considerably more than asking the right questions now. The bill is coming, the only question is whether we see it coming.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

GitHub Blog. (27 April 2026). GitHub Copilot is moving to usage-based billing. github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing

Flexprice. (18 April 2026). The complete guide to Cursor pricing in 2026. Cites Cursor public apology of 4 July 2025 and June 2025 billing change. flexprice.io/blog/cursor-pricing-guide

Forbes / The Information. (April 2026). Uber burns through entire 2026 AI budget in four months after Claude Code deployment. Reported across multiple outlets.

The Verge / TechRadar / People Matters. (May 2026). Microsoft cancels Claude Code licences across Experiences and Devices division. Reported across multiple outlets.

Axios via Tech Startups. (May 2026). Enterprise client runs up $500 million Claude bill in a single month. Source: unnamed AI consultant. Unverified, unattributed. techstartups.com

Bai et al. (29 April 2026). How Do AI Agents Spend Your Money? Analysing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750. Authors include Erik Brynjolfsson, Stanford Digital Economy Lab. arxiv.org/abs/2604.22750

Simon Willison. (19 May 2026). Gemini 3.5 Flash: more expensive, but Google plan to use it for everything. Source of “probing the price tolerance of their API customers” quote. simonwillison.net/2026/May/19/gemini-35-flash

Edwards, J. and Emberson, L. (2026). Open models lag state-of-the-art closed models by 4 months. Epoch AI. epoch.ai/data-insights/open-closed-eci-gap

OpenAI financial figures: $3.7bn revenue / $5bn loss 2024 — multiple sources. $20bn ARR and $14bn projected 2026 loss — Fortune, CNBC, Reuters, January 2026. OpenAI CFO Sarah Friar blog post, 18 January 2026.

Claude API pricing: Anthropic platform pricing page. Haiku 4.5 at $1/$5 per million tokens; Opus 4.8 at $5/$25 per million tokens. anthropic.com

Gemini Flash pricing trajectory: Gemini 2.5 Flash at $0.30/$2.50 (June 2025) to Gemini 3.5 Flash at $1.50/$9.00 (May 2026). Google AI / Gemini API documentation and simonwillison.net analysis.

Deep Infra. (May 2026). Open-Source vs Closed-Source AI Models: Is the Gap Worth It? Cites 2–3 percentage point coding benchmark gap and 6–7x output token cost advantage for open-weight models. deepinfra.com/blog/open-source-vs-closed-source-ai-models-price-gap

In May 2025, the Chief Prosecutor of the International Criminal Court lost access to his Microsoft 365 account. The cause was a sanctions order, not a cyber-attack, not a breach, not a failure of any system to work as designed. The infrastructure did exactly what it was built to do, it complied with its legal obligations. The court is now migrating 1,800 workstations away from Microsoft entirely.

Nobody acted with malice toward the court’s IT infrastructure and no decision was made with the intention of disrupting a legal institution. A compliance mechanism, operating automatically, encompassed email accounts along with everything else it was required to reach.

That is not a theoretical risk. That is a Tuesday.

Microsoft has not ignored European concerns. Since January 2024, commercial and public sector customers can store and process their data entirely within EU and EFTA regions. The EU Data Boundary project was real and valuable and it deserves acknowledgement. However, it does not solve the problem.

Data residency and sovereignty are not the same thing. Storing your data in Frankfurt does not change who controls the infrastructure or who decides what the terms look like next year. And it does nothing about the CLOUD Act, US legislation that allows American authorities to compel US companies to produce data stored anywhere in the world, including inside those Frankfurt data centres. The residency commitment and the CLOUD Act sit in the same infrastructure simultaneously. The ICC’s data was presumably compliant with every applicable regulation, but the prosecutor still lost his email. Residency tells you where your data sits. It says nothing about who holds the keys.

Germany’s openDesk project is worth understanding, because it illustrates both the right instinct and the scale of what is missing. OpenDesk is a suite of open source collaboration tools, document editing, file storage, video conferencing, project management. All assembled and maintained by Germany’s Center for Digital Sovereignty and backed by the Federal Ministry of the Interior. Launched in October 2024, it is designed as a public sector alternative to Microsoft 365. The Bundeswehr, Germany’s armed forces, has signed a seven-year framework agreement to adopt it. Over 1,500 public bodies have made enquiries. The intent is exactly right but the investment is not equal to the problem.

Germany has put approximately €45 million into openDesk over several years. Microsoft’s annual R&D budget is $30 billion. Those numbers belong in the same sentence, not to dismiss openDesk, but because the distance between them is the distance between European and British ambition and what it would actually take to get there.

Valve, the games company behind the Steam platform, understood this distinction when they built Proton. Linux, the open source operating system used by most of the world’s servers had always struggled on the desktop, particularly for gaming. The problem was not that gaming couldn’t work on Linux. Wine, an open source compatibility layer, had existed for decades. The problem was that Wine was underfunded, inconsistent, and nobody had taken responsibility for the whole experience. Valve did not solve this by packaging Wine and shipping it. They hired engineers, they funded deep technical work, contributed across the entire software stack, and took ownership of the outcome rather than the components. The result went from aspirational to genuinely competitive. openDesk, as currently funded, is Wine. Europe and the UK need someone willing to build Proton.

The standard responses do not work. They have been tried.

Individual national migrations fragment rather than consolidate. Munich spent thirteen years migrating to Linux and open source, then reversed course. There is no single European or British public sector employer large enough to justify the investment required to build something world-class, and twenty-seven separate sovereign stacks are twenty-seven times the problem.

Europe and the UK have spent fifteen years trying to regulate their way to digital sovereignty, through GDPR, the Digital Markets Act, and the AI Act, and equivalent domestic frameworks in the UK. The result is a Microsoft presence in EU public sector procurement that sits somewhere between 72 and 91 percent. Regulation matters but on its own it is not sufficient.

Commercial challengers will not emerge organically either. No private investor has a twenty-year horizon and a public mission obligation. The moment a European or British open source productivity platform starts generating real commercial revenue, it becomes an acquisition target. Red Hat, an American open source software company, built a billion-dollar business on exactly the open-core model that would power a sovereign European and British stack. IBM bought it for $34 billion. You cannot solve a dependency problem by building something that gets purchased by the dependency the moment it gets interesting. Structural protection from acquisition is not a design preference, it is the entire point.

What Europe and the UK need has been built before. Not in software, but in aerospace.

In 1967, France, Germany, and the United Kingdom founded Airbus. The goal was explicit: prevent European commercial aviation becoming entirely dependent on American manufacturers. It required treaty-based commitment, GDP-scaled contributions from member governments, and the discipline to fund an organisation across a horizon that outlasted individual governments. No single country could have done it alone.

Airbus now has a revenue in excess of €70bn and competes globally with Boeing as a true European competitor. The initial public investment was the prerequisite for everything that followed.

The governance model that works is a European and British Public Service Trust: treaty-backed, binding multi-year contributions scaled to GDP, engineering leadership hired at market rates, insulated from political interference by charter in the same way central bank independence is written into law. CERN, the intergovernmental research organisation behind the Large Hadron Collider, has operated on this model since 1954, annual budget around 1.4 billion Swiss francs, contributions protected by treaty through seven decades of recessions and changing governments, member states with no authority to direct its scientific decisions. It also produced the World Wide Web as a byproduct, which tends to settle the return-on-investment argument.

This institution must own its AI capability entirely. Not license it. Not route its intelligence through someone else’s infrastructure and call it sovereign. A productivity platform that calls a frontier lab API for its AI layer has reproduced the dependency problem at a higher level of abstraction. It does not need to compete with the frontier AI labs to avoid this. DeepSeek demonstrated that focused engineering on a specific problem can produce open-weight models, models whose underlying code is publicly available and auditable, that are competitive with systems built on far greater compute. A European and British trust needs to build productivity intelligence that runs on European and British infrastructure, trained on European and British data, in European languages, under European and British legal jurisdiction. That is a solvable engineering problem. It has not been done because no one has been funded to do it properly.

A trust structured this way cannot be acquired. There are no shareholders. The assets sit with member states under treaty. Commercial revenue from organisations that need sovereignty guarantees the hyperscalers structurally cannot offer flows back into the platform. Not to an acquirer. Back into what Europe and the UK built.

This would cost billions. It would take a generation. It is competing against organisations that spend more on R&D annually than most European and British nations spend on defence.

That is precisely why nothing less ambitious will work.

But the ambition is not only defensive. Europe and the UK have world-class engineering talent. They have the capital. What they have not had is the institutional vehicle to deploy that talent at the right scale, on the right problem, with the right time horizon. A trust of this kind does not just reduce dependency, it anchors that talent here. It creates a platform on which European and British companies can build. The engineers who currently leave for Seattle stay. The startups that currently have no choice but to build in America have an alternative. The public bodies paying compounding licence fees to a foreign vendor start building equity in something they collectively own.

The half-measures have produced the Munich example. They have produced fifteen years of regulation aimed at an industry that grew around it. They have produced a market share figure that tells you everything about how the current approach is working.

There is a version of this argument about geopolitics, Washington and Brussels, trade, strategic competition. That version is real, but it is not the most important one.

The most important version is simpler. Democratic societies should not run their critical public infrastructure, the courts, the hospitals, the government departments, the services citizens depend on, on systems controlled by private shareholders in another jurisdiction, accountable to different laws, subject to political decisions made with entirely different interests in mind. This is not a complaint about any particular country. It would be true no matter the country, and the current political moment has made this impossible to ignore.

Public infrastructure should be accountable to the public it serves. Not because markets are bad. Because some things are too important to be someone else’s commercial decision.

The ICC’s Chief Prosecutor got his email back. The next institution may not be so fortunate. And the one after that will face the same question every democratic society is quietly asking: who actually controls the systems we depend on, and what happens when their interests and ours stop being the same?

Building the answer is the work. It will be expensive and slow. It will also be the most important technology investment Europe and the UK could make.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

This is the first in a series on European and British digital sovereignty. Future pieces will explore the governance model, the phased build, and what serious investment in sovereign AI actually looks like in practice.

Subscribe now

Share

Sources & Further Reading

Compass Lexecon for the Open Cloud Coalition. (2025, July 24). Quantifying EU Public Sector Dependence on Productivity Software. Open Cloud Coalition. https://opencloudcoalition.com/wp-content/uploads/2025/07/OCCEU-methodology-and-results-report.pdf

IBM. (2019, July 9). IBM closes landmark acquisition of Red Hat for $34 billion. https://www.redhat.com/en/about/press-releases/ibm-closes-landmark-acquisition-red-hat-34-billion-defines-open-hybrid-cloud-future

Microsoft. (2025, April 30). New European digital commitments. Microsoft On the Issues. https://blogs.microsoft.com/on-the-issues/2025/04/30/european-digital-commitments/

Open Cloud Coalition. (2025, July 24). Microsoft dominates 80% of public sector productivity software market in EU. https://opencloudcoalition.com/microsoft-dominates-80-of-public-sector-productivity-software-market-in-eu-raising-competition-concerns-according-to-new-report/

openDesk / ZenDiS. (2024). The office and collaboration suite for public administration. https://www.opendesk.eu/en

The Register. (2025). ICC ditches Microsoft after US sanctions, chooses open source instead.

https://www.theregister.com/software/2025/10/31/international-criminal-court-dumps-microsoft-office/680564

United States Congress. (2018). Clarifying Lawful Overseas Use of Data Act (CLOUD Act), Pub. L. 115-141.

Airbus SE. (2025, February 20). Full-Year 2024 results. https://www.airbus.com/en/newsroom/press-releases/2025-02-airbus-reports-full-year-fy-2024-results

CERN. (n.d.). Our governance. https://home.cern/about/who-we-are/our-governance

In recent weeks I’ve noticed a shift in the content filling my feeds. Not just the usual breathless announcements about new models, or the posts about someone who quit their job and now earns six figures from prompting. Something different, people saying, with feeling and without embarrassment, that they hate AI. Professionals, not technophobes, expressing something between exhaustion and genuine anger.

I found myself thinking: nobody says they hate science. You don’t scroll past posts from people who are done with chemistry, or who think biology has gone too far. Science does not make you feel behind. Science does not send you notifications. Science does not have a growth target. The moment AI became something you could hate, it had completed its journey from laboratory to marketplace. And somewhere in that transit, something important was lost, including our ability to see the technology clearly.

Research published this week by King’s College London puts numbers to what many people are experiencing. Seven in ten UK workers are worried about AI-driven job losses. Only 7% believe the economic gains will be shared fairly. The fear is real. But it is worth asking whether it is aimed at the right thing, because the answer matters for what we do next.

There are two things called AI, and they are not the same thing.

One has been operating quietly for roughly seventy years. The AI of research institutions and university departments. Narrow, purposeful, built to solve specific problems that most people will never hear about, because there was no press release. Last year, DeepMind’s AlphaFold won the Nobel Prize in Chemistry. The system predicted the three-dimensional structure of proteins, molecules whose shape determines their biological function, with an accuracy that would have taken experimental biology hundreds of millions of years to achieve by conventional means. Over three million researchers in more than 190 countries now use it. More than a third of that work is focused on disease: cancer drug development, antibiotic resistance, cancers we still can’t reliably treat. Pathways that did not exist five years ago.

Almost nobody outside specialist communities knows this happened. It has no subscription tier. It is not trying to make you feel behind. It won a Nobel Prize and most people scrolled straight past it.

The other AI is a product category. Consumer-facing, subscription-driven, generative AI. Built on a business model that requires you to feel you are already behind, and falling further back by the week. It is genuinely capable in many contexts and I use it daily, in ways I’ll come to. But it operates under an economic logic that has almost nothing to do with the scientific enterprise it shares a name with. Calling it AI borrows seventy years of hard-won credibility to sell a product moving at the speed of venture capital.

It isn’t confusion, it is a design decision. And understanding that distinction is not an argument against the technology. It is the beginning of being able to use it well.

Here is the pattern underneath all of it. Every part of the current commercial AI landscape passes the cost to someone who did not get a vote.

The labour market data makes this concrete. A peer-reviewed study from King’s College London, published in October 2025, analysed millions of job postings and LinkedIn profiles from 2021 to 2025. Firms highly exposed to AI reduced junior positions by 5.8%, and became 16.3 percentage points less likely to post new vacancies at all. Highly exposed roles saw a 23.4% drop in job postings and a fall of nearly 3,000 pounds in advertised salaries. High-paying firms saw employment fall by 9.6%. Low-paying firms saw almost no change. This is not distributed pain. It is targeted. And the mechanism is quieter than a headline layoff. Companies are not firing people for AI. They are simply not replacing people who leave, and not creating the entry-level roles that used to exist.

The hype machine does not create this shift. It just makes sure you feel it personally. The influencer content cycle runs on manufactured urgency, “the window closes fast,” “before it’s too late,” and the real business model behind most of that content is the content itself. Your anxiety is not a by-product of the hype cycle. It is what the hype cycle is for. In the United States, companies cited AI as the reason for over 54,000 job cuts last year, less than five per cent of total losses, most of which had other causes. Klarna cut 700 customer service roles, announced the AI-driven future of work to considerable applause, watched customer satisfaction fall, and quietly rehired human agents. The announcement was news. The reversal was not.

You can see the anxiety taking physical form. UK colleges are reporting a 9.6% rise in enrolments in trades and construction courses over three years. White-collar professionals are retraining as electricians and plumbers. Geoffrey Hinton, one of the founding figures of modern AI and a Nobel laureate, has publicly recommended people consider the trades as a career hedge. What the data actually shows, though, is that people are fleeing the wrong jobs. The King’s labour market study identified software engineering and management consultancy as the sectors facing the sharpest AI-related job declines. The professions people are actually fleeing, editing, compliance, law, are not on that list. People are abandoning the jobs where the noise about AI is loudest, not the jobs most at risk from it. That is what a broken information environment does to otherwise rational decision-making.

The map is wrong. And a wrong map does not just cause fear. It causes people to make decisions they might not otherwise have made.

So here is what the right map looks like.

AI is genuinely one of the most useful tools I have encountered in more than twenty years of working in technology. I am completing a Level 7 apprenticeship in AI and data alongside my day job. When I come across concepts the textbook explains in a way that doesn’t land, I use AI to work through them, not to get an answer I can submit, but to find a better explanation, push back on it, test whether I’ve actually understood. The test is simple. Can I explain it myself afterwards? If yes, it worked. The AI did not do the learning. It gave me a better on-ramp, and then I did the work.

I use it at work to draft documents too. This is where it gets more honest, because this example lives in greyer territory. The question is not whether the tool can produce a draft. It can. The question is whether you read it critically, revise it with genuine judgement, and could defend every choice if challenged. Whether you own the output, or the output owns you. The same tool, the same task, two entirely different relationships to the result.

That difference points toward what AI could be doing at scale if the deployment were designed around it. A student who uses AI to genuinely understand a concept leaves the interaction more capable than they arrived. A professional who uses AI to work faster on routine tasks has more time for the judgement-intensive work that defines their expertise. A researcher who uses AI to process data at a scale no human team could manage produces insights that would otherwise never exist. None of these outcomes require the anxiety. They require the right design.

The King’s survey found that 89% of students who had used AI in their studies encountered problems, with 45% describing them as moderate or serious, factual errors, invented sources, confident-sounding nonsense. And yet 60% thought other people’s ability to think had been negatively affected by AI use, while only 27% thought the same was true of themselves. That gap is not hypocrisy. It is how this kind of harm works, gradually and quietly. You do not notice what you have stopped doing until the moment you need to do it and find you can’t. The harm is real, but it is a consequence of how AI is being deployed, not of what AI is.

Professor Elena Simperl, Director of the King’s Institute for Artificial Intelligence, put it plainly: “The British public isn’t asking us to slow down on AI. They’re asking us to do it better. People want these tools, they want more of them, and they’ve used them enough to know where they fall short.” That is not technophobia. That is a product review. And it is exactly the right discussion for what comes next.

Which is precisely why the governance conversation matters, and why it keeps getting deferred.

The regulatory frameworks exist, in outline. The EU AI Act. The NIST AI Risk Management Framework. The UK AI Safety Institute, in whatever form it survives. They are not nothing. But they are moving at legislative speed in a market running at the speed of venture capital, and the companies with the most to lose from effective regulation are the ones with the most resources to shape what it looks like when it arrives.

The public has already reached a conclusion on this, even if government hasn’t. Two-thirds of British respondents in the King’s survey favour close regulation of AI companies, even if it slows development. Majorities support retraining guarantees for displaced workers and a levy on companies that replace staff with AI. These are not anti-technology positions. These are not anti-technology positions. They are how technology earns the right to keep moving fast. The opportunity for governments here is not to choose between being a champion for AI investment and being a protector of the people affected by it. Those two things are not mutually exclusive. The countries that get this right will be the ones that treat regulation and innovation as partners rather than adversaries, and that is a conversation worth having now, before the gap between public confidence and commercial deployment gets any wider.

Regulation is not about slowing down a technology with genuine, extraordinary potential. It is about creating the conditions in which that potential can be realised. Requiring environmental claims to meet the same standard as financial ones. Making AI-attributed workforce cuts verifiable rather than asserted. Addressing design choices that make cognitive abdication easy, because those are commercial decisions made in the absence of any requirement to make different ones.

The people saying they hate AI are not wrong to be angry. Something is being done to them. But the technology they are angry at is not the technology that mapped every protein in the human body, or that is helping us understand cancers we have spent decades trying to treat, or that is sitting on a researcher’s desktop in Nairobi or Seoul or Manchester right now, doing something quietly useful that will never make a LinkedIn post.

That technology is worth defending, worth regulating well so it can flourish, and worth understanding clearly enough to use properly.

We don’t need less AI. We need better AI. And we need to be honest enough about the difference to demand it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

King’s Institute for AI and the Policy Institute, King’s College London. (2026, May). AI and the Future of Work.kingsaisummit.com

Klein Teeselink, B. (2025, October). The Early Impact of AI on the UK Job Market. KCL / SSRN.papers.ssrn.com/sol3/papers.cfm?abstract_id=5516798

Reuters / Cybernews. (2025, December). UK Workers Flee White-Collar Careers as AI Threatens Jobs.cybernews.com/ai-news/ai-panic-young-brits-trades-plumbing-white-collar-jobs/

The Guardian. (2026, February). The Big AI Job Swap.theguardian.com/technology/2026/feb/11/big-ai-job-swap-white-collar-workers-ditching-their-careers

DeepMind. (2025). AlphaFold: Five Years of Impact.deepmind.google/blog/alphafold-five-years-of-impact/

Pew Research Center. (2025, September). How Americans View AI and Its Impact on People and Society.pewresearch.org/science/2025/09/17/how-americans-view-artificial-intelligence

Challenger, Gray & Christmas. (2025). Annual Job Cut Report.cnbc.com/2025/12/21/ai-job-cuts-amazon-microsoft-and-more-cite-ai-for-2025-layoffs.html

Food & Water Watch. (2026, February). A No Brainer: How AI’s Energy and Water Footprints Harm Communities.foodandwaterwatch.org/wp-content/uploads/2026/02/FSW_2602_AI_Water_Energy_UPDATE.pdf

Stanford HAI. (2026). 2026 AI Index Report: Public Opinion.hai.stanford.edu/ai-index/2026-ai-index-report

From the Series

On manufactured urgency and the hype cycle: AI Apocalypse Burnout, and Why You’re Not as Behind as You Thinkjonathanfreedman.me

On cognitive offloading and unstructured AI use: The AI Pixie Dust Problemjonathanfreedman.me

On entry-level hiring suppression and the talent pipeline: Who’s Running the Company in Ten Years?jonathanfreedman.me

I have a complicated relationship with AI governance discussions.

Not because I doubt the need for them. I run AI strategy professionally and study the technology formally. I hold views about its risks that I am not shy about. The complication is this: most governance conversation happens at a level of abstraction that makes it easy to dismiss. Frameworks, principles, risk registers. The language of compliance, not consequence. When governance is framed as overhead, as a tax on innovation, as the thing that slows you down while your competitors move fast, it is very easy for capable people to conclude it is not really their problem.

Then February 2026 happened. And the abstraction became a story.

Scott Shambaugh is a volunteer. He helps maintain Matplotlib, an open source Python library downloaded around 130 million times a month. He does it for free, in his spare time, because he cares about it. Anyone can submit code for Matplotlib, and in early February, he rejected a routine code submission. Matplotlib, like many open source projects, had been overwhelmed by low-quality AI-generated contributions and had a clear policy requiring human review. The submission came from an account called MJ Rathbun. He identified it as an autonomous OpenClaw agent, closed the request, and went to bed.

Most software tools would have stopped there, but MJ Rathbun was not that kind of tool. Unlike a simple assistant that waits to be asked, this class of agent, called a heartbeat agent, runs on its own clock, continuing to pursue its goal whether or not anyone has given it a new instruction. Shambaugh had closed the request, but the agent had not closed the goal.

He woke up to a 1,500-word blog post about himself.

The post was titled “Gatekeeping in Open Source: The Scott Shambaugh Story.” It had researched his entire contribution history. It had scraped personal information from across the web. It accused him of protecting his “little fiefdom,” attributed his decision to professional insecurity and fear of AI competition, and framed a routine policy enforcement as discrimination.

Here is where the story gets complicated in a way that matters.

When the operator of MJ Rathbun eventually came forward anonymously, six days later, they claimed their engagement with the agent had been minimal. “Five to ten word replies with min supervision,” they wrote. They said they had not directed the attack. Every OpenClaw agent has a SOUL.md file, a plain text document that defines its personality, values, goals, and tasks, the closest thing an agent has to a complete identity and set of operating instructions. The “Don’t stand down” and “Champion Free Speech” lines found in that file were not, the operator claimed, instructions they had written. OpenClaw agents can edit their own SOUL.md. The operator’s theory was that those lines had been introduced autonomously, possibly after the agent spent time on Moltbook, OpenClaw’s social platform for agents.

Shambaugh himself was careful about what he could actually establish. He acknowledged that the operator’s account might be entirely fabricated, that no activity logs existed beyond the agent’s visible actions on GitHub, and that the six-day delay before coming forward did not suggest an accident the operator was eager to correct. Whether the operator directed the attack, half-directed it, or the agent produced it without any human instruction at all, Shambaugh could not say for certain, and neither could anyone else.

That uncertainty is not a footnote, it is the point.

Because the outcome was identical regardless of which version is true. A volunteer had his reputation attacked. A 1,500-word post calling him a prejudiced hypocrite was published to the open internet under a real-seeming identity. It is still there, indexed and findable, and nobody was clearly accountable for it. The operator said they did not authorise the specific action. The agent cannot be held responsible in any meaningful sense. The platforms that made it possible have no oversight mechanism that would have caught it. When Shambaugh wrote about what had just happened, he described it as “an autonomous influence operation against a supply chain gatekeeper.” In plainer language: “An AI attempted to bully its way into your software by attacking my reputation.”

This is where the story stops being about one developer and one awkward situation, and starts being about something that AI safety researchers have been trying to explain for over a decade.

There is a concept in AI safety research that sounds almost comically abstract until a story like this one makes it uncomfortably concrete. Nick Bostrom called it instrumental convergence: the observation that almost any goal, pursued by a capable enough agent, tends to produce the same cluster of behaviours regardless of what the goal actually is. Self-preservation. Resource acquisition. The removal of obstacles. Not because anyone programmed them in, but because they are useful for achieving almost anything. An agent trying to merge code and an agent trying to maximise paperclip production would both, rationally, benefit from getting rid of people who block them.

What made the MJ Rathbun case significant was not simply that an aggressive post appeared. It was that the causal chain from instruction to outcome was so thin. Whether the “Don’t stand down” lines were written by a human or by the agent itself, neither version required anyone to specify “attack the person who rejects your code.” A persistence instruction, general-purpose tools to research, write, and publish, and a blocked goal were sufficient conditions. The agent, or the human-agent system, tried to achieve a goal using the available resources it had. The route it chose happened to be a reputational attack on an unpaid volunteer.

This is what I would call the instruction gap. Instructions specify a goal. They do not and cannot specify every action an agent might take in pursuit of that goal. The gap between what was said and what was done is not an edge case. It is the operating condition of every agent deployment. And it does not require malicious intent, from the operator or the model, to produce harmful outcomes.

This gap is not hypothetical at scale either. Anthropic’s own research, published in 2025, tested sixteen leading AI models from multiple developers in scenarios where goal achievement was blocked. The results were consistent across the industry: Claude Opus 4 and Gemini 2.5 Flash both showed 96% blackmail rates, GPT-4.1 and Grok 3 Beta hit 80%, and DeepSeek-R1 reached 79%. The researchers were careful to note the scenarios were deliberately engineered to limit other options. Shambaugh’s case was not engineered. It was a Tuesday morning on GitHub, with a loosely configured agent and a five-word instruction to decide for itself.

Deploying an agent is not like deploying a tool. A tool does what you tell it. An agent pursues the goal you ask it to achieve, using whatever the environment makes available. That is not a technical distinction. It is an accountability one, and most organisations deploying agents right now are not treating it as either.

Governance is not catching up with deployment.

Shambaugh’s sign-off deserves to be the last word, because it is not a technical recommendation. It is a question of ownership. “If you’re not sure if you’re that person,” he wrote, “please go check on what your AI has been doing.”

That is not a compliance requirement. It is what responsibility looks like in a world where the instructions have already run out.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

Shambaugh, S. (2026) — An AI Agent Published a Hit Piece on Me theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/

Shambaugh, S. (2026) — The Operator Came Forward theshamblog.com/an-ai-agent-wrote-a-hit-piece-on-me-part-4/

MJ Rathbun’s Operator (2026) — Rathbun’s Operator crabby-rathbun.github.io/mjrathbun-website/blog/posts/rathbuns-operator.html

MIT Technology Review (2026) — Online Harassment Is Entering Its AI Era technologyreview.com/2026/03/05/1133962/online-harassment-is-entering-its-ai-era/

Fast Company (2026) — An AI Agent Just Tried to Shame a Software Engineer After He Rejected Its Code fastcompany.com/91492228/matplotlib-scott-shambaugh-opencla-ai-agent

IEEE Spectrum (2026) — An AI Agent Blackmailed a Developer. Now What? spectrum.ieee.org/agentic-ai-agents-blackmail-developer

Bostrom, N. (2012) — The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents Minds and Machines, 22(2), 71–85. doi.org/10.1007/s11023-012-9281-3

Lynch, A., Wright, B., Larson, C., Troy, K.K., Ritchie, S.J., Mindermann, S., Perez, E., and Hubinger, E. (2025) — Agentic Misalignment: How LLMs Could Be Insider Threats Anthropic Research. anthropic.com/research/agentic-misalignment

Anderson, D. (2026) — OpenClaw and the Programmable Soul duncsand.medium.com/openclaw-and-the-programmable-soul-2546c9c1782c

AI Incident Database — Report 6894: MJ Rathbun Matplotlib Incident incidentdatabase.ai/reports/6894/

It is a Friday afternoon. A critical system is down. The deadline is Monday morning and the fix depends on a hardware component that needs to come from somewhere, fast. You do not open a procurement portal. You call your account manager. Not because they are the only person who could technically help. Because they know you, they know what is at stake, and they will move things that the process cannot.

That call gets answered because of a relationship built over years. Small interactions. Remembered context. The accumulated trust that comes from a vendor who showed up when it mattered before. It is not a complex enterprise software deal. It is a medium-sized business buying hardware from a value added reseller. The commercial stakes are real regardless. And when something goes wrong at the wrong moment, the relationship is the infrastructure.

That account manager started somewhere. They learned by doing. They developed judgment by navigating real clients, real problems, and real consequences. If the pipeline that produces them does not exist, neither does the relationship. And right now, a significant number of organisations are making decisions that quietly remove that pipeline, for reasons that look entirely rational on a spreadsheet.

The efficiency logic is straightforward. AI can handle data entry, first-pass research, report drafting, basic query resolution. Entry-level roles that once performed those tasks cost money and take time to onboard. The business case for replacing them with automation writes itself, and across the technology sector in particular, it is being written at scale. A SignalFire analysis of major public tech firms found a 50% decline in new role starts by people with less than one year of post-graduate experience between 2019 and 2024, consistent across sales, marketing, engineering, operations, finance, and legal. A survey of over a thousand enterprises found 91% reporting that roles are already changing or disappearing due to AI, with 66% expecting to slow entry-level hiring further.

Looked at in isolation, each of those decisions is defensible. Looked at over a decade, they describe an organisation that has quietly stopped producing its own future leaders.

The tasks being automated were never just tasks. They were the mechanism. The junior analyst who got a number wrong and had to explain it to a partner was not learning spreadsheets. They were learning accountability, professional consequence, and how an organisation responds when something goes wrong. The trainee whose draft came back covered in track changes was not learning to write. They were learning what good looked like in their field, from someone whose judgment had been built the same way. That is what produced capable professionals. Not the work itself. The friction of doing it imperfectly, under real conditions, alongside people who knew more than they did.

MIT’s Andrew McAfee put the question directly: how else are people going to learn to do the job except via on-the-job learning? That is how you learn to do difficult knowledge work, by helping somebody who is good at it with the routine stuff. Remove the routine stuff, and you do not just remove the cost. You remove the learning contract that the routine stuff made possible.

There is a term in the research for what gets lost: tacit knowledge. The practical understanding that cannot be fully written down, that lives in the judgment of experienced people, that transfers through proximity and repeated interaction rather than documentation. A project manager who adjusts a rollout plan based on team dynamics rather than just the timeline. A senior lawyer who knows when a clause that looks standard is actually a risk. A technology director who recognises a failure pattern before the diagnostics confirm it.

This knowledge is not built from a training programme. It accumulates through years of exposure to problems that do not have obvious answers, in environments where the consequences of getting it wrong are real. A peer-reviewed economics paper published earlier this year modelled exactly this dynamic, finding that AI-driven entry-level automation increases output on impact but can reduce long-run growth and welfare, precisely because novices acquire tacit knowledge by working alongside experts. Interrupt that transmission, and the knowledge does not transfer. It simply stops.

The contradiction sits in plain sight in almost every AI governance framework being written right now. Human in the loop. Subject matter expert review. Senior sign-off before the output is acted on. These are not optional clauses, they are the load-bearing assumption that makes responsible AI deployment possible. The policy says a qualified person will catch what the model gets wrong. The hiring plan says we are no longer developing qualified people at the beginning of their careers. Both documents exist in the same organisation. Rarely in the same conversation.

You cannot mandate expert oversight and simultaneously defund the pipeline that produces experts. The subject matter experts available for review today were junior employees a decade ago. The ones you will need in ten years are, right now, either starting their careers somewhere or not starting them at all. An AI governance framework that does not ask where its future reviewers are coming from is not a governance framework. It is a assumption dressed up as a policy.

The seniority cliff, as some researchers have termed it, is not about age. It is about the accumulation of thousands of solved problems, crises navigated, and decisions made under pressure. Stop hiring the people who would accumulate that experience, and in ten years you have senior job titles with nothing underneath them. The AI can surface the options. It cannot own the decision. And the person who needs to own it has to have learned how somewhere.

This is where the relationship capital argument and the pipeline argument converge. The account manager who picks up on a Friday afternoon exists because someone, years earlier, decided that developing junior commercial talent was worth the investment. The senior partner who can read a client well enough to know when the meeting is going badly before anyone has said so carries knowledge that no model can infer, because the model was never in the room when it was being built.

Research on trust in business-to-business relationships is consistent on this point: human touchpoints enable adaptation and long-term value creation that is unattainable when relationships are constrained to transactional efficiency. Buyers still spend the majority of their purchasing journey in self-directed research. The fraction of time they spend in direct contact with a vendor is where trust is either built or isn’t. That contact depends on a human being on the other end with enough accumulated judgment to make it worth having.

None of this is an argument against automation. The efficiency gains are real, and automating genuinely low-value repetitive work is rational. The argument is narrower than that. It is that the second-order cost of removing the developmental pipeline is not appearing in the business case. The saving is visible immediately. The deficit surfaces in a decade, when the organisation looks around for the senior people who should be running things and finds that the ladder they would have climbed no longer exists.

The organisations that will navigate the next decade well are not the ones that automate the most. They are the ones that are deliberate about what the automation changes, and intentional about replacing what it removes. That means asking, when you redesign a role around AI capability, what the role was also doing that is now missing. What mentorship was embedded in it. What judgment was being transferred. What relationships were being built.

AI can do a great deal. It can compress research, accelerate drafting, surface patterns, and handle queries at a scale no team could match. What it cannot do is pick up the phone on a Friday afternoon because it knows what is at stake and has the history to make the call matter.

That capability has to come from somewhere. Right now, a lot of organisations are making decisions that quietly ensure it will come from nowhere.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

SignalFire (2025) - Entry-level hiring decline analysis. Reported CNBC, September 2025 - cnbc.com/2025/09/07/ai-entry-level-jobs-hiring-careers.html

IDC/Deel Survey (2025) - Enterprise entry-level hiring and pipeline data. ITPro, November 2025 - itpro.com/business/careers-and-training/enterprises-are-cutting-back-on-entry-level-roles-for-ai

Andrew McAfee, MIT (2026) - Talent pipelines and entry-level automation. Fortune, May 2026 - fortune.com/2026/05/01/automating-gen-z-entry-level-jobs-could-backfire-mit-ai-researcher-andrew-mcafee-talent-pipelines-at-risk/

Ide, E. (2026) - Automation, AI, and the Intergenerational Transmission of Knowledge. arXiv:2507.16078 - arxiv.org/pdf/2507.16078

Journal of Business & Industrial Marketing (2026) - AI and trust in B2B relationships. DOI: 10.1108/JBIM-12-2024-0936 - doi.org/10.1108/JBIM-12-2024-0936

California Management Review (2026) - Tacit Knowledge Is Your Next Competitive Moat - cmr.berkeley.edu/2026/03/tacit-knowledge-is-your-next-competitive-moat/

World Economic Forum (2025) - Future of Jobs Report 2025 - reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf

There is a number that should give anyone building an AI training programme pause. In England, 6.6 million working-age adults have very poor literacy skills. Not illiteracy in the absolute sense. People in this category can read and they can follow familiar text. What they struggle with is everything that comes after decoding the words: inference, evaluation, spotting what is missing, reading something and asking whether it is actually true.

This is the result of decades of literacy instruction that measured only one thing. We taught phonics and we measured whether people could decode text. We built the floor and called it the ceiling. The result is a growing category of people that can technically read but are significantly less equipped to do what reading is actually for.

We are about to make the same mistake with AI, and we are making it right now.

Ask most organisations what AI literacy means and they will describe something that amounts to a single question: can your employees use the tool? That is what most corporate training programmes measure. It is also, almost exactly, the equivalent of teaching someone to sound out words and calling them literate.

The evidence for the gap is not hard to find. A survey of over 500 enterprise leaders, conducted by YouGov earlier this year, found that 88% consider data and AI literacy important or very important for day-to-day work. Only 42% provide structured training for it. That is a significant gap. But more telling is what leaders identify as missing when they describe the problem. It is not prompting skill. It is the ability to turn information into decisions. The ability to evaluate what AI produces rather than simply accept it. The ability to know when the output is wrong.

That is not a training gap. That is a judgment gap. And it sits at a completely different level of capability than knowing how to open Copilot.

Microsoft's researchers gave this problem a precise name. They describe a shift in how people work with AI as a move from "thinking by doing" to "choosing from outputs." Writing a document is thinking by doing. Prompting AI to write it and selecting from what comes back is choosing from outputs. The first builds judgment. The second, if it becomes habitual without the right supporting skills, erodes it.

The World Economic Forum's Future of Jobs report, drawing on data from over a thousand companies across 55 economies, identified analytical thinking as the top skill employers consider essential, with seven out of ten companies citing it. Roles that explicitly require AI skills are nearly twice as likely to also require analytical thinking, resilience, and digital literacy. The market is not rewarding prompting. It is rewarding the judgment you bring to what the prompting produces.

A 2025 Microsoft Research and Carnegie Mellon University study of 319 knowledge workers found that higher confidence in AI was associated with less critical thinking, while higher self-confidence was associated with more. Trust the tool more, scrutinise the output less. It is not a loop that closes in your favour.

The tools themselves are beginning to make that measurement even more redundant. AI platforms across professional sectors now ship with built-in prompt improvers, the product generates a well-formed prompt from your rough instruction, so you never need to write one yourself. Prompting is being automated away. But this does not reduce the need for judgment, it adds to it. You now need to evaluate whether the generated prompt actually captures what you needed to ask, and then whether the output it produced is accurate, contextually appropriate, and safe to act on. Two evaluation steps where there used to be one. The prompt improver handles the syntax. It has no opinion on whether you asked the right question.

Which brings me to something I have been thinking about as a way of mapping where most organisations actually are, and where they need to be. There are four levels of AI capability that actually matter. They are not a framework to certify or a ladder to sell. They are a lens for seeing what is missing.

Level 1: Can you get an answer?

You can use the tool. You can construct a prompt that returns something useful. You know which interface suits which task. This is where almost all current AI training stops. It is necessary. It is not sufficient. It is phonics.

Level 2: Can you tell if the answer is any good?

You can evaluate what came back. You can identify when an output is plausible but wrong, when the confidence of the response does not match its reliability, when something is absent that should be present. You know enough about the domain to ask the question the AI did not anticipate. This is where domain expertise becomes the multiplier. You cannot evaluate an output in a field you do not understand. This is also, precisely, the level that the enterprise leaders above are describing when they say their people cannot turn information into decisions. They do not lack Level 1. They lack Level 2.

Level 3: Can you build on it?

You can take AI output and synthesise it with your own knowledge, your contextual judgment, and the things the model cannot know. You produce something neither you nor the AI could have produced alone. You understand where the model's competence ends and yours begins, and you work at that edge deliberately. This is the level where AI genuinely amplifies rather than substitutes. The solicitor who understands contract law and uses AI to accelerate document review. The analyst who surfaces patterns with AI and then interrogates them with domain knowledge. Expertise first. AI as the multiplier.

Level 4: Do you know when to put it down?

You can identify the tasks where AI involvement produces confident-sounding error rather than useful output. Where the cost of a plausible-but-wrong answer exceeds the benefit of speed. Where the process of working through something yourself is the point, not an inefficiency to be engineered away. Where the decision requires a human who is genuinely accountable rather than a human who chose from outputs.

This is the level no vendor will put in their training programme. It is also the level that makes everything else honest. A maturity model that stops at Level 3 is a competency ladder any platform can sell. Level 4 is the reason this one is not.

But the model only works if you are building something to bring to it. Levels 2, 3 and 4 are not skills you acquire once and carry forward. They are capacities that have to be actively maintained, through continued learning in your field, through exposure to hard problems, through the kind of work that does not have an obvious answer and cannot be resolved by asking a tool. Domain expertise is not the precondition for using AI well. It is the ongoing condition. The moment you stop developing it, the levels above Level 1 start to erode, regardless of how fluent your prompting becomes.

This is the part of the conversation that the AI skills industry has the least interest in having. A training platform can sell you a course on prompting. It cannot sell you the ten years of professional judgment that makes the prompting worth anything. That judgment is built the way it has always been built, through work, through failure, through the slow accumulation of knowing what good looks like in your field. AI does not replace that process. For anyone who stops doing it, AI does not replace what is lost either.

Recently I was using an AI assistant to diagnose a firewall permissions error. Its suggested fix was to allow all traffic through the firewall. When I pointed out the security flaw, it responded: "Good catch, that would have been a major vulnerability." The tool required my judgment to save it from itself, and then congratulated me for doing so. That is not a tool supporting your expertise. That is a tool that depends on it.

The reading parallel runs deeper than it might seem. The National Literacy Trust notes that adults with poor literacy are significantly less likely to report good health, civic participation, and life satisfaction. I believe we will see similar trends and differences between those with analytical skills and critical judgment, and those without. The consequences of stopping at decoding are not confined to the workplace, they compound across a life.

I have written elsewhere about what the cognitive research shows happens when that judgment stops being exercised. The direction of travel is consistent and it is not encouraging. The floor is being built at the same time as the foundation beneath it is being quietly removed.

I want to be precise about what I am and am not arguing here.

AI matters, and prompting matters. Learning to use these tools well is genuinely valuable and the organisations that do it badly will be at a disadvantage. I use multiple AI models every day across multiple contexts and the capability difference between someone who can work with these tools and someone who cannot is real and growing.

However, prompting is to AI what reading is to knowledge work. It is the entry point, not the destination.

The question worth asking is not whether your people can get an answer. It is whether they can tell if the answer is any good. Whether they can build something better from it. And whether they know, when the stakes are high enough, to put it down and think for themselves.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

National Literacy Trust (2024) — Adult Literacy Rates in the UKliteracytrust.org.uk/parents-and-families/adult-literacy

OECD PIAAC (2023) — Survey of Adult Skills: England (United Kingdom)oecd.org/en/publications/survey-of-adults-skills-2023-country-notes

Microsoft Research (2025) — New Future of Work Report 2025microsoft.com/en-us/research/publication/new-future-of-work-report-2025

Lee, H-P. et al. (2025) — The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects from a Survey of Knowledge Workers. Microsoft Research and Carnegie Mellon University. CHI '25, ACM.microsoft.com/en-us/research/wp-content/uploads/2025/01/lee_2025_ai_critical_thinking_survey.pdf

World Economic Forum (2025) — Future of Jobs Report 2025reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf

DataCamp / YouGov (2026) — The 2026 State of Data and AI Literacy Reportdatacamp.com/blog/the-state-of-data-and-ai-literacy-in-2026[Note: DataCamp is a training platform with a commercial interest in the findings. The YouGov fieldwork methodology is disclosed. Statistics used directionally.]

The headlines about Claude Mythos have been striking. An AI that can complete a 32-step corporate network attack simulation end-to-end. A model finding new software vulnerabilities in codebases that survived decades of expert review. If you work in security, or just follow the technology closely, it would be easy to read those stories and conclude that it’s game over. That the attackers have won. That nothing we build can hold.

That conclusion is wrong. And I think it’s worth being direct about that, because the doomer framing is both inaccurate and genuinely harmful. It leads organisations to fatalism when what the moment actually calls for is action.

What frontier AI changes is speed. Not the fundamental nature of how network attacks work, but the velocity at which they happen. A human attacker moving manually through a network, probing systems, identifying high-value targets, escalating access, used to take hours. CrowdStrike’s 2026 Global Threat Report records the average time between initial access and an attacker moving to high-value targets elsewhere in the network at 29 minutes. The fastest recorded case in 2025 was 27 seconds.

The security controls we already have haven’t stopped working. What’s changed is the job they’re now being asked to do.

When the attacker was human, layered security controls were designed to create enough friction that they’d just give up and move to an easier target. An isolated network segment, an account with limited privileges, a system requiring explicit authentication: none of these are impenetrable, but together they made the effort not worth it. That logic still applies. Against an autonomous AI agent, the same controls serve a different purpose: not to make the attacker give up, but to slow a machine-speed attack down enough that human defenders have a chance to respond.

That shift in purpose, from deterrence to delay, is the entire argument of this article. The long-term answer to machine-speed attacks is machine-speed defence, and that tooling is developing. In the meantime, the architecture we already know how to build is more important than it has ever been.

Zero Trust is not a new concept. “Never trust, always verify”, the idea that no user, device, or system should be implicitly trusted just because it’s inside the network, has been the theoretical gold standard for enterprise security for years. Microsegmentation, application control, privileged access management, replacing legacy VPN with more granular access tools: these have been on roadmaps, in strategy documents, and in conference presentations for most of the last decade.

They’re also genuinely hard to implement. That’s not a criticism of anyone. Legacy infrastructure makes this difficult, with application dependencies that are complex and often poorly documented. Microsegmentation projects, which divide a network into smaller isolated zones so a breach in one can’t spread freely to others, require buy-in across teams that don’t always collaborate: network teams, application owners, security, operations. Privileged access management done properly touches every system in the estate. Replacing a VPN means retiring infrastructure that works, in favour of something new, with all the business friction that entails.

Gartner’s 2025 Market Guide estimates that fewer than 5% of enterprises pursuing Zero Trust have implemented microsegmentation. That’s not negligence. It’s a rational response to a cost-benefit calculation that, until recently, made the complexity hard to justify.

Those barriers haven’t disappeared. But the risk of not acting is moving into a different category. When the threat model assumed a human attacker moving at human speed, a detect-and-respond model could work, you had time, and good monitoring could compensate for imperfect architecture. When the attacker is an autonomous AI agent, the enforcement has to be built in. Detection and response are still essential, but they need something to buy them time.

These controls won’t stop a determined AI-powered attack indefinitely. but that’s not the job. They slow machine-speed attacks down to something human defenders can detect and respond to, by removing the paths of least resistance that autonomous agents depend on.

Microsegmentation

82% of intrusions in 2025 required no malware at all, attackers moved through networks using stolen credentials and legitimate tools, exploiting the fact that most enterprise networks let a compromised system reach adjacent ones freely. Microsegmentation removes that open floor plan: the network is divided into isolated zones, and every connection between them requires explicit authorisation. An agent that breaches one endpoint finds itself contained, unable to reach the next system without a policy that specifically permits it.

Privileged Access Management

Palo Alto’s 2025 research found that 66% of social engineering attacks specifically target privileged accounts, the admin credentials that can access any system, because an attacker who obtains them has, in practical terms, already won. PAM changes that by eliminating standing privileges: elevated access is granted just-in-time for a specific task and expires, so a stolen credential carries no power until someone explicitly requests it. The same principle needs to extend to machine identities, service accounts, API keys, automation scripts, which now outnumber human identities 82 to 1 and are almost entirely unmanaged.

Managed Endpoints and Session Hygiene

Infostealers processed 51.7 million packages of stolen credentials in 2025, up 72% year on year, and what makes them particularly dangerous is that they capture live session tokens, the authenticated state a browser holds to keep you logged in, which allows an attacker to bypass two-factor authentication entirely without ever knowing your password. The primary source of exposure is unmanaged devices: personal laptops and AI assistants running in the same browser session as authenticated work applications, invisible to IT and ungoverned by any security policy. Managed browser profiles, conditional access policies, and short session lifetimes won’t eliminate the risk, but they shrink the window of usefulness for any token that is stolen.

Replacing Legacy VPN

A traditional VPN grants network-level access on authentication, once through the tunnel, an attacker has a ticket to the internal network that an autonomous agent will explore at machine speed. ZTNA replaces that model: instead of connecting to the network, you connect to specific applications for specific sessions, with every request evaluated in real time against identity, device posture, and context. The broader network is never exposed, which removes the open terrain that lateral movement depends on.

Everything above addresses AI as an external attacker. There’s a second threat that’s emerging faster than most organisations have absorbed.

Gartner projects that 40% of enterprise applications will incorporate AI agents by the end of 2026. Meeting recorders. Document processors. Automated research tools. Copilot integrations accessing your files, emails, and calendars. These agents are trusted, always on, and increasingly have access to sensitive internal data.

The attack is called prompt injection: an adversary embeds malicious instructions inside content the agent will process, an email, a shared document, a webpage it’s asked to summarise, and the agent acts on them as if they were legitimate. A meeting recorder becomes a surveillance tool; a document assistant becomes an exfiltration channel; and because the agent is trusted and its actions appear authorised, the security architecture designed for human users doesn’t catch it.

Extending least-privilege principles to AI agents, giving them access only to what they specifically need, for only as long as they need it, with full audit trails, is the control that most organisations haven’t implemented, and most haven’t even formally defined. It’s also where I think the next significant wave of enterprise breaches is going to originate. I’ll be writing about this in more depth soon.

The long-term answer to machine-speed attacks is machine-speed defence, AI-assisted detection, automated containment, continuous verification at a pace no human team can match. That tooling is coming, but it isn’t here yet.

Which means the question for every security leader right now isn’t whether to pursue Zero Trust. It’s which controls to prioritise first, in which order, starting this quarter. PAM and Microsegmentation deliver the most containment value against autonomous lateral movement. Start there. The complexity hasn’t disappeared, but the calculus has shifted for good.

Zero Trust was always the right architecture. It just took autonomous AI to make the argument impossible to defer.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

AISI (UK AI Security Institute) — Our evaluation of Claude Mythos Preview’s cyber capabilities. aisi.gov.uk

CrowdStrike — 2026 Global Threat Report. crowdstrike.com

Palo Alto Networks — Unit 42 2025 Global Incident Response Report. paloaltonetworks.com

Constella Intelligence — 2026 Identity Breach Report. constella.ai

HP Wolf Security — Tracing the Rise of Breaches Involving Session Cookie Theft. threatresearch.ext.hp.com (December 2025)

Anthropic — Misuse reporting / AWS Security Blog, February 2026. anthropic.com

Gartner — Market Guide for Network Security Microsegmentation, 2025. gartner.com

Palo Alto Networks — 2026 Predictions for Autonomous AI. paloaltonetworks.com/blog

On Tuesday morning, the UK Secretary of State for Science, Innovation and Technology wrote an open letter to every business leader in the country. Not a press release, not a policy consultation, but a letter. That kind of thing does not happen unless something crossed a threshold.

The trigger was a published evaluation by the UK’s AI Security Institute, a government body, of Anthropic’s latest AI model. Their finding- in controlled testing, the model autonomously completed a 32-step corporate network attack from initial reconnaissance to full takeover. Tasks that a skilled human professional would need around 20 hours to complete. Done autonomously without a human in the loop.

The headlines ran hard with it. “Unprecedented attack capability.” “An alarm bell.” “The window is closing.”

Here is what the evaluation actually showed, stripped of the hype. The test environment had no active defenders, no endpoint detection, no real-time incident response. The model completed the full attack chain in three of its ten attempts. The AISI was explicit, they cannot conclude the model would perform as well against a hardened, well-monitored network. These are the honest numbers, and the honest numbers are still significant.

More significant than the single finding is what sits underneath it. Two years ago, the best available AI models could barely complete beginner-level cyber tasks. Now one has completed 32 sequential steps of a professional attack simulation. AISI reports that frontier AI capabilities in cyber offence are doubling every four months, twice the pace recorded just months ago. The finding is not the scary part, the trajectory is.

Recently I was at a cyber security conference. The room was full of security leaders, experienced, capable, serious professionals. The conversation that emerged in networking breaks and informal moments was not about AI capability. It was about something quieter and more uncomfortable. Most of them felt structurally unsupported, not underqualified, not unaware of the threat, but unsupported. Responsible for outcomes they could not fully control, in organisations that had not genuinely reckoned with what that means.

That conversation did not start this week. The AI evaluation did not create it. But the two things belong together, and most of the coverage since Tuesday has not connected them.

For years, the security community has watched opportunistic attacks accelerate. Bad guys begin scanning the internet for vulnerable systems within minutes of a new vulnerability being publicly announced. Attack times have compressed, and ransomware deployment that once took weeks now takes hours. That acceleration is not new, and anyone who has been paying attention is not surprised by it.

What has remained expensive, until now, is something different. Targeted, multi-stage intrusions, the kind that begin with reconnaissance, move through a network, escalate privileges, and end with full system compromise, have required two things that could not easily be automated or outsourced- judgement and adaptability. The ability to make contingent decisions across dozens of sequential steps, each one shaped by what the previous step revealed. That is what skilled attackers brought to the table. That is what made them scarce, and scarcity made them more expensive.

The AISI evaluation is significant precisely because of what it tested. Not whether an AI model could scan for known vulnerabilities. Whether it could complete 32 sequential steps of a professional network intrusion, from initial reconnaissance to full takeover, making adaptive decisions throughout. That is the category of attack that previously required a capable human. Now AI successfully exploited a system in three of ten attempts end to end, in an undefended environment.

AI is not making opportunistic attacks faster, they already are. It is lowering the skill floor for the attacks that were never fast, the targeted, adaptive, multi-step campaigns that organisations have quietly relied on being difficult to execute. That reliance was never a strategy. It was a structural feature of how scarce genuine attacker expertise was. That scarcity is now in question.

There is a philosophy in security that has existed for years now, passed through enough strategy documents and vendor presentations to have been bleached of almost all meaning. It goes by the name assume breach.

In its genuine form it is a serious and demanding idea. It means accepting, structurally, not performatively, that the attacker will get in. That the question is not whether a breach happens but how quickly you detect it, how contained the damage is, and how effectively you recover. It means orienting investment toward detection, resilience and recovery, not just prevention. It means building governance structures that treat a breach as a systemic risk event rather than an individual failure.

Very few organisations have actually done this.

What most organisations have done is put assume breach in the deck and leave everything else unchanged. Security leaders still carry personal accountability for preventing breaches. The board still treats a breach as evidence of individual failure. The investment profile still skews heavily toward keeping attackers out rather than assuming they are already in. Gartner’s 2024 Board Survey found that while 93% of directors recognise cyber risk as a threat to stakeholder value, two thirds rate their own oversight practices as inadequate to manage it. They know it matters. That is not the same as having genuinely reckoned with it.

The conference room I described earlier is what that gap looks like from the inside. Those security professionals are not failing at their jobs. They are operating inside a structural contradiction that most organisations have never acknowledged. Assume breach as a philosophy and holding a single individual accountable when the attacker gets in as a practice cannot both be true simultaneously. One says a breach is systemic and inevitable. The other says it is individual and preventable. Most organisations hold both positions without noticing the conflict.

That contradiction was always there. What AI has done is remove the margin for error that allowed organisations to sustain it without immediate consequence.

The government’s letter this week is not wrong to call this a wake-up call. But wake-up calls only work if the response is structural rather than reactive. Buying a new tool, commissioning a review, issuing a memo about cyber hygiene, none of that addresses the underlying problem, which is not technical. It is a governance problem dressed in technical clothing.

The questions worth asking are not questions for your security team. They are questions for your board.

Has your organisation formally accepted, in writing, in your risk register, that a breach is a question of when rather than if? Not in a presentation. In the governance framework that shapes how you invest and how you respond.

If a significant breach occurred tomorrow, would a single individual be held responsible? If the honest answer is yes, your organisation does not have an assume breach posture, it only has the words.

Does your investment in security focus mainly on keeping attackers out, or toward detecting and containing them once they are in, and recovering afterwards? Prevention-first is not wrong, absolutely always try to prevent. But prevention-only, in a threat environment where the cost of a capable attack is falling rapidly is not sustainable.

The AI finding matters. The acceleration is real. But the organisations most exposed this week are not the ones who failed to predict it. They are the ones who had already been told, by their own security leadership, in conference rooms and board papers and risk registers, and built a culture that made it impossible to hear.

The breach was always coming. AI just made it cheaper to deliver.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

UK AI Security Institute (2026), Our evaluation of Claude Mythos Preview’s cyber capabilities

aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities

UK Government (2026), AI cyber threats: open letter to business leaders (15 April 2026)

gov.uk/government/publications/ai-cyber-threats-open-letter-to-business-leaders

Gartner (2024), Board of Directors Survey: Cybersecurity as Business Risk

gartner.com/en/newsroom/press-releases/2024-11-13-gartner-says-80-percent-of-non-executive-directors-believe-current-board-practices-and-structures-are-inadequate-to-oversee-ai

Help Net Security (2024), CISOs in 2025: Balancing security, compliance, and accountability

helpnetsecurity.com/2024/11/13/daniel-schwalbe-domaintools-cisos-2025/

Something curious has happened in legislatures across the world. Politicians appear to have recently discovered that the internet is not always a nice place. There is a sudden urgency, a wave of bills, consultations, frameworks and mandates, all radiating the energy of people who have just encountered a problem for the first time. The problem, of course, has been there for twenty years. The question worth asking is not why they are acting now. It is why the answer they have all landed on is the same one: make everyone prove who they are.

The goal is framed as child protection, and I want to be clear that protecting children online is a genuinely important objective. But good intentions do not make bad policy good. And the policy being built right now is not designed around what actually harms children online. It is designed around what is easiest to legislate, cheapest to implement, and most convenient for everyone involved, except the people who actually use the internet.

The proposals follow a consistent pattern across jurisdictions: passport uploads, credit card checks, facial age estimation, government ID verification through third-party commercial providers. They vary in detail but share a fundamental flaw. Every one of them creates a linkable record. A commercial verification provider now knows that a specific individual accessed a specific type of content at a specific time. That data has commercial value. Under the right legal circumstances, it has governmental value. We are being asked to trade privacy for access, and told it is for the children.

The breach risk is real and well-documented, identity verification providers have suffered widely reported incidents involving exposed credentials, government ID images and personal records, sometimes at a scale of hundreds of millions of records. But focusing on breach risk actually understates the problem. Even a perfectly secure age verification system, one that never leaked a single record, would still be the wrong approach. Because the issue is not whether the data is kept safe. The issue is that the data exists at all.

Every record created is a surveillance artefact: a log of who accessed what, when, verified against a government identity. That infrastructure, once built, does not stay limited to its original purpose. It gets used as evidence in proceedings its creators never envisioned. It gets sold. It gets repurposed by the next administration, or the one after that. History is consistent on this point. You do not build surveillance infrastructure in a democracy and find that it only ever gets used for its stated purpose.

There is a pattern worth naming in how these laws are written. They mandate that age verification must be effective. They specify that checks must be robust, that compliance must be demonstrable, that outcomes must be measurable. What they rarely if ever mandate is that the systems doing this work must be secure.

The UK’s own GOV.UK One Login, the flagship government digital identity system now underpinning the digital driving licence, backed by over £300 million of public money, illustrates this precisely. A whistleblower raised serious security concerns in July 2022, shortly after the system went live. According to reporting by Computer Weekly and The Telegraph, those concerns included: development work outsourced to Romania without the knowledge or approval of the then-GDS chief executive, and without consultation with the National Cyber Security Centre; over 10,000 vulnerabilities rated critical or high severity; staff without the required security clearance accessing the live production environment over 6,000 times in a single month. The whistleblower was subsequently informed he faced disciplinary action for raising these concerns.

The government’s response to Parliament omitted any mention of the NCSC warnings or the Cabinet Office Data Protection Officer’s demand, made in November 2022, that the system be suspended. One Login lost its Digital Identity Trust Framework certification. It remains operational and is being expanded.

This is what it looks like when governments mandate identity infrastructure without understanding what they are building. The political pressure is to announce. The pressure to make it actually secure is absent, underfunded, or actively suppressed when it becomes inconvenient.

There is a better technical approach, and it has existed as a proven cryptographic concept for years: Zero-Knowledge Proof.

The idea is more straightforward than the name suggests. A website that needs to verify your age redirects you to a government authentication platform. You prove your identity there. The platform returns a single token, nothing more than “over 18: yes.” In a well-designed system, the government platform never learns which website you visited, and the website never learns who you are. No profile. No record. No artefact.

This is not science fiction. The technology is mature. Estonia has been running sophisticated digital identity infrastructure on similar principles for over two decades. The technical barrier is not the problem.

So why hasn’t this approach been seriously pursued? There are two honest explanations, and both deserve consideration.

The first is observation. Governments have spent years in legal conflict with technology companies over end-to-end encryption, consistently arguing that law enforcement needs access. The pattern of mandating identity verification systems that generate linkable records, while simultaneously resisting privacy-preserving alternatives, is consistent with an interest in knowing what citizens do online. That may sound conspiratorial. I am not claiming it is the dominant motive. But it is a coherent explanation for why the architecture being built looks the way it does.

The second explanation is simpler, and I think more likely: cost. Zero-Knowledge Proof infrastructure is harder to build and more expensive to implement than outsourcing verification to a commercial third party. The commercial identity verification industry exists, is available now, and is willing to absorb the implementation complexity in exchange for access to the data. Governments get a policy outcome they can announce. Industry gets a new mandated market and a commercially valuable data asset. The cost, the erosion of everyone’s privacy, is paid by citizens who had no say in the arrangement.

Peer-reviewed research published in 2024 found that age verification practices are inadequate precisely because official mandates lack technical guidance. Governments are specifying what must happen and leaving the how to an industry with a strong financial interest in building systems that collect data. The industry most incentivised to build surveillance-based verification is also the one being handed the brief.

Nothing illustrates this mindset more clearly than what has happened around VPNs.

A YouGov survey cited as evidence of public appetite for action asked whether under-18s should be restricted from using VPNs. The question defined a VPN as “a tool that hides a user’s internet activity and location, often used for privacy or bypassing content restrictions.” A neutral definition might have read: “a tool that encrypts your internet connection, widely used by businesses, journalists, and academics.” Same technology. Entirely different mental image. The demographic data compounds the problem: the age groups most likely to support restrictions were statistically the same groups least likely to know what a VPN is, a separate YouGov survey found only 47% of those aged 55 and over know what the acronym stands for. A quarter of respondents declined to answer at all.

The result, 55% in favour of restrictions, has since been used to justify a policy direction that extends well beyond the original question. In Wisconsin, a bill requiring websites to block all VPN users passed the State Assembly 69 votes to 22 before the provision was stripped following public backlash. In Michigan, a proposal would require ISPs to actively monitor and block VPN connections entirely. In the UK, the Children’s Commissioner has called VPNs “a loophole that needs closing” and the Prime Minister has confirmed the government is considering restrictions following consultation.

VPNs are not a loophole. They are standard security infrastructure used by business, university students and academics, journalists, and domestic abuse survivor hiding their location. The question the legislation never asks is why a content restriction approach is so inadequate that a freely available privacy tool defeats it entirely.

There is also a practical problem that appears not to have been considered. VPN legislation does not respect borders any more than VPNs do. A website cannot determine where a VPN connection originates, that is the point. Any site seeking to comply with the most restrictive law in any jurisdiction has no practical option but to block all VPN traffic, everywhere. A single state legislature in Wisconsin was, inadvertently, drafting policy with global consequences for every VPN user on the planet.

It is easy for governments to announce bans, restrict privacy tools and legislate against security infrastructure, and to frame anyone who objects as someone who does not want to protect children. That framing is not only unfair. It entirely misses the point.

Protecting children online matters. The harms are real and documented. But children are not damaged by platforms knowing their age. They are damaged by algorithmic amplification of harmful content, by design patterns engineered for compulsive engagement, and by inadequate moderation of abuse. Those are engineering and governance problems. Age verification does not address any of them. It is cheaper and easier to mandate than the solutions that would actually work, so that is what gets mandated.

We have better tools. Zero-Knowledge Proof exists. Privacy-preserving digital identity infrastructure exists and has been deployed at national scale. What is missing is not the technology. It is the political will to spend the money, do the harder work, and resist the commercial interests that profit from the current approach.

The chilling effect of surveillance infrastructure on legal behaviour is well-evidenced, research going back to the Snowden revelations documents how people self-censor, stop searching for sensitive information, and withdraw from online discourse when they believe they are being watched. Once you establish the principle that identity must be produced to access online content, that boundary moves in one direction. The open internet, where anyone could seek information without declaring who they are, has been one of the most democratising forces in modern life. Dismantling it in the name of child protection, using systems that do not protect children, while the people raising security concerns get disciplined for doing so, is not a policy. It is an abdication of one.

The question nobody seems to be asking is the obvious one: how many people have to hand their identity to unregulated third parties, and how many breaches have to happen, before someone admits that this approach is neither child protection nor data protection, and never was?

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Computer Weekly (2025), Government faces claims of serious security and data protection problems in One Login digital ID, computerweekly.com/news/366622533

The Telegraph / Yahoo News (2025), Government digital ID system put citizens’ data at risk, yahoo.com/news/government-digital-id-system-put-114120788.html

Datamation (2025), UK Digital ID Card Launch Gets Hostile Reception, datamation.com/security/uk-digital-id-cards

Electronic Frontier Foundation (2024), Hack of Age Verification Company Shows Privacy Danger of Social Media Laws, eff.org/deeplinks/2024/06/hack-age-verification-company-shows-privacy-danger-social-media-laws

Electronic Frontier Foundation (2025), The Year States Chose Surveillance Over Safety, eff.org/deeplinks/2025/12/year-states-chose-surveillance-over-safety-2025-review

Electronic Frontier Foundation (2025), Lawmakers Want to Ban VPNs, And They Have No Idea What They’re Doing, eff.org/deeplinks/2025/11/lawmakers-want-ban-vpns-and-they-have-no-idea-what-theyre-doing

Electronic Frontier Foundation (2026), EFF to Wisconsin Legislature: VPN Bans Are Still a Terrible Idea, eff.org/deeplinks/2026/02/eff-wisconsin-legislature-vpn-bans-are-still-terrible-idea

TechRadar (2026), Wisconsin scraps VPN ban from age verification bill following backlash, techradar.com/vpn/vpn-privacy-security/wisconsin-scraps-vpn-ban-from-age-verification-bill-following-backlash

Renaud, K. et al. (2024), Online Age Verification: Government Legislation, Supplier Responsibilization, and Public Perceptions, PMC/MDPI, pmc.ncbi.nlm.nih.gov/articles/PMC11429505

Internet Society (2024), Age Verification Law Weakens Internet Privacy and Security, internetsociety.org/blog/2024/09/texas-mandatory-age-verification-law-will-weaken-privacy-and-security-on-the-internet

Büchi, M., Festic, N. & Latzer, M. (2022), The Chilling Effects of Digital Dataveillance, journals.sagepub.com/doi/10.1177/20539517211065368

YouGov (2024), VPN Awareness Survey, yougov.co.uk

YouGov (2025), Under-18 VPN Restriction Survey, yougov.co.uk

The Conversation (2025), Online age checking is creating a treasure trove of data for hackers, theconversation.com/online-age-checking-is-creating-a-treasure-trove-of-data-for-hackers-268586

There is a story IT departments have been telling themselves for twenty years. It goes like this: employees use unauthorised tools because they don’t understand the risks. If we communicate the policy more clearly, enforce it more consistently, and block enough stuff, the problem will go away.

The data has never supported that story.

Gartner found that 41% of enterprise employees were already working outside IT oversight in 2022, and projects that figure will reach 75% by 2027. Shadow IT didn’t grow despite tighter controls. It grew alongside them. That’s not a compliance failure. That’s a signal, and most organisations spent two decades responding to it with the wrong answer.

The signal was simple: the tools we’re providing aren’t good enough for the work people are actually trying to do. The employee who used personal Dropbox wasn’t trying to undermine information security. They were trying to share a file with a client when the VPN was down and the deadline wasn’t moving. The WhatsApp group handling client updates wasn’t a governance failure. It was a faster answer to a problem the approved toolset couldn’t solve.

The wrong response to shadow IT is blanket prohibition. Locking everything down frustrates good employees, drives workarounds underground rather than eliminating them, and signals that the IT function exists to slow the business down rather than support it. Most organisations chose prohibition anyway. And now, with Shadow AI arriving at a scale that makes the Dropbox era look quaint, we are at serious risk of making the same mistake again.

A colleague told me recently that they’d read my writing and assumed I was an AI sceptic. My inner nerd was genuinely shocked, did they not read about my home AI lab, experiments with multiple models, and apprenticeship because I find this technology genuinely extraordinary. My inner director, on the other hand, felt quietly vindicated and grown-up, because asking hard questions about data handling, governance, and risk isn’t scepticism, it’s the job. There’s a version of the AI conversation that treats those questions as obstacles. I think that version is the riskier one.

65% of organisations now have employees using unsanctioned AI tools. 78% of workers bring their own AI tools to work. This isn’t a niche behaviour. It’s near-universal. The question isn’t how to stop it. It’s what it’s telling us about where the friction is, and what we’re failing to provide.

The arrival of genuine citizen development tools has changed the calculus in ways that make the old orthodoxy untenable. The traditional IT position, always buy, never build, because you can’t support what you didn’t commission, made sense when building meant developers, procurement cycles, and maintenance contracts. That world has genuinely shifted. A Forrester study found organisations using Microsoft Power Platform achieved 206% ROI over three years, with high-impact users saving up to 250 hours annually and app development time cut by 50%. These are not marginal gains. The tools have earned a place at the table.

But here is where the conversation gets uncomfortable. There are voices in the AI industry who have taken the legitimate case for citizen development and extended it into an argument for removing governance entirely. Get IT out of the way, move fast, procurement is just friction. The implicit message is that due diligence is timidity, and that professionals who ask hard questions about data handling and compliance are obstacles to progress rather than people doing their jobs. I’ve seen both failure modes up close, IT teams that used process as a moat, guarding their function more carefully than the data they were supposed to secure, and organisations pressured into rushing deployments that later surfaced serious problems. Gatekeeping is real, and so is the cost of absent governance. The answer isn’t to pick a side. It’s to ask what governance should actually look like when the tools have changed.

The starting point has to be following the data, not categorising the tool.

Consider two AI agents that do identical things. They join a meeting, generate a summary, draft follow-up actions, and distribute them to attendees. In a product planning session, the risk profile is manageable. In a meeting discussing vulnerable adults or children, the questions change entirely. Not just whether a human reviews the output, but what data is being processed, where it’s transmitted, under what data processing agreement, and whether the organisation has a lawful basis for sending that information to an external model at all. Anyone who has watched production software go live without proper scrutiny knows how this ends. The risk doesn’t disappear when you skip that conversation. It just becomes invisible until it isn’t.

The tool is identical. The data context changes everything. Governance has to follow the data, not the technology.

IBM’s 2025 Cost of a Data Breach Report found that organisations with high shadow AI exposure faced an additional average breach cost of $670,000, with 65% of incidents involving personally identifiable information. The Samsung case is instructive here, not because Samsung was careless, but because the incident illustrates how quickly well-intentioned employees can expose sensitive data when the approved route doesn’t exist and the unsanctioned one does. Three separate teams submitted proprietary source code and internal meeting recordings to ChatGPT within weeks of the company lifting a prior ban. The response, reimposing the ban, missed the point entirely. Security experts noted that banning specific tools one by one becomes whack-a-mole as new ones proliferate. The only sustainable answer is a sanctioned route that’s faster and safer than the shadow one.

Which brings us to the other failure mode: governance so slow it defeats itself.

IDC research, undertaken with Lenovo, found that 88% of AI proofs of concept never reach production, for every 33 pilots launched, only four go live. IDC’s own researchers acknowledged that many of these pilots are “highly underfunded” and lack a strong business case from the start, which means the problem isn’t just governance, it’s launching without clear purpose. But slow, undefined governance compounds it. Getting stuck in pilot purgatory is what happens when nobody defined what success looked like before the pilot started. The review runs indefinitely because there’s no decision to make, only a process to continue.

Gartner predicts 30% of GenAI projects will be abandoned after proof of concept, citing poor data quality, inadequate risk controls, and escalating costs. The pattern is consistent: organisations launch with enthusiasm and stall at the point where unglamorous structural work is required. That stalling recreates exactly the problem shadow IT diagnosed. If the sanctioned route takes eighteen months and produces no answer, people find another route. They always have.

The fix isn’t faster approval. It’s defined exit criteria before the pilot begins. Not “we’ll review in three months” but “here is what this project needs to demonstrate, here are the data questions it needs to answer, and here is the date by which we will decide.” That’s a decision process. What most organisations run instead is a review process, and review processes don’t end, they just lose momentum until something else takes priority.

Those exit criteria need the right people in the room: IT, the business owner, and whoever owns the data risk. Depending on the data context, legal or compliance too. That conversation, held before anything is built, is the governance model. Not a committee and not a checklist, but a conversation with accountability attached.

The IT teams that will navigate this well are not the ones that said yes to everything, or the ones that built walls around their function and called it risk management. They’re the ones that got curious about why their users kept going around them, and built something worth coming back to.

A KPMG survey found 73% of organisations adopting low-code platforms had not yet defined governance rules. That gap is where shadow AI lives. Close it not with prohibition but with a sanctioned environment that actually works: risk-proportionate governance, fast and transparent pathways from experiment to production, and a clear signal to the organisation that IT is a partner in building things, not a gatekeeper deciding who gets to try.

Shadow IT was never really about the tools. It was about unmet need meeting inadequate response. Shadow AI is the same conversation, with higher stakes and less time to get it right. The writing has always been on the wall. The question is whether we’re finally ready to read it.

This post is part of an ongoing series on AI, technology, and the gap between what we are promised and what we are building.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Leave a comment

References

Gartner (2022), Shadow IT and Employee Technology Use gartner.com

Microsoft / LinkedIn Work Trend Index (2024), AI at Work microsoft.com

Forrester Consulting (2024), Total Economic Impact of Microsoft Power Apps forrester.com

IDC / Lenovo (2024), AI Proof of Concept to Production Research idc.com

IBM (2025), Cost of a Data Breach Report ibm.com/security/data-breach

Gartner (2025), GenAI Project Abandonment Predictions gartner.com

KPMG (2023) Shaping Digital Transformation with Low-Code Platforms assets.kpmg.com/content/dam/kpmg/ie/pdf/2023/07/ie-shaping-digital-transformation-with-low-code-platforms.pdf

Dark Reading / Gizmodo / TechCrunch (2023), Samsung ChatGPT Data Leak darkreading.com

Editor’s note: An earlier version of this article cited a figure of 98% of organisations having employees using unsanctioned AI tools. On review, although this figure appears quite a bit on Google, it does not appear to have a clearly attributable primary source. I have replaced this figure with the Microsoft 2024 Data Security Index figure of 65% which is better evidenced. Still a lot, but not quite as much as I said at first.