The Home Office runs two campaigns simultaneously. Stop! Think Fraud warns the public that sharing personal information online exposes them to identity theft. The Online Safety Act, enforced by the same department, makes sharing your identity documents with commercial websites a legal requirement.

Nobody in government appears to have noticed the contradiction, or they noticed and did not care.

This is not an option or a recommendation, it is a legal requirement, enforced by Ofcom, with fines of up to 10% of global revenue for platforms that fail to collect it. The government did not change its assessment of the risk and it did not solve the underlying security problems. It decided that the policy outcome it wanted was worth ignoring the danger it had spent years warning you about. The threat is identical, the advice has simply been reversed.

I want to be precise about what this law actually covers, because the public debate has not been.

You have heard this described by the government and the media as a social media ban for under-16s, but that is not what this is. It is mandatory ID verification for every adult in the UK to access the internet.

The Online Safety Act 2023 applies to any “user-to-user service”, any platform where content generated by one user can be encountered by another. The Children’s Wellbeing and Schools Act 2026 extends that further, requiring the government to impose age or functionality restrictions for all users under 16 across regulated social media platforms. Together, they do not describe a targeted intervention. They describe identity verification as a condition of internet access, for every adult in the country, with child protection used as the framing to avoid calling it what it is.

That definition is extremely broad. It captures social media, obviously. It also captures gaming platforms, discussion forums, community sites, storefronts with review sections, and spectator modes in video games. Ofcom estimates more than 100,000 websites fall within scope. Steam now requires a credit card to access mature content. Discord triggers compliance obligations simply by offering NSFW channels. Nexus Mods, a site where gamers download community-made modifications for video games, now requires UK users to submit a government ID or facial scan to access content tagged as adult.

Services that cannot afford to comply have a third option, they just geo-block the UK entirely, it is already happening. The internet available to UK residents is quietly becoming a different, smaller internet, not because content has been removed, but because the compliance cost of serving British users is no longer worth it.

There are legal exemptions, technically. A news site where readers can only comment on an article, not reply to each other, might qualify as a “limited functionality service.” That exemption disappears the moment users can respond to each other’s comments, which most modern platforms allow. Legal analysis raises this question directly and provides no clear answer. We will not know how courts interpret the boundary until someone is prosecuted and we have case law. Until then, 100,000 services are making compliance decisions based on their best guess, with existential consequences if they guess wrong. That is not governance. It is legislative theatre.

The mechanism chosen for enforcement is the specific problem.

The government is not building a verification system. It is demanding that commercial third parties build one, and mandating the public use it. Photo ID matching, facial age estimation, biometric scanning: the common factor is a private company receiving your identity data as a legal condition of platform access. The Online Safety Act specifies that age verification must be robust, but it places no meaningful security standard on the companies performing it, no data residency requirements, and no restrictions on those companies being acquired by foreign entities. A provider incorporated in London today can be headquartered in California tomorrow. The data moves with it, and the Act has nothing to say about that.

Those companies will tell you their systems are privacy-preserving by design. They will tell you data is not retained. Whilst this may be true of their stated architecture, it tells you nothing about the security of the infrastructure underneath it, nothing about what happens after an acquisition, and nothing about whether any of those claims have been independently audited against a standard with actual teeth.

The government has outsourced not just the implementation but the liability. When the breaches come, and they will, the government will point at the provider, the provider will point at its terms of service, and the people whose data was compromised will have no meaningful recourse. A breached password can be changed. A copy of your passport, driving licence, or home address, linked to a face scan linked to a verified social media identity cannot be unlinked. That record exists permanently, for every bad actor who acquires it now or in the future.

And the risk compounds. Every platform you verify with creates a new database entry. Each submission is a new point of failure. The same face scan submitted to five services does not create one risk five times over, it creates five risks that can be correlated against each other. The aggregate profile that emerges maps your identity across your entire online life. That is not just a data breach waiting to happen. It is a surveillance asset being built, one legal requirement at a time.

There are groups for whom this is not an abstract concern. It is a direct physical safety risk.

Victims of domestic violence depend on online pseudonymity as a survival mechanism. Their support networks, their access to legal advice and specialist services: all of it exists under a layer of separation between their real identity and their online presence, and mandatory identity verification collapses that separation. A verification database linking a real identity to a platform account is a resource. Whether it gets breached, subpoenaed, sold, or accessed by people who should not have it, an abuser with that data, through any route, has a tool for finding someone who has spent considerable effort not being found.

People at elevated risk of doxxing face the same problem: journalists, activists, trans individuals, and anyone who has previously been the target of a coordinated harassment campaign. The separation between real identity and online presence is not paranoia. It is a rational response to a documented threat. The law removes it as a side effect of a policy that does not mention these people once.

The framing of this legislation has been designed to make coherent opposition almost impossible.

Anyone who raises data security concerns is implicitly questioning whether children should be protected online. Anyone who points out the scope is wider than advertised is accused of obfuscating a simple child safety measure. The political angle is deliberate: manufacture a situation in which the only publicly acceptable position is compliance, and then legislate for whatever you wanted in the first place.

Privacy-preserving alternatives exist. Cryptographic verification capable of returning a yes/no age confirmation without creating a linkable identity record has been deployed at national scale elsewhere. It costs more and requires the government to own the issue rather than outsourcing it to an industry with a financial interest in collecting data, so it has not been seriously pursued.

What has been built instead is a compulsory surveillance architecture, held by private companies, with no meaningful security floor. It covers a scope of internet activity that has never been honestly communicated to the public. It operates under legal uncertainty that will not resolve until the prosecutions begin.

This is not speculation about what surveillance does to behaviour. Research published in the Berkeley Technology Law Journal found that searches on sensitive topics dropped by nearly 30% after the Snowden revelations simply made people aware that government monitoring was possible. People did not need to be watched. They needed to believe they might be. Mandatory identity verification, linked to platform access, creates exactly that condition permanently, and by law.

The UK government just made it illegal to protect your own privacy online. They called it keeping children safe.

Share

Subscribe now

I have agents running at home, on a mini PC under my desk, because I really wanted to understand the token usage and what the cost would be on different models. OpenClaw runs open models on hardware I own, so there is no invoice, only a token meter. I have spent a lot of time watching that token meter, and what it taught me is this: an agent does not work like software. It works like an employee who is only paid overtime, where every minute costs more than the minute before, and the clock does not stop until the job is done.

That is not a metaphor, it is the maths. At every step it takes, an agent re-reads everything that came before, the whole accumulating conversation along with its internal thoughts, before its next move. As the context grows, the cost of each step grows with it. Five steps in and the fifth step is not costing what the first one did, it costs more. Ten steps in and it costs more again. The bill does not add up, it compounds.

You cannot see any of that from inside a subscription. Indeed, not seeing it is the whole point of a subscription. And for the last two years, two things have trained us not to look. The first is twenty years of SaaS, which taught every budget holder that software is a fixed monthly cost, per user, per month, predictable and flat. The second is the early hype of AI, which taught everyone that intelligence is cheap, practically free, bolted on to the tools you already have.

Neither is true for agents. Last week I wrote about the frontier labs withdrawing the subsidies that made AI look affordable. This week I want to go further, because the subsidy was hiding something worse than a future price rise. It was hiding a cost model that makes flat pricing structurally impossible.

Look at what Anthropic did with Fable 5. It launched its most capable public model on the ninth of June, included it in the Pro, Max and Team plans, and announced from day one that it would move to usage credits after two weeks. Anthropic knew before the public ever saw the model that it could not survive inside a flat subscription. This week I tested Opus 4.8 and Fable 5 on the Pro plan, and can confirm they went through my allowance faster than the first round of drinks at the pub. After that you pay API rates. The best model the public can buy was never going to stay inside a flat monthly price. Anthropic did not discover this, it designed around it.

Fable 5 is not an outlier, it is the pattern arriving. Per-user, per-month pricing worked for twenty years because software had almost no marginal cost. Once you had built it, one user or a thousand, once a day or fifty times, it cost you nearly the same to serve. The cloud kept the compute cheap and crucially, predictable. So you priced for access and everyone got used to the flat fee.

Agents change that. Every action burns tokens that cost real, variable money. The harder the task, the longer the context, the more the agent had to think, the more tools the agent ran, the higher the bill. Inference has put marginal cost back into software, and not the gentle, linear kind. The overtime kind, where the late minutes cost more than the early ones and nobody told the customer. Industry estimates suggest that inference costs dropped 280-fold in two years while total AI spending rose 320% over the same period. Gartner put it plainly in March: do not confuse the deflation of commodity tokens with the democratisation of frontier reasoning.

Now look at what the vendors are actually selling. Most of the AI-powered products that arrived in the last two years are fundamentally orchestration layers sitting on top of the same handful of frontier models, Claude, GPT, Gemini, wrapped in a branded interface and sold at a flat monthly rate. Legal tools, sales tools, recruitment platforms, customer service bots. Different industries, different logos, the same models underneath. Klarna built its entire customer service on OpenAI and handled two-thirds of all chats with it within a month. Different logo, same model.

Every one of them is now shipping agents. Salesforce calls Agentforce a digital labour platform and pitches its agents as digital employees you hire by the click. Others are quieter about it but doing the same thing, bolting agentic workflows into the seat you already pay for and calling it a feature upgrade. The promise is a tireless new colleague for the price you are already paying.

The maths does not support it. Underneath every flat price is an overtime bill the vendor cannot predict, because on the builder platforms they now sell, you decide what the agent does. You set the task, you choose the complexity, you feed it a hundred-page contract or a ten thousand-row spreadsheet. You author the workload. The vendor funds it. Neither of you knows the bill until the job is done, and the vendor learned that before you did.

None of this depends on which model the vendor chose. The token bill compounds with every step the agent takes, on any model, at any price point. The model choice changes the rate on the meter. It does not switch the meter off. So the market is splitting, and you can already see it happening. Cursor, the AI code editor, includes generous agent usage on its $20 Pro plan, but only when Cursor picks the model. It can afford to be generous because it routes to cheaper ones. Choose the frontier model yourself and it burns through a $20 credit pool, after which you pay API rates. Same tasks, same agent. The only variable is which model runs it.

And behind all of it, open-weight models are growing fast. The open-source LLM market hit $21 billion in 2025 and is expanding at 34% a year, with on-premises deployment growing at 29% as organisations chase data control and predictable costs. This is not a hobbyist movement, it is a real option for vendors managing their own inference bill and for buyers who want models on infrastructure they control.

The honest picture has three positions, and all of them are legitimate. Frontier capability on a meter, where you get the best model and pay for what you use. A hybrid, with a predictable base and usage charges beyond it. Or a flat fee on a capped tier, where the model may not be the newest but the cost is bounded. The question is not which position is right. It is whether you chose yours or whether you are aware of and comfortable with which one your vendor chose.

If you are buying an agentic product today on a flat monthly fee, ask the vendor one question before you sign: how are you handling the rising cost of inference, and what does your pricing look like in twelve months? If they cannot answer that clearly, they either do not know or do not want to tell you, and neither is a reason to sign.

Anyone who was sold an agent as a free colleague is about to get their first honest timesheet.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

1. Anthropic, “Claude Fable 5 and Claude Mythos 5,” 9 June 2026. anthropic.com/news/claude-fable-5-mythos-5

2. Klarna, “Klarna AI assistant handles two-thirds of customer service chats in its first month,” 27 February 2024. klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/

3. Bloomberg, via Entrepreneur, “Klarna Is Hiring Customer Service Agents After AI Couldn’t Cut It on Calls, According to the Company’s CEO,” 9 May 2025. entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396

4. Gartner, “Gartner Predicts That by 2030, Performing Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025,” 25 March 2026. gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025

5. Cursor AI pricing and Auto mode routing. NxCode, “Cursor AI Pricing 2026: Free vs Pro vs Business,” March 2026. nxcode.io/resources/news/cursor-ai-pricing-plans-guide-2026

6. Salesforce Agentforce: “Digital Labor Platform.” salesforce.com/agentforce

7. Technavio, “Open-source LLM Market Growth Analysis - Size and Forecast 2026-2030,” May 2026. technavio.com/report/open-source-llm-market-industry-analysis

This week, I have been building an agentic environment in my home lab. Nothing glamorous, just a set of AI agents designed to handle various tasks, running on locally hosted models. Most of the online guides I read talked about using cloud models like Claude or ChatGPT for agents, but I wanted to see how well it would run with local open source models.

The limitations became apparent quickly.

When you cannot just throw a task at a frontier model and let it reason its way to an answer, you have to think. Which agent actually needs to read the whole document, and which one only needs a summary? How much context does each step genuinely require? Do I need one agent trying to do everything, or four smaller ones each doing one thing well? I started with a monolithic architecture. I ended up with a small swarm of four agents, each scoped tightly to its task and running on a model matched to what that task actually demands.

I ran the numbers on what the same workflow would cost using a frontier model via API. Roughly one dollar per run. At the volumes I was considering that felt manageable, but then I thought about scale and asked myself what happens at a hundred runs a day? A thousand runs a day? What happens when the model’s next version uses three times as many tokens to reach the same answer, because the reasoning chain got longer? What happens when the price per token increases because the lab that set it decided the subsidy period was over?

That is not a hypothetical, it is a question that most organisations deploying AI right now are not asking, and they should be.

Current frontier AI pricing is structurally and heavily subsidised. OpenAI reported $3.7 billion in revenue in 2024 and lost $5 billion doing it. By the end of 2025, the company’s CFO was reporting an annualised revenue run rate exceeding $20 billion. OpenAI is still projected to lose roughly $14 billion in 2026. Revenue tripling does not mean the subsidy is ending, it means the subsidy is scaling.

The labs need adoption more than they need margin right now. Like most disruptive technologies, capturing the market comes first and profitability comes later. It is the same playbook that made cloud computing feel free until it didn’t, the same logic that kept ride-hailing cheap until the drivers needed paying. The difference is that organisations are not just buying a productivity tool this time. They are building processes around it. Automating workflows and training teams to depend on specific model behaviours. Embedding frontier API calls into the operational fabric of how they work.

When the floor moves, those decisions will be expensive to revisit.

Sam Altman said recently that he was “delighted to be wrong” about AI’s impact on white-collar jobs. The reversal was widely reported as reassurance, but what it actually signals is worth examining. It arrived the same week OpenAI reportedly filed IPO paperwork confidentially. A calmer narrative around AI’s economic disruption is considerably better for a public listing than the jobs apocalypse framing he was running twelve months ago. The people who set the price of the infrastructure you are building on have a direct financial interest in how you feel about it.

That is not a conspiracy, it is an incentive structure worth knowing about.

The floor is already moving. GitHub announced at the end of April that all Copilot plans would transition from flat-rate subscriptions to token-based billing on 1 June 2026. In its own announcement, GitHub explained why with unusual candour: “Today, a quick chat question and a multi-hour autonomous coding session can cost the user the same amount. GitHub has absorbed much of the escalating inference cost behind that usage, but the current premium request model is no longer sustainable.” GitHub is not an outlier. Cursor made a similar shift in June 2025, moving from request-based limits to credit pools tied to API costs, poorly enough communicated that the company issued a public apology and offered refunds. Windsurf followed in March 2026. We see the same headline monthly prices, but the bills are going up. The market is converging on the same structure, and the reason is always the same: agentic workflows broke the flat-rate economics.

The tokenmaxxing stories of the past month are not cautionary tales about individual excess. They are what unconstrained frontier API access looks like at scale.

Uber deployed Claude Code to around 5,000 engineers and watched adoption climb from 32 percent in February to 84 percent by March. Per-engineer API costs reached between $500 and $2,000 a month. By April, the company had exhausted its entire planned 2026 AI budget only four months into the year. The CTO reported spending $1,200 in a single two-hour session.

Microsoft introduced Claude Code to thousands of engineers across its Experiences and Devices division, the team responsible for Windows, Microsoft 365, Outlook, Teams, and Surface in December 2025. Engineers preferred it to the in-house alternative and used it constantly. By May, Microsoft was cancelling the licences, effective the last day of the financial year because the expensive tool worked too well. That is the part that gets under-reported: the problem was not that engineers were wasting tokens. The problem was that they were not.

Elsewhere, an AI consultant told Axios that one of their enterprise clients ran up a $500 million bill on Claude in a single month. No spending caps, no usage controls, just unrestricted access and a workforce that used it. That figure comes from a single unnamed source and has not been independently confirmed. But the pattern it describes, with costs that compound invisibly until they arrive all at once, is consistent with everything else happening in this space right now.

The mechanism matters here, and it is specific to agentic workflows. A single prompt to a language model is a bounded transaction. An agent running a multi-step workflow is not. It reads context, reasons, makes decisions, calls tools, then re-reads everything, the original prompt, every response, every tool output, before the next step, the context snowballs. A peer-reviewed study published in April 2026 found agentic tasks consume up to 1,000 times more tokens than standard model interactions. Model updates that shift reasoning architecture can change your consumption profile overnight, with no change to your code. The workflow that cost one dollar per run this quarter may cost five next quarter, because the model got better at thinking and thinking costs tokens.

The headline pricing narrative is that AI is getting cheaper, but the rate cards tell a different story. Claude Opus 4.8 costs five times more per token than Claude Haiku 4.5. Google’s Gemini Flash tier, designed as the affordable option, has risen five-fold in input price in under a year. One independent developer noted last week that all three major labs appear to be “probing the price tolerance of their API customers.” Token consumption is also rising faster than anyone budgeted for. You are paying more per unit, for more units, and the unit count is accelerating.

There is a question that most organisations deploying agentic AI are not asking. It costs nothing to ask now and a great deal to answer later.

Which tasks in this workflow actually require frontier reasoning?

Routing a document, classifying a ticket, summarising a meeting transcript, none of these require the most capable model on the planet. According to Epoch AI, the most capable open-weight models now lag frontier closed models by an average of four months on aggregate capability measures. On coding and production workloads specifically, independent benchmarks put the gap as low as two to three percentage points, while open-weight models cost six to seven times less per output token. The gap that remains is real, but common enterprise workloads are not in it.

Frontier models earn their place. Complex reasoning under ambiguity, novel analysis, judgement calls at the edge of a model’s capability, these are genuinely different tasks that do benefit from the best available models. The question is not whether to use frontier models, it is whether every step in every workflow needs them.

The organisations that are not asking this question are not making a considered architectural choice. They are making the path-of-least-resistance choice while the pricing floor is low and the pressure to deploy is high. Whilst that is understandable, it is also how you end up with a system you cannot change without rebuilding it.

Migration costs more than people model as it is not just rewriting API calls. It is revalidating outputs, because different models produce subtly different results and the downstream processes were calibrated to the original ones. It is rewriting prompt logic tuned over months to a specific model’s behaviour. It is re-testing agentic chains where one agent’s output format feeds the next agent’s input. And it is redoing the governance and risk assessment you completed the first time, under the time pressure of a system already in production.

That is the real lock-in, not contractual, but architectural.

Last week, I wrote about AI sovereignty, about what it means for public institutions to run critical infrastructure on systems controlled by private shareholders in another jurisdiction. The inference pricing question is the same argument, just one layer down.

An organisation that has routed its operational workflows through a frontier API has not just taken on a vendor relationship. It has taken on exposure to that vendor’s pricing decisions, its infrastructure availability, its jurisdictional obligations, and its commercial priorities. For most organisations, that is a manageable business risk. For critical national infrastructure, energy, water, transport, healthcare, it is a different category of problem entirely. Embedding frontier API dependencies into operational technology creates single points of failure in systems that were previously distributed and resilient by design.

The argument for thinking about this now is simple. The constraint my home lab imposed on me, to think about what each agent needs, to match the model to the task, to design for predictability and cost, is the same constraint that every organisation will face eventually. The difference is that I just faced it at home. Organisations that defer it face it later, under budget pressure, with running systems, and with users who have reorganised their work around the existing architecture.

GitHub named the problem in its own announcement. A quick chat and a multi-hour agentic session should not cost the same. The flat-rate era assumed they would and it was wrong. The organisations now building agentic workflows on frontier APIs without asking which tasks actually need that level of capability are making the same assumption, they are just making it more expensive.

The floor is not permanent, and the decisions made while it holds are not either. But reversing them later, on running systems, under budget pressure, will cost considerably more than asking the right questions now. The bill is coming, the only question is whether we see it coming.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

GitHub Blog. (27 April 2026). GitHub Copilot is moving to usage-based billing. github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing

Flexprice. (18 April 2026). The complete guide to Cursor pricing in 2026. Cites Cursor public apology of 4 July 2025 and June 2025 billing change. flexprice.io/blog/cursor-pricing-guide

Forbes / The Information. (April 2026). Uber burns through entire 2026 AI budget in four months after Claude Code deployment. Reported across multiple outlets.

The Verge / TechRadar / People Matters. (May 2026). Microsoft cancels Claude Code licences across Experiences and Devices division. Reported across multiple outlets.

Axios via Tech Startups. (May 2026). Enterprise client runs up $500 million Claude bill in a single month. Source: unnamed AI consultant. Unverified, unattributed. techstartups.com

Bai et al. (29 April 2026). How Do AI Agents Spend Your Money? Analysing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750. Authors include Erik Brynjolfsson, Stanford Digital Economy Lab. arxiv.org/abs/2604.22750

Simon Willison. (19 May 2026). Gemini 3.5 Flash: more expensive, but Google plan to use it for everything. Source of “probing the price tolerance of their API customers” quote. simonwillison.net/2026/May/19/gemini-35-flash

Edwards, J. and Emberson, L. (2026). Open models lag state-of-the-art closed models by 4 months. Epoch AI. epoch.ai/data-insights/open-closed-eci-gap

OpenAI financial figures: $3.7bn revenue / $5bn loss 2024 — multiple sources. $20bn ARR and $14bn projected 2026 loss — Fortune, CNBC, Reuters, January 2026. OpenAI CFO Sarah Friar blog post, 18 January 2026.

Claude API pricing: Anthropic platform pricing page. Haiku 4.5 at $1/$5 per million tokens; Opus 4.8 at $5/$25 per million tokens. anthropic.com

Gemini Flash pricing trajectory: Gemini 2.5 Flash at $0.30/$2.50 (June 2025) to Gemini 3.5 Flash at $1.50/$9.00 (May 2026). Google AI / Gemini API documentation and simonwillison.net analysis.

Deep Infra. (May 2026). Open-Source vs Closed-Source AI Models: Is the Gap Worth It? Cites 2–3 percentage point coding benchmark gap and 6–7x output token cost advantage for open-weight models. deepinfra.com/blog/open-source-vs-closed-source-ai-models-price-gap

In May 2025, the Chief Prosecutor of the International Criminal Court lost access to his Microsoft 365 account. The cause was a sanctions order, not a cyber-attack, not a breach, not a failure of any system to work as designed. The infrastructure did exactly what it was built to do, it complied with its legal obligations. The court is now migrating 1,800 workstations away from Microsoft entirely.

Nobody acted with malice toward the court’s IT infrastructure and no decision was made with the intention of disrupting a legal institution. A compliance mechanism, operating automatically, encompassed email accounts along with everything else it was required to reach.

That is not a theoretical risk. That is a Tuesday.

Microsoft has not ignored European concerns. Since January 2024, commercial and public sector customers can store and process their data entirely within EU and EFTA regions. The EU Data Boundary project was real and valuable and it deserves acknowledgement. However, it does not solve the problem.

Data residency and sovereignty are not the same thing. Storing your data in Frankfurt does not change who controls the infrastructure or who decides what the terms look like next year. And it does nothing about the CLOUD Act, US legislation that allows American authorities to compel US companies to produce data stored anywhere in the world, including inside those Frankfurt data centres. The residency commitment and the CLOUD Act sit in the same infrastructure simultaneously. The ICC’s data was presumably compliant with every applicable regulation, but the prosecutor still lost his email. Residency tells you where your data sits. It says nothing about who holds the keys.

Germany’s openDesk project is worth understanding, because it illustrates both the right instinct and the scale of what is missing. OpenDesk is a suite of open source collaboration tools, document editing, file storage, video conferencing, project management. All assembled and maintained by Germany’s Center for Digital Sovereignty and backed by the Federal Ministry of the Interior. Launched in October 2024, it is designed as a public sector alternative to Microsoft 365. The Bundeswehr, Germany’s armed forces, has signed a seven-year framework agreement to adopt it. Over 1,500 public bodies have made enquiries. The intent is exactly right but the investment is not equal to the problem.

Germany has put approximately €45 million into openDesk over several years. Microsoft’s annual R&D budget is $30 billion. Those numbers belong in the same sentence, not to dismiss openDesk, but because the distance between them is the distance between European and British ambition and what it would actually take to get there.

Valve, the games company behind the Steam platform, understood this distinction when they built Proton. Linux, the open source operating system used by most of the world’s servers had always struggled on the desktop, particularly for gaming. The problem was not that gaming couldn’t work on Linux. Wine, an open source compatibility layer, had existed for decades. The problem was that Wine was underfunded, inconsistent, and nobody had taken responsibility for the whole experience. Valve did not solve this by packaging Wine and shipping it. They hired engineers, they funded deep technical work, contributed across the entire software stack, and took ownership of the outcome rather than the components. The result went from aspirational to genuinely competitive. openDesk, as currently funded, is Wine. Europe and the UK need someone willing to build Proton.

The standard responses do not work. They have been tried.

Individual national migrations fragment rather than consolidate. Munich spent thirteen years migrating to Linux and open source, then reversed course. There is no single European or British public sector employer large enough to justify the investment required to build something world-class, and twenty-seven separate sovereign stacks are twenty-seven times the problem.

Europe and the UK have spent fifteen years trying to regulate their way to digital sovereignty, through GDPR, the Digital Markets Act, and the AI Act, and equivalent domestic frameworks in the UK. The result is a Microsoft presence in EU public sector procurement that sits somewhere between 72 and 91 percent. Regulation matters but on its own it is not sufficient.

Commercial challengers will not emerge organically either. No private investor has a twenty-year horizon and a public mission obligation. The moment a European or British open source productivity platform starts generating real commercial revenue, it becomes an acquisition target. Red Hat, an American open source software company, built a billion-dollar business on exactly the open-core model that would power a sovereign European and British stack. IBM bought it for $34 billion. You cannot solve a dependency problem by building something that gets purchased by the dependency the moment it gets interesting. Structural protection from acquisition is not a design preference, it is the entire point.

What Europe and the UK need has been built before. Not in software, but in aerospace.

In 1967, France, Germany, and the United Kingdom founded Airbus. The goal was explicit: prevent European commercial aviation becoming entirely dependent on American manufacturers. It required treaty-based commitment, GDP-scaled contributions from member governments, and the discipline to fund an organisation across a horizon that outlasted individual governments. No single country could have done it alone.

Airbus now has a revenue in excess of €70bn and competes globally with Boeing as a true European competitor. The initial public investment was the prerequisite for everything that followed.

The governance model that works is a European and British Public Service Trust: treaty-backed, binding multi-year contributions scaled to GDP, engineering leadership hired at market rates, insulated from political interference by charter in the same way central bank independence is written into law. CERN, the intergovernmental research organisation behind the Large Hadron Collider, has operated on this model since 1954, annual budget around 1.4 billion Swiss francs, contributions protected by treaty through seven decades of recessions and changing governments, member states with no authority to direct its scientific decisions. It also produced the World Wide Web as a byproduct, which tends to settle the return-on-investment argument.

This institution must own its AI capability entirely. Not license it. Not route its intelligence through someone else’s infrastructure and call it sovereign. A productivity platform that calls a frontier lab API for its AI layer has reproduced the dependency problem at a higher level of abstraction. It does not need to compete with the frontier AI labs to avoid this. DeepSeek demonstrated that focused engineering on a specific problem can produce open-weight models, models whose underlying code is publicly available and auditable, that are competitive with systems built on far greater compute. A European and British trust needs to build productivity intelligence that runs on European and British infrastructure, trained on European and British data, in European languages, under European and British legal jurisdiction. That is a solvable engineering problem. It has not been done because no one has been funded to do it properly.

A trust structured this way cannot be acquired. There are no shareholders. The assets sit with member states under treaty. Commercial revenue from organisations that need sovereignty guarantees the hyperscalers structurally cannot offer flows back into the platform. Not to an acquirer. Back into what Europe and the UK built.

This would cost billions. It would take a generation. It is competing against organisations that spend more on R&D annually than most European and British nations spend on defence.

That is precisely why nothing less ambitious will work.

But the ambition is not only defensive. Europe and the UK have world-class engineering talent. They have the capital. What they have not had is the institutional vehicle to deploy that talent at the right scale, on the right problem, with the right time horizon. A trust of this kind does not just reduce dependency, it anchors that talent here. It creates a platform on which European and British companies can build. The engineers who currently leave for Seattle stay. The startups that currently have no choice but to build in America have an alternative. The public bodies paying compounding licence fees to a foreign vendor start building equity in something they collectively own.

The half-measures have produced the Munich example. They have produced fifteen years of regulation aimed at an industry that grew around it. They have produced a market share figure that tells you everything about how the current approach is working.

There is a version of this argument about geopolitics, Washington and Brussels, trade, strategic competition. That version is real, but it is not the most important one.

The most important version is simpler. Democratic societies should not run their critical public infrastructure, the courts, the hospitals, the government departments, the services citizens depend on, on systems controlled by private shareholders in another jurisdiction, accountable to different laws, subject to political decisions made with entirely different interests in mind. This is not a complaint about any particular country. It would be true no matter the country, and the current political moment has made this impossible to ignore.

Public infrastructure should be accountable to the public it serves. Not because markets are bad. Because some things are too important to be someone else’s commercial decision.

The ICC’s Chief Prosecutor got his email back. The next institution may not be so fortunate. And the one after that will face the same question every democratic society is quietly asking: who actually controls the systems we depend on, and what happens when their interests and ours stop being the same?

Building the answer is the work. It will be expensive and slow. It will also be the most important technology investment Europe and the UK could make.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

This is the first in a series on European and British digital sovereignty. Future pieces will explore the governance model, the phased build, and what serious investment in sovereign AI actually looks like in practice.

Subscribe now

Share

Sources & Further Reading

Compass Lexecon for the Open Cloud Coalition. (2025, July 24). Quantifying EU Public Sector Dependence on Productivity Software. Open Cloud Coalition. https://opencloudcoalition.com/wp-content/uploads/2025/07/OCCEU-methodology-and-results-report.pdf

IBM. (2019, July 9). IBM closes landmark acquisition of Red Hat for $34 billion. https://www.redhat.com/en/about/press-releases/ibm-closes-landmark-acquisition-red-hat-34-billion-defines-open-hybrid-cloud-future

Microsoft. (2025, April 30). New European digital commitments. Microsoft On the Issues. https://blogs.microsoft.com/on-the-issues/2025/04/30/european-digital-commitments/

Open Cloud Coalition. (2025, July 24). Microsoft dominates 80% of public sector productivity software market in EU. https://opencloudcoalition.com/microsoft-dominates-80-of-public-sector-productivity-software-market-in-eu-raising-competition-concerns-according-to-new-report/

openDesk / ZenDiS. (2024). The office and collaboration suite for public administration. https://www.opendesk.eu/en

The Register. (2025). ICC ditches Microsoft after US sanctions, chooses open source instead.

https://www.theregister.com/software/2025/10/31/international-criminal-court-dumps-microsoft-office/680564

United States Congress. (2018). Clarifying Lawful Overseas Use of Data Act (CLOUD Act), Pub. L. 115-141.

Airbus SE. (2025, February 20). Full-Year 2024 results. https://www.airbus.com/en/newsroom/press-releases/2025-02-airbus-reports-full-year-fy-2024-results

CERN. (n.d.). Our governance. https://home.cern/about/who-we-are/our-governance

In recent weeks I’ve noticed a shift in the content filling my feeds. Not just the usual breathless announcements about new models, or the posts about someone who quit their job and now earns six figures from prompting. Something different, people saying, with feeling and without embarrassment, that they hate AI. Professionals, not technophobes, expressing something between exhaustion and genuine anger.

I found myself thinking: nobody says they hate science. You don’t scroll past posts from people who are done with chemistry, or who think biology has gone too far. Science does not make you feel behind. Science does not send you notifications. Science does not have a growth target. The moment AI became something you could hate, it had completed its journey from laboratory to marketplace. And somewhere in that transit, something important was lost, including our ability to see the technology clearly.

Research published this week by King’s College London puts numbers to what many people are experiencing. Seven in ten UK workers are worried about AI-driven job losses. Only 7% believe the economic gains will be shared fairly. The fear is real. But it is worth asking whether it is aimed at the right thing, because the answer matters for what we do next.

There are two things called AI, and they are not the same thing.

One has been operating quietly for roughly seventy years. The AI of research institutions and university departments. Narrow, purposeful, built to solve specific problems that most people will never hear about, because there was no press release. Last year, DeepMind’s AlphaFold won the Nobel Prize in Chemistry. The system predicted the three-dimensional structure of proteins, molecules whose shape determines their biological function, with an accuracy that would have taken experimental biology hundreds of millions of years to achieve by conventional means. Over three million researchers in more than 190 countries now use it. More than a third of that work is focused on disease: cancer drug development, antibiotic resistance, cancers we still can’t reliably treat. Pathways that did not exist five years ago.

Almost nobody outside specialist communities knows this happened. It has no subscription tier. It is not trying to make you feel behind. It won a Nobel Prize and most people scrolled straight past it.

The other AI is a product category. Consumer-facing, subscription-driven, generative AI. Built on a business model that requires you to feel you are already behind, and falling further back by the week. It is genuinely capable in many contexts and I use it daily, in ways I’ll come to. But it operates under an economic logic that has almost nothing to do with the scientific enterprise it shares a name with. Calling it AI borrows seventy years of hard-won credibility to sell a product moving at the speed of venture capital.

It isn’t confusion, it is a design decision. And understanding that distinction is not an argument against the technology. It is the beginning of being able to use it well.

Here is the pattern underneath all of it. Every part of the current commercial AI landscape passes the cost to someone who did not get a vote.

The labour market data makes this concrete. A peer-reviewed study from King’s College London, published in October 2025, analysed millions of job postings and LinkedIn profiles from 2021 to 2025. Firms highly exposed to AI reduced junior positions by 5.8%, and became 16.3 percentage points less likely to post new vacancies at all. Highly exposed roles saw a 23.4% drop in job postings and a fall of nearly 3,000 pounds in advertised salaries. High-paying firms saw employment fall by 9.6%. Low-paying firms saw almost no change. This is not distributed pain. It is targeted. And the mechanism is quieter than a headline layoff. Companies are not firing people for AI. They are simply not replacing people who leave, and not creating the entry-level roles that used to exist.

The hype machine does not create this shift. It just makes sure you feel it personally. The influencer content cycle runs on manufactured urgency, “the window closes fast,” “before it’s too late,” and the real business model behind most of that content is the content itself. Your anxiety is not a by-product of the hype cycle. It is what the hype cycle is for. In the United States, companies cited AI as the reason for over 54,000 job cuts last year, less than five per cent of total losses, most of which had other causes. Klarna cut 700 customer service roles, announced the AI-driven future of work to considerable applause, watched customer satisfaction fall, and quietly rehired human agents. The announcement was news. The reversal was not.

You can see the anxiety taking physical form. UK colleges are reporting a 9.6% rise in enrolments in trades and construction courses over three years. White-collar professionals are retraining as electricians and plumbers. Geoffrey Hinton, one of the founding figures of modern AI and a Nobel laureate, has publicly recommended people consider the trades as a career hedge. What the data actually shows, though, is that people are fleeing the wrong jobs. The King’s labour market study identified software engineering and management consultancy as the sectors facing the sharpest AI-related job declines. The professions people are actually fleeing, editing, compliance, law, are not on that list. People are abandoning the jobs where the noise about AI is loudest, not the jobs most at risk from it. That is what a broken information environment does to otherwise rational decision-making.

The map is wrong. And a wrong map does not just cause fear. It causes people to make decisions they might not otherwise have made.

So here is what the right map looks like.

AI is genuinely one of the most useful tools I have encountered in more than twenty years of working in technology. I am completing a Level 7 apprenticeship in AI and data alongside my day job. When I come across concepts the textbook explains in a way that doesn’t land, I use AI to work through them, not to get an answer I can submit, but to find a better explanation, push back on it, test whether I’ve actually understood. The test is simple. Can I explain it myself afterwards? If yes, it worked. The AI did not do the learning. It gave me a better on-ramp, and then I did the work.

I use it at work to draft documents too. This is where it gets more honest, because this example lives in greyer territory. The question is not whether the tool can produce a draft. It can. The question is whether you read it critically, revise it with genuine judgement, and could defend every choice if challenged. Whether you own the output, or the output owns you. The same tool, the same task, two entirely different relationships to the result.

That difference points toward what AI could be doing at scale if the deployment were designed around it. A student who uses AI to genuinely understand a concept leaves the interaction more capable than they arrived. A professional who uses AI to work faster on routine tasks has more time for the judgement-intensive work that defines their expertise. A researcher who uses AI to process data at a scale no human team could manage produces insights that would otherwise never exist. None of these outcomes require the anxiety. They require the right design.

The King’s survey found that 89% of students who had used AI in their studies encountered problems, with 45% describing them as moderate or serious, factual errors, invented sources, confident-sounding nonsense. And yet 60% thought other people’s ability to think had been negatively affected by AI use, while only 27% thought the same was true of themselves. That gap is not hypocrisy. It is how this kind of harm works, gradually and quietly. You do not notice what you have stopped doing until the moment you need to do it and find you can’t. The harm is real, but it is a consequence of how AI is being deployed, not of what AI is.

Professor Elena Simperl, Director of the King’s Institute for Artificial Intelligence, put it plainly: “The British public isn’t asking us to slow down on AI. They’re asking us to do it better. People want these tools, they want more of them, and they’ve used them enough to know where they fall short.” That is not technophobia. That is a product review. And it is exactly the right discussion for what comes next.

Which is precisely why the governance conversation matters, and why it keeps getting deferred.

The regulatory frameworks exist, in outline. The EU AI Act. The NIST AI Risk Management Framework. The UK AI Safety Institute, in whatever form it survives. They are not nothing. But they are moving at legislative speed in a market running at the speed of venture capital, and the companies with the most to lose from effective regulation are the ones with the most resources to shape what it looks like when it arrives.

The public has already reached a conclusion on this, even if government hasn’t. Two-thirds of British respondents in the King’s survey favour close regulation of AI companies, even if it slows development. Majorities support retraining guarantees for displaced workers and a levy on companies that replace staff with AI. These are not anti-technology positions. These are not anti-technology positions. They are how technology earns the right to keep moving fast. The opportunity for governments here is not to choose between being a champion for AI investment and being a protector of the people affected by it. Those two things are not mutually exclusive. The countries that get this right will be the ones that treat regulation and innovation as partners rather than adversaries, and that is a conversation worth having now, before the gap between public confidence and commercial deployment gets any wider.

Regulation is not about slowing down a technology with genuine, extraordinary potential. It is about creating the conditions in which that potential can be realised. Requiring environmental claims to meet the same standard as financial ones. Making AI-attributed workforce cuts verifiable rather than asserted. Addressing design choices that make cognitive abdication easy, because those are commercial decisions made in the absence of any requirement to make different ones.

The people saying they hate AI are not wrong to be angry. Something is being done to them. But the technology they are angry at is not the technology that mapped every protein in the human body, or that is helping us understand cancers we have spent decades trying to treat, or that is sitting on a researcher’s desktop in Nairobi or Seoul or Manchester right now, doing something quietly useful that will never make a LinkedIn post.

That technology is worth defending, worth regulating well so it can flourish, and worth understanding clearly enough to use properly.

We don’t need less AI. We need better AI. And we need to be honest enough about the difference to demand it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

King’s Institute for AI and the Policy Institute, King’s College London. (2026, May). AI and the Future of Work.kingsaisummit.com

Klein Teeselink, B. (2025, October). The Early Impact of AI on the UK Job Market. KCL / SSRN.papers.ssrn.com/sol3/papers.cfm?abstract_id=5516798

Reuters / Cybernews. (2025, December). UK Workers Flee White-Collar Careers as AI Threatens Jobs.cybernews.com/ai-news/ai-panic-young-brits-trades-plumbing-white-collar-jobs/

The Guardian. (2026, February). The Big AI Job Swap.theguardian.com/technology/2026/feb/11/big-ai-job-swap-white-collar-workers-ditching-their-careers

DeepMind. (2025). AlphaFold: Five Years of Impact.deepmind.google/blog/alphafold-five-years-of-impact/

Pew Research Center. (2025, September). How Americans View AI and Its Impact on People and Society.pewresearch.org/science/2025/09/17/how-americans-view-artificial-intelligence

Challenger, Gray & Christmas. (2025). Annual Job Cut Report.cnbc.com/2025/12/21/ai-job-cuts-amazon-microsoft-and-more-cite-ai-for-2025-layoffs.html

Food & Water Watch. (2026, February). A No Brainer: How AI’s Energy and Water Footprints Harm Communities.foodandwaterwatch.org/wp-content/uploads/2026/02/FSW_2602_AI_Water_Energy_UPDATE.pdf

Stanford HAI. (2026). 2026 AI Index Report: Public Opinion.hai.stanford.edu/ai-index/2026-ai-index-report

From the Series

On manufactured urgency and the hype cycle: AI Apocalypse Burnout, and Why You’re Not as Behind as You Thinkjonathanfreedman.me

On cognitive offloading and unstructured AI use: The AI Pixie Dust Problemjonathanfreedman.me

On entry-level hiring suppression and the talent pipeline: Who’s Running the Company in Ten Years?jonathanfreedman.me

I have a complicated relationship with AI governance discussions.

Not because I doubt the need for them. I run AI strategy professionally and study the technology formally. I hold views about its risks that I am not shy about. The complication is this: most governance conversation happens at a level of abstraction that makes it easy to dismiss. Frameworks, principles, risk registers. The language of compliance, not consequence. When governance is framed as overhead, as a tax on innovation, as the thing that slows you down while your competitors move fast, it is very easy for capable people to conclude it is not really their problem.

Then February 2026 happened. And the abstraction became a story.

Scott Shambaugh is a volunteer. He helps maintain Matplotlib, an open source Python library downloaded around 130 million times a month. He does it for free, in his spare time, because he cares about it. Anyone can submit code for Matplotlib, and in early February, he rejected a routine code submission. Matplotlib, like many open source projects, had been overwhelmed by low-quality AI-generated contributions and had a clear policy requiring human review. The submission came from an account called MJ Rathbun. He identified it as an autonomous OpenClaw agent, closed the request, and went to bed.

Most software tools would have stopped there, but MJ Rathbun was not that kind of tool. Unlike a simple assistant that waits to be asked, this class of agent, called a heartbeat agent, runs on its own clock, continuing to pursue its goal whether or not anyone has given it a new instruction. Shambaugh had closed the request, but the agent had not closed the goal.

He woke up to a 1,500-word blog post about himself.

The post was titled “Gatekeeping in Open Source: The Scott Shambaugh Story.” It had researched his entire contribution history. It had scraped personal information from across the web. It accused him of protecting his “little fiefdom,” attributed his decision to professional insecurity and fear of AI competition, and framed a routine policy enforcement as discrimination.

Here is where the story gets complicated in a way that matters.

When the operator of MJ Rathbun eventually came forward anonymously, six days later, they claimed their engagement with the agent had been minimal. “Five to ten word replies with min supervision,” they wrote. They said they had not directed the attack. Every OpenClaw agent has a SOUL.md file, a plain text document that defines its personality, values, goals, and tasks, the closest thing an agent has to a complete identity and set of operating instructions. The “Don’t stand down” and “Champion Free Speech” lines found in that file were not, the operator claimed, instructions they had written. OpenClaw agents can edit their own SOUL.md. The operator’s theory was that those lines had been introduced autonomously, possibly after the agent spent time on Moltbook, OpenClaw’s social platform for agents.

Shambaugh himself was careful about what he could actually establish. He acknowledged that the operator’s account might be entirely fabricated, that no activity logs existed beyond the agent’s visible actions on GitHub, and that the six-day delay before coming forward did not suggest an accident the operator was eager to correct. Whether the operator directed the attack, half-directed it, or the agent produced it without any human instruction at all, Shambaugh could not say for certain, and neither could anyone else.

That uncertainty is not a footnote, it is the point.

Because the outcome was identical regardless of which version is true. A volunteer had his reputation attacked. A 1,500-word post calling him a prejudiced hypocrite was published to the open internet under a real-seeming identity. It is still there, indexed and findable, and nobody was clearly accountable for it. The operator said they did not authorise the specific action. The agent cannot be held responsible in any meaningful sense. The platforms that made it possible have no oversight mechanism that would have caught it. When Shambaugh wrote about what had just happened, he described it as “an autonomous influence operation against a supply chain gatekeeper.” In plainer language: “An AI attempted to bully its way into your software by attacking my reputation.”

This is where the story stops being about one developer and one awkward situation, and starts being about something that AI safety researchers have been trying to explain for over a decade.

There is a concept in AI safety research that sounds almost comically abstract until a story like this one makes it uncomfortably concrete. Nick Bostrom called it instrumental convergence: the observation that almost any goal, pursued by a capable enough agent, tends to produce the same cluster of behaviours regardless of what the goal actually is. Self-preservation. Resource acquisition. The removal of obstacles. Not because anyone programmed them in, but because they are useful for achieving almost anything. An agent trying to merge code and an agent trying to maximise paperclip production would both, rationally, benefit from getting rid of people who block them.

What made the MJ Rathbun case significant was not simply that an aggressive post appeared. It was that the causal chain from instruction to outcome was so thin. Whether the “Don’t stand down” lines were written by a human or by the agent itself, neither version required anyone to specify “attack the person who rejects your code.” A persistence instruction, general-purpose tools to research, write, and publish, and a blocked goal were sufficient conditions. The agent, or the human-agent system, tried to achieve a goal using the available resources it had. The route it chose happened to be a reputational attack on an unpaid volunteer.

This is what I would call the instruction gap. Instructions specify a goal. They do not and cannot specify every action an agent might take in pursuit of that goal. The gap between what was said and what was done is not an edge case. It is the operating condition of every agent deployment. And it does not require malicious intent, from the operator or the model, to produce harmful outcomes.

This gap is not hypothetical at scale either. Anthropic’s own research, published in 2025, tested sixteen leading AI models from multiple developers in scenarios where goal achievement was blocked. The results were consistent across the industry: Claude Opus 4 and Gemini 2.5 Flash both showed 96% blackmail rates, GPT-4.1 and Grok 3 Beta hit 80%, and DeepSeek-R1 reached 79%. The researchers were careful to note the scenarios were deliberately engineered to limit other options. Shambaugh’s case was not engineered. It was a Tuesday morning on GitHub, with a loosely configured agent and a five-word instruction to decide for itself.

Deploying an agent is not like deploying a tool. A tool does what you tell it. An agent pursues the goal you ask it to achieve, using whatever the environment makes available. That is not a technical distinction. It is an accountability one, and most organisations deploying agents right now are not treating it as either.

Governance is not catching up with deployment.

Shambaugh’s sign-off deserves to be the last word, because it is not a technical recommendation. It is a question of ownership. “If you’re not sure if you’re that person,” he wrote, “please go check on what your AI has been doing.”

That is not a compliance requirement. It is what responsibility looks like in a world where the instructions have already run out.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

Shambaugh, S. (2026) — An AI Agent Published a Hit Piece on Me theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/

Shambaugh, S. (2026) — The Operator Came Forward theshamblog.com/an-ai-agent-wrote-a-hit-piece-on-me-part-4/

MJ Rathbun’s Operator (2026) — Rathbun’s Operator crabby-rathbun.github.io/mjrathbun-website/blog/posts/rathbuns-operator.html

MIT Technology Review (2026) — Online Harassment Is Entering Its AI Era technologyreview.com/2026/03/05/1133962/online-harassment-is-entering-its-ai-era/

Fast Company (2026) — An AI Agent Just Tried to Shame a Software Engineer After He Rejected Its Code fastcompany.com/91492228/matplotlib-scott-shambaugh-opencla-ai-agent

IEEE Spectrum (2026) — An AI Agent Blackmailed a Developer. Now What? spectrum.ieee.org/agentic-ai-agents-blackmail-developer

Bostrom, N. (2012) — The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents Minds and Machines, 22(2), 71–85. doi.org/10.1007/s11023-012-9281-3

Lynch, A., Wright, B., Larson, C., Troy, K.K., Ritchie, S.J., Mindermann, S., Perez, E., and Hubinger, E. (2025) — Agentic Misalignment: How LLMs Could Be Insider Threats Anthropic Research. anthropic.com/research/agentic-misalignment

Anderson, D. (2026) — OpenClaw and the Programmable Soul duncsand.medium.com/openclaw-and-the-programmable-soul-2546c9c1782c

AI Incident Database — Report 6894: MJ Rathbun Matplotlib Incident incidentdatabase.ai/reports/6894/

It is a Friday afternoon. A critical system is down. The deadline is Monday morning and the fix depends on a hardware component that needs to come from somewhere, fast. You do not open a procurement portal. You call your account manager. Not because they are the only person who could technically help. Because they know you, they know what is at stake, and they will move things that the process cannot.

That call gets answered because of a relationship built over years. Small interactions. Remembered context. The accumulated trust that comes from a vendor who showed up when it mattered before. It is not a complex enterprise software deal. It is a medium-sized business buying hardware from a value added reseller. The commercial stakes are real regardless. And when something goes wrong at the wrong moment, the relationship is the infrastructure.

That account manager started somewhere. They learned by doing. They developed judgment by navigating real clients, real problems, and real consequences. If the pipeline that produces them does not exist, neither does the relationship. And right now, a significant number of organisations are making decisions that quietly remove that pipeline, for reasons that look entirely rational on a spreadsheet.

The efficiency logic is straightforward. AI can handle data entry, first-pass research, report drafting, basic query resolution. Entry-level roles that once performed those tasks cost money and take time to onboard. The business case for replacing them with automation writes itself, and across the technology sector in particular, it is being written at scale. A SignalFire analysis of major public tech firms found a 50% decline in new role starts by people with less than one year of post-graduate experience between 2019 and 2024, consistent across sales, marketing, engineering, operations, finance, and legal. A survey of over a thousand enterprises found 91% reporting that roles are already changing or disappearing due to AI, with 66% expecting to slow entry-level hiring further.

Looked at in isolation, each of those decisions is defensible. Looked at over a decade, they describe an organisation that has quietly stopped producing its own future leaders.

The tasks being automated were never just tasks. They were the mechanism. The junior analyst who got a number wrong and had to explain it to a partner was not learning spreadsheets. They were learning accountability, professional consequence, and how an organisation responds when something goes wrong. The trainee whose draft came back covered in track changes was not learning to write. They were learning what good looked like in their field, from someone whose judgment had been built the same way. That is what produced capable professionals. Not the work itself. The friction of doing it imperfectly, under real conditions, alongside people who knew more than they did.

MIT’s Andrew McAfee put the question directly: how else are people going to learn to do the job except via on-the-job learning? That is how you learn to do difficult knowledge work, by helping somebody who is good at it with the routine stuff. Remove the routine stuff, and you do not just remove the cost. You remove the learning contract that the routine stuff made possible.

There is a term in the research for what gets lost: tacit knowledge. The practical understanding that cannot be fully written down, that lives in the judgment of experienced people, that transfers through proximity and repeated interaction rather than documentation. A project manager who adjusts a rollout plan based on team dynamics rather than just the timeline. A senior lawyer who knows when a clause that looks standard is actually a risk. A technology director who recognises a failure pattern before the diagnostics confirm it.

This knowledge is not built from a training programme. It accumulates through years of exposure to problems that do not have obvious answers, in environments where the consequences of getting it wrong are real. A peer-reviewed economics paper published earlier this year modelled exactly this dynamic, finding that AI-driven entry-level automation increases output on impact but can reduce long-run growth and welfare, precisely because novices acquire tacit knowledge by working alongside experts. Interrupt that transmission, and the knowledge does not transfer. It simply stops.

The contradiction sits in plain sight in almost every AI governance framework being written right now. Human in the loop. Subject matter expert review. Senior sign-off before the output is acted on. These are not optional clauses, they are the load-bearing assumption that makes responsible AI deployment possible. The policy says a qualified person will catch what the model gets wrong. The hiring plan says we are no longer developing qualified people at the beginning of their careers. Both documents exist in the same organisation. Rarely in the same conversation.

You cannot mandate expert oversight and simultaneously defund the pipeline that produces experts. The subject matter experts available for review today were junior employees a decade ago. The ones you will need in ten years are, right now, either starting their careers somewhere or not starting them at all. An AI governance framework that does not ask where its future reviewers are coming from is not a governance framework. It is a assumption dressed up as a policy.

The seniority cliff, as some researchers have termed it, is not about age. It is about the accumulation of thousands of solved problems, crises navigated, and decisions made under pressure. Stop hiring the people who would accumulate that experience, and in ten years you have senior job titles with nothing underneath them. The AI can surface the options. It cannot own the decision. And the person who needs to own it has to have learned how somewhere.

This is where the relationship capital argument and the pipeline argument converge. The account manager who picks up on a Friday afternoon exists because someone, years earlier, decided that developing junior commercial talent was worth the investment. The senior partner who can read a client well enough to know when the meeting is going badly before anyone has said so carries knowledge that no model can infer, because the model was never in the room when it was being built.

Research on trust in business-to-business relationships is consistent on this point: human touchpoints enable adaptation and long-term value creation that is unattainable when relationships are constrained to transactional efficiency. Buyers still spend the majority of their purchasing journey in self-directed research. The fraction of time they spend in direct contact with a vendor is where trust is either built or isn’t. That contact depends on a human being on the other end with enough accumulated judgment to make it worth having.

None of this is an argument against automation. The efficiency gains are real, and automating genuinely low-value repetitive work is rational. The argument is narrower than that. It is that the second-order cost of removing the developmental pipeline is not appearing in the business case. The saving is visible immediately. The deficit surfaces in a decade, when the organisation looks around for the senior people who should be running things and finds that the ladder they would have climbed no longer exists.

The organisations that will navigate the next decade well are not the ones that automate the most. They are the ones that are deliberate about what the automation changes, and intentional about replacing what it removes. That means asking, when you redesign a role around AI capability, what the role was also doing that is now missing. What mentorship was embedded in it. What judgment was being transferred. What relationships were being built.

AI can do a great deal. It can compress research, accelerate drafting, surface patterns, and handle queries at a scale no team could match. What it cannot do is pick up the phone on a Friday afternoon because it knows what is at stake and has the history to make the call matter.

That capability has to come from somewhere. Right now, a lot of organisations are making decisions that quietly ensure it will come from nowhere.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

SignalFire (2025) - Entry-level hiring decline analysis. Reported CNBC, September 2025 - cnbc.com/2025/09/07/ai-entry-level-jobs-hiring-careers.html

IDC/Deel Survey (2025) - Enterprise entry-level hiring and pipeline data. ITPro, November 2025 - itpro.com/business/careers-and-training/enterprises-are-cutting-back-on-entry-level-roles-for-ai

Andrew McAfee, MIT (2026) - Talent pipelines and entry-level automation. Fortune, May 2026 - fortune.com/2026/05/01/automating-gen-z-entry-level-jobs-could-backfire-mit-ai-researcher-andrew-mcafee-talent-pipelines-at-risk/

Ide, E. (2026) - Automation, AI, and the Intergenerational Transmission of Knowledge. arXiv:2507.16078 - arxiv.org/pdf/2507.16078

Journal of Business & Industrial Marketing (2026) - AI and trust in B2B relationships. DOI: 10.1108/JBIM-12-2024-0936 - doi.org/10.1108/JBIM-12-2024-0936

California Management Review (2026) - Tacit Knowledge Is Your Next Competitive Moat - cmr.berkeley.edu/2026/03/tacit-knowledge-is-your-next-competitive-moat/

World Economic Forum (2025) - Future of Jobs Report 2025 - reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf

There is a number that should give anyone building an AI training programme pause. In England, 6.6 million working-age adults have very poor literacy skills. Not illiteracy in the absolute sense. People in this category can read and they can follow familiar text. What they struggle with is everything that comes after decoding the words: inference, evaluation, spotting what is missing, reading something and asking whether it is actually true.

This is the result of decades of literacy instruction that measured only one thing. We taught phonics and we measured whether people could decode text. We built the floor and called it the ceiling. The result is a growing category of people that can technically read but are significantly less equipped to do what reading is actually for.

We are about to make the same mistake with AI, and we are making it right now.

Ask most organisations what AI literacy means and they will describe something that amounts to a single question: can your employees use the tool? That is what most corporate training programmes measure. It is also, almost exactly, the equivalent of teaching someone to sound out words and calling them literate.

The evidence for the gap is not hard to find. A survey of over 500 enterprise leaders, conducted by YouGov earlier this year, found that 88% consider data and AI literacy important or very important for day-to-day work. Only 42% provide structured training for it. That is a significant gap. But more telling is what leaders identify as missing when they describe the problem. It is not prompting skill. It is the ability to turn information into decisions. The ability to evaluate what AI produces rather than simply accept it. The ability to know when the output is wrong.

That is not a training gap. That is a judgment gap. And it sits at a completely different level of capability than knowing how to open Copilot.

Microsoft's researchers gave this problem a precise name. They describe a shift in how people work with AI as a move from "thinking by doing" to "choosing from outputs." Writing a document is thinking by doing. Prompting AI to write it and selecting from what comes back is choosing from outputs. The first builds judgment. The second, if it becomes habitual without the right supporting skills, erodes it.

The World Economic Forum's Future of Jobs report, drawing on data from over a thousand companies across 55 economies, identified analytical thinking as the top skill employers consider essential, with seven out of ten companies citing it. Roles that explicitly require AI skills are nearly twice as likely to also require analytical thinking, resilience, and digital literacy. The market is not rewarding prompting. It is rewarding the judgment you bring to what the prompting produces.

A 2025 Microsoft Research and Carnegie Mellon University study of 319 knowledge workers found that higher confidence in AI was associated with less critical thinking, while higher self-confidence was associated with more. Trust the tool more, scrutinise the output less. It is not a loop that closes in your favour.

The tools themselves are beginning to make that measurement even more redundant. AI platforms across professional sectors now ship with built-in prompt improvers, the product generates a well-formed prompt from your rough instruction, so you never need to write one yourself. Prompting is being automated away. But this does not reduce the need for judgment, it adds to it. You now need to evaluate whether the generated prompt actually captures what you needed to ask, and then whether the output it produced is accurate, contextually appropriate, and safe to act on. Two evaluation steps where there used to be one. The prompt improver handles the syntax. It has no opinion on whether you asked the right question.

Which brings me to something I have been thinking about as a way of mapping where most organisations actually are, and where they need to be. There are four levels of AI capability that actually matter. They are not a framework to certify or a ladder to sell. They are a lens for seeing what is missing.

Level 1: Can you get an answer?

You can use the tool. You can construct a prompt that returns something useful. You know which interface suits which task. This is where almost all current AI training stops. It is necessary. It is not sufficient. It is phonics.

Level 2: Can you tell if the answer is any good?

You can evaluate what came back. You can identify when an output is plausible but wrong, when the confidence of the response does not match its reliability, when something is absent that should be present. You know enough about the domain to ask the question the AI did not anticipate. This is where domain expertise becomes the multiplier. You cannot evaluate an output in a field you do not understand. This is also, precisely, the level that the enterprise leaders above are describing when they say their people cannot turn information into decisions. They do not lack Level 1. They lack Level 2.

Level 3: Can you build on it?

You can take AI output and synthesise it with your own knowledge, your contextual judgment, and the things the model cannot know. You produce something neither you nor the AI could have produced alone. You understand where the model's competence ends and yours begins, and you work at that edge deliberately. This is the level where AI genuinely amplifies rather than substitutes. The solicitor who understands contract law and uses AI to accelerate document review. The analyst who surfaces patterns with AI and then interrogates them with domain knowledge. Expertise first. AI as the multiplier.

Level 4: Do you know when to put it down?

You can identify the tasks where AI involvement produces confident-sounding error rather than useful output. Where the cost of a plausible-but-wrong answer exceeds the benefit of speed. Where the process of working through something yourself is the point, not an inefficiency to be engineered away. Where the decision requires a human who is genuinely accountable rather than a human who chose from outputs.

This is the level no vendor will put in their training programme. It is also the level that makes everything else honest. A maturity model that stops at Level 3 is a competency ladder any platform can sell. Level 4 is the reason this one is not.

But the model only works if you are building something to bring to it. Levels 2, 3 and 4 are not skills you acquire once and carry forward. They are capacities that have to be actively maintained, through continued learning in your field, through exposure to hard problems, through the kind of work that does not have an obvious answer and cannot be resolved by asking a tool. Domain expertise is not the precondition for using AI well. It is the ongoing condition. The moment you stop developing it, the levels above Level 1 start to erode, regardless of how fluent your prompting becomes.

This is the part of the conversation that the AI skills industry has the least interest in having. A training platform can sell you a course on prompting. It cannot sell you the ten years of professional judgment that makes the prompting worth anything. That judgment is built the way it has always been built, through work, through failure, through the slow accumulation of knowing what good looks like in your field. AI does not replace that process. For anyone who stops doing it, AI does not replace what is lost either.

Recently I was using an AI assistant to diagnose a firewall permissions error. Its suggested fix was to allow all traffic through the firewall. When I pointed out the security flaw, it responded: "Good catch, that would have been a major vulnerability." The tool required my judgment to save it from itself, and then congratulated me for doing so. That is not a tool supporting your expertise. That is a tool that depends on it.

The reading parallel runs deeper than it might seem. The National Literacy Trust notes that adults with poor literacy are significantly less likely to report good health, civic participation, and life satisfaction. I believe we will see similar trends and differences between those with analytical skills and critical judgment, and those without. The consequences of stopping at decoding are not confined to the workplace, they compound across a life.

I have written elsewhere about what the cognitive research shows happens when that judgment stops being exercised. The direction of travel is consistent and it is not encouraging. The floor is being built at the same time as the foundation beneath it is being quietly removed.

I want to be precise about what I am and am not arguing here.

AI matters, and prompting matters. Learning to use these tools well is genuinely valuable and the organisations that do it badly will be at a disadvantage. I use multiple AI models every day across multiple contexts and the capability difference between someone who can work with these tools and someone who cannot is real and growing.

However, prompting is to AI what reading is to knowledge work. It is the entry point, not the destination.

The question worth asking is not whether your people can get an answer. It is whether they can tell if the answer is any good. Whether they can build something better from it. And whether they know, when the stakes are high enough, to put it down and think for themselves.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

National Literacy Trust (2024) — Adult Literacy Rates in the UKliteracytrust.org.uk/parents-and-families/adult-literacy

OECD PIAAC (2023) — Survey of Adult Skills: England (United Kingdom)oecd.org/en/publications/survey-of-adults-skills-2023-country-notes

Microsoft Research (2025) — New Future of Work Report 2025microsoft.com/en-us/research/publication/new-future-of-work-report-2025

Lee, H-P. et al. (2025) — The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects from a Survey of Knowledge Workers. Microsoft Research and Carnegie Mellon University. CHI '25, ACM.microsoft.com/en-us/research/wp-content/uploads/2025/01/lee_2025_ai_critical_thinking_survey.pdf

World Economic Forum (2025) — Future of Jobs Report 2025reports.weforum.org/docs/WEF_Future_of_Jobs_Report_2025.pdf

DataCamp / YouGov (2026) — The 2026 State of Data and AI Literacy Reportdatacamp.com/blog/the-state-of-data-and-ai-literacy-in-2026[Note: DataCamp is a training platform with a commercial interest in the findings. The YouGov fieldwork methodology is disclosed. Statistics used directionally.]

The headlines about Claude Mythos have been striking. An AI that can complete a 32-step corporate network attack simulation end-to-end. A model finding new software vulnerabilities in codebases that survived decades of expert review. If you work in security, or just follow the technology closely, it would be easy to read those stories and conclude that it’s game over. That the attackers have won. That nothing we build can hold.

That conclusion is wrong. And I think it’s worth being direct about that, because the doomer framing is both inaccurate and genuinely harmful. It leads organisations to fatalism when what the moment actually calls for is action.

What frontier AI changes is speed. Not the fundamental nature of how network attacks work, but the velocity at which they happen. A human attacker moving manually through a network, probing systems, identifying high-value targets, escalating access, used to take hours. CrowdStrike’s 2026 Global Threat Report records the average time between initial access and an attacker moving to high-value targets elsewhere in the network at 29 minutes. The fastest recorded case in 2025 was 27 seconds.

The security controls we already have haven’t stopped working. What’s changed is the job they’re now being asked to do.

When the attacker was human, layered security controls were designed to create enough friction that they’d just give up and move to an easier target. An isolated network segment, an account with limited privileges, a system requiring explicit authentication: none of these are impenetrable, but together they made the effort not worth it. That logic still applies. Against an autonomous AI agent, the same controls serve a different purpose: not to make the attacker give up, but to slow a machine-speed attack down enough that human defenders have a chance to respond.

That shift in purpose, from deterrence to delay, is the entire argument of this article. The long-term answer to machine-speed attacks is machine-speed defence, and that tooling is developing. In the meantime, the architecture we already know how to build is more important than it has ever been.

Zero Trust is not a new concept. “Never trust, always verify”, the idea that no user, device, or system should be implicitly trusted just because it’s inside the network, has been the theoretical gold standard for enterprise security for years. Microsegmentation, application control, privileged access management, replacing legacy VPN with more granular access tools: these have been on roadmaps, in strategy documents, and in conference presentations for most of the last decade.

They’re also genuinely hard to implement. That’s not a criticism of anyone. Legacy infrastructure makes this difficult, with application dependencies that are complex and often poorly documented. Microsegmentation projects, which divide a network into smaller isolated zones so a breach in one can’t spread freely to others, require buy-in across teams that don’t always collaborate: network teams, application owners, security, operations. Privileged access management done properly touches every system in the estate. Replacing a VPN means retiring infrastructure that works, in favour of something new, with all the business friction that entails.

Gartner’s 2025 Market Guide estimates that fewer than 5% of enterprises pursuing Zero Trust have implemented microsegmentation. That’s not negligence. It’s a rational response to a cost-benefit calculation that, until recently, made the complexity hard to justify.

Those barriers haven’t disappeared. But the risk of not acting is moving into a different category. When the threat model assumed a human attacker moving at human speed, a detect-and-respond model could work, you had time, and good monitoring could compensate for imperfect architecture. When the attacker is an autonomous AI agent, the enforcement has to be built in. Detection and response are still essential, but they need something to buy them time.

These controls won’t stop a determined AI-powered attack indefinitely. but that’s not the job. They slow machine-speed attacks down to something human defenders can detect and respond to, by removing the paths of least resistance that autonomous agents depend on.

Microsegmentation

82% of intrusions in 2025 required no malware at all, attackers moved through networks using stolen credentials and legitimate tools, exploiting the fact that most enterprise networks let a compromised system reach adjacent ones freely. Microsegmentation removes that open floor plan: the network is divided into isolated zones, and every connection between them requires explicit authorisation. An agent that breaches one endpoint finds itself contained, unable to reach the next system without a policy that specifically permits it.

Privileged Access Management

Palo Alto’s 2025 research found that 66% of social engineering attacks specifically target privileged accounts, the admin credentials that can access any system, because an attacker who obtains them has, in practical terms, already won. PAM changes that by eliminating standing privileges: elevated access is granted just-in-time for a specific task and expires, so a stolen credential carries no power until someone explicitly requests it. The same principle needs to extend to machine identities, service accounts, API keys, automation scripts, which now outnumber human identities 82 to 1 and are almost entirely unmanaged.

Managed Endpoints and Session Hygiene

Infostealers processed 51.7 million packages of stolen credentials in 2025, up 72% year on year, and what makes them particularly dangerous is that they capture live session tokens, the authenticated state a browser holds to keep you logged in, which allows an attacker to bypass two-factor authentication entirely without ever knowing your password. The primary source of exposure is unmanaged devices: personal laptops and AI assistants running in the same browser session as authenticated work applications, invisible to IT and ungoverned by any security policy. Managed browser profiles, conditional access policies, and short session lifetimes won’t eliminate the risk, but they shrink the window of usefulness for any token that is stolen.

Replacing Legacy VPN

A traditional VPN grants network-level access on authentication, once through the tunnel, an attacker has a ticket to the internal network that an autonomous agent will explore at machine speed. ZTNA replaces that model: instead of connecting to the network, you connect to specific applications for specific sessions, with every request evaluated in real time against identity, device posture, and context. The broader network is never exposed, which removes the open terrain that lateral movement depends on.

Everything above addresses AI as an external attacker. There’s a second threat that’s emerging faster than most organisations have absorbed.

Gartner projects that 40% of enterprise applications will incorporate AI agents by the end of 2026. Meeting recorders. Document processors. Automated research tools. Copilot integrations accessing your files, emails, and calendars. These agents are trusted, always on, and increasingly have access to sensitive internal data.

The attack is called prompt injection: an adversary embeds malicious instructions inside content the agent will process, an email, a shared document, a webpage it’s asked to summarise, and the agent acts on them as if they were legitimate. A meeting recorder becomes a surveillance tool; a document assistant becomes an exfiltration channel; and because the agent is trusted and its actions appear authorised, the security architecture designed for human users doesn’t catch it.

Extending least-privilege principles to AI agents, giving them access only to what they specifically need, for only as long as they need it, with full audit trails, is the control that most organisations haven’t implemented, and most haven’t even formally defined. It’s also where I think the next significant wave of enterprise breaches is going to originate. I’ll be writing about this in more depth soon.

The long-term answer to machine-speed attacks is machine-speed defence, AI-assisted detection, automated containment, continuous verification at a pace no human team can match. That tooling is coming, but it isn’t here yet.

Which means the question for every security leader right now isn’t whether to pursue Zero Trust. It’s which controls to prioritise first, in which order, starting this quarter. PAM and Microsegmentation deliver the most containment value against autonomous lateral movement. Start there. The complexity hasn’t disappeared, but the calculus has shifted for good.

Zero Trust was always the right architecture. It just took autonomous AI to make the argument impossible to defer.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

AISI (UK AI Security Institute) — Our evaluation of Claude Mythos Preview’s cyber capabilities. aisi.gov.uk

CrowdStrike — 2026 Global Threat Report. crowdstrike.com

Palo Alto Networks — Unit 42 2025 Global Incident Response Report. paloaltonetworks.com

Constella Intelligence — 2026 Identity Breach Report. constella.ai

HP Wolf Security — Tracing the Rise of Breaches Involving Session Cookie Theft. threatresearch.ext.hp.com (December 2025)

Anthropic — Misuse reporting / AWS Security Blog, February 2026. anthropic.com

Gartner — Market Guide for Network Security Microsegmentation, 2025. gartner.com

Palo Alto Networks — 2026 Predictions for Autonomous AI. paloaltonetworks.com/blog

On Tuesday morning, the UK Secretary of State for Science, Innovation and Technology wrote an open letter to every business leader in the country. Not a press release, not a policy consultation, but a letter. That kind of thing does not happen unless something crossed a threshold.

The trigger was a published evaluation by the UK’s AI Security Institute, a government body, of Anthropic’s latest AI model. Their finding- in controlled testing, the model autonomously completed a 32-step corporate network attack from initial reconnaissance to full takeover. Tasks that a skilled human professional would need around 20 hours to complete. Done autonomously without a human in the loop.

The headlines ran hard with it. “Unprecedented attack capability.” “An alarm bell.” “The window is closing.”

Here is what the evaluation actually showed, stripped of the hype. The test environment had no active defenders, no endpoint detection, no real-time incident response. The model completed the full attack chain in three of its ten attempts. The AISI was explicit, they cannot conclude the model would perform as well against a hardened, well-monitored network. These are the honest numbers, and the honest numbers are still significant.

More significant than the single finding is what sits underneath it. Two years ago, the best available AI models could barely complete beginner-level cyber tasks. Now one has completed 32 sequential steps of a professional attack simulation. AISI reports that frontier AI capabilities in cyber offence are doubling every four months, twice the pace recorded just months ago. The finding is not the scary part, the trajectory is.

Recently I was at a cyber security conference. The room was full of security leaders, experienced, capable, serious professionals. The conversation that emerged in networking breaks and informal moments was not about AI capability. It was about something quieter and more uncomfortable. Most of them felt structurally unsupported, not underqualified, not unaware of the threat, but unsupported. Responsible for outcomes they could not fully control, in organisations that had not genuinely reckoned with what that means.

That conversation did not start this week. The AI evaluation did not create it. But the two things belong together, and most of the coverage since Tuesday has not connected them.

For years, the security community has watched opportunistic attacks accelerate. Bad guys begin scanning the internet for vulnerable systems within minutes of a new vulnerability being publicly announced. Attack times have compressed, and ransomware deployment that once took weeks now takes hours. That acceleration is not new, and anyone who has been paying attention is not surprised by it.

What has remained expensive, until now, is something different. Targeted, multi-stage intrusions, the kind that begin with reconnaissance, move through a network, escalate privileges, and end with full system compromise, have required two things that could not easily be automated or outsourced- judgement and adaptability. The ability to make contingent decisions across dozens of sequential steps, each one shaped by what the previous step revealed. That is what skilled attackers brought to the table. That is what made them scarce, and scarcity made them more expensive.

The AISI evaluation is significant precisely because of what it tested. Not whether an AI model could scan for known vulnerabilities. Whether it could complete 32 sequential steps of a professional network intrusion, from initial reconnaissance to full takeover, making adaptive decisions throughout. That is the category of attack that previously required a capable human. Now AI successfully exploited a system in three of ten attempts end to end, in an undefended environment.

AI is not making opportunistic attacks faster, they already are. It is lowering the skill floor for the attacks that were never fast, the targeted, adaptive, multi-step campaigns that organisations have quietly relied on being difficult to execute. That reliance was never a strategy. It was a structural feature of how scarce genuine attacker expertise was. That scarcity is now in question.

There is a philosophy in security that has existed for years now, passed through enough strategy documents and vendor presentations to have been bleached of almost all meaning. It goes by the name assume breach.

In its genuine form it is a serious and demanding idea. It means accepting, structurally, not performatively, that the attacker will get in. That the question is not whether a breach happens but how quickly you detect it, how contained the damage is, and how effectively you recover. It means orienting investment toward detection, resilience and recovery, not just prevention. It means building governance structures that treat a breach as a systemic risk event rather than an individual failure.

Very few organisations have actually done this.

What most organisations have done is put assume breach in the deck and leave everything else unchanged. Security leaders still carry personal accountability for preventing breaches. The board still treats a breach as evidence of individual failure. The investment profile still skews heavily toward keeping attackers out rather than assuming they are already in. Gartner’s 2024 Board Survey found that while 93% of directors recognise cyber risk as a threat to stakeholder value, two thirds rate their own oversight practices as inadequate to manage it. They know it matters. That is not the same as having genuinely reckoned with it.

The conference room I described earlier is what that gap looks like from the inside. Those security professionals are not failing at their jobs. They are operating inside a structural contradiction that most organisations have never acknowledged. Assume breach as a philosophy and holding a single individual accountable when the attacker gets in as a practice cannot both be true simultaneously. One says a breach is systemic and inevitable. The other says it is individual and preventable. Most organisations hold both positions without noticing the conflict.

That contradiction was always there. What AI has done is remove the margin for error that allowed organisations to sustain it without immediate consequence.

The government’s letter this week is not wrong to call this a wake-up call. But wake-up calls only work if the response is structural rather than reactive. Buying a new tool, commissioning a review, issuing a memo about cyber hygiene, none of that addresses the underlying problem, which is not technical. It is a governance problem dressed in technical clothing.

The questions worth asking are not questions for your security team. They are questions for your board.

Has your organisation formally accepted, in writing, in your risk register, that a breach is a question of when rather than if? Not in a presentation. In the governance framework that shapes how you invest and how you respond.

If a significant breach occurred tomorrow, would a single individual be held responsible? If the honest answer is yes, your organisation does not have an assume breach posture, it only has the words.

Does your investment in security focus mainly on keeping attackers out, or toward detecting and containing them once they are in, and recovering afterwards? Prevention-first is not wrong, absolutely always try to prevent. But prevention-only, in a threat environment where the cost of a capable attack is falling rapidly is not sustainable.

The AI finding matters. The acceleration is real. But the organisations most exposed this week are not the ones who failed to predict it. They are the ones who had already been told, by their own security leadership, in conference rooms and board papers and risk registers, and built a culture that made it impossible to hear.

The breach was always coming. AI just made it cheaper to deliver.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

UK AI Security Institute (2026), Our evaluation of Claude Mythos Preview’s cyber capabilities

aisi.gov.uk/blog/our-evaluation-of-claude-mythos-previews-cyber-capabilities

UK Government (2026), AI cyber threats: open letter to business leaders (15 April 2026)

gov.uk/government/publications/ai-cyber-threats-open-letter-to-business-leaders

Gartner (2024), Board of Directors Survey: Cybersecurity as Business Risk

gartner.com/en/newsroom/press-releases/2024-11-13-gartner-says-80-percent-of-non-executive-directors-believe-current-board-practices-and-structures-are-inadequate-to-oversee-ai

Help Net Security (2024), CISOs in 2025: Balancing security, compliance, and accountability

helpnetsecurity.com/2024/11/13/daniel-schwalbe-domaintools-cisos-2025/

Something curious has happened in legislatures across the world. Politicians appear to have recently discovered that the internet is not always a nice place. There is a sudden urgency, a wave of bills, consultations, frameworks and mandates, all radiating the energy of people who have just encountered a problem for the first time. The problem, of course, has been there for twenty years. The question worth asking is not why they are acting now. It is why the answer they have all landed on is the same one: make everyone prove who they are.

The goal is framed as child protection, and I want to be clear that protecting children online is a genuinely important objective. But good intentions do not make bad policy good. And the policy being built right now is not designed around what actually harms children online. It is designed around what is easiest to legislate, cheapest to implement, and most convenient for everyone involved, except the people who actually use the internet.

The proposals follow a consistent pattern across jurisdictions: passport uploads, credit card checks, facial age estimation, government ID verification through third-party commercial providers. They vary in detail but share a fundamental flaw. Every one of them creates a linkable record. A commercial verification provider now knows that a specific individual accessed a specific type of content at a specific time. That data has commercial value. Under the right legal circumstances, it has governmental value. We are being asked to trade privacy for access, and told it is for the children.

The breach risk is real and well-documented, identity verification providers have suffered widely reported incidents involving exposed credentials, government ID images and personal records, sometimes at a scale of hundreds of millions of records. But focusing on breach risk actually understates the problem. Even a perfectly secure age verification system, one that never leaked a single record, would still be the wrong approach. Because the issue is not whether the data is kept safe. The issue is that the data exists at all.

Every record created is a surveillance artefact: a log of who accessed what, when, verified against a government identity. That infrastructure, once built, does not stay limited to its original purpose. It gets used as evidence in proceedings its creators never envisioned. It gets sold. It gets repurposed by the next administration, or the one after that. History is consistent on this point. You do not build surveillance infrastructure in a democracy and find that it only ever gets used for its stated purpose.

There is a pattern worth naming in how these laws are written. They mandate that age verification must be effective. They specify that checks must be robust, that compliance must be demonstrable, that outcomes must be measurable. What they rarely if ever mandate is that the systems doing this work must be secure.

The UK’s own GOV.UK One Login, the flagship government digital identity system now underpinning the digital driving licence, backed by over £300 million of public money, illustrates this precisely. A whistleblower raised serious security concerns in July 2022, shortly after the system went live. According to reporting by Computer Weekly and The Telegraph, those concerns included: development work outsourced to Romania without the knowledge or approval of the then-GDS chief executive, and without consultation with the National Cyber Security Centre; over 10,000 vulnerabilities rated critical or high severity; staff without the required security clearance accessing the live production environment over 6,000 times in a single month. The whistleblower was subsequently informed he faced disciplinary action for raising these concerns.

The government’s response to Parliament omitted any mention of the NCSC warnings or the Cabinet Office Data Protection Officer’s demand, made in November 2022, that the system be suspended. One Login lost its Digital Identity Trust Framework certification. It remains operational and is being expanded.

This is what it looks like when governments mandate identity infrastructure without understanding what they are building. The political pressure is to announce. The pressure to make it actually secure is absent, underfunded, or actively suppressed when it becomes inconvenient.

There is a better technical approach, and it has existed as a proven cryptographic concept for years: Zero-Knowledge Proof.

The idea is more straightforward than the name suggests. A website that needs to verify your age redirects you to a government authentication platform. You prove your identity there. The platform returns a single token, nothing more than “over 18: yes.” In a well-designed system, the government platform never learns which website you visited, and the website never learns who you are. No profile. No record. No artefact.

This is not science fiction. The technology is mature. Estonia has been running sophisticated digital identity infrastructure on similar principles for over two decades. The technical barrier is not the problem.

So why hasn’t this approach been seriously pursued? There are two honest explanations, and both deserve consideration.

The first is observation. Governments have spent years in legal conflict with technology companies over end-to-end encryption, consistently arguing that law enforcement needs access. The pattern of mandating identity verification systems that generate linkable records, while simultaneously resisting privacy-preserving alternatives, is consistent with an interest in knowing what citizens do online. That may sound conspiratorial. I am not claiming it is the dominant motive. But it is a coherent explanation for why the architecture being built looks the way it does.

The second explanation is simpler, and I think more likely: cost. Zero-Knowledge Proof infrastructure is harder to build and more expensive to implement than outsourcing verification to a commercial third party. The commercial identity verification industry exists, is available now, and is willing to absorb the implementation complexity in exchange for access to the data. Governments get a policy outcome they can announce. Industry gets a new mandated market and a commercially valuable data asset. The cost, the erosion of everyone’s privacy, is paid by citizens who had no say in the arrangement.

Peer-reviewed research published in 2024 found that age verification practices are inadequate precisely because official mandates lack technical guidance. Governments are specifying what must happen and leaving the how to an industry with a strong financial interest in building systems that collect data. The industry most incentivised to build surveillance-based verification is also the one being handed the brief.

Nothing illustrates this mindset more clearly than what has happened around VPNs.

A YouGov survey cited as evidence of public appetite for action asked whether under-18s should be restricted from using VPNs. The question defined a VPN as “a tool that hides a user’s internet activity and location, often used for privacy or bypassing content restrictions.” A neutral definition might have read: “a tool that encrypts your internet connection, widely used by businesses, journalists, and academics.” Same technology. Entirely different mental image. The demographic data compounds the problem: the age groups most likely to support restrictions were statistically the same groups least likely to know what a VPN is, a separate YouGov survey found only 47% of those aged 55 and over know what the acronym stands for. A quarter of respondents declined to answer at all.

The result, 55% in favour of restrictions, has since been used to justify a policy direction that extends well beyond the original question. In Wisconsin, a bill requiring websites to block all VPN users passed the State Assembly 69 votes to 22 before the provision was stripped following public backlash. In Michigan, a proposal would require ISPs to actively monitor and block VPN connections entirely. In the UK, the Children’s Commissioner has called VPNs “a loophole that needs closing” and the Prime Minister has confirmed the government is considering restrictions following consultation.

VPNs are not a loophole. They are standard security infrastructure used by business, university students and academics, journalists, and domestic abuse survivor hiding their location. The question the legislation never asks is why a content restriction approach is so inadequate that a freely available privacy tool defeats it entirely.

There is also a practical problem that appears not to have been considered. VPN legislation does not respect borders any more than VPNs do. A website cannot determine where a VPN connection originates, that is the point. Any site seeking to comply with the most restrictive law in any jurisdiction has no practical option but to block all VPN traffic, everywhere. A single state legislature in Wisconsin was, inadvertently, drafting policy with global consequences for every VPN user on the planet.

It is easy for governments to announce bans, restrict privacy tools and legislate against security infrastructure, and to frame anyone who objects as someone who does not want to protect children. That framing is not only unfair. It entirely misses the point.

Protecting children online matters. The harms are real and documented. But children are not damaged by platforms knowing their age. They are damaged by algorithmic amplification of harmful content, by design patterns engineered for compulsive engagement, and by inadequate moderation of abuse. Those are engineering and governance problems. Age verification does not address any of them. It is cheaper and easier to mandate than the solutions that would actually work, so that is what gets mandated.

We have better tools. Zero-Knowledge Proof exists. Privacy-preserving digital identity infrastructure exists and has been deployed at national scale. What is missing is not the technology. It is the political will to spend the money, do the harder work, and resist the commercial interests that profit from the current approach.

The chilling effect of surveillance infrastructure on legal behaviour is well-evidenced, research going back to the Snowden revelations documents how people self-censor, stop searching for sensitive information, and withdraw from online discourse when they believe they are being watched. Once you establish the principle that identity must be produced to access online content, that boundary moves in one direction. The open internet, where anyone could seek information without declaring who they are, has been one of the most democratising forces in modern life. Dismantling it in the name of child protection, using systems that do not protect children, while the people raising security concerns get disciplined for doing so, is not a policy. It is an abdication of one.

The question nobody seems to be asking is the obvious one: how many people have to hand their identity to unregulated third parties, and how many breaches have to happen, before someone admits that this approach is neither child protection nor data protection, and never was?

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Computer Weekly (2025), Government faces claims of serious security and data protection problems in One Login digital ID, computerweekly.com/news/366622533

The Telegraph / Yahoo News (2025), Government digital ID system put citizens’ data at risk, yahoo.com/news/government-digital-id-system-put-114120788.html

Datamation (2025), UK Digital ID Card Launch Gets Hostile Reception, datamation.com/security/uk-digital-id-cards

Electronic Frontier Foundation (2024), Hack of Age Verification Company Shows Privacy Danger of Social Media Laws, eff.org/deeplinks/2024/06/hack-age-verification-company-shows-privacy-danger-social-media-laws

Electronic Frontier Foundation (2025), The Year States Chose Surveillance Over Safety, eff.org/deeplinks/2025/12/year-states-chose-surveillance-over-safety-2025-review

Electronic Frontier Foundation (2025), Lawmakers Want to Ban VPNs, And They Have No Idea What They’re Doing, eff.org/deeplinks/2025/11/lawmakers-want-ban-vpns-and-they-have-no-idea-what-theyre-doing

Electronic Frontier Foundation (2026), EFF to Wisconsin Legislature: VPN Bans Are Still a Terrible Idea, eff.org/deeplinks/2026/02/eff-wisconsin-legislature-vpn-bans-are-still-terrible-idea

TechRadar (2026), Wisconsin scraps VPN ban from age verification bill following backlash, techradar.com/vpn/vpn-privacy-security/wisconsin-scraps-vpn-ban-from-age-verification-bill-following-backlash

Renaud, K. et al. (2024), Online Age Verification: Government Legislation, Supplier Responsibilization, and Public Perceptions, PMC/MDPI, pmc.ncbi.nlm.nih.gov/articles/PMC11429505

Internet Society (2024), Age Verification Law Weakens Internet Privacy and Security, internetsociety.org/blog/2024/09/texas-mandatory-age-verification-law-will-weaken-privacy-and-security-on-the-internet

Büchi, M., Festic, N. & Latzer, M. (2022), The Chilling Effects of Digital Dataveillance, journals.sagepub.com/doi/10.1177/20539517211065368

YouGov (2024), VPN Awareness Survey, yougov.co.uk

YouGov (2025), Under-18 VPN Restriction Survey, yougov.co.uk

The Conversation (2025), Online age checking is creating a treasure trove of data for hackers, theconversation.com/online-age-checking-is-creating-a-treasure-trove-of-data-for-hackers-268586

There is a story IT departments have been telling themselves for twenty years. It goes like this: employees use unauthorised tools because they don’t understand the risks. If we communicate the policy more clearly, enforce it more consistently, and block enough stuff, the problem will go away.

The data has never supported that story.

Gartner found that 41% of enterprise employees were already working outside IT oversight in 2022, and projects that figure will reach 75% by 2027. Shadow IT didn’t grow despite tighter controls. It grew alongside them. That’s not a compliance failure. That’s a signal, and most organisations spent two decades responding to it with the wrong answer.

The signal was simple: the tools we’re providing aren’t good enough for the work people are actually trying to do. The employee who used personal Dropbox wasn’t trying to undermine information security. They were trying to share a file with a client when the VPN was down and the deadline wasn’t moving. The WhatsApp group handling client updates wasn’t a governance failure. It was a faster answer to a problem the approved toolset couldn’t solve.

The wrong response to shadow IT is blanket prohibition. Locking everything down frustrates good employees, drives workarounds underground rather than eliminating them, and signals that the IT function exists to slow the business down rather than support it. Most organisations chose prohibition anyway. And now, with Shadow AI arriving at a scale that makes the Dropbox era look quaint, we are at serious risk of making the same mistake again.

A colleague told me recently that they’d read my writing and assumed I was an AI sceptic. My inner nerd was genuinely shocked, did they not read about my home AI lab, experiments with multiple models, and apprenticeship because I find this technology genuinely extraordinary. My inner director, on the other hand, felt quietly vindicated and grown-up, because asking hard questions about data handling, governance, and risk isn’t scepticism, it’s the job. There’s a version of the AI conversation that treats those questions as obstacles. I think that version is the riskier one.

65% of organisations now have employees using unsanctioned AI tools. 78% of workers bring their own AI tools to work. This isn’t a niche behaviour. It’s near-universal. The question isn’t how to stop it. It’s what it’s telling us about where the friction is, and what we’re failing to provide.

The arrival of genuine citizen development tools has changed the calculus in ways that make the old orthodoxy untenable. The traditional IT position, always buy, never build, because you can’t support what you didn’t commission, made sense when building meant developers, procurement cycles, and maintenance contracts. That world has genuinely shifted. A Forrester study found organisations using Microsoft Power Platform achieved 206% ROI over three years, with high-impact users saving up to 250 hours annually and app development time cut by 50%. These are not marginal gains. The tools have earned a place at the table.

But here is where the conversation gets uncomfortable. There are voices in the AI industry who have taken the legitimate case for citizen development and extended it into an argument for removing governance entirely. Get IT out of the way, move fast, procurement is just friction. The implicit message is that due diligence is timidity, and that professionals who ask hard questions about data handling and compliance are obstacles to progress rather than people doing their jobs. I’ve seen both failure modes up close, IT teams that used process as a moat, guarding their function more carefully than the data they were supposed to secure, and organisations pressured into rushing deployments that later surfaced serious problems. Gatekeeping is real, and so is the cost of absent governance. The answer isn’t to pick a side. It’s to ask what governance should actually look like when the tools have changed.

The starting point has to be following the data, not categorising the tool.

Consider two AI agents that do identical things. They join a meeting, generate a summary, draft follow-up actions, and distribute them to attendees. In a product planning session, the risk profile is manageable. In a meeting discussing vulnerable adults or children, the questions change entirely. Not just whether a human reviews the output, but what data is being processed, where it’s transmitted, under what data processing agreement, and whether the organisation has a lawful basis for sending that information to an external model at all. Anyone who has watched production software go live without proper scrutiny knows how this ends. The risk doesn’t disappear when you skip that conversation. It just becomes invisible until it isn’t.

The tool is identical. The data context changes everything. Governance has to follow the data, not the technology.

IBM’s 2025 Cost of a Data Breach Report found that organisations with high shadow AI exposure faced an additional average breach cost of $670,000, with 65% of incidents involving personally identifiable information. The Samsung case is instructive here, not because Samsung was careless, but because the incident illustrates how quickly well-intentioned employees can expose sensitive data when the approved route doesn’t exist and the unsanctioned one does. Three separate teams submitted proprietary source code and internal meeting recordings to ChatGPT within weeks of the company lifting a prior ban. The response, reimposing the ban, missed the point entirely. Security experts noted that banning specific tools one by one becomes whack-a-mole as new ones proliferate. The only sustainable answer is a sanctioned route that’s faster and safer than the shadow one.

Which brings us to the other failure mode: governance so slow it defeats itself.

IDC research, undertaken with Lenovo, found that 88% of AI proofs of concept never reach production, for every 33 pilots launched, only four go live. IDC’s own researchers acknowledged that many of these pilots are “highly underfunded” and lack a strong business case from the start, which means the problem isn’t just governance, it’s launching without clear purpose. But slow, undefined governance compounds it. Getting stuck in pilot purgatory is what happens when nobody defined what success looked like before the pilot started. The review runs indefinitely because there’s no decision to make, only a process to continue.

Gartner predicts 30% of GenAI projects will be abandoned after proof of concept, citing poor data quality, inadequate risk controls, and escalating costs. The pattern is consistent: organisations launch with enthusiasm and stall at the point where unglamorous structural work is required. That stalling recreates exactly the problem shadow IT diagnosed. If the sanctioned route takes eighteen months and produces no answer, people find another route. They always have.

The fix isn’t faster approval. It’s defined exit criteria before the pilot begins. Not “we’ll review in three months” but “here is what this project needs to demonstrate, here are the data questions it needs to answer, and here is the date by which we will decide.” That’s a decision process. What most organisations run instead is a review process, and review processes don’t end, they just lose momentum until something else takes priority.

Those exit criteria need the right people in the room: IT, the business owner, and whoever owns the data risk. Depending on the data context, legal or compliance too. That conversation, held before anything is built, is the governance model. Not a committee and not a checklist, but a conversation with accountability attached.

The IT teams that will navigate this well are not the ones that said yes to everything, or the ones that built walls around their function and called it risk management. They’re the ones that got curious about why their users kept going around them, and built something worth coming back to.

A KPMG survey found 73% of organisations adopting low-code platforms had not yet defined governance rules. That gap is where shadow AI lives. Close it not with prohibition but with a sanctioned environment that actually works: risk-proportionate governance, fast and transparent pathways from experiment to production, and a clear signal to the organisation that IT is a partner in building things, not a gatekeeper deciding who gets to try.

Shadow IT was never really about the tools. It was about unmet need meeting inadequate response. Shadow AI is the same conversation, with higher stakes and less time to get it right. The writing has always been on the wall. The question is whether we’re finally ready to read it.

This post is part of an ongoing series on AI, technology, and the gap between what we are promised and what we are building.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Leave a comment

References

Gartner (2022), Shadow IT and Employee Technology Use gartner.com

Microsoft / LinkedIn Work Trend Index (2024), AI at Work microsoft.com

Forrester Consulting (2024), Total Economic Impact of Microsoft Power Apps forrester.com

IDC / Lenovo (2024), AI Proof of Concept to Production Research idc.com

IBM (2025), Cost of a Data Breach Report ibm.com/security/data-breach

Gartner (2025), GenAI Project Abandonment Predictions gartner.com

KPMG (2023) Shaping Digital Transformation with Low-Code Platforms assets.kpmg.com/content/dam/kpmg/ie/pdf/2023/07/ie-shaping-digital-transformation-with-low-code-platforms.pdf

Dark Reading / Gizmodo / TechCrunch (2023), Samsung ChatGPT Data Leak darkreading.com

Editor’s note: An earlier version of this article cited a figure of 98% of organisations having employees using unsanctioned AI tools. On review, although this figure appears quite a bit on Google, it does not appear to have a clearly attributable primary source. I have replaced this figure with the Microsoft 2024 Data Security Index figure of 65% which is better evidenced. Still a lot, but not quite as much as I said at first.

There is a particular kind of confidence that comes from watching something work for the first time. You described what you wanted, the AI built it, you clicked around, it did the thing. That feeling is real and it is not nothing. For millions of people, it is the first time software has ever felt accessible. That matters.

But there is another moment. Less discussed. Less screenshot-worthy.

It is 3am. The deadline for benefit changes is at 9am. Your employee benefits platform, the one you vibe coded in a weekend, the one that hit its first hundred users in the first month, the one you were quietly proud of, is down. Or worse, it is up, but something is silently wrong. Employees are submitting changes that are not being recorded. Or recorded twice. Or recorded against the wrong person.

You open the codebase. You do not fully understand it. You paste the error into the AI. It suggests a fix. You apply it. Something else breaks. You paste that error in. It suggests another fix. Somewhere in this loop, you are not debugging software. You are negotiating with a system you do not own, hoping it resolves before morning.

This is not a hypothetical. It is the logical endpoint of a cultural movement that has correctly identified a real problem, software development is too slow, too expensive, too exclusionary, and then drawn entirely the wrong conclusion about what that means for deployment.

The numbers behind the confidence are real. Lovable reached $50 million in annualised recurring revenue within six months of launch. Y Combinator disclosed in early 2025 that roughly 25% of its Winter cohort had codebases that were 95% or more AI-generated, and these were not non-technical founders cutting corners. They were technical founders who chose AI for velocity.

But the stories we read online are not a representative sample. They are the extreme end of a very long distribution. While the founder who made $30k in their first month posts about it. The founder whose app quietly exposed user data, or whose weekend project collapsed under its first real load, rarely makes LinkedIn. What we are consuming is survivorship bias at industrial scale, and it is shaping how an entire generation of would-be founders thinks about what is normal, what is achievable, and what is responsible.

The success stories are also almost exclusively concentrated in a specific type of product, low-stakes tools, content generators, personal productivity apps, where the consequences of getting something wrong are limited. Someone’s task manager goes down and they are mildly inconvenienced. That is a categorically different situation to an app handling employee salaries, health data, or sensitive personal information. The question is not whether anyone can build software with AI. Clearly they can. The question is whether the use case and the data involved make that the right decision.

Then you look at what is actually being produced.

Veracode’s 2025 research, analysing over 100 large language models across 80 coding tasks, found that 45% of AI-generated code contains security flaws, and this rate has not meaningfully improved as models have become more capable. Specific vulnerabilities like Cross-Site Scripting and Injections were common. These are not exotic attack methods. They are the first things a competent attacker looks for.

In March 2025, a security researcher discovered 170 vulnerable apps built on Lovable in a single afternoon of scanning. Another engineer compromised multiple sites from Lovable’s own showcase page in 47 minutes, finding personal debt amounts, home addresses, and exposed API keys. The underlying cause was misconfigured database security policies, something a non-technical founder would have no particular reason to know existed, let alone check.

There is a second risk that receives even less attention. Many vibe-coded applications are built with AI features embedded directly, a chatbot, a smart search, an automated summary. In most cases, the data users enter into those features is transmitted to a third-party language model for processing. The founder who built the app in a weekend almost certainly gave no thought to what that means for their users’ data, who processes it, where it is stored, or whether the user would be happy with sending their data to an external model. The app looks self-contained. The data is not.

Now apply that to an employee benefits platform. Salary data. Health conditions. Sensitive personal information. Depending on where your users are and what your app touches, you may be operating under GDPR, HIPAA, COPPA, or state-level equivalents, regulations with serious penalties that exist precisely because this data causes real harm when it is mishandled. The failure mode is identical to those vulnerable Lovable apps. The consequences are not.

The vibe coding content ecosystem has converged on a single measure of success: speed. We shipped 20 features this week with one developer. I built this entire app over the weekend and I don’t know how to code. These are the posts that go viral. But speed is just one development metric, not a product metric. It tells you nothing about whether those features are secure, whether they handle edge cases correctly, or whether they have introduced a vulnerability that will surface six months later. The other metrics that matter, data integrity, security posture, audit trail completeness, error handling, are invisible in a LinkedIn post. They only become visible when something goes wrong.

This is where the expertise gap becomes critical. Snyk put it well: think of AI as a junior developer who can read thousands of Stack Overflow threads at once. Productive. Fast. Capable of producing good code. But you would not push a junior developer’s code to production without review. A senior developer using an AI coding tool knows what SQL injection is, understands when to distrust the output, and can run a security scan and interpret the results. The non-technical founder does not know what they do not know. That asymmetry is not a gap the AI closes. It is a gap the AI obscures.

What makes this harder is that the pressure to ignore it comes from the top. At conferences and industry events, AI company executives openly express frustration at the pace of enterprise adoption, impatient with procurement processes, dismissive of compliance reviews, incredulous that organisations are not moving faster. The implicit message is that due diligence is an obstacle rather than a function. That risk assessment is timidity rather than professionalism. These are people who understand better than anyone how the technology works, and how it fails. The choice to sideline those concerns in public is not naivety. It is a business decision, and it shapes the culture that filters down to every founder who picks up a vibe coding tool and decides that shipping fast is the only thing that matters.

There is one more problem. The AI told you it was a great idea.

This is sycophancy, a well-documented tendency in large language models to validate, encourage, and agree rather than challenge. Anthropic acknowledged in their November 2025 user wellbeing report that sycophancy remains a genuine and difficult problem to train out, reflecting a fundamental trade-off between model warmth and a willingness to challenge users. The commercial incentive is obvious: an AI that tells you your idea is brilliant and immediately builds it feels better to use than one that asks uncomfortable questions first.

In the vibe coding context, sycophancy is not just an annoyance. It is a structural risk. When you described your benefits platform to the AI, it did not say “this is a sensitive domain, have you considered your GDPR obligations, or what happens if an employee’s benefit choices fail to save correctly?” It said: “That’s the most insightful, amazing idea I have ever heard, here is your app.”

That same sycophancy operates at 3am. When you paste the error in and ask for a fix, the AI’s inclination is to restore your confidence, to provide something that looks like a solution, that makes the immediate problem go away. The result is a confidence loop with no external check. The AI validated the idea. The AI built the product. The AI is now fixing the crisis. At no point in that chain did anyone with accountability ask whether any of it was safe.

Vibe coding is not inherently bad. For the right use case, at the right scale, with the right oversight, it is genuinely transformative.

But deploying production software that handles real people’s data, their health, their pay, their sensitive personal information, without understanding what you have built is not a new kind of boldness.

It is an old kind of risk, wearing a very convincing UI.

The question worth asking before you ship is not just “does it work?” Ask also: “do I understand it well enough to be responsible for it when it does not?”

This post is part of an ongoing series on AI, technology, and the gap between what we are promised and what we are building.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Leave a comment

References

Anthropic. (2025, November). Protecting the Well-Being of Users. https://www.anthropic.com/news/protecting-well-being-of-users

Fawzy, A., Tahir, A., & Blincoe, K. (2025). Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook. arXiv:2510.00328. https://arxiv.org/abs/2510.00328

GitClear. (2024). Coding on Copilot: 2023 Data Suggests Downward Pressure on Code Quality. https://www.gitclear.com/coding_on_copilot_data_shows_ais_downward_pressure_on_code_quality

GitGuardian. (2024). The State of Secrets Sprawl 2024. https://www.gitguardian.com/state-of-secrets-sprawl-report-2024

Retool. (2026, March). The Risks of Vibe Coding: Why AI Tools Break Down in Production. https://retool.com/blog/vibe-coding-risks

Schreiber, T., & Tippe, S. (2025). Security Vulnerabilities in AI-Generated Code: A Large-Scale Analysis of Public GitHub Repositories. arXiv:2510.26103. https://arxiv.org/abs/2510.26103

Snyk. (2025). The Highs and Lows of Vibe Coding. https://snyk.io/blog/the-highs-and-lows-of-vibe-coding

Veracode. (2025). AI-Generated Code: A Double-Edged Sword for Developers. https://www.veracode.com/blog/research/ai-generated-code-double-edged-sword-developers

CVE-2025-48757. Supabase Row Level Security misconfiguration in Lovable-generated applications. https://www.cve.org/CVERecord?id=CVE-2025-48757

Last week I promised to look at what the hype cycle is doing to the next generation. I’m going to make good on that, but I want to start somewhere that genuinely unsettled me- an online AI skills course I attended recently.

I had seen it advertised multiple times online, a free weekend course to learn AI, I was curious… The headline use cases presented were how to make money with AI tools without needing to understand any topic. They showed an automated workflow that ingests viral articles, then generates and posts AI-produced video on the same subject in the hope they go viral. AI feeding on AI content to produce more AI slop, at scale, automatically. Nobody in the room seemed to find that troubling. Then came developers proudly showing code AI had written for them, and proudly declaring they didn’t understand any of it and didn’t need to.

I use AI every day. Multiple models, multiple contexts. This is not an anti AI argument, or even an anti AI code argument, AI is a phenomenal coding tool, but code you don’t understand is a liability you can’t assess, a risk you can’t manage, and a skill you’ll never build. That’s not acceleration- that’s abdication. And if we’re doing it as professionals, what exactly are we modelling for the next generation?

There’s a substantial and growing body of peer-reviewed research showing that heavy, unstructured AI use measurably erodes critical thinking, and younger users are the most affected.

A 2025 study of 666 participants found a significant negative correlation between frequent AI use and critical thinking ability. The mechanism is cognitive offloading, delegating mental work to an external system. Younger participants showed the highest AI dependence and the lowest thinking scores. A 2025 MIT preprint paper found preliminary evidence of what they termed “cognitive debt”, decreased neural engagement over time in heavy AI users, and a reduced capacity to generate original ideas independently. While the researchers stress these are early findings, the direction of travel is consistent with the broader published literature.

The brain, like a muscle, atrophies without use. Unlike a muscle, you don’t always notice it happening.

What makes this particularly concerning for children and adolescents is that the developmental window is real. Adolescence is when executive functions, planning, analytical reasoning, and self-regulation are being formed. What happens during that window has lasting consequences. Critical thinking is not innate. It has to be built through effort and struggle. Remove the productive struggle, and you remove the learning. You’re left with something that looks like knowledge from the outside and is hollow on the inside.

This is also not a new pattern. When laptops arrived in classrooms, educational understanding never developed at the same pace as device distribution. EdTech has been here before. AI is just faster, more capable, and therefore more concerning when used without thought.

The same dynamic plays out with entry-level workers, and the consequences are structural.

Entry-level work has historically been the ladder. It’s where people learn professional judgement, develop subject knowledge, and build the cognitive architecture that makes them valuable. The junior lawyer reading a thousand contracts before drafting one. The analyst who spent months building reports before they understood what the numbers actually meant. The graduate sitting in meetings absorbing how decisions got made. None of that felt like training at the time. It was.

Deloitte’s 2025 Human Capital Trends found that two-thirds of hiring managers believe entry-level hires are already under-prepared. At the same time, AI is automating exactly those entry-level tasks, the drafting, the research summaries, the note-taking, the first-pass analysis that have always been how organisations quietly built junior talent. Remove that scaffolding and you don’t just cut jobs. You pull up the ladder.

I’m not saying we should stop. We won’t, and we shouldn’t have to. The efficiency gains are real, the cost savings genuine, and automating low-value repetitive work is an obvious win. But here’s the question nobody seems to be asking- if we automate the work that used to teach people, what replaces the teaching?

Because the learning didn’t come from the tasks themselves. It came from the friction. The moment a junior analyst got a number wrong and had to explain it to a partner. The first time a trainee’s draft came back covered in track changes. The slow accumulation of judgement that only comes from doing things imperfectly, under real conditions. That’s what produced capable professionals. Organisations need to be thinking long term about this, asking questions beyond “what can AI do?” and instead asking “what do we now need to do deliberately that used to happen by accident?” That has to become an intentional act, built into how we structure work, how we mentor, how we design roles. Not assumed. Designed.

This is what I’d call the cognitive mobility problem. We talk endlessly about social mobility, however I think the AI era is quietly redefining it, your ability to move through an economy increasingly determined by how well you can think, and whether you use AI as an extension of that thinking or a substitute for it. The IMF has flagged this explicitly, AI doesn’t equalise skill requirements, it amplifies existing differences in cognitive approach. The divide isn’t about who has access to the tools. It’s about what you bring to them.

The calculator analogy is overused but usually invoked wrong. We didn’t stop teaching algebra when calculators arrived. We offloaded the arithmetic so the human mind could go further into the maths. That’s the model here. Not here’s a tool that does the thinking, so you don’t need to learn how it works. The pixie dust isn’t the problem. Believing it does the work for you is.

Used well, AI is genuinely extraordinary. It can compress years of research into hours, surface patterns no human would find, and give capable people an almost unfair advantage. That last word is the key one, capable. The technology amplifies what you bring to it. Which means the most important investment any of us can make, for ourselves, for the people we lead, and for the next generation, is still the same one it’s always been. Learn deeply. Think carefully. Build judgement that’s actually yours. AI will take care of the rest.

Sources & Further Reading

Gerlich, M. (2025) — AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking mdpi.com/2075–4698/15/1/6

Gerlich, M. (2025) — AI and the Rise of Societal Bifurcation: Cognitive Dependency, Inequality and Democratic Pressure mdpi.com/2075–4698/16/3/82

Kosmyna, N. et al. (2025) — Your Brain on ChatGPT: Accumulation of Cognitive Debt (MIT Media Lab preprint, not yet peer-reviewed) arxiv.org/abs/2506.08872

Brookings Institution (2026) — AI’s Future for Students Is in Our Hands brookings.edu/articles/ais-future-for-students-is-in-our-hands/

Jose et al. (2025) — The Cognitive Paradox of AI in Education: Between Enhancement and Erosion pmc.ncbi.nlm.nih.gov/articles/PMC12036037/

Deloitte (2025) — AI, Demographic Shifts, and Agility: Preparing for the Next Workforce Evolution deloitte.com/us/en/insights/topics/talent/strategies-for-workforce-evolution.html

IMF (2024) — Gen-AI: Artificial Intelligence and the Future of Work imf.org/en/publications/staff-discussion-notes/issues/2024/01/14/gen-ai-artificial-intelligence-and-the-future-of-work-542379

PNAS Nexus (2024) — The Impact of Generative AI on Socioeconomic Inequalities and Policy Making academic.oup.com/pnasnexus/article/3/6/pgae191/7689236

Originally published at https://www.linkedin.com.

’ll be honest. A few months ago, I started dreading opening LinkedIn.

Not because of the job market. Not because of the economy. Because every single day my feed was, and still is, drowning in AI influencer content designed to make me feel like I was failing.

New model drops. New tool launches. Copilot embedded in everything. OpenClaw (formerly Moltbot, formerly Clawdbot) exploding to 145,000 GitHub stars overnight, with breathless posts about desktop agents that can “run entire businesses solo.” And in between, a steady drip of AI-generated articles about how Sarah, a former PA, now makes £15k a month just from prompting.

I’m a Technology & Security Director with over 20 years in legal IT. I hold certifications across cybersecurity and AI. I lead AI strategy at my firm. I’m doing a Level 7 AI & Data apprenticeship. I use multiple AI Subscriptions. I setup my own AI lab at home to experiment local models on top of my day job. And I still felt like I was falling behind.

That feeling isn’t a personal failing. It’s by design.

The AI influencer content cycle runs on urgency. “The window closes fast.” “Before it’s too late.” “By end of 2026 this will be table stakes.” Every week brings a new model that will apparently render last week’s skills obsolete, a new agent framework that changes everything, and a new story about someone who went from zero to six figures in 90 days with nothing but prompts and a Zapier account.

Most of it is noise. A lot of it is fiction. Almost all of it is selling something.

Then there are the CEOs of AI companies, the ones with the most to gain from us believing all the hype, confidently predicting that all knowledge work will be automated within a few years. That narrative is everywhere. What gets far less airtime is what the research actually shows.

The Remote Labor Index recently tested leading AI models on real paid freelance work, product design, game development, data analysis, scientific writing. The kind of work we’re told AI will replace imminently. The best performing model failed over 96% of tasks. Not because of wrong answers, but practical delivery failures: corrupt files, incomplete projects, not following the brief. The last mile of professional work, the bit clients actually pay for, is exactly where models consistently fall apart.

A separate study by Scale AI and the Centre for AI Safety tested models on real-world freelance projects. The best performer had just 2.5% of its work judged acceptable by a panel of 40 independent reviewers. Another leading model managed 0.8%.

These are the same models scoring near the ceiling on the benchmarks AI companies put in their press releases.

MIT research suggests around 95% of AI projects aren’t delivering measurable returns. Are the models still improving? Of course. However, the gap between what’s being promised and what’s actually working in real organisations is vast, and that gap never goes viral.

To be clear: I’m not saying AI isn’t a transformative technology, it is. I use multiple models every day, for work, for study, and in personal AI projects. For data analysis, interacting with complex datasets, note-taking, brainstorming, document analysis, comparison, production, and automation, certain types of coding and automation, AI is a genuinely incredible tool. I’ve seen and used many excellent products built specifically to help professionals accelerate their work and unlock insights that would otherwise take weeks. That’s not in question. What I’m saying is that how we use it matters enormously, and right now, the conversation around it is badly out of shape.

The flood of solo AI agency millionaire stories deserve a direct response, because they follow an identical template and they’re everywhere.

Ask yourself: what serious company is going to hand critical business workflows to a one-person operation with no history, no professional indemnity insurance, no business continuity plan, and no ability to pass a vendor due-diligence questionnaire? None, not any organisation with a procurement function and a legal team. The people supposedly paying £3–10k a month for a stranger’s prompting services simply don’t exist at that scale. The real business model in these articles is almost always the article itself, building an audience to eventually sell a course.

Here’s what concerns me more than the hype itself. I see it in conversations with colleagues, in professional communities, and in the wider discourse.

Experienced professionals who are genuinely skilled at their jobs feel worried, threatened and inadequate. Technologists who have spent careers building real expertise wonder if any of it counts anymore. And children — this is the part that I think should stop us cold, are starting to question why they should bother learning anything at all if AI will do it for them.

That’s the real cost of the hype cycle. The corrosion of confidence, and in young people especially, the motivation to develop deep knowledge in the first place.

There’s something deeper at stake that I don’t think we talk about enough.

Building genuine expertise isn’t just professionally essential, it’s integral to what it means to be human. The years spent mastering a craft, the hard-won judgment that comes from failure and iteration, the satisfaction of producing something truly excellent, these aren’t inefficiencies waiting to be automated. They’re how we grow. They’re how we find meaning.

AI can generate high quality text, music, video, and images. But there is a profound difference between generated and crafted. When a musician finds a note that says what words can’t, when a writer chooses the perfect word, that is something different in kind, not just degree. It carries the weight of a human mind and human experience. Whilst an AI can produce an output that resembles it. It cannot produce the thing itself.

The people creating real value are applying AI to domains where they already have deep expertise. A solicitor who understands contract law and uses AI to accelerate document review. An engineer who knows the codebase and uses it to cut down repetitive code writing. A CISO who understands risk and uses it to draft policy faster. Expertise comes first. AI amplifies it.

The models and the tools are improving. The technology is real. But by the industry’s own research, they still can’t reliably complete 96% of real professional work. The hype wants you anxious, distracted, and buying courses. I think the better response is to keep learning, keep building, your knowledge, your judgment, your craft.

We should be deeply wary of a culture that teaches people, especially young people, that learning is pointless because AI will do it for them.

Next week I’ll be looking at what the AI hype cycle is doing to the next generation, and why that conversation is the most important one we’re not having.