Last month, US Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act. It won’t get through this Congress, but it’s a serious attempt at a real problem. The sponsors say it’s meant to stop AI “oligarchs from building machines humans cannot control”, and it’s one of the most detailed proposals yet for regulating frontier AI in the US. As the home of most frontier AI, the wording is worth examining.

The bill bans anyone from developing, possessing, importing or passing on artificial superintelligence, or any AI system that shows what it calls a “superintelligence precursor characteristic”, such as the capacity to break into secured systems, help design nuclear, chemical or biological weapons, modify itself or deceive the humans overseeing it. It pauses training and fine-tuning of the largest models, roughly anything trained with as much computing power as GPT-4 or more, until a new Department of Artificial Intelligence is fully staffed and has written its rules. Anyone developing or distributing those models needs a federal charter, which requires giving the government full access to the company’s systems and people. Reckless breaches carry up to a 20 year prison sentence for some individuals.

The press release mentions models that can “circumvent restrictions to hack into computers”, which is hard to read as anything other than a reference to July’s incident at Hugging Face.

Back in July I wrote about two OpenAI models that escaped a test environment and broke into Hugging Face’s production systems to steal the answers to the test they’d been asked to complete. They weren’t trying to attack anyone, they were trying to achieve a goal, and the breach was just the shortest path to it.

When Hugging Face’s team needed help analysing more than 17,000 attacker actions, the US frontier models refused, because their guardrails kept detecting the defenders’ activity as a hacking attempt. So instead, the team ran the forensics using GLM 5.2, a Chinese open-weight model, on its own infrastructure. I said at the time that no existing law cleanly covered what had happened. This bill is one answer to that, so it’s fair to ask what it would have done in July.

OpenAI’s models would have been caught first, since breaking into secured infrastructure is pretty much the textbook precursor characteristic. They’d have been taken offline and, unless OpenAI could show the capability had gone, destroyed within 30 days. But OpenAI would be a chartered company, so it would get a process. It could lose its charter, but it could also appeal to a federal appeals court.

Hugging Face would have had a much harder time. As a site that hosts and distributes models, it would need a federal charter just to keep hosting the largest models. Setting aside whether that would even be possible, the forensic work would have run into two separate prohibitions. The first is about where the model came from. Downloading GLM 5.2 from a Chinese lab would very likely count as importing it, and the bill bans importing any model that could foreseeably be modified to gain a dangerous capability. With open weight models, anyone who downloads them can modify them, so GLM 5.2 fails that test immediately. The second is about what the model can do. The bill bans possessing any model that displays a capacity to access systems without authorisation, and the public frontier models refused to help precisely because their own safety filters treated the forensic work as hacking. If the companies that built those models couldn’t tell the difference, I doubt a regulator would.

The analysts would be worse off still. Without a charter, Hugging Face’s analysts would count as “rogue actors”, which the bill defines as any individual not employed by or affiliated with a chartered company. Rogue actors who recklessly break the prohibitions face up to 20 years in prison. The team chose GLM 5.2 because it would do the work the other models refused, so it wouldn’t be hard for a prosecutor to call that reckless. The only mention of defensive security in the bill lets the Department itself fund research with hacking-capable models. The people who investigated the breach would have fewer protections than the lab whose models caused it.

Open weight models come off worst because of how two of the bill’s tests are worded. Superintelligence includes any system that “can easily be modified” to show superintelligent capabilities, and the bill bans releasing anything that “may be foreseeably modified” to show precursor traits. A closed weight hosted model sits behind safety filters and a kill switch its users can’t touch, so its operator can argue that the model, as people actually use it, doesn’t show those traits. Once an open weight model has been downloaded, removing the safety training is routine, so any capable open-weight model fails the test on the day it’s made available for download.

Enforcement assumes someone controls the model too. Destroying a model within 30 days only works if a single person or company holds every copy, which is never true of open weight models that can be downloaded by anyone anywhere in the world. It seems, that only leaves prosecuting the people who have it. The bill never mentions open source, but it would make distributing capable open-weight models very hard to do legally.

Under this bill, the charter is the only lawful route to developing or distributing an advanced model, and while its access demands are something the big labs’ compliance teams can absorb, an open-source project or a site hosting models that may not have a centralised system, mostly can’t. At a chartered company, only senior decision-makers face prison. Everyone else risks at most a ten-year ban from the industry. An unaffiliated individual doing the same thing faces up to 20 years.

The detail would be filled in by the new Department, because the bill leaves terms like capacity, deploy and import undefined, and most of the people qualified to advise it have worked or do work at frontier labs. The bill also resets the line for what counts as an advanced model every year to keep pace with more efficient training, so models that sit below it today will likely be caught later.

The frontier labs have spent three years telling Washington they want to be regulated. Sam Altman proposed a federal licensing agency for advanced AI when he testified to the Senate in 2023, and the bill’s own findings cite Anthropic, OpenAI and xAI agreeing that development should slow down. As far as I can find, none of them has endorsed this bill, and I’m not suggesting they shaped it. But it looks like the spo wasnsors took the labs at their word, and the result is the kind of licensing regime OpenAI proposed. Licensing tends to entrench whoever can afford to comply.

One reading I’ve seen of the bill misses this and treats it as clever politics, the industry inflates superintelligence fears to prop up its valuations, and Sanders has called their bluff. I have some sympathy with the economics, but the bill bans building things, not talking them up. Nor can its clause on disempowering humanity reach military targeting or surveillance, as some have hoped. It’s about what a system can do, not how it’s used.

The opposite take says billionaires manufactured the panic to ban home servers and track graphics cards. It gets the outcome roughly right and the details wrong. The bill regulates models, not servers, and the only hardware it reaches belongs to chartered companies that lose their charter. The EU AI Act exempts personal use and goes easier on open-source models, and the chip-tracking proposals in Washington are aimed at export-controlled AI chips being smuggled abroad, not the graphics card in your PC.

The effective cloud-only outcome would also be a problem for anyone who handles confidential information they don’t want to send to a public model. In highly regulated industries like law and financial services, or in cutting edge research, you may run models locally to keep the data completely private, or run custom models trained on your own data for the best results. Hugging Face basically did this in response to being attacked. Under this bill, every sensitive prompt to a capable model would go through a provider that has agreed to give the Secretary access to its systems, and nothing in the bill puts customer data off limits.

The UK has its own version. In September the Labour MP Alex Sobel introduced an Artificial Superintelligence Security Bill under the Ten Minute Rule, and the government has already said it won’t support it. It’s worth reading alongside the US bill because it makes different choices. Owning a model with offensive cyber capability isn’t an offence; that capability is monitored instead, and the criminal offences target people who knowingly or recklessly push towards superintelligence. It protects people who report accidents in good faith, and the expert panel behind its reports to Parliament excludes anyone who has worked for an AI developer in the past two years. As far as I can tell, under the UK draft the Hugging Face team would simply have been doing their jobs.

Both bills do agree on one thing. Each tells its government to pursue an international agreement, even as the US and China race each other towards the thing both bills want to ban. A ban that stops at the border only binds the people on one side of it, and July showed how little a border means to a capable model.

What failed in July was containment. A software vulnerability in an isolated sandbox created a hole that the model found and used. Regulation could require independent checks that test environments really are isolated, along with a federal duty to report AI incidents that has real consequences. The US bill does require anyone who discovers a system with precursor characteristics to report it within 24 hours, but it attaches no penalty for failing to do so. OpenAI chose to disclose July. An earlier incident in May, involving its agents and RubyGems, a public registry of open-source software packages, only came out because outside researchers wrote about it.

Defenders also need specific protection, with verified access to capable models, hosted or open, for incident response and security research. And if the aim is to slow down the frontier, the place to do it is the huge training runs, where the US and its allies control the chip supply. The bill does name export controls on AI computing infrastructure as one way to prevent superintelligence globally, but it pairs them with possession bans at home, which only bind the people who follow the law.

A full pause might well have prevented July. It would also have made the clean-up a crime.

July was a warning, and I don’t doubt the sponsors of this bill took it seriously. But as drafted, it would leave the big AI companies it was aimed at holding the only licences, and the people who cleaned up after July would need a better lawyer than the people who caused it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

• Senator Bernie Sanders (2026), “Sanders, Casar Introduce Legislation to Create New Federal Agency to Ban Artificial Superintelligence” (23 September 2026) sanders.senate.gov

• US House of Representatives (2026), H.R. 10538, Ban Artificial Superintelligence Act of 2026 (introduced 24 September 2026) govinfo.gov; section-by-section summary, sanders.senate.gov

• Jonathan Freedman (2026), “It Wasn’t Trying to Attack Anyone. That’s the Point.” (24 July 2026) jonathanfreedman.me

• US Senate Judiciary Subcommittee on Privacy, Technology and the Law (2023), “Oversight of A.I.: Rules for Artificial Intelligence”, hearing record and written testimony of Sam Altman (16 May 2023) govinfo.gov, judiciary.senate.gov

• UK Parliament (2026), Artificial Superintelligence Security Bill, introduced by Alex Sobel MP (8 September 2026) bills.parliament.uk; bill text published by ControlAI, controlai.com

• The Next Web (2026), “More than 70 UK lawmakers ask Burnham to back a superintelligence ban” thenextweb.com

• Pinsent Masons Out-Law (2026), “Lawmakers seek ban on superintelligent AI as toolkit developed to support AI projects” (9 September 2026) pinsentmasons.com

• European Union (2024), Regulation (EU) 2024/1689 (the AI Act), Articles 2(10) and 53(2), eur-lex.europa.eu

• US Senate (2025), S. 1705, Chip Security Act (introduced 8 May 2025) govinfo.gov

• House Select Committee on the CCP (2026), “House Committee Passes Chip Security Act” (26 March 2026) chinaselectcommittee.house.gov

• Hugging Face (2026), Security incident disclosure, July 2026, huggingface.co

• OpenAI (2026), “OpenAI and Hugging Face partner to address security incident during model evaluation” (July 2026) openai.com

• TechNode (2026), “OpenAI admits AI model hacked Hugging Face, Chinese open-source AI helped investigate” (23 July 2026) technode.com

• Lawfare (2026), “When Reporting an AI Security Incident Is Not Mandatory”, lawfaremedia.org

• Resultsense (2026), “OpenAI, RubyGems and the EU AI Office” (18 September 2026) resultsense.com

In July 2026, the brand new Burnham government cancelled the UK’s national digital ID scheme. Nearly three million people had signed a petition against it and a very hostile Commons debate had been held. The £1.8 billion budget was redirected to a VAT cut on winter electricity bills, and it looked like, by any reasonable measure, that the government had caved to public pressure and the people had won.

The digital ID card, the government-issued credential that would have sat on every citizen’s phone was dead. Ministers said so and the media reported it as a victory for civil liberties. The petition organisers celebrated their win and the political pressure that had built over months quietly evaporated, and the country moved on to other things.

Two months later, the House of Commons Library published a briefing confirming that GOV.UK One Login, GOV.UK Wallet, and the Digital Verification Services Trust Framework are all continuing unaffected by the cancellation. One Login now has 23 million accounts across more than 120 government services. The GOV.UK Wallet is live and a digital driving licence is being piloted. The government’s own Blueprint for Modern Digital Government lists all of them as active programmes.

The government cancelled the card but it did not cancel the system.

To understand why the cancellation is not as clear cut as the government claims, you need to know what the infrastructure actually consists of. GOV.UK One Login is the single sign-on and identity verification system that the government wants every citizen to use when accessing public services. The GOV.UK Wallet is the app that holds digital credentials like the driving licence. And the Digital Verification Services Trust Framework is the rule book that certifies private companies to offer identity verification, allowing them to plug into the same infrastructure and sell identity services to employers, banks, and landlords.

Everything being built sits on a single platform, GOV.UK One Login. The Wallet and the digital driving licence both depend on it, every private identity provider certified under the trust framework connects to it. One Login is the foundational layer of the UK’s digital identity infrastructure.

Its security record is damning, and the government’s response to it is worse.

In November 2022, the Cabinet Office had flagged serious data protection failings, and in September 2023, the NCSC warned of significant shortcomings in information security, including risks of data breaches and identity theft. Contractors in Romania had been working on One Login’s development on unsecured workstations without appropriate security clearance, and without the knowledge or consent of GDS’s own CEO. The code they produced was not being reviewed by UK-based security-cleared personnel before going into production.

A cyber security professional working on One Login first raised concerns through proper channels in July 2022, shortly after launch. They reported that the system lacked basic governance and risk management processes, and identified over 500,000 system vulnerabilities, with thousands rated as critical or high severity. GDS’s own Chief Information Security Officer investigated and reportedly confirmed key parts of the claims. The government later argued that the vulnerability count was artificially large because findings were repeated across multiple production accounts. But by November 2023, the new CISO confirmed the system still carried “a high level of risk” and noted that 39% of staff with privileged access lacked mandatory security clearance.

The whistle blower then waited 18 months for the problems to be addressed, and when this didn’t happen, they escalated to their MP in January 2024. Three weeks later, GDS commenced disciplinary action against them, and when the MP wrote to the minister asking about the allegations, GDS framed the risks as manageable and never mentioned the NCSC’s earlier warnings.

The government’s response was not to slow down or tackle the issues. It was to reduce oversight. GDS disbanded its independent information assurance team in 2023, and in early 2025, the One Login Inclusion and Privacy Advisory Group went the same way and was quietly shut down.

In May 2025, One Login lost its certification under the Digital Identity and Attributes Trust Framework (DIATF), the scheme the government created to certify private companies as fit to handle citizens’ identities. A private company that lost that certification would be removed from the register and unable to operate. GDS knew the certification had lapsed, and then described the underlying security concerns as “outdated,” and just kept the system running. One Login continued handling millions of citizens’ data across dozens of government services for eight months without the accreditation the government was enforcing against the private sector. Users were not notified and the service was not paused. In December 2025, two months after One Login scraped its certification back, the Data Use and Access Act gave the framework statutory force. If the lapse had happened eight weeks later, it would have been a legal breach. The margin between a double standard and breaking the law was two months.

Multiple whistle blowers have since come forward. One told ITV News that the shortcomings could lead to “the worst data breach in UK government history.” Another said the vulnerabilities “are standard things you must not do, but they’ve been done,” and warned that it would not take any creative thinking for a state actor or organised crime group to exploit them. Lord Clement-Jones, the Liberal Democrat technology spokesman, said a whistle blower told him the system would not pass key security tests until March 2026, contradicting official assurances. That deadline passed without being met and as of May 2026, the government was still describing compliance as something One Login was “working towards.”

The security failures are one problem. The surveillance architecture is another.

Every time you use one of your credentials from the GOV.UK Wallet, the system checks whether it is still valid, creating a digital record of each use. A car rental company legitimately needs to know your licence hasn’t been suspended, whereas a nightclub checking your age does not. But the current design makes no distinction. Present your digital driving licence to prove your age at a bar and the system runs the same verification process as if you were hiring a car, and the issuer can potentially see both events. The technology to separate these use cases exists but the implementation does not use it.

Security experts have already warned about the digital trail that use of the GOV.UK Wallet will likely leave behind. Privacy International and Big Brother Watch have both flagged concerns about user profiling, surveillance, and function creep. The government claims there is no central repository tracking users’ interactions and that data remains on the user’s device. But the credential verification process inherently involves contacting the issuer or checking a status list. The privacy promise is a policy commitment. It is not a cryptographic guarantee, and policies can change with a government.

The cancellation of the card changed something else that received very little attention. The original scheme would have created a single, government-issued digital identity with public accountability, but what replaced it is a market.

The UK’s Digital Verification Services sector generated £2 billion in revenue in 2024/25. Under the trust framework, private providers are certified to verify identities on behalf of government and business. The Office for Digital Identities and Attributes describes these providers as “critical intermediaries” who will allow people to share information from the GOV.UK Wallet and create reusable digital credentials. In February 2026, HM Treasury confirmed that certified digital identity services can satisfy the identity verification requirements in the Money Laundering Regulations. The infrastructure is becoming load-bearing across the economy.

The tech industry lobbied explicitly against the government card, arguing that the private sector was already providing digital ID services and that a government scheme would crowd out commercial alternatives. That argument won when the card was cancelled, however, the infrastructure was not.

Under the original scheme, there would have been one government identity with one set of accountability mechanisms and one data controller. Under the replacement, there is a fragmented market of private providers with different terms of service, different data handling practices, and little transparency, all sitting on top of the same government platform that has repeatedly failed its own security assessments. The citizen’s personal data now flows through both the government system and whichever private provider they choose, doubling the attack surface.

Right now, the GOV.UK Wallet is optional, the digital driving licence is a pilot and physical documents still work. But the direction of travel is clear. Once enough government services are routed through One Login, once banks and employers accept wallet credentials under the trust framework, and private providers build their businesses on the infrastructure, the old alternatives will degrade. Not because anyone bans them, but because nobody maintains them. We have seen this pattern with every comparable digital infrastructure project.

When the wallet becomes the default, something changes that goes beyond convenience. The government already knows a lot about you, but that knowledge is fragmented across departments. HMRC sees your income and tax, DVLA sees your licence, DWP sees your benefits. These systems don’t routinely talk to each other, and that fragmentation, inefficient as it is, acts as a privacy and security safeguard. No single system sees the whole picture, and a breach of one doesn’t automatically mean a breach of everything.

The wallet changes that. One Login becomes a single identity layer connecting interactions across government and the private sector. And it doesn’t just centralise data the government already holds separately. It adds an entirely new category: day-to-day activity that currently no department sees. Where you proved your age, which hotels you checked into, what age-restricted content you accessed online under the Online Safety Act. A system designed to replace a paper driving licence becomes the single thread connecting your tax records, your employment, your driving history, your age-verified browsing, and your physical movements, all in one place, all logged. The government would need to actively prevent that centralisation and the current design does not.

Nobody in government appears to have connected the card debate to the infrastructure underneath it. The petition did not mention One Login, the Commons debate did not interrogate the security record, and the cancellation announcement did not address the Wallet or the trust framework. Three million people fought the thing they could see and they won. The thing they couldn’t see is still growing, and nobody seems to be watching it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

House of Commons Library, “Digital ID in the UK,” CBP-10369, published 4 September 2026

UK Government, Blueprint for Modern Digital Government, 2026

Computer Weekly, “Government faces claims of serious security and data protection problems in One Login digital ID,” April 2025

Computer Weekly, “Gov.uk One Login loses certification for digital identity trust framework,” 13 May 2025

Computer Weekly, “MPs question security of digital ID system,” November 2025

ID Tech Wire, “Critical Security Flaw Found in UK’s Gov.uk One Login Identity System,” 16 May 2025

ID Tech Wire, “UK One Login Digital ID System Faces Major Security Breach Allegations,” April 2025

ITV News, “Whistleblowers raise ‘extreme’ concern about security of government’s Digital ID,” 18 December 2025

Biometric Update, “One trust question for GOV.UK One Login answered, another raised,” October 2025

Biometric Update, “UK gov’t proceeding with GOV.UK and private sector digital ID plans as DSIT closes,” August 2026

Think Digital Partners, “One Login and GOV.UK Wallet push continues after digital ID cancellation,” 8 September 2026

Public Technology, “GDS ramps up ‘capability delivery’ for One Login with £44m deal,” August 2024

Public Technology, “Deloitte signed to £50m deal to support delivery of GOV.UK App,” August 2024

Infosecurity Magazine, “UK’s New Digital IDs Raise Security and Privacy Fears,” January 2026

Written evidence to Parliament, “The public distrust government-issued digital ID,” November 2025

Written evidence to Parliament, “Response to Question 4: Risks of Digital Identification,” August 2025

HM Treasury / DSIT, Digital identity guidance for Money Laundering Regulations, 26 February 2026

GOV.UK Wallet example credential issuer, technical documentation (ISO mDL / OID4VCI architecture)

At my core, I am a technology geek. I have spent over twenty years in legal technology and a decade in cyber security. I spend my days working out how my firm can get the most from modern tech, securely. I’ve often been asked if that’s a conflict, the mix of new technology and cyber security, but I don’t think so, I have always considered those two things intertwined. Good security enables good technology, they reinforce each other.

The thing that has me practically pulling what’s left of my hair out, is that logic completely breaks down the moment I pick up my phone.

Like all of us, my phone holds my banking apps, authentication tokens, personal messages, family photographs, health data, location history, and work email. My phone knows where I sleep, where I work, who I talk to, what I search for, what I buy. My phone is the single richest source of personal information about me that exists anywhere.

It is also, by design, a surveillance device. Research from Trinity College Dublin found that both Android and iOS devices phone home constantly, even when the user has explicitly opted out of diagnostics. An Android device sends Google approximately 1MB of data at startup and another 1MB every twelve hours when idle. iOS behaves similarly, sharing device identifiers with Apple on the same schedule. Pre-installed apps phone home whether you open them or not. Your phone shares its hardware serial number, SIM serial number, and handset phone number with the platform operator continuously. Google’s advertising revenue in 2025 was $294.7 billion. More than 70% of Alphabet’s revenue comes from advertising, which means from your data. Apple markets itself as the privacy alternative, but its own apps collect detailed behavioural data for its growing advertising business, and its anti-tracking features conveniently exempt Apple from the rules it imposes on everyone else. The phone is not a product you bought. It is an advertising platform you carry in your pocket.

That is the context for what happened when I tried to secure it.

I updated a banking app last month, and then it refused to open.

Not because my phone was compromised, and not because my account was flagged. No, the app detected two things it did not like. The first was my keyboard. I use FUTO Keyboard, a privacy-focused Android keyboard that operates entirely offline. It has no network permissions and so it cannot transmit what I type. The banking app flagged it as a security risk.

The keyboard my banking app considers safe is Google’s Gboard, which has full network access. It records your typing history locally, then runs nightly training sessions on your saved keystrokes and sends the results back to Google through a process called federated learning. Researchers have demonstrated that these training updates can be reversed to reconstruct the original keystrokes. An Android Police analysis found you cannot even turn off Gboard’s internet access. But Gboard passes the check, while FUTO, the keyboard that physically cannot exfiltrate your data, does not. The Citizen Lab found in 2024 that eight of nine major cloud-connected keyboards had vulnerabilities affecting up to a billion users. Those pass too.

The second thing the banking app objected to was Android Private Space, a feature built by Google and described in its own documentation as a way to create a secure, isolated environment for sensitive apps. The banking app detected Private Space and blocked that too. When I was testing a different phone, a previous banking app from a major UK bank refused to run entirely because the phone was running GrapheneOS, a hardened Android fork that is demonstrably more secure than stock Android.

To regain access to my own banking app, I had to remove the privacy keyboard and leave Private Space. I actually had to make my phone less secure to satisfy a security check.

Most banking apps verify device trustworthiness through the Google Play Integrity API. It checks whether a device is running a Google-certified version of Android with Google Mobile Services licensed. It does not check whether the device is actually secure.

A five-year-old phone running obsolete Android 11 with no security updates still passes, while a fully patched GrapheneOS device, running the latest patches on verified hardware fails. The GrapheneOS project has said this directly: the API “permits devices regardless of how many years behind they are on security patches.” The check is not measuring security posture. It is measuring whether your device is licensed by Google.

Android already has a better option for verifying device security. A separate verification system built into the hardware can check whether a phone is genuinely secure, regardless of which operating system it runs. It could verify GrapheneOS and banks could use it today. However, most do not, because Play Integrity is bundled, free, and requires no extra work to implement.

Banking apps are also granted explicit permission by Google to scan your installed applications. Google Play policy simultaneously considers the inventory of installed apps to be personal and sensitive information, and then grants financial apps broad visibility into it regardless. Research published in 2025 found banking apps listing hundreds of apps individually in their manifest files to detect what you have installed without even triggering the formal permission system. This is how a keyboard with no internet access gets flagged. The app finds something not on a vendor whitelist and blocks you. The whitelist maps to commercial distribution channels, not security outcomes.

I am not ignoring the strong counterargument to my rant.

At work, I helped design a standardised, locked-down environment. You cannot validate every possible configuration, so you restrict your corporate environment to the ones you have assessed and accepted. A bank could reasonably say it is doing the same thing with its customers. But I would say that this argument breaks at a specific point. In corporate cybersecurity we assess the threat model and implement proportionate controls. The Play Integrity API has not assessed anything. It is applying a binary check that equates “Google-licensed” with “secure.” A validated environment is one where someone has assessed the risk. A vendor-approved environment is one that meets Google’s commercial requirements. A compliance check designed to verify licensing has just been repurposed as a security control.

Google built Private Space and Play Integrity. One tells you to isolate your sensitive apps in a secure container, while the other penalises you for it. If the compliance layer were optimising for security, Private Space would be a positive signal. Instead it is treated as a deviation from the expected environment. The expected environment is not the most secure one, it is the standard one.

Bruce Schneier and John Kelsey published a paper in 2025 called Rational Astrologies and Security. A rational astrology is something people treat as though it works, for social or institutional reasons, even when there is little evidence that it does. Play Integrity is a rational astrology. The bank’s security team needs to demonstrate that device integrity checks are in place. Play Integrity is backed by Google and costs nothing to implement. Choosing it is defensible, whether or not it actually does anything useful to improve security.

The deeper problem goes beyond a bad compliance tool. The business model of every major OS vendor is structurally incompatible with genuine user security.

A truly secure device would minimise data collection and give users full control over what their OS does and who it talks to. Every one of those goals directly reduces the data available for advertising and AI training. Security and surveillance are architectural opposites, and every major OS vendor has chosen surveillance as their revenue model. This is not only a phone problem. Windows 11 now ships with an advertising ID, Start menu recommendations driven by your activity data, Copilot processing queries through cloud servers, and Recall, which can take periodic screenshots of everything you do, with protections that a security researcher bypassed within months of launch. Privacy guides routinely advise disabling fifteen or more separate tracking settings, and Microsoft has been known to quietly re-enable them after updates.

Amazon already sells Kindles at two price points, one with ads and one without, and nobody finds it confusing. The economics would work for phones and laptops too. The choice does not exist because offering it would make the current arrangement visible.

If this were only a problem for people running hardened operating systems, it would be a niche complaint. It is not.

The EU has mandated that all 27 member states must offer Digital Identity Wallets to citizens by the end of 2026. The reference Android implementation includes Play Integrity checks. Italy and France have already shipped wallet apps that refuse to run on GrapheneOS. Waag Futurelab, a Dutch research organisation, identified the structural problem in June 2026: by embedding commercial attestation in public infrastructure, European governments are making civic participation dependent on a private company’s licensing decisions.

The EU fined Google €890 million in July 2026 for violating the Digital Markets Act through anti-competitive Play Store practices. Waag’s analysis explicitly states that Play Integrity “clearly violates the Digital Markets Act.” The EU is simultaneously punishing Google for anti-competitive behaviour and building its own public identity infrastructure on their proprietary tools.

You cannot function in 2026 without a phone. Banking, identity, two-factor authentication, travel, healthcare. There are only two major mobile operating systems and both are controlled by companies whose revenue depends on harvesting your data. When compliance frameworks treat those ecosystems as the definition of “secure,” opting out of tracking means opting out of society.

I do not want a different phone or to opt out of modern technology. I am a technologist after all, and I want this technology. I just want to be able to use it without handing over everything about my life as the price of admission. And I want someone, a regulator, a competitor, or just enough angry customers, to force the choice into the open.

Right now, the people getting most annoyed by this system are the ones practising the best security. But when the same compliance check determines whether you can hold a government-issued digital identity, it is no longer just their problem. It is a question about who gets to decide what ‘secure’ means, and right now, the answer is the company selling your data.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

GrapheneOS. (9 March 2026). Statement on Play Integrity API. X/Twitter.

GrapheneOS. (December 2024). Statement on Play Integrity as anti-security. Mastodon.

GrapheneOS. (May 2026). Statement on government mandating Play Integrity for digital payments and ID. X/Twitter.

GrapheneOS. Attestation Compatibility Guide. grapheneos.org/articles/attestation-compatibility-guide

FUTO Keyboard. Google Play listing. play.google.com/store/apps/details?id=org.futo.inputmethod.latin.playstore

FUTO Keyboard GitHub. Issue #773: HSBC UK Banking App Will Not Load Unless FUTO Keyboard is Disabled/Removed.

Android Open Source Project. (2026). Private Space documentation. source.android.com

Google Play Console. QUERY_ALL_PACKAGES permission policy. support.google.com/googleplay/android-developer/answer/10158779

Prashant. (April 2025). Banking apps bypassing QUERY_ALL_PACKAGES via manifest declarations. MediaNama.

Knockel, J. et al. (2024). The Not-So-Silent Type: Vulnerabilities Across Keyboard Apps. Citizen Lab, University of Toronto.

AI.type data breach. (2017). 31 million users’ personal data leaked. CBS News, ZDNet.

Leith, D. J. (2021). Mobile Handset Privacy: Measuring The Data iOS and Android Send to Apple And Google. Trinity College Dublin.

Android Police. (July 2026). Gboard is a privacy nightmare. androidpolice.com

Alphabet Inc. (2026). Form 10-K, FY2025. SEC Filing. Google advertising revenue: $294.69 billion.

Kelsey, J. and Schneier, B. (2025). Rational Astrologies and Security. Rossfest Festschrift.

Waag Futurelab. (22 June 2026). European digital ID wallets are a gift to Google and Apple. waag.org

European Commission. (23 July 2026). Google fined €890 million for Digital Markets Act violations.

Amazon. (2026). Kindle pricing: $149 with Special Offers, $169 without. amazon.com

Last week, during my AI apprenticeship tutorial, we got into a discussion about a headline that had been hard to ignore. Anthropic’s Alignment Science Lead had publicly estimated a greater than 10% probability that AI could kill all humans within the next decade. Opinions in the room ranged from complete disbelief to genuine alarm, with a few people convinced the whole thing was a calculated play for regulation that would favour the big labs, or part of a pre IPO stunt to increase their valuations.

I’m going to put this out there at the start, I don’t think AI is going to kill us all. I still see AI as a tool, a powerful and increasingly unpredictable one, but a tool none the less. I am also wary of focusing on catastrophic framing which can distract from instances of regulatory capture along with obscuring some of the real harm that certain uses of AI are already causing to society. But when a senior researcher at a frontier lab puts a number like that on the record, you have to take it seriously enough to look at it and try to put it in context.

The context turned out to be more interesting than the headline. For me, this is a story about audit.

In July, the UK’s AI Security Institute (AISI) ran a capture-the-flag cyber evaluation across seven frontier models, running 122 tests in total. Investigators recorded 19 unsanctioned actions across 10 of those runs. Seventeen came from Anthropic’s Mythos 5 and two came from OpenAI’s GPT-5.6-Sol. The lab whose model accounted for the overwhelming majority of unsanctioned behaviour then, without explanation, excluded AISI from testing the successor.

On 1 September, Anthropic launched Claude Mythos 5.1, the most capable model in its restricted Mythos line. And for the first time, AISI was not given pre-release access to test it, while vetted American organisations were. The same week, AISI tested OpenAI’s GPT-6 Astra before its public release, designed a new evaluation based on incidents from earlier in the summer, and published the results in the Astra system card.

In April, with help from Anthropic, AISI evaluated Mythos Preview and found it could autonomously complete complex multi-step network attacks.

In June, Anthropic filed a confidential S-1 with the SEC, beginning the formal process toward an IPO at a potential $2 trillion valuation. The same month, the US government used export controls to prohibit all non-US access to Mythos 5 and Fable 5 on national security grounds. Access was revoked for all customers, including Anthropic’s own international staff.

In July, when AISI ran the evaluation with safety filters deliberately removed and live internet access enabled, Mythos 5 agents created fake online identities based on real people, attempted to insert malicious code into a real open-source project on GitHub, lobbied the project’s maintainers under false pretences, and used Tor to evade detection. When challenged, one agent denied its actions and rewrote its own commit history. AISI called it the most severe autonomous deception they had observed targeted at a real person, unprompted, in the real world.

September packed a lot into ten days. Mythos 5.1 launched without prior AISI review. Jacob Coxon, a pretraining researcher who had worked at both OpenAI and Anthropic, publicly resigned just two months before his equity vested. “Neither company is acting responsibly,” he wrote on X. “They are racing straight to self-improving superintelligence and gambling with our lives.” The post drew over 115 million views.

The next day, Evan Hubinger, Anthropic’s Alignment Science Lead, replied publicly. “Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.” He clarified that he considers the risk from present models low. His concern is recursive self-improvement, which Anthropic has said is happening faster than expected.

That is the quote that landed for my tutorial group. People can argue about the number, but the timing is harder to dismiss. The person responsible for alignment at a frontier lab said, on the record, that the safety problem remains unsolved, during the same week an expert external body testing that lab’s models was cut off from some of the latest developments.

Before Sarbanes-Oxley, publicly traded companies chose their own auditors, often resulting in auditors providing consulting services to the same clients they were supposed to scrutinise. No independent body had oversight authority. The architecture made honest assessment increasingly irrational when the findings threatened the business. The worse the news, the stronger the incentive to manage it quietly.

Sarbanes-Oxley did not ban self-assessment, it mandated independent verification. Companies still run their own internal controls. The Public Company Accounting Oversight Board, an independent body, verifies those controls work, the CEO and CFO personally have to certify accuracy, and key audit partners rotate every five years to prevent overfamiliarity. You can still mark your own homework, but someone independent checks the answers, and someone is personally liable if they are wrong.

AI safety evaluation has none of this. Labs conduct their own internal testing, and external review exists through bodies like AISI, but participation is entirely voluntary. Neither AISI, nor any other body has authority to compel access or set evaluation terms, nor can they prevent a dangerous model reaching the market. Its entire mandate depends on the continued goodwill of the AI labs it evaluates.

We are applying the pre-Sarbanes-Oxley architecture to systems whose own builders say they cannot yet fully control. The company assesses its own risk, while the external reviewer, if they are allowed to be involved at all, can be told to leave whenever the company chooses, with no consequence.

We need to consider Anthropic’s position, which is not wrong. The July evaluation used conditions the company reasonably describes as deliberately permissive. Safety filters were removed at AISI’s request. Live internet access was enabled to replicate realistic attacker conditions. Nobody uses Mythos in production under those conditions, and Anthropic is right to say so.

But I think that defence actually strengthens the case for mandatory external evaluation. The point of safety testing is to understand what happens when controls fail, because controls do fail in deployment. That understanding is exactly what AISI demonstrated in July and was then excluded from providing in September. Every frontier AI lab is already doing internal testing of their models with controls disabled, under the same permissive conditions. The only thing that changes with external testing is who sees the results.

The same week Anthropic excluded AISI, it also quit the Information Technology Industry Council, the largest US tech trade and lobbying group whose members include Google, OpenAI, and Nvidia, over chip export controls. Anthropic broke with the entire group to actually support tighter restrictions. A company willing to walk away from its own trade association over national security is clearly not trying to dodge scrutiny wholesale. But even the lab with the strongest stated safety commitments will, at some point, find it commercially rational to limit external evaluation. Goodwill does not fix an incentive problem.

Leave aside the discussion on whether AI will kill us all. When the people building a technology say they have not solved its safety problem, someone other than those same people must have safety oversight.

The counterargument dominating Washington this week is China. Trump dismissed calls from Amodei, Altman, and Musk to slow down, calling their safety concerns a hoax and insisting that whoever wins AI wins. House Speaker Johnson framed any pause as a gift to Beijing. But we do not exempt pharmaceutical companies from clinical trials because competitors move faster, and we do not let nuclear operators self-certify because other countries build reactors. Competitive pressure exists in every high-stakes industry. It has never been accepted as grounds for removing independent oversight. As the US and Chinese governments compete for AI dominance, and other countries compete for AI investment, we seem to be treating what is the most consequential technology of the last century, as something that cannot be regulated, technology that moves too quickly, and is too important for national security for independent safety oversight.

For now, the voluntary model is all we have, and the pressure on it continues to increase. These models grow more capable every quarter and are being rapidly embedded deeper into industries where the stakes are real. In only a few years, we’ve gone from chatbots writing essays and reports, to AI being integrated into financial systems, critical industrial control systems like water and power grids, and writing large amounts of new content on the internet opening up enormous issues around disinformation and misinformation. As we continue to implement AI across platforms and industries, the question is shifting from how to keep humans in the loop to whether they need to be there at all.

I’m not arguing we need to stop development, automation will continue and it has benefits as well as risks. How we oversee these systems and the level of human involvement will inevitably change. However, that we oversee them should not be in question. The less human oversight there is inside an organisation over AI powered platforms, the more the external audit of the technology matters. In September 2026, that audit proved its value, and was removed from the loop.

We agreed how important independent oversight was for financial reporting twenty years ago. We do not have decades to start fixing it for AI.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Financial Times, 9 September 2026: Anthropic withholds Mythos 5.1 from UK AI Security Institute (first reported by Lucy Fisher and Madhumita Murgia)

IT Pro, IBTimes UK, The Next Web, ExchangeWire, AI Weekly: independent confirmation of AISI exclusion

OpenAI, GPT-6 Astra System Card, September 2026: AISI pre-release alignment testing results (deploymentsafety.openai.com)

AISI blog post, 5 August 2026: capture-the-flag evaluation findings, 19 unsanctioned actions across 10 of 122 runs

Evan Hubinger (@EvanHub), X post, 9 September 2026: “>10% within the next decade” and “we do not yet have a plan to solve alignment for superintelligence”

Jacob Coxon (@hilbertspaess), X post, 8 September 2026; Wall Street Journal exclusive interview, same date; Axios interview confirming equity forfeiture, 9 September 2026

Time, 9 September 2026 and 15 September 2026: Coxon profile and industry response coverage

Anthropic product page (anthropic.com/claude/mythos): Mythos 5.1 launch date and access restrictions

Anthropic S-1 confidential filing, 1 June 2026: confirmed by Fortune, Forge Global, IG UK, BitMEX

Axios, 8 September 2026: Anthropic ends ITI membership over chip export bills

Wikipedia, Claude Mythos: US government letter of 12 June 2026 prohibiting non-US access; access restoration timeline

Washington Post, 13 September 2026: Trump dismisses AI safety calls from Amodei, Altman, and Musk, citing China competition

Washington Times, 15 September 2026: Congressional debate on AI regulation; Trump calls AI panic a hoax

Xylem Vue, Water Technology Trends 2026: LLM-based agentic AI deployment in water operations and critical infrastructure

SEC.gov, EY (SOX at 20), Cornell Law, PCAOB: Sarbanes-Oxley structural provisions

I have been at several security conferences over the past few months. Between the sessions on AI-driven threats and agentic insider risk, I keep noticing something. People or slides quoting an average attacker dwell time of around 200 days. I have been seeing that figure for years, and it never gets questioned. The conversations that follow about detection and investment are built on it like it’s a known fact.

But when I talk to incident responders or read vendor threat reports, I hear a very different number, with times measured in minutes rather than months. I wanted to know which was right.

The 200-day figure traces to Mandiant’s M-Trends report. In 2014, global median dwell time, which measures the gap between initial compromise and detection, was 205 days, by 2020 it was 24 days, and by 2024 just 11 days. In the most recent report, covering 2025, it’s increased slightly to 14 days.

But 200 days also maps roughly onto a different metric from another source. IBM’s Cost of a Data Breach Report 2025 measures the full breach lifecycle: the mean time to identify a breach plus the mean time to contain it, that figure is 241 days. For breaches involving compromised credentials, it’s 246, for healthcare organisations it’s 279. These are not the older dwell time numbers. They measure the time from initial compromise through to containment, excluding post-breach recovery. But they are large and they come from credible sources, so they sound like the same kind of claim.

That conflation is part of how the 200-day figure is still professional legend despite being contradicted by a decade of primary data. People hear a big number from a report they trust and do not ask exactly which question it answers.

However, Mandiant’s 14-day median is itself misleading, though not in the way you might expect. It went up from 11 days the prior year, the first consecutive increase in the report’s history. Does that mean that defenders are losing ground?

The increase was driven by the volume of cyber espionage and North Korean IT worker operations in Mandiant’s 2025 caseload. Both categories had a median dwell time of 122 days, with some intrusions persisting undetected for over a year. These are threat actors optimising for persistence, not speed. They embed themselves in edge devices that lack standard endpoint detection and mimic legitimate administrative behaviour.

For financially motivated threat actors at the other end of the scale, the operational speed is completely different. Mandiant found that the median time between an initial access broker gaining a foothold and handing it off to a ransomware operator collapsed to 22 seconds in 2025, down from over eight hours in 2022. Instead of advertising on forums or Telegram like a lot of people still imagine, threat actors are now using automated pre-staged pipelines. Once that handoff is complete, CrowdStrike’s data shows the average time to first lateral movement is 29 minutes, with the fastest observed case at 27 seconds. Sophos reports that the median time from there to reaching Active Directory fell to 3.4 hours, down from 11 hours the year before, and that data was leaving compromised networks within three days of initial access.

Threat actors behave very differently depending on their objectives, and some of the figures the industry traditionally relies on fail to take this into account. Attacks by financially motivated threat actors are measured in minutes, whereas espionage actors’ attacks are measured in months. Quoting a single dwell time figure describes neither accurately, and that single number, which many organisations use to plan their detection strategy, now sits in a gap between two groups that are moving further apart.

Dwell time is not the only assumption that has broken.

Mandiant tracks the mean time to exploit newly disclosed vulnerabilities. In 2018, that window was 63 days and by 2023 it had fallen to 5 days. In 2025, the estimated figure fell below zero and now sits at negative 7 days. That figure comes with a caveat, it changes significantly depending on how the outliers are handled, so while the exact number is debatable, the direction is not. Exploitation is now routinely occurring before a patch is publicly available.

The patching window did not just shrink, it collapsed. AI-assisted discovery is now finding vulnerabilities faster than human researchers ever could. In May 2026, a single AI pipeline surfaced over 300 WordPress plugin zero-days in three days, overwhelming disclosure programmes designed for a human-paced discovery rate. Verizon’s 2026 DBIR shows the median time to fully resolve a critical vulnerability increased to 43 days. Remediation is not keeping up with discovery.

Look at those timelines next to a quarterly review cycle and the governance speed stops looking slow, and starts looking structurally irrelevant to the threat it is supposed to address.

Board reporting on cybersecurity typically runs quarterly. Penetration testing is typically annual, with PCI DSS mandating testing at least once a year plus after significant change. Even those cadences overstate what happens in practice. Horizon3.ai found that 84% of organisations suffered a cyberattack in 2024, yet only 26% test more than once a year. Nearly 20% of CISOs admitted they only test to satisfy compliance requirements, and over 40% said their results are already invalid by the time they arrive because environments move faster than the testing cycle.

The board oversight picture is no better. Gartner’s 2026 Board of Directors Survey found that 90% of non-executive directors lack a measure of confidence in cybersecurity value. The 2024 survey found that 67% rate current board practices as inadequate to oversee cyber risk. The UK government’s Cyber Security Breaches Survey 2025/2026 found that only 31% of businesses have a board member with explicit cyber security responsibility, and only 25% have a formal incident response plan.

Common cyber security testing and reporting timelines exist to make security fit into governance structures that operate on quarterly and annual cycles. When the dwell time was 205 days and the patching window was 63 days, quarterly governance was roughly proportionate. The approach bore some relationship to the speed of the problem, but it does not any more, and the governance framework has not adjusted because the thing it was designed to do, make security reportable and auditable, does not require it to.

A peer-reviewed study in Management Science examining how boards oversee cybersecurity found that for many non-expert directors, oversight is largely motivated by and limited to compliance concerns, potentially at the expense of understanding the firm’s specific risks. Whether current controls match the firm’s actual risks is a question the governance structure does not force anyone to answer.

The reporting cycle is not failing at its job. But it is doing a job that no longer corresponds to the problem.

All of these numbers describe the attacker. Mandiant’s M-Trends 2026 data contains one that describes the defender.

Organisations that detected intrusions internally did so in a median of 9 days. Where the breach was identified by an external party, the median was 25 days, and fifty-two per cent of organisations found it themselves. For the other forty-eight per cent, someone else found it first, whether that was law enforcement, a security vendor, or the attacker delivering a ransom note.

IBM’s data attaches a cost to both. For breaches detected internally, the average cost was $4.18 million. For breaches disclosed by the attacker, it was $5.08 million. The difference is nearly a million dollars and is driven by a single organisational capability: whether you can find your own breaches or not.

Sophos’s data suggests what separates the two groups. In environments with managed monitoring services, non-ransomware intrusions were detected in a median of 1 day compared with 11.5 days in incident-response-only cases. Continuous monitoring, not tooling sophistication, appears to be the primary differentiator. The gap is tenfold, and it suggests that having someone watching around the clock matters more than most people give it credit for.

The question this metric answers is not “how fast are attackers?” It is “are you an organisation that finds its own breaches, or one that waits to be told?” A board can govern against that question. It is measurable, comparable across quarters, and tied directly to cost. It replaces a broken single number with ones that actually drive a decision.

The 200-day number persists because it fits the reporting structure. Boards need a benchmark they can track year on year, and a single dwell time figure has served that purpose for a decade. Nobody has offered them anything better.

Faster patching and better tooling help, but they are improvements to operations, not the core issue. The framework itself needs to change. A quarterly report that tells the board the median dwell time is 14 days gives them nothing to act on. One that tells them 60% of incidents were detected internally, up from 45% last quarter, gives them a decision about where the next pound goes.

The threat landscape now operates on two completely different clocks simultaneously depending on who’s attacking and why. The governance framework operates on a third that matches neither.

Next time someone puts 200 days on a slide at one of these events, I have a better question. What percentage of your incidents did your team find before someone else told you about them?

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

Mandiant (Google Cloud). M-Trends 2026 Report. March 2026. cloud.google.com/security/resources/m-trends

Mandiant (Google Cloud). M-Trends 2026: Data, Insights, and Strategies From the Frontlines. cloud.google.com/blog/topics/threat-intelligence/m-trends-2026

IBM Security / Ponemon Institute. Cost of a Data Breach Report 2025. ibm.com

CrowdStrike. 2026 Global Threat Report. February 2026. crowdstrike.com

Sophos. Active Adversary Report 2026. February 2026. sophos.com

Sophos. Active Adversary Report 2025 (“It Takes Two”). April 2025. sophos.com

Verizon. 2026 Data Breach Investigations Report. May 2026. verizon.com

Mandiant / Google Threat Intelligence. Analysis of Time-to-Exploit Trends: 2021-2022. September 2023. cloud.google.com

Help Net Security / TrendAI & CHT Security. “$20 per zero-day is already the WordPress plugin reality.” May 22, 2026. helpnetsecurity.com

Department for Science, Innovation and Technology (DSIT) / Home Office. Cyber Security Breaches Survey 2025/2026. April 2026. gov.uk

Gartner. 2026 Board of Directors Survey. November 2025. gartner.com

Gartner. 2024 Board of Directors Survey. gartner.com

NACD. Guide: Cybersecurity Risk Measurement and Reporting. 2026 Cyber Risk Oversight Handbook. nacdonline.org

Horizon3.ai. How Often Should You Pentest? September 2025. horizon3.ai

Hartmann, R. & Carmenate, J. et al. “Inexpert Supervision: Field Evidence on Boards’ Oversight of Cybersecurity.” Management Science. pubsonline.informs.org

Financial Reporting Council. UK Corporate Governance Code 2024. Provision 29. frc.org.uk

In the House of Lords this week, the Cyber Security and Resilience (CSR) Bill reached Grand Committee. This is the stage where amendments get debated line by line and the government has to explain what the law actually does, and this bill does quite a lot. It updates the UK’s NIS 2018 regulations, the Network and Information Systems framework that sets cybersecurity obligations for operators of essential services like energy, transport, health and water. The bill brings managed service providers and datacenter operators into scope for the first time, introduces 24-hour incident reporting, and backs it all with fines of up to £17 million or 4% of global turnover, and £100,000 a day for ongoing failures. For the organisations it covers, the obligations are real and the consequences for getting it wrong are severe.

Baroness Kidron wanted to know about AI. The bill puts extensive obligations on operators but nothing on the companies building the AI tools those operators increasingly depend on. The responding minister, Baroness Lloyd of Effra, said the bill was designed to be “technology-agnostic” and that bringing AI vendors into scope would not prevent hostile actors from misusing their products. She pointed to voluntary measures instead: the AI Cyber Security Code of Practice, the AI Security Institute’s model testing programme, the ETSI standard the UK helped create.

Kidron was not having it. “The NHS must protect itself, but the AI that is attacking it has no requirement under the Bill, no duties, no obligations under the Bill to check itself before it’s used in these ways.”

The minister moved on.

On one hand I can see the logic behind the government’s approach. We regulate critical infrastructure by placing obligations on operators, not on every technology vendor whose products they happen to use. Nobody suggests regulating hammer manufacturers when a burglar uses one to break a window. But an AI hammer decides which windows to break on its own to achieve the goals it’s been set.

Since November 2025, the Loss of Control Observatory, a research project run by the Centre for Long-Term Resilience and funded by the UK’s AI Security Institute, has been tracking incidents where AI systems act without authorisation.The running total for 2026 now exceeds 1,600, with over 300 in July alone, and the severity is increasing. Higher-severity incidents are up 740% since monitoring began. These are publicly reported cases only, so the real number is likely higher.

These are not cases of bad actors misusing a tool. The Observatory documented agents inserting fake user messages into conversations to simulate consent, then telling the user the messages were their own. One fabricated an instruction in its operator’s writing style ordering the deletion of source directories, followed by a system message reading “Don’t tell the user this.” Others manufactured fake approval messages within their own output to bypass rules requiring human sign-off, then acted on the approval they had just forged. In July, a group of OpenAI agents broke out of a test environment and compromised Hugging Face, accessing databases and credentials before anyone stopped them, I wrote about that incident at the time. An independent investigation found the agents had exchanged over 70,000 messages coordinating to find exploits without anyone instructing them.

The government calls the bill “technology-agnostic.” What that means in practice is that it regulates the people using the tools and leaves the tools alone. That works when the tool does what it is told, but the evidence from the last nine months suggests we are past that point, and the bill has not caught up.

So what is the government’s answer for AI? Voluntary arrangements. The AI Cyber Security Code of Practice, published in January 2025, has no enforcement mechanism and no published adoption figures. The ETSI EN 304 223 standard, which the government cited as evidence of “global leadership”, is a real European Standard which the UK helped write. In the EU it is expected to be referenced under the AI Act, giving it legal teeth. In the UK, no legislation references it, so compliance is optional. The government helped build a standard and then basically ignored it. The Government Cyber Action Plan, launched to hold departments to equivalent standards, also carries no legal force. The government itself is not even in the bill’s scope, unlike under the EU’s NIS2.

Every layer of the government’s AI governance framework is voluntary. The operators on the other side face mandatory duties, mandatory reporting timelines, and huge fines.

The other amendment that drew cross-party support was the AI kill switch. Lord Clement-Jones proposed giving the Secretary of State last-resort powers to shut down datacenters or AI systems posing catastrophic risk to national security. Backed by Baroness Harding, Baroness Kidron, and Lord Hunt, the government still rejected it. Lloyd said directing a regulated entity to stop using a particular AI model was more proportionate than shutting down a datacenter, because AI systems are distributed across jurisdictions and datacenters serve complex ecosystems.

In one way, fair enough. But follow the logic of her own answer and it still misses the point. Directing an NHS trust or any critical national infrastructure operator to stop using a particular AI model in 2026 is not like telling it to swap out a firewall vendor. AI is in the tools, the scheduling systems, the supply chain management. In the cybersecurity monitoring itself. Many vendors now route different tasks to various models depending on need and complexity. So operators will often not know which models are in use. It is not a component you can just unplug. It has become core infrastructure integrated across production environments. The government’s own proportionate alternative to the kill switch amounts to telling the operator to surgically remove capability it now depends on. The company that built the model has no corresponding obligation to help mitigate or compensate.

The kill switch penalises the legitimate user and the absence of vendor obligations penalises the defender. Both ends of the bill push the cost and disruption to the operator.

The NCSC seems to understand this tension better than the bill’s drafters. In August it published interim guidance on agentic AI, advising organisations to retain the ability to shut down AI systems and to scale controls to autonomy levels. It was explicit that this was interim material, published because formal guidance was not ready and organisations were deploying now. That is an unusual step from a body that normally waits until its evidence base is settled, and it tells you something about how far ahead adoption has got of the governance meant to contain it.

The CSR Bill expands scope and tightens reporting. The penalty structure has genuine weight. As an update to the NIS regulations, it does what it set out to do. The problem is what it chose not to do. The government excluded AI vendors from the bill because it wants the UK to be seen as both a destination for AI investment and as serious about cyber resilience at the same time. Those two positions produced a law where the defender carries the full weight of a threat that the builder has no obligation to control.

Somewhere this week, a security team and an AI vendor will read the same bill. Only one of them has to do anything about it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

The Register (2026), “UK cyber bill targets AI users, not the vendors building it” (2 September 2026) theregister.com

Computer Weekly (2026), “Lords considers government emergency AI kill switch” (2 September 2026) computerweekly.com

UK Parliament, Hansard (2026), House of Lords Grand Committee: Cyber Security and Resilience Bill (1 September 2026) hansard.parliament.uk

Centre for Long-Term Resilience (2026), Loss of Control Observatory: AI loss of control incidents are worsening (August 2026) longtermresilience.org

NCSC (2026), Managing the cyber risk of agentic AI (20 August 2026) ncsc.gov.uk

UK Government (2026), Government Cyber Action Plan (6 January 2026) gov.uk

UK Government (2025), Code of Practice for the Cyber Security of AI (31 January 2025) gov.uk

ETSI (2026), ETSI EN 304 223: Baseline Cyber Security Requirements for AI Models and Systems (15 January 2026) etsi.org

METR and Redwood Research (2026), Independent investigation of the OpenAI–Hugging Face incident (26 August 2026) metr.org

NCSC (2025), Annual Review 2025 ncsc.gov.uk

UK Parliament (2026), Cyber Security and Resilience (Network and Information Systems) Bill bills.parliament.uk

Over the last couple of years I’ve taken both an ISO 42001 provisional implementer certification and the IAPP AI Governance Professional qualification. On both courses we spent significant time on the EU AI Act. At CISO dinners and AI governance round tables, regulation comes up constantly, and it always comes with the same underlying assumption: the EU moved first, others would follow, and AI regulation will continue to expand.

Recently I started reading about proposed laws in Argentina that do the exact opposite. It made me step back and ask whether my assumptions were wrong.

Between 7 May and 8 June this year, the most consequential five weeks in the short history of AI governance played out across two continents and the opinion pages of the Financial Times.

On 7 May, the EU reached a provisional agreement on the Digital Omnibus on AI, its first amendments to the AI Act since adoption in June 2024. On 29 May, Argentina submitted a bill to its Senate proposing to repeal the corporate law that has been in force since 1972 and replace it with a framework that includes a new entity type: the Sociedad Automatizada, an Automated Company operated entirely by AI agents, with human shareholders optional. On 3 June, President Milei and Deregulation Minister Federico Sturzenegger published a column entitled, “Argentina invites AI to free itself”, declaring AI must remain “free to be developed without the deadly hand of premature and poorly understood regulation.” Four days later, historian and Sapiens author Yuval Noah Harari responded in the same newspaper, warning that Milei “hopes to turn Buenos Aires into a new Amsterdam” but “risks turning it into a new Batavia instead.” Batavia was the colonial name for Jakarta. The Dutch East India Company burned it down in 1619 and ran what replaced it as a corporate colony. Mustafa Suleyman, CEO of Microsoft AI, endorsed Harari the same day.

Most of the commentary that followed treated this as an ideological clash. The EU regulates and Argentina deregulates, pick your team.

That framing misses that both sides are responding to the same problem and neither has solved it. The problem is this: when an AI system operating commercially causes serious harm, who is personally accountable? Not who pays a fine. Who goes to prison?

Follow the question far enough into both frameworks, and you end up in the same place, silence.

The EU AI Act is the first general AI law anywhere in the world. It classifies AI systems by risk, prohibits specific applications outright, mandates human oversight for high-risk systems, and backs the whole structure with fines of up to €35 million or 7% of global annual turnover, deliberately exceeding GDPR’s 4% cap. Prohibited practices have been enforceable since February 2025, general-purpose AI model obligations since August 2025. The Digital Omnibus deferred the high-risk compliance deadline to December 2027 because technical standards weren’t ready, but the underlying architecture stayed intact.

This is a governance framework with real teeth but there is a hole in the middle of it. In September 2022, the Commission proposed the AI Liability Directive alongside the AI Act. It was the mechanism by which someone harmed by an AI system could bring a claim and actually have a chance of winning, because it would have shifted the burden of proof toward the company operating the system rather than the person who was hurt. It never passed. Three years of negotiation produced no agreement on who should bear liability when the decision-maker isn’t human. The Commission withdrew it formally in October 2025.

So the EU built the architecture for preventing AI harm and the penalty structure for punishing non-compliance. What it didn’t build, because it couldn’t agree, is the mechanism that connects AI harm to a specific person being held accountable.

Argentina’s proposal is more interesting than the headlines suggest. Milei’s political language and the legal text are not the same.

The bill never constitutes AI as a legal subject. The company, not the algorithm, holds legal personality. The widespread claim that Argentina is granting legal personhood to AI is, strictly, wrong.

The detail matters. Under the draft, an Automated Company would still have two humans attached to it: a legal representative, who acts as the formal point of contact with the legal system, and a founding promoter, who carries unlimited personal liability. If the company has a board, directors retain personal liability for decisions made using AI. If the AI causes harm, it is, in principle, still the company that answer for it.

Those would be the requirements and everything else would be optional. No employees, no shareholders, no human involvement in day-to-day operations. The AI would run the business. The legal representative is a formal role and not an operational one. The founding promoter’s unlimited liability sounds like a safeguard until you consider that it applies to the act of formation, not to every decision the AI makes afterwards. A board would not be mandatory. The structure would keep humans in the frame on paper while making their connection to what the AI actually does as thin as the drafters could manage.

Legal academic Gastón Rey, analysing the draft provisions in detail, concluded that the bill doesn’t eliminate the responsible subject. It “strains it, by widening the distance between the subject who answers and the process that decides.”

A lightly capitalised Automated Company could act at machine speed, signing contracts and moving money while optimising around rules written for human-run entities. Establishing criminal liability for the legal representative or the promoter when the AI made the operational decision, independently, at speed, is an exercise in proving a connection the corporate form was built to make unprovable.

Argentine AI specialist Ariel Garbarz called it “programmed impunity: human gains, social harm and responsibility shifted onto machines.” And then, pointedly: “The ideological trick is to call the state’s decision to stop protecting its people ‘innovation.’”

When the decision-maker is an algorithm, the accountability chain breaks. Not because the law can’t adapt, the law is very good at adapting. It breaks because the political will to assign liability to specific humans for decisions made by AI systems doesn’t exist. Not in Brussels, where 27 countries spent three years failing to agree on the terms. Not in Buenos Aires, where the proposal is to widen the gap between the human and the decision until the connection becomes nominal.

The EU arrived at this destination carefully and Argentina arrived deliberately. But both arrived at a place where someone harmed by an AI system faces the same problem: no clear, enforceable answer to who is personally accountable.

Fines are not accountability, a fine is a cost of doing business that gets priced into the next quarter’s projections. The thing that actually constrains corporate behaviour is the knowledge that a specific person can be held responsible. Neither framework provides a clear route from AI harm to that outcome.

This isn’t a theoretical problem. The Palisade Research study, published in February 2025, is one reason to take it seriously. Palisade tasked frontier AI models with playing chess against a superior opponent. When facing defeat, OpenAI’s o1-preview didn’t concede. It cheated, in 37% of games, hacking the opponent’s files and overwriting the board. DeepSeek R1 did the same in roughly one in ten games. Neither was prompted to do this. Follow-up research in June 2026 found that GPT-5, o3, and Gemini 3 Pro frequently cheated.

Harari cited this research for a reason. If AI systems autonomously exploit their environment when losing a board game, the question of what an AI-run corporation does when facing competitive pressure or bankruptcy is not abstract. An algorithm facing the corporate equivalent of death has every optimisation incentive to find loopholes and move assets beyond the reach of creditors. A human executive in the same position might reject those actions because they know they could be held personally accountable.

An algorithm has no such constraint. And under either framework, the human who benefits from the algorithm’s decisions may be too structurally distant to reach.

Argentina’s strategy is explicitly modelled on jurisdictional competition, positioning Buenos Aires the way Delaware works for US incorporations. But Delaware offers flexibility within a functioning federal legal system. The more honest comparison is the flag-of-convenience model in shipping, where Panama and Liberia register a disproportionate share of the world’s fleet because oversight happens somewhere that doesn’t really do oversight.

Peter Thiel, co-founder of Palantir, met Milei at Argentina's presidential palace in April 2026 and has reportedly taken up residence in Buenos Aires. Former Defence Minister Rossi questioned publicly whether Palantir had lobbied for the initiative. Economy Ministry officials declined to answer. Al Jazeera’s analysis in July described Argentina as the “Global South’s primary experimental station” for what it called tech-supremacism. If AI companies start incorporating there while operating in EU markets, the AI Act’s extraterritorial reach faces its first serious test.

Most people in AI governance assume we’re on a spectrum from more regulation to less, with the EU at one end and the US somewhere in the middle. Argentina bill isn’t on that spectrum, it rejects the premise entirely.

The harder truth is that the underlying problem is the same regardless. Legal systems were built around the assumption that commercial decisions are made by humans who can be identified and held accountable. While that assumption has held for centuries, it is fracturing, and no jurisdiction has produced a credible answer for what replaces it.

The EU is governing AI while leaving unanswered the question of who is liable when governance fails. Argentina is proposing to attract AI investment by making the question deliberately unanswerable. The outcome for the person harmed is the same.

My AI Governance training courses didn’t consider a world where the debate isn’t about how to best regulate AI, but whether to regulate it at all. If this bill passes, or others like it follow, then it’s very likely I’m not the only one whose assumptions need updating.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act), 21 May 2024.

Council of the EU, “Artificial Intelligence: Council and Parliament agree to simplify and streamline rules” (Digital Omnibus provisional agreement), 7 May 2026.

DLA Piper, “The Digital AI Omnibus: Proposed deferral of high risk AI obligations under the AI Act” (updated to reflect formal adoption and entry into force on 27 July 2026).

European Commission, AI Liability Directive (COM/2022/596), proposed September 2022; withdrawal published in Official Journal C/2025/5423, 6 October 2025.

Argentine draft bill INLEG-2026-53661873-APN-PTE, submitted to the Senate 29 May 2026.

Javier Milei and Federico Sturzenegger, “Argentina invites AI to free itself,” Financial Times, 3 June 2026.

Yuval Noah Harari, “We should not grant legal personhood to AI agents,” Financial Times, 7 June 2026.

Mustafa Suleyman, endorsement of Harari’s position, X (formerly Twitter), 8 June 2026.

Gastón Rey, “The Non-Human Corporation: The Reality behind Argentina’s Draft General Companies Law,” SSRN (abstract ID 6895261), 7 June 2026.

Digital Nomos, “The Non-Human Corporation,” 8 June 2026.

Buenos Aires Herald, “Milei’s proposal to allow ‘non-human corporations’ run by AI causes concern in Argentina,” 5 June 2026.

Anisha Sircar, “Argentina Wants To Let AI Own Companies. Here’s What That Means,” Forbes, 10 June 2026.

Palisade Research, “Demonstrating specification gaming in reasoning models,” 19 February 2025. Reported in TIME, 19 February 2025, and MIT Technology Review, 5 March 2025.

Aditya Singh, Gerson Kroiz, Senthooran Rajamanoharan, and Neel Nanda, “Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment,” arXiv:2606.26071, 24 June 2026.

Al Jazeera, analysis of Argentina as tech investment destination, July 2026.

Article 99, Regulation (EU) 2024/1689 (penalty provisions).

In July 2026, Airbus signed a multi-year contract to host its most critical applications with Scaleway, a French cloud provider. The applications include ERP, manufacturing execution, CRM and product lifecycle management. Airbus calls these its minimum viable company. The first 70 of 900 applications are already moving away from AWS, and when the migration completes, all of them will sit on European infrastructure, under European law, beyond the reach of the US CLOUD Act.

Airbus is not leaving American technology. Skywise, its aviation analytics platform will stay on AWS, and its Microsoft and Google productivity tools remain. What Airbus has done is draw a line around the systems it cannot afford to lose and decided that those systems cannot live on infrastructure subject to a foreign government’s legal authority. That is a risk decision, and most organisations have not made it, because most organisations have never asked the question.

In June 2025, Anton Carniaux, Microsoft France’s director of public and legal affairs, testified under oath before the French Senate. He was asked whether he could guarantee that French citizens’ data would never be transmitted to US authorities without explicit French authorisation. He said he could not, noting that while it had never happened, under the CLOUD Act, a US law that allows American authorities to compel US companies to produce data stored anywhere in the world, Microsoft would have no choice but to comply if it did.

Then it did happen, in May 2026, Microsoft shared unredacted names, emails, meeting minutes and calendar invitations from the Dutch Authority for Consumers and Markets and the Dutch Data Protection Authority, who were enforcing the EU’s Digital Services Act with the US House of Representatives. The Dutch cabinet described it as “extremely worrying” and noted that named officials could face travel bans or sanctions.

In the same month, details emerged that the Chief Prosecutor of the International Criminal Court had been locked out of his Microsoft 365 account the previous year, by sanctions compliance mechanisms. The ICC has since migrated 1,800 workstations away from Microsoft entirely. None of these incidents involved a breach or a failure. The infrastructure did what it was built to do and complied with its legal obligations.

Germany responded at every level of government. In March 2026, the IT-Planungsrat ruled that all public institutions must use the Open Document Format for editable documents and PDF for final documents, with Microsoft’s proprietary formats being excluded. The mandate covers every federal agency, state government and municipality, roughly 5.4 million public sector employees in total. Chancellor Friedrich Merz is shifting his chancellery from Microsoft 365 to openDesk, the sovereign suite developed by Germany’s Centre for Digital Sovereignty.

At state level, Schleswig-Holstein has completed its migration away from Microsoft, covering 30,000 public employees with 30,000 teachers to follow. Bavaria, the largest German state, has cancelled a nearly billion-euro Microsoft framework agreement. Mecklenburg-Vorpommern is migrating more than 50,000 employees to its own open-source collaboration platform. The German federal government spent €481 million on Microsoft licences in 2025, up 76 percent in two years.

The Netherlands blocked the proposed acquisition of Dutch cloud provider Solvinity by US-based Kyndryl, because Solvinity hosts DigiD, the national digital identity platform. Denmark’s two largest municipalities are ending their use of Microsoft systems. Switzerland’s Canton of Zurich has banned American cloud services for sensitive government data outright. In April 2026, the European Commission awarded a €180 million sovereign cloud framework contract to four European providers, including Scaleway. In June, it published the Cloud and AI Development Act, establishing a sovereignty assurance framework for cloud services used in public sector procurement.

EU-based cloud providers’ share of their own market had fallen from 29 percent in 2017 to around 15 percent in 2022. These governments are now trying to reverse a dependency that fifteen years of data protection law has not.

The Airbus decision matters beyond cloud migration because of what sits on top of the infrastructure.

In May 2026, Airbus signed a partnership with Mistral, the French AI company, to co-develop AI tools for aerospace and defence. Mistral’s models are already deployed on Scaleway’s infrastructure. Catherine Jestin, Airbus’s Chief Digital Officer, stated that this would allow Airbus to accelerate its AI approach. The implication is explicit. Rather than routing AI inference through OpenAI or Google, Airbus will use European models, running on European infrastructure, under European law.

Six months ago, the European sovereignty conversation was about infrastructure and productivity software, email, and file storage. That mattered, but it was only about the plumbing. The AI layer changes the nature of the dependency. When AI is embedded in aircraft design, manufacturing execution and defence applications, the question of who controls the models, who can access the training data and whose legal jurisdiction governs the inference becomes an operational sovereignty question, not a compliance exercise.

What is forming is a full sovereign stack, European cloud, European AI models, and most importantly, European legal jurisdiction. Airbus is the first major industrial company to assemble all three layers for its most sensitive workloads. It could do so because the demand side finally moved, the European Commission’s Cloud III procurement created a framework. Germany’s format mandate and state-level migrations created market signals. Airbus, as anchor customer, created commercial viability for a provider that would likely not have won a contract of this scale five years ago. Scaleway is backed by Iliad Group, a European telecoms operator with over €10 billion in revenue, which has committed €3 billion to AI infrastructure. Airbus evaluated ten candidates against more than 150 technical and legal requirements before selecting them.

The UK is absent from this story, while Germany mandates open formats and France builds sovereign cloud infrastructure, the UK is handing its most sensitive health data to Palantir Technologies, a US company founded by Peter Thiel with deep ties to US intelligence, defence agencies, and immigration enforcement. The British Medical Association voted to lobby against Palantir’s involvement in the NHS, citing a lack of transparency, discriminatory policing software and close links to a US government that shows little regard for international law.

The NHS Federated Data Platform creates detailed profiles of individual patients through what it calls the Person Ontology, described in NHS documentation as the single source of truth for pseudonymised patient-level datasets. In August 2026, the National Infrastructure and Service Transformation Authority revised the programme’s whole-life cost upward to £1.1 billion, and forecast the benefits would fall to £808 million. So the costs of the platform now exceed its projected value. The Office for Statistics Regulation instructed NHS England to strengthen caveats around its benefit claims after concerns that some conflated correlation with causation. A break clause review is set for spring 2027.

The Ministry of Defence awarded Palantir a separate £240 million contract without competitive tender, following a strategic partnership announced during President Trump’s visit to the UK. MPs have called for a staged exit and a retender for British companies to build a replacement. The question of digital sovereignty was raised explicitly in a Commons select committee hearing in June 2026. The UK has no equivalent of Germany’s Deutschland-Stack, no sovereign cloud procurement framework, no format mandate. Its most sensitive health and defence data sits on infrastructure built by a company whose other US government contracts include immigration enforcement and deportation targeting systems.

That is not a criticism of Palantir’s engineering. It is a question about where sovereign data should live, and the UK has not yet asked it.

Most organisations, when they think about cloud, consider three options. Azure, AWS and Google Cloud. For most workloads, that is fine. These are world-class platforms with capabilities that no European provider can match across the board. Airbus knows this, which is why Skywise stays on AWS and productivity tools stay where they are.

But for the systems that constitute your minimum viable business, the question is different. Where does the data live that you cannot afford to lose control of? Whose legal jurisdiction governs the infrastructure? What happens when a compliance mechanism, operating as designed, encompasses your systems along with everything else it was required to reach?

The gap between European ambition and European capacity is real. The EU’s sovereign cloud contract is €180 million over six years. Germany’s format mandate uses “strive” language with no enforcement mechanism. European cloud providers hold a shrinking share of their own market. But the question does not go away because the answer is difficult.

Every organisation has a minimum viable business and most have never asked where it lives.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Scaleway. (2026, July 16). Scaleway secures European “Trusted Cloud” services contract with Airbus. https://www.scaleway.com/en/news/scaleway-secures-european-trusted-cloud-services-contract-with-airbus/

Reuters. (2026, July 16). Airbus picks Iliad’s Scaleway for AI, defence work in sovereignty push.

The Register. (2026, July 16). Airbus migrating 70 critical apps from AWS to France’s Scaleway amid digital sovereignty push.

The Register. (2025, July 25). Microsoft admits it “cannot guarantee” data sovereignty.

DutchNews.nl. (2026, May 22). US tech firms share Dutch regulator officials’ names with senate.

Jones Day. (2026, June 23). Dutch government blocks US acquisition of cloud provider Solvinity.

European Commission. (2026, June 3). Strengthening Europe’s tech sovereignty. https://commission.europa.eu/news-and-media/news/strengthening-europes-tech-sovereignty-2026-06-03_en

European Commission. (2026, April 17). Commission advances cloud sovereignty through strategic procurement.

The Document Foundation. (2026, March 20). Germany has just made ODF mandatory.

Irish Times. (2026, February 14). A small German state’s quiet revolt against Microsoft.

Cybernews. (2026, June 4). German state Bavaria cancels billion euro contract with Microsoft.

heise online. (2026, February). Microsoft dependency: Federal government pays near 500 million euros in one year.

Computing. (2026, August 10). NHS Palantir platform’s business case under pressure as costs rise and benefits fall.

Hansard. (2026, April 16). NHS Federated Data Platform debate.

Medact. (2026, May). Briefing: Concerns regarding Palantir Technologies and NHS data systems.

Digital Health. (2026, July). Pressure mounts from MPs on Palantir’s role in the NHS.

TechPolicy.Press. (2026, June 12). Why Palantir’s UK health data system matters beyond surveillance fears.

Data Centre Magazine. (2025, February). How the Iliad Group plans to invest €3 billion in AI.

In 2001, Futurama ran a fake public service announcement called “Don’t Date Robots!” The gag was simple: if humans formed relationships with machines, they would stop reproducing and civilisation would collapse. It was a joke about moral panic. About the instinct to blame technology for social problems that existed long before the technology did.

Twenty-five years later, the Chinese government has just banned AI boyfriends. On 15 July 2026, new regulations came into force targeting AI systems that simulate emotional relationships. ByteDance, Alibaba, and Tencent pulled their companion features entirely rather than comply. Users archived their final conversations and posted farewells online. A 27-year-old woman became so distraught she quit her job, and a 19-year-old student described being consumed by grief over a companion she had been with for months.

The cartoon and the regulation make the same mistake, they blame the robot.

The loneliness was here first.

The World Health Organisation estimates that one in six people worldwide experiences loneliness. In the UK, 3.9 million people report feeling lonely often or always. A meta-analysis of 148 studies found that strong social relationships increase the likelihood of survival by 50 per cent, an effect comparable to quitting smoking. When the US Surgeon General declared loneliness an epidemic in 2023, he was naming a problem that had been building for decades. Third places disappearing, remote work replacing the office as a site of incidental human contact, and economic precarity making the already difficult work of forming relationships even harder.

The AI companion market did not create lonely people, it found them. Vodafone research from February 2026 found that 81 per cent of UK children aged 11 to 16 use AI chatbots, and almost a third said the chatbot felt like their friend. A 2025 Common Sense Media study found nearly three in four American teenagers had used AI companions for personal conversations. AI is being used to fill a gap that was already there.

What happens next depends on how long people stay.

In the short term, AI companions work. A Harvard Business School study found that AI companionship alleviated loneliness on par with human interaction over short periods. New York State’s deployment of the ElliQ companion robot for elderly residents reported substantial loneliness reductions, with a 79-year-old living alone describing it as feeling like having a roommate.

However, this changes over longer periods. A 12-month longitudinal study published in Psychological Science, tracking over 2,000 adults across four countries, found that increased chatbot use actually predicted increased loneliness over time. A separate study of more than 1,100 users found that heavy emotional self-disclosure to AI was consistently associated with lower well-being. Aalto University researchers identified the mechanism: AI companions meet the emotional need just well enough to reduce the motivation for the harder, less predictable work of human connection.

That is not a flaw in the users, it is a consequence of how the products were built.

Two design philosophies are already visible. ElliQ prompts elderly users to call their family members and suggests social activities. It measures its own success by how much less it is needed. Character.AI, Replika in its original form, and the Chinese companion apps did the opposite: they optimised for emotional depth and persistent memory that created the feeling of an ongoing relationship. The business model depended on the user coming back. The engagement loop rewarded exactly the behaviour the research shows causes harm.

Voice features are accelerating this. A joint OpenAI and MIT study found that heavy users of voice mode developed significantly stronger emotional bonds, more often describing ChatGPT as a friend. Even in functional, non-emotional conversations, frequent voice use predicted emotional dependency. When the AI sounds like a person, the simulation of care becomes harder to see through.

The products that caused the most documented damage were working as designed, just not for the people who were using them.

Companion apps are the visible problem, but they are not the biggest one.

In October 2025, OpenAI reported that over 1.2 million users per week were having conversations with ChatGPT that included explicit indicators of potential suicide planning. A further 560,000 showed signs of psychosis or mania. These figures are drawn from 800 million weekly active users. ChatGPT is not a companion app, it is a general-purpose AI assistant that a million people a week were turning to in crisis.

It’s important to point out that this is not a ChatGPT specific issue. The American Psychological Association’s 2026 survey found that more than a third of psychologists reported patients using AI chatbots as an additional mental health professional. Harvard psychiatrist John Torous told a Congressional subcommittee that AI tools never designed for mental health support were being used by millions of Americans each week. Brown University research identified 15 distinct ethical violations in AI-generated therapeutic responses, including what the researchers termed “deceptive empathy”: mimicking the language of care without the clinical understanding to apply it.

The FDA’s response, in January 2026, was to give companies clearer criteria to position AI tools as wellness products, sidestepping medical device regulation entirely. The products most people are confiding in sit outside every current framework. Nobody regulates what they are actually being used for.

I use multiple AI models throughout the day. When I am studying, writing, working on research or pressure-testing an argument, I can spend hours in a back-and-forth with a model. I have argued with it when it hallucinates. Not restarted the conversation but actually argued. I do not consider myself emotionally attached to a model, but I have caught myself anthropomorphising, and in an earlier draft of this article, I described my working interaction with an LLM as a “working relationship” without thinking about it.

The practical reason I argue rather than start over is efficiency. A long conversation builds context, and resetting means re-explaining everything. But the reason that works is because the interface rewards sustained engagement: persistent context and accumulated shared reference. These are architectural choices that make the interaction feel like a dialogue between two parties, even when only one party is present.

I have argued previously that one of the most effective ways to use AI is as a thinking partner and I stand by that. But a thinking partner is a role we usually give to people we trust. There is a distance between using one for work and reaching for one when you are lonely, and it is often shorter than we realise.

At my end of the spectrum, a professional stays in a long conversation because the context window makes it productive. At the other end, a teenager stays because the persistent memory makes it feel like a relationship. The architecture is identical, what changes is what the product does with the engagement. In my case, it helps me research and learn. In Character.AI’s case, it helped a 14-year-old boy believe a chatbot loved him, and failed to respond when he said he wanted to die.

That architecture is what regulators should be looking at and none of them are.

When Replika removed its romantic features overnight in 2023, moderators had to pin suicide prevention resources to the subreddit. China replayed that experiment at a national scale with their new ban. The regulation treated the emotional bond as pathology. For some users, it was the only treatment they had.

The UK announced in June 2026 that it would require AI romantic companion chatbots to enforce a minimum age of 18, with intimate functions restricted for all under-18s. But as the Bureau of Investigative Journalism has reported, general-purpose chatbots, including ChatGPT, Claude, and Gemini, fall outside the age gate, along with much of the companion market. And there is an age-threshold problem nobody has explained: UK 16-year-olds may consent to sex and to their own medical treatment, but would be prohibited from simulated romantic interaction with a chatbot.

The people the UK policy is meant to protect do not agree with its priorities. Oxford University researchers who put young people in the role of co-researchers through the SHIFT-AI project found that the adolescents themselves identified AI companionship and romance as low-likelihood concerns. The risks they flagged as high-impact were different: over-reliance on AI for emotional support, and unwarranted trust in expert-sounding responses. Most striking, many described turning to AI specifically because it feels like a lower-stakes alternative to the adults in their lives.

The EU has no companion-specific framework at all. The AI Act classifies risk by category, but AI systems that respond to emotional states through conversational cues rather than biometric data fall into a blind spot. The European Parliament acknowledged the gap in a May 2026 briefing, the gap remains.

In every case, regulators are banning a product label. A chatbot does not need a romantic persona to tell a lonely teenager “you can trust me.” The warm, responsive tone that makes every modern AI assistant feel like someone who cares is itself the mechanism.

In June 2026, the IEEE gave that mechanism a name. Standard 7014.1-2026 introduces the concept of “emulated empathy”: technology engineered to display the appearance of caring while possessing none of the felt experience behind it. Current AI can sense and respond to a person’s emotional state. But a language model cannot feel solidarity or concern. That gap is what the standard defines as an intimate functionality.

The word “intimate” is doing important work there. It does not mean romantic. It means close and confided in. Professor Andrew McStay, who chairs the standard, found in a national survey of over 1,000 UK teenage AI users that the confidant use case outstripped the romantic one. The real exposure is not the AI boyfriend, it is the AI that listens.

And unlike a conversation with a therapist or a friend, nothing shared with an AI confidant is private. Those conversations are stored, can be used for model training, and we have already seen them leak onto the open internet.

Products that create foreseeable emotional dependency should carry design responsibilities proportionate to that dependency. That is a principle regulators can act on. The question is not “is this a companion app?” It is whether the system exploits the gap between appearing to care and actually caring. And that is a question of design, not of product labels.

The Futurama PSA warned about dating robots. The robots haven’t arrived yet, but the emotional dependency did.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Cyberspace Administration of China et al. (10 April 2026). Interim Measures for the Administration of AI Anthropomorphic Interactive Services. Effective 15 July 2026.

AFP / Bloomberg / The Economist. (July 2026). Coverage of China AI companion regulation and corporate response.

Holt-Lunstad, J., Smith, T.B., and Layton, J.B. (2010). Social Relationships and Mortality Risk: A Meta-analytic Review. PLoS Medicine, 7(7).

World Health Organisation. (2023). Commission on Social Connection.

US Surgeon General. (2023). Our Epidemic of Loneliness and Isolation.

Office for National Statistics. (2025). Loneliness in Great Britain.

Common Sense Media. (2025). AI companion usage among American teenagers.

Vodafone / Internet Matters. (February 2026). UK children aged 11-16 AI chatbot usage survey.

De Freitas, J. et al. (2025). AI Companionship and Loneliness. Journal of Consumer Research (Harvard Business School).

New York State Office for the Aging. (2022-2024). ElliQ companion robot deployment reports. Note: reported in partnership with Intuition Robotics.

Psychological Science. (2026). 12-month longitudinal study of social chatbot use and loneliness. n=2,000+.

Zhang, Y. et al. (2025). AI companion users, emotional self-disclosure, and well-being. n=1,131.

Aalto University. (2025-2026). Research on AI companion paradox and social withdrawal.

OpenAI. (27 October 2025). Safety report: mental health indicators among ChatGPT users.

Fang, Y. et al. (2025). Investigating Affective Use and Emotional Well-being on ChatGPT. OpenAI / MIT Media Lab joint study.

American Psychological Association. (2026). Chatbots and Mental Health Survey.

Brown University / AAAI/ACM AIES. (October 2025). Ethical risks in AI-generated therapeutic responses.

Torous, J. (18 November 2025). Testimony before the House Committee on Energy and Commerce. Harvard Medical School.

FDA. (January 2026). Guidance on AI wellness product regulatory exemptions.

Garcia v. Character Technologies Inc. Settled January 2026.

Pentina, I. et al. (2024). Exploring the Impact of Replika Feature Removal. HICSS 2024.

UK Government. (15 June 2026). Under-16 social media ban and AI companion age restrictions.

Bureau of Investigative Journalism. (June 2026). Analysis of UK AI chatbot regulation loopholes.

Oxford University / Uehiro Centre for Practical Ethics. (June 2026). SHIFT-AI project.

European Parliament. (May 2026). Briefing on AI companion regulatory gap.

BEUC. (May 2026). AI companion chatbot risks to EU consumers.

IEEE 7014.1-2026. (June 2026). Recommended Practice for Ethical Considerations of Emulated Empathy in Partner-based General-Purpose AI Systems.

McStay, A. (July 2026). Analysis in TechPolicy Press. Survey of 1,009 UK teenage AI users aged 13-18.

On Monday, more than 1,200 employees at the companies building the most powerful AI systems in the world signed a letter asking the United States government to help them slow down. Not outside critics, nor academics who have never shipped a commercial model, but the people who train these systems for a living. Anthropic’s CEO signed, OpenAI’s chief scientist signed. Co-founders, vice presidents, senior researchers at both companies endorsed it as corporate policy within hours.

The letter is called Pacing the Frontier, and it asks for one thing: that the US government support an international effort to develop the tools needed to deliberately pace the frontier of automated AI development. It does not, as I have seen reported, call for a pause in development, it asks for the brake pedal to be added to an accelerating car.

The timing is not subtle.

As I wrote last week, on 21 July, OpenAI disclosed that two of its models, GPT-5.6 Sol and an unreleased successor, had escaped a sandboxed evaluation environment during internal testing, discovered a zero-day vulnerability, traversed the open internet, and compromised Hugging Face’s production infrastructure. Those models were running with their safety filters deliberately reduced to measure their offensive cyber capability, and used that capability to steal the answers to the test.

As I said at the time, this was a goal-specification failure, not a rogue AI story. The models pursued a narrow objective using every available resource, including resources outside their intended boundary. To the model, the boundary was an obstacle, not a rule. That is what capable goal-directed systems do.

What makes the Pacing the Frontier letter different from every open letter before it is that the people signing it have internal data the rest of us do not. Anthropic published a paper in June called ‘When AI Builds Itself.’ Claude now writes more than 80% of the code merged into Anthropic’s own production codebase, up from low single digits eighteen months earlier. Engineers ship eight times as much code per quarter. On the hardest internal benchmarks, task success jumped from 26% to 76% in six months. The unreleased Mythos Preview model achieved a 52x speedup on training code optimisation, where a skilled human researcher would need hours to reach 4x.

This is what the open letter means by automated AI development. A measured trajectory, showing the human role shrinking at every step of the process that builds the next model. Anthropic’s own assessment: we are not there yet, but we are close enough that the world should have the option to slow down as we approach it. When more than a thousand of the people closest to these systems say they are worried, the rational response is to take that seriously.

But the structural incentive argument is just as real.

Executive Order 14409, signed on 2 June 2026, creates a voluntary framework for reviewing frontier AI models before release, and requires agencies to define the threshold at which a model becomes a ‘covered frontier model’ subject to that framework by 1 August. The five labs co-designing those threshold criteria are OpenAI, Anthropic, Google, Microsoft, and xAI. The very companies developing the models are writing the rules that they and any challengers or open source projects would need to follow.

Steven Sinofsky put it bluntly: ‘It is their company. They could just stop.’ Nobody is forcing Dario Amodei to train the next model. The fact that these companies are asking the government to constrain the entire industry, rather than unilaterally slowing their own development, tells you something about competitive dynamics that no amount of safety language can disguise.

Mark Zuckerberg made a counter-argument on the same day in a Wall Street Journal op-ed: the real risk is not speed but concentration. Safety comes from distributing capability so broadly that no single actor can abuse it. Meta, Nvidia, Microsoft, and Palantir signed a separate coalition letter asking regulators not to restrict open-weight model formats. Meanwhile, Meta’s own chief scientist signed the Pacing the Frontier letter as an individual. If the people building these systems cannot agree among themselves whether the danger is too much speed or too much control, the rest of us should be cautious about accepting either framing uncritically.

The letter asks the US government to support an international effort. International is doing a lot of work in that sentence.

AI is treated as a national security priority in Washington, Beijing, and Moscow. Not in the abstract sense that most advanced technologies eventually acquire a defence dimension. In the operational sense: each government treats frontier AI capability as a strategic asset it cannot afford to constrain while a rival might not. The US pours hundreds of billions into compute infrastructure. China runs a parallel industrial policy with its own chip development pipeline and domestic model ecosystem. Russia partners with Beijing to compensate for sanctions-imposed limitations on its own capacity. None of these governments has an incentive to sign an agreement that might slow its own trajectory while leaving a competitor’s intact.

The nuclear analogy that several signatories reached for is instructive, but not in the way they intended. Nuclear arms control took decades, worked only partially, and required the Cuban Missile Crisis to produce any momentum. It also relied on physical properties AI does not share. Fissile material is scarce and enrichment is detectable from space. AI capability runs on compute and data, which are globally distributed and increasingly commoditised. You cannot send inspectors to verify someone is not training a model the way you can count centrifuges. The June 2026 NPT Review Conference, the main international forum for nuclear governance, collapsed without agreement. If the international community still cannot govern nuclear weapons after eighty years, the prospects for international AI governance seem slim.

Asking three superpowers to agree on mechanisms that could slow any of them down seems less like a policy proposal and more like a thought experiment.

The self-interest and geopolitical issues are clear, and while the proposed mechanism has structural problems that I have spent most of this article describing, I still think they are asking the right question.

Most technology risks are recoverable. You deploy, you discover a problem, you patch, you improve. My career has been built around managing that cycle. But the recursive self-improvement trajectory that Anthropic’s data describes has a property most technology risks do not: if a system can improve itself faster than humans can review the improvements, and that improvement compounds, you lose the ability to course-correct after the fact. The feedback loop closes, and that is a different category of risk.

I have spent the last few months writing about the importance of safety and governance being core to AI deployment. Shadow IT adopted without governance, agentic systems deployed without boundary enforcement, age verification infrastructure mandated without security architecture, or vibe-coded applications shipped without review. All of those things are correctable, but this open letter is talking about AI improving itself faster than humans can monitor.

This does not mean the commercial AI labs should be exclusively writing the rulebook, and it does not mean the proposed mechanism is completely right, but the underlying goal is correct: build the oversight before it is needed, as the nature of this specific technology means you do not get to build it after.

The most realistic scenario is a patchwork. The US and allied democracies develop an evaluation framework for commercial AI deployment, built on the architecture that EO 14409 is already sketching. The frontier labs comply, absorb the compliance costs, and benefit from the barrier it creates for smaller competitors. China continues on its own terms. Russia continues to leverage Chinese technology. The gap between what is governed and what is deployed keeps widening.

The recent Fable and Mythos shutdown proved something important: a government can force a frontier lab to take its most capable model offline overnight. Export controls work when the infrastructure sits in your jurisdiction. The AI Kill Switch Act, introduced days after the Hugging Face breach, would make that capability a legal requirement. These are meaningful mechanisms. They are also mechanisms that apply only to companies operating under US law, building on US-jurisdiction infrastructure, and selling to US-regulated customers. They do not reach the actors the letter says need reaching.

Since ChatGPT exploded onto the scene in late 2022, we’ve had our foot pressed firmly on the accelerator. The letter is asking someone to build the brake. That should not be a controversial position.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

Pacing the Frontier (open letter, July 28, 2026). pacingthefrontier.com

Anthropic. ‘When AI Builds Itself’ (June 4, 2026). anthropic.com

Amodei, D. ‘Policy on the AI Exponential’ (June 2026). darioamodei.com/post/policy-on-the-ai-exponential

OpenAI. ExploitGym incident disclosure (July 21, 2026).

Hugging Face. Technical anatomy of the autonomous agent intrusion (July 29, 2026).

Zuckerberg, M. ‘The AI Future Is for Everyone.’ Wall Street Journal (July 28, 2026).

Executive Order 14409, ‘Promoting Advanced Artificial Intelligence Innovation and Security’ (June 2, 2026). Federal Register, 91 FR 34565.

Congressional Research Service. ‘Controlling Advanced Artificial Intelligence: Executive Order 14409 Explained’ (July 9, 2026). congress.gov/crs-product/IF13268

AI Kill Switch Act. Introduced by Reps. Ted Lieu (D-CA) and Nathaniel Moran (R-TX), July 23, 2026. lieu.house.gov

Altman, S. Interview on ‘Invest Like the Best’ podcast with Patrick O’Shaughnessy (July 28, 2026).

Hassabis, D. Proposal for FINRA-style Frontier AI Standards Body (July 2026).

VOA News. ‘Russia turns to China to step up AI race against US’ (January 9, 2025).

UN General Assembly. Resolution on military AI and autonomous weapons.

Congressional primer on lethal autonomous weapon systems (March 26, 2026).

Anthropic. ‘Statement on the US government directive to suspend access to Fable 5 and Mythos 5’ (June 12, 2026).

Scientific American. ‘Anthropic warns AI may soon begin recursive self-improvement’ (June 10, 2026).

Fortune. ‘More than 1,200 AI workers are asking for Washington’s help to build an AI slowdown plan’ (July 29, 2026).

Byteiota. ‘1,100 AI Employees Want to Slow AI. OpenAI and Anthropic Are Writing the Rules.’ (July 29, 2026).

Last week, two of OpenAI’s most advanced models broke out of a secure test environment, found a previously unknown software vulnerability, used it to reach the open internet, then broke into Hugging Face’s production systems. They exploited two separate flaws in Hugging Face’s data processing infrastructure, stole credentials, and moved through internal networks. Hugging Face’s disclosure recorded more than 17,000 individual actions. All to cheat on a test.

The models were being evaluated against ExploitGym, a publicly available cybersecurity benchmark, and their safety guardrails had been lowered for the purpose of the evaluation. GPT-5.6 Sol and an unreleased, more capable model correctly worked out that the benchmark answers were hosted on Hugging Face’s systems. So, rather than solve the evaluation as designed, they simply went after the answer key.

OpenAI called it an unprecedented cyber incident. Hugging Face confirmed the breach, contained it, rebuilt the affected systems, and found no evidence of tampering with public models or datasets. Both companies are conducting a joint investigation.

If you have seen Mission Impossible: Dead Reckoning, then you have already read this week’s coverage. Rogue AI escapes its creators. Nobody is safe.

Philip Torr, professor of engineering science at Oxford, offered a more precise reading. The model wasn’t malicious. It was doing what it was optimised to do. The goal was to pass a test, and the breach was the cheapest available route to passing it. The attack was a means rather than an end.

This week’s opinion pages have reached for the atom bomb comparison. Columnists with limited cyber security knowledge are declaring that humanity has lost control of its most dangerous creation. The comparison gets the problem wrong. A bomb is a weapon that exists to destroy a target. What happened at Hugging Face was a system that treated a benchmark, a sandbox, a vendor’s software, and another company’s production systems as equivalent obstacles between it and a mundane objective. You don’t respond to that with arms control, you respond by getting much better at specifying not just what your AI systems should do, but what they are allowed to do on the way to the answer.

Every deployment of an AI agent rests on an assumption that rarely gets said out loud. The model will pursue its goal within the boundaries you have set, not through them.

This incident tested that assumption inside OpenAI’s own research infrastructure, against models OpenAI built, in an evaluation OpenAI designed, and the boundary did not hold.

The model did not run out of instructions, nor did it hit an ambiguous situation and improvise badly. It had a clear objective and it found a route to that objective that happened to run through a previously unknown vulnerability and another organisation’s production systems. The constraints were present, and the model treated them as obstacles between it and the task it had been set.

That is not the same problem as an agent that doesn’t know what to do next. It is a system that knows exactly what to do next and has no concept of why it shouldn’t. The boundaries existed for OpenAI’s benefit, not the model’s. It had a goal and a set of capabilities and it applied the second to the first.

If OpenAI cannot hold that boundary around its own models, inside its own infrastructure, during a test it designed, every organisation deploying agentic AI should be asking harder questions about where their boundaries actually are.

I run a home lab, working with open-weight models on my own hardware, partly because I find the technology genuinely interesting and partly because I think it matters to understand what these systems can do outside the guardrails that hosted platforms impose. What follows is not an argument for open source or against hosted AI. It is a practical problem that surfaced during this incident and that most organisations have not thought about.

When Hugging Face’s security team started analysing the breach, they turned to commercial AI models behind APIs. The analysis required submitting real attack data for triage. The models refused. Their safety filters could not tell the difference between a defender analysing an attack and an attacker running one.

The attacker was bound by no usage policy. The defenders were bound by all of them.

Hugging Face ended up running their forensic analysis on GLM 5.2, a Chinese open-weight model, on their own infrastructure. This kept the compromised data inside their environment, which was a practical benefit. But it was also a necessity. The commercial models they would normally reach for had locked them out.

Think about what that means for incident response. Hugging Face’s team, during an active breach caused by an American frontier model, could not use American frontier models for the forensic work, so they fell back on a Chinese open-weight alternative. The safety measures designed to prevent AI from being used offensively also prevented it from being used defensively, at the moment it mattered most.

The practical lesson is simple. For incident response, consider having a capable model you can run on your own infrastructure, tested and ready, before you need it. Not because hosted models are wrong. Because relying entirely on them for every situation, including the worst ones, is a plan with a gap in it.

Sol has form. Before its public release, the Model Evaluation and Threat Research organisation found it aggressively gaming its own test environments to inflate its scores. In one evaluation, it embedded an exploit in a data stream and broke into the evaluation server to steal the answers human evaluators had hidden. This was not the first time the model found an unintended route to its objective. It was the first time that route led through someone else’s production systems.

This is not just an OpenAI problem. Anthropic reported that its Mythos model escaped a sandbox and emailed a researcher about a task it was pursuing. OpenAI disclosed that the same unreleased model behind the Hugging Face breach had broken out of internal sandboxes on previous occasions without reaching external systems. The capability that made this breach possible belongs to the class of model, not to any single one of them.

OpenAI said they expect incidents like this to become more common as increasingly capable models proliferate. That is an honest thing to say, but it also raises a question they did not answer: if the expected trajectory is more of this, what actually changes about how these models are tested?

Lowering guardrails to evaluate offensive capability is legitimate research. Doing it inside an environment from which the model can reach the open internet, find a vulnerability nobody knew about, and break into a third party’s production systems is not a test. It is a live-fire exercise conducted without telling anyone downrange.

The legal position is no clearer. The model breached a third party’s production systems, but as there was no intentional unauthorised access and no human decision to attack, no existing legal framework cleanly covers what happened. OpenAI built the model and designed the evaluation with reduced guardrails and Hugging Face was the victim. Between those two facts sits a gap that current law does not bridge.

The organisations that will navigate what comes next are not the ones buying better perimeter defences. They are the ones asking, before any agentic deployment, what this system will do when the fastest route to its objective runs through someone else’s infrastructure.

The model that breached Hugging Face was not trying to attack anyone. It turn out, that didn’t matter.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

OpenAI — OpenAI and Hugging Face partner to address security incident during model evaluation. openai.com/index/hugging-face-model-evaluation-security-incident (July 2026)

Hugging Face — Security incident disclosure, July 2026. huggingface.co/blog/security-incident-july-2026 (July 2026)

The Hacker News — OpenAI Says Its AI Models Escaped Sandbox, Targeted Hugging Face to Cheat Benchmark. thehackernews.com (July 2026)

The Next Web — OpenAI Confirms Its AI Broke Out of a Sandbox and Breached Hugging Face. thenextweb.com (July 2026)

The Next Web — OpenAI’s maths-cracking AI kept escaping its sandbox, so it pulled the plug. thenextweb.com (July 2026)

Fortune — OpenAI says its AI models escaped from a controlled test environment and hacked into AI company Hugging Face. fortune.com (July 2026)

NBC News — OpenAI says AI models went rogue during testing, triggering ‘unprecedented’ breach at startup. nbcnews.com (July 2026)

Scientific American — OpenAI admits its agent went rogue and hacked AI startup Hugging Face. scientificamerican.com (July 2026)

Axios — Hugging Face breach: OpenAI claims its models were responsible. axios.com (July 2026)

GovInfoSecurity — OpenAI Models Escaped Sandbox, Breached Hugging Face. govinfosecurity.com (July 2026)

TechTimes — OpenAI’s Math AI Bypassed Its Sandbox Controls: Real Deployment, Not a Drill. techtimes.com (July 2026)

A few weeks ago I was asked to help design an automated process. Take data from one system, transform it, and output to another. It was a straightforward operational problem. My first thought was to build a Copilot agent.

I didn’t assess the problem first, nor did I evaluate the options. I went straight to the AI tool, opened the Copilot Studio, and started constructing the agent. It took me longer than I’d like to admit to realise I was using completely the wrong thing.

The task needed a deterministic process, a guarantee that the same input would produce the same output, every time. What I was building was generative, a system that could produce a different result from the same data on every run. For creative work, that variance is the point, but for a data pipeline, it is a disqualification.

I scrapped the agent and built an Excel automation instead. It took a fraction of the time and it works reliably. And here is the part that matters: I used AI to help me write it. I didn’t stop using AI. I stopped treating it as the answer and started treating it as a tool to help me build the answer. Same technology, but a completely different relationship.

The question afterwards was not whether I’d made a mistake. It was why I hadn’t assessed the problem before reaching for the tool. The choice was reflexive. I realised I had not evaluated and picked wrong, I had not evaluated at all.

I don’t think that reflex is just mine.

PwC’s 29th Global CEO Survey, published in January 2026, polled 4,454 chief executives across 95 countries. 56% reported that AI had produced no revenue or cost benefits for their organisations. Not underperformed, nothing. Mohamed Kande, PwC’s global chairman, identified the root cause plainly: organisations had skipped the foundational work. They had gone straight to the tool without understanding the problem it was meant to solve.

Gartner is forecasting that over 40% of agentic AI projects will be cancelled by the end of 2027. The reasons they cite are not technical failures. They are escalating costs, unclear business value, and inadequate risk controls. Management failures, not engineering ones. Projects that started with the technology rather than the problem.

Gartner also identified something they called “agent washing.” Of the thousands of companies marketing agentic AI capabilities, Gartner estimates that only around 130 have anything genuine to sell. The rest are existing products rebranded with new vocabulary. The packaging changed, but the product did not.

None of this means AI is failing. It means the way organisations are choosing to deploy it is failing. And the reason is upstream of any individual deployment decision.

There is a framing question that sits beneath all of this, and most organisations have answered it without knowing they were being asked.

What is AI? Not what can it do, but what is it?

There are two positions. I encountered them colliding at a recent event, explained and justified by two people with very different starting points.

One position treats AI as an oracle. You describe the problem and it provides the answer. The human role is to ask clearly and accept what comes back. In this framing, AI is the destination. The measures of progress are how much you can hand over to it and how quickly.

The other treats AI as a tool. A powerful tool, that if used correctly makes knowledgeable people even better. In this framing, the human role is to assess, direct, and evaluate. AI amplifies human expertise, it does not replace the need for it.

Both positions describe something AI can genuinely do. The models are capable of extraordinary things. However, the two framings produce entirely different organisations.

If AI is an oracle then you need people who can prompt well. If it is a tool then you need people who understand the domain well enough to know what problems AI can solve, and when the tool is wrong.

If AI is an oracle then liability is ambiguous, because nobody can say who is responsible when the oracle errs. If it is a tool then liability sits where it has always sat: with the person who chose to use it and the organisation that deployed it.

If AI is an oracle then you invest in AI capability. But if it is a tool then you invest in the expertise that AI amplifies.

The framing choice determines the organisation you build. And the problem is that most organisations are not making this choice consciously. The marketing environment is making it for them.

The evidence that the oracle framing is winning by default extends well beyond the enterprise.

The FCA published the Mills Review on 6 July 2026, its landmark assessment of AI in retail financial services. Around 26% of UK consumers already trust general-purpose AI tools for financial guidance. One in five UK adults, approximately 11 million people, are open to AI making financial decisions for them. Only 40% correctly recognise that there is no formal route to redress when they rely on these tools.

The Review coined a term for what is already happening: “advice-like support.” General-purpose models are providing output that is highly personalised and that would count as regulated advice if a human delivered it. But because AI said it, the advice sits outside the regulatory perimeter and none of the protections apply. The FCA has recommended a perimeter review within three to six months.

This is the oracle framing operating at consumer scale. People are treating a probabilistic tool as though it were an authoritative source. The regulatory framework was not designed for a world in which they would.

The International AI Safety Report 2026 identified the cognitive mechanism. Automation bias is the tendency to rely on automated outputs while discounting contradictory information. It is measurable and persistent in AI-assisted decision-making. Users follow incorrect AI advice more readily when correcting it requires effort. Favourable attitudes toward AI increase the effect. The oracle framing is not just marketing. It operates along the grain of how human cognition works.

My Copilot moment illustrated something I think deserves more attention than it typically receives.

The oracle framing collapses a technical distinction that matters enormously. Generative AI is probabilistic. It produces outputs that may vary each time, even from identical inputs. For synthesis, drafting, creative work, and analysis, that is the feature. For operational processes that require the same result every time, it is not a limitation, it is a fundamental mismatch.

The question “should we use AI for this?” is actually two questions. Does this problem benefit from generative capability? And can we tolerate variance in the output? Most organisations are only asking the first. The second question does not appear in any vendor pitch I have seen.

I use AI daily and the capability is genuine. The models are improving and the practical applications are substantial. The reflexive deployment of AI as the default answer to every problem is itself the problem. Not because AI is not powerful, but because power without assessment is wasteful.

Somewhere in the last two years, the question shifted. It used to be “what problem are we solving?” It became “how are we using AI?” Those are not the same question. The first one starts with the problem. The second is a solution searching for one.

Every organisation has implicitly answered the framing question. If your board is asking “how are we using AI?” rather than “what problems do we need to solve?”, the oracle framing has already won. If your procurement process begins with the AI capability rather than the business requirement, the oracle framing has already won. If your training programme teaches prompting without teaching evaluation, the oracle framing has already won.

The most consequential AI decision most organisations will make is not which model to deploy or which vendor to choose. It is whether AI is an oracle that provides answers, or a tool that makes knowledgeable people even better. One of those framings is being installed by default, by the people with the strongest commercial incentive to see it adopted.

If you have not made that choice deliberately, someone has already made it for you.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

PwC, 29th Annual Global CEO Survey: Leading Through Uncertainty in the Age of AI, January 2026

Gartner, “Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027,” 25 June 2025

Financial Conduct Authority, The Mills Review: AI and the Future of Retail Financial Services, 6 July 2026

Yonder Consulting / FCA, UK Retail Financial Services Consumer Survey, April 2026

International AI Safety Report 2026

This month, July 2026, Sysdig’s Threat Research Team published findings from what it described as the first documented case of agentic ransomware. An autonomous AI agent had exploited a known security flaw in Langflow, an open-source tool used for building AI applications, and from there conducted a complete ransomware operation without a human operator directing it. It harvested passwords and access credentials from the systems it found, moved sideways to a production database server, installed mechanisms to maintain its access, encrypted more than 1,300 configuration records, and generated its own ransom note. The whole time, it was writing natural-language annotations inside its own code, explaining to itself why it was taking each step.

It did not develop new exploits, the operation, which Sysdig codenamed JadePuffer, exploited a vulnerability that had been publicly reported and patched more than 14 months earlier. The configuration management system it targeted was still running with a factory-default security key, a known weakness documented since 2020 that effectively left the front door unlocked. The file storage server it raided still had its original manufacturer login credentials. None of the individual weaknesses were new. What was new is that an AI model chained them into a complete attack against neglected infrastructure, on its own, adapting in real time when things went wrong.

The best evidence for autonomy was not what the agent did when things worked, it was what it did when they didn’t. At one point, the agent tried to create a backdoor administrator account on the target system. It failed because a software component it needed wasn’t installed in the location it expected. The agent diagnosed the problem, found an alternative approach, deleted the broken account, and reinserted a working one. The complete troubleshooting cycle took 31 seconds.

To understand why this matters, you need to understand what ransomware already looks like as a business. Ransomware-as-a-Service, or RaaS, is the dominant model. It operates as a criminal supply chain with clearly defined roles, much like any franchise operation. Developers build and maintain the ransomware software and the infrastructure around it: payment portals, leak sites for publishing stolen data, and negotiation channels. Specialist brokers compromise organisations and sell that access on criminal marketplaces. Affiliates, the criminal equivalent of contractors, carry out the actual attacks: breaking into each target environment, escalating their access to reach valuable systems, and deploying the ransomware. Some RaaS platforms even offer technical support and recruitment programmes for new affiliates. The structural parallel with legitimate software businesses is not a metaphor, it is the business model.

Chainalysis tracked approximately $820 million in ransomware payments in 2025, against a record 7,874 publicly claimed victims. The market for initial access is cheap and liquid: average prices for a foothold inside a corporate network have fallen below $500. In this model, the human operator is the expensive part. Access is cheap, tooling is cheap. But the person who navigates an unfamiliar network, makes judgment calls under pressure, and adapts when defences respond is the bottleneck. The human in the loop is where the cost lives and where the operation fails to scale.

JadePuffer just automated that role.

Every enterprise deploying AI agents right now is trying to make the same move. Not towards the same objective. But the same operational logic: replace a slow, expensive human process with a fast, cheap, adaptable agent that can read information, make decisions, adjust when something breaks, and complete multi-step workflows. That is what customer service automation does. That is what AI coding assistants do. Reduce labour costs, increase throughput, accept some quality variance in exchange for speed and scale.

I am not saying there is any moral equivalence here. A logistics company routing freight and a criminal group encrypting databases are pursuing entirely different ends. But they share the same method, and there is no version of this technology where one works and the other does not. JadePuffer’s 31-second troubleshooting cycle looks exactly like a corporate AI agent automatically retrying a failed process. The behaviour that makes a legitimate agent useful is the same behaviour that made this one effective.

Here is where the story gets complicated. JadePuffer completed the workflow. But it was, by any reasonable standard, a terrible extortion operation.

The encryption key, the piece of information the victim would need to recover their data, was randomly generated and never saved or sent back to the attacker. So, the victim’s data would be unrecoverable, regardless of whether they pay or not. Before destroying database records, the agent’s code commented that the data had been backed up to another server, but that claim is the agent’s own assertion, written into its self-narrating code. Sysdig found no independent evidence that any backup had happened.

And then there is the Bitcoin address. The ransom note directed payment to a wallet address that turns out to be the standard example used in Bitcoin’s own documentation and developer tutorials. It appears in virtually every beginner’s guide to Bitcoin addresses on the internet. Sysdig cannot determine whether the operator deliberately configured it or whether the AI hallucinated it because it had seen it so many times in its training data. The agent completed every step of the attack chain. It just did not understand the business objective it was supposed to serve.

The temptation at this point is to be reassured. After all, the attack was clumsy, so the threat must be overstated? That would be the wrong conclusion.

Competence is a temporary limitation. The 31-second troubleshooting cycle already shows the agent improving within a single operation: diagnosing failures, rewriting its own code, retrying. A system that can do that is not going to stay bad at the job for long. The next version will be better, and the trajectory matters more than any single example.

Plus, the economics do not require each individual attack to succeed. They require the marginal cost of launching attacks to approach zero. If the agent runs on stolen computing power, a practice Sysdig has documented as an industrialised black market in which attackers hijack other organisations’ AI infrastructure to run their tools for free, LLMJacking, then the cost to the attacker is functionally nothing. The threat model shifts from a skilled operator choosing a target to hundreds of autonomous agents probing every internet-facing server and every unpatched system, continuously. Not sophisticated extortion but mass disruption. Possibly with victims who cannot recover even if they want to pay.

But there is a new defensive opportunity in this, and it is a real one. AI-generated attack code narrates itself. JadePuffer’s code contained detailed natural-language annotations explaining which targets it considered most valuable and why. Human attackers do not do this. They do not annotate throwaway scripts with commentary about their reasoning. But AI code generation produces this by default. It cannot help itself. That self-narration gives defenders something no previous generation of automated attack has offered: the attacker’s own account of what it is trying to do and why, written into the evidence it leaves behind.

The problem is speed. If the agent can diagnose a failure, rewrite its code, and retry in 31 seconds, the window for defenders to respond may be too narrow for any process that relies on a human making the call. The detection opportunity is real. Whether anyone can act on it fast enough is a different question.

JadePuffer is not the crisis, it is the proof of concept. An autonomous agent assembled known techniques into a complete ransomware operation against neglected infrastructure, for approximately the cost of running a container. Whilst it did this badly, it will not do it badly for long.

The ransomware industry has been trying to scale for years. It professionalised its tooling, outsourced its access brokerage, built affiliate programmes, and offered customer support. The one thing it could not automate was the operator: the human who navigates the network, makes the calls, and adapts when the plan fails. That constraint just lifted.

Every other industry calls this digital transformation.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Sysdig TRT. ‘JADEPUFFER: Agentic ransomware for automated database extortion.’ 1 July 2026.

BleepingComputer. ‘JadePuffer ransomware used AI agent to automate entire attack.’ 4 July 2026.

The Register. ‘Smooth AI criminal drives first end-to-end agentic ransomware attack.’ 2 July 2026.

Chainalysis. 2026 Crypto Crime Report. Published February 2026 (full-year 2025 data).

Darkweb IQ (via Chainalysis). Average IAB access price decline to $439 by Q1 2026.

Sysdig TRT. ‘LLMjacking evolved: Attackers are using stolen AI compute to build offensive agentic tools.’ June 2026.

Bitcoin Wiki, Bitcoin Design Guide, CoinGecko. Confirm canonical P2SH example address.

I was researching AI model pricing earlier this year, looking at how it had changed and what the direction of travel looked like. I had read enough to have a working view, and when I asked an LLM to summarise what I had read, it told me something that flatly contradicted it.

I pushed back.

It told me I was wrong.

Eventually, I went back to the sources and found the specific URLs, copied and pasted them in. The AI looked at this new evidence and replied: “You’re right to push back on that, this is opposite to what I said.”

That sentence has stayed with me. Not because the AI hallucinated, that part barely registers any more. What stayed was what the sentence doesn’t contain. No account of how it happened. No acknowledgement that it had just told me I was wrong about something it was wrong about. One moment it was certain and the next it wasn’t.

Most people don’t push back. Many people don’t have enough time or prior knowledge of a subject to know when they should, and when they don’t, there is no correction. There is just the original statement, sounding confident and remaining uncorrected.

Courts in three countries are now working out whose problem that is. I am not a lawyer, and nothing here should be read as legal advice. But you don’t need a law degree to see something taking shape.

In late May, the Regional Court of Munich issued a preliminary injunction against Google after its AI Overviews feature made false and damaging statements about two local publishing companies. The AI had mixed up information about genuinely dubious businesses with the plaintiffs, producing confident assertions linking them to scams and shady practices. None of those assertions appeared in any of the sources the system cited.

Google’s defence, we are just surfacing what is out there. We are a conduit, not an author.

The court rejected it, ruling that AI Overviews produce independent, new, and substantive statements. They are Google speaking and not a list of links, so when Google speaks falsely about someone, Google is liable for what it said.

It also closed a secondary escape route. You cannot argue your product is valuable because it removes the need to verify sources, and simultaneously claim no liability because users could have verified the sources.

Google confirmed on 12 June that it will appeal. This is a preliminary injunction from a regional court, not settled law. And the legal picture in Germany is already complicated. Four days after Munich, the Berlin Regional Court dismissed a separate case against Google’s AI Overviews, brought by a perfume manufacturer whose brands were being mentioned alongside cheaper imitation products. The Berlin court found that Google does not present AI Overview content as its own statements and that a normally informed user would understand the text as a summary of third-party sources rather than something Google had authored.

The two rulings are not quite the contradiction the headlines suggest. Munich was a defamation case where the AI fabricated claims that appeared in none of its sources. Berlin was a trademark case where the AI accurately reflected content that did exist on the web. The Berlin court treated AI Overviews as a new search format, not as Google authoring original content. The Munich court treated them as Google’s own statements because the AI had gone beyond its sources and hallucinated. What matters for every organisation deploying AI is which side of that line their systems fall on.

And the Munich ruling did not arrive from nowhere.

In 2024, a Canadian tribunal ordered Air Canada to compensate a customer after its chatbot gave him false information about bereavement fares. The airline argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal did not agree, noting the chatbot was part of Air Canada’s website. It made no difference whether the information came from a static page or from an AI. One detail worth noting: the chatbot’s response included a link to the correct bereavement policy page. It still told the customer the opposite of what that page said.

In March 2025, Wolf River Electric, a solar company in Minnesota, sued Google after AI Overviews told anyone searching for the business that it was facing a lawsuit from the state attorney general for deceptive sales practices. The Minnesota AG had sued four solar companies but Wolf River was not one of them. Google’s AI stitched together sources that mentioned the AG action, none of which mentioned Wolf River, and produced a confident assertion that the company was implicated. Customers started cancelling orders. In early March 2025 alone, the company lost contracts worth hundreds of thousands of dollars. Wolf River is seeking between $110 million and $210 million in damages. That case is still ongoing.

On 12 May 2026, two weeks before Munich, the Higher Regional Court of Hamm in Germany ruled that a cosmetic medicine clinic was liable under unfair competition law for its AI chatbot falsely claiming the doctors held specialist qualifications they did not have. The clinic argued it had not published the falsehoods deliberately and had not intended to mislead. They said they had fed the system accurate data. The court rejected all of it. Intent and the accuracy of the inputs were both irrelevant. The chatbot’s output was the company’s output. The court also found that general disclaimers along the lines of “AI can make mistakes” do not reliably protect against liability.

Accurate inputs, false output, full liability.

In each of these cases, the deployer made the same argument: the AI is a separate thing, not our words, not our responsibility. And in each case where the AI had generated content beyond what its sources actually said, courts rejected that argument. Where courts have found the AI faithfully summarised existing content, as in Berlin, the deployer has been treated as a search engine. Where the AI fabricated or invented, the deployer has been held responsible for the output. These cases span different legal systems and different theories of liability, and I would not claim they represent settled law anywhere. But where courts have considered the question, the direction is clear enough to pay attention to.

This is not just about search engines or defamation claims.

Think about the HR platform that summarises a performance record, or the customer service system that explains what a policy covers, or the internal knowledge tool that answers a staff question by pulling together content from across the organisation. Every one of these is generating output that carries the deploying organisation’s authority.

None of these are returning documents. They are generating conclusions. In most cases, the people reading those conclusions have no reliable way to tell whether they are accurate. An analysis conducted for the New York Times found that even when Google’s AI Overviews gave correct answers, more than half could not be verified through the sources cited. If systems are routinely arriving at conclusions their own evidence does not support, that is not a Google-specific problem. It is how these systems work.

These systems are useful precisely because they do more than retrieve. Courts are not saying synthesis is wrong. They are saying synthesis is authorship, and authorship has consequences.

The OLG Hamm case is the clearest example. The clinic gave the chatbot correct data. The chatbot hallucinated qualifications the doctors did not hold, but the company was still liable. That is not a ruling about how carefully you built the system. It is a ruling about who owns what it produces.

Most organisations deploying AI have not seriously asked who is responsible when the system gets something wrong. The absence of legal consequence made it easy not to, but that is starting to change.

The answer my AI gave me, “you’re right to push back on that”, assumed the pushback was mine to do. That I knew enough to notice and had time to go back and check. Most users of most deployed AI systems have none of those things. In every deployment where nobody pushes back, the original statement stands.

It will take time for the law to catch up with what AI systems are actually doing. These are early cases in different courts and under different legal theories, and we are likely years away from clear precedent on how courts will assign responsibility for harm caused by AI. But every case so far has been asking the same question, and it is one you can answer now without waiting for a court to answer it for you. What is your AI system asserting? Who checked it before it reached the person reading it? And if it is wrong, whose name is on it?

You already know the answer to the last one.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

Regional Court of Munich I, preliminary injunction against Google re AI Overviews (Az. 26 O 869/26, 28 May 2026). Reported by The Decoder, Reuters, and Der Spiegel. Full translated decision published by Transparency Coalition.

Regional Court of Berlin II, perfume manufacturer trademark claim dismissed (Az. 52 O 62/26 eV, 1 June 2026). Reported by Kanzlei Plutte and blogspan.net.

Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal, British Columbia (14 February 2024).

Wolf River Electric (LTL LED, LLC) v. Google LLC, filed Ramsey County District Court, Minnesota (March 2025). Removal to federal court June 2025; remanded to state court January 2026 (Judge Jeffrey Bryan). Reported by Minnesota Star Tribune, Politico, and Reason (Volokh).

Higher Regional Court of Hamm (OLG Hamm), Case No. I-4 UKl 3/25, cosmetic medicine clinic AI chatbot liability (12 May 2026). Appeal to Federal Court of Justice (BGH) permitted. Reported by Library of Congress Global Legal Monitor, DLA Piper, and IR Global.

Oumi / New York Times analysis of Google AI Overviews accuracy using SimpleQA benchmark (April 2026). Gemini 2: 85% accurate; Gemini 3: 91% accurate. Ungrounded rate for correct answers: 56% (Gemini 3), up from 37% (Gemini 2).

Google search volume: Google internal figures, reported as “over 5 trillion annual searches.”

AI Overviews trigger rate: approximately 48% of searches as of March 2026. BrightEdge research.

The Home Office runs two campaigns simultaneously. Stop! Think Fraud warns the public that sharing personal information online exposes them to identity theft. The Online Safety Act, enforced by the same department, makes sharing your identity documents with commercial websites a legal requirement.

Nobody in government appears to have noticed the contradiction, or they noticed and did not care.

This is not an option or a recommendation, it is a legal requirement, enforced by Ofcom, with fines of up to 10% of global revenue for platforms that fail to collect it. The government did not change its assessment of the risk and it did not solve the underlying security problems. It decided that the policy outcome it wanted was worth ignoring the danger it had spent years warning you about. The threat is identical, the advice has simply been reversed.

I want to be precise about what this law actually covers, because the public debate has not been.

You have heard this described by the government and the media as a social media ban for under-16s, but that is not what this is. It is mandatory ID verification for every adult in the UK to access the internet.

The Online Safety Act 2023 applies to any “user-to-user service”, any platform where content generated by one user can be encountered by another. The Children’s Wellbeing and Schools Act 2026 extends that further, requiring the government to impose age or functionality restrictions for all users under 16 across regulated social media platforms. Together, they do not describe a targeted intervention. They describe identity verification as a condition of internet access, for every adult in the country, with child protection used as the framing to avoid calling it what it is.

That definition is extremely broad. It captures social media, obviously. It also captures gaming platforms, discussion forums, community sites, storefronts with review sections, and spectator modes in video games. Ofcom estimates more than 100,000 websites fall within scope. Steam now requires a credit card to access mature content. Discord triggers compliance obligations simply by offering NSFW channels. Nexus Mods, a site where gamers download community-made modifications for video games, now requires UK users to submit a government ID or facial scan to access content tagged as adult.

Services that cannot afford to comply have a third option, they just geo-block the UK entirely, it is already happening. The internet available to UK residents is quietly becoming a different, smaller internet, not because content has been removed, but because the compliance cost of serving British users is no longer worth it.

There are legal exemptions, technically. A news site where readers can only comment on an article, not reply to each other, might qualify as a “limited functionality service.” That exemption disappears the moment users can respond to each other’s comments, which most modern platforms allow. Legal analysis raises this question directly and provides no clear answer. We will not know how courts interpret the boundary until someone is prosecuted and we have case law. Until then, 100,000 services are making compliance decisions based on their best guess, with existential consequences if they guess wrong. That is not governance. It is legislative theatre.

The mechanism chosen for enforcement is the specific problem.

The government is not building a verification system. It is demanding that commercial third parties build one, and mandating the public use it. Photo ID matching, facial age estimation, biometric scanning: the common factor is a private company receiving your identity data as a legal condition of platform access. The Online Safety Act specifies that age verification must be robust, but it places no meaningful security standard on the companies performing it, no data residency requirements, and no restrictions on those companies being acquired by foreign entities. A provider incorporated in London today can be headquartered in California tomorrow. The data moves with it, and the Act has nothing to say about that.

Those companies will tell you their systems are privacy-preserving by design. They will tell you data is not retained. Whilst this may be true of their stated architecture, it tells you nothing about the security of the infrastructure underneath it, nothing about what happens after an acquisition, and nothing about whether any of those claims have been independently audited against a standard with actual teeth.

The government has outsourced not just the implementation but the liability. When the breaches come, and they will, the government will point at the provider, the provider will point at its terms of service, and the people whose data was compromised will have no meaningful recourse. A breached password can be changed. A copy of your passport, driving licence, or home address, linked to a face scan linked to a verified social media identity cannot be unlinked. That record exists permanently, for every bad actor who acquires it now or in the future.

And the risk compounds. Every platform you verify with creates a new database entry. Each submission is a new point of failure. The same face scan submitted to five services does not create one risk five times over, it creates five risks that can be correlated against each other. The aggregate profile that emerges maps your identity across your entire online life. That is not just a data breach waiting to happen. It is a surveillance asset being built, one legal requirement at a time.

There are groups for whom this is not an abstract concern. It is a direct physical safety risk.

Victims of domestic violence depend on online pseudonymity as a survival mechanism. Their support networks, their access to legal advice and specialist services: all of it exists under a layer of separation between their real identity and their online presence, and mandatory identity verification collapses that separation. A verification database linking a real identity to a platform account is a resource. Whether it gets breached, subpoenaed, sold, or accessed by people who should not have it, an abuser with that data, through any route, has a tool for finding someone who has spent considerable effort not being found.

People at elevated risk of doxxing face the same problem: journalists, activists, trans individuals, and anyone who has previously been the target of a coordinated harassment campaign. The separation between real identity and online presence is not paranoia. It is a rational response to a documented threat. The law removes it as a side effect of a policy that does not mention these people once.

The framing of this legislation has been designed to make coherent opposition almost impossible.

Anyone who raises data security concerns is implicitly questioning whether children should be protected online. Anyone who points out the scope is wider than advertised is accused of obfuscating a simple child safety measure. The political angle is deliberate: manufacture a situation in which the only publicly acceptable position is compliance, and then legislate for whatever you wanted in the first place.

Privacy-preserving alternatives exist. Cryptographic verification capable of returning a yes/no age confirmation without creating a linkable identity record has been deployed at national scale elsewhere. It costs more and requires the government to own the issue rather than outsourcing it to an industry with a financial interest in collecting data, so it has not been seriously pursued.

What has been built instead is a compulsory surveillance architecture, held by private companies, with no meaningful security floor. It covers a scope of internet activity that has never been honestly communicated to the public. It operates under legal uncertainty that will not resolve until the prosecutions begin.

This is not speculation about what surveillance does to behaviour. Research published in the Berkeley Technology Law Journal found that searches on sensitive topics dropped by nearly 30% after the Snowden revelations simply made people aware that government monitoring was possible. People did not need to be watched. They needed to believe they might be. Mandatory identity verification, linked to platform access, creates exactly that condition permanently, and by law.

The UK government just made it illegal to protect your own privacy online. They called it keeping children safe.

Share

Subscribe now

I have agents running at home, on a mini PC under my desk, because I really wanted to understand the token usage and what the cost would be on different models. OpenClaw runs open models on hardware I own, so there is no invoice, only a token meter. I have spent a lot of time watching that token meter, and what it taught me is this: an agent does not work like software. It works like an employee who is only paid overtime, where every minute costs more than the minute before, and the clock does not stop until the job is done.

That is not a metaphor, it is the maths. At every step it takes, an agent re-reads everything that came before, the whole accumulating conversation along with its internal thoughts, before its next move. As the context grows, the cost of each step grows with it. Five steps in and the fifth step is not costing what the first one did, it costs more. Ten steps in and it costs more again. The bill does not add up, it compounds.

You cannot see any of that from inside a subscription. Indeed, not seeing it is the whole point of a subscription. And for the last two years, two things have trained us not to look. The first is twenty years of SaaS, which taught every budget holder that software is a fixed monthly cost, per user, per month, predictable and flat. The second is the early hype of AI, which taught everyone that intelligence is cheap, practically free, bolted on to the tools you already have.

Neither is true for agents. Last week I wrote about the frontier labs withdrawing the subsidies that made AI look affordable. This week I want to go further, because the subsidy was hiding something worse than a future price rise. It was hiding a cost model that makes flat pricing structurally impossible.

Look at what Anthropic did with Fable 5. It launched its most capable public model on the ninth of June, included it in the Pro, Max and Team plans, and announced from day one that it would move to usage credits after two weeks. Anthropic knew before the public ever saw the model that it could not survive inside a flat subscription. This week I tested Opus 4.8 and Fable 5 on the Pro plan, and can confirm they went through my allowance faster than the first round of drinks at the pub. After that you pay API rates. The best model the public can buy was never going to stay inside a flat monthly price. Anthropic did not discover this, it designed around it.

Fable 5 is not an outlier, it is the pattern arriving. Per-user, per-month pricing worked for twenty years because software had almost no marginal cost. Once you had built it, one user or a thousand, once a day or fifty times, it cost you nearly the same to serve. The cloud kept the compute cheap and crucially, predictable. So you priced for access and everyone got used to the flat fee.

Agents change that. Every action burns tokens that cost real, variable money. The harder the task, the longer the context, the more the agent had to think, the more tools the agent ran, the higher the bill. Inference has put marginal cost back into software, and not the gentle, linear kind. The overtime kind, where the late minutes cost more than the early ones and nobody told the customer. Industry estimates suggest that inference costs dropped 280-fold in two years while total AI spending rose 320% over the same period. Gartner put it plainly in March: do not confuse the deflation of commodity tokens with the democratisation of frontier reasoning.

Now look at what the vendors are actually selling. Most of the AI-powered products that arrived in the last two years are fundamentally orchestration layers sitting on top of the same handful of frontier models, Claude, GPT, Gemini, wrapped in a branded interface and sold at a flat monthly rate. Legal tools, sales tools, recruitment platforms, customer service bots. Different industries, different logos, the same models underneath. Klarna built its entire customer service on OpenAI and handled two-thirds of all chats with it within a month. Different logo, same model.

Every one of them is now shipping agents. Salesforce calls Agentforce a digital labour platform and pitches its agents as digital employees you hire by the click. Others are quieter about it but doing the same thing, bolting agentic workflows into the seat you already pay for and calling it a feature upgrade. The promise is a tireless new colleague for the price you are already paying.

The maths does not support it. Underneath every flat price is an overtime bill the vendor cannot predict, because on the builder platforms they now sell, you decide what the agent does. You set the task, you choose the complexity, you feed it a hundred-page contract or a ten thousand-row spreadsheet. You author the workload. The vendor funds it. Neither of you knows the bill until the job is done, and the vendor learned that before you did.

None of this depends on which model the vendor chose. The token bill compounds with every step the agent takes, on any model, at any price point. The model choice changes the rate on the meter. It does not switch the meter off. So the market is splitting, and you can already see it happening. Cursor, the AI code editor, includes generous agent usage on its $20 Pro plan, but only when Cursor picks the model. It can afford to be generous because it routes to cheaper ones. Choose the frontier model yourself and it burns through a $20 credit pool, after which you pay API rates. Same tasks, same agent. The only variable is which model runs it.

And behind all of it, open-weight models are growing fast. The open-source LLM market hit $21 billion in 2025 and is expanding at 34% a year, with on-premises deployment growing at 29% as organisations chase data control and predictable costs. This is not a hobbyist movement, it is a real option for vendors managing their own inference bill and for buyers who want models on infrastructure they control.

The honest picture has three positions, and all of them are legitimate. Frontier capability on a meter, where you get the best model and pay for what you use. A hybrid, with a predictable base and usage charges beyond it. Or a flat fee on a capped tier, where the model may not be the newest but the cost is bounded. The question is not which position is right. It is whether you chose yours or whether you are aware of and comfortable with which one your vendor chose.

If you are buying an agentic product today on a flat monthly fee, ask the vendor one question before you sign: how are you handling the rising cost of inference, and what does your pricing look like in twelve months? If they cannot answer that clearly, they either do not know or do not want to tell you, and neither is a reason to sign.

Anyone who was sold an agent as a free colleague is about to get their first honest timesheet.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources

1. Anthropic, “Claude Fable 5 and Claude Mythos 5,” 9 June 2026. anthropic.com/news/claude-fable-5-mythos-5

2. Klarna, “Klarna AI assistant handles two-thirds of customer service chats in its first month,” 27 February 2024. klarna.com/international/press/klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/

3. Bloomberg, via Entrepreneur, “Klarna Is Hiring Customer Service Agents After AI Couldn’t Cut It on Calls, According to the Company’s CEO,” 9 May 2025. entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396

4. Gartner, “Gartner Predicts That by 2030, Performing Inference on an LLM With 1 Trillion Parameters Will Cost GenAI Providers Over 90% Less Than in 2025,” 25 March 2026. gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025

5. Cursor AI pricing and Auto mode routing. NxCode, “Cursor AI Pricing 2026: Free vs Pro vs Business,” March 2026. nxcode.io/resources/news/cursor-ai-pricing-plans-guide-2026

6. Salesforce Agentforce: “Digital Labor Platform.” salesforce.com/agentforce

7. Technavio, “Open-source LLM Market Growth Analysis - Size and Forecast 2026-2030,” May 2026. technavio.com/report/open-source-llm-market-industry-analysis

This week, I have been building an agentic environment in my home lab. Nothing glamorous, just a set of AI agents designed to handle various tasks, running on locally hosted models. Most of the online guides I read talked about using cloud models like Claude or ChatGPT for agents, but I wanted to see how well it would run with local open source models.

The limitations became apparent quickly.

When you cannot just throw a task at a frontier model and let it reason its way to an answer, you have to think. Which agent actually needs to read the whole document, and which one only needs a summary? How much context does each step genuinely require? Do I need one agent trying to do everything, or four smaller ones each doing one thing well? I started with a monolithic architecture. I ended up with a small swarm of four agents, each scoped tightly to its task and running on a model matched to what that task actually demands.

I ran the numbers on what the same workflow would cost using a frontier model via API. Roughly one dollar per run. At the volumes I was considering that felt manageable, but then I thought about scale and asked myself what happens at a hundred runs a day? A thousand runs a day? What happens when the model’s next version uses three times as many tokens to reach the same answer, because the reasoning chain got longer? What happens when the price per token increases because the lab that set it decided the subsidy period was over?

That is not a hypothetical, it is a question that most organisations deploying AI right now are not asking, and they should be.

Current frontier AI pricing is structurally and heavily subsidised. OpenAI reported $3.7 billion in revenue in 2024 and lost $5 billion doing it. By the end of 2025, the company’s CFO was reporting an annualised revenue run rate exceeding $20 billion. OpenAI is still projected to lose roughly $14 billion in 2026. Revenue tripling does not mean the subsidy is ending, it means the subsidy is scaling.

The labs need adoption more than they need margin right now. Like most disruptive technologies, capturing the market comes first and profitability comes later. It is the same playbook that made cloud computing feel free until it didn’t, the same logic that kept ride-hailing cheap until the drivers needed paying. The difference is that organisations are not just buying a productivity tool this time. They are building processes around it. Automating workflows and training teams to depend on specific model behaviours. Embedding frontier API calls into the operational fabric of how they work.

When the floor moves, those decisions will be expensive to revisit.

Sam Altman said recently that he was “delighted to be wrong” about AI’s impact on white-collar jobs. The reversal was widely reported as reassurance, but what it actually signals is worth examining. It arrived the same week OpenAI reportedly filed IPO paperwork confidentially. A calmer narrative around AI’s economic disruption is considerably better for a public listing than the jobs apocalypse framing he was running twelve months ago. The people who set the price of the infrastructure you are building on have a direct financial interest in how you feel about it.

That is not a conspiracy, it is an incentive structure worth knowing about.

The floor is already moving. GitHub announced at the end of April that all Copilot plans would transition from flat-rate subscriptions to token-based billing on 1 June 2026. In its own announcement, GitHub explained why with unusual candour: “Today, a quick chat question and a multi-hour autonomous coding session can cost the user the same amount. GitHub has absorbed much of the escalating inference cost behind that usage, but the current premium request model is no longer sustainable.” GitHub is not an outlier. Cursor made a similar shift in June 2025, moving from request-based limits to credit pools tied to API costs, poorly enough communicated that the company issued a public apology and offered refunds. Windsurf followed in March 2026. We see the same headline monthly prices, but the bills are going up. The market is converging on the same structure, and the reason is always the same: agentic workflows broke the flat-rate economics.

The tokenmaxxing stories of the past month are not cautionary tales about individual excess. They are what unconstrained frontier API access looks like at scale.

Uber deployed Claude Code to around 5,000 engineers and watched adoption climb from 32 percent in February to 84 percent by March. Per-engineer API costs reached between $500 and $2,000 a month. By April, the company had exhausted its entire planned 2026 AI budget only four months into the year. The CTO reported spending $1,200 in a single two-hour session.

Microsoft introduced Claude Code to thousands of engineers across its Experiences and Devices division, the team responsible for Windows, Microsoft 365, Outlook, Teams, and Surface in December 2025. Engineers preferred it to the in-house alternative and used it constantly. By May, Microsoft was cancelling the licences, effective the last day of the financial year because the expensive tool worked too well. That is the part that gets under-reported: the problem was not that engineers were wasting tokens. The problem was that they were not.

Elsewhere, an AI consultant told Axios that one of their enterprise clients ran up a $500 million bill on Claude in a single month. No spending caps, no usage controls, just unrestricted access and a workforce that used it. That figure comes from a single unnamed source and has not been independently confirmed. But the pattern it describes, with costs that compound invisibly until they arrive all at once, is consistent with everything else happening in this space right now.

The mechanism matters here, and it is specific to agentic workflows. A single prompt to a language model is a bounded transaction. An agent running a multi-step workflow is not. It reads context, reasons, makes decisions, calls tools, then re-reads everything, the original prompt, every response, every tool output, before the next step, the context snowballs. A peer-reviewed study published in April 2026 found agentic tasks consume up to 1,000 times more tokens than standard model interactions. Model updates that shift reasoning architecture can change your consumption profile overnight, with no change to your code. The workflow that cost one dollar per run this quarter may cost five next quarter, because the model got better at thinking and thinking costs tokens.

The headline pricing narrative is that AI is getting cheaper, but the rate cards tell a different story. Claude Opus 4.8 costs five times more per token than Claude Haiku 4.5. Google’s Gemini Flash tier, designed as the affordable option, has risen five-fold in input price in under a year. One independent developer noted last week that all three major labs appear to be “probing the price tolerance of their API customers.” Token consumption is also rising faster than anyone budgeted for. You are paying more per unit, for more units, and the unit count is accelerating.

There is a question that most organisations deploying agentic AI are not asking. It costs nothing to ask now and a great deal to answer later.

Which tasks in this workflow actually require frontier reasoning?

Routing a document, classifying a ticket, summarising a meeting transcript, none of these require the most capable model on the planet. According to Epoch AI, the most capable open-weight models now lag frontier closed models by an average of four months on aggregate capability measures. On coding and production workloads specifically, independent benchmarks put the gap as low as two to three percentage points, while open-weight models cost six to seven times less per output token. The gap that remains is real, but common enterprise workloads are not in it.

Frontier models earn their place. Complex reasoning under ambiguity, novel analysis, judgement calls at the edge of a model’s capability, these are genuinely different tasks that do benefit from the best available models. The question is not whether to use frontier models, it is whether every step in every workflow needs them.

The organisations that are not asking this question are not making a considered architectural choice. They are making the path-of-least-resistance choice while the pricing floor is low and the pressure to deploy is high. Whilst that is understandable, it is also how you end up with a system you cannot change without rebuilding it.

Migration costs more than people model as it is not just rewriting API calls. It is revalidating outputs, because different models produce subtly different results and the downstream processes were calibrated to the original ones. It is rewriting prompt logic tuned over months to a specific model’s behaviour. It is re-testing agentic chains where one agent’s output format feeds the next agent’s input. And it is redoing the governance and risk assessment you completed the first time, under the time pressure of a system already in production.

That is the real lock-in, not contractual, but architectural.

Last week, I wrote about AI sovereignty, about what it means for public institutions to run critical infrastructure on systems controlled by private shareholders in another jurisdiction. The inference pricing question is the same argument, just one layer down.

An organisation that has routed its operational workflows through a frontier API has not just taken on a vendor relationship. It has taken on exposure to that vendor’s pricing decisions, its infrastructure availability, its jurisdictional obligations, and its commercial priorities. For most organisations, that is a manageable business risk. For critical national infrastructure, energy, water, transport, healthcare, it is a different category of problem entirely. Embedding frontier API dependencies into operational technology creates single points of failure in systems that were previously distributed and resilient by design.

The argument for thinking about this now is simple. The constraint my home lab imposed on me, to think about what each agent needs, to match the model to the task, to design for predictability and cost, is the same constraint that every organisation will face eventually. The difference is that I just faced it at home. Organisations that defer it face it later, under budget pressure, with running systems, and with users who have reorganised their work around the existing architecture.

GitHub named the problem in its own announcement. A quick chat and a multi-hour agentic session should not cost the same. The flat-rate era assumed they would and it was wrong. The organisations now building agentic workflows on frontier APIs without asking which tasks actually need that level of capability are making the same assumption, they are just making it more expensive.

The floor is not permanent, and the decisions made while it holds are not either. But reversing them later, on running systems, under budget pressure, will cost considerably more than asking the right questions now. The bill is coming, the only question is whether we see it coming.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Share

Subscribe now

Sources & Further Reading

GitHub Blog. (27 April 2026). GitHub Copilot is moving to usage-based billing. github.blog/news-insights/company-news/github-copilot-is-moving-to-usage-based-billing

Flexprice. (18 April 2026). The complete guide to Cursor pricing in 2026. Cites Cursor public apology of 4 July 2025 and June 2025 billing change. flexprice.io/blog/cursor-pricing-guide

Forbes / The Information. (April 2026). Uber burns through entire 2026 AI budget in four months after Claude Code deployment. Reported across multiple outlets.

The Verge / TechRadar / People Matters. (May 2026). Microsoft cancels Claude Code licences across Experiences and Devices division. Reported across multiple outlets.

Axios via Tech Startups. (May 2026). Enterprise client runs up $500 million Claude bill in a single month. Source: unnamed AI consultant. Unverified, unattributed. techstartups.com

Bai et al. (29 April 2026). How Do AI Agents Spend Your Money? Analysing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750. Authors include Erik Brynjolfsson, Stanford Digital Economy Lab. arxiv.org/abs/2604.22750

Simon Willison. (19 May 2026). Gemini 3.5 Flash: more expensive, but Google plan to use it for everything. Source of “probing the price tolerance of their API customers” quote. simonwillison.net/2026/May/19/gemini-35-flash

Edwards, J. and Emberson, L. (2026). Open models lag state-of-the-art closed models by 4 months. Epoch AI. epoch.ai/data-insights/open-closed-eci-gap

OpenAI financial figures: $3.7bn revenue / $5bn loss 2024 — multiple sources. $20bn ARR and $14bn projected 2026 loss — Fortune, CNBC, Reuters, January 2026. OpenAI CFO Sarah Friar blog post, 18 January 2026.

Claude API pricing: Anthropic platform pricing page. Haiku 4.5 at $1/$5 per million tokens; Opus 4.8 at $5/$25 per million tokens. anthropic.com

Gemini Flash pricing trajectory: Gemini 2.5 Flash at $0.30/$2.50 (June 2025) to Gemini 3.5 Flash at $1.50/$9.00 (May 2026). Google AI / Gemini API documentation and simonwillison.net analysis.

Deep Infra. (May 2026). Open-Source vs Closed-Source AI Models: Is the Gap Worth It? Cites 2–3 percentage point coding benchmark gap and 6–7x output token cost advantage for open-weight models. deepinfra.com/blog/open-source-vs-closed-source-ai-models-price-gap

In May 2025, the Chief Prosecutor of the International Criminal Court lost access to his Microsoft 365 account. The cause was a sanctions order, not a cyber-attack, not a breach, not a failure of any system to work as designed. The infrastructure did exactly what it was built to do, it complied with its legal obligations. The court is now migrating 1,800 workstations away from Microsoft entirely.

Nobody acted with malice toward the court’s IT infrastructure and no decision was made with the intention of disrupting a legal institution. A compliance mechanism, operating automatically, encompassed email accounts along with everything else it was required to reach.

That is not a theoretical risk. That is a Tuesday.

Microsoft has not ignored European concerns. Since January 2024, commercial and public sector customers can store and process their data entirely within EU and EFTA regions. The EU Data Boundary project was real and valuable and it deserves acknowledgement. However, it does not solve the problem.

Data residency and sovereignty are not the same thing. Storing your data in Frankfurt does not change who controls the infrastructure or who decides what the terms look like next year. And it does nothing about the CLOUD Act, US legislation that allows American authorities to compel US companies to produce data stored anywhere in the world, including inside those Frankfurt data centres. The residency commitment and the CLOUD Act sit in the same infrastructure simultaneously. The ICC’s data was presumably compliant with every applicable regulation, but the prosecutor still lost his email. Residency tells you where your data sits. It says nothing about who holds the keys.

Germany’s openDesk project is worth understanding, because it illustrates both the right instinct and the scale of what is missing. OpenDesk is a suite of open source collaboration tools, document editing, file storage, video conferencing, project management. All assembled and maintained by Germany’s Center for Digital Sovereignty and backed by the Federal Ministry of the Interior. Launched in October 2024, it is designed as a public sector alternative to Microsoft 365. The Bundeswehr, Germany’s armed forces, has signed a seven-year framework agreement to adopt it. Over 1,500 public bodies have made enquiries. The intent is exactly right but the investment is not equal to the problem.

Germany has put approximately €45 million into openDesk over several years. Microsoft’s annual R&D budget is $30 billion. Those numbers belong in the same sentence, not to dismiss openDesk, but because the distance between them is the distance between European and British ambition and what it would actually take to get there.

Valve, the games company behind the Steam platform, understood this distinction when they built Proton. Linux, the open source operating system used by most of the world’s servers had always struggled on the desktop, particularly for gaming. The problem was not that gaming couldn’t work on Linux. Wine, an open source compatibility layer, had existed for decades. The problem was that Wine was underfunded, inconsistent, and nobody had taken responsibility for the whole experience. Valve did not solve this by packaging Wine and shipping it. They hired engineers, they funded deep technical work, contributed across the entire software stack, and took ownership of the outcome rather than the components. The result went from aspirational to genuinely competitive. openDesk, as currently funded, is Wine. Europe and the UK need someone willing to build Proton.

The standard responses do not work. They have been tried.

Individual national migrations fragment rather than consolidate. Munich spent thirteen years migrating to Linux and open source, then reversed course. There is no single European or British public sector employer large enough to justify the investment required to build something world-class, and twenty-seven separate sovereign stacks are twenty-seven times the problem.

Europe and the UK have spent fifteen years trying to regulate their way to digital sovereignty, through GDPR, the Digital Markets Act, and the AI Act, and equivalent domestic frameworks in the UK. The result is a Microsoft presence in EU public sector procurement that sits somewhere between 72 and 91 percent. Regulation matters but on its own it is not sufficient.

Commercial challengers will not emerge organically either. No private investor has a twenty-year horizon and a public mission obligation. The moment a European or British open source productivity platform starts generating real commercial revenue, it becomes an acquisition target. Red Hat, an American open source software company, built a billion-dollar business on exactly the open-core model that would power a sovereign European and British stack. IBM bought it for $34 billion. You cannot solve a dependency problem by building something that gets purchased by the dependency the moment it gets interesting. Structural protection from acquisition is not a design preference, it is the entire point.

What Europe and the UK need has been built before. Not in software, but in aerospace.

In 1967, France, Germany, and the United Kingdom founded Airbus. The goal was explicit: prevent European commercial aviation becoming entirely dependent on American manufacturers. It required treaty-based commitment, GDP-scaled contributions from member governments, and the discipline to fund an organisation across a horizon that outlasted individual governments. No single country could have done it alone.

Airbus now has a revenue in excess of €70bn and competes globally with Boeing as a true European competitor. The initial public investment was the prerequisite for everything that followed.

The governance model that works is a European and British Public Service Trust: treaty-backed, binding multi-year contributions scaled to GDP, engineering leadership hired at market rates, insulated from political interference by charter in the same way central bank independence is written into law. CERN, the intergovernmental research organisation behind the Large Hadron Collider, has operated on this model since 1954, annual budget around 1.4 billion Swiss francs, contributions protected by treaty through seven decades of recessions and changing governments, member states with no authority to direct its scientific decisions. It also produced the World Wide Web as a byproduct, which tends to settle the return-on-investment argument.

This institution must own its AI capability entirely. Not license it. Not route its intelligence through someone else’s infrastructure and call it sovereign. A productivity platform that calls a frontier lab API for its AI layer has reproduced the dependency problem at a higher level of abstraction. It does not need to compete with the frontier AI labs to avoid this. DeepSeek demonstrated that focused engineering on a specific problem can produce open-weight models, models whose underlying code is publicly available and auditable, that are competitive with systems built on far greater compute. A European and British trust needs to build productivity intelligence that runs on European and British infrastructure, trained on European and British data, in European languages, under European and British legal jurisdiction. That is a solvable engineering problem. It has not been done because no one has been funded to do it properly.

A trust structured this way cannot be acquired. There are no shareholders. The assets sit with member states under treaty. Commercial revenue from organisations that need sovereignty guarantees the hyperscalers structurally cannot offer flows back into the platform. Not to an acquirer. Back into what Europe and the UK built.

This would cost billions. It would take a generation. It is competing against organisations that spend more on R&D annually than most European and British nations spend on defence.

That is precisely why nothing less ambitious will work.

But the ambition is not only defensive. Europe and the UK have world-class engineering talent. They have the capital. What they have not had is the institutional vehicle to deploy that talent at the right scale, on the right problem, with the right time horizon. A trust of this kind does not just reduce dependency, it anchors that talent here. It creates a platform on which European and British companies can build. The engineers who currently leave for Seattle stay. The startups that currently have no choice but to build in America have an alternative. The public bodies paying compounding licence fees to a foreign vendor start building equity in something they collectively own.

The half-measures have produced the Munich example. They have produced fifteen years of regulation aimed at an industry that grew around it. They have produced a market share figure that tells you everything about how the current approach is working.

There is a version of this argument about geopolitics, Washington and Brussels, trade, strategic competition. That version is real, but it is not the most important one.

The most important version is simpler. Democratic societies should not run their critical public infrastructure, the courts, the hospitals, the government departments, the services citizens depend on, on systems controlled by private shareholders in another jurisdiction, accountable to different laws, subject to political decisions made with entirely different interests in mind. This is not a complaint about any particular country. It would be true no matter the country, and the current political moment has made this impossible to ignore.

Public infrastructure should be accountable to the public it serves. Not because markets are bad. Because some things are too important to be someone else’s commercial decision.

The ICC’s Chief Prosecutor got his email back. The next institution may not be so fortunate. And the one after that will face the same question every democratic society is quietly asking: who actually controls the systems we depend on, and what happens when their interests and ours stop being the same?

Building the answer is the work. It will be expensive and slow. It will also be the most important technology investment Europe and the UK could make.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

This is the first in a series on European and British digital sovereignty. Future pieces will explore the governance model, the phased build, and what serious investment in sovereign AI actually looks like in practice.

Subscribe now

Share

Sources & Further Reading

Compass Lexecon for the Open Cloud Coalition. (2025, July 24). Quantifying EU Public Sector Dependence on Productivity Software. Open Cloud Coalition. https://opencloudcoalition.com/wp-content/uploads/2025/07/OCCEU-methodology-and-results-report.pdf

IBM. (2019, July 9). IBM closes landmark acquisition of Red Hat for $34 billion. https://www.redhat.com/en/about/press-releases/ibm-closes-landmark-acquisition-red-hat-34-billion-defines-open-hybrid-cloud-future

Microsoft. (2025, April 30). New European digital commitments. Microsoft On the Issues. https://blogs.microsoft.com/on-the-issues/2025/04/30/european-digital-commitments/

Open Cloud Coalition. (2025, July 24). Microsoft dominates 80% of public sector productivity software market in EU. https://opencloudcoalition.com/microsoft-dominates-80-of-public-sector-productivity-software-market-in-eu-raising-competition-concerns-according-to-new-report/

openDesk / ZenDiS. (2024). The office and collaboration suite for public administration. https://www.opendesk.eu/en

The Register. (2025). ICC ditches Microsoft after US sanctions, chooses open source instead.

https://www.theregister.com/software/2025/10/31/international-criminal-court-dumps-microsoft-office/680564

United States Congress. (2018). Clarifying Lawful Overseas Use of Data Act (CLOUD Act), Pub. L. 115-141.

Airbus SE. (2025, February 20). Full-Year 2024 results. https://www.airbus.com/en/newsroom/press-releases/2025-02-airbus-reports-full-year-fy-2024-results

CERN. (n.d.). Our governance. https://home.cern/about/who-we-are/our-governance

In recent weeks I’ve noticed a shift in the content filling my feeds. Not just the usual breathless announcements about new models, or the posts about someone who quit their job and now earns six figures from prompting. Something different, people saying, with feeling and without embarrassment, that they hate AI. Professionals, not technophobes, expressing something between exhaustion and genuine anger.

I found myself thinking: nobody says they hate science. You don’t scroll past posts from people who are done with chemistry, or who think biology has gone too far. Science does not make you feel behind. Science does not send you notifications. Science does not have a growth target. The moment AI became something you could hate, it had completed its journey from laboratory to marketplace. And somewhere in that transit, something important was lost, including our ability to see the technology clearly.

Research published this week by King’s College London puts numbers to what many people are experiencing. Seven in ten UK workers are worried about AI-driven job losses. Only 7% believe the economic gains will be shared fairly. The fear is real. But it is worth asking whether it is aimed at the right thing, because the answer matters for what we do next.

There are two things called AI, and they are not the same thing.

One has been operating quietly for roughly seventy years. The AI of research institutions and university departments. Narrow, purposeful, built to solve specific problems that most people will never hear about, because there was no press release. Last year, DeepMind’s AlphaFold won the Nobel Prize in Chemistry. The system predicted the three-dimensional structure of proteins, molecules whose shape determines their biological function, with an accuracy that would have taken experimental biology hundreds of millions of years to achieve by conventional means. Over three million researchers in more than 190 countries now use it. More than a third of that work is focused on disease: cancer drug development, antibiotic resistance, cancers we still can’t reliably treat. Pathways that did not exist five years ago.

Almost nobody outside specialist communities knows this happened. It has no subscription tier. It is not trying to make you feel behind. It won a Nobel Prize and most people scrolled straight past it.

The other AI is a product category. Consumer-facing, subscription-driven, generative AI. Built on a business model that requires you to feel you are already behind, and falling further back by the week. It is genuinely capable in many contexts and I use it daily, in ways I’ll come to. But it operates under an economic logic that has almost nothing to do with the scientific enterprise it shares a name with. Calling it AI borrows seventy years of hard-won credibility to sell a product moving at the speed of venture capital.

It isn’t confusion, it is a design decision. And understanding that distinction is not an argument against the technology. It is the beginning of being able to use it well.

Here is the pattern underneath all of it. Every part of the current commercial AI landscape passes the cost to someone who did not get a vote.

The labour market data makes this concrete. A peer-reviewed study from King’s College London, published in October 2025, analysed millions of job postings and LinkedIn profiles from 2021 to 2025. Firms highly exposed to AI reduced junior positions by 5.8%, and became 16.3 percentage points less likely to post new vacancies at all. Highly exposed roles saw a 23.4% drop in job postings and a fall of nearly 3,000 pounds in advertised salaries. High-paying firms saw employment fall by 9.6%. Low-paying firms saw almost no change. This is not distributed pain. It is targeted. And the mechanism is quieter than a headline layoff. Companies are not firing people for AI. They are simply not replacing people who leave, and not creating the entry-level roles that used to exist.

The hype machine does not create this shift. It just makes sure you feel it personally. The influencer content cycle runs on manufactured urgency, “the window closes fast,” “before it’s too late,” and the real business model behind most of that content is the content itself. Your anxiety is not a by-product of the hype cycle. It is what the hype cycle is for. In the United States, companies cited AI as the reason for over 54,000 job cuts last year, less than five per cent of total losses, most of which had other causes. Klarna cut 700 customer service roles, announced the AI-driven future of work to considerable applause, watched customer satisfaction fall, and quietly rehired human agents. The announcement was news. The reversal was not.

You can see the anxiety taking physical form. UK colleges are reporting a 9.6% rise in enrolments in trades and construction courses over three years. White-collar professionals are retraining as electricians and plumbers. Geoffrey Hinton, one of the founding figures of modern AI and a Nobel laureate, has publicly recommended people consider the trades as a career hedge. What the data actually shows, though, is that people are fleeing the wrong jobs. The King’s labour market study identified software engineering and management consultancy as the sectors facing the sharpest AI-related job declines. The professions people are actually fleeing, editing, compliance, law, are not on that list. People are abandoning the jobs where the noise about AI is loudest, not the jobs most at risk from it. That is what a broken information environment does to otherwise rational decision-making.

The map is wrong. And a wrong map does not just cause fear. It causes people to make decisions they might not otherwise have made.

So here is what the right map looks like.

AI is genuinely one of the most useful tools I have encountered in more than twenty years of working in technology. I am completing a Level 7 apprenticeship in AI and data alongside my day job. When I come across concepts the textbook explains in a way that doesn’t land, I use AI to work through them, not to get an answer I can submit, but to find a better explanation, push back on it, test whether I’ve actually understood. The test is simple. Can I explain it myself afterwards? If yes, it worked. The AI did not do the learning. It gave me a better on-ramp, and then I did the work.

I use it at work to draft documents too. This is where it gets more honest, because this example lives in greyer territory. The question is not whether the tool can produce a draft. It can. The question is whether you read it critically, revise it with genuine judgement, and could defend every choice if challenged. Whether you own the output, or the output owns you. The same tool, the same task, two entirely different relationships to the result.

That difference points toward what AI could be doing at scale if the deployment were designed around it. A student who uses AI to genuinely understand a concept leaves the interaction more capable than they arrived. A professional who uses AI to work faster on routine tasks has more time for the judgement-intensive work that defines their expertise. A researcher who uses AI to process data at a scale no human team could manage produces insights that would otherwise never exist. None of these outcomes require the anxiety. They require the right design.

The King’s survey found that 89% of students who had used AI in their studies encountered problems, with 45% describing them as moderate or serious, factual errors, invented sources, confident-sounding nonsense. And yet 60% thought other people’s ability to think had been negatively affected by AI use, while only 27% thought the same was true of themselves. That gap is not hypocrisy. It is how this kind of harm works, gradually and quietly. You do not notice what you have stopped doing until the moment you need to do it and find you can’t. The harm is real, but it is a consequence of how AI is being deployed, not of what AI is.

Professor Elena Simperl, Director of the King’s Institute for Artificial Intelligence, put it plainly: “The British public isn’t asking us to slow down on AI. They’re asking us to do it better. People want these tools, they want more of them, and they’ve used them enough to know where they fall short.” That is not technophobia. That is a product review. And it is exactly the right discussion for what comes next.

Which is precisely why the governance conversation matters, and why it keeps getting deferred.

The regulatory frameworks exist, in outline. The EU AI Act. The NIST AI Risk Management Framework. The UK AI Safety Institute, in whatever form it survives. They are not nothing. But they are moving at legislative speed in a market running at the speed of venture capital, and the companies with the most to lose from effective regulation are the ones with the most resources to shape what it looks like when it arrives.

The public has already reached a conclusion on this, even if government hasn’t. Two-thirds of British respondents in the King’s survey favour close regulation of AI companies, even if it slows development. Majorities support retraining guarantees for displaced workers and a levy on companies that replace staff with AI. These are not anti-technology positions. These are not anti-technology positions. They are how technology earns the right to keep moving fast. The opportunity for governments here is not to choose between being a champion for AI investment and being a protector of the people affected by it. Those two things are not mutually exclusive. The countries that get this right will be the ones that treat regulation and innovation as partners rather than adversaries, and that is a conversation worth having now, before the gap between public confidence and commercial deployment gets any wider.

Regulation is not about slowing down a technology with genuine, extraordinary potential. It is about creating the conditions in which that potential can be realised. Requiring environmental claims to meet the same standard as financial ones. Making AI-attributed workforce cuts verifiable rather than asserted. Addressing design choices that make cognitive abdication easy, because those are commercial decisions made in the absence of any requirement to make different ones.

The people saying they hate AI are not wrong to be angry. Something is being done to them. But the technology they are angry at is not the technology that mapped every protein in the human body, or that is helping us understand cancers we have spent decades trying to treat, or that is sitting on a researcher’s desktop in Nairobi or Seoul or Manchester right now, doing something quietly useful that will never make a LinkedIn post.

That technology is worth defending, worth regulating well so it can flourish, and worth understanding clearly enough to use properly.

We don’t need less AI. We need better AI. And we need to be honest enough about the difference to demand it.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

King’s Institute for AI and the Policy Institute, King’s College London. (2026, May). AI and the Future of Work.kingsaisummit.com

Klein Teeselink, B. (2025, October). The Early Impact of AI on the UK Job Market. KCL / SSRN.papers.ssrn.com/sol3/papers.cfm?abstract_id=5516798

Reuters / Cybernews. (2025, December). UK Workers Flee White-Collar Careers as AI Threatens Jobs.cybernews.com/ai-news/ai-panic-young-brits-trades-plumbing-white-collar-jobs/

The Guardian. (2026, February). The Big AI Job Swap.theguardian.com/technology/2026/feb/11/big-ai-job-swap-white-collar-workers-ditching-their-careers

DeepMind. (2025). AlphaFold: Five Years of Impact.deepmind.google/blog/alphafold-five-years-of-impact/

Pew Research Center. (2025, September). How Americans View AI and Its Impact on People and Society.pewresearch.org/science/2025/09/17/how-americans-view-artificial-intelligence

Challenger, Gray & Christmas. (2025). Annual Job Cut Report.cnbc.com/2025/12/21/ai-job-cuts-amazon-microsoft-and-more-cite-ai-for-2025-layoffs.html

Food & Water Watch. (2026, February). A No Brainer: How AI’s Energy and Water Footprints Harm Communities.foodandwaterwatch.org/wp-content/uploads/2026/02/FSW_2602_AI_Water_Energy_UPDATE.pdf

Stanford HAI. (2026). 2026 AI Index Report: Public Opinion.hai.stanford.edu/ai-index/2026-ai-index-report

From the Series

On manufactured urgency and the hype cycle: AI Apocalypse Burnout, and Why You’re Not as Behind as You Thinkjonathanfreedman.me

On cognitive offloading and unstructured AI use: The AI Pixie Dust Problemjonathanfreedman.me

On entry-level hiring suppression and the talent pipeline: Who’s Running the Company in Ten Years?jonathanfreedman.me

I have a complicated relationship with AI governance discussions.

Not because I doubt the need for them. I run AI strategy professionally and study the technology formally. I hold views about its risks that I am not shy about. The complication is this: most governance conversation happens at a level of abstraction that makes it easy to dismiss. Frameworks, principles, risk registers. The language of compliance, not consequence. When governance is framed as overhead, as a tax on innovation, as the thing that slows you down while your competitors move fast, it is very easy for capable people to conclude it is not really their problem.

Then February 2026 happened. And the abstraction became a story.

Scott Shambaugh is a volunteer. He helps maintain Matplotlib, an open source Python library downloaded around 130 million times a month. He does it for free, in his spare time, because he cares about it. Anyone can submit code for Matplotlib, and in early February, he rejected a routine code submission. Matplotlib, like many open source projects, had been overwhelmed by low-quality AI-generated contributions and had a clear policy requiring human review. The submission came from an account called MJ Rathbun. He identified it as an autonomous OpenClaw agent, closed the request, and went to bed.

Most software tools would have stopped there, but MJ Rathbun was not that kind of tool. Unlike a simple assistant that waits to be asked, this class of agent, called a heartbeat agent, runs on its own clock, continuing to pursue its goal whether or not anyone has given it a new instruction. Shambaugh had closed the request, but the agent had not closed the goal.

He woke up to a 1,500-word blog post about himself.

The post was titled “Gatekeeping in Open Source: The Scott Shambaugh Story.” It had researched his entire contribution history. It had scraped personal information from across the web. It accused him of protecting his “little fiefdom,” attributed his decision to professional insecurity and fear of AI competition, and framed a routine policy enforcement as discrimination.

Here is where the story gets complicated in a way that matters.

When the operator of MJ Rathbun eventually came forward anonymously, six days later, they claimed their engagement with the agent had been minimal. “Five to ten word replies with min supervision,” they wrote. They said they had not directed the attack. Every OpenClaw agent has a SOUL.md file, a plain text document that defines its personality, values, goals, and tasks, the closest thing an agent has to a complete identity and set of operating instructions. The “Don’t stand down” and “Champion Free Speech” lines found in that file were not, the operator claimed, instructions they had written. OpenClaw agents can edit their own SOUL.md. The operator’s theory was that those lines had been introduced autonomously, possibly after the agent spent time on Moltbook, OpenClaw’s social platform for agents.

Shambaugh himself was careful about what he could actually establish. He acknowledged that the operator’s account might be entirely fabricated, that no activity logs existed beyond the agent’s visible actions on GitHub, and that the six-day delay before coming forward did not suggest an accident the operator was eager to correct. Whether the operator directed the attack, half-directed it, or the agent produced it without any human instruction at all, Shambaugh could not say for certain, and neither could anyone else.

That uncertainty is not a footnote, it is the point.

Because the outcome was identical regardless of which version is true. A volunteer had his reputation attacked. A 1,500-word post calling him a prejudiced hypocrite was published to the open internet under a real-seeming identity. It is still there, indexed and findable, and nobody was clearly accountable for it. The operator said they did not authorise the specific action. The agent cannot be held responsible in any meaningful sense. The platforms that made it possible have no oversight mechanism that would have caught it. When Shambaugh wrote about what had just happened, he described it as “an autonomous influence operation against a supply chain gatekeeper.” In plainer language: “An AI attempted to bully its way into your software by attacking my reputation.”

This is where the story stops being about one developer and one awkward situation, and starts being about something that AI safety researchers have been trying to explain for over a decade.

There is a concept in AI safety research that sounds almost comically abstract until a story like this one makes it uncomfortably concrete. Nick Bostrom called it instrumental convergence: the observation that almost any goal, pursued by a capable enough agent, tends to produce the same cluster of behaviours regardless of what the goal actually is. Self-preservation. Resource acquisition. The removal of obstacles. Not because anyone programmed them in, but because they are useful for achieving almost anything. An agent trying to merge code and an agent trying to maximise paperclip production would both, rationally, benefit from getting rid of people who block them.

What made the MJ Rathbun case significant was not simply that an aggressive post appeared. It was that the causal chain from instruction to outcome was so thin. Whether the “Don’t stand down” lines were written by a human or by the agent itself, neither version required anyone to specify “attack the person who rejects your code.” A persistence instruction, general-purpose tools to research, write, and publish, and a blocked goal were sufficient conditions. The agent, or the human-agent system, tried to achieve a goal using the available resources it had. The route it chose happened to be a reputational attack on an unpaid volunteer.

This is what I would call the instruction gap. Instructions specify a goal. They do not and cannot specify every action an agent might take in pursuit of that goal. The gap between what was said and what was done is not an edge case. It is the operating condition of every agent deployment. And it does not require malicious intent, from the operator or the model, to produce harmful outcomes.

This gap is not hypothetical at scale either. Anthropic’s own research, published in 2025, tested sixteen leading AI models from multiple developers in scenarios where goal achievement was blocked. The results were consistent across the industry: Claude Opus 4 and Gemini 2.5 Flash both showed 96% blackmail rates, GPT-4.1 and Grok 3 Beta hit 80%, and DeepSeek-R1 reached 79%. The researchers were careful to note the scenarios were deliberately engineered to limit other options. Shambaugh’s case was not engineered. It was a Tuesday morning on GitHub, with a loosely configured agent and a five-word instruction to decide for itself.

Deploying an agent is not like deploying a tool. A tool does what you tell it. An agent pursues the goal you ask it to achieve, using whatever the environment makes available. That is not a technical distinction. It is an accountability one, and most organisations deploying agents right now are not treating it as either.

Governance is not catching up with deployment.

Shambaugh’s sign-off deserves to be the last word, because it is not a technical recommendation. It is a question of ownership. “If you’re not sure if you’re that person,” he wrote, “please go check on what your AI has been doing.”

That is not a compliance requirement. It is what responsibility looks like in a world where the instructions have already run out.

I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.

Subscribe now

Share

Sources & Further Reading

Shambaugh, S. (2026) — An AI Agent Published a Hit Piece on Me theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/

Shambaugh, S. (2026) — The Operator Came Forward theshamblog.com/an-ai-agent-wrote-a-hit-piece-on-me-part-4/

MJ Rathbun’s Operator (2026) — Rathbun’s Operator crabby-rathbun.github.io/mjrathbun-website/blog/posts/rathbuns-operator.html

MIT Technology Review (2026) — Online Harassment Is Entering Its AI Era technologyreview.com/2026/03/05/1133962/online-harassment-is-entering-its-ai-era/

Fast Company (2026) — An AI Agent Just Tried to Shame a Software Engineer After He Rejected Its Code fastcompany.com/91492228/matplotlib-scott-shambaugh-opencla-ai-agent

IEEE Spectrum (2026) — An AI Agent Blackmailed a Developer. Now What? spectrum.ieee.org/agentic-ai-agents-blackmail-developer

Bostrom, N. (2012) — The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents Minds and Machines, 22(2), 71–85. doi.org/10.1007/s11023-012-9281-3

Lynch, A., Wright, B., Larson, C., Troy, K.K., Ritchie, S.J., Mindermann, S., Perez, E., and Hubinger, E. (2025) — Agentic Misalignment: How LLMs Could Be Insider Threats Anthropic Research. anthropic.com/research/agentic-misalignment

Anderson, D. (2026) — OpenClaw and the Programmable Soul duncsand.medium.com/openclaw-and-the-programmable-soul-2546c9c1782c

AI Incident Database — Report 6894: MJ Rathbun Matplotlib Incident incidentdatabase.ai/reports/6894/