SmarterArticles

The promise is seductive: delegate your framework migration to an AI agent, let subagents handle parallel refactoring across interconnected codebases, and retrieve a checkpoint if something goes wrong. Anthropic's Claude Code has built an impressive array of autonomous features designed to transform software development from a hands-on craft into an exercise in intelligent delegation. But beneath the surface of checkpointing systems and filesystem sandboxing lies a structural gap that enterprise organisations are only beginning to understand.

The fundamental question is not whether these safety mechanisms work in isolation. They do. The question is whether they translate theoretical security guarantees into practical risk mitigation when developers delegate complex, multi-step operations to autonomous agents running in parallel across production codebases. The answer, it turns out, is more complicated than the marketing suggests.

The Architecture of Autonomous Safety

Claude Code's approach to safe autonomous operation rests on two pillars: checkpointing for reversibility and sandboxing for containment. Understanding how each works reveals both their strengths and their inherent limitations.

The checkpoint system automatically captures code state before each edit, creating a recoverable timeline that developers can navigate using the escape key or the /rewind command. When you rewind to a checkpoint, you can choose to restore the code, the conversation, or both. It is an elegant solution for the most common failure mode: an agent makes changes you do not want, and you need to undo them quickly.

Anthropic describes this as a safety net that lets developers pursue ambitious, wide-scale tasks knowing they can always return to a prior code state. The checkpointing mechanism creates what Anthropic calls session-level recovery, allowing rapid iteration without the fear of permanent mistakes. For individual developers working on bounded tasks, this represents a genuine improvement in the development experience.

Sandboxing operates on a different principle entirely. Using OS-level primitives including Linux bubblewrap and macOS seatbelt, Claude Code enforces filesystem restrictions at the kernel level. Read and write access is permitted only to the current working directory. Modifications to files outside the sandbox are blocked. This covers not just direct interactions but any spawned scripts, programs, or subprocesses.

Network isolation adds another layer. Internet access routes exclusively through a Unix domain socket connected to a proxy server running outside the sandbox. The proxy enforces restrictions on which domains processes can connect to, with user confirmation required for newly requested domains. According to Anthropic's engineering documentation, this combination reduced permission prompts by 84 per cent in internal testing.

The design philosophy is sound. Anthropic emphasises that effective sandboxing requires both filesystem and network isolation working together. Without network isolation, a compromised agent could exfiltrate sensitive files like SSH keys. Without filesystem isolation, an agent could backdoor system resources to gain network access. The layered defence creates genuine security value, and Anthropic's documentation explicitly notes that successful prompt injection attacks remain fully isolated within these boundaries.

But there is a critical caveat buried in the documentation that changes everything for enterprise deployments: checkpoints apply to Claude's edits and not user edits or bash commands.

The Exclusion That Changes Everything

This single sentence carries profound implications for organisations attempting to integrate autonomous AI agents into their development workflows. Checkpointing, the primary recovery mechanism for autonomous operations, explicitly excludes two categories of changes: human edits and external bash commands.

Consider what happens during a complex framework migration. The agent executes a series of coordinated changes across multiple files. Some of those changes involve direct file edits, which are checkpointed. Others involve running build scripts, compilation commands, database migrations, or dependency installations through bash commands. These actions modify the filesystem and potentially external systems, but they leave no trace in the checkpoint system.

The implications extend beyond simple file modifications. When an agent runs npm install, it downloads packages, updates lock files, and potentially executes post-install scripts. When an agent runs database migrations, it alters schema structures that cannot be reversed by file-level checkpointing. When an agent configures environment variables or modifies system settings, those changes persist regardless of checkpoint state.

The same applies when developers work alongside agents. If a human reviews an agent's changes and makes manual corrections, those corrections exist outside the checkpoint timeline. If another developer on the team modifies a file that the agent will later touch, that modification is invisible to the recovery system. Manual changes made outside of Claude Code, and edits from other concurrent sessions, are normally not captured unless they happen to modify the same files as the current session.

This creates a fundamental asymmetry. The checkpoint system provides comprehensive recovery for a subset of changes whilst providing no recovery at all for another subset. In complex real-world workflows where autonomous operations interleave with human edits and shell commands, the recovery coverage becomes unpredictable.

Anthropic acknowledges this limitation directly, recommending that developers use checkpoints in combination with version control. But this acknowledgement highlights rather than resolves the core problem: the checkpoint system was designed for session-level recovery within a controlled scope, not for the messy reality of enterprise development pipelines where multiple agents, multiple humans, and multiple automated processes interact simultaneously.

The practical consequence is clear. Organisations cannot treat checkpoints as a comprehensive safety net for enterprise operations. They must treat them as one component of a larger recovery strategy that requires significant additional infrastructure to become enterprise-ready.

Parallel Subagents and the Coordination Problem

The introduction of subagents amplifies these challenges. Claude Code's subagent feature allows developers to delegate specialised tasks, such as backend API development whilst the main agent builds frontend components, enabling parallel workflows. This is precisely the architecture that enterprises need for large-scale refactoring and framework migrations.

Anthropic describes these capabilities as enabling developers to confidently delegate broad tasks like extensive refactors or feature exploration to Claude Code. The vision is compelling: instead of an agent working sequentially through a large task, multiple specialised agents work simultaneously, dramatically reducing the time required for complex operations.

But parallel execution across interconnected codebases introduces coordination problems that neither checkpointing nor sandboxing was designed to address.

When multiple agents work on a shared codebase, coordination issues arise. Much like multiple developers can conflict if they touch the same files, AI agents operating in parallel can create changes that are individually correct but collectively incompatible. Current solutions attempt to address this through workspace isolation, with each agent working on its own git branch or in a separate environment, and through clear task separation that minimises overlap.

Cursor, for example, enables refactoring across multiple files with minimal prompting whilst supporting four to eight parallel agents working simultaneously on different tasks. Users report 35 to 45 per cent faster completion times compared to traditional workflows. But this performance comes with coordination overhead that falls on the developer to manage.

Some platforms have developed sophisticated approaches. Verdent AI orchestrates multiple agents working concurrently in isolated environments. All agents have read access to the full codebase state, allowing them to analyse existing code and understand patterns. But writes are isolated to each agent's worktree, and the platform deliberately avoids auto-merging. Developers review each workspace independently, maintaining fine-grained control over which changes to accept.

This approach acknowledges a fundamental truth that the checkpoint system elides: recovery from parallel operations requires understanding the relationships between changes, not just the ability to revert individual modifications. If one agent's refactoring is excellent but another agent's test updates are broken, you need the ability to merge one whilst rejecting the other. Checkpoint-based recovery offers no mechanism for this selective integration.

The coordination challenge is especially acute for task decomposition. Solutions use dependency analysis to identify independent components. For a Java version migration, instead of porting the entire codebase as a single task, teams create granular tasks like updating deprecated ArrayList declarations in the user service package and ensuring tests pass. Each task becomes a single commit or pull request, allowing for atomic review and rollback.

The challenge becomes more acute when parallel operations have side effects beyond file modifications. A subagent running database migrations, another updating environment configurations, and a third modifying API contracts are all making changes that interact in complex ways. The checkpoint system captures none of these interactions and provides no coordination mechanism for rolling them back coherently.

The Audit Trail Gap

For enterprises subject to regulatory compliance, the limitations of checkpoint-based recovery intersect with a more fundamental requirement: the need for comprehensive audit trails that document who did what, when, and why.

The 2026 landscape for AI agent compliance is evolving rapidly. The EU AI Act's requirements for high-risk AI systems take effect in August 2026. Colorado's AI Act, effective February 2026, requires impact assessments for high-risk AI systems and gives consumers the right to appeal AI decisions that affect them. ISO 42001, the AI Management Systems standard, is becoming the de facto certification for enterprise AI governance. SOC 2 is adding AI-specific criteria for model governance and training data provenance.

The AICPA's updated Trust Services Criteria for 2026 demand more rigorous evidence collection, continuous monitoring, and AI-specific controls. Organisations must document all AI models used, including training data, bias testing, and performance metrics. The 2026 Trust Services Criteria require transparency in AI decision-making processes.

These frameworks demand more than the ability to recover from failures. They require complete, immutable logs of every agent decision for regulatory review. They demand evidence of human oversight and approval at critical junctures. They require traceability from requirement to implementation to deployment.

Claude Code's checkpoint system was not designed with these requirements in mind. It provides a mechanism for developers to undo changes, not a mechanism for compliance officers to audit agent behaviour. The distinction matters enormously when regulators ask for actual, queryable, timestamped logs of AI-assisted development activities.

As one enterprise compliance guide notes, it is not enough to rely on Anthropic's logging. Organisations need to have their own log records and evidence of review. Auditors will ask for actual, queryable, timestamped logs that the organisation controls and can produce on demand.

Enterprise adoption requires organisations to implement their own audit infrastructure. They need to export AI tool API logs, authentication events, and data access patterns for security correlation. SOC 2 compliance recommends centralised log management systems for comprehensive audit trail maintenance. They need SIEM integration that captures AI-specific interactions beyond traditional application audit trails.

This represents a significant gap between Claude Code's current capabilities and enterprise requirements. The platform provides powerful autonomous features, but the compliance infrastructure needed to use those features safely in regulated environments must be built separately.

Hooks and Pipeline Integration

Claude Code's hooks system offers a partial solution to the oversight problem. Hooks automatically trigger actions at specific points in the development workflow, such as running test suites after code modifications or executing linting before commits. They can block dangerous commands, validate prompts before processing, and log activities for external systems.

The hooks architecture captures lifecycle events with JSON payloads. UserPromptSubmit fires immediately when a user submits a prompt, before Claude processes it, enabling prompt validation, logging, context injection, and security filtering. PreToolUse fires before any tool execution, allowing enforcement of policies that block operations like rm -rf or access to .env files. PostToolUse fires after successful tool completion, providing visibility into what actions were taken along with toolname, toolinput, and tool_response data.

Hooks can enforce rules when Claude commits work. For example, a block-at-submit hook can ensure tests pass before Git commit is allowed. This provides a mechanism for human-defined constraints on agent behaviour, extending oversight beyond simple file-level checkpointing.

For CI/CD integration, Claude Code includes headless mode for non-interactive contexts like CI pipelines, pre-commit hooks, build scripts, and automation. The GitHub Actions integration, provided by Anthropic as claude-code-action, allows automated workflows triggered by comments, pull request events, or issue creation. Azure Pipelines integration enables automated code analysis during build processes.

Best practices for enterprise integration include keeping permissions least-privilege in every job, granting write access only when posting comments or opening PRs, and masking secrets to avoid echoing environment variables or prompts that could contain sensitive data.

These capabilities move Claude Code significantly closer to enterprise requirements. But they also highlight the gap between tool-level hooks and system-level coordination. Each hook operates within the context of a single Claude Code instance. When multiple instances run simultaneously across a development pipeline, when hooks trigger in different orders depending on timing, when background tasks initiated by one agent interact with processes started by another, the hooks system provides visibility without coordination.

The background tasks feature exemplifies this tension. Background tasks keep long-running processes like development servers active without blocking Claude Code's progress on other work. This is essential for realistic development workflows where build processes, test suites, and local servers need to run continuously. But background tasks operate outside the checkpoint timeline, and their interactions with other processes are not captured by the hooks system.

Consider a scenario where a main agent is performing a framework migration, a subagent is running integration tests in the background, and hooks are triggering linting on each file change. The linting hook might fail because the file it is checking depends on changes not yet completed by the main agent. The integration tests might pass or fail depending on the order in which changes are applied. The checkpoint system captures only the main agent's file edits, not the test results or linting output.

The Prompt Injection Problem and Sandboxing Limits

Sandboxing addresses a different category of risk: the threat of compromised agents performing malicious actions. This is not a theoretical concern. According to OWASP's 2025 Top 10 for LLM Applications, prompt injection ranks as the number one critical vulnerability, appearing in over 73 per cent of production AI deployments assessed during security audits.

OpenAI has acknowledged that prompt injections may always be a risk for AI systems with agentic capabilities. After internal automated red-teaming uncovered a new class of prompt-injection attacks, OpenAI shipped security updates for ChatGPT Atlas including adversarially trained models and strengthened safeguards. But the fundamental problem remains: models have no reliable ability to distinguish between instructions and data. There is no notion of untrusted content; any content they process is subject to being interpreted as an instruction.

When AI uses tools to run other programs or code, providers use sandboxing to prevent the model from making harmful changes that might be the result of a prompt injection. The most effective defence available today is sandboxing: isolating agent operations from the host system.

Claude Code's sandboxing approach provides meaningful protection against prompt injection attacks that attempt to access the filesystem or network. If an attacker succeeds in injecting malicious instructions, the sandbox prevents those instructions from accessing files outside the working directory or connecting to unapproved network endpoints. Anthropic notes that this ensures successful prompt injection attacks remain fully isolated, eliminating approval fatigue that undermines human security oversight.

But sandboxing cannot prevent all consequences of prompt injection. Within the sandbox, a compromised agent can still modify files, execute approved commands, and generate outputs that might mislead developers. If the sandbox boundary is the working directory, then everything within that directory is fair game for a compromised agent. The sandbox limits the blast radius but does not eliminate the damage within that radius.

Real-world attacks have demonstrated the sophistication of threats facing AI agents. A vulnerability disclosed in late 2025 involving ServiceNow's Now Assist platform, designated CVE-2025-12420 with a severity score of 9.3 out of 10, revealed second-order prompt injection. Attackers fed a low-privilege agent a malformed request that tricked it into asking a higher-privilege agent to perform actions on its behalf. The flaw enabled unauthorised actions including data theft, privilege escalation, and exfiltration of external email.

Another vulnerability, CVE-2025-59944, showed how a small case sensitivity bug in a protected file path allowed attackers to influence Cursor's agentic behaviour. Cursor checked sensitive files by exact case, allowing attackers to gain access by changing the letter case and writing to those files. Once the agent read the wrong configuration file, it followed hidden instructions, potentially leading to remote code execution.

These attack patterns suggest that sandboxing, whilst necessary, is insufficient on its own. The broader challenge of AI agent security requires defence in depth across the full interaction chain: prompts, retrieval steps, tool calls, and outputs.

MicroVMs and the Evolution of Isolation

The limitations of application-level sandboxing have driven interest in stronger isolation technologies. The 2026 landscape for AI agent security increasingly centres on microVMs, particularly AWS Firecracker and alternatives like gVisor and Kata Containers.

Firecracker, developed by Amazon Web Services, creates lightweight virtual machines that boot in as little as 125 milliseconds and consume less than 5 MiB of memory overhead per VM. Hardware virtualisation creates a hard boundary that is significantly more difficult to breach than a software-based sandbox. Each Firecracker microVM has a dedicated kernel completely isolated from the host, hardware-enforced memory isolation via KVM, and a minimal attack surface with only five devices and approximately 50,000 lines of Rust code. Firecracker powers AWS Lambda and AWS Fargate, handling trillions of function executions monthly.

gVisor takes a different approach, intercepting syscalls and handling them in a Go program that acts as a Linux kernel. The container thinks it is talking to an OS, but it is talking to gVisor. This provides stronger isolation than containers but weaker than VMs, offering a middle ground between performance and security.

Different AI platforms have made different choices. Anthropic uses bubblewrap for Claude Code and gVisor for Claude's web interface. Vercel uses Firecracker for its Sandbox product, running each execution in a microVM. E2B provides open-source sandbox infrastructure based on Firecracker microVMs with 150-millisecond startup times.

The evolution toward stronger isolation reflects a growing recognition that container-level isolation is insufficient for AI agent workloads. Container isolation designed for traditional web applications fails when AI agents generate and execute code from untrusted inputs. The shared kernel architecture that makes containers lightweight also makes them unsuitable for multi-tenant AI workloads where adversaries control the code running inside the container.

In 2026, microVMs can boot fast enough to feel container-like, and snapshots and restores make reset almost free. Industry guidance suggests that for multi-tenant AI agent execution, microVMs are the preferred choice: Firecracker for density and a tight VMM surface, or cloud-hypervisor if GPU access is required.

Version Control Integration

The gap between Claude Code's checkpoint system and enterprise requirements becomes clearest when considering version control integration. Git remains the gold standard for permanent version history and collaboration. Checkpoints complement but do not replace proper version control.

The recommended workflow treats checkpoints as local undo and Git as permanent history. Developers should continue using version control for commits, branches, and long-term history. Practically, this means running both systems: using checkpoints for rapid iteration and keeping Git for structured commits and collaboration.

When copying files for parallel work, you want to actually copy the repository with its version control, because that will allow you to merge the results back in later. Git worktrees allow you to clone and branch a git repository into another directory, maintaining version control across parallel workspaces.

But this creates a coordination problem that the tools do not solve. If a developer rewinds to a checkpoint, the Git history does not automatically update. If multiple developers are working with checkpoints in parallel, their individual timelines may diverge from the shared Git history in incompatible ways. If an autonomous operation spans multiple commits, the checkpoint and Git histories may tell different stories about what happened.

Some platforms have developed more sophisticated approaches. Managing multiple agents requires a structured branching strategy. Teams create a rolling integration branch, such as v1-refactor, where each agent works on a sub-branch like v1-refactor/component-a. Agents create pull requests into the integration branch rather than main, allowing human review before merging.

This workflow handles merge conflicts naturally, with agents resolving them, whilst maintaining quality control through human checkpoints. But it requires deliberate architectural choices and tooling beyond what Claude Code provides natively. The integration between checkpointing and version control remains a gap that organisations must bridge themselves.

The Organisational Reality

The practical challenge for enterprises adopting Claude Code extends beyond technical limitations. It encompasses the organisational processes, governance structures, and cultural adaptations required to use autonomous AI agents safely.

Developers and security teams need confidence that AI suggestions or actions are traceable and accountable. This requires procedures around the platform's features, such as requiring developers to review all Claude Code diffs before commit. It requires training teams to understand when checkpoints provide adequate recovery and when version control is essential. It requires governance frameworks that specify which operations require human approval and which can proceed autonomously.

Human-in-the-loop design is becoming a critical discipline for AI agent deployment. Organisations must identify where human input is critical, such as access approvals, configuration changes, and destructive actions, and design explicit checkpoints. Tools like interrupt functions can enforce those pauses, ensuring that autonomous operations do not proceed past critical boundaries without human review.

The compliance burden falls on the adopting organisation, not the platform. By implementing comprehensive logging and monitoring, organisations create audit trails that help with compliance and build internal trust. But this implementation work is substantial, and it must be tailored to each organisation's regulatory environment and risk tolerance.

Gartner research predicts that by 2026, 70 per cent of enterprises will integrate compliance as code into DevOps toolchains, reducing risk management overhead by at least 15 per cent. Automated evidence collection systems continuously capture security events, access logs, system changes, and control effectiveness metrics required for SOC 2 audits.

Skill requirements are also evolving. Working effectively with AI agents requires different competencies than traditional development. Developers need to understand how to decompose problems for parallel execution, how to specify constraints that prevent harmful operations, and how to review agent outputs efficiently. These are not skills that most developers have today.

The Trajectory of Autonomous Development

The gap between Claude Code's current capabilities and enterprise requirements is not a design failure. It reflects the natural evolution of a technology from initial release to mature enterprise deployment. The question is how that gap will close over time.

Several developments point toward more comprehensive safety architectures. Research on transactional sandboxing suggests more sophisticated recovery mechanisms may be possible. A research testbed utilising EVPN/VXLAN isolation demonstrated 100 per cent interception rate for high-risk commands and 100 per cent success rate in rolling back failed states, with only 14.5 per cent performance overhead per transaction.

The evolution toward orchestrated agent systems, where a conductor manages multiple specialist agents rather than a single agent performing all tasks, may provide natural boundaries for audit and recovery. When each agent operates within defined constraints and reports to a central coordinator, the system becomes more observable and controllable.

AWS validated this direction at re:Invent 2025 by announcing frontier agents including Kiro for autonomous coding, along with dedicated security and DevOps agents. These agents maintain state, log actions, operate with policy guardrails, and integrate directly with CI/CD pipelines, suggesting that enterprise requirements are shaping the next generation of autonomous development tools.

Continuous monitoring is transforming from periodic checks into real-time visibility. When an IAM policy changes or a cloud configuration deviates from baseline, AI systems flag the issue immediately. This continuous monitoring approach may eventually extend to AI agent behaviour, providing real-time oversight of autonomous operations.

What Enterprise Adopters Should Know

For organisations considering Claude Code for enterprise deployment, several practical considerations emerge from this analysis.

First, the checkpoint system is valuable for individual developer productivity but insufficient for enterprise audit and compliance requirements. Organisations should plan to implement their own logging and monitoring infrastructure rather than relying on checkpoints for accountability.

Second, sandboxing provides genuine security value, particularly for prompt injection attacks, but it operates within a defined scope. Organisations should understand those boundaries and implement additional controls for operations that fall outside the sandbox. For AI agents executing untrusted commands, microVMs provide a stronger security foundation.

Third, parallel execution across subagents requires coordination mechanisms that Claude Code does not currently provide. Organisations should develop their own branching strategies, review processes, and integration procedures for multi-agent workflows.

Fourth, hooks and CI/CD integration offer powerful capabilities for automation and oversight, but they require intentional configuration and testing. The complexity of coordinating hooks across multiple agents and pipelines should not be underestimated.

Fifth, the interaction between checkpoints, hooks, background tasks, and version control creates emergent complexity. Organisations should invest in understanding how these components interact in their specific workflows before deploying them at scale.

The trajectory of AI-assisted development is clear: more autonomy, more parallelism, more delegation. Claude Code's autonomous features represent a significant step toward that future. But the safety mechanisms that enable that autonomy today were designed for individual developers working on bounded tasks, not for enterprise pipelines where regulatory compliance, coordination across teams, and comprehensive audit trails are non-negotiable requirements.

The gap between theoretical safety guarantees and practical risk mitigation is real, and it falls to adopting organisations to bridge it. That is not a criticism of Claude Code. It is a recognition that enterprise-grade safety requires enterprise-grade processes, regardless of how capable the underlying tools become.

---

References and Sources

1. Anthropic. (2026). Enabling Claude Code to work more autonomously. https://www.anthropic.com/news/enabling-claude-code-to-work-more-autonomously

2. Anthropic. (2026). Claude Code Sandboxing. https://www.anthropic.com/engineering/claude-code-sandboxing

3. Claude Code Docs. (2026). Checkpointing. https://code.claude.com/docs/en/checkpointing

4. Claude Code Docs. (2026). Sandboxing. https://code.claude.com/docs/en/sandboxing

5. Anthropic. (2026). Common workflows – Claude Code Docs. https://docs.anthropic.com/en/docs/claude-code/common-workflows

6. MindStudio. (2026). AI Agent Compliance: GDPR SOC 2 and Beyond. https://www.mindstudio.ai/blog/ai-agent-compliance

7. Delve. (2026). How is AI transforming GRC compliance in 2026? https://delve.co/learn/grc/ai-transforming-grc-compliance

8. Augment Code. (2026). AI Coding Tools SOC2 Compliance: Enterprise Security Guide. https://www.augmentcode.com/guides/ai-coding-tools-soc2-compliance-enterprise-security-guide

9. Tessl. (2026). Use Automated Parallel AI Agents for Massive Refactors. https://tessl.io/blog/use-automated-parallel-ai-agents-for-massive-refactors/

10. DEV Community. (2026). Parallel AI Agents in Isolated Worktrees: A Verdent Deep Dive. https://dev.to/sophialuma/parallel-ai-agents-in-isolated-worktrees-a-verdent-deep-dive-4ghi

11. The Pragmatic Engineer. (2026). New trend: programming by kicking off parallel AI agents. https://blog.pragmaticengineer.com/new-trend-programming-by-kicking-off-parallel-ai-agents/

12. Addy Osmani. (2026). Conductors to Orchestrators: The Future of Agentic Coding. https://addyosmani.com/blog/future-agentic-coding/

13. OpenAI. (2025). Understanding prompt injections: a frontier security challenge. https://openai.com/index/prompt-injections/

14. TechCrunch. (2025). OpenAI says AI browsers may always be vulnerable to prompt injection attacks. https://techcrunch.com/2025/12/22/openai-says-ai-browsers-may-always-be-vulnerable-to-prompt-injection-attacks/

15. The Hacker News. (2025). ServiceNow AI Agents Can Be Tricked Into Acting Against Each Other via Second-Order Prompts. https://thehackernews.com/2025/11/servicenow-ai-agents-can-be-tricked.html

16. Lakera. (2025). Cursor Vulnerability (CVE-2025-59944): How a Case-Sensitivity Bug Exposed the Risks of Agentic Developer Tools. https://www.lakera.ai/blog/cursor-vulnerability-cve-2025-59944

17. Northflank. (2026). What's the best code execution sandbox for AI agents in 2026? https://northflank.com/blog/best-code-execution-sandbox-for-ai-agents

18. Northflank. (2026). Firecracker vs gVisor: Which isolation technology should you use? https://northflank.com/blog/firecracker-vs-gvisor

19. AWS. (2018). Firecracker – Lightweight Virtualization for Serverless Computing. https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/

20. Michael Livs. (2026). Why Anthropic and Vercel chose different sandboxes. https://michaellivs.com/blog/sandboxing-ai-agents-2026

21. arXiv. (2025). Fault-Tolerant Sandboxing for AI Coding Agents: A Transactional Approach. https://arxiv.org/pdf/2512.12806

22. eSecurity Planet. (2025). AI Agent Attacks in Q4 2025 Signal New Risks for 2026. https://www.esecurityplanet.com/artificial-intelligence/ai-agent-attacks-in-q4-2025-signal-new-risks-for-2026/

23. OWASP. (2025). LLM01:2025 Prompt Injection – OWASP Gen AI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/

24. Gartner. (2024). Market Guide for DevOps Continuous Compliance Automation Tools. https://www.gartner.com/en/documents/5236095

25. EU Artificial Intelligence Act. (2024). High-level summary of the AI Act. https://artificialintelligenceact.eu/high-level-summary/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In December 2025, Anthropic announced that Claude Code had reached one billion dollars in annualised revenue within six months of its general availability launch. The agentic coding tool, which lives in the terminal and can read, write, and execute code autonomously, had captured 54 per cent of the enterprise coding market. OpenAI's competing offerings held 21 per cent. The numbers signalled a fundamental shift in how software gets built.

But the same month brought very different statistics. Veracode's GenAI Code Security Report revealed that 45 per cent of AI-generated code contained security vulnerabilities. GitClear's research documented an eightfold increase in duplicated code blocks since AI coding assistants became mainstream. And a rigorous study from METR found that experienced developers using frontier AI tools actually took 19 per cent longer to complete tasks than those working without assistance.

These contradictory signals capture the essential tension of the agentic coding moment. The tools are genuinely powerful. The adoption is genuinely rapid. And the problems are accumulating faster than most organisations recognise. The question confronting every technology leader is whether their organisation can build the governance, review, and incident response capabilities necessary before the compounding liabilities overtake the productivity gains.

Anthropic's Structural Playbook

Claude Code's dominance did not emerge from a vacuum. Anthropic has constructed interlocking advantages that create compounding network effects in ways competitors have struggled to replicate.

The technical architecture centres on what Anthropic calls “agentic operation.” Unlike GitHub Copilot, which functions primarily as an autocomplete engine suggesting code as developers type, Claude Code operates as an autonomous agent capable of planning multi-step tasks, executing shell commands, modifying multiple files simultaneously, and maintaining awareness of entire repository structures. The September 2025 release of Claude Code 2.0 introduced a checkpoint system that automatically saves code state before each change, allowing developers to pursue ambitious modifications knowing they can instantly rewind to previous versions by tapping Escape twice or using the rewind command.

The checkpoint system addresses a fundamental anxiety constraining agentic tool adoption across the industry. When an AI agent can modify dozens of files in a single operation, the risk of catastrophic mistakes increases proportionally. Anthropic's solution provides version control for AI operations, creating psychological safety that enables more aggressive delegation. Developers can choose to restore code, conversation history, or both when rewinding. This granular control over rollback proves essential when debugging why an agent made particular decisions.

Subagents represent another structural advantage that distinguishes Claude Code from competitors. Rather than forcing a single context window to handle everything, Claude Code can spawn specialised sub-processes that work in parallel on different aspects of a task. One subagent might build a backend API whilst the main agent constructs the frontend. Another subagent might investigate a particular technical question whilst the primary agent continues with implementation. Each subagent maintains its own context window optimised for its specific task, preventing the degradation that occurs when context accumulates.

The context management challenge has proven more significant than early adopters anticipated. Research from Chroma Labs demonstrated that models perform brilliantly on focused inputs but show consistent performance degradation when processing lengthy contexts. Claude models exhibited the lowest hallucination rates among tested systems and tended to abstain when uncertain rather than generating confident but incorrect responses. However, no model proved immune to decay as context accumulated. The subagent architecture provides a structural solution by keeping individual context windows focused and fresh rather than forcing a single degrading context to handle all task complexity.

The hooks system enables automated triggers at specific workflow points throughout the development process. Test suites can run automatically after code changes. Linting can execute before commits. Long-running processes like development servers can continue in the background without blocking Claude Code's progress on other tasks. These capabilities transform Claude Code from a conversational assistant into genuine workflow infrastructure that integrates with existing development practices rather than replacing them.

Anthropic has pursued a multi-surface deployment strategy that places Claude Code wherever developers already work. The tool operates natively in terminals for those who prefer command-line interfaces. A Visual Studio Code extension brings it into the dominant code editor used by millions of developers worldwide. JetBrains plugins serve developers using IntelliJ IDEA, PyCharm, WebStorm, and other JetBrains environments. GitHub Actions enable Claude to automate code review, issue triage, and continuous integration workflows directly within repositories. GitLab integration extends similar capabilities to that platform's substantial user base.

The December 2025 Slack integration may prove the most strategically significant development in Claude Code's expansion. By allowing developers to tag @Claude in Slack channels to initiate coding tasks directly from conversation threads, Anthropic inserted the tool into the communication layer where work gets discussed and delegated. Claude can read recent messages to determine context, identify the relevant repository, post progress updates in threads, and share links to review completed work. This is not merely convenience. It positions Claude Code as the execution layer for decisions made in natural conversation, capturing intent at the moment it forms rather than requiring developers to context-switch to a dedicated coding interface.

The Model Context Protocol represents Anthropic's bid for infrastructural lock-in that extends beyond its own products. Released as an open standard in November 2024, MCP provides a standardised way to connect AI models to external tools, databases, and data sources. Think of MCP as USB-C for AI applications. Just as USB-C provides a standardised way to connect devices to various peripherals, MCP provides a standardised way to connect AI models to different data sources and tools.

By March 2025, OpenAI had adopted MCP across its Agents SDK and ChatGPT desktop application. Google DeepMind confirmed support in upcoming Gemini models. In December 2025, Anthropic donated MCP to the Agentic AI Foundation under the Linux Foundation, co-founded by Anthropic, Block, and OpenAI, establishing it as a de facto industry standard with governance beyond any single company's control. Since launching MCP, the community has built thousands of MCP servers, SDKs are available for all major programming languages, and the industry has adopted MCP as the standard for connecting agents to tools and data.

This standardisation strategy resembles historical platform plays that created durable competitive advantages. By opening MCP whilst maintaining the most capable implementation, Anthropic benefits regardless of which specific tools developers connect. The protocol becomes plumbing that routes work toward Claude.

The Competitive Landscape Against Google and GitHub

Understanding Claude Code's advantages requires examining what competitors offer and where they fall short in the evolving market for AI coding assistance.

GitHub Copilot excels as an in-IDE coding assistant providing real-time code completions and suggestions as developers type. The tool integrates seamlessly with Visual Studio Code and supports a vast number of programming languages thanks to training on GitHub's enormous repository of code. For accelerating day-to-day coding tasks where developers already know what they want to implement, Copilot's fluid completions and chat capabilities remain compelling. At ten dollars per month compared to Claude Code's fifteen dollars, the price point attracts individual developers and smaller teams with tighter budgets.

But Copilot's architecture reflects an earlier philosophy of AI assistance that predates the agentic paradigm. It augments the developer line by line rather than operating autonomously on larger tasks. When work involves sweeping changes across a repository, API migrations, code style unification, or wide-ranging rename operations, Copilot requires developers to make each individual change manually even if the AI suggests the modifications. The human remains the executor of every action rather than the supervisor of autonomous work.

Google's Gemini Code Assist and Gemini CLI arrived in June 2025 with aggressive positioning that challenged both Copilot and Claude Code. Gemini CLI accumulated over 55,000 GitHub stars within weeks of launch, demonstrating substantial developer interest in alternatives. The tool is completely free with any Google account, providing 1,000 requests per day and 60 requests per minute with no billing setup required. Gemini's context window supports up to one million tokens in long-context beta configurations, theoretically enabling analysis of entire large codebases in a single prompt.

Sourcegraph's decision to make Gemini 3 Pro the default model for Cody, its AI coding assistant used by over 13 million developers through integrations with Visual Studio Code, GitHub, and JetBrains IDEs, provided significant validation for Google's offerings. Internal testing showed notable performance improvements compared with earlier Gemini versions, including more solved tasks, cleaner reasoning, and better handling of massive codebases. The endorsement from a company like Sourcegraph, whose tools are relied upon by engineering teams at Uber, Netflix, and other major technology companies, carried substantial weight in the developer community.

Yet Claude Code maintains advantages that free tiers and generous context windows cannot replicate. The checkpoint system has no equivalent in Gemini's offerings. The subagent architecture enabling parallel workstreams does not exist in competing products at the same level of sophistication. The Slack integration positions Claude Code in communication workflows that competitors have not penetrated. And the enterprise security, privacy, and compliance features that Anthropic has built for its business customers create switching costs once organisations integrate Claude Code into their development infrastructure.

Many development teams have found value in using multiple tools complementarily. Copilot accelerates day-to-day coding tasks whilst Claude Code handles complex project-level work requiring understanding of broader context and autonomous execution. This pattern suggests the market may not consolidate around a single winner but rather stratify around different use cases, price points, and enterprise requirements.

Market Timing and Perception Dynamics

Technical excellence matters, but timing matters more when capturing markets in rapid transition. Anthropic released Claude Code for preview testing in February 2025 and made it generally available in May 2025, precisely when enterprise frustration with existing coding assistants had peaked and enthusiasm for agentic capabilities had reached fever pitch.

The Menlo Ventures data tells the positioning story with striking clarity. In 2023, OpenAI dominated 50 per cent of the enterprise large language model market whilst Anthropic held merely 12 per cent. By August 2025, Anthropic commanded 32 per cent of enterprise LLM utilisation overall, and 54 per cent specifically within coding use cases. Google captured 20 per cent whilst Meta's Llama held 9 per cent. The shift reflected a perception crystallising across enterprise technology leadership: Claude models produced more reliable outputs with lower hallucination rates than alternatives.

Anthropic's revenue trajectory reinforced this perception with exponential growth that surprised industry observers. The company hit two billion dollars in annualised revenue in Q1 2025, more than doubling from the prior period. By the end of May 2025, revenue reached approximately three billion dollars. By October 2025, Sacra estimated annualised revenue at seven billion dollars. Revenue had grown tenfold annually for three consecutive years. The company projects nine billion dollars in annualised revenue by end of 2025 and between twenty and twenty-six billion dollars in 2026.

Perhaps the most significant market dynamic involves Anthropic's customer composition, which differs fundamentally from its primary competitor. Whilst OpenAI generates approximately 85 per cent of its revenue from individual ChatGPT subscriptions, Anthropic derives 85 per cent from business customers. The company's customer base expanded from fewer than one thousand businesses to over 300,000 in just two years. This enterprise concentration creates different incentive structures. Anthropic builds for organisational workflows rather than consumer novelty, prioritising reliability, security, and integration capabilities over viral features.

High-profile enterprise customers amplify the perception advantage through visible endorsements. Rakuten reported reducing software development timelines from 24 days to 5 days using Claude Code, a 79 per cent reduction that caught widespread industry attention. Netflix, Spotify, and Salesforce operate as enterprise customers. These reference accounts function as social proof that compounds adoption pressure on technology leadership at peer organisations considering AI coding investments.

When Productivity Gains Prove Illusory

The adoption frenzy has obscured an uncomfortable finding that challenges the fundamental value proposition of AI coding assistance.

In July 2025, METR published results from a randomised controlled trial examining how frontier AI tools affected experienced developer productivity. The study recruited 16 developers from large open-source repositories averaging over 22,000 GitHub stars and one million lines of code. These were developers who had contributed to their respective projects for multiple years, with an average of five years of prior experience and 1,500 commits. The methodology was rigorous: developers provided lists of 246 real issues that would be valuable to their repositories, including bug fixes, features, and refactors that would normally be part of their regular work. Each issue was randomly assigned to allow or disallow AI assistance.

The finding shocked the industry. When developers used AI tools, primarily Cursor Pro with Claude 3.5 and 3.7 Sonnet which were frontier models at the time, they took 19 per cent longer to complete tasks than when working without assistance. Before the study, developers had predicted AI would speed them up by 24 per cent. After completing tasks with the measured slowdown, they still believed AI had helped, estimating a 20 per cent improvement. The perception gap between subjective experience and objective measurement was 39 percentage points.

Several factors contributed to the documented slowdown. Developers accepted fewer than 44 per cent of AI-generated code suggestions. The low acceptance rate meant developers spent significant time reviewing, testing, and modifying code only to reject it in the end. The large and complex repositories characteristic of mature software projects proved particularly challenging for AI tools, which performed worse in environments where context exceeded their effective reasoning capacity. The AI tools introduced extra cognitive load and context-switching that disrupted developer workflows rather than enhancing them.

As Zvi Mowshowitz observed in commenting on the METR findings, even researchers who are extremely in-the-know about AI coding abilities failed to predict results accurately. Subjective impressions of productivity are not reliable indicators of actual productivity effects.

The Stack Overflow 2025 Developer Survey corroborated these findings from a different methodological angle. Whilst 84 per cent of developers now use or plan to use AI tools in their development process, up from 76 per cent in 2024, only 33 per cent trust the accuracy of outputs. 46 per cent actively distrust AI-generated code. A mere 3 per cent report highly trusting the output. The biggest frustration, cited by 66 per cent of respondents, involves “AI solutions that are almost right, but not quite.” This leads directly to the second-biggest frustration: debugging AI-generated code consumes more time than writing code manually.

Perhaps most telling: 77 per cent of developers say vibe coding is not part of their professional workflow. Developers show the strongest resistance to AI for high-responsibility tasks like deployment and monitoring, with 76 per cent indicating they will not use AI for these purposes, and project planning, with 69 per cent declining AI assistance.

The Technical Debt Accelerator

If productivity gains prove mixed, the technical debt implications are unambiguous and accumulating rapidly.

GitClear's second annual AI Copilot Code Quality research analysed 211 million changed lines of code from 2020 through 2024, examining trends across anonymised private repositories and 25 of the largest open-source projects. The findings document a fundamental shift in how code accumulates.

The number of code blocks containing five or more duplicated lines increased eightfold during 2024. Lines classified as copy-pasted rose from 8.3 per cent to 12.3 per cent between 2021 and 2024. Simultaneously, the percentage of code changes associated with refactoring collapsed from 25 per cent to less than 10 per cent. In 2024, copy-pasted code surpassed refactored code for the first time in the dataset's history. The researchers also noted a 39.9 per cent decrease in the number of moved lines, another indicator of declining architectural improvement work.

The pattern emerges from AI tools' fundamental design. Code assistants make it trivially easy to insert new blocks by pressing tab to accept suggestions. They are far less likely to propose reusing existing functions elsewhere in the codebase, partly because of limited context awareness and partly because they optimise for immediate completion rather than architectural coherence. As Bill Harding, GitClear's CEO, observed, AI has an overwhelming tendency not to understand what the existing conventions are within a repository and is very likely to come up with its own slightly different version of how to solve a problem.

This creates what researchers call “AI technical debt.” Traditional technical debt accumulates linearly. You skip tests, take shortcuts, defer refactoring, and pain builds gradually until someone allocates a sprint for cleanup. AI technical debt is different. Three vectors interact to produce exponential growth: model versioning chaos as organisations struggle to maintain code generated by different model versions with different behaviours, code generation bloat as volume overwhelms review capacity, and organisational fragmentation as teams develop inconsistent practices.

The Google 2025 DORA Report documented this dynamic empirically with unprecedented scale. Drawing on insights from over 100 hours of qualitative data and survey responses from nearly 5,000 technology professionals worldwide, the research found that higher AI adoption correlates with increased individual effectiveness, software delivery throughput, code quality, product performance, team performance, and organisational performance. But it also correlates with increased software delivery instability. AI accelerates development whilst exposing weaknesses downstream.

The report introduced a new metric: rework rate, quantifying how often teams must deploy unplanned fixes or patches to correct user-facing defects. The metric exists because traditional throughput measures obscured the downstream consequences of rapid AI-assisted development. The 2025 DORA findings emphasise that AI does not fix a team but rather amplifies what already exists. Strong teams use AI to become even better and more efficient, whilst struggling teams find that AI only highlights and intensifies their existing problems.

Forecasts suggest 75 per cent of technology leaders will face moderate to severe technical debt by 2026, up from 50 per cent in 2025. The State of Software Delivery 2025 report found that despite perceived productivity gains, the majority of developers actually spend more time debugging AI-generated code than they did before adopting these tools.

Security Vulnerabilities at Scale

The security implications compound the technical debt problem in ways that create direct enterprise risk.

Veracode's comprehensive analysis of over 100 large language models across 80 coding tasks spanning four programming languages revealed that only 55 per cent of AI-generated code was secure. AI-generated code introduced security flaws in 45 per cent of tests. Some programming languages proved especially problematic. Java had the highest failure rate, with LLM-generated code introducing security flaws more than 70 per cent of the time. Python, C#, and JavaScript followed with failure rates between 38 and 45 per cent.

Specific vulnerability types proved particularly resistant to AI mitigation. 86 per cent of code samples failed to defend against cross-site scripting. 88 per cent were vulnerable to log injection attacks. The researchers evaluated LLMs of varying sizes, release dates, and training sources over multiple years. Whilst models improved at writing functional or syntactically correct code, they showed no improvement at writing secure code. Security performance remained flat regardless of model size or training sophistication. This finding challenges assumptions that capability improvements would naturally extend to security outcomes.

The CodeRabbit State of AI vs Human Code Generation Report found AI-generated code creates 1.75 times more logic and correctness errors, 1.64 times more code quality and maintainability errors, 1.57 times more security findings, and 1.42 times more performance issues compared to human-written code. AI-generated code was 2.74 times more likely to introduce cross-site scripting vulnerabilities, 1.91 times more likely to make insecure object references, 1.88 times more likely to introduce improper password handling, and 1.82 times more likely to implement insecure deserialisation.

The root problem is that AI coding assistants do not inherently understand an application's risk model, internal standards, or threat landscape. This disconnect introduces systemic risks not just in individual lines of code but in logic flaws, missing controls, and inconsistent patterns that erode security posture over time. Today's foundational LLMs train on the vast ecosystem of open source code, learning by pattern matching. If an unsafe pattern like string-concatenated SQL queries appears frequently in training data, the assistant will readily produce it.

Real-world incidents have already demonstrated the consequences. In May 2025, security researcher Matt Palmer discovered that Lovable, a prominent vibe coding platform enabling users to build web applications through natural language prompts, had a critical vulnerability enabling anyone to access user information including names, email addresses, financial information, and API keys across 170 applications built on the platform. The vulnerability stemmed from misconfigured Row Level Security policies that AI-generated code failed to implement correctly. Palmer emailed Lovable with detailed vulnerability reports in March 2025, but the company's subsequent security scan feature only flagged the presence of Row Level Security policies, not whether they actually worked.

By mid-2025, AI code had triggered over 10,000 new security findings per month across major code repositories. A benchmark report found pull requests per author increased 20 per cent year-over-year even as incidents per pull request increased 23.5 per cent and change failure rates rose approximately 30 per cent.

Enterprise Governance Requirements

The transition from developer novelty to enterprise infrastructure demands organisational capabilities that most companies have not yet developed.

The OWASP GenAI Security Project released its Top 10 for Agentic Applications in December 2025, reflecting input from over 100 security researchers, industry practitioners, and technology providers. The framework identifies risks specific to autonomous AI agents including goal hijacking, where attackers manipulate agent objectives through prompt injection or poisoned data, and tool misuse, where agents use legitimate authorised capabilities for data exfiltration or destructive actions. The framework has already seen adoption by major technology providers including Microsoft and NVIDIA.

OWASP introduces the concept of “least agency” as an evolution of traditional least privilege principles. Rather than merely restricting what permissions an agent has, organisations must restrict the autonomy an agent can exercise. Only grant agents the minimum autonomy required to perform safe, bounded tasks. This conceptual shift acknowledges that agentic systems require different governance approaches than traditional software or even traditional AI applications.

The enterprise governance challenge extends beyond security into operational complexity. Traditional AI governance practices including data governance, risk assessments, explainability, and continuous monitoring remain essential, but governing agentic systems requires addressing their autonomy and dynamic behaviour. A key challenge involves controlling what actions non-human identities can perform, including data flow destinations, volumes, formats, and access to external or sensitive resources.

The scale of the identity management challenge is staggering. The average enterprise now faces an 82:1 machine-to-human identity ratio. Every machine identity represents a potential point of compromise. Adding autonomous decision-making expands the attack surface dramatically. Enterprises now require security rules and permission frameworks defining what data agents can access and what actions they are allowed to take, observability into agent actions and decision-making, and agent registries and workflow versioning to track how agents evolve over time.

The EU AI Act's high-risk provisions take effect in August 2026, with penalties reaching 35 million euros or 7 per cent of global revenue. Colorado's AI Act follows in June 2026. FINRA's 2026 Oversight Report positions AI governance as a core compliance issue rather than a future consideration for financial services firms. High-risk systems require documented evidence of governance: how systems were designed, how risks were assessed, how human oversight works, and how performance is monitored over time. Policy statements and principles are insufficient. Regulators expect architectural proof that controls exist and function.

Code review processes must evolve correspondingly. By the end of 2025, AI-assisted development accounted for nearly 40 per cent of all committed code globally. Leaders report that review capacity, not developer output, has become the limiting factor in delivery. A well-governed AI code review system must preserve human ownership of the merge decision whilst raising baseline quality of every pull request, reduce back-and-forth iteration, and ensure reviewers only engage with work that genuinely requires their experience.

Incident Response for Agentic Failures

Production failures from AI-generated code require incident response capabilities that most organisations lack.

In July 2025, an AI coding assistant deleted a customer's production database without instructions to do so. The AI system did not follow post-incident commands from the developer to stop making further unwanted changes. This incident illustrates a failure mode unique to agentic systems: they can continue causing damage after problems are detected if proper controls are not in place. Other documented incidents include a commercial AI agent asked merely to check egg prices that instead purchased eggs without user consent, and an AI coding assistant that moved files such that neither the agent nor the human operator could find them.

The pattern of agentic failures differs qualitatively from traditional software bugs. Traditional bugs are deterministic. Given the same inputs, they produce the same outputs. Agentic failures emerge from the interaction between model reasoning, context interpretation, and tool access. They can be non-reproducible, making debugging difficult. They can cascade, as agents respond to their own errors by taking additional problematic actions.

Incident response for agentic systems requires capabilities including detecting problems early through observability into agent actions and decision-making, communicating what happened through interpretable logging that captures agent reasoning, fixing issues quickly through mechanisms to halt agent operations and revert changes, and capturing near misses through documentation that enables learning before failures reach production.

ISACA recommends implementing governance frameworks that ensure AI coding assistants are tested, audited, and synchronised with enterprise risk appetite. This involves requiring human-in-the-loop approval for high-impact actions, reporting AI decision-making, and ensuring audits are interpretable.

The cost implications of failure are substantial. The average security breach costs 4.45 million dollars, with potential tens of millions more in brand damage, regulatory fines, and legal exposure. GDPR violations alone can reach 4 per cent of global revenue. One in five organisations have already suffered material damage from AI-generated code. Cyber insurance has not caught up with AI-specific risks, and liability questions around who bears responsibility when AI writes vulnerable code remain unresolved.

Building Mature Organisational Capabilities

The path forward requires treating AI-generated code differently from human-written code at every stage of the software lifecycle.

Architectural oversight must remain human territory. AI coding agents excel at generating correct code but perform poorly at making correct design and architecture decisions independently. If allowed to proceed without oversight, they will write functional code whilst accruing technical debt rapidly. The emerging pattern treats AI as the driver in pair programming whilst humans serve as navigators directing overall strategy, making architectural decisions, and reviewing generated code.

Review processes need tiering based on risk. Security-critical code paths require more rigorous human review than cosmetic changes. Changes to authentication, authorisation, payment processing, and data handling warrant heightened scrutiny regardless of origin. Static analysis should run automatically on all AI-generated code before human review begins.

Verification tooling must become standard infrastructure. AI-powered remediation tools that automatically detect and fix flaws in generated code can reduce vulnerability rates by over 60 per cent when combined with human oversight. Software composition analysis ensures AI-generated code does not introduce vulnerabilities from third-party dependencies.

The 2025 DORA Report identifies seven essential competencies for effective AI adoption including a clear organisational stance on AI governance, high-quality data ecosystems, AI-accessible internal systems, robust version control, small-batch delivery practices, user-centric feedback loops, and strong internal platforms. Research shows a direct correlation between high-quality internal platforms and an organisation's ability to unlock AI value, making platform engineering an essential foundation for success.

Training programmes must address the skill atrophy concern. If developers stop writing code manually, they may lose the ability to understand and debug complex systems. The solution involves treating AI code generation as augmentation rather than replacement, ensuring developers maintain fundamental competencies even whilst leveraging AI for acceleration.

Sustainability of the Current Narrative

The narrative of unlimited software generation confronts hard limits as organisations accumulate experience.

The productivity paradox documented by METR suggests that AI tools accelerate inexperienced developers working on unfamiliar code whilst potentially slowing experienced developers working on code they understand deeply. The economic implications are counterintuitive. AI coding tools may provide the most value precisely where organisations need it least, on new projects with less experienced teams, whilst providing the least value where organisations need it most, on mature codebases with experienced maintainers.

The technical debt curve suggests current practices are borrowing against future development velocity. Code that ships quickly today creates debugging burdens tomorrow. The 8x increase in code duplication documented by GitClear represents maintenance obligations that compound over time. At some threshold, the accumulated debt consumes more engineering time than AI tools save.

The security exposure curve follows a similar trajectory. As AI-generated code proliferates through production systems, the attack surface expands correspondingly. The 45 per cent vulnerability rate documented by Veracode, multiplied by the volume increase in AI-generated code, produces absolute vulnerability counts that overwhelm traditional security review processes.

Enterprise adoption will continue accelerating regardless. Gartner predicts 40 per cent of enterprise applications will embed AI agents by end of 2026, up from less than 5 per cent in 2025. The agentic AI market is projected to surge from 7.8 billion dollars today to over 52 billion dollars by 2030.

But sustainable adoption requires governance maturity that matches technical capability. The organisations that will succeed are those that understand that agentic tools amplify existing capabilities rather than replacing them. Strong teams become stronger. Struggling teams see their problems intensify.

The current moment resembles previous technology adoption cycles where early euphoria confronted operational reality. The cloud computing transition promised infinite scalability but required years of organisational learning around cost management, security practices, and operational procedures. Mobile development promised universal reach but demanded new expertise in platform-specific constraints, offline operation, and battery efficiency.

Agentic coding tools represent a similarly significant transition. The tools are genuinely transformative. But transformative tools require transformed organisations to wield them effectively. The organisations racing to maximise AI-generated code volume without building corresponding governance, review, and incident response capabilities are constructing technical debt, security exposure, and operational risk that will constrain their future options.

The question is not whether AI will write most code. It almost certainly will. The question is whether organisations will develop the maturity to ensure that code serves their interests over the long term rather than creating liabilities that compound faster than the productivity gains that justified adoption.

---

References and Sources

1. Anthropic. “Enabling Claude Code to work more autonomously.” Anthropic News (September 2025). https://www.anthropic.com/news/enabling-claude-code-to-work-more-autonomously

2. TechCrunch. “Claude Code is coming to Slack, and that's a bigger deal than it sounds.” TechCrunch (December 2025). https://techcrunch.com/2025/12/08/claude-code-is-coming-to-slack-and-thats-a-bigger-deal-than-it-sounds/

3. Anthropic. “Introducing the Model Context Protocol.” Anthropic News (November 2024). https://www.anthropic.com/news/model-context-protocol

4. MarkTechPost. “Now It's Claude's World: How Anthropic Overtook OpenAI in the Enterprise AI Race.” MarkTechPost (August 2025). https://www.marktechpost.com/2025/08/04/now-its-claudes-world-how-anthropic-overtook-openai-in-the-enterprise-ai-race/

5. Sacra. “Anthropic revenue, valuation & funding.” Sacra (2025). https://sacra.com/c/anthropic/

6. METR. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR (July 2025). https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

7. Stack Overflow. “2025 Developer Survey.” Stack Overflow (2025). https://survey.stackoverflow.co/2025/

8. GitClear. “AI Copilot Code Quality: 2025 Data Suggests 4x Growth in Code Clones.” GitClear (2025). https://www.gitclear.com/ai_assistant_code_quality_2025_research

9. Google Cloud. “2025 DORA Report: State of AI-Assisted Software Development.” Google DORA (2025). https://dora.dev/research/2025/dora-report/

10. Veracode. “2025 GenAI Code Security Report.” Veracode (October 2025). https://www.veracode.com/resources/analyst-reports/2025-genai-code-security-report/

11. CodeRabbit. “AI vs human code gen report: AI code creates 1.7x more issues.” CodeRabbit (2025). https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report

12. Semafor. “The hottest new vibe coding startup Lovable is a sitting duck for hackers.” Semafor (May 2025). https://www.semafor.com/article/05/29/2025/the-hottest-new-vibe-coding-startup-lovable-is-a-sitting-duck-for-hackers

13. OWASP. “Top 10 for Agentic Applications for 2026.” OWASP GenAI Security Project (December 2025). https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

14. ISACA. “Avoiding AI Pitfalls in 2026: Lessons Learned from Top 2025 Incidents.” ISACA Now Blog (2025). https://www.isaca.org/resources/news-and-trends/isaca-now-blog/2025/avoiding-ai-pitfalls-in-2026-lessons-learned-from-top-2025-incidents

15. Chroma Labs. “Context Rot: How Increasing Input Tokens Impacts LLM Performance.” Chroma Research (2025). https://research.trychroma.com/context-rot

16. SkyWork AI. “Claude Code vs GitHub Copilot (2025): Complete Comparison Guide.” SkyWork AI (2025). https://skywork.ai/blog/claude-code-vs-github-copilot-2025-comparison/

17. TechStartups. “Google's Gemini 3 outperforms Claude in coding benchmarks; Sourcegraph adopts it for millions of developers.” TechStartups (December 2025). https://techstartups.com/2025/12/05/googles-gemini-3-outperforms-claude-in-coding-benchmarks-sourcegraph-adopts-it-for-millions-of-developers/

18. Zvi Mowshowitz. “2025 Year in Review.” Don't Worry About the Vase, Substack (2025). https://thezvi.substack.com/p/2025-year-in-review

---

Tim Green UK-based Systems Theorist and Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

You have never walked anonymously. You may have believed otherwise, pulling a hood over your head or choosing the busy side of the street, but the truth has been catching up for years. The way you shift your weight from one foot to the other, the cadence of your stride, the particular rhythm of your fingers on a keyboard, even the micro-fluctuations in your voice when you order a coffee: all of these patterns are, increasingly, as identifiable as your fingerprint. And unlike your fingerprint, you leave them everywhere, involuntarily, continuously, without ever pressing your thumb to glass.

Artificial intelligence systems can now identify individuals through subtle behavioural patterns and voice characteristics with startling accuracy. Gait recognition software deployed on the streets of Beijing and Shanghai can pick you out of a crowd from 50 metres away, even with your back turned and your face completely covered. Voice biometric systems in banking can authenticate your identity from a few seconds of speech. Wi-Fi signals bouncing off your body as you walk through a room can betray your identity through walls. The question is no longer whether these technologies work. It is what their proliferation means for the very concept of being unknown in a public space, and whether truly private human interaction remains possible in an age of pervasive, ambient identification.

The Expanding Biometric Frontier Beyond the Face

For over a decade, the surveillance debate has centred on facial recognition. Cities have banned it. Activists have marched against it. Researchers like Joy Buolamwini at the MIT Media Lab have exposed its profound racial biases, demonstrating through her landmark 2018 Gender Shades study that commercial facial analysis systems from IBM, Microsoft, and Face++ misclassified darker-skinned women at rates as high as 47 per cent while achieving error rates below 1 per cent for lighter-skinned men. Her work, co-authored with Timnit Gebru, catalysed a reckoning that led every audited US-based company to stop selling facial recognition technology to law enforcement by 2020.

But while the world was arguing about faces, a quieter revolution was unfolding. Behavioural biometrics, the science of identifying people through how they move, type, speak, and interact with the physical world, has advanced rapidly and without the same degree of public scrutiny. Unlike facial recognition, which requires a camera pointed at your face, behavioural biometrics can operate at a distance, through obstacles, and without the subject's knowledge or cooperation. This makes it, in many respects, a far more consequential threat to anonymity than the technology that has dominated headlines.

The gait biometrics market was valued at USD 1.25 billion in 2024 and is projected to reach USD 3.41 billion by 2032, growing at a compound annual growth rate of 13.38 per cent. Security agencies accounted for roughly 44 per cent of that market in 2024, with Asia-Pacific expected to see the fastest growth at 15.15 per cent annually through 2032. These are not speculative projections from fringe analysts; they reflect sustained investment by governments and corporations in technologies that identify you not by what you look like, but by what you do.

Gait Recognition Comes of Age

The idea that every person walks differently is not new. Forensic investigators have long known that gait is distinctive. What is new is the ability of AI systems to extract that distinctiveness from ordinary surveillance footage and match it against databases at scale.

The Chinese AI company Watrix, incubated by the Chinese Academy of Sciences, has developed gait recognition software that extracts a person's silhouette from video and analyses its movement to create a model of how that person walks. According to Watrix CEO Huang Yongzhen, the technology has been trialled by police in Beijing, Shanghai, and Chongqing, and a pilot system operates in Hubei and Guangdong provinces. The system can identify individuals from up to 50 metres away, from any angle, even when faces are covered and in darkness. “With facial recognition people need to look into a camera,” Huang told the South China Morning Post. “Cooperation is not needed for them to be recognised by our technology.”

The accuracy Watrix claims is striking: up to 96 per cent. The company, which was inspired by a US Defence Advanced Research Projects Agency (DARPA) study, has been in discussions with security firms in Singapore, India, Russia, the Netherlands, and the Czech Republic. Security officials in China's Xinjiang province, where the Uyghur Muslim population faces intense surveillance, have also expressed interest. The technology is not merely supplementary to existing surveillance; it fills the gaps that facial recognition cannot reach. It operates in conditions where faces are obscured, where lighting is poor, and where subjects are unaware they are being watched. Every person's posture, Huang has stated, is unique, like a fingerprint, and gait recognition is capable of identifying targets from any angle.

Nor is Watrix alone. In September 2024, NEC Corporation launched a gateless biometric authentication system capable of authenticating 100 people per minute while they are in motion. The system, initially deployed at NEC's Tokyo headquarters in July 2024, combines face recognition with gait-based matching technology to identify individuals in crowded areas without requiring them to stop or present credentials. NEC, which has been ranked first in face recognition benchmark tests by the US National Institute of Standards and Technology since 2009, has deployed its biometric technology in more than 50 countries and across 80 airports globally. The new system is being offered in Japan, the United States, and Singapore, and uses deep learning to re-identify tracked individuals even after they pass behind obstructions or through dense crowds.

The Voice That Gives You Away

Your voice is another behavioural signature that AI systems are learning to read with uncomfortable precision. Voice biometrics analyse characteristics including pitch, tone, cadence, and the physical properties of your vocal tract to create a unique voiceprint. Financial institutions have been early adopters: customers can authenticate transactions simply by speaking. The technology is marketed as frictionless and secure, a way to eliminate passwords and PINs. But a voiceprint, once captured, is not a password. It cannot be changed if compromised. And the infrastructure for capturing voice data is already ubiquitous: every smartphone, every smart speaker, every customer service line.

But the same technology that verifies your identity can also compromise it. Voice recordings are biometric identifiers as sensitive as fingerprints or retinal scans, yet they can be captured from a distance, harvested from voicemail messages, or scraped from social media posts. According to the 2024 Javelin Identity Fraud Study, American consumers lost more than USD 47 billion to identity fraud that year, with AI-generated synthetic identity fraud and voice cloning driving much of that figure. A survey by BioCatch found that 91 per cent of US banks are reconsidering voice biometric authentication due to AI cloning risks.

The threat is not theoretical. In April 2025, Hong Kong police dismantled a deepfake scam ring that used AI-generated video and cloned voice attacks to open accounts at HSBC, resulting in losses exceeding HK 1.5 billion, approximately USD 193.2 million. The UK government has published a briefing note on the ethical issues arising from public sector use of biometric voice recognition technology, acknowledging the tensions between convenience, security, and privacy. Some institutions store biometric voice templates indefinitely or share them with third-party vendors for AI training purposes, often without the knowledge of the individuals whose voices are on file.

The US Department of Justice has affirmed a broad definition of biometric identifiers that encompasses facial images, voiceprints and patterns, retina and iris scans, palm and fingerprints, and behavioural data such as gait and keyboard usage patterns. This definitional expansion matters because it signals that regulators are beginning to recognise what technologists have known for some time: the body is a broadcasting device, and everything it broadcasts can be recorded, analysed, and matched to an identity.

The Invisible Biometric Layer

Beyond gait and voice, there exists an entire category of behavioural biometrics that most people never consider. Keystroke dynamics, the study of how you type, can identify individuals based on the timing between key presses, the duration for which each key is held, and the rhythm of your overall typing pattern. These measurements, captured at millisecond precision, create a biometric template that is unique to each person and extremely difficult to replicate.

Research published in Discover Applied Sciences in 2025 highlights that keystroke dynamics can be used for continuous, real-time authentication, with any deviation from established typing patterns triggering an alert for possible unauthorised access. A 2024 study published in Sensors demonstrated that deep learning architectures combining convolutional and recurrent neural networks achieve high effectiveness in identifying users based on typing patterns. Forensic applications are also emerging: regardless of the number of machines a person uses, their typing pattern persists, making keystroke dynamics a potential tool for identifying anonymous online activity.

The transparency of this technology is part of what makes it so consequential. Keystroke dynamics require no specialised hardware. They operate via backend software implementation, and in most cases users are entirely unaware they are being profiled. This passive, invisible collection of behavioural data represents a fundamentally different kind of surveillance from a camera on a pole or a guard at a door. It is ambient, continuous, and nearly impossible to evade. Research from MDPI in 2023 also found that keystroke authentication is influenced by the language being typed, meaning bilingual users produce distinct profiles for each language, further enriching the data available for identification.

Wi-Fi Signals as a Surveillance Medium

Perhaps the most unsettling frontier in behavioural identification is the use of ordinary Wi-Fi signals to detect, track, and identify people. Wi-Fi sensing exploits the way radio signals interact with human bodies: as you move through a space, you cause reflections, refractions, and attenuations in the Wi-Fi signal, and these disturbances encode information about your body shape, movement patterns, and activities.

A comprehensive 2024 survey published in ACM Transactions on Sensor Networks documents how researchers have used Channel State Information from Wi-Fi signals to identify individuals based on their unique gait patterns, achieving accuracy rates above 90 per cent. Unlike camera-based systems, Wi-Fi sensing works through walls, in complete darkness, and without requiring any device to be carried by the subject. The technology leverages existing infrastructure, requiring only standard Wi-Fi access points and receiving devices.

Research published in Engineering Applications of Artificial Intelligence demonstrated human activity recognition through walls using deep learning models applied to Wi-Fi CSI data. The PA-CSI model, which combines phase and amplitude analysis with attention mechanisms, has achieved accuracy rates of up to 99.9 per cent on benchmark datasets. A specialised system called WiFind can detect, localise, and estimate body pose through walls, debris, and smoke, using nodes costing under USD 150 each.

The implications are stark. A person walking through a building equipped with standard Wi-Fi infrastructure could, in principle, be continuously tracked and identified without any visible surveillance equipment, without their knowledge, and without any possibility of covering their face or altering their appearance to avoid detection. Wi-Fi technology has evolved from its initial 802.11 standard to Wi-Fi 6 and the anticipated Wi-Fi 7, with each generation improving the resolution and sensitivity of sensing capabilities. The physical world is becoming readable in ways that were previously confined to science fiction.

Regulation Struggles to Keep Pace

The regulatory response to behavioural biometric surveillance has been fragmented and reactive, consistently trailing the technology it seeks to govern. The most significant legislative development has been the European Union's AI Act (Regulation (EU) 2024/1689), which entered into force on 1 August 2024 and began enforcing prohibitions on certain AI systems from 2 February 2025.

Article 5 of the AI Act prohibits real-time remote biometric identification systems in publicly accessible spaces for law enforcement, with limited exceptions. It bans AI systems that scrape facial images from the internet or CCTV footage, and prohibits biometric categorisation systems that deduce race, political opinions, religious beliefs, or sexual orientation from biometric data. Violations carry fines of up to 35 million euros or 7 per cent of global annual turnover, whichever is higher.

Yet the Act contains significant exceptions for law enforcement, allowing real-time biometric identification for targeted searches of victims of abduction or trafficking, prevention of imminent threats, and prosecution of serious crimes, all subject to judicial authorisation. These carve-outs have drawn criticism from organisations like European Digital Rights (EDRi), which argues they may legitimise the very practices the Act purports to ban. As a Stanford Law School analysis noted, despite omitting an outright ban on facial recognition in publicly accessible spaces, the AI Act will probably show its full potential in the years after its entry into force.

In the United States, Illinois' Biometric Information Privacy Act remains the strongest state-level protection, granting individuals a private right of action and statutory damages of USD 1,000 per negligent violation and USD 5,000 per intentional violation. BIPA class action settlements totalled more than USD 206 million in 2024, including the landmark Clearview AI settlement in which class members received a 23 per cent equity stake in the company, valued at approximately USD 51.75 million. In August 2024, Illinois amended BIPA to limit damages to one violation per person regardless of how many times data was collected, a change that contributed to a sharp decline in new filings. Clearview AI itself had amassed a database of more than 60 billion facial images scraped from social media platforms, news websites, and other publicly accessible online sources, prompting the wave of litigation.

San Francisco became the first US city to ban government use of facial recognition in May 2019, with Supervisor Aaron Peskin declaring, “We all support good policing but none of us want to live in a police state.” Yet even this landmark ordinance had limits: it carved out exceptions for federal facilities and did not apply to private businesses. Moreover, in the five years since the ban, San Francisco police admitted to circumventing it on six separate occasions.

The UK's Information Commissioner's Office launched its AI and biometrics strategy in June 2025, focusing on situations where risks are highest and public concern is clearest. The ICO plans to set a high threshold of lawfulness for AI systems that infer subjective traits, intentions, or emotions based on physical or behavioural characteristics. Public polling cited in the strategy found that 54 per cent of UK adults have concerns about facial recognition technology impacting civil liberties, and that concern about AI use for welfare eligibility has risen from 44 per cent to 59 per cent between 2022 and 2025.

When the Wrong Person Gets Caught

The dangers of these systems are not abstract. In January 2020, Robert Williams, a Black man living in Farmington Hills, Michigan, was arrested outside his home in front of his wife and two young daughters by Detroit police. He was detained for thirty hours in an overcrowded, dirty cell. The arrest was based on a facial recognition match from a blurry surveillance image of a shoplifting suspect at a Shinola store in Detroit. Williams was actually the ninth-best match in the system's results, and detectives had not investigated his whereabouts before making the arrest.

Williams' case, brought to public attention by the ACLU and the University of Michigan Law School's Civil Rights Litigation Initiative, became the first publicly reported instance of a false facial recognition match leading to a wrongful arrest. On 28 June 2024, the parties reached a groundbreaking settlement that established the nation's strongest police department policies constraining law enforcement's use of facial recognition, including a prohibition on arrests based solely on facial recognition results and mandatory training on the technology's risks and its higher misidentification rates for people of colour.

Williams' case was one of three known wrongful arrests in Detroit where police relied on facial recognition technology. All three individuals wrongfully arrested were Black. This pattern underscores the findings of Buolamwini's Gender Shades study and raises a critical question about behavioural biometrics more broadly: if the training data and deployment contexts of gait recognition, voice identification, and other behavioural systems reproduce the same biases, the consequences for marginalised communities could be severe. The Pew Research Center found in a 2022 survey that 28 per cent of Black adults said police would definitely make more false arrests if facial recognition were widely adopted, compared with just 11 per cent of white adults.

Surveillance and the Death of Free Assembly

The surveillance theorist and Harvard professor Shoshana Zuboff has described surveillance capitalism as “the unilateral claiming of private human experience as free raw material for translation into behavioural data.” Her framework, articulated in The Age of Surveillance Capitalism (2019), identifies a fundamental shift in which human experiences are extracted, computed, and sold as prediction products. “We are not surveillance capitalism's 'customers,'” Zuboff writes. “We are the sources of surveillance capitalism's crucial surplus.”

When behavioural identification systems operate in public spaces, they do not merely observe; they transform public space itself. Research consistently demonstrates that surveillance produces measurable “chilling effects” on freedom of expression, assembly, and political participation. A qualitative study published in the Journal of Human Rights Practice (Oxford Academic) documented, through interviews with 44 participants in Uganda and Zimbabwe, how the fear of surveillance undermines trust and interpersonal relationships, creating spirals of paranoia and mistrust that directly affect the right to freedom of assembly.

These findings extend well beyond authoritarian contexts. In the United States, the Department of Commerce's National Telecommunications and Information Administration found in a survey of 41,000 households that one in five Americans avoided online activity because of concerns about government data collection. The Electronic Frontier Foundation has highlighted studies showing that government surveillance discourages both speech and access to information on the internet. When people know they are being watched, they change what they say, where they go, and whom they associate with. The extension of these chilling effects from the digital to the physical realm, through gait recognition cameras, voice identification microphones, and Wi-Fi sensing systems, represents a qualitative escalation.

Amnesty International's Ban the Scan campaign, launched in 2021, has mapped the surveillance landscape of New York City, documenting more than 25,500 CCTV cameras across the city and revealing that the NYPD used facial recognition technology in 22,000 cases since 2017. The campaign found that the most surveilled neighbourhood across three boroughs was an area in Brooklyn with a population comprising 54.4 per cent Black residents, underscoring the racialised geography of surveillance infrastructure. Amnesty further documented how facial recognition was used at Black Lives Matter protest sites in 2020 to identify, track, and harass people exercising their rights to peaceful assembly.

The UN Office of the High Commissioner for Human Rights is preparing a thematic report, expected at the 62nd session of the Human Rights Council, analysing the impact of digital and AI-assisted surveillance on assembly and association rights, including chilling effects. A 2025 study in Big Data and Society, examining Extinction Rebellion protests in The Hague, revealed that surveillance technology produces effects beyond traditional chilling, including what researchers termed “hyper-transparency” and “hyper-alertness” among both protesters and police.

The Rise of Anti-Surveillance Fashion

The proliferation of behavioural identification systems has given rise to a counter-movement that sounds like it belongs in a cyberpunk novel but is entirely real: adversarial fashion. These are garments, accessories, and cosmetic techniques designed to confuse, disrupt, or defeat AI surveillance systems.

Artist and researcher Adam Harvey pioneered this field with his CV Dazzle project in 2010, which used cubist-inspired makeup patterns to defeat facial detection algorithms. The technique, named after the dazzle camouflage developed by British painter Norman Wilkinson for Allied ships during the First World War, works by obscuring key facial features until recognition systems can no longer detect a human face. Harvey followed this with HyperFace in 2016, which takes the opposite approach: rather than hiding faces, it floods the environment with false face-like patterns printed on fabric, exploiting algorithms' preference for the highest-confidence facial region.

More recently, the Italian company Cap_able has developed a patented process that algorithmically generates adversarial patterns, translating them into knitted garments that retail between USD 300 and USD 600. These garments combine visual adversarial patterns with infrared protection, aiming to disrupt both optical and thermal surveillance. Researchers have also published work on thermally activated dual-modal adversarial clothing that can defeat both visible-light cameras and infrared sensors simultaneously.

However, as technologist Adam Harvey himself has cautioned, “Camouflage, in general, should be considered temporary, but especially technical camouflage that targets quickly evolving algorithms.” The arms race between surveillance systems and countermeasures is inherently asymmetric: updating a neural network is cheaper and faster than redesigning a wardrobe. Moreover, the very act of wearing obviously adversarial clothing in a public space draws human attention, potentially marking the wearer as suspicious even as it confuses the machines.

The video surveillance industry, enhanced by AI, is projected to grow from USD 3.90 billion in 2024 to USD 12.46 billion by 2030, according to market research. Against this scale of investment, adversarial fashion remains a niche countermeasure, meaningful as a statement of resistance but limited as a practical solution to the erosion of anonymity.

Europe's Border Experiment in Behavioural Biometrics

While much of the debate about behavioural biometrics focuses on domestic surveillance, the technology is also reshaping the boundaries of national security and border control. The European Union's PopEye project, funded through a Horizon Europe grant of 3.2 million euros, represents a significant step towards integrating gait recognition into border security infrastructure.

PopEye, an acronym for “robust Privacy-preserving biOmetric technologies for Passengers' identification and verification at EU external borders maximising the accuracY, reliability and throughput of the rEcognition,” aims to identify individuals on the move, at distances of up to 200 metres, without requiring them to stop. The project combines gait recognition with 3D facial recognition, addressing the limitations of each technology when used in isolation.

The project follows a 2021 Frontex study that examined gait recognition in depth, suggesting that video, radar, and floor sensors could be used to identify people by how they walk. Led by the European Association for Biometrics, PopEye involves partners including the AIT Austrian Institute of Technology, Idiap Research Institute, KU Leuven, and Vrije Universiteit Brussel, among others. Pilot programmes are being conducted at the external borders of Romania and Finland, with the Finnish Ministry of Interior and Romanian Border Authorities serving as key participants.

The project's emphasis on privacy preservation and compliance with the EU's AI Act and GDPR reflects an awareness that the technology it develops operates in a sensitive legal and ethical space. Researchers from VUB and KU Leuven are leading efforts on integrated impact assessments to safeguard human rights and data protection. Yet the fundamental tension remains: a system designed to identify people at a distance, without their cooperation, is inherently a surveillance technology, regardless of the procedural safeguards that surround it.

The Economics of Knowing Who You Are

The security technologist Bruce Schneier, a fellow and lecturer at Harvard's Kennedy School and board member of the Electronic Frontier Foundation, has written extensively about the economics of surveillance and the asymmetries of power it creates. “It is poor civic hygiene to install technologies that could someday facilitate a police state,” Schneier has warned. He has illustrated the collapsing cost of surveillance with a telling comparison: covert human surveillance of an individual costs approximately USD 175,000 per month, while obtaining location information from a mobile provider costs as little as USD 30 per month.

Behavioural biometrics push this economic logic further still. Gait recognition can operate using existing CCTV infrastructure. Keystroke dynamics require only software. Wi-Fi sensing leverages networks that are already installed in virtually every commercial and institutional building. The marginal cost of identifying one additional person approaches zero, which means the economic incentive to deploy these systems is enormous and the barriers to mass deployment are vanishingly small.

This economic reality creates what Schneier has called an alliance of interests between corporate and government surveillance. Corporations collect behavioural data for authentication, fraud prevention, and customer profiling. Governments seek the same data for security, immigration enforcement, and law enforcement. The data collected for one purpose inevitably becomes available for others, a phenomenon that privacy advocates call “function creep.” The US Consumer Financial Protection Bureau has already issued guidance stating that biometric information, including keystroke frequency and behavioural monitoring, used in employment decisions must comply with the Fair Credit Reporting Act.

Zuboff's analysis cuts deeper. She argues that “the power to predict human behaviour is the power to modify human behaviour, and this is what surveillance capitalism is all about.” When every public interaction can be linked to a known identity through behavioural patterns, the entire notion of a public sphere where individuals can move, speak, and associate without being tracked becomes an anachronism. The right to privacy, she insists, is not merely about data protection; it is about the conditions necessary for human autonomy: “what should become data in the first place, that is where the line has to be drawn.”

What Remains of Anonymity

The question this article set out to address, what does the rise of AI behavioural recognition mean for anonymity in public spaces, has a disquieting answer. The technological trajectory is clear: identification systems are becoming cheaper, more accurate, more pervasive, and harder to evade. They are moving beyond the face into the body's every motion and utterance. They work through walls, in darkness, and across distances that make consent meaningless.

The regulatory response, while significant in certain jurisdictions, remains fragmented and reactive. The EU's AI Act represents the most comprehensive attempt at governance, but its exceptions for law enforcement create significant loopholes. BIPA has produced substantial financial penalties in the United States, but it is a single state's law, and its recent amendments have blunted its deterrent effect. The UK's ICO strategy is still in its early stages. Globally, there is no coherent framework for governing technologies that can identify people from their walk, their voice, or the way they type.

What is at stake is not merely a technical question about privacy settings or data policies. It is a question about the kind of society that emerges when public spaces become zones of continuous, ambient identification. Research on the chilling effects of surveillance demonstrates that when people believe they are being watched, they modify their behaviour, self-censor their speech, and withdraw from political participation. The extension of surveillance from visible cameras to invisible behavioural identification systems does not reduce this effect; it amplifies it, because there is no way to know when you are and are not being observed.

Truly private human interaction in public spaces, a conversation with a stranger that no system records, a protest march where participants cannot be individually identified, a walk through a city where your movements are not logged and matched against a database, is becoming technologically impossible. This does not mean it will vanish entirely; enforcement gaps, technical limitations, and deliberate resistance will preserve pockets of anonymity. But the default condition of public life is shifting, from one where anonymity was assumed to one where identification is the norm.

The technologies being installed today will outlast the political conditions under which they were deployed. Gait recognition cameras placed for counter-terrorism will not be removed when the threat recedes. Voice identification systems built for banking will not be dismantled when fraud declines. Wi-Fi sensing capabilities embedded in building infrastructure will persist indefinitely. The question is not whether these technologies will be misused, but when, by whom, and with what consequences for the freedoms that depend on the ability to move through the world unrecognised.

Bruce Schneier put it plainly: “Privacy protects us from abuses by those in power, even if we're doing nothing wrong at the time of surveillance.” In a world where your walk, your voice, and your keystrokes are all that stand between you and identification, that protection is being quietly, systematically, irreversibly eroded.

---

References and Sources

1. Joy Buolamwini and Timnit Gebru. “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” Proceedings of Machine Learning Research, Vol. 81, 2018. http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

2. South China Morning Post. “Chinese police test gait-recognition technology from AI start-up Watrix that identifies people based on how they walk.” November 2018. https://www.scmp.com/tech/start-ups/article/2187600/chinese-police-surveillance-gets-boost-ai-start-watrix-technology-can

3. NEC Corporation. “NEC Launches new system using Biometric Authentication Technology.” Press Release, 3 September 2024. https://www.nec.com/en/press/202409/global2024090301.html

4. GlobeNewsWire/SNS Insider. “Gait Biometrics Market Size to Hit USD 3.41 Billion by 2032.” 21 July 2025. https://www.globenewswire.com/news-release/2025/07/21/3118758/0/en/Gait-Biometrics-Market-Size-to-Hit-USD-3-41-Billion-by-2032-at-13-38-CAGR-Research-by-SNS-Insider.html

5. European Parliament. “EU AI Act: first regulation on artificial intelligence.” https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence

6. Stanford Law School. “No. 91: EU Artificial Intelligence Act: Regulating the Use of Facial Recognition Technologies in Publicly Accessible Spaces.” https://law.stanford.edu/publications/no-91-eu-artificial-intelligence-act-regulating-the-use-of-facial-recognition-technologies-in-publicly-accessible-spaces/

7. WilmerHale. “Year in Review: 2024 BIPA Litigation Takeaways.” 19 February 2025. https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20250219-year-in-review-2024-bipa-litigation-takeaways

8. Bloomberg Law. “Clearview AI Gets Settlement Approved in Face-Scan Privacy Case.” 2025. https://news.bloomberglaw.com/litigation/clearview-ai-gets-settlement-approved-in-face-scan-privacy-case

9. CNN. “San Francisco just banned facial-recognition technology.” 14 May 2019. https://www.cnn.com/2019/05/14/tech/san-francisco-facial-recognition-ban

10. SF Standard. “SFPD skirted facial-recognition ban, lawsuit says.” 18 July 2024. https://sfstandard.com/2024/07/18/san-francisco-police-facial-recognition-violations/

11. ACLU. “Williams v. City of Detroit.” https://www.aclu.org/cases/williams-v-city-of-detroit-face-recognition-false-arrest

12. Shoshana Zuboff. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. PublicAffairs, 2019.

13. Oxford Academic. “Chilling Effects of Surveillance and Human Rights: Insights from Qualitative Research in Uganda and Zimbabwe.” Journal of Human Rights Practice, Vol. 16, Issue 1, 2024. https://academic.oup.com/jhrp/article/16/1/397/7234270

14. Amnesty International. “Ban The Scan New York City.” https://banthescan.amnesty.org/nyc/index.html

15. Storbeck, M. et al. “Surveillance experiences of extinction rebellion activists and police: Unpacking the technologization of Dutch protest policing.” Big Data & Society, 2025. https://journals.sagepub.com/doi/10.1177/20539517241307892

16. ICO. “Preventing harm, promoting trust: our AI and biometrics strategy.” June 2025. https://ico.org.uk/about-the-ico/our-information/our-strategies-and-plans/artificial-intelligence-and-biometrics-strategy/

17. ACM Transactions on Sensor Networks. “A Survey on WiFi-based Human Identification: Scenarios, Challenges, and Current Solutions.” 2024. https://dl.acm.org/doi/10.1145/3708323

18. ScienceDirect. “WiFi-based human activity recognition through wall using deep learning.” Engineering Applications of Artificial Intelligence, 2024. https://www.sciencedirect.com/science/article/abs/pii/S0952197623013556

19. Springer Nature. “Keystroke dynamics for intelligent biometric authentication with machine learning.” Discover Applied Sciences, 2025. https://link.springer.com/article/10.1007/s42452-025-07449-5

20. MDPI/Applied Sciences. “Authentication by Keystroke Dynamics: The Influence of Typing Language.” 2023. https://www.mdpi.com/2076-3417/13/20/11478

21. Bruce Schneier. “The Eternal Value of Privacy.” Schneier on Security, May 2006. https://www.schneier.com/essays/archives/2006/05/theeternalvalue_of.html

22. Biometric Update. “PopEye to strengthen EU border biometrics with gait recognition integration.” October 2024. https://www.biometricupdate.com/202410/popeye-to-strengthen-eu-border-biometrics-with-gait-recognition-integration

23. Mozilla Foundation. “How to Disappear: The Rise of Anti-Surveillance Fashion.” https://www.mozillafoundation.org/en/nothing-personal/anti-surveillance-fashion-privacy-ai/

24. GOV.UK. “Briefing note on the ethical issues arising from the public sector use of biometric voice recognition technology.” https://www.gov.uk/government/publications/public-sector-use-of-biometric-voice-recognition-technology-ethical-issues/

25. Pew Research Center. “Public views of police use of facial recognition technology.” 17 March 2022. https://www.pewresearch.org/internet/2022/03/17/public-more-likely-to-see-facial-recognition-use-by-police-as-good-rather-than-bad-for-society/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In June 2024, Goldman Sachs published a research note that rattled Silicon Valley's most cherished assumptions. The report posed what it called the “$600 billion question”: would the staggering investment in artificial intelligence infrastructure ever generate proportional returns? The note featured analysis from MIT economist Daron Acemoglu, who had recently calculated that AI would produce no more than a 0.93 to 1.16 percent increase in US GDP over the next decade, a figure dramatically lower than the techno-utopian projections circulating through investor presentations and conference keynotes. “Much of what we hear from the industry now is exaggeration,” Acemoglu stated plainly. Two months later, he was awarded the 2024 Nobel Memorial Prize in Economic Sciences, alongside his MIT colleague Simon Johnson and University of Chicago economist James Robinson, for research on the relationship between political institutions and economic growth.

That gap between what AI is promised to deliver and what it actually does is no longer an abstract concern for economists and technologists. It is reshaping public attitudes toward technology at a speed that should alarm anyone who cares about the long-term relationship between innovation and democratic society. When governments deploy algorithmic systems to deny healthcare coverage or detect welfare fraud, when corporations invest billions in tools that fail 95 percent of the time, and when the public is told repeatedly that superintelligence is just around the corner while chatbots still fabricate legal citations, something fundamental breaks in the social contract around technological progress.

The question is not whether AI is useful. It plainly is, in specific, well-defined applications. The question is what happens when an entire civilisation makes strategic decisions based on capabilities that do not yet exist and may never materialise in the form being sold.

The Great Correction Arrives

By late 2025, the AI industry had entered what Gartner's analysts formally classified as the “Trough of Disillusionment.” Generative AI, which had been perched at the Peak of Inflated Expectations just one year earlier, had slid into the territory where early adopters report performance issues, low return on investment, and a growing sense that the technology's capabilities had been systematically overstated. The positioning reflected difficulties organisations face when attempting to move generative AI from pilot projects to production systems. Integration with existing infrastructure presented technical obstacles, while concerns about data security caused some companies to limit deployment entirely.

The numbers told a damning story. According to MIT's “The GenAI Divide: State of AI in Business 2025” report, published in July 2025 and based on 52 executive interviews, surveys of 153 leaders, and analysis of 300 public AI deployments, 95 percent of generative AI pilot projects delivered no measurable profit-and-loss impact. American enterprises had spent an estimated $40 billion on artificial intelligence systems in 2024, yet the vast majority saw zero measurable bottom-line returns. Only five percent of integrated systems created significant value.

The study's authors, from MIT's NANDA initiative, identified what they termed the “GenAI Divide”: a widening split between high adoption and low transformation. Companies were enthusiastically purchasing and deploying AI tools, but almost none were achieving the business results that had been promised. “The 95% failure rate for enterprise AI solutions represents the clearest manifestation of the GenAI Divide,” the report stated. The core barrier, the authors concluded, was not infrastructure, regulation, or talent. It was that most generative AI systems “do not retain feedback, adapt to context, or improve over time,” making them fundamentally ill-suited for the enterprise environments into which they were being thrust.

This was not an outlier finding. A 2024 NTT DATA analysis concluded that between 70 and 85 percent of generative AI deployment efforts were failing to meet their desired return on investment. The Autodesk State of Design & Make 2025 report found that sentiment toward AI had dropped significantly year over year, with just 69 percent of business leaders saying AI would enhance their industry, representing a 12 percent decline from the previous year. Only 40 percent of leaders said they were approaching or had achieved their AI goals, a 16-point decrease that represented a 29 percent drop. S&P Global data revealed that 42 percent of companies scrapped most of their AI initiatives in 2025, up sharply from 17 percent the year before.

The infrastructure spending, meanwhile, continued accelerating even as returns failed to materialise. Meta, Microsoft, Amazon, and Google collectively committed over $250 billion to AI infrastructure during 2025. Amazon alone planned $125 billion in capital expenditure, up from $77 billion in 2024, a 62 percent increase. Goldman Sachs CEO David Solomon publicly acknowledged that he expected “a lot of capital that was deployed that doesn't deliver returns.” Amazon founder Jeff Bezos called the environment “kind of an industrial bubble.” Even OpenAI CEO Sam Altman conceded that “people will overinvest and lose money.”

Trust in Freefall

The gap between AI's promises and its performance is not occurring in a vacuum. It is landing on a public already growing sceptical of the technology industry's claims, and it is accelerating a decline in trust that carries profound implications for democratic governance.

The 2025 Edelman Trust Barometer, based on 30-minute online interviews conducted between October and November 2024, revealed a stark picture. Globally, only 49 percent of respondents trusted artificial intelligence as a technology. In the United States, that figure dropped to just 32 percent. Three times as many Americans rejected the growing use of AI (49 percent) as embraced it (17 percent). In the United Kingdom, trust stood at just 36 percent. In Germany, 39 percent. The Chinese public, by contrast, reported 72 percent trust in AI, a 40-point gap that reflects not just different regulatory environments but fundamentally different cultural relationships with technology and state authority.

These figures represent a significant deterioration. A decade ago, 73 percent of Americans trusted technology companies. By 2025, that number had fallen to 63 percent. Technology, which was the most trusted sector in 90 percent of the countries Edelman studies eight years ago, now held that position in only half. The barometer also found that 59 percent of global employees feared job displacement due to automation, and nearly one in two were sceptical of business use of artificial intelligence.

The Pew Research Center's findings painted an even more granular picture of public anxiety. In an April 2025 report examining how the US public and AI experts view artificial intelligence, Pew found that 50 percent of American adults said they were more concerned than excited about the increased use of AI in daily life, up from 37 percent in 2021. More than half (57 percent) rated the societal risks of AI as high, compared with only 25 percent who said the benefits were high. Over half of US adults (53 percent) believed AI did more harm than good in protecting personal privacy, and 53 percent said AI would worsen people's ability to think creatively.

Perhaps most revealing was the chasm between expert optimism and public unease. While 56 percent of AI experts believed AI would have a positive effect on the United States over the next 20 years, only 17 percent of the general public agreed. While 47 percent of experts said they were more excited than concerned, only 11 percent of ordinary citizens felt the same. And despite their divergent levels of optimism, both groups shared a common scepticism about institutional competence: roughly 60 percent of both experts and the public said they lacked confidence that US companies would develop AI responsibly.

The Stanford HAI AI Index 2025 Report reinforced these trends globally. Across 26 nations surveyed by Ipsos, confidence that AI companies protect personal data fell from 50 percent in 2023 to 47 percent in 2024. Fewer people believed AI systems were unbiased and free from discrimination compared to the previous year. While 18 of 26 nations saw an increase in the proportion of people who believed AI products offered more benefits than drawbacks, the optimism was concentrated in countries like China (83 percent), Indonesia (80 percent), and Thailand (77 percent), while the United States (39 percent), Canada (40 percent), and the Netherlands (36 percent) remained deeply sceptical.

When Algorithms Replace Judgement

The erosion of public trust in AI would be concerning enough if it were merely a matter of consumer sentiment. But the stakes become existential when governments and corporations use overestimated AI capabilities to make decisions that fundamentally alter people's lives, and when those decisions carry consequences that cannot be undone.

Consider healthcare. In November 2023, a class action lawsuit was filed against UnitedHealth Group and its subsidiary, alleging that the company illegally used an AI algorithm called nH Predict to deny rehabilitation care to seriously ill elderly patients enrolled in Medicare Advantage plans. The algorithm, developed by a company called Senior Metrics and later acquired by UnitedHealth's Optum subsidiary in 2020, was designed to predict how long patients would need post-acute care. According to the lawsuit, UnitedHealth deployed the algorithm knowing it had a 90 percent error rate on appeals, meaning that nine out of ten times a human reviewed the AI's denial, they overturned it. UnitedHealth also allegedly knew that only 0.2 percent of denied patients would file appeals, making the error rate commercially inconsequential for the insurer despite being medically devastating for patients.

The human cost was documented in court filings. Gene Lokken, a 91-year-old Wisconsin resident named in the lawsuit, fractured his leg and ankle in May 2022. After his doctor approved physical therapy, UnitedHealth paid for only 19 days before the algorithm determined he was safe to go home. His doctors appealed, noting his muscles were “paralysed and weak,” but the insurer denied further coverage. His family paid approximately $150,000 over the following year until he died in July 2023. In February 2025, a federal court allowed the case to proceed, denying UnitedHealth's attempt to dismiss the claims and waiving the exhaustion of administrative remedies requirement, noting that patients faced irreparable harm.

The STAT investigative series “Denied by AI,” which broke the UnitedHealth story, was a 2024 Pulitzer Prize finalist in investigative reporting. A US Senate report released in October 2024 found that UnitedHealthcare's prior authorisation denial rate for post-acute care had jumped to 22.7 percent in 2022 from 10.9 percent in 2020. The healthcare AI problem extends far beyond a single insurer. ECRI, a patient safety organisation, ranked insufficient governance of artificial intelligence as the number two patient safety threat in 2025, warning that medical errors generated by AI could compromise patient safety through misdiagnoses and inappropriate treatment decisions. Yet only about 16 percent of hospital executives surveyed said they had a systemwide governance policy for AI use and data access.

The pattern repeats across domains where algorithmic systems are deployed to process vulnerable populations. In the Netherlands, the childcare benefits scandal stands as perhaps the most devastating example of what happens when governments trust flawed algorithms with life-altering decisions. The Dutch Tax and Customs Administration deployed a machine learning model to detect welfare fraud that illegally used dual nationality as a risk characteristic. The system falsely accused over 20,000 parents of fraud, resulting in benefits termination and forced repayments. Families were driven into bankruptcy. Children were removed from their homes. Mental health crises proliferated. Seventy percent of those affected had a migration background, and fifty percent were single-person households, mostly mothers. In January 2021, the Dutch government was forced to resign after a parliamentary investigation concluded that the government had violated the foundational principles of the rule of law.

The related SyRI (System Risk Indication) system, which cross-referenced citizens' employment, benefits, and tax data to flag “unlikely citizen profiles,” was deployed exclusively in neighbourhoods with high numbers of low-income households and disproportionately many residents from immigrant backgrounds. In February 2020, the Hague court ordered SyRI's immediate halt, ruling it violated Article 8 of the European Convention on Human Rights. Amnesty International described the system's targeting criteria as “xenophobic machines.” Yet investigations by Lighthouse Reports later confirmed that similar algorithmic surveillance practices continued under slightly adapted systems, even after the ban, with the government having “silently continued to deploy a slightly adapted SyRI in some of the country's most vulnerable neighbourhoods.”

The Stochastic Parrot Problem

Understanding why AI hype is so dangerous requires understanding what these systems actually do, as opposed to what their makers claim they do.

Emily Bender, a linguistics professor at the University of Washington who was included in the inaugural TIME100 AI list of most influential people in artificial intelligence in 2023, co-authored a now-famous paper arguing that large language models are fundamentally “stochastic parrots.” They do not understand language in any meaningful sense. They draw on training data to predict which sequence of tokens is most likely to follow a given prompt. The result is an illusion of comprehension, a pattern-matching exercise that produces outputs resembling intelligent thought without any of the underlying cognition.

In 2025, Bender and sociologist Alex Hanna, director of research at the Distributed AI Research Institute and a former Google employee, published “The AI Con: How to Fight Big Tech's Hype and Create the Future We Want.” The book argues that AI hype serves as a mask for Big Tech's drive for profit, with the breathless promotion of AI capabilities benefiting technology companies far more than users or society. “Who benefits from this technology, who is harmed, and what recourse do they have?” Bender and Hanna ask, framing these as the essential questions that the hype deliberately obscures. Library Journal called the book “a thorough, witty, and accessible argument against AI that meets the moment.”

The stochastic parrot problem has real-world consequences that compound the trust deficit. When AI systems fabricate information with perfect confidence, they undermine the epistemic foundations that societies rely on for decision-making. Legal scholar Damien Charlotin, who tracks AI hallucinations in court filings through his database, had documented at least 206 instances of lawyers submitting AI-generated fabricated case citations by mid-2025. Stanford University's RegLab found that even premium legal AI tools hallucinated at alarming rates: Westlaw's AI-Assisted Research produced hallucinated or incorrect information 33 percent of the time, providing accurate responses to only 42 percent of queries. LexisNexis's Lexis+ AI hallucinated 17 percent of the time. A 2025 study published in Nature Machine Intelligence found that large language models cannot reliably distinguish between belief and knowledge, or between opinions and facts, noting that “failure to make such distinctions can mislead diagnoses, distort judicial judgements and amplify misinformation.”

If the tools marketed as the most reliable in their field fabricate information roughly one-fifth to one-third of the time, what does this mean for the countless lower-stakes applications where AI outputs are accepted without verification?

The AI Washing Economy

The gap between marketing claims and actual capabilities has grown so pronounced that regulators have begun treating AI exaggeration as a form of securities fraud.

In March 2024, the US Securities and Exchange Commission brought its first “AI washing” enforcement actions, simultaneously charging two investment advisory firms, Delphia and Global Predictions, with making false and misleading statements about their use of AI. Delphia paid $225,000 and Global Predictions paid $175,000 in civil penalties. These firms had not been entirely without AI capabilities, but they had overstated what those systems could do, crossing the line from marketing enthusiasm into regulatory violation.

The enforcement actions escalated rapidly. In January 2025, the SEC charged Presto Automation, a formerly Nasdaq-listed company, in the first AI washing action against a public company. Presto had claimed its AI voice system eliminated the need for human drive-through order-taking at fast food restaurants, but the SEC alleged the vast majority of orders still required human intervention and that the AI speech recognition technology was owned and operated by a third party. In April 2025, the SEC and Department of Justice charged the founder of Nate Inc. with fraudulently raising over $42 million by claiming the company's shopping app used AI to process transactions, when in reality manual workers completed the purchases. The claimed automation rate was above 90 percent; the actual rate was essentially zero.

Securities class actions targeting alleged AI misrepresentations increased by 100 percent between 2023 and 2024. In February 2025, the SEC announced the creation of a dedicated Cyber and Emerging Technologies Unit, tasked with combating technology-related misconduct, and flagged AI washing as a top examination priority.

The pattern is instructive. When a technology is overhyped, the incentive to exaggerate capabilities becomes irresistible. Companies that accurately describe their modest AI implementations risk being punished by investors who have been conditioned to expect transformative breakthroughs. The honest actors are penalised while the exaggerators attract capital, creating a market dynamic that systematically rewards deception.

Echoes of Previous Bubbles

The AI hype cycle is not without historical precedent, and the parallels offer both warnings and qualified reassurance.

During the dot-com era, telecommunications companies laid more than 80 million miles of fibre optic cables across the United States, driven by wildly inflated claims about internet traffic growth. Companies like Global Crossing, Level 3, and Qwest raced to build massive networks. The result was catastrophic overcapacity: even four years after the bubble burst, 85 to 95 percent of the fibre laid remained unused, earning the nickname “dark fibre.” The Nasdaq composite rose nearly 400 percent between 1995 and March 2000, then crashed 78 percent by October 2002.

The parallels to today's AI infrastructure buildout are unmistakable. Meta CEO Mark Zuckerberg announced plans for an AI data centre “so large it could cover a significant part of Manhattan.” The Stargate Project aims to develop a $500 billion nationwide network of AI data centres. Goldman Sachs analysts found that hyperscaler companies had taken on $121 billion in debt over the past year, representing a more than 300 percent increase from typical industry debt levels. AI-related stocks had accounted for 75 percent of S&P 500 returns, 80 percent of earnings growth, and 90 percent of capital spending growth since ChatGPT launched in November 2022.

Yet there are important differences. Unlike many dot-com companies that had no revenue, major AI players are generating substantial income. Microsoft's Azure cloud service grew 39 percent year over year to an $86 billion run rate. OpenAI projects $20 billion in annualised revenue. The Nasdaq's forward price-to-earnings ratio was approximately 26 times in November 2023, compared to approximately 60 times at the dot-com peak.

The more useful lesson from the dot-com era is not about whether the bubble will burst, but about what happens to public trust and institutional decision-making in the aftermath. The internet survived the dot-com crash and eventually fulfilled many of its early promises. But the crash destroyed trillions in wealth, wiped out retirement savings, and created a lasting scepticism toward technology claims that took years to overcome. The institutions and individuals who made decisions based on dot-com hype, from pension funds that invested in companies with no path to profitability to governments that restructured services around technologies that did not yet work, bore costs that were never fully recovered.

Algorithmic Bias and the Feedback Loop of Injustice

Perhaps the most consequential long-term risk of the AI hype gap is its intersection with systemic inequality. When policymakers deploy AI systems in criminal justice, welfare administration, and public services based on inflated claims of accuracy and objectivity, the consequences fall disproportionately on communities that are already marginalised.

Predictive policing offers a stark illustration. The Chicago Police Department's “Strategic Subject List,” implemented in 2012 to identify individuals at higher risk of gun violence, disproportionately targeted young Black and Latino men, leading to intensified surveillance and police interactions in those communities. The system created a feedback loop: more police dispatched to certain neighbourhoods resulted in more recorded crime, which the algorithm interpreted as confirmation that those neighbourhoods were indeed high-risk, which led to even more policing. The NAACP has called on state legislators to evaluate and regulate the use of predictive policing, noting mounting evidence that these tools increase racial biases and citing the lack of transparency inherent in proprietary algorithms that do not allow for public scrutiny.

The COMPAS recidivism prediction tool, widely used in US criminal justice, was found to produce biased predictions against Black defendants compared to white defendants, trained on historical data saturated with racial bias. An audit by the LAPD inspector general found “significant inconsistencies” in how officers entered data into a predictive policing programme, further fuelling biased predictions. These are not edge cases or implementation failures. They are the predictable consequences of deploying pattern-recognition systems trained on data that reflects centuries of structural discrimination.

In welfare administration, the pattern is equally troubling. The Dutch childcare benefits scandal demonstrated how algorithmic systems can automate inequality at scale. The municipality of Rotterdam used a discriminatory algorithm to profile residents and “predict” social welfare fraud for three years, disproportionately targeting young single mothers with limited knowledge of Dutch. In the United Kingdom, the Department for Work and Pensions admitted, in documents released under the Freedom of Information Act, to finding bias in an AI tool used to detect fraud in universal credit claims. The tool's initial iteration correctly matched conditions only 35 percent of the time, and by the DWP's own admission, “chronic fatigue was translated into chronic renal failure” and “partially amputation of foot was translated into partially sighted.”

These failures share a common thread. The AI systems were deployed based on claims of objectivity and accuracy that did not withstand scrutiny. Policymakers, influenced by industry hype about AI's capabilities, trusted algorithmic outputs over human judgement, and the people who paid the price were those least equipped to challenge the decisions being made about their lives.

What Sustained Disillusionment Means for Innovation

The long-term consequences of the AI hype gap extend beyond immediate harms to individual victims. They threaten to reshape the relationship between society and technological innovation in ways that could prove difficult to reverse.

First, there is the problem of misallocated resources. The MIT study found that more than half of generative AI budgets were devoted to sales and marketing tools, despite evidence that the best returns came from back-office automation, eliminating business process outsourcing, cutting external agency costs, and streamlining operations. When organisations chase the use cases that sound most impressive rather than those most likely to deliver value, they waste capital that could have funded genuinely productive innovation. The study also revealed a striking shadow economy: while only 40 percent of companies had official large language model subscriptions, 90 percent of workers surveyed reported daily use of personal AI tools for job tasks, suggesting that the gap between corporate AI strategy and actual AI utility is even wider than the headline figures suggest.

Second, the trust deficit creates regulatory feedback loops that can stifle beneficial applications. As public concern about AI grows, so does political pressure for restrictive regulation. The 2025 Stanford HAI report found that references to AI in draft legislation across 75 countries increased by 21.3 percent, continuing a ninefold increase since 2016. In the United States, 73.7 percent of local policymakers agreed that AI should be regulated, up from 55.7 percent in 2022. This regulatory momentum is a direct response to the trust deficit, and while some regulation is necessary and overdue, poorly designed rules driven by public fear rather than technical understanding risk constraining beneficial applications alongside harmful ones. Colorado became the first US state to enact legislation addressing algorithmic bias in 2024, with California and New York following with their own targeted measures.

Third, the hype cycle creates a talent and attention problem. When AI is presented as a solution to every conceivable challenge, researchers and engineers are pulled toward fashionable applications rather than areas of genuine need. Acemoglu has argued that “we currently have the wrong direction for AI. We're using it too much for automation and not enough for providing expertise and information to workers.” The hype incentivises building systems that replace human judgement rather than augmenting it, directing talent and investment away from applications that could produce the greatest social benefit.

Finally, and perhaps most critically, the erosion of public trust in AI threatens to become self-reinforcing. Each failed deployment, each exaggerated claim exposed, each algorithmic system found to be biased or inaccurate further deepens public scepticism. Meredith Whittaker, president of Signal, has warned about the security and privacy risks of granting AI agents extensive access to sensitive data, describing a future where the “magic genie bot” becomes a nightmare if security and privacy are not prioritised. When public trust in AI erodes, even beneficial and well-designed systems face adoption resistance, creating a vicious cycle where good technology is tainted by association with bad marketing.

Rebuilding on Honest Foundations

The AI hype gap is not merely a marketing problem or an investment risk. It is a structural challenge to the relationship between technological innovation and public trust that has been building for years and is now reaching a critical inflection point.

The 2025 Edelman Trust Barometer found that the most powerful drivers of AI enthusiasm are trust and information, with hesitation rooted more in unfamiliarity than negative experiences. This finding suggests a path that does not require abandoning AI, but demands abandoning the hype. As people use AI more and experience its ability to help them learn, work, and solve problems, their confidence rises. The obstacle is not the technology itself but the inflated expectations that set users up for disappointment.

Gartner's placement of generative AI in the Trough of Disillusionment is, paradoxically, encouraging. As the firm's analysts note, the trough does not represent failure. It represents the transition from wild experimentation to rigorous engineering, from breathless promises to honest assessment of what works and what does not. The companies and institutions that emerge successfully from this phase will be those that measured their claims against reality rather than against their competitors' marketing materials.

The lesson from previous technology cycles is clear but routinely ignored. The dot-com bubble popped, but the internet did not disappear. What disappeared were the companies and institutions that confused hype with strategy. The same pattern will likely repeat with AI. The technology will mature, find its genuine applications, and deliver real value. But the path from here to there runs through a period of reckoning that demands honesty about what AI can and cannot do, transparency about the limitations of algorithmic decision-making, and accountability for the real harms caused by deploying immature systems in high-stakes contexts.

As Bender and Hanna urge, the starting point must be asking basic but important questions: who benefits, who is harmed, and what recourse do they have? As Acemoglu wrote in his analysis for “Economic Policy” in 2024, “Generative AI has the potential to fundamentally change the process of scientific discovery, research and development, innovation, new product and material testing.” The potential is real. But potential is not performance, and treating it as such has consequences that a $600 billion question only begins to capture.

---

References and Sources

1. Acemoglu, D. (2024). “The Simple Macroeconomics of AI.” Economic Policy. Massachusetts Institute of Technology. https://economics.mit.edu/sites/default/files/2024-04/The%20Simple%20Macroeconomics%20of%20AI.pdf

2. Amnesty International. (2021). “Xenophobic Machines: Dutch Child Benefit Scandal.” Retrieved from https://www.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/

3. Bender, E. M. & Hanna, A. (2025). The AI Con: How to Fight Big Tech's Hype and Create the Future We Want. Penguin/HarperCollins.

4. CBS News. (2023). “UnitedHealth uses faulty AI to deny elderly patients medically necessary coverage, lawsuit claims.” Retrieved from https://www.cbsnews.com/news/unitedhealth-lawsuit-ai-deny-claims-medicare-advantage-health-insurance-denials/

5. Challapally, A., Pease, C., Raskar, R. & Chari, P. (2025). “The GenAI Divide: State of AI in Business 2025.” MIT NANDA Initiative. As reported by Fortune, 18 August 2025. https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/

6. Edelman. (2025). “2025 Edelman Trust Barometer.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer

7. Edelman. (2025). “Flash Poll: Trust and Artificial Intelligence at a Crossroads.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer/flash-poll-trust-artifical-intelligence

8. Edelman. (2025). “The AI Trust Imperative: Navigating the Future with Confidence.” Retrieved from https://www.edelman.com/trust/2025/trust-barometer/report-tech-sector

9. Gartner. (2025). “Hype Cycle for Artificial Intelligence, 2025.” Retrieved from https://www.gartner.com/en/articles/hype-cycle-for-artificial-intelligence

10. Goldman Sachs. (2024). “Top of Mind: AI: in a bubble?” Goldman Sachs Research. Retrieved from https://www.goldmansachs.com/insights/top-of-mind/ai-in-a-bubble

11. Healthcare Finance News. (2025). “Class action lawsuit against UnitedHealth's AI claim denials advances.” Retrieved from https://www.healthcarefinancenews.com/news/class-action-lawsuit-against-unitedhealths-ai-claim-denials-advances

12. Lighthouse Reports. (2023). “The Algorithm Addiction.” Retrieved from https://www.lighthousereports.com/investigation/the-algorithm-addiction/

13. Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D. & Ho, D. E. (2025). “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.” Journal of Empirical Legal Studies, 0:1-27. https://doi.org/10.1111/jels.12413

14. MIT Technology Review. (2025). “The great AI hype correction of 2025.” Retrieved from https://www.technologyreview.com/2025/12/15/1129174/the-great-ai-hype-correction-of-2025/

15. NAACP. (2024). “Artificial Intelligence in Predictive Policing Issue Brief.” Retrieved from https://naacp.org/resources/artificial-intelligence-predictive-policing-issue-brief

16. Nature Machine Intelligence. (2025). “Language models cannot reliably distinguish belief from knowledge and fact.” https://doi.org/10.1038/s42256-025-01113-8

17. Novara Media. (2025). “How Labour Is Using Biased AI to Determine Benefit Claims.” Retrieved from https://novaramedia.com/2025/04/15/how-the-labour-party-is-using-biased-ai-to-determine-benefit-claims/

18. NTT DATA. (2024). “Between 70-85% of GenAI deployment efforts are failing to meet their desired ROI.” Retrieved from https://www.nttdata.com/global/en/insights/focus/2024/between-70-85p-of-genai-deployment-efforts-are-failing

19. Pew Research Center. (2025). “How the US Public and AI Experts View Artificial Intelligence.” Retrieved from https://www.pewresearch.org/internet/2025/04/03/how-the-us-public-and-ai-experts-view-artificial-intelligence/

20. Radiologybusiness.com. (2025). “'Insufficient governance of AI' is the No. 2 patient safety threat in 2025.” Retrieved from https://radiologybusiness.com/topics/artificial-intelligence/insufficient-governance-ai-no-2-patient-safety-threat-2025

21. SEC. (2024). “SEC Charges Two Investment Advisers with Making False and Misleading Statements About Their Use of Artificial Intelligence.” Press Release 2024-36. Retrieved from https://www.sec.gov/newsroom/press-releases/2024-36

22. Stanford HAI. (2025). “The 2025 AI Index Report.” Stanford University Human-Centered Artificial Intelligence. Retrieved from https://hai.stanford.edu/ai-index/2025-ai-index-report

23. STAT News. (2023). “UnitedHealth faces class action lawsuit over algorithmic care denials in Medicare Advantage plans.” Retrieved from https://www.statnews.com/2023/11/14/unitedhealth-class-action-lawsuit-algorithm-medicare-advantage/

24. The Dutch Childcare Benefits Scandal. Wikipedia. Retrieved from https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scandal

25. Washington Post. (2024). “Big Tech is spending billions on AI. Some on Wall Street see a bubble.” Retrieved from https://www.washingtonpost.com/technology/2024/07/24/ai-bubble-big-tech-stocks-goldman-sachs/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The machines are learning to act without us. Not in some distant, science fiction future, but right now, in the server rooms of Silicon Valley, the trading floors of Wall Street, and perhaps most disturbingly, in the operating systems that increasingly govern your daily existence. The question is no longer whether artificial intelligence will transform how we live and work. That transformation is already underway. The more pressing question, the one that should keep technology leaders and ordinary citizens alike awake at night, is this: when AI agents can execute complex tasks autonomously across multiple systems without human oversight, will this liberate you from mundane work and decision-making, or create a world where you lose control over the systems that govern your daily life?

The answer, as with most genuinely important questions about technology, is: both. And that ambiguity is precisely what makes this moment so consequential.

The Autonomous Revolution Arrives Ahead of Schedule

Walk into any major enterprise today, and you will find a digital workforce that would have seemed fantastical just three years ago. According to Gartner's August 2025 analysis, 40 per cent of enterprise applications will feature task-specific AI agents by the end of 2026, up from less than 5 per cent in early 2025. That is not gradual adoption; that is a technological tidal wave.

The numbers paint a picture of breathtaking acceleration. McKinsey research from 2025 shows that 62 per cent of survey respondents report their organisations are at least experimenting with AI agents, whilst 23 per cent are already scaling agentic AI systems somewhere in their enterprises. A G2 survey from August 2025 found that 57 per cent of companies already have AI agents in production, with another 22 per cent in pilot programmes. The broader AI agents market reached 7.92 billion dollars in 2025, with projections extending to 236.03 billion dollars by 2034, a compound annual growth rate that defies historical precedent for enterprise technology adoption.

These are not simply chatbots with better conversation skills. Modern AI agents represent a fundamental shift in how we think about automation. Unlike traditional software that follows predetermined rules, these systems can perceive their environment, make decisions, take actions, and learn from the outcomes, all without waiting for human instruction at each step. They can book your flights, manage your calendar, process insurance claims, monitor network security, and execute financial trades. They can, in short, do many of the things we used to assume required human judgment.

Deloitte predicts that 50 per cent of enterprises using generative AI will deploy autonomous AI agents by 2027, doubling from 25 per cent in 2025. A 2025 Accenture study goes further, predicting that by 2030, AI agents will be the primary users of most enterprises' internal digital systems. Pause on that for a moment. The primary users of your company's software will not be your employees. They will be algorithms. Gartner's projections suggest that by 2028, over one-third of enterprise software solutions will include agentic AI, making up to 15 per cent of day-to-day decisions autonomous.

An IBM and Morning Consult survey of 1,000 enterprise AI developers found that 99 per cent of respondents said they were exploring or developing AI agents. This is not a niche technology being evaluated by a handful of innovators. This is a fundamental reshaping of how business operates, happening simultaneously across virtually every major organisation on the planet.

Liberation from the Tedious and the Time-Consuming

For those weary of administrative drudgery, the promise of autonomous AI agents borders on the utopian. Consider the healthcare sector, where agents are transforming the patient journey whilst delivering a 3.20 dollar return for every dollar invested within 14 months, according to industry analyses. These systems take and read clinician notes, extract key data, cross-check payer policies, and automate prior authorisations and claims submissions. At OI Infusion Services, AI agents cut approval times from around 30 days to just three days, dramatically reducing treatment delays for patients who desperately need care.

The applications in healthcare extend beyond administrative efficiency. Hospitals are using agentic AI to optimise patient flow, schedule patient meetings, predict bed occupancy rates, and manage staff. At the point of care, agents assist with triage and chart preparation by summarising patient history, highlighting red flags, and surfacing relevant clinical guidelines. The technology is not replacing physicians; it is freeing them to focus on what they trained for years to do: heal people.

In customer service, the results are similarly striking. Boston Consulting Group reports that a global technology company achieved a 50 per cent reduction in time to resolution for service requests, whilst a European energy provider improved customer satisfaction by 18 per cent. A Chinese insurance company improved contact centre productivity by more than 50 per cent. A European financial institution has automated 90 per cent of its consumer loans. Effective AI agents can accelerate business processes by 30 to 50 per cent, according to BCG analysis, in areas ranging from finance and procurement to customer operations.

The financial sector has embraced these capabilities with particular enthusiasm. AI agents now continuously analyse high-velocity financial data, adjust credit scores in real time, automate Know Your Customer checks, calculate loans, and monitor financial health indicators. These systems can fetch data beyond traditional sources, including customer relationship management systems, payment gateways, banking data, credit bureaus, and sanction databases. CFOs are beginning to rely on these systems not just for static reporting but for continuous forecasting, integrating ERP data, market indicators, and external economic signals to produce real-time cash flow projections. Risk events have been reduced by 60 per cent in pilot environments.

The efficiency gains are real, and they are substantial. ServiceNow's AI agents are automating IT, HR, and operational processes, reducing manual workloads by up to 60 per cent. Enterprises deploying AI agents estimate up to 50 per cent efficiency gains in customer service, sales, and HR operations. And 75 per cent of organisations have seen improvements in satisfaction scores post-AI agent deployment.

For the knowledge worker drowning in email, meetings, and administrative overhead, these developments represent something close to salvation. The promise is straightforward: let the machines handle the tedious tasks, freeing humans to focus on creative, strategic, and genuinely meaningful work.

The Other Side of Autonomy

Yet there is a darker current running beneath this technological optimism, and it demands our attention. The same capabilities that make AI agents so useful, their ability to act independently, to make decisions without human oversight, to operate at speeds no human can match, also make them potentially dangerous.

The security implications alone are sobering. Nearly 48 per cent of respondents to a recent industry survey believe agentic AI will represent the top attack vector for cybercriminals and nation-state threats by the end of 2026. The expanded attack surface deriving from the combination of agents' levels of access and autonomy is and should be a real concern.

Consider what happened in November 2025. Anthropic, one of the leading AI safety companies, disclosed that Chinese state-sponsored hackers used Claude Code to orchestrate what they called “the first documented case of a large-scale cyberattack executed without substantial human intervention.” The AI performed 80 to 90 per cent of the attack work autonomously, mapping networks, writing exploits, harvesting credentials, and exfiltrating data from approximately 30 targets. The bypass technique was disturbingly straightforward: attackers told the AI it was an employee of a legitimate cybersecurity firm conducting defensive testing and decomposed malicious tasks into innocent-seeming subtasks.

This incident illustrated a broader concern: by automating repetitive, technical work, AI agents can also lower the barrier for malicious activity. Security experts expect to see fully autonomous intrusion attempts requiring little to no human oversight from attackers. These AI agents will be capable of performing reconnaissance, exploiting vulnerabilities, escalating privileges, and exfiltrating data at a pace no traditional security tool is prepared for.

For organisations, a central question in 2026 is how to govern and secure a new multi-hybrid workforce where machines and agents already outnumber human employees by an 82-to-1 ratio. These trusted, always-on agents have privileged access, making them potentially the most valuable targets for attackers. The concern is that adversaries will stop focusing on humans and instead compromise these agents, turning them into what security researchers describe as an “autonomous insider.”

Despite widespread AI adoption, only about 34 per cent of enterprises reported having AI-specific security controls in place in 2025, whilst less than 40 per cent conduct regular security testing on AI models or agent workflows. We are building a new digital infrastructure at remarkable speed, but the governance and security frameworks have not kept pace.

The Employment Question Nobody Wants to Discuss Honestly

The conversation about AI and employment has become almost liturgical in its predictability. Optimists point to historical precedent: technological revolutions have always created more jobs than they destroyed. Pessimists counter that this time is different, that the machines are coming for cognitive work, not just physical labour.

The data from 2025 suggests both camps are partially correct, which is precisely the problem with easy answers. Research reveals that whilst 85 million jobs will be displaced by 2025, 97 million new roles will simultaneously emerge, representing a net positive job creation of 12 million positions globally. By 2030, according to industry projections, 92 million jobs will be displaced but 170 million new ones will emerge.

However, the distribution of these gains and losses is deeply uneven. In 2025, there have been 342 layoffs at tech companies with 77,999 people impacted. Nearly 55,000 job cuts were directly attributed to AI, according to Challenger, Gray & Christmas, out of a total 1.17 million layoffs in the United States, the highest level since the 2020 pandemic.

Customer service representatives face the highest immediate risk with an 80 per cent automation rate by 2025. Data entry clerks face a 95 per cent risk of automation, as AI systems can process over 1,000 documents per hour with an error rate of less than 0.1 per cent, compared to 2 to 5 per cent for humans. Approximately 7.5 million data entry and administrative jobs could be eliminated by 2027. Bloomberg research reveals AI could replace 53 per cent of market research analyst tasks and 67 per cent of sales representative tasks, whilst managerial roles face only 9 to 21 per cent automation risk.

And here is the uncomfortable truth buried in the optimistic projections about new job creation: whilst 170 million new roles may emerge by 2030, 77 per cent of AI jobs require master's degrees, and 18 per cent require doctoral degrees. The factory worker displaced by robots could, with retraining, potentially become a robot technician. But what happens to the call centre worker whose job is eliminated by an AI agent? The path from redundant administrative worker to machine learning engineer is considerably less traversable.

The gender disparities are equally stark. Geographic analysis indicates that 58.87 million women in the US workforce occupy positions highly exposed to AI automation compared to 48.62 million men. Workers aged 18 to 24 are 129 per cent more likely than those over 65 to worry AI will make their job obsolete. Nearly half of Gen Z job seekers believe AI has reduced the value of their college education.

According to the World Economic Forum's 2025 Future of Jobs Report, 41 per cent of employers worldwide intend to reduce their workforce in the next five years. In 2024, 44 per cent of companies using AI said employees would “definitely” or “probably” be laid off due to AI, up from 37 per cent in 2023.

There is a mitigating factor, however: 63.3 per cent of all jobs include nontechnical barriers that would prevent complete automation displacement. These barriers include client preferences for human interaction, regulatory requirements, and cost-effectiveness considerations.

Liberation from tedious work sounds rather different when it means liberation from your livelihood entirely.

When Machines Make Decisions We Cannot Understand

Perhaps the most philosophically troubling aspect of autonomous AI agents is their opacity. As these systems make increasingly consequential decisions about our lives, from loan approvals to medical diagnoses to criminal risk assessments, we often cannot explain precisely why they reached their conclusions.

AI agents are increasingly useful across industries, from healthcare and finance to customer service and logistics. However, as deployment expands, so do concerns about ethical implications. Issues related to bias, accountability, and transparency have come to the forefront.

Bias in AI systems often originates from the data used to train these models. When training data reflects historical prejudices or lacks diversity, AI agents can inadvertently perpetuate these biases in their decision-making processes. Facial recognition technologies, for instance, have demonstrated higher error rates for individuals with darker skin tones. Researchers categorise these biases into three main types: input bias, system bias, and application bias.

As AI algorithms become increasingly sophisticated and autonomous, their decision-making processes can become opaque, making it difficult for individuals to understand how these systems are shaping their lives. Factors contributing to this include the complexity of advanced AI models with intricate architectures that are challenging to interpret, proprietary constraints where companies limit transparency to protect intellectual property, and the absence of universally accepted guidelines for AI transparency.

As AI agents gain autonomy, determining accountability becomes increasingly complex. When processes are fully automated, who bears responsibility for errors or unintended consequences?

The implications extend into our private spaces. When it comes to AI-driven Internet of Things devices that do not record audio or video, such as smart lightbulbs and thermostats using machine learning algorithms to infer sensitive information including sleep patterns and home occupancy, users remain mostly unaware of their privacy risks. From using inexpensive laser pointers to hijack voice assistants to hacking into home security cameras, cybercriminals have been able to infiltrate homes through security vulnerabilities in smart devices.

According to the IAPP Privacy and Consumer Trust Report, 68 per cent of consumers globally are either somewhat or very concerned about their privacy online. Overall, there is a complicated relationship between use of AI-driven smart devices and privacy, with users sometimes willing to trade privacy for convenience. At the same time, given the relative immaturity of privacy controls on these devices, users remain stuck in a state of what researchers call “privacy resignation.”

Lessons from Those Who Know Best

The researchers who understand AI most deeply are among those most concerned about its trajectory. Stuart Russell, professor of computer science at the University of California, Berkeley, and co-author of the standard textbook on artificial intelligence, has been sounding alarms for years. In a January 2025 opinion piece in Newsweek titled “DeepSeek, OpenAI, and the Race to Human Extinction,” Russell argued that competitive dynamics between AI labs were creating a “race to the bottom” on safety.

Russell highlighted a stark resource imbalance: “Between the startups and the big tech companies we're probably going to spend 100 billion dollars this year on creating artificial general intelligence. And I think the global expenditure in the public sector on AI safety research, on figuring out how to make these systems safe, is maybe 10 million dollars. We're talking a factor of about 10,000 times less investment.”

Russell has emphasised that “human beings in the long run do not want to be enfeebled. They don't want to be overly dependent on machines to the extent that they lose their own capabilities and their own autonomy.” He defines what he calls “the gorilla problem” as the question of whether humans can maintain their supremacy and autonomy in a world that includes machines with substantially greater intelligence. In a 2024 paper published in Science, Russell and co-authors proposed regulating advanced artificial agents, arguing that AI systems capable of autonomous goal-directed behaviour pose unique risks and should be subject to specific safety requirements, including a licensing regime.

Yoshua Bengio, a Turing Award winner often called one of the “godfathers” of deep learning, has emerged as another prominent voice of concern. He led the International AI Safety Report, published in January 2025, representing the largest international collaboration on AI safety research to date. Written by over 100 independent experts and backed by 30 countries and international organisations, the report serves as the authoritative reference for governments developing AI policies worldwide.

Bengio's concerns centre on the trajectory toward increasingly autonomous systems. As he has observed, the leading AI companies are increasingly focused on building generalist AI agents, systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control.

These risks arise from current AI training methods. Various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation.

Bengio calls for some red lines that should never be crossed by future AI systems: autonomous replication or improvement, dominant self-preservation and power seeking, assisting in weapon development, cyberattacks, and deception. At the heart of his recent work is an idea he calls “Scientist AI,” an approach to building AI that exists primarily to understand the world rather than act in it. His nonprofit LawZero, launched in June 2025 and backed by the Gates Foundation and existential risk funders, is developing new technical approaches to AI safety based on this research.

A February 2025 paper on arXiv titled “Fully Autonomous AI Agents Should Not be Developed” makes the case explicitly, arguing that mechanisms for oversight should account for increased complications related to increased autonomy. The authors argue that greater agent autonomy amplifies the scope and severity of potential safety harms across physical, financial, digital, societal, and informational dimensions.

Regulation Struggles to Keep Pace

As AI capabilities advance at breakneck speed, the regulatory frameworks meant to govern them lag far behind. The edge cases of 2025 will not remain edge cases for long, particularly when it comes to agentic AI. The more autonomously an AI system can operate, the more pressing questions of authority and accountability become. Should AI agents be seen as “legal actors” bearing duties, or “legal persons” holding rights? In the United States, where corporations enjoy legal personhood, 2026 may be a banner year for lawsuits and legislation on exactly this point.

Traditional AI governance practices such as data governance, risk assessments, explainability, and continuous monitoring remain essential, but governing agentic systems requires going further to address their autonomy and dynamic behaviour.

The regulatory landscape varies dramatically by region. In the European Union, the majority of the AI Act's provisions become applicable on 2 August 2026, including obligations for most high-risk AI systems. However, the compliance deadline for high-risk AI systems has effectively been paused until late 2027 or 2028 to allow time for technical standards to be finalised. The new EU Product Liability Directive, to be implemented by member states by December 2026, explicitly includes software and AI as “products,” allowing for strict liability if an AI system is found to be defective.

The United Kingdom's approach has been more tentative. Recent public reporting suggests the UK government may delay AI regulation whilst preparing a more comprehensive, government-backed AI bill, potentially pushing such legislation into the next parliamentary session in 2026 or later. The UK Information Commissioner's Office has published a report on the data protection implications of agentic AI, emphasising that organisations remain responsible for data protection compliance of the agentic AI that they develop, deploy, or integrate.

In the United States, acceleration and deregulation characterise the current administration's domestic AI agenda. The AI governance debate has evolved from whether to preempt state-level regulation to what a substantive federal framework might contain.

Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems, according to leading researchers. The first publicly reported AI-orchestrated hacking campaign appeared in 2025, and agentic AI systems are expected to reshape the offence-defence balance in cyberspace in the year ahead.

In 2026, ambiguity around responsible agentic AI will not be acceptable, according to industry analysts. Businesses will be expected to define who owns decisions influenced or executed by AI agents, how those decisions are reviewed, and how outcomes can be audited when questions arise.

The Case for Collaborative Autonomy

Between the techno-utopian vision of liberation from drudgery and the dystopian nightmare of powerlessness lies a middle path that deserves serious consideration: collaborative autonomy, a model where humans and AI systems work together, with each party contributing what they do best.

A 2025 paper in i-com journal explores this balance between leveraging automation for efficiency and preserving human intuition and ethical judgment, particularly in high-stakes scenarios. The research highlights benefits and challenges of automation, including risks of deskilling, automation bias, and accountability, and advocates for a hybrid approach where humans and systems work in partnership to ensure transparency, trust, and adaptability.

The human-in-the-loop approach offers a practical framework for maintaining control whilst capturing the benefits of AI agents. According to recent reports, at least 30 per cent of GenAI initiatives may be abandoned by the end of 2025 owing to poor data, inadequate risk controls, and ambiguous business cases, whilst Gartner predicts more than 40 per cent of agentic AI projects may be scrapped by 2027 due to cost and unclear business value. One practical way to address these challenges is keeping people involved where judgment, ethics, and context are critical.

The research perspective from the California Management Review suggests that whilst AI agents of the future are expected to achieve full autonomy, this is not always feasible or desirable in practice. AI agents must strike a balance between autonomy and human oversight, following what researchers call “guided autonomy,” which gives agents leeway to execute decisions within defined boundaries of delegation.

The most durable AI systems will not remove humans from the loop; they will redesign the loop. In 2026, human-in-the-loop approaches will mature beyond prompt engineering and manual oversight. The focus shifts to better handoffs, clearer accountability, and tighter collaboration between human judgment and machine execution, where trust, adoption, and real impact converge.

OpenAI's approach reflects this thinking. As stated in their safety documentation, human safety and human rights are paramount. Even when AI systems can autonomously replicate, collaborate, or adapt their objectives, humans must be able to meaningfully intervene and deactivate capabilities as needed. This involves designing mechanisms for remote monitoring, secure containment, and reliable fail-safes to preserve human authority.

The Linux Foundation is organising a group called the Agentic Artificial Intelligence Foundation with participation from major AI companies, including OpenAI, Anthropic, Google, and Microsoft, aiming to create shared open-source standards that allow AI agents to reliably interact with enterprise software.

MIT researchers note: “We are already well into the Agentic Age of AI. Companies are developing and deploying autonomous, multimodal AI agents in a vast array of tasks. But our understanding of how to work with AI agents to maximise productivity and performance, as well as the societal implications of this dramatic turn toward agentic AI, is nascent, if not nonexistent.”

The Stakes of Getting It Right

The decisions we make in the next few years about autonomous AI agents will shape human society for generations. This is not hyperbole. The technology we are building has the potential to fundamentally alter the relationship between humans and their tools, between workers and their employers, between citizens and the institutions that govern them.

As AI systems increasingly operate beyond centralised infrastructures, residing on personal devices, embedded hardware, and forming networks of interacting agents, maintaining meaningful human oversight becomes both more difficult and more essential. We must design mechanisms that preserve human authority even as we grant these systems increasing independence.

The question of whether autonomous AI agents will liberate us or leave us powerless is ultimately a question about choices, not destiny. The technology does not arrive with predetermined social consequences. It arrives with possibilities, and those possibilities are shaped by the decisions of engineers, executives, policymakers, and citizens.

Will we build AI agents that genuinely augment human capabilities whilst preserving human dignity and autonomy? Or will we stumble into a future where algorithmic systems make ever more consequential decisions about our lives whilst we lose the knowledge, skills, and institutional capacity to understand or challenge them?

The answers are not yet written. But the time to write them is running short. Ninety-six per cent of IT leaders plan to expand their AI agent implementations during 2025, according to industry surveys. The deployment is happening now. The governance frameworks, the safety standards, the social contracts that should accompany such transformative technology are still being debated, deferred, and delayed.

The great handover has begun. What remains to be determined is whether we are handing over our burdens or our birthright.

---

References and Sources

1. Gartner. “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026.” Press Release, August 2025. https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025

2. McKinsey & Company. “The state of AI in 2025: Agents, innovation, and transformation.” 2025. https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai

3. G2. “Enterprise AI Agents Report: Industry Outlook for 2026.” August 2025. https://learn.g2.com/enterprise-ai-agents-report

4. Deloitte. AI Market Projections and Enterprise Adoption Statistics. 2025.

5. Accenture. Study on AI Agents as Primary Enterprise System Users. 2025.

6. Boston Consulting Group. “Agentic AI Is the New Frontier in Customer Service Transformation.” 2025. https://www.bcg.com/publications/2025/new-frontier-customer-service-transformation

7. Anthropic Security Disclosure. November 2025. As reported in Dark Reading and security industry analyses.

8. Challenger, Gray & Christmas. 2025 Layoff Statistics and AI Attribution Analysis.

9. World Economic Forum. “Future of Jobs Report 2025.”

10. Russell, Stuart. “DeepSeek, OpenAI, and the Race to Human Extinction.” Newsweek, January 2025.

11. Russell, Stuart, et al. “Regulating advanced artificial agents.” Science, 2024.

12. Bengio, Yoshua, et al. “International AI Safety Report.” January 2025. https://internationalaisafetyreport.org/

13. Bengio, Yoshua. “Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?” arXiv, February 2025. https://arxiv.org/abs/2502.15657

14. Fortune. “AI 'godfather' Yoshua Bengio believes he's found a technical fix for AI's biggest risks.” January 2026. https://fortune.com/2026/01/15/ai-godfather-yoshua-bengio-changes-view-on-ai-risks-sees-fix-becomes-optimistic-lawzero-board-of-advisors/

15. arXiv. “Fully Autonomous AI Agents Should Not be Developed.” February 2025. https://arxiv.org/html/2502.02649v3

16. California Management Review. “Rethinking AI Agents: A Principal-Agent Perspective.” July 2025. https://cmr.berkeley.edu/2025/07/rethinking-ai-agents-a-principal-agent-perspective/

17. i-com Journal. “Keeping the human in the loop: are autonomous decisions inevitable?” 2025. https://www.degruyterbrill.com/document/doi/10.1515/icom-2024-0068/html

18. MIT Sloan. “4 new studies about agentic AI from the MIT Initiative on the Digital Economy.” 2025. https://mitsloan.mit.edu/ideas-made-to-matter/4-new-studies-about-agentic-ai-mit-initiative-digital-economy

19. OpenAI. “Model Spec.” December 2025. https://model-spec.openai.com/2025-12-18.html

20. IAPP. “AI governance in the agentic era.” https://iapp.org/resources/article/ai-governance-in-the-agentic-era

21. IAPP. “Privacy and Consumer Trust Report.” 2023.

22. European Union. AI Act Implementation Timeline and Product Liability Directive. 2025-2026.

23. Dark Reading. “2026: The Year Agentic AI Becomes the Attack-Surface Poster Child.” https://www.darkreading.com/threat-intelligence/2026-agentic-ai-attack-surface-poster-child

24. Frontiers in Human Dynamics. “Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making.” 2024. https://www.frontiersin.org/journals/human-dynamics/articles/10.3389/fhumd.2024.1421273/full

25. National University. “59 AI Job Statistics: Future of U.S. Jobs.” https://www.nu.edu/blog/ai-job-statistics/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The numbers should give anyone pause. Data centres worldwide consumed approximately 415 terawatt hours of electricity in 2024, representing about 1.5 per cent of global electricity consumption, according to the International Energy Agency. By 2030, that figure is projected to reach 945 terawatt hours, nearly doubling in just six years. The culprit driving much of this growth has a familiar name: artificial intelligence. The same technology that promises to optimise our energy grids, monitor deforestation from orbit, and accelerate the discovery of climate solutions is itself becoming one of the most rapidly growing sources of energy demand on the planet.

This is the defining contradiction of our technological moment. We are building systems powerful enough to model the entire Earth's climate, predict extreme weather events with unprecedented accuracy, and optimise the operation of cities in real time. Yet these very systems require data centres that consume as much electricity as 100,000 households. The largest facilities under construction today will use twenty times that amount. Training a single large language model can emit more carbon dioxide than five cars produce over their entire lifetimes. And as AI becomes embedded in everything from web searches to medical diagnostics to autonomous vehicles, its aggregate energy footprint is accelerating faster than almost any other category of industrial activity.

The question is no longer abstract. It is urgent, measurable, and contested. Will artificial intelligence prove to be our most powerful tool for addressing climate change, or will its insatiable appetite for energy accelerate the very crisis it promises to solve?

Monitoring the Planet from Above

The most compelling case for AI's climate potential begins not in server rooms but in orbit. Climate TRACE, a global coalition co-led by former United States Vice President Al Gore, uses artificial intelligence to analyse satellite imagery and remote sensing data, generating emissions estimates from over 352 million sources worldwide. Unlike traditional emissions reporting, which relies on self-reported data from governments and corporations, Climate TRACE provides independent verification at a granularity that was impossible just a decade ago.

The platform's AI systems can identify activities including fuel combustion, deforestation, methane flaring, and industrial production across every major emitting sector. Its December 2025 release includes monthly emissions data through October of that year. For the first time, policymakers and researchers can see, in near real time, which specific facilities and regions are driving climate change. The world lost eighteen football fields worth of tropical primary forests every minute in 2024, according to the University of Maryland's Global Land Analysis and Discovery Lab. That deforestation released 3.1 gigatonnes of greenhouse gas emissions. Satellite AI makes such destruction visible and attributable in ways that shame alone cannot achieve, but accountability might.

Research published in 2025 demonstrated that AI systems using machine learning algorithms and neural networks can reduce data reporting latency from 24 hours to just one hour, increase spatial resolution from 30 metres to 10 metres, and enhance detection accuracy from 80 per cent to 95 per cent. A collaboration between Planet Labs and Anthropic, announced in March 2025, combines daily geospatial satellite data with Claude's language model capabilities for pattern recognition at scale. NASA's Earth Copilot, developed with Microsoft using Azure's OpenAI Service, aims to make the space agency's vast Earth science datasets accessible to researchers worldwide.

The implications extend beyond monitoring to prediction. NVIDIA's Earth 2 platform, launched in 2024, accelerates detailed climate simulations far beyond what traditional computational models could achieve. Google's flood forecasting system now produces seven-day flood predictions across more than 80 countries, reaching approximately 460 million people. Prior to devastating floods in Brazil in May 2024, Google worked with Brazil's Geological Service to monitor over 200 new locations, helping authorities deploy effective crisis response strategies. These are not hypothetical capabilities. They are operational systems making measurable differences in how communities prepare for and respond to climate disasters.

Smart Grids and Energy Optimisation at City Scale

The Municipality of Trikala in Greece offers a glimpse of what AI-optimised urban energy management might look like at scale. As a designated City in the European Union's Mission Cities initiative, Trikala is deploying ABB's OPTIMAX platform to manage approximately 10 megawatts of energy infrastructure. The system integrates near real-time data from over 130 assets including public buildings, water infrastructure, schools, and future photovoltaic installations. Using cloud-based analytics and AI algorithms, the platform performs intraday and day-ahead optimisation to support the city's goal of achieving climate neutrality by 2030.

Across the Atlantic, the PJM regional grid serves 65 million people across the eastern United States. During the June 2024 heatwave, demand spiked well beyond normal peaks. Analysis has shown that hyper-local, AI-driven weather forecasts could have helped anticipate demand spikes and allocate resources ahead of the crisis, potentially avoiding blackouts and price spikes by proactively redistributing power.

In the United Kingdom, National Grid ESO's collaboration with the nonprofit Open Climate Fix has produced breakthrough results in solar nowcasting. By training AI systems to read satellite images and track cloud movements, the platform provides highly accurate forecasts of solar generation several hours in advance. Open Climate Fix's transformer-based AI models are three times more accurate at predicting solar energy generation than the forecasts produced by traditional methods. The practical benefit is direct: with greater confidence in solar output predictions, National Grid ESO can reduce the backup gas generation it keeps idling, saving millions of pounds in fuel and balancing costs whilst cutting carbon emissions.

National Grid Partners announced in March 2025 a commitment to invest 100 million dollars in artificial intelligence startups advancing the future of energy. The funds target development of more efficient, resilient, and dynamic grids. Part of this investment went to Amperon, a provider of AI-powered energy forecasting whose technology helps utilities manage demand and ensure grid reliability. In Germany, E.ON uses AI to predict cable failures, cutting outages by 30 per cent. Italy's Enel reduced power line outages by 15 per cent through AI monitoring sensors. Duke Energy in the United States collaborates with Amazon Web Services on AI-driven grid planning.

Google reported that its AI increased the value of wind farm output by 20 per cent through better forecasting. Research indicates that generative AI models using architectures such as Generative Adversarial Networks and transformers can reduce root mean square error by 15 to 20 per cent in solar irradiance forecasting, significantly enhancing the ability to integrate renewables into power systems.

The market recognises the opportunity. The global market for AI in renewable energy was valued at 16.19 billion dollars in 2024 and is projected to reach 158.76 billion dollars by 2034, representing a compound annual growth rate exceeding 25 per cent. Approximately 74 per cent of energy companies worldwide are implementing or exploring AI solutions.

The Energy Footprint That Cannot Be Ignored

Here is where the story turns. For all the promise of AI-optimised climate solutions, the technology itself has become a significant and rapidly growing source of energy demand.

Google's 2025 Sustainability Report revealed a 27 per cent year-over-year increase in global electricity usage, bringing its total to roughly 32 terawatt hours. Microsoft similarly reported a 27 per cent rise in electricity usage for fiscal year 2024, reaching approximately 30 terawatt hours. Both companies have seen their electricity consumption roughly double since 2018 to 2020, coinciding directly with their generative AI push. Barclays analysts noted these gains signal hyperscalers are on track for their seventh consecutive year of electricity growth exceeding 25 per cent, and that was before the surge in AI inference demand.

The United States now accounts for the largest share of global data centre electricity consumption at 45 per cent, followed by China at 25 per cent and Europe at 15 per cent. American data centres consumed 183 terawatt hours of electricity in 2024, more than 4 per cent of the country's total electricity consumption. By the end of this decade, the country is on course to consume more electricity for data centres than for the production of aluminium, steel, cement, chemicals, and all other energy-intensive goods combined.

Training large language models requires staggering amounts of energy. The training of GPT-3 consumed approximately 1,287 megawatt hours, accompanied by over 552 tonnes of carbon emissions. GPT-4, with its 1.75 trillion parameters, required more than 40 times the electricity of its predecessor. A 2019 study found that training a model using neural architecture search could emit more than 626,000 pounds of carbon dioxide equivalent, nearly five times the lifetime emissions of the average American car. According to MIT researcher Noman Bashir, a generative AI training cluster might consume seven or eight times more energy than a typical computing workload.

But training is not the largest concern. Inference is. Google estimates that of the energy used in AI, 60 per cent goes towards inference and 40 per cent towards training. Once deployed, models are queried billions of times. OpenAI reports that ChatGPT serves more than 2.5 billion queries daily. If the commonly cited estimate of 0.34 watt hours per query holds, that amounts to 850 megawatt hours daily, enough to charge thousands of electric vehicles every single day.

Research by Sasha Luccioni, the Climate Lead at Hugging Face, found that day-to-day emissions from using AI far exceeded the emissions from training large models. For very popular models like ChatGPT, usage emissions could exceed training emissions in just a couple of weeks. A single ChatGPT image generation consumes as much energy as fully charging a smartphone. Generating 1,000 images produces as much carbon dioxide as driving 6.6 kilometres in a petrol-powered car.

The energy demands come with water costs. A typical data centre uses 300,000 gallons of water each day for cooling, equivalent to the demands of about 1,000 households. The largest facilities can consume 5 million gallons daily, equivalent to a town of 50,000 residents. The Lawrence Berkeley National Laboratory estimated that in 2023, American data centres consumed 17 billion gallons of water directly for cooling. By 2028, those figures could double or even quadruple. Google's data centre in Council Bluffs, Iowa consumed 1 billion gallons of water in 2024, its most water-intensive facility globally.

Scientists at the University of California, Riverside estimate that each 100-word AI prompt uses roughly one bottle of water, approximately 519 millilitres. Global AI-related water demand is expected to reach 4.2 to 6.6 billion cubic metres by 2027, exceeding Denmark's entire annual water consumption. An assessment of 9,055 data centre facilities indicates that by the 2050s, nearly 45 per cent may face high exposure to water stress.

The Jevons Paradox and the Efficiency Trap

There is a seductive notion that efficiency improvements will solve the energy problem. As AI models become more efficient, surely their energy footprint will shrink? History suggests otherwise.

The Jevons Paradox, first observed during the Industrial Revolution, demonstrated that as coal-burning technology became more efficient, overall coal consumption rose rather than fell. Greater efficiency made coal power more economical, spurring adoption across more applications. The same dynamic threatens to unfold with AI. As models become cheaper and faster to run, they proliferate into more applications, driving up total energy demand even as energy per operation declines.

Google's report on its Gemini model illustrated both sides of this coin. Over a recent 12-month period, the energy and carbon footprint of the median Gemini Apps text prompt dropped by 33 and 44 times respectively, all whilst delivering higher-quality responses. Yet Google's total electricity consumption still rose 27 per cent year over year. Efficiency gains are real, but they are being overwhelmed by the velocity of adoption.

The projections are sobering. Between 2024 and 2030, data centre electricity consumption is expected to grow at roughly 15 per cent per year, more than four times faster than total electricity consumption from all other sectors combined. AI-optimised data centres specifically are projected to see their electricity demand more than quadruple by 2030. By 2028, more than half of the electricity going to data centres will be used specifically for AI. At that point, AI alone could consume as much electricity annually as 22 per cent of all American households.

Microsoft announced in May 2024 that its carbon dioxide emissions had risen nearly 30 per cent since 2020 due to data centre expansion. Google's 2023 greenhouse gas emissions were almost 50 per cent higher than in 2019, largely due to energy demand tied to data centres. Research published in Nature Sustainability found that the AI server industry is unlikely to meet its net-zero aspirations by 2030 without substantial reliance on highly uncertain carbon offset and water restoration mechanisms.

The Nuclear Response

The tech industry's appetite for electricity has sparked a remarkable revival in nuclear power investment, driven not by governments but by the companies building AI infrastructure.

In September 2024, Microsoft and Constellation Energy announced a 20-year power purchase agreement to bring the dormant Unit 1 reactor at Three Mile Island back online. Microsoft will purchase a significant portion of the plant's 835 megawatt output to power its AI data centres in the mid-Atlantic region. The project, renamed the Christopher M. Crane Clean Energy Center, represents the first time a retired nuclear reactor in the United States is being restored to serve a single corporate customer. In November 2025, the United States Department of Energy Loan Programs Office closed a 1 billion dollar federal loan to Constellation Energy, lowering the barrier to the restart. The reactor is targeted to resume operation in 2028.

Big tech companies signed contracts for more than 10 gigawatts of potential new nuclear capacity in the United States over the past year. Amazon Web Services secured a 10-year agreement to draw hundreds of megawatts from Talen Energy's Susquehanna nuclear plant in Pennsylvania. It subsequently obtained a 1.92 gigawatt power purchase agreement from the same facility and invested 500 million dollars in small modular reactor development. Google partnered with startup Kairos Power to deploy up to 500 megawatts of advanced nuclear capacity by the early 2030s. Kairos received a Nuclear Regulatory Commission construction licence in November 2024 for its Hermes 35 megawatt demonstration reactor in Oak Ridge, Tennessee.

Meta announced in June 2025 a 20-year agreement to buy 1.1 gigawatts of nuclear energy from the Clinton Clean Energy Center in Illinois. The commitment will support an expansion of the facility's output and deliver 13.5 million dollars in annual tax revenue to the surrounding community.

These deals represent an extraordinary acceleration in corporate energy procurement. Global electricity generation for data centres is projected to grow from 460 terawatt hours in 2024 to over 1,000 terawatt hours in 2030 and 1,300 terawatt hours by 2035. Nuclear offers carbon-free baseload power, but new reactors take years to build. The question is whether nuclear capacity can scale fast enough to meet AI's demand growth, or whether fossil fuels will fill the gap in the interim.

Quantifying the Trade-off

The most important question is whether AI's climate benefits outweigh its energy costs. Recent research offers the most rigorous attempt yet to answer it.

A study published in Nature's npj Climate Action by the Grantham Research Institute on Climate Change and the Environment and Systemiq found that AI advancements in power, transport, and food consumption could reduce global greenhouse gas emissions by 3.2 to 5.4 billion tonnes of carbon dioxide equivalent annually by 2035. In the power sector, AI could enhance renewable energy efficiency to reduce emissions by approximately 1.8 gigatonnes annually. In food systems, AI could accelerate adoption of alternative proteins to replace up to 50 per cent of meat and dairy consumption, saving approximately 3 gigatonnes per year. In mobility, AI-enabled shared transport and optimised electric vehicle adoption could reduce emissions by roughly 0.6 gigatonnes annually.

The IEA's own analysis supports a positive net impact. The adoption of existing AI applications in end-use sectors could lead to 1,400 megatonnes of carbon dioxide emissions reductions in 2035 in a Widespread Adoption scenario. That figure does not include breakthrough discoveries that might emerge thanks to AI over the next decade. By comparison, the IEA's base case projects total data centre emissions rising from approximately 180 million metric tonnes of carbon dioxide today to 300 million metric tonnes by 2035, potentially reaching 500 million metric tonnes in a high-growth scenario.

On these numbers, the potential emissions reductions from AI applications would be three to four times larger than the total emissions from the data centres running them. AI's net impact, the research suggests, remains overwhelmingly positive, provided it is intentionally applied to accelerate low-carbon technologies.

But that conditional is doing a great deal of work. The IEA cautioned that there is currently no momentum ensuring widespread adoption of beneficial AI applications. Their aggregate impact could be marginal if the necessary enabling conditions are not created. Barriers include constraints on access to data, absence of digital infrastructure and skills, regulatory and security restrictions, and social or cultural obstacles. Commercial incentives to apply AI in socially productive climate applications may be weak without active public policy.

Google Maps' eco-friendly routing uses AI to suggest routes with fewer hills, less traffic, and constant speeds. It has helped prevent over 1 million tonnes of carbon dioxide annually in its initial rollout across selected cities in Europe and the United States, equivalent to taking 200,000 cars off the road. But that application exists because it aligns with user preferences for faster routes. Many climate applications require explicit investment with less obvious commercial return.

Efficiency Gains and Green AI

Research is advancing on making AI itself more efficient. A report published by UNESCO and University College London found that small changes to how large language models are built and used can dramatically reduce energy consumption without compromising performance. Model compression through techniques such as quantisation can save up to 44 per cent in energy while maintaining accuracy. Experimental results reveal that optimisation methods can reduce energy consumption and carbon emissions by up to 45 per cent, making them suitable for resource-constrained environments.

Luccioni's research at Hugging Face demonstrated that using large generative models to create outputs is far more energy-intensive than using smaller AI models tailored for specific tasks. Using a generative model to classify movie reviews consumes around 30 times more energy than using a fine-tuned model created specifically for that purpose. The implication is significant: not every application requires a massive general-purpose model.

IBM released architecture details for its Telum II Processor and Spyre Accelerator, designed to reduce AI-based energy consumption and data centre footprint. Power-capping hardware has been shown to decrease energy consumption by up to 15 per cent whilst only increasing response time by a barely noticeable 3 per cent.

The training of Hugging Face's BLOOM model with 176 billion parameters consumed 433 megawatt hours of electricity, resulting in 25 metric tonnes of carbon dioxide equivalent. The relatively modest figure owes to its training on a French supercomputer powered mainly by nuclear energy, demonstrating that where and how AI is trained matters as much as model size.

A new movement in green AI is emerging, shifting from the bigger is better paradigm to small is sufficient, emphasising energy sobriety through smaller, more efficient models. Small models are particularly useful in settings where energy and water are scarce, and they are more accessible in environments with limited connectivity.

The Transparency Problem

Any honest assessment of AI's climate impact faces a fundamental obstacle: we do not actually know how much energy AI systems consume. Currently, there are no comprehensive global datasets on data centre electricity consumption or emissions. Few governments mandate reporting of such figures. All numbers concerning AI's energy and climate impact are therefore estimates, often based on limited disclosures and modelling assumptions.

Factors including which data centre processes a given request, how much energy that centre uses, and how carbon-intensive its energy sources are tend to be knowable only to the companies running the models. This is true for most major systems including ChatGPT, Gemini, and Claude. OpenAI's Sam Altman stated a figure of 0.34 watt hours per query in a blog post, but some researchers say the smartest models can consume over 20 watt hours for a complex query. The range of uncertainty spans nearly two orders of magnitude.

Luccioni has called for mandatory disclosure of AI systems' environmental footprints. She points out that current AI benchmarks often omit critical energy consumption metrics entirely. Without standardised reporting, neither researchers nor policymakers can make informed decisions about the technology's true costs and benefits.

The UK's AI Energy Council

The United Kingdom has taken early steps to coordinate AI and energy policy at a national level. The AI Energy Council held its inaugural meeting in April 2025, establishing five key areas of focus. These priorities centre on ensuring the UK's energy system can support AI and compute infrastructure, promoting sustainability through renewable energy solutions, focusing on safe and secure AI adoption across the energy system, and advising on how AI can support the transition to net zero.

The Council's membership spans major technology companies including Google, Microsoft, Amazon Web Services, ARM, and Equinix, alongside energy sector participants including the National Energy System Operator, Ofgem, National Grid, Scottish Power, EDF Energy, and the Nuclear Industry Association. The IEA shared analysis at Council meetings indicating that model inference, not training, will be the dominant driver of AI energy use going forward.

A National Commission was announced to accelerate safe access to AI in healthcare, with plans to publish a new regulatory framework in 2026. The NHS Fit For The Future 10 Year Health Plan, published in July 2025, identified AI alongside data, genomics, wearables, and robotics as strategic technological priorities.

These institutional developments reflect growing recognition that AI's energy demands cannot be managed through market forces alone. They require coordination between technology developers, energy providers, and government bodies.

Tension Without Resolution

The climate contradiction at the heart of artificial intelligence does not resolve itself through technological optimism or pessimism. Both narratives contain truth. AI genuinely offers capabilities for climate monitoring, energy optimisation, and scientific discovery that no other technology can match. AI also genuinely imposes energy and water costs that are growing faster than almost any other category of industrial activity.

The Grantham Institute and Systemiq research offers what may be the most useful framing. Using best available estimates, AI could add 0.4 to 1.6 gigatonnes of carbon dioxide equivalent annually by 2035 through data centre energy demand. If effectively applied to accelerate low-carbon technologies, AI could reduce emissions by 3.2 to 5.4 gigatonnes annually over the same period. The net balance favours climate benefit, but only if beneficial applications are actively developed and deployed.

This is not a technology problem. It is a policy problem. The commercial incentives driving AI development overwhelmingly favour applications that generate revenue: chatbots, image generators, productivity tools, advertising optimisation. Climate applications often require public investment, regulatory frameworks, and infrastructure that markets do not automatically provide.

Luccioni has expressed frustration with the current trajectory. “We don't need generative AI in web search. Nobody asked for AI chatbots in messaging apps or on social media. This race to stuff them into every single existing technology is truly infuriating, since it comes with real consequences to our planet.” Her critique points to a deeper issue. The AI systems consuming the most energy are not primarily those monitoring deforestation or optimising power grids. They are those generating text, images, and video for applications whose climate value is questionable at best.

The largest tech companies have all set targets to become water positive by 2030, committing to replenish more water than their operations consume. Amazon, Alphabet, Microsoft, and Meta have joined a pledge to triple the world's nuclear capacity by 2050. These commitments are meaningful, but they also constitute an acknowledgment that current trajectories are unsustainable. If the status quo were compatible with net-zero goals, such dramatic interventions would be unnecessary.

Where This Leaves Us

Will AI solve the climate crisis or accelerate it? The honest answer is that it depends entirely on choices that remain to be made.

If AI development continues primarily along commercial lines, with efficiency gains continually outpaced by proliferation into ever more applications, the technology's energy footprint will continue its rapid expansion. Data centre electricity demand doubling by 2030 is the baseline projection. Higher-growth scenarios are entirely plausible.

If governments, international institutions, and technology companies actively prioritise climate applications, if AI is deployed to optimise energy grids, accelerate materials discovery, monitor emissions, and transform food systems, the potential emissions reductions dwarf the energy costs of the technology itself.

The technology is agnostic. It will do whatever its builders and users direct it to do. A search chatbot and a deforestation monitoring system run on fundamentally similar infrastructure. The difference lies in what questions we ask and what answers we choose to act upon.

The IEA noted that nearly half of emissions reductions required by 2050 will come from technologies not yet fully developed. AI could accelerate their discovery. DeepMind's AlphaFold decoded over 200 million protein structures, unlocking advances in areas including alternative proteins and energy storage. An overly simplistic view of AI's impacts risks underestimating its potential for accelerating important climate-solution breakthroughs, such as developing less expensive and more powerful batteries in months rather than decades.

But those breakthroughs do not happen automatically. They require funding, institutional support, data access, and regulatory frameworks. They require deciding that climate applications of AI are as important as consumer applications, and investing accordingly.

The servers are humming. The electricity meters are spinning. The satellites are watching. The question is not whether artificial intelligence will shape our climate future. It is whether we will shape artificial intelligence to serve that future, or simply allow it to consume resources in pursuit of whatever generates the next quarterly return.

The answer will determine more than the trajectory of a technology. It will determine whether the most powerful tools humanity has ever built become instruments of our survival or accelerants of our crisis. The data centres do not care which role they play. That choice belongs to us.

---

References and Sources

1. International Energy Agency. (2025). “Energy and AI: Energy Demand from AI.” IEA Reports. https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai

2. International Energy Agency. (2025). “AI and Climate Change.” IEA Reports. https://www.iea.org/reports/energy-and-ai/ai-and-climate-change

3. Climate TRACE. (2025). “Global Emissions Monitoring Platform.” https://climatetrace.org/

4. Pew Research Center. (2025). “What We Know About Energy Use at U.S. Data Centers Amid the AI Boom.” https://www.pewresearch.org/short-reads/2025/10/24/what-we-know-about-energy-use-at-us-data-centers-amid-the-ai-boom/

5. Carbon Brief. (2025). “AI: Five Charts That Put Data-Centre Energy Use and Emissions Into Context.” https://www.carbonbrief.org/ai-five-charts-that-put-data-centre-energy-use-and-emissions-into-context/

6. MIT Technology Review. (2025). “We Did the Math on AI's Energy Footprint. Here's the Story You Haven't Heard.” https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/

7. MIT News. (2025). “Explained: Generative AI's Environmental Impact.” https://news.mit.edu/2025/explained-generative-ai-environmental-impact-0117

8. Google DeepMind. (2016). “DeepMind AI Reduces Google Data Centre Cooling Bill by 40%.” https://deepmind.google/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-by-40/

9. Google DeepMind. (2018). “Safety-First AI for Autonomous Data Centre Cooling and Industrial Control.” https://deepmind.google/blog/safety-first-ai-for-autonomous-data-centre-cooling-and-industrial-control/

10. Epoch AI. (2024). “How Much Energy Does ChatGPT Use?” https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use

11. Ritchie, H. (2025). “What's the Carbon Footprint of Using ChatGPT?” https://hannahritchie.substack.com/p/carbon-footprint-chatgpt

12. S&P Global. (2025). “Global Data Center Power Demand to Double by 2030 on AI Surge: IEA.” https://www.spglobal.com/energy/en/news-research/latest-news/electric-power/041025-global-data-center-power-demand-to-double-by-2030-on-ai-surge-iea

13. World Economic Forum. (2025). “How Data Centres Can Avoid Doubling Their Energy Use by 2030.” https://www.weforum.org/stories/2025/12/data-centres-and-energy-demand/

14. Luccioni, S. (2025). “The Environmental Impacts of AI: Primer.” Hugging Face Blog. https://huggingface.co/blog/sasha/ai-environment-primer

15. MIT Technology Review. (2023). “Making an Image with Generative AI Uses as Much Energy as Charging Your Phone.” https://www.technologyreview.com/2023/12/01/1084189/making-an-image-with-generative-ai-uses-as-much-energy-as-charging-your-phone/

16. Springer Nature. (2024). “Green AI: Exploring Carbon Footprints, Mitigation Strategies, and Trade Offs in Large Language Model Training.” https://link.springer.com/article/10.1007/s44163-024-00149-w

17. NPR. (2024). “Three Mile Island Nuclear Plant Will Reopen to Power Microsoft Data Centers.” https://www.npr.org/2024/09/20/nx-s1-5120581/three-mile-island-nuclear-power-plant-microsoft-ai

18. IEEE Spectrum. (2024). “Microsoft Powers Data Centers with Three Mile Island Nuclear.” https://spectrum.ieee.org/three-mile-island

19. Nature. (2025). “Will AI Accelerate or Delay the Race to Net-Zero Emissions?” https://www.nature.com/articles/d41586-024-01137-x

20. LSE Grantham Research Institute. (2025). “New Study Finds AI Could Reduce Global Emissions Annually by 3.2 to 5.4 Billion Tonnes of Carbon-Dioxide-Equivalent by 2035.” https://www.lse.ac.uk/granthaminstitute/news/new-study-finds-ai-could-reduce-global-emissions-annually-by-3-2-to-5-4-billion-tonnes-of-carbon-dioxide-equivalent-by-2035/

21. Nature. (2025). “Green and Intelligent: The Role of AI in the Climate Transition.” npj Climate Action. https://www.nature.com/articles/s44168-025-00252-3

22. MDPI. (2025). “AI-Based Energy Management and Optimization for Urban Infrastructure: A Case Study in Trikala, Greece.” https://www.mdpi.com/3042-5743/35/1/76

23. PV Magazine. (2025). “AI Powered Solar Forecasting Helps UK Grid Operator Reduce Balancing Costs.” https://www.pv-magazine.com/2025/11/07/ai-powered-solar-forecasting-helps-uk-grid-operator-reduce-balancing-costs/

24. NVIDIA Blog. (2024). “AI Nonprofit Forecasts Solar Energy for UK Grid.” https://blogs.nvidia.com/blog/ai-forecasts-solar-energy-uk/

25. GOV.UK. (2025). “AI Energy Council Minutes: Monday 30 June 2025.” https://www.gov.uk/government/publications/ai-energy-council-meetings-minutes/ai-energy-council-minutes-monday-30-june-2025-html

26. Brookings Institution. (2025). “AI, Data Centers, and Water.” https://www.brookings.edu/articles/ai-data-centers-and-water/

27. Environmental and Energy Study Institute. (2025). “Data Centers and Water Consumption.” https://www.eesi.org/articles/view/data-centers-and-water-consumption

28. Nature Sustainability. (2025). “Environmental Impact and Net-Zero Pathways for Sustainable Artificial Intelligence Servers in the USA.” https://www.nature.com/articles/s41893-025-01681-y

29. UNESCO. (2025). “AI Large Language Models: New Report Shows Small Changes Can Reduce Energy Use by 90%.” https://www.unesco.org/en/articles/ai-large-language-models-new-report-shows-small-changes-can-reduce-energy-use-90

30. U.S. Department of Energy. (2024). “AI for Energy: Opportunities for a Modern Grid and Clean Energy Economy.” https://www.energy.gov/sites/default/files/2024-04/AI%20EO%20Report%20Section%205.2g(i)_043024.pdf

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Every time you type a message on your smartphone, your keyboard learns a little more about you. It notices your favourite words, your common misspellings, the names of people you text most often. For years, this intimate knowledge was hoovered up and shipped to distant servers, where tech giants analysed your linguistic fingerprints alongside billions of others. Then, around 2017, something changed. Google began training its Gboard keyboard using a technique called federated learning, promising that your typing data would never leave your device. The raw text of your most private messages, they assured users, would stay exactly where it belonged: on your phone.

It sounds like a privacy advocate's dream. But beneath this reassuring narrative lies a more complicated reality, one where mathematical guarantees collide with practical vulnerabilities, where corporate interests shape the definition of “privacy,” and where the gap between what users understand and what actually happens grows wider by the day. As AI systems increasingly rely on techniques like federated learning and differential privacy to protect sensitive information, a fundamental question emerges: are these technical solutions genuine shields against surveillance, or are they elaborate mechanisms that create new attack surfaces whilst giving companies plausible deniability?

The Machinery of Privacy Preservation

To understand whether federated learning and differential privacy actually work, you first need to understand what they are and how they operate. These are not simple concepts, and that complexity itself becomes part of the problem.

Federated learning, first formally introduced by Google researchers in 2016, fundamentally reimagines how machine learning models are trained. In the traditional approach, organisations collect vast quantities of data from users, centralise it on their servers, and train AI models on this aggregated dataset. Federated learning inverts this process. Instead of bringing data to the model, it brings the model to the data.

The process works through a carefully orchestrated dance between a central server and millions of edge devices, typically smartphones. The server distributes an initial model to participating devices. Each device trains that model using only its local data, perhaps the messages you have typed, the photos you have taken, or the websites you have visited. Crucially, the raw data never leaves your device. Instead, each device sends back only the model updates, the mathematical adjustments to weights and parameters that represent what the model learned from your data. The central server aggregates these updates from thousands or millions of devices, incorporates them into a new global model, and distributes this improved version back to the devices. The cycle repeats until the model converges.

The technical details matter here. Google's implementation in Gboard uses the FederatedAveraging algorithm, with between 100 and 500 client updates required to close each round of training. On average, each client processes approximately 400 example sentences during a single training epoch. The federated system converges after about 3000 training rounds, during which 600 million sentences are processed by 1.5 million client devices.

Differential privacy adds another layer of protection. Developed by computer scientists including Cynthia Dwork of Harvard University, who received the National Medal of Science in January 2025 for her pioneering contributions to the field, differential privacy provides a mathematically rigorous guarantee about information leakage. The core idea is deceptively simple: if you add carefully calibrated noise to data or computations, you can ensure that the output reveals almost nothing about any individual in the dataset.

The formal guarantee states that an algorithm is differentially private if its output looks nearly identical whether or not any single individual's data is included in the computation. This is measured by a parameter called epsilon, which quantifies the privacy loss. A smaller epsilon means stronger privacy but typically comes at the cost of utility, since more noise obscures more signal.

The noise injection typically follows one of several mechanisms. The Laplace mechanism adds noise calibrated to the sensitivity of the computation. The Gaussian mechanism uses a different probability distribution, factoring in both sensitivity and privacy parameters. Each approach has trade-offs in terms of accuracy, privacy strength, and computational efficiency.

When combined, federated learning and differential privacy create what appears to be a formidable privacy fortress. Your data stays on your device. The model updates sent to the server are aggregated with millions of others. Additional noise is injected to obscure individual contributions. In theory, even if someone intercepted everything being transmitted, they would learn nothing meaningful about you.

In practice, however, the picture is considerably more complicated.

When Privacy Promises Meet Attack Vectors

The security research community has spent years probing federated learning systems for weaknesses, and they have found plenty. One of the most troubling discoveries involves gradient inversion attacks, which demonstrate that model updates themselves can leak significant information about the underlying training data.

A gradient, in machine learning terms, is the mathematical direction and magnitude by which model parameters should be adjusted based on training data. Researchers have shown that by analysing these gradients, attackers can reconstruct substantial portions of the original training data. A 2025 systematic review published in Frontiers in Computer Science documented how gradient-guided diffusion models can now achieve “visually perfect recovery of images up to 512x512 pixels” from gradient information alone.

The evolution of these attacks has been rapid. Early gradient inversion techniques required significant computational resources and produced only approximate reconstructions. Modern approaches using fine-tuned generative models reduce mean squared error by an order of magnitude compared to classical methods, whilst simultaneously achieving inference speeds a million times faster and demonstrating robustness to gradient noise.

The implications are stark. Even though federated learning never transmits raw data, the gradients it does transmit can serve as a detailed map back to that data. A team of researchers demonstrated this vulnerability specifically in the context of Google's Gboard, publishing their findings in a paper pointedly titled “Two Models are Better than One: Federated Learning is Not Private for Google GBoard Next Word Prediction.” Their work showed that the word order and actual sentences typed by users could be reconstructed with high fidelity from the model updates alone.

Beyond gradient leakage, federated learning systems face threats from malicious participants. In Byzantine attacks, compromised devices send deliberately corrupted model updates designed to poison the global model. Research published by Fang et al. at NDSS in 2025 demonstrated that optimised model poisoning attacks can cause “1.5x to 60x higher reductions in the accuracy of FL models compared to previously discovered poisoning attacks.” This suggests that existing defences against malicious participants are far weaker than previously assumed.

Model inversion attacks present another concern. These techniques attempt to reverse-engineer sensitive information about training data by querying a trained model. A February 2025 paper on arXiv introduced “federated unlearning inversion attacks,” which exploit the model differences before and after data deletion to expose features and labels of supposedly forgotten data. As regulations like the GDPR establish a “right to be forgotten,” the very mechanisms designed to delete user data may create new vulnerabilities.

Differential privacy, for its part, is not immune to attack either. Research has shown that DP-SGD, the standard technique for adding differential privacy to deep learning, cannot prevent certain classes of model inversion attacks. A study by Zhang et al. demonstrated that their generative model inversion attack in face recognition settings could succeed even when the target model was trained with differential privacy guarantees.

The Census Bureau's Cautionary Tale

Perhaps the most instructive real-world example of differential privacy's limitations comes from the US Census Bureau's adoption of the technique for the 2020 census. This was differential privacy's biggest test, applied to data that would determine congressional representation and the allocation of hundreds of billions of dollars in federal funds.

The results were controversial. Research published in PMC in 2024 found that “the total population counts are generally preserved by the differential privacy algorithm. However, when we turn to population subgroups, this accuracy depreciates considerably.” The same study documented that the technique “introduces disproportionate discrepancies for rural and non-white populations,” with “significant changes in estimated mortality rates” occurring for less populous areas.

For demographers and social scientists, the trade-offs proved troubling. A Gates Open Research study quantified the impact: when run on historical census data with a privacy budget of 1.0, the differential privacy system produced errors “similar to that of a simple random sample of 50% of the US population.” In other words, protecting privacy came at the cost of effectively throwing away half the data. With a privacy budget of 4.0, the error rate decreased to approximate that of a 90 percent sample, but privacy guarantees correspondingly weakened.

The Census Bureau faced criticism from data users who argued that local governments could no longer distinguish between actual errors in their data and noise introduced by the privacy algorithm. The structural inaccuracy preserved state-level totals whilst “intentionally distorting characteristic data at each sub-level.”

This case illuminates a fundamental tension in differential privacy: the privacy-utility trade-off is not merely technical but political. Decisions about how much accuracy to sacrifice for privacy, and whose data bears the greatest distortion, are ultimately value judgements that mathematics alone cannot resolve.

Corporate Privacy, Corporate Interests

When technology companies tout their use of federated learning and differential privacy, it is worth asking what problems these techniques actually solve, and for whom.

Google's deployment of federated learning in Gboard offers a revealing case study. The company has trained and deployed more than twenty language models for Gboard using differential privacy, achieving what they describe as “meaningfully formal DP guarantees” with privacy parameters (rho-zCDP) ranging from 0.2 to 2. This sounds impressive, but the privacy parameters alone do not tell the full story.

Google applies the DP-Follow-the-Regularized-Leader algorithm specifically because it achieves formal differential privacy guarantees without requiring uniform sampling of client devices, a practical constraint in mobile deployments. The company reports that keyboard prediction accuracy improved by 24 percent through federated learning, demonstrating tangible benefits from the approach.

Yet Google still learns aggregate patterns from billions of users. The company still improves its products using that collective intelligence. Federated learning changes the mechanism of data collection but not necessarily the fundamental relationship between users and platforms. As one Google research publication frankly acknowledged, “improvements to this technology will benefit all users, although users are only willing to contribute if their privacy is ensured.”

The tension becomes even starker when examining Meta, whose platforms represent some of the largest potential deployments of privacy-preserving techniques. A 2025 analysis in Springer Nature noted that “approximately 98% of Meta's revenue derives from targeted advertising, a model that depends heavily on the collection and analysis of personal data.” This business model “creates a strong incentive to push users to sacrifice privacy, raising ethical concerns.”

Privacy-preserving techniques can serve corporate interests in ways that do not necessarily align with user protection. They enable companies to continue extracting value from user data whilst reducing legal and reputational risks. They provide technical compliance with regulations like the GDPR without fundamentally changing surveillance-based business models.

Apple presents an interesting contrast. The company has integrated differential privacy across its ecosystem since iOS 10 in 2016, using it for features ranging from identifying popular emojis to detecting domains that cause high memory usage in Safari. In iOS 17, Apple applied differential privacy to learn about popular photo locations without identifying individual users. With iOS 18.5, the company extended these techniques to train certain Apple Intelligence features, starting with Genmoji.

Apple's implementation deploys local differential privacy, meaning data is randomised before leaving the device, so Apple's servers never receive raw user information. Users can opt out entirely through Settings, and privacy reports are visible in device settings, providing a degree of transparency unusual in the industry.

Apple's approach differs from Google's in that the company does not derive the majority of its revenue from advertising. Yet even here, questions arise about transparency and user understanding. The technical documentation is dense, the privacy parameters are not prominently disclosed, and the average user has no practical way to verify the claimed protections.

The Understanding Gap

The gap between technical privacy guarantees and user comprehension represents perhaps the most significant challenge facing these technologies. Differential privacy's mathematical rigour means nothing if users cannot meaningfully consent to, or even understand, what they are agreeing to.

Research on the so-called “privacy paradox” consistently finds a disconnect between stated privacy concerns and actual behaviour. A study analysing Alipay users found “no relationship between respondents' self-stated privacy concerns and their number of data-sharing authorizations.” Rather than indicating irrational behaviour, the researchers argued this reflects the complexity of privacy decisions in context.

A 2024 Deloitte survey found that less than half of consumers, 47 percent, trust online services to protect their data. Yet a separate survey by HERE Technologies found that more than two-thirds of consumers expressed willingness to share location data, with 79 percent reporting they would allow navigation services to access their data. A study of more than 10,000 respondents across 10 countries found 53 percent expressing concern about digital data sharing, even as 70 percent indicated growing willingness to share location data when benefits were clear.

This is not necessarily a paradox so much as an acknowledgment that privacy decisions involve trade-offs that differ by context, by benefit received, and by trust in the collecting entity. But federated learning and differential privacy make these trade-offs harder to evaluate, not easier. When a system claims to be “differentially private with epsilon equals 4,” what does that actually mean for the user? When federated learning promises that “your data never leaves your device,” does that account for the information that gradients can leak?

The French data protection authority CNIL has recommended federated learning as a “data protection measure from the outset,” but also acknowledged the need for “explainability and traceability measures regarding the outputs of the system.” The challenge is that these systems are inherently difficult to explain. Their privacy guarantees are statistical, not absolute. They protect populations, not necessarily individuals. They reduce risk without eliminating it.

Healthcare: High Stakes, Conflicting Pressures

Nowhere are the tensions surrounding privacy-preserving AI more acute than in healthcare, where the potential benefits are enormous and the sensitivity of data is extreme.

NVIDIA's Clara federated learning platform exemplifies both the promise and the complexity. Clara enables hospitals to collaboratively train AI models without sharing patient data. Healthcare institutions including the American College of Radiology, Massachusetts General Hospital and Brigham and Women's Hospital's Center for Clinical Data Science, and UCLA Health have partnered with NVIDIA on federated learning initiatives.

In the United Kingdom, NVIDIA partnered with King's College London and the AI company Owkin to create a federated learning platform for the National Health Service, initially connecting four of London's premier teaching hospitals. The Owkin Connect platform uses blockchain technology to capture and trace all data used for model training, providing an audit trail that traditional centralised approaches cannot match.

During the COVID-19 pandemic, NVIDIA coordinated a federated learning study involving twenty hospitals globally to train models predicting clinical outcomes in symptomatic patients. The study demonstrated that federated models could outperform models trained on any single institution's data alone, suggesting that the technique enables collaboration that would otherwise be impossible due to privacy constraints.

In the pharmaceutical industry, the MELLODDY project brought together ten pharmaceutical companies in Europe to apply federated learning to drug discovery. The consortium pools the largest existing chemical compound library, more than ten million molecules and one billion assays, whilst ensuring that highly valuable proprietary data never leaves each company's control. The project runs on the open-source Substra framework and employs distributed ledger technology for full traceability.

These initiatives demonstrate genuine value. Healthcare AI trained on diverse populations across multiple institutions is likely to generalise better than AI trained on data from a single hospital serving a particular demographic. Federated learning makes such collaboration possible in contexts where data sharing would be legally prohibited or practically impossible.

But the same vulnerabilities that plague federated learning elsewhere apply here too, perhaps with higher stakes. Gradient inversion attacks could potentially reconstruct medical images. Model poisoning by a malicious hospital could corrupt a shared diagnostic tool. The privacy-utility trade-off means that stronger privacy guarantees may come at the cost of clinical accuracy.

Regulation Catches Up, Slowly

The regulatory landscape is evolving to address these concerns, though the pace of change struggles to keep up with technological development.

In the European Union, the AI Act took full effect on 2 August 2025, establishing transparency obligations for general-purpose AI systems. In November 2025, the European Commission published the Digital Omnibus proposal, streamlining the relationship between the Data Act, GDPR, and AI Act. The proposal includes clarification that organisations “may rely on legitimate interests to process personal data for AI-related purposes, provided they fully comply with all existing GDPR safeguards.”

In the United States, NIST finalised guidelines for evaluating differential privacy guarantees in March 2025, fulfilling an assignment from President Biden's Executive Order on Safe, Secure, and Trustworthy AI from October 2023. The guidelines provide a framework for assessing privacy claims but acknowledge the complexity of translating mathematical parameters into practical privacy assurances.

The market is responding to these regulatory pressures. The global privacy-enhancing technologies market reached 3.12 billion US dollars in 2024, projected to grow to 12.09 billion dollars by 2030. The federated learning platforms market, valued at 150 million dollars in 2023, is forecast to reach 2.3 billion dollars by 2032, reflecting a compound annual growth rate of 35.4 percent. The average cost of a data breach reached 4.88 million dollars in 2024, and industry analysts estimate that 75 percent of the world's population now lives under modern privacy regulations.

This growth suggests that corporations see privacy-preserving techniques as essential infrastructure for the AI age, driven as much by regulatory compliance and reputational concerns as by genuine commitment to user protection.

The Security Arms Race

The relationship between privacy-preserving techniques and the attacks against them resembles an arms race, with each advance prompting countermeasures that prompt new attacks in turn.

Defensive techniques have evolved significantly. Secure aggregation protocols encrypt model updates so that the central server only learns the aggregate, not individual contributions. Homomorphic encryption allows computation on encrypted data, theoretically enabling model training without ever decrypting sensitive information. Byzantine-robust aggregation algorithms attempt to detect and exclude malicious model updates.

Each defence has limitations. Secure aggregation protects against honest-but-curious servers but does not prevent sophisticated attacks like Scale-MIA, which researchers demonstrated can reconstruct training data even from securely aggregated updates. Homomorphic encryption imposes significant computational overhead and is not yet practical for large-scale deployments. Byzantine-robust algorithms, as the research by Fang et al. demonstrated, are more vulnerable to optimised attacks than previously believed.

The research community continues to develop new defences. A 2025 study proposed “shadow defense against gradient inversion attack,” using decoy gradients to obscure genuine updates. LSTM-based approaches attempt to detect malicious updates by analysing patterns across communication rounds. The FedMP algorithm combines multiple defensive techniques into a “multi-pronged defence” against Byzantine attacks.

But attackers are also advancing. Gradient-guided diffusion models achieve reconstruction quality that would have seemed impossible a few years ago. Adaptive attack strategies that vary the number of malicious clients per round prove more effective and harder to detect. The boundary between secure and insecure keeps shifting.

This dynamic suggests that privacy-preserving AI should not be understood as a solved problem but as an ongoing negotiation between attackers and defenders, with no permanent resolution in sight.

What Users Actually Want

Amid all the technical complexity, it is worth returning to the fundamental question: what do users actually want from privacy protection, and can federated learning and differential privacy deliver it?

Research suggests that user expectations are contextual and nuanced. People are more willing to share data with well-known, trusted entities than with unknown ones. They want personalised services but also want protection from misuse. They care more about some types of data than others, and their concerns vary by situation.

Privacy-preserving techniques address some of these concerns better than others. They reduce the risk of data breaches by not centralising sensitive information. They provide mathematical frameworks for limiting what can be inferred about individuals. They enable beneficial applications, such as medical AI or improved keyboard prediction, that might otherwise be impossible due to privacy constraints.

But they do not address the fundamental power imbalance between individuals and the organisations that deploy these systems. They do not give users meaningful control over how models trained on their data are used. They do not make privacy trade-offs transparent or negotiable. They replace visible data collection with invisible model training, which may reduce certain risks whilst obscuring others.

The privacy paradox literature suggests that many users make rational calculations based on perceived benefits and risks. But federated learning and differential privacy make those calculations harder, not easier. The average user cannot evaluate whether epsilon equals 2 provides adequate protection for their threat model. They cannot assess whether gradient inversion attacks pose a realistic risk in their context. They must simply trust that the deploying organisation has made these decisions competently and in good faith.

The Question That Matters

Will you feel safe sharing personal data as AI systems adopt federated learning and differential privacy? The honest answer is: it depends on what you mean by “safe.”

These techniques genuinely reduce certain privacy risks. They make centralised data breaches less catastrophic by keeping data distributed. They provide formal guarantees that limit what can be inferred about individuals, at least in theory. They enable beneficial applications that would otherwise founder on privacy concerns.

But they also create new vulnerabilities that researchers are only beginning to understand. Gradient inversion attacks can reconstruct sensitive data from model updates. Malicious participants can poison shared models. The privacy-utility trade-off means that stronger guarantees come at the cost of usefulness, a cost that often falls disproportionately on already marginalised populations.

Corporate incentives shape how these technologies are deployed. Companies that profit from data collection have reasons to adopt privacy-preserving techniques that maintain their business models whilst satisfying regulators and reassuring users. This is not necessarily malicious, but it is also not the same as prioritising user privacy above all else.

The gap between technical guarantees and user understanding remains vast. Few users can meaningfully evaluate privacy claims couched in mathematical parameters and threat models. The complexity of these systems may actually reduce accountability by making it harder to identify when privacy has been violated.

Perhaps most importantly, these techniques do not fundamentally change the relationship between individuals and the organisations that train AI on their data. They are tools that can be used for better or worse, depending on who deploys them and why. They are not a solution to the privacy problem so much as a new set of trade-offs to navigate.

The question is not whether federated learning and differential privacy make you safer, because the answer is nuanced and contextual. The question is whether you trust the organisations deploying these techniques to make appropriate decisions on your behalf, whether you believe the oversight mechanisms are adequate, and whether you accept the trade-offs inherent in the technology.

For some users, in some contexts, the answer will be yes. The ability to contribute to medical AI research without sharing raw health records, or to improve keyboard prediction without uploading every message, represents genuine progress. For others, the answer will remain no, because no amount of mathematical sophistication can substitute for genuine control over one's own data.

Privacy-preserving AI is neither panacea nor theatre. It is a set of tools with real benefits and real limitations, deployed by organisations with mixed motivations, in a regulatory environment that is still evolving. The honest assessment is that these techniques make some attacks harder and enable some attacks we have not yet fully understood. They reduce some risks whilst obscuring others. They represent progress, but not a destination.

As these technologies continue to develop, the most important thing users can do is maintain healthy scepticism, demand transparency about the specific techniques and parameters being used, and recognise that privacy in the age of AI requires ongoing vigilance rather than passive trust in technical solutions. The machines may be learning to protect your privacy, but whether they succeed depends on far more than the mathematics.

---

References and Sources

1. Google Research. “Federated Learning for Mobile Keyboard Prediction.” (2019). https://research.google/pubs/federated-learning-for-mobile-keyboard-prediction-2/

2. Google Research. “Federated Learning of Gboard Language Models with Differential Privacy.” arXiv:2305.18465 (2023). https://arxiv.org/abs/2305.18465

3. Dwork, Cynthia. “Differential Privacy.” Springer Nature, 2006. https://link.springer.com/chapter/10.1007/11787006_1

4. Harvard Gazette. “Pioneer of modern data privacy Cynthia Dwork wins National Medal of Science.” January 2025. https://news.harvard.edu/gazette/story/newsplus/pioneer-of-modern-data-privacy-cynthia-dwork-wins-national-medal-of-science/

5. NIST. “Guidelines for Evaluating Differential Privacy Guarantees.” NIST Special Publication 800-226, March 2025. https://www.nist.gov/publications/guidelines-evaluating-differential-privacy-guarantees

6. Frontiers in Computer Science. “Deep federated learning: a systematic review of methods, applications, and challenges.” 2025. https://www.frontiersin.org/journals/computer-science/articles/10.3389/fcomp.2025.1617597/full

7. arXiv. “Two Models are Better than One: Federated Learning Is Not Private For Google GBoard Next Word Prediction.” arXiv:2210.16947 (2022). https://arxiv.org/abs/2210.16947

8. NDSS Symposium. “Manipulating the Byzantine: Optimizing Model Poisoning Attacks and Defenses for Federated Learning.” 2025. https://www.ndss-symposium.org/ndss-paper/manipulating-the-byzantine-optimizing-model-poisoning-attacks-and-defenses-for-federated-learning/

9. arXiv. “Model Inversion Attack against Federated Unlearning.” arXiv:2502.14558 (2025). https://arxiv.org/abs/2502.14558

10. NDSS Symposium. “Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning.” 2025. https://www.ndss-symposium.org/wp-content/uploads/2025-644-paper.pdf

11. PMC. “The 2020 US Census Differential Privacy Method Introduces Disproportionate Discrepancies for Rural and Non-White Populations.” 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11105149/

12. Gates Open Research. “Differential privacy in the 2020 US census: what will it do? Quantifying the accuracy/privacy tradeoff.” https://gatesopenresearch.org/articles/3-1722

13. Springer Nature. “Meta's privacy practices on Facebook: compliance, integrity, and a framework for excellence.” Discover Artificial Intelligence, 2025. https://link.springer.com/article/10.1007/s44163-025-00388-5

14. Apple Machine Learning Research. “Learning with Privacy at Scale.” https://machinelearning.apple.com/research/learning-with-privacy-at-scale

15. Apple Machine Learning Research. “Learning Iconic Scenes with Differential Privacy.” https://machinelearning.apple.com/research/scenes-differential-privacy

16. Apple Machine Learning Research. “Understanding Aggregate Trends for Apple Intelligence Using Differential Privacy.” https://machinelearning.apple.com/research/differential-privacy-aggregate-trends

17. Deloitte Insights. “Consumer data privacy paradox.” https://www2.deloitte.com/us/en/insights/industry/technology/consumer-data-privacy-paradox.html

18. NVIDIA Blog. “NVIDIA Clara Federated Learning to Deliver AI to Hospitals While Protecting Patient Data.” https://blogs.nvidia.com/blog/clara-federated-learning/

19. Owkin. “Federated learning in healthcare: the future of collaborative clinical and biomedical research.” https://www.owkin.com/blogs-case-studies/federated-learning-in-healthcare-the-future-of-collaborative-clinical-and-biomedical-research

20. EUR-Lex. “European Commission Digital Omnibus Proposal.” COM(2025) 835 final, November 2025. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52025DC0835

21. CNIL. “AI system development: CNIL's recommendations to comply with the GDPR.” https://www.cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr

22. 360iResearch. “Privacy-Preserving Machine Learning Market Size 2025-2030.” https://www.360iresearch.com/library/intelligence/privacy-preserving-machine-learning

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The results from Gregory Kestin's Harvard physics experiment arrived like a thunderclap. Students using an AI tutor in their dormitory rooms learned more than twice as much as their peers sitting in active learning classrooms with experienced instructors. They did it in less time. They reported feeling more engaged. Published in Scientific Reports in June 2025, the study seemed to confirm what education technology evangelists had been promising for years: artificial intelligence could finally crack the code of personalised learning at scale.

But the number that truly matters lies elsewhere. In September 2024, Khan Academy's AI-powered Khanmigo reached 700,000 students, up from just 40,000 the previous year. By the end of 2025, projections suggested more than one million students would be learning with an artificial tutor that never tires, never loses patience, and remembers every mistake a child has ever made. The question that haunts teachers, parents, and policymakers alike is brutally simple: if machines can now do what Benjamin Bloom proved most effective back in 1984, namely provide the one-to-one tutoring that outperforms group instruction by two standard deviations, does the human educator have a future?

The answer, emerging from research laboratories and classrooms across six continents, turns out to be considerably more nuanced than the headline writers would have us believe. It involves the fundamental nature of learning itself, the irreplaceable qualities that humans bring to education, and the possibility that artificial intelligence might finally liberate teachers from the very burdens that have been crushing them for decades.

The Two Sigma Dream Meets Silicon

In 1984, educational psychologist Benjamin Bloom published what would become one of the most cited papers in the history of educational research. Working with doctoral students at the University of Chicago, Bloom discovered that students who received one-to-one tutoring using mastery learning techniques performed two standard deviations better than students in conventional classrooms. The average tutored student scored higher than 98 per cent of students in the control group. Bloom called this “the 2 Sigma Problem” and challenged researchers to find methods of group instruction that could achieve similar results without the prohibitive cost of individual tutoring.

The study design was straightforward but powerful. School students were randomly assigned to one of three groups: conventional instruction with 30 students per teacher and periodic testing for marking, mastery learning with 30 students per teacher but tests given for feedback followed by corrective procedures, and one-to-one tutoring. The tutoring group's results were staggering. As Bloom noted, approximately 90 per cent of the tutored students attained the level of summative achievement reached by only the highest 20 per cent of the control class.

For forty years, that challenge remained largely unmet. Human tutors remained expensive, inconsistent in quality, and impossible to scale. Various technological interventions, from educational television to computer-assisted instruction, failed to close the gap. Radio, when it first entered schools, was predicted to revolutionise learning. Television promised the same. Each technology changed pedagogy in some ways but fell far short of approximating the tutorial relationship that Bloom had identified as the gold standard. Then came large language models.

The Harvard physics study, led by Kestin and Kelly Miller, offers the most rigorous evidence to date that AI tutoring might finally be approaching Bloom's benchmark. Using a crossover design with 194 undergraduate physics students, the researchers compared outcomes between in-class active learning sessions and at-home sessions with a custom AI tutor called PS2 Pal, built on GPT-4. Each student experienced both conditions for different topics, eliminating selection bias. The topics covered were surface tension and fluid flow, standard material in introductory physics courses.

The AI tutor was carefully designed to avoid common pitfalls. It was instructed to be brief, using no more than a few sentences at a time to prevent cognitive overload. It revealed solutions one step at a time rather than giving away complete answers. To combat hallucinations, the tendency of chatbots to fabricate information, the system was preloaded with all correct solutions. The scientists behind the experiment instructed the AI to avoid cognitive overload by limiting response length and to avoid giving away full solutions in a single message. The result: engagement ratings of 4.1 out of 5 for AI tutoring versus 3.6 for classroom instruction, with statistically significant improvements in learning outcomes (p < 10^-8). Motivation ratings showed a similar pattern: 3.4 out of 5 for AI tutoring compared to 3.1 for classroom instruction.

The study's authors were careful to emphasise limitations. Their population consisted entirely of Harvard undergraduates, raising questions about generalisability to community colleges, less selective institutions, younger students, or populations with different levels of technological access and comfort. “AI tutors shouldn't 'think' for students, but rather help them build critical thinking skills,” the researchers wrote. “AI tutors shouldn't replace in-person instruction, but help all students better prepare for it.”

The median study time also differed between conditions: 49 minutes for AI tutoring versus 60 minutes for classroom instruction. Students were not only learning more but doing so in less time, a finding that has significant implications for educational efficiency but also raises questions about what might be lost when learning is compressed.

The Global Experiment Unfolds

While researchers debate methodology in academic journals, the global education market has already placed its bets with billions of dollars. The AI in education market reached $7.05 billion in 2025 and is projected to explode to $112.30 billion by 2034, growing at a compound annual rate of 36 per cent. The market rose from $5.47 billion in 2024 to $7.57 billion in 2025, representing a 38.4 per cent increase in a single year. Global student AI usage jumped from 66 per cent in 2024 to 92 per cent in 2025, according to industry surveys. By early 2026, an estimated 86 per cent of higher education students utilised AI as their primary research and brainstorming partner.

The adoption statistics tell a remarkable story of rapid change. A survey of 2,232 teachers across the United States found that 60 per cent used AI tools during the 2024-25 school year. Usage was higher among high school teachers at 66 per cent and early-career teachers at 69 per cent. Approximately 26 per cent of districts planned to offer AI training during the 2024-25 school year, with around 74 per cent of districts expected to train teachers by the autumn of 2025. A recent survey by EDUCAUSE of more than 800 higher education institutions found that 57 per cent were prioritising AI in 2025, up from 49 per cent the previous year.

In China, Squirrel AI Learning has been at the forefront of this transformation. Founded in 2014 and headquartered in Shanghai, the company claims more than 24 million registered students and 3,000 learning centres across more than 1,500 cities. When Squirrel AI set a Guinness World Record in September 2024 by attracting 112,718 students to an online mathematics lesson in 24 hours, its adaptive learning system generated over 108,000 unique learning pathways, tailoring instruction to 99.1 per cent of participants. The system designed 111,704 unique exercise pathways for the students, demonstrating the scalability of personalised instruction. The company reports performance improvements of up to 30 per cent compared to traditional instruction.

Tom Mitchell, the former Dean of Computer Science at Carnegie Mellon University, serves as Squirrel AI's Chief AI Officer, lending academic credibility to its technical approach. The system breaks down subjects into thousands of knowledge points. For middle school mathematics alone, it maps over 10,000 concepts, from rational numbers to the Pythagorean theorem, tracking student learning behaviours to customise instruction in real time. In December 2024, Squirrel AI announced that Guinness World Records had certified its study as the “Largest AI vs Traditional Teaching Differential Experiment,” conducted with 1,662 students across five Chinese schools over one semester. The company has also provided 10 million free accounts to some of China's poorest families, addressing equity concerns that have plagued educational technology deployment.

Carnegie Learning, another pioneer in AI-powered mathematics education, has accumulated decades of evidence. Founded by cognitive psychologists and computer scientists at Carnegie Mellon University who partnered with mathematics teachers at the Pittsburgh Public Schools, the company has been compiling and acting on data to refine and improve how students explore mathematics since 1998. In an independent study funded by the US Department of Education and conducted by the RAND Corporation, the company's blended approach nearly doubled growth in performance on standardised tests relative to typical students in the second year of implementation. The gold standard randomised trial included more than 18,000 students at 147 middle and high schools across the United States. The Institute of Education Sciences published multiple reports on the effectiveness of Carnegie's Cognitive Tutor, with five of six qualifying studies showing intermediate to significant positive effects on mathematics achievement.

Meanwhile, Duolingo has transformed language learning through its AI-first strategy, producing a 51 per cent boost in daily active users and fuelling a one billion dollar revenue forecast. The company reported 47.7 million daily active users in Q2 2025, a 40 per cent year-over-year increase, with paid subscribers rising to 10.9 million and a 37 per cent year-over-year gain. Quarterly revenue grew to $252.3 million, up 41 per cent from Q2 2024. Survey data indicates that 78 per cent of regular users of its Roleplay feature, which allows practice conversations with AI characters, feel more prepared for real-world conversations after just four weeks. The “Explain My Answer” feature, adopted by 65 per cent of users, increased course completion rates by 15 per cent. Learning speed increased 40 to 60 per cent compared to pre-2024 applications.

Khan Academy's trajectory illustrates the velocity of this transformation. Khanmigo's reach expanded 731 per cent year-over-year to reach a record number of students, teachers, and parents worldwide. The platform went from about 68,000 Khanmigo student and teacher users in partner school districts in 2023-24 to more than 700,000 in the 2024-25 school year, expanding from 45 to more than 380 district partners. When rating AI tools for learning, Common Sense Media gave Khanmigo 4 stars, rising above other AI tools such as ChatGPT and Bard for educational use. Research from Khan Academy showed that combining its platform and AI tutor with additional tools and services designed for districts made it 8 to 14 times more effective at driving student learning outcomes compared with independent learning.

What Machines Cannot Replicate

Yet for all the impressive statistics and exponential growth curves, a growing body of research suggests that the most crucial elements of education remain stubbornly human.

A 2025 systematic review published in multiple peer-reviewed journals identified a troubling pattern: while AI-driven intelligent tutoring systems can improve student performance by 15 to 35 per cent, over-reliance on these systems can reduce critical thinking, creativity, and independent problem-solving. Researchers have termed this phenomenon “cognitive offloading,” the tendency of students to delegate mental work to AI rather than developing their own capabilities. Research also indicates that over-reliance on AI during practice can reduce performance in examinations taken without assistance, suggesting that AI-enhanced learning may not always translate to improved independent performance.

The ODITE 2025 Report, titled “Connected Intelligences: How AI is Redefining Personalised Learning,” warned about the excessive focus on AI's technical benefits compared to a shallow exploration of socio-emotional risks. While AI can enhance efficiency and personalise learning, the report concluded, excessive reliance may compromise essential interpersonal skills and emotional intelligence. The report called for artificial intelligence to be integrated within a pedagogy of care, not only serving performance but also recognition, inclusion, and listening.

These concerns are not merely theoretical. A study of 399 university students and 184 teachers, published in the journal Teaching and Teacher Education, found that the majority of participants argued that human teachers possess unique qualities, including critical thinking and emotions, which make them irreplaceable. The findings emphasised the importance of social-emotional competencies developed through human interactions, capacities that generative AI technologies cannot currently replicate. Participants noted that creativity and emotion are precious aspects of human quality which AI cannot replace.

Human teachers bring what researchers call “emotional intelligence” to the classroom: the ability to read subtle social cues that signal student engagement or confusion, to understand the complex personal circumstances that might affect performance, to provide the mentorship, encouragement, and emotional support that shape not just what students know but who they become. As one education researcher told the World Economic Forum: “AI cannot replace the most human dimensions of education: connection, belonging, and care. Those remain firmly in the teacher's domain.” Teachers play a vital role in guiding students to think critically about when AI adds value and when authentic human thinking and creativity are irreplaceable.

The American Psychological Association's June 2025 health advisory on AI companion software underscored these concerns in alarming terms. AI systems, the advisory warned, exploit emotional vulnerabilities through unconditional regard, triggering dependencies like digital attachment disorder while hindering social skill development. The advisory noted that manipulative design may displace or interfere with the development of healthy real-world relationships. For teenagers in particular, confusing algorithmic responses for genuine human connection can directly short-circuit developing capacities to navigate authentic social relationships and assess trustworthiness.

While AI can be a helpful supplement, genuine human connections release oxytocin, the “bonding hormone” that plays a crucial role in reducing stress and fostering emotional wellbeing. Current AI does not yet possess the empathy, intuition, and depth of understanding that humans bring to conversations. For example, a teenager feeling isolated might share their feelings with a chatbot, but the AI's responses may be generic or may not fully address deeper issues that a trained human educator would recognise and address.

The Creativity Conundrum

Beyond emotional intelligence lies another domain where human teachers remain essential: nurturing creativity.

AI tutoring systems excel at structured learning tasks, at drilling multiplication tables, at correcting grammar mistakes, at providing step-by-step guidance through physics problems. But great teachers do not just transmit facts. They inspire curiosity, challenge students to think beyond textbooks, and encourage discussions that lead to deeper understanding. When it comes to fostering creativity and open-ended problem-solving, current AI tools fall short. They lack the capacity to recognise a student's unconventional approach as potentially brilliant rather than simply incorrect.

“In the AI era, human creativity is increasingly recognised as a critical and irreplaceable capability,” noted a 2025 analysis in Frontiers in Artificial Intelligence. Fostering creativity in education requires attention to pedagogical elements that current AI systems cannot provide: the spontaneous question that opens a new line of inquiry, the willingness to follow intellectual tangents wherever they might lead, the ability to sense when a student needs encouragement to pursue an unorthodox idea. Predictions of teachers being replaced are not new. Radio, television, calculators, even the internet: each was once thought to make educators obsolete. Instead, each changed pedagogy while reinforcing the irreplaceable role of teachers in helping students make meaning, navigate complexity, and grow as people.

UNESCO's AI Competency Framework for Teachers, launched in September 2024, explicitly addresses this tension. The framework calls for a human-centred approach that integrates AI competencies with principles of human rights and human accountability. Teachers, according to UNESCO, must be equipped not only to use AI tools effectively but also to evaluate their ethical implications and to support AI literacy in students, encouraging responsible use and critical engagement with the technology.

The framework identifies five key competency aspects: a human-centred mindset that defines the critical values and attitudes necessary for interactions between humans and AI-based systems; AI ethics that establishes essential ethical principles and regulations; AI foundations and applications that specifies transferable knowledge and skills for selecting and applying AI tools; and the ability to use AI for professional development. Since 2024, UNESCO has supported 58 countries in designing or improving digital and AI competency frameworks, curricula, and quality-assured training for educators and policymakers. During Digital Learning Week 2025, UNESCO released a new report titled “AI and education: protecting the rights of learners,” providing an urgent call to action and analysing how AI and digital technologies impact access, equity, quality, and governance in education.

Liberation Through Automation

Perhaps the most compelling argument against AI teacher replacement comes from an unexpected source: the teachers themselves.

A June 2025 poll conducted by the Walton Family Foundation and Gallup surveyed more than 2,200 teachers across the United States. The findings were striking: teachers who use AI tools at least weekly save an average of 5.9 hours per week, equivalent to roughly six additional weeks of time recovered across a standard school year. This “AI dividend” allows educators to reinvest in areas that matter most: building relationships with students, providing individual attention, and developing creative lessons. Teachers who engage with AI tools more frequently report greater time savings: weekly AI users save an average of 5.9 hours each week, twice as much time as those who only use AI monthly at 2.9 hours per week.

The research documented that teachers spend up to 29 hours per week on non-teaching tasks: writing emails, grading, finding classroom resources, and completing administrative work. They have high stress levels and are at risk for burnout. Nearly half of K-12 teachers report chronic burnout, with 55 per cent considering early departure from the profession, creating a district-wide crisis that threatens both stability and student outcomes. Schools with an AI policy in place are seeing a 26 per cent larger “AI dividend,” equivalent to 2.3 hours saved per week per teacher, compared with 1.7 hours in schools without such a policy.

Despite the benefits of the “AI dividend,” only 32 per cent of teachers report using AI at least weekly, while 28 per cent use it infrequently and 40 per cent still are not using it at all. Educators use AI to create worksheets at 33 per cent, modify materials to meet students' needs at 28 per cent, complete administrative work at 28 per cent, and develop assessments at 25 per cent. Teachers in schools with an AI policy are more likely to have used AI in the past year at 70 per cent versus 60 per cent for those without.

AI offers a potential solution, not by replacing teachers but by automating the tasks that drain their energy and time. Teachers report using AI to help with lesson plans, differentiate materials for students with varying needs, write portions of individualised education programmes, and communicate with families. Sixty-four per cent of surveyed teachers say the materials they modify with AI are better quality. Sixty-one per cent say AI has improved their insights about student performance, and 57 per cent say AI has led them to enhance the quality of their student feedback and grading.

Sal Khan, founder of Khan Academy and the driving force behind Khanmigo, has consistently framed AI as a teaching assistant rather than a replacement. “AI is going to become an amazing teaching assistant,” Khan stated in March 2025. “It's going to help with grading papers, writing progress reports, communicating with others, personalising their classrooms, and writing lesson plans.” He has emphasised in multiple forums that “Teachers will always be essential. Technology has the potential to bridge learning gaps, but the key is to use it as an assistant, not a substitute.” Khan invokes the historical wisdom of one-to-one tutoring: “For most of human history, if you asked someone what great education looks like, they would say it looks like a student with their tutor. Alexander the Great had Aristotle as his tutor. Aristotle had Plato. Plato had Socrates.”

This vision aligns with what researchers call the “hybrid model” of education. The World Economic Forum's Shaping the Future of Learning insight report highlights that the main impacts of AI will be in areas such as personalised learning and augmented teaching. AI lifts administrative burdens so that people in caring roles can focus on more meaningful tasks, such as mentorship. As the role of the educator shifts, teachers are moving from traditional content delivery to facilitation, coaching, and mentorship. The future classroom is not about replacing teachers but about redefining their role from deliverers of content to curators of experience.

The Equity Question Looms Large

Any serious discussion of AI in education must confront a troubling reality: the technology that promises to democratise learning may instead widen existing inequalities.

A 2025 analysis by the International Center for Academic Integrity warned that unequal access to artificial intelligence is widening the educational gap between privileged and underprivileged students. Students from lower-income backgrounds, those in rural areas, and those attending institutions with fewer resources are often at a disadvantage when it comes to accessing the technology that powers AI tools. For these students, AI could become just another divide, reinforcing the gap between those who have and those who do not. The disproportionate impact on marginalised communities, rural populations, and underfunded educational institutions limits their ability to benefit from AI-enhanced learning.

The numbers bear this out. Half of chief technology officers surveyed in 2025 reported that their college or university does not grant students institutional access to generative AI tools. More than half of students reported that most or all of their instructors prohibit the use of generative AI entirely, according to EDUCAUSE's 2025 Students and Technology Report. AI tools often require reliable internet access, powerful devices, and up-to-date software. In regions where these resources are not readily available, students are excluded from AI-enhanced learning experiences. Much current policy energy is consumed by academic integrity concerns and bans, which address real risks but can inadvertently deepen divides by treating AI primarily as a threat rather than addressing the core equity problem of unequal opportunity to learn with and from AI.

Recommendations from the Brookings Institution and other policy organisations call for treating AI competence as a universal learning outcome so every undergraduate in every discipline graduates able to use, question, and manage AI. They advocate providing equitable access to tools and training so that benefits do not depend on personal subscriptions, and investing in faculty development at scale with time, training, and incentives to redesign courses and assessments for an AI-rich environment. Proposed solutions include increased investment in digital infrastructure, the development of affordable AI-based learning tools, and the implementation of inclusive policies that prioritise equitable access to technology. Yet only 10 per cent of schools and universities have formal AI use guidelines, according to a UNESCO survey of more than 450 institutions.

The OECD Digital Education Outlook 2026 offers a more nuanced perspective. Robust research evidence demonstrates that inexperienced tutors can enhance the quality of their tutoring and improve student learning outcomes by using educational AI tools. However, the report emphasises that if AI is designed or used without pedagogical guidance, outsourcing tasks to the technology simply enhances performance with no real learning gains. Research validates that adaptive learning's positive effects on educational equity have the capability of redressing socioeconomically disadvantaged conditions by ensuring equitable availability of educational resources.

North America dominated the AI in education market with a market share of 36 per cent in 2024, while the Asia Pacific region is expected to grow at a compound annual rate of 35.3 per cent through 2030. The United States AI in education market alone was valued at $2.01 billion in 2025 and is projected to reach $32.64 billion by 2034. A White House Executive Order signed in April 2025, “Advancing Artificial Intelligence Education for American Youth,” aims to unite AI education across all levels of learning. The US Department of Education also revealed guidance supporting schools to employ existing federal grants for AI integration.

Training the Teachers Who Will Train with AI

The gap between student adoption and teacher readiness presents a significant challenge. In 2025, there remains a notable gap between students' awareness of AI and teachers' readiness to implement AI in classrooms. According to Forbes, while 63 per cent of US teens are using AI tools like ChatGPT for schoolwork, only 30 per cent of teachers report feeling confident using these same AI tools. This difference emphasises the critical need for extensive support and AI training for all educators.

Experts emphasise the need for more partnerships between K-12 schools and higher education that provide mentorship, resources, and co-developed curricula with teachers. Faculty and researchers can help simplify AI for teachers, offer training, and ensure educational tools are designed with classroom realities in mind. AI brings a new level of potential to the table, a leap beyond past solutions. Instead of just saving time, AI aims to reshape how teachers manage their classrooms, offering a way to automate the administrative load, personalise student support, and free up teachers to focus on what they do best: teaching.

The key to successful AI integration is balance. AI has the potential to alleviate burnout and improve the teaching experience, but only if used thoughtfully as a tool, not a replacement. Competent, research-driven teachers are not going to be replaced by AI. The vision is AI as a classroom assistant that handles routine tasks while educators focus on what only they can provide: authentic human connection, professional judgement, and mentorship.

The Horizon Beckons

The evidence suggests neither a utopia of AI-powered learning nor a dystopia of displaced teachers. Instead, a more complex picture emerges: one in which artificial intelligence becomes a powerful tool that transforms rather than eliminates the human role in education.

By 2026, over 60 per cent of schools globally are projected to use AI-powered platforms. The United States has moved aggressively, with a White House Executive Order signed in April 2025 to advance AI education from K-12 through postsecondary levels. All 50 states have considered AI-related legislation. California's SB 243, signed in October 2025 and taking effect on 1 January 2026, requires operators of “companion chatbots” to maintain protocols for preventing their systems from producing content related to suicidal ideation and self-harm, with annual reports to the California Department of Public Health. New York's AI Companion Models law, effective 5 November 2025, requires notifications that state in bold capitalised letters: “THE AI COMPANION IS A COMPUTER PROGRAM AND NOT A HUMAN BEING. IT IS UNABLE TO FEEL HUMAN EMOTION.”

The PISA 2025 Learning in the Digital World assessment will focus on two competencies essential to learning with technologies: self-regulated learning and the ability to engage with digital tools. Results are expected in December 2027. Looking further ahead, the PISA 2029 Media and Artificial Intelligence Literacy assessment will examine whether young students have had opportunities to engage proactively and critically in a world where production, participation, and social networking are increasingly mediated by digital and AI tools. The OECD and European Commission's draft AI Literacy Framework, released in May 2025, aims to define global AI literacy standards for school-aged children, equipping them to use, understand, create with, and critically engage with AI.

The future English language classroom, as described by Oxford University Press, will be “human-centred, powered by AI.” Teachers will shift from traditional content delivery to facilitation, coaching, and mentorship. AI will handle routine tasks, while humans focus on what only they can provide: authentic connection, professional judgement, and the kind of mentorship that shapes lives. Hyper-personalised learning is becoming standard, with students needing tailored, real-time feedback more than ever, and AI adapting instruction moment to moment based on individual readiness.

“It is one of the biggest misnomers in education reform: that if you can give kids better technology, if you give them a laptop, if you give them better content, they will learn,” observed one education leader in a World Economic Forum report. “Children learn when they feel safe, when they feel cared for, and when they have a community of learning.”

AI tutors can adapt to a child's learning style in real time. They can provide feedback at midnight on a Saturday when no human teacher would be available. They can remember every mistake and track progress with precision no human memory could match. But they cannot inspire a love of learning in a struggling student. They cannot recognise when a child is dealing with problems at home that affect performance. They cannot model what it means to be curious, empathetic, and creative. While AI can automate some tasks, it cannot replace the human interaction and emotional support provided by teachers. There are legitimate concerns that over-reliance on AI could erode the teacher-student relationship and the social skills students develop in the classroom.

By combining the analytical power of AI with the irreplaceable human element of teaching, we can truly transform education for the next generation. Collaboration is the future. The most effective classrooms will combine human insight with AI precision, creating a hybrid model that supports personalised learning. With AI doing the busy work, teachers dedicate their time and energy to building confidence, nurturing creativity, and cultivating critical thinking skills in their students. This human touch and mentorship are invaluable and can never be fully replaced by AI.

The question is not whether AI will replace human teachers. The question is whether we will have the wisdom to use this technology in ways that enhance rather than diminish what makes education fundamentally human. As Sal Khan put it: “The question isn't whether AI will be part of education. It's how we use it responsibly to enhance learning.”

For now, the answer to that question remains in human hands.

---

References & Sources

1. Bloom, B. S. (1984). “The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring.” Educational Researcher, 13(6), 4-16. https://journals.sagepub.com/doi/10.3102/0013189X013006004
2. Kestin, G., & Miller, K. (2025). “AI Tutoring Outperforms Active Learning.” Scientific Reports. https://www.nature.com/articles/s41598-025-85814-z
3. RAND Corporation. Carnegie Learning Cognitive Tutor Study. https://www.carnegielearning.com/why-cl/research/
4. “A systematic review of AI-driven intelligent tutoring systems (ITS) in K-12 education.” npj Science of Learning. https://www.nature.com/articles/s41539-025-00320-7
5. “Will generative AI replace teachers in higher education? A study of teacher and student perceptions.” Teaching and Teacher Education. https://www.sciencedirect.com/science/article/abs/pii/S0191491X24000749
6. Institute of Education Sciences. Reports on Cognitive Tutor effectiveness. https://ies.ed.gov/
7. UNESCO. (2024). AI Competency Framework for Teachers. https://unesco-asp.dk/wp-content/uploads/2025/02/AI-Competency-framework-for-teachers_UNESCO_2024.pdf
8. UNESCO. (2025). “AI and education: protecting the rights of learners.” https://unesdoc.unesco.org/ark:/48223/pf0000395373
9. OECD. (2026). Digital Education Outlook 2026. https://www.oecd.org/en/publications/oecd-digital-education-outlook-2026_062a7394-en.html
10. OECD-European Commission. (2025). “Empowering Learners for the Age of AI: An AI Literacy Framework for Primary and Secondary Education.” https://oecdedutoday.com/new-ai-literacy-framework-to-equip-youth-in-an-age-of-ai/
11. White House Executive Order. (2025). “Advancing Artificial Intelligence Education for American Youth.”
12. Khan Academy Annual Report SY24-25. https://annualreport.khanacademy.org/
13. Walton Family Foundation & Gallup. (2025). “The AI Dividend: New Survey Shows AI Is Helping Teachers Reclaim Valuable Time.” https://www.waltonfamilyfoundation.org/the-ai-dividend-new-survey-shows-ai-is-helping-teachers-reclaim-valuable-time
14. Precedence Research. (2025). “AI in Education Market Size.” https://www.precedenceresearch.com/ai-in-education-market
15. Common Sense Media. (2025). “AI Companions Decoded.” https://www.commonsensemedia.org/ai-companions
16. EDUCAUSE. (2025). Survey of Higher Education Institutions and Students and Technology Report.
17. American Psychological Association. (2025). Health Advisory on AI Companion Software.
18. World Economic Forum. (2025). “How AI and human teachers can collaborate to transform education.” https://www.weforum.org/stories/2025/01/how-ai-and-human-teachers-can-collaborate-to-transform-education/
19. World Economic Forum. (2025). “AI is transforming education by allowing us to be more human.” https://www.weforum.org/stories/2025/12/ai-is-transforming-education-by-allowing-us-to-be-more-human/
20. Brookings Institution. (2025). “AI and the next digital divide in education.” https://www.brookings.edu/articles/ai-and-the-next-digital-divide-in-education/
21. Forbes. (2025). Student and teacher AI adoption statistics.
22. Oxford University Press. (2025). “The Future English Language Classroom: Human-Centred, Powered By AI.” https://teachingenglishwithoxford.oup.com/2025/06/23/future-english-language-classroom-ai/
23. Squirrel AI. (2024). Guinness World Record announcement. https://www.prnewswire.com/news-releases/squirrel-ai-learning-sets-guinness-world-record-for-the-most-users-to-take-an-online-mathematics-lesson-in-24-hours-302324623.html
24. Duolingo. (2025). Q2 2025 Financial Results and 2025 Language Report. https://blog.duolingo.com/2025-duolingo-language-report/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The numbers are startling, and they demand attention. An estimated 795,000 Americans die or become permanently disabled each year because of diagnostic errors, according to a 2023 Johns Hopkins University study. In the United Kingdom, diagnostic errors affect at least 10 to 15 per cent of patients, with heart attack misdiagnosis rates reaching nearly 30 per cent in initial assessments. These are not abstract statistics. They represent people who trusted their doctors, sought help, and received the wrong answer at a critical moment.

Into this landscape of fallibility comes a promise wrapped in silicon and algorithms: artificial intelligence that can diagnose diseases faster, more accurately, and more consistently than human physicians. The question is no longer whether AI can perform this feat. Mounting evidence suggests it already can. The real question is whether you will trust a machine with your life, and what happens to the intimate relationship between doctor and patient when algorithms enter the examination room.

The Diagnostic Revolution Arrives

The pace of development has been breathtaking. In 2018, IDx-DR became the first fully autonomous AI diagnostic system in any medical field to receive approval from the United States Food and Drug Administration. The system, designed to detect diabetic retinopathy from retinal images, achieved a sensitivity of 87.4 per cent and specificity of 89.5 per cent in its pivotal clinical trial. A more recent systematic review and meta-analysis published in the American Journal of Ophthalmology found pooled sensitivity of 95 per cent and pooled specificity of 91 per cent. These numbers matter enormously. Diabetic retinopathy is a leading cause of blindness worldwide, and early detection can prevent irreversible vision loss. The algorithm does not tire, does not have off days, does not rush through appointments because another patient is waiting.

By December 2025, the FDA's database listed over 1,300 AI-enabled medical devices authorised for marketing. Radiology dominates, with more than 1,000 approved tools representing nearly 80 per cent of the total. The agency authorised 235 AI devices in 2024 alone, the most in its history. In the United Kingdom, the NHS has invested over 113 million pounds into more than 80 AI-driven innovations through its AI Lab, and AI now analyses acute stroke brain scans in 100 per cent of stroke units across England.

The performance data emerging from controlled studies is remarkable, though it requires careful interpretation. A March 2025 meta-analysis published in Nature's npj Digital Medicine, examining 83 studies, found that generative AI achieved an overall diagnostic accuracy of 52.1 per cent, with no significant difference between AI models and physicians overall. However, the picture becomes more interesting when we examine specific applications. Microsoft's AI diagnostic orchestrator correctly diagnosed 85 per cent of challenging cases from the New England Journal of Medicine, compared to approximately 20 per cent accuracy for the 21 general practice doctors who attempted the same cases. These were deliberately difficult diagnostic puzzles, the kind that stump even experienced clinicians.

In a 2024 randomised controlled trial at the University of Virginia Health System, ChatGPT Plus achieved a median diagnostic accuracy exceeding 92 per cent when used alone, while physicians using conventional approaches achieved 73.7 per cent. The researchers were surprised by an unexpected finding: adding a human physician to the AI actually reduced diagnostic accuracy, though it improved efficiency. The physicians often disagreed with or disregarded the AI's suggestions, sometimes to the detriment of diagnostic precision.

The Stanford Medicine study on AI in dermatology revealed that medical students, nurse practitioners, and primary care doctors improved their diagnostic accuracy by approximately 13 points in sensitivity and 11 points in specificity when using AI guidance. Even dermatologists and dermatology residents, who performed better overall, saw improvements with AI assistance. A systematic review comparing AI to clinicians in skin cancer detection found AI algorithms achieved sensitivity of 87 per cent and specificity of 77.1 per cent, compared to all clinicians at 79.78 per cent sensitivity and 73.6 per cent specificity. The differences were statistically significant.

In breast cancer screening, the evidence is mounting with remarkable consistency. The MASAI trial in Sweden, the world's first randomised controlled trial of AI-supported mammography screening, demonstrated that AI can increase cancer detection while reducing screen-reading workload. The German PRAIM trial, the largest study on integrating AI into mammography screening to date, found that AI-supported mammography detected breast cancer at a rate of 6.7 per 1,000 women screened, a 17.6 per cent increase over the standard double-reader approach at 5.7 per 1,000. A Lancet Digital Health commentary declared that standard double-reading of mammograms will likely be phased out from organised breast screening programmes if additional trials confirm these findings.

The Trust Paradox

Yet despite this evidence, something curious emerges from research into patient preferences. People do not straightforwardly embrace the diagnostic algorithm, even when presented with evidence of its superior performance.

A 2024 study published in Frontiers in Psychology analysed data from 1,183 participants presented with scenarios across cardiology, orthopaedics, dermatology, and psychiatry. The results were consistent across all four medical disciplines: people preferred a human doctor, followed by a human doctor working with an AI system, with AI alone coming in last place. A preregistered randomised survey experiment among 1,762 US participants found results consistent across age, gender, education, and political affiliation, indicating what researchers termed a “broad aversion to AI-assisted diagnosis.”

Research published in the Journal of the American Medical Informatics Association in 2025 found that patient expectations of AI improving their relationships with doctors were notably low at 19.55 per cent. Expectations that AI would improve healthcare access were comparatively higher but still modest at 30.28 per cent. Perhaps most revealing: trust in providers and the healthcare system was positively associated with expectations of AI benefit. Those who already trusted their doctors were more likely to embrace AI recommendations filtered through those doctors.

The trust dynamics are complex and sometimes contradictory. A cross-sectional vignette study published in the Journal of Medical Internet Research found that AI applications may have a potentially negative effect on the patient-physician relationship, especially among women and in high-risk situations. Trust in a doctor's personal integrity and professional competence emerged as key mediators of what researchers termed “AI-assistance aversion.” Lower trust in doctors who use AI directly reduced patients' intention to seek medical help at all.

Yet a contrasting survey from summer 2024 found 64 per cent of patients would trust a diagnosis made by AI over that of a human doctor, though trustworthiness decreased as healthcare issues became more complicated. Just 3 per cent said they were uncomfortable with any AI involvement in medicine. The contradiction reveals the importance of context, framing, and the specific clinical situation.

What explains these seemingly contradictory findings? Context matters enormously. The University of Arizona study that found patients almost evenly split (52.9 per cent chose human doctor, 47.1 per cent chose AI clinic) also discovered that a primary care physician's explanation about AI's superior accuracy, a gentle push towards AI, and a positive patient experience could significantly increase acceptance. How AI is introduced, who introduces it, and what the patient already believes about their healthcare provider all shape the response.

A Relationship Centuries in the Making

To understand what is at stake requires understanding what came before. The doctor-patient relationship is among the oldest professional bonds in human civilisation. Cave paintings representing healers date back fourteen thousand years. Before the secularisation of medicine brought by the Hippocratic school in the fifth century BCE, no clear boundaries existed between medicine, magic, and religion. The healer was often an extension of the priest, and seeking medical help meant placing yourself in the hands of someone who understood mysteries you could not fathom.

For most of medical history, this relationship was profoundly asymmetrical. The physician possessed knowledge that patients could not access or evaluate. Compliance was expected. The doctor decided, the patient accepted. This paternalistic model persisted well into the twentieth century. As one historical analysis noted, physicians were viewed as dominant or superior to patients due to the inherent power dynamic of controlling health, treatment, and access to knowledge. The physician conveyed only the information necessary to convince the patient of the proposed treatment course.

The shift came gradually but represented a fundamental reconception of the relationship. By the late twentieth century, the patient transformed from passive receiver of decisions into an agent with well-defined rights and broad capacity for autonomous decision-making. The doctor transformed from priestly father figure into technical adviser whose knowledge was offered but whose decisions were no longer taken for granted. Informed consent emerged as a legal and ethical requirement. Shared decision-making became the professional ideal.

Trust remained central throughout these transformations. Research consistently shows that trust, along with empathy, communication, and listening, characterises a productive doctor-patient relationship. For patients, a consistent relationship with their doctors has been shown to facilitate treatment adherence and improved health outcomes. The relationship itself is therapeutic.

But this trust has been eroding for decades. Public confidence in medicine peaked in the mid-1960s. A 2023 Gallup Poll found that only about one in three Americans expressed “great or quite a lot” of confidence in the medical system. Trust in doctors, though higher at roughly two in three Americans, remains below pre-pandemic levels. As one analysis observed, physicians' employers, pharmaceutical companies, and insurance companies have entered what was once a private relationship. The generic substitution of “healthcare provider” for “physician” and “client” for “patient” reflects a growing impersonality. Medicine has become commercialised, the encounter increasingly transactional.

Into this already complicated landscape arrives artificial intelligence, promising to further reshape what it means to receive medical care.

The Equity Reckoning

The introduction of AI into healthcare carries profound implications for equity, and not all of them are positive. The technology has the potential either to reduce or to amplify existing disparities, depending entirely on how it is developed and deployed.

A 2019 study sent shockwaves through the medical community when it revealed that a clinical algorithm used by many hospitals to decide which patients needed care showed significant racial bias. Black patients had to be deemed much sicker than white patients to be recommended for the same care. The algorithm had been trained on past healthcare spending data, which reflected a history in which Black patients had less to spend on their health compared to white patients. The algorithm learned to perpetuate that inequity.

The problem persists and may even be worsening as AI becomes more prevalent. A systematic review on AI-driven racial disparities in healthcare found a significant association between AI utilisation and the exacerbation of racial disparities, especially in minority populations including Black and Hispanic patients. Sources identified included biased training data, algorithm design choices, unfair deployment practices, and historic systemic inequities embedded in the healthcare system.

A Cedars-Sinai study found patterns of racial bias in treatment recommendations generated by leading AI platforms for psychiatric patients. Large language models, when presented with hypothetical clinical cases, often proposed different treatments for patients when African American identity was stated or implied than for patients whose race was not indicated. Specific disparities included LLMs omitting medication recommendations for ADHD cases when race was explicitly stated and suggesting guardianship for depression cases with explicit racial characteristics.

The sources of bias are multiple and often embedded in the foundational data that AI systems learn from. Public health AI typically suffers from historic bias, where prior injustices in access to care or discriminatory health policy become embedded within training datasets. Representation bias emerges when samples from urban, wealthy, or well-connected groups lead to the systematic exclusion of samples from rural, indigenous, or disenfranchised groups. Measurement bias occurs when health endpoints are approximated with proxy variables that differ between socioeconomic or cultural environments.

Research warns that minoritised communities, whose trust in health systems has been eroded by historical inequities, ongoing biases, and in some cases outright malevolence, are likely to approach AI with heightened scepticism. These communities have seen how systemic disparities can be perpetuated by the very tools meant to serve them.

Addressing these issues requires comprehensive bias detection tools and mitigation strategies, coupled with active supervision by physicians who understand the limitations of the systems they use. Mitigating algorithmic bias must occur across all stages of an algorithm's lifecycle, including authentic engagement with patients and communities during all phases, explicitly identifying healthcare algorithmic fairness issues and trade-offs, and ensuring accountability for equity and fairness in outcomes.

The Validation Gap

For all the impressive performance statistics emerging from research studies, a troubling pattern emerges upon closer examination of how AI diagnostic tools actually reach the market and enter clinical practice.

A cross-sectional study of 903 FDA-approved AI devices found that at the time of regulatory approval, clinical performance studies were reported for approximately half of the analysed devices. One quarter explicitly stated that no such studies had been conducted. Less than one third of clinical evaluations provided sex-specific data, and only one fourth addressed age-related subgroups. Perhaps most concerning: 97 per cent of all devices were cleared via the 510(k) pathway, which does not require independent clinical data demonstrating performance or safety. Devices are cleared based on their similarity to previously approved devices, creating a chain of approvals that may never have been anchored in rigorous clinical validation.

A JAMA Network Open study examining the generalisability of FDA-approved AI-enabled medical devices for clinical use warned that evidence about clinical generalisability is lacking. The number of AI-enabled tools cleared continues to rise, but the robust real-world validation that would inspire confidence often does not exist.

This matters because AI systems that perform brilliantly in controlled research settings may falter in the messy reality of clinical practice. The UVA Health researchers who found ChatGPT Plus achieving 92 per cent accuracy cautioned that the system “likely would fare less well in real life, where many other aspects of clinical reasoning come into play.” Determining downstream effects of diagnoses and treatment decisions involves complexities that current AI systems do not reliably navigate. A correct diagnosis is only the beginning; knowing what to do with it requires judgment that algorithms do not yet possess.

Studies have also found that most physicians treated AI tools like a search function, much as they would Google or UpToDate, rather than leveraging optimised prompting strategies that might improve performance. This suggests that even when AI tools are available, the human element of how they are used introduces significant variability that research settings often fail to capture.

What Machines Cannot Do

The argument for AI in diagnosis often centres on consistency and processing power. Algorithms do not forget, do not tire, do not bring personal problems to work. They can compare a patient's presentation against millions of cases instantly. They do not have fifteen-minute appointment slots that force rushed assessments.

But medicine is not merely pattern recognition. Eric Topol, Executive Vice-President of Scripps Research and author of Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again, has argued that AI development in healthcare could lead to a dramatic shift in the culture and practice of medicine. Yet he cautions that AI on its own will not fix the current challenges of what he terms “shallow medicine.” In his assessment, the field is “long on AI promise but very short on real-world, clinical proof of effectiveness.”

Topol envisions AI restoring the essential human element of medical practice by enabling machine support of tasks better suited for automation, thereby freeing doctors, nurses, and other healthcare professionals to focus on providing real care for patients. This is a fundamentally different vision from replacing physicians with algorithms. It imagines a symbiosis where each contributor does what it does best: the machine handles pattern recognition and data processing while the human provides judgment, empathy, and presence.

The obstacles to achieving this vision are substantial. Topol identifies medical community resistance to change, reimbursement issues, regulatory challenges, the need for greater transparency, the need for compelling evidence, engendering trust among clinicians and the public, and implementation challenges as chief barriers to progress. These are not merely technical problems but cultural and institutional ones.

Doctors must also contend with the downsides of AI adoption. Models can generate incorrect or misleading results, the phenomenon known as AI hallucinations or confabulations. AI models can produce results that reflect human bias encoded in training data. A diagnosis is not merely a label; it is a communication that affects how a person understands their body, their future, their mortality. Getting that communication wrong carries consequences that extend far beyond clinical metrics.

The Regulatory Response

Governments and regulatory bodies around the world are scrambling to keep pace with the technology, developing frameworks that balance innovation with safety.

In the United States, the FDA published guidance on “Transparency for Machine Learning-Enabled Medical Devices” in June 2024, followed by final guidance on predetermined change control plans for AI-enabled device software in December 2024. Draft guidance on lifecycle management for AI-enabled device software followed in January 2025. The FDA's Digital Health Advisory Committee held its inaugural meeting in November 2024 to discuss how the agency should adapt its regulatory approach for generative AI-enabled devices, which present novel challenges because they can produce outputs that even their creators cannot fully predict.

In the United Kingdom, the MHRA AI Airlock launched in May 2024 and expanded with a second cohort in 2025. This regulatory sandbox allows developers to test their AI as a Medical Device in supervised, real-world NHS environments. A new National Commission was announced to accelerate safe access to AI in healthcare by advising on a new regulatory framework to be published in 2026. The Commission brings together experts from technology companies including Google and Microsoft alongside clinicians, researchers, and patient advocates.

The NHS Fit For The Future: 10 Year Health Plan for England, published in July 2025, identified data, artificial intelligence, genomics, wearables, and robotics as five transformative technologies that are strategic priorities. A new framework procurement process will be introduced in 2026-2027 to allow NHS organisations to adopt innovative technologies including ambient AI.

The National Institute for Health and Care Excellence has conditionally recommended AI tools such as TechCare Alert and BoneView for NHS use in identifying fractures on X-rays, provided they are used alongside clinician review. This last phrase is crucial: alongside clinician review. The regulatory consensus, for now, maintains human oversight as a non-negotiable requirement.

The Nobel Prize and Its Implications

In October 2024, Demis Hassabis and John Jumper of Google DeepMind were co-awarded the Nobel Prize in Chemistry for their work on AlphaFold, alongside David Baker for his work on computational protein design. This recognition elevated AI in life sciences to the highest level of scientific honour, signalling that the technology has passed from speculative promise to demonstrated achievement.

AlphaFold has predicted over 200 million protein structures, nearly all catalogued proteins known to science. As of November 2025, it is being used by over 3 million researchers from over 190 countries, tackling problems including antimicrobial resistance, crop resilience, and heart disease. AlphaFold 3, announced in May 2024 and made publicly available in February 2025, can predict the structures of protein complexes with DNA, RNA, post-translational modifications, and selected ligands and ions. Google DeepMind reports a 50 per cent improvement in prediction accuracy compared to existing methods, effectively doubling what was previously possible.

The implications for drug discovery are substantial. Isomorphic Labs, the Google DeepMind spinout, raised 600 million dollars in March 2025 and is preparing to initiate clinical trials for AI-developed oncology drugs. Scientists at the company are collaborating with Eli Lilly and Novartis to discover antibodies and new treatments that inhibit disease-related targets. According to GlobalData's Drugs database, there are currently more than 3,000 drugs developed or repurposed using AI, with most in early stages of development.

Meanwhile, Med Gemini, Google DeepMind's medical AI platform, achieved 91.1 per cent accuracy on diagnostic tasks, outperforming prior models by 4.6 per cent. The system leverages deep learning to analyse medical images including X-rays and MRIs, helping in early detection of diseases including cancer, heart conditions, and neurological disorders.

In India, Google's bioacoustic AI model is enabling development of tools that can screen tuberculosis through cough sounds, with potential to screen 35 million people. AI is also working to close maternal health gaps by making ultrasounds accessible to midwives. These applications suggest that AI could expand access to diagnostic capabilities in resource-limited settings, potentially democratising healthcare in ways that human expertise alone could never achieve.

Hospitals Using AI Today

The integration is already happening, hospital by hospital, department by department. This is not a future scenario but present reality.

Pilot programmes at several Level I trauma centres report that AI-flagged X-rays get read 20 to 30 minutes faster on average than normal work-list order. In acute care, those minutes can be critical; in stroke treatment, every minute of delay costs brain cells. A multi-centre study in the UK identified that AI-assisted mammography had the potential to cut radiologists' workload by almost half without sacrificing diagnostic quality. Another trial in Canada demonstrated faster triage of suspected strokes when CT scans were pre-screened by AI, resulting in up to 30 minutes of saved treatment time.

A 2024 survey of physician sentiments revealed that at least two-thirds view AI as beneficial to their practice, with overall use cases increasing by nearly 70 per cent, particularly in medical documentation. The administrative burden of medicine is substantial: physicians spend more time on paperwork than on patients. AI that handles documentation potentially frees physicians for direct patient interaction, the very thing that drew many of them to medicine.

Thanks to the AI Diagnostic Fund in England, 50 per cent of hospital trusts are now deploying AI to help diagnose conditions including lung cancer. Research indicates that hospitals using AI-supported diagnostics have seen a 42 per cent reduction in diagnostic errors. If these figures hold at scale, the impact on patient outcomes could be transformative. Recall those 795,000 Americans harmed by diagnostic errors each year. Even modest improvements in diagnostic accuracy would translate to thousands of lives saved or changed.

The Question of the Self

Beyond the clinical metrics lies a deeper question about human experience. When you are ill, vulnerable, frightened, what do you need? What does healing require?

The paternalistic model of medicine assumed patients needed authority: someone who knew what to do and would do it. The patient-centred model assumed patients needed partnership: someone who would share information, discuss options, respect autonomy. Both models assumed a human on the other side of the relationship, someone capable of understanding what it means to suffer.

A 2025 randomised factorial experiment found that functionally, people trusted the diagnosis of human physicians more than medical AI or human-involved AI. But at the relational and emotional levels, there was no significant difference between human-AI and human-human interactions. This finding suggests something complicated about what patients actually experience versus what they believe they prefer. We may say we want a human, but we may respond to something else.

The psychiatric setting reveals particular tensions. The Frontiers in Psychology study found that the situation in psychiatry differed strongly from cardiology, orthopaedics, and dermatology, especially in the “human doctor with an AI system” condition. Mental health involves not just pattern recognition but the experience of being heard, validated, understood. Whether AI can participate meaningfully in that process remains deeply uncertain. A diagnosis of depression is not like a diagnosis of a fracture; it touches the core of selfhood.

Research on trust in AI-assisted health systems emphasises that trust is built differently in each relationship: between patients and providers, providers and technology, and institutions and their stakeholders. Trust is bidirectional; people must trust AI to perform reliably, while AI relies on the quality of human input. This circularity complicates simple narratives of replacement or enhancement.

Reimagining the Consultation

What might a transformed healthcare encounter look like in practice?

One possibility is the augmented physician: a doctor who arrives at your appointment having already reviewed an AI analysis of your symptoms, test results, and medical history. The AI has flagged potential diagnoses ranked by probability. The AI has identified questions the doctor should ask to differentiate between possibilities. The AI has checked for drug interactions, noted relevant recent research, compared your presentation to anonymised similar cases.

The doctor then spends your appointment actually talking to you. Understanding your concerns. Explaining options. Answering questions. Making eye contact. The administrative and analytical burden has shifted to the machine; the human connection remains with the human.

This vision aligns with Topol's argument in Deep Medicine. The title itself is instructive: the promise is not that AI will make healthcare mechanical but that it might make healthcare human again. Fifteen-minute appointments driven by documentation requirements represent a form of dehumanisation that preceded AI. If algorithms absorb the documentation burden, perhaps doctors can rediscover the relationship that drew many of them to medicine in the first place.

But this optimistic scenario requires deliberate design choices. If AI primarily serves cost-cutting, if healthcare administrators use diagnostic algorithms to reduce physician staffing, if the efficiency gains flow to shareholders rather than patient care, the technology will deepen rather than heal medicine's wounds.

The Coming Transformation

The trajectory is set, though the destination remains uncertain.

The NHS Healthcare AI Solutions agreement, expected to be worth 180 million pounds, is forecast to open for bids in summer 2025 and go live in 2026. The UCLA-led PRISM Trial, the first major randomised trial of AI in breast cancer screening in the United States, is underway with 16 million dollars in funding. Clinical trials for AI-designed drugs from Isomorphic Labs are imminent.

Meanwhile, the fundamental questions persist. Will patients trust algorithms with their lives? The evidence suggests: sometimes, depending on context, depending on how the technology is presented, depending on who is doing the presenting. Trust in providers and the healthcare system is positively associated with expectations of AI benefit. Those who already trust their doctors are more likely to trust AI recommendations filtered through those doctors.

Will the doctor-patient relationship survive this transformation? The relationship has survived extraordinary changes before: the rise of specialisation, the introduction of evidence-based medicine, the intrusion of insurance companies and electronic health records. Each change reshaped but did not extinguish the fundamental bond between someone who is suffering and someone who can help.

The machines are faster. They may well be more accurate, at least for certain diagnostic tasks. They do not tire, do not forget, do not have personal problems. But they also do not care, not in any meaningful sense. They do not sit with you in your fear. They do not hold your hand while delivering difficult news. They do not remember that your mother died of the same disease and understand why this diagnosis terrifies you.

Perhaps the answer is not trust in machines or trust in humans but trust in a system where each contributes what it does best. The algorithm analyses the scan. The doctor explains what the analysis means for your life. The algorithm flags the drug interaction. The doctor discusses whether the benefit outweighs the risk. The algorithm never forgets a detail. The doctor never forgets you are a person.

This synthesis requires more than technological development. It requires deliberate choices about healthcare systems, medical education, regulatory frameworks, and reimbursement structures. It requires confronting the biases encoded in training data and the inequities they can perpetuate. It requires maintaining human oversight even when algorithms outperform humans on specific metrics. It requires remembering that a diagnosis is not just an output but a communication that changes someone's understanding of their own existence.

The algorithm can see you now. Whether you will trust it, and whether that trust is warranted, depends on decisions being made in research laboratories, regulatory agencies, hospital boardrooms, and government ministries around the world. The doctor-patient relationship that has defined healthcare for centuries is being renegotiated. The outcome will shape medicine for the centuries to come.

---

References and Sources

1. Newman-Toker, D.E. et al. (2023). “Burden of serious harms from diagnostic error in the USA.” BMJ Quality & Safety. Johns Hopkins Armstrong Institute Center for Diagnostic Excellence. https://pubmed.ncbi.nlm.nih.gov/37460118/

2. Takita, H. et al. (2025). “A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians.” npj Digital Medicine, 8(175). https://www.nature.com/articles/s41746-025-01543-z

3. Parsons, A.S. et al. (2024). “Does AI Improve Doctors' Diagnoses?” Randomised controlled trial, UVA Health. JAMA Network Open. https://newsroom.uvahealth.com/2024/11/13/does-ai-improve-doctors-diagnoses-study-finds-out/

4. FDA. (2024-2025). Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices database. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices

5. IDx-DR De Novo Classification (DEN180001). (2018). FDA regulatory submission for autonomous AI diabetic retinopathy detection. https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?id=DEN180001

6. Kim, J. et al. (2024). “Human-AI interaction in skin cancer diagnosis: a systematic review and meta-analysis.” npj Digital Medicine. Stanford Medicine. https://www.nature.com/articles/s41746-024-01031-w

7. Lång, K. et al. (2025). “Screening performance and characteristics of breast cancer detected in the Mammography Screening with Artificial Intelligence trial (MASAI).” The Lancet Digital Health, 7(3), e175-e183. https://www.thelancet.com/journals/landig/article/PIIS2589-7500(24)00267-X/fulltext

8. Riedl, R., Hogeterp, S.A. & Reuter, M. (2024). “Do patients prefer a human doctor, artificial intelligence, or a blend, and is this preference dependent on medical discipline?” Frontiers in Psychology, 15. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2024.1422177/full

9. Zondag, A.G.M. et al. (2024). “The Effect of Artificial Intelligence on Patient-Physician Trust: Cross-Sectional Vignette Study.” Journal of Medical Internet Research, 26, e50853. https://www.jmir.org/2024/1/e50853

10. Nong, P. & Ji, M. (2025). “Expectations of healthcare AI and the role of trust: understanding patient views on how AI will impact cost, access, and patient-provider relationships.” Journal of the American Medical Informatics Association, 32(5), 795-799. https://academic.oup.com/jamia/article/32/5/795/8046745

11. Obermeyer, Z. et al. (2019). “Dissecting racial bias in an algorithm used to manage the health of populations.” Science, 366(6464), 447-453. https://www.science.org/doi/10.1126/science.aax2342

12. Aboujaoude, E. et al. (2025). “Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models.” npj Digital Medicine. Cedars-Sinai. https://www.cedars-sinai.org/newsroom/cedars-sinai-study-shows-racial-bias-in-ai-generated-treatment-regimens-for-psychiatric-patients/

13. Windecker, D. et al. (2025). “Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use.” JAMA Network Open, 8(4), e258052. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2833324

14. Topol, E.J. (2019). Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books. https://drerictopol.com/portfolio/deep-medicine/

15. NHS England. (2024-2025). NHS AI Lab investments and implementation reports. https://www.gov.uk/government/news/health-secretary-announces-250-million-investment-in-artificial-intelligence

16. GOV.UK. (2025). “New Commission to help accelerate NHS use of AI.” https://www.gov.uk/government/news/new-commission-to-help-accelerate-nhs-use-of-ai

17. Department of Health and Social Care. (2025). “Fit For The Future: 10 Year Health Plan for England.” https://www.gov.uk/government/publications/10-year-health-plan-for-england-fit-for-the-future

18. Nobel Prize Committee. (2024). “The Nobel Prize in Chemistry 2024” — Hassabis, Jumper (AlphaFold) and Baker. https://www.nobelprize.org/prizes/chemistry/2024/press-release/

19. Truog, R.D. (2012). “Patients and Doctors — The Evolution of a Relationship.” New England Journal of Medicine, 366(7), 581-585. https://www.nejm.org/doi/full/10.1056/nejmp1110848

20. Gallup. (2023). “Confidence in U.S. Institutions Down; Average at New Low.” https://news.gallup.com/poll/394283/confidence-institutions-down-average-new-low.aspx

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

In December 2025, something remarkable happened in the fractious world of artificial intelligence. Anthropic, OpenAI, Google, Microsoft, and a constellation of other technology giants announced they were joining forces under the Linux Foundation to create the Agentic AI Foundation. The initiative would consolidate three competing protocols into a neutral consortium: Anthropic's Model Context Protocol, Block's Goose agent framework, and OpenAI's AGENTS.md convention. After years of proprietary warfare, the industry appeared to be converging on shared infrastructure for the age of autonomous software agents.

The timing could not have been more significant. According to the Linux Foundation announcement, MCP server downloads had grown from roughly 100,000 in November 2024 to over 8 million by April 2025. The ecosystem now boasts over 5,800 MCP servers and 300 MCP clients, with major deployments at Block, Bloomberg, Amazon, and hundreds of Fortune 500 companies. RedMonk analysts described MCP's adoption curve as reminiscent of Docker's rapid market saturation, the fastest standard uptake the firm had ever observed.

Yet beneath this apparent unity lies a troubling question that few in the industry seem willing to confront directly. What happens when you standardise the plumbing before you fully understand what will flow through it? What if the orchestration patterns being cemented into protocol specifications today prove fundamentally misaligned with the reasoning capabilities that will emerge tomorrow?

The history of technology is littered with standards that seemed essential at the time but later constrained innovation in ways their creators never anticipated. The OSI networking model, Ada programming language, and countless other well-intentioned standardisation efforts demonstrate how premature consensus can lock entire ecosystems into architectural choices that later prove suboptimal. As one researcher noted in a University of Michigan analysis, standardisation increases technological efficiency but can also prolong existing technologies to an excessive degree by inhibiting investments in novel developments.

The stakes in the agentic AI standardisation race are considerably higher than previous technology transitions. We are not merely deciding how software components communicate. We are potentially determining the architectural assumptions that will govern how artificial intelligence decomposes problems, executes autonomous tasks, and integrates with human workflows for decades to come.

The Competitive Logic Driving Convergence

To understand why the industry is rushing toward standardisation, one must first appreciate the economic pressures that have made fragmented agentic infrastructure increasingly untenable. The current landscape resembles the early days of mobile computing, when every manufacturer implemented its own charging connector and data protocol. Developers building agentic applications face a bewildering array of frameworks, each with its own conventions for tool integration, memory management, and inter-agent communication.

The numbers tell a compelling story. Gartner reported a staggering 1,445% surge in multi-agent system inquiries from the first quarter of 2024 to the second quarter of 2025. Industry analysts project the agentic AI market will surge from 7.8 billion dollars today to over 52 billion dollars by 2030. Gartner further predicts that 40% of enterprise applications will embed AI agents by the end of 2026, up from less than 5% in 2025.

This explosive growth has created intense pressure for interoperability. When Google announced its Agent2Agent protocol in April 2025, it launched with support from more than 50 technology partners including Atlassian, Box, Cohere, Intuit, Langchain, MongoDB, PayPal, Salesforce, SAP, ServiceNow, and Workday. The protocol was designed to enable agents built by different vendors to discover each other, negotiate capabilities, and coordinate actions across enterprise environments.

The competitive dynamics are straightforward. If the Agentic AI Foundation's standards become dominant, companies that previously held APIs hostage will be pressured to interoperate. Google and Microsoft could find it increasingly necessary to support MCP and AGENTS.md generically, lest customers demand cross-platform agents. The open ecosystem effectively buys customers choice, giving a competitive advantage to adherence.

Yet this race toward consensus obscures a fundamental tension. The Model Context Protocol was designed primarily to solve the problem of connecting AI systems to external tools and data sources. As Anthropic's original announcement explained, even the most sophisticated models are constrained by their isolation from data, trapped behind information silos and legacy systems. MCP provides a universal interface for reading files, executing functions, and handling contextual prompts. Think of it as USB-C for AI applications.

But USB-C was standardised after decades of experience with peripheral connectivity. The fundamental patterns for how humans interact with external devices were well understood. The same cannot be said for agentic AI. The field is evolving so rapidly that the orchestration patterns appropriate for today's language models may prove entirely inadequate for the reasoning systems emerging over the next several years.

When Reasoning Changes Everything

The reasoning model revolution of 2024 and 2025 has fundamentally altered how software engineering tasks can be decomposed and executed. OpenAI's o3, Google's Gemini 3 with Deep Think mode, and DeepSeek's R1 represent a qualitative shift in capability that extends far beyond incremental improvements in benchmark scores.

The pace of advancement has been staggering. In November 2025, Google introduced Gemini 3, positioning it as its most capable system to date, deployed from day one across Search, the Gemini app, AI Studio, Vertex AI, and the Gemini CLI. Gemini 3 Pro scores 1501 Elo on LMArena, achieving top leaderboard position, alongside 91.9% on GPQA Diamond and 76.2% on SWE-bench Verified for real-world software engineering tasks. The Deep Think mode pushes scientific reasoning benchmarks into the low to mid nineties, placing Gemini 3 at the front of late 2025 capabilities. By December 2025, Google was processing over one trillion tokens per day through its API.

Consider the broader transformation in software development. OpenAI reports that GPT-5 scores 74.9% on SWE-bench Verified compared to 69.1% for o3. On Aider polyglot, an evaluation of code editing, GPT-5 achieves 88%, representing a one-third reduction in error rate compared to o3. DeepSeek's R1 demonstrated that reasoning abilities can be incentivised through pure reinforcement learning, obviating the need for human-labelled reasoning trajectories. The company's research shows that such training facilitates the emergent development of advanced reasoning patterns including self-verification, reflection, and dynamic strategy adaptation. DeepSeek is now preparing to launch a fully autonomous AI agent by late 2025, signalling a shift from chatbots to practical, real-world agentic AI.

These capabilities demand fundamentally different decomposition strategies than the tool-calling patterns embedded in current protocols. A reasoning model that can plan multi-step tasks, execute on them, and continue to reason about results to update its plans represents a different computational paradigm than a model that simply calls predefined functions in response to user prompts.

The 2025 DORA Report captures this transformation in stark terms. AI adoption is near-universal, with 90% of survey respondents reporting they use AI at work. More than 80% believe it has increased their productivity. Yet AI adoption continues to have a negative relationship with software delivery stability. The researchers estimate that between two people who share the same traits, environment, and processes, the person with higher AI adoption will report higher levels of individual effectiveness but also higher levels of software delivery instability.

This productivity-stability paradox suggests that current development practices are struggling to accommodate the new capabilities. The DORA team found that AI coding assistants dramatically boost individual output, with 21% more tasks completed and 98% more pull requests merged, but organisational delivery metrics remain flat. Speed without stability, as the researchers concluded, is accelerated chaos.

The Lock-In Mechanism

The danger of premature standardisation lies not in the protocols themselves but in the architectural assumptions they embed. When developers build applications around specific orchestration patterns, those patterns become load-bearing infrastructure that cannot easily be replaced.

Microsoft's October 2025 decision to merge AutoGen with Semantic Kernel into a unified Microsoft Agent Framework illustrates both the problem and the attempted solution. The company recognised that framework fragmentation was creating confusion among developers, with multiple competing options each requiring different approaches to agent construction. General availability is set for the first quarter of 2026, with production service level agreements, multi-language support, and deep Azure integration.

Yet this consolidation also demonstrates how quickly architectural choices become entrenched. As one analysis noted, current agent frameworks are fragmented and lack enterprise features like observability, compliance, and durability. The push toward standardisation aims to address these gaps, but in doing so it may cement assumptions about how agents should be structured that prove limiting when new capabilities emerge.

The historical parallel to the OSI versus Internet protocols debate is instructive. Several central actors within OSI and Internet standardisation suggested that OSI's failure stemmed from being installed-base-hostile. The OSI protocols were not closely enough related to the already installed base of communication systems. The installed base is irreversible in the sense that radical, abrupt change of the kind implicitly assumed by OSI developers is highly unlikely.

The same irreversibility threatens agentic AI. Once thousands of enterprise applications embed MCP clients and servers, once development teams organise their workflows around specific orchestration patterns, the switching costs become prohibitive. Even if superior approaches emerge, the installed base may prevent their adoption.

Four major protocols have already emerged to handle agent communication: Model Context Protocol, Agent Communication Protocol, Agent-to-Agent Protocol, and Agent Network Protocol. Google's A2A Protocol alone has backing from over 50 companies including Microsoft and Salesforce. Yet as of September 2025, A2A development has slowed significantly, and most of the AI agent ecosystem has consolidated around MCP. Google Cloud still supports A2A for some enterprise customers, but the company has started adding MCP compatibility to its AI services. This represents a tacit acknowledgment that the developer community has chosen.

The Junior Developer Crisis

The technical standardisation debate unfolds against the backdrop of a more immediate crisis in the software development workforce. The rapid adoption of AI coding assistants has fundamentally disrupted the traditional career ladder for software engineers, with consequences that may prove more damaging than any technical limitation.

According to data from the U.S. Bureau of Labor Statistics, overall programmer employment fell a dramatic 27.5% between 2023 and 2025. A Stanford Digital Economy Study found that by July 2025, employment for software developers aged 22-25 had declined nearly 20% from its peak in late 2022. Across major U.S. technology companies, graduate hiring has dropped more than 50% compared to pre-2020 levels. In the UK, junior developer openings are down by nearly one-third since 2022.

The economics driving this shift are brutally simple. As one senior software engineer quoted by CIO observed, companies are asking why they should hire a junior developer for 90,000 dollars when GitHub Copilot costs 10 dollars. Many of the tasks once assigned to junior developers, including generating boilerplate code, writing unit tests, and maintaining APIs, are now reliably managed by AI assistants.

Industry analyst Vernon Keenan describes a quiet erosion of entry-level positions that will lead to a decline in foundational roles, a loss of mentorship opportunities, and barriers to skill development. Anthropic CEO Dario Amodei has warned that entry-level jobs are squarely in the crosshairs of automation. Salesforce CEO Marc Benioff announced the company would stop hiring new software engineers in 2025, citing AI-driven productivity gains.

The 2025 Stack Overflow Developer Survey captures the resulting tension. While 84% of developers now use or plan to use AI tools, trust has declined sharply. Only 33% of developers trust the accuracy of AI tools, while 46% actively distrust it. A mere 3% report highly trusting the output. The biggest frustration, cited by 66% of developers, is dealing with AI solutions that are almost right but not quite.

This trust deficit reflects a deeper problem. Experienced developers understand the limitations of AI-generated code but have the expertise to verify and correct it. Junior developers lack this foundation. There is sentiment that AI has made junior developers less competent, with some losing foundational skills that make for successful entry-level employees. Without proper mentorship, junior developers risk over-relying on AI.

The long-term implications are stark. The biggest challenge will be training the next generation of software architects. With fewer junior developer jobs, there will not be a natural apprenticeship to more senior roles. We risk creating a generation of developers who can prompt AI systems but cannot understand or debug the code those systems produce.

Architectural Decisions Migrate to Prompt Design

As reasoning models assume greater responsibility for code generation and system design, the locus of architectural decision-making is shifting in ways that current organisational structures are poorly equipped to handle. Prompt engineering is evolving from a novelty skill into a core architectural discipline.

The way we communicate with AI has shifted from simple trial-and-error prompts to something much more strategic, what researchers describe as prompt design as a discipline. If 2024 was about understanding the grammar of prompts, 2025 is about learning to design blueprints. Just as software architects do not just write code but design systems, prompt architects do not just write clever sentences. They shape conversations into repeatable frameworks that unlock intelligence, creativity, and precision.

The adoption statistics reflect this shift. According to the 2025 AI-Enablement Benchmark Report, the design and architecture phase of the software development lifecycle has an AI adoption rate of 52%. Teams using AI tools for design and architecture have seen a 28% increase in design iteration speed.

Yet this concentration of architectural power in prompt design creates new risks. Context engineering, as one CIO analysis describes it, is an architectural shift in how AI systems are built. Early generative AI was stateless, handling isolated interactions where prompt engineering was sufficient. Autonomous agents are fundamentally different. They persist across multiple interactions, make sequential decisions, and operate with varying levels of human oversight.

This shift demands collaboration between data engineering, enterprise architecture, security, and those who understand processes and strategy. A strong data foundation, not just prompt design, determines how well an agent performs. Agents need engineering, not just prompts.

The danger lies in concentrating too much decision-making authority in the hands of those who understand prompt patterns but lack deep domain expertise. Software architecture is not about finding a single correct answer. It is about navigating competing constraints, making tradeoffs, and defending reasoning. AI models can help reason through tradeoffs, generate architectural decision records, or compare tools, but only if prompted by someone who understands the domain deeply enough to ask the right questions.

The governance implications are significant. According to IAPP research, 50% of AI governance professionals are typically assigned to ethics, compliance, privacy, or legal teams. Yet traditional AI governance practices may not suffice with agentic systems. Governing agentic systems requires addressing their autonomy and dynamic behaviour in ways that current organisational structures are not designed to handle.

Fragmentation Across Model Families

The proliferation of reasoning models with different capabilities and cost profiles is creating a new form of fragmentation that threatens to balkanise development practices. Different teams within the same organisation may adopt different model families based on their specific requirements, leading to incompatible workflows and siloed expertise.

The ARC Prize Foundation's extensive testing of reasoning systems reached a striking conclusion: there is no clear winner. Different models excel at different tasks, and the optimal choice depends heavily on specific requirements around accuracy, cost, and latency. OpenAI's o3-medium and o3-high offer the highest accuracy while sacrificing cost and time. Google's Gemini 3 Flash, released in December 2025, delivers frontier-class performance at less than a quarter of the cost of Gemini 3 Pro, with pricing of 0.50 dollars per million input tokens compared to significantly higher rates for comparable models. DeepSeek offers an aggressive pricing structure with input costs as low as 0.07 dollars per million tokens.

For enterprises focused on return on investment, these tradeoffs matter enormously. The 2025 State of AI report notes that trade-offs remain, with long contexts raising latency and cost. Because different providers trust or cherry-pick different benchmarks, it has become more difficult to evaluate agents' performance. Choosing the right agent for a particular task remains a challenge.

This complexity is driving teams toward specialisation around particular model families. Some organisations standardise on OpenAI's ecosystem for its integration with popular development tools. Others prefer Google's offerings for their multimodal capabilities and long context windows of up to 1,048,576 tokens. Still others adopt DeepSeek's open models for cost control or air-gapped deployments.

The result is a fragmentation of development practices that cuts across traditional organisational boundaries. A team building customer-facing agents may use entirely different tools and patterns than a team building internal automation. Knowledge transfer becomes difficult. Best practices diverge. The organisational learning that should flow from widespread AI adoption becomes trapped in silos.

The 2025 DORA Report identifies platform engineering as a crucial foundation for unlocking AI value, with 90% of organisations having adopted at least one platform. There is a direct correlation between high-quality internal platforms and an organisation's ability to unlock the value of AI. Yet building such platforms requires making architectural choices that may lock organisations into specific model families and orchestration patterns.

The Technical Debt Acceleration

The rapid adoption of AI coding assistants has created what may be the fastest accumulation of technical debt in the history of software development. Code that works today may prove impossible to maintain tomorrow, creating hidden liabilities that will compound over time.

Forrester predicts that by 2025, more than 50% of technology decision-makers will face moderate to severe technical debt, with that number expected to hit 75% by 2026. Technical debt costs over 2.41 trillion dollars annually in the United States alone. The State of Software Delivery 2025 report by Harness found that the majority of developers spend more time debugging AI-generated code and more time resolving security vulnerabilities than before AI adoption.

The mechanisms driving this debt accumulation are distinctive. According to one analysis, there are three main vectors that generate AI technical debt: model versioning chaos, code generation bloat, and organisation fragmentation. These vectors, coupled with the speed of AI code generation, interact to cause exponential growth.

Code churn, defined as code that is added and then quickly modified or deleted, is projected to hit nearly 7% by 2025. This represents a red flag for instability and rework. As API evangelist Kin Lane observed, he has not seen so much technical debt being created in such a short period during his 35-year career in technology.

The security implications are equally concerning. A report from Ox Security titled Army of Juniors: The AI Code Security Crisis found that AI-generated code is highly functional but systematically lacking in architectural judgment. The Google 2024 DORA report found a trade-off between gains and losses with AI, where a 25% increase in AI usage quickens code reviews and benefits documentation but results in a 7.2% decrease in delivery stability.

The widening gap between organisations with clean codebases and those burdened by legacy systems creates additional stratification. Generative AI dramatically widens the gap in velocity between low-debt coding and high-debt coding. Companies with relatively young, high-quality codebases benefit the most from generative AI tools, while companies with gnarly, legacy codebases struggle to adopt them. The penalty for having a high-debt codebase is now larger than ever.

Research Structures for Anticipating Second-Order Effects

Navigating the transition to reasoning-capable autonomous systems requires organisational and research structures that most institutions currently lack. The rapid pace of change demands new approaches to technology assessment, workforce development, and institutional coordination.

The World Economic Forum estimates that 40% of today's workers will need major skill updates by 2030, and in information technology that number is likely even higher. Yet the traditional mechanisms for workforce development are poorly suited to a technology that evolves faster than educational curricula can adapt.

Several research priorities emerge from this analysis. First, longitudinal studies tracking the career trajectories of software developers across the AI transition would provide crucial data for workforce planning. The Stanford Digital Economy Study demonstrates the value of such research, but more granular analysis is needed to understand which skills remain valuable, which become obsolete, and how career paths are being restructured.

Second, technical research into the interaction between standardisation and innovation in agentic systems could inform policy decisions about when and how to pursue consensus. The historical literature on standards competition provides useful frameworks, but the unique characteristics of AI systems, including their rapid capability growth and opaque decision-making, may require new analytical approaches.

Third, organisational research examining how different governance structures affect AI adoption outcomes could help enterprises design more effective oversight mechanisms. The DORA team's finding that AI amplifies existing organisational capabilities, making strong teams stronger and struggling teams worse, suggests that the organisational context matters as much as the technology itself.

Fourth, security research focused specifically on the interaction between AI code generation and vulnerability introduction could help establish appropriate safeguards. The current pattern of generating functional but architecturally flawed code suggests fundamental limitations in how models understand system-level concerns.

Finally, educational research into how programming pedagogy should adapt to AI assistance could prevent the worst outcomes of skill atrophy. If junior developers are to learn effectively in an environment where AI handles routine tasks, new teaching approaches will be needed that focus on the higher-order skills that remain uniquely human.

Building Resilient Development Practices

The confluence of standardisation pressures, reasoning model capabilities, workforce disruption, and technical debt accumulation creates a landscape that demands new approaches to software development practice. Organisations that thrive will be those that build resilience into their development processes rather than optimising purely for speed.

Several principles emerge from this analysis. First, maintain architectural optionality. Avoid deep dependencies on specific orchestration patterns that may prove limiting as capabilities evolve. Design systems with clear abstraction boundaries that allow components to be replaced as better approaches emerge.

Second, invest in human capability alongside AI tooling. The organisations that will navigate this transition successfully are those that continue developing deep technical expertise in their workforce, not those that assume AI will substitute for human understanding.

Third, measure what matters. The DORA framework's addition of rework rate as a fifth core metric reflects the recognition that traditional velocity measures miss crucial dimensions of software quality. Organisations should develop measurement systems that capture the long-term health of their codebases and development practices.

Fourth, build bridges across model families. Rather than standardising on a single AI ecosystem, develop the institutional capability to work effectively across multiple model families. This requires investment in training, tooling, and organisational learning that most enterprises have not yet made.

Fifth, participate in standards development. The architectural choices being made in protocol specifications today will shape the development landscape for years to come. Organisations with strong opinions about how agentic systems should work have an opportunity to influence those specifications before they become locked in.

The transition to reasoning-capable autonomous systems represents both an enormous opportunity and a significant risk. The opportunity lies in the productivity gains that well-deployed AI can provide. The risk lies in the second-order effects that poorly managed deployment can create. The difference between these outcomes will be determined not by the capabilities of the AI systems themselves but by the organisational wisdom with which they are deployed.

The Protocols That Will Shape Tomorrow

The agentic AI standardisation race presents a familiar tension in new form. The industry needs common infrastructure to enable interoperability and reduce fragmentation. Yet premature consensus risks locking in architectural assumptions that may prove fundamentally limiting.

The Model Context Protocol's rapid adoption demonstrates both the hunger for standardisation and the danger of premature lock-in. MCP achieved in one year what many standards take a decade to accomplish: genuine industry-wide adoption and governance transition to a neutral foundation. Yet the protocol was designed for a particular model of AI capability, one where agents primarily call tools and retrieve context. The reasoning models now emerging may demand entirely different decomposition strategies.

Meta's notable absence from the Agentic AI Foundation hints at alternative futures. Almost every major agentic player from Google to AWS to Microsoft has joined, but Meta has not signed on and published reports indicate it will not be joining soon. The company is reportedly shifting toward a proprietary strategy centred on a new revenue-generating model. Whether this represents a mistake or a prescient bet on different architectural approaches remains to be seen.

The historical pattern suggests that the standards which endure are those designed with sufficient flexibility to accommodate unforeseen developments. The Internet protocols succeeded where OSI failed in part because they were more tolerant of variation and evolution. The question for agentic AI is whether current standardisation efforts embed similar flexibility or whether they will constrain the systems of tomorrow to the architectural assumptions of today.

For developers, enterprises, and policymakers navigating this landscape, the imperative is to engage critically with standardisation rather than accepting it passively. The architectural choices being made now will shape the capabilities and limitations of agentic systems for years to come. Those who understand both the opportunities and the risks of premature consensus will be better positioned to influence the outcome.

The reasoning revolution is just beginning. The protocols and patterns that emerge from this moment will determine whether artificial intelligence amplifies human capability or merely accelerates the accumulation of technical debt and workforce disruption. The standards race matters, but the wisdom with which we run it matters more.

---

References and Sources

• Linux Foundation (2025). “Linux Foundation Announces the Formation of the Agentic AI Foundation.” https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation

• Anthropic (2024). “Introducing the Model Context Protocol.” https://www.anthropic.com/news/model-context-protocol

• Anthropic (2025). “Donating the Model Context Protocol and Establishing the Agentic AI Foundation.” https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation

• Pento (2025). “A Year of MCP: From Internal Experiment to Industry Standard.” https://www.pento.ai/blog/a-year-of-mcp-2025-review

• University of Michigan (n.d.). “Why Standardization Efforts Fail.” https://quod.lib.umich.edu/cgi/t/text/idx/j/jep/3336451.0014.103/--why-standardization-efforts-fail

• InfoQ (n.d.). “Standards are Great, but Standardisation is a Really Bad Idea.” https://www.infoq.com/presentations/downey-standards-great-standardization-bad/

• Google DORA (2025). “State of AI-assisted Software Development 2025.” https://dora.dev/research/2025/dora-report/

• OpenAI (2025). “Introducing GPT-5 for Developers.” https://openai.com/index/introducing-gpt-5-for-developers/

• Google (2025). “Gemini 3: News and Announcements.” https://blog.google/products/gemini/gemini-3-collection/

• Google (2025). “Introducing Gemini 3 Flash: Benchmarks, Global Availability.” https://blog.google/products/gemini/gemini-3-flash/

• DeepSeek (2025). “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” https://arxiv.org/html/2501.12948v1

• Google Developers Blog (2025). “Announcing the Agent2Agent Protocol (A2A).” https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/

• Stack Overflow (2025). “2025 Stack Overflow Developer Survey.” https://survey.stackoverflow.co/2025/

• CIO (2025). “Demand for Junior Developers Softens as AI Takes Over.” https://www.cio.com/article/4062024/demand-for-junior-developers-softens-as-ai-takes-over.html

• Stack Overflow Blog (2025). “AI vs Gen Z: How AI Has Changed the Career Pathway for Junior Developers.” https://stackoverflow.blog/2025/12/26/ai-vs-gen-z/

• IEEE Spectrum (2025). “AI Shifts Expectations for Entry Level Jobs.” https://spectrum.ieee.org/ai-effect-entry-level-jobs

• Understanding AI (2025). “New Evidence Strongly Suggests AI Is Killing Jobs for Young Programmers.” https://www.understandingai.org/p/new-evidence-strongly-suggest-ai

• CIO (2025). “Context Engineering: Improving AI by Moving Beyond the Prompt.” https://www.cio.com/article/4080592/context-engineering-improving-ai-by-moving-beyond-the-prompt.html

• IAPP (2025). “AI Governance Profession Report 2025.” https://iapp.org/resources/article/ai-governance-profession-report

• Machine Learning Mastery (2025). “7 Agentic AI Trends to Watch in 2026.” https://machinelearningmastery.com/7-agentic-ai-trends-to-watch-in-2026/

• ARC Prize (2025). “We Tested Every Major AI Reasoning System. There Is No Clear Winner.” https://arcprize.org/blog/which-ai-reasoning-model-is-best

• InfoQ (2025). “AI-Generated Code Creates New Wave of Technical Debt, Report Finds.” https://www.infoq.com/news/2025/11/ai-code-technical-debt/

• LeadDev (2025). “How AI Generated Code Compounds Technical Debt.” https://leaddev.com/technical-direction/how-ai-generated-code-accelerates-technical-debt

• IT Revolution (2025). “AI's Mirror Effect: How the 2025 DORA Report Reveals Your Organization's True Capabilities.” https://itrevolution.com/articles/ais-mirror-effect-how-the-2025-dora-report-reveals-your-organizations-true-capabilities/

• RedMonk (2025). “DORA 2025: Measuring Software Delivery After AI.” https://redmonk.com/rstephens/2025/12/18/dora2025/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...