Human in the Loop

Human in the Loop

When Anthropic released Claude's “computer use” feature in October 2024, the AI could suddenly navigate entire computers by interpreting screen content and simulating keyboard and mouse input. OpenAI followed in January 2025 with Operator, powered by its Computer-Using Agent model. Google deployed Gemini 2.0 with Astra for low-latency multimodal perception. The age of agentic AI, systems capable of autonomous decision-making without constant human oversight, had arrived. So had the regulatory panic.

Across government offices in Brussels, London, Washington, and beyond, policymakers face an uncomfortable truth: their legal frameworks were built for software that follows instructions, not AI that makes its own choices. When an autonomous agent can book flights, execute financial transactions, manage customer relationships, or even write and deploy code independently, who bears responsibility when things go catastrophically wrong? The answer, frustratingly, depends on which jurisdiction you ask, and whether you ask today or six months from now.

This regulatory fragmentation isn't just an academic concern. It's reshaping how technology companies build products, where they deploy services, and whether smaller competitors can afford to play the game at all. The stakes extend far beyond compliance costs: they touch questions of liability, data sovereignty, competitive dynamics, and whether innovation happens in regulatory sandboxes or grey market jurisdictions with looser rules.

The European Vanguard

The European Union's AI Act, which entered into force on 1 August 2024, represents the world's first comprehensive attempt to regulate artificial intelligence through binding legislation. Its risk-based approach categorises AI systems from prohibited to minimal risk, with agentic AI likely falling into the “high-risk” category depending on deployment context. The Act's phased implementation means some requirements already apply: prohibited AI practices and AI literacy obligations took effect on 2 February 2025, whilst full compliance obligations arrive on 2 August 2026.

For agentic AI, the EU framework presents unprecedented challenges. Article 9's risk management requirements mandate documented processes extending beyond one-time validation to include ongoing testing, real-time monitoring, and clearly defined response strategies. Because agentic systems engage in multi-step decision-making and operate autonomously, they require continuous safeguards, escalation protocols, and oversight mechanisms throughout their lifecycle. Traditional “deploy and monitor” approaches simply don't suffice when an AI agent might encounter novel situations requiring judgement calls.

Documentation requirements under Article 11 prove equally demanding. Whilst the provision requires detailed technical documentation for high-risk AI systems, agentic AI demands comprehensive transparency beyond traditional practices like model cards or AI Bills of Materials. Organisations must document not just initial model architecture but also decision-making processes, reasoning chains, tool usage patterns, and behavioural evolution over time. This depth proves essential for auditing and compliance, especially when systems behave dynamically or interact with third-party APIs in ways developers cannot fully anticipate.

Article 12's event recording requirements create similar challenges at scale. Agentic systems make independent decisions and generate logs across diverse environments, from cloud infrastructure to edge devices. Structured logs including timestamps, reasoning chains, and tool usage become critical for post-incident analysis, compliance verification, and accountability attribution. The European Commission's proposed amendments even introduce “AIH Codes” covering underlying AI technologies, explicitly including agentic AI as a distinct category requiring regulatory attention.

The penalties for non-compliance carry genuine teeth: up to €35 million or 7% of global annual turnover for prohibited practices, €15 million or 3% for violations involving high-risk AI systems, and €7.5 million or 1% for providing false information. These aren't hypothetical fines but real financial exposures that force board-level attention.

Yet implementation questions abound. The European Data Protection Board has highlighted that “black-box AI” cannot justify failure to comply with transparency requirements, particularly challenging for agentic AI where actions may derive from intermediate steps or model outputs not directly supervised or even understood by human operators. How organisations demonstrate compliance whilst maintaining competitive advantages in proprietary algorithms remains an open question, one the European Commission's July 2025 voluntary Code of Practice for general-purpose AI developers attempts to address through standardised disclosure templates.

The British Experiment

Across the Channel, the United Kingdom pursues a markedly different strategy. As of 2025, no dedicated AI law exists in force. Instead, the UK maintains a flexible, principles-based approach through existing legal frameworks applied by sectoral regulators. Responsibility for AI oversight falls to bodies like the Information Commissioner's Office (ICO) for data protection, the Financial Conduct Authority (FCA) for financial services, and the Competition and Markets Authority (CMA) for market competition issues.

This sectoral model offers advantages in agility and domain expertise. The ICO published detailed guidance on AI and data protection, operates a Regulatory Sandbox for AI projects, and plans a statutory code of practice for AI and automated decision-making. The FCA integrates AI governance into existing risk management frameworks, expecting Consumer Duty compliance, operational resilience measures, and Senior Manager accountability. The CMA addresses AI as a competition issue, warning that powerful incumbents might restrict market entry through foundation model control whilst introducing new merger thresholds specifically targeting technology sector acquisitions.

Coordination happens through the Digital Regulation Cooperation Forum, a voluntary alliance of regulators with digital economy remits working together on overlapping issues. The DRCF launched an AI and Digital Hub pilot to support innovators facing complex regulatory questions spanning multiple regulators' remits. This collaborative approach aims to prevent regulatory arbitrage whilst maintaining sector-specific expertise.

Yet critics argue this fragmented structure creates uncertainty. Without central legislation, organisations face interpretative challenges across different regulatory bodies. The proposed Artificial Intelligence (Regulation) Bill, reintroduced in the House of Lords in March 2025, would establish a new “AI Authority” and codify five AI principles into binding duties, requiring companies to appoint dedicated “AI Officers.” The UK government has indicated a comprehensive AI Bill could arrive in 2026, drawing lessons from the EU's experience.

For agentic AI specifically, the UK's 2025 AI Opportunities Action Plan and earlier White Paper identified “autonomy risks” as requiring further regulatory attention. Sectoral regulators like the FCA, ICO, and CMA are expected to issue guidance capturing agentic behaviours within their domains. This creates a patchwork where financial services AI agents face different requirements than healthcare or employment screening agents, even when using similar underlying technologies.

The American Reversal

The United States regulatory landscape underwent dramatic shifts within weeks of the Trump administration's January 2025 inauguration. The Biden administration's Executive Order 14110 on “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” issued in October 2023, represented a comprehensive federal approach addressing transparency, bias mitigation, explainability, and privacy. It required federal agencies to appoint Chief AI Officers, mandated safety testing for advanced AI systems, and established guidelines for AI use in critical infrastructure.

Executive Order 14179, “Removing Barriers to American Leadership in Artificial Intelligence,” issued in the Trump administration's first days, reversed course entirely. The new order eliminated many Biden-era requirements, arguing they would “stifle American innovation and saddle companies with burdensome new regulatory requirements.” The AI Diffusion Rule, issued on 15 January 2025 by the Bureau of Industry and Security, faced particular criticism before its May 2025 implementation deadline. Industry giants including Nvidia and Microsoft argued the rules would result in billions in lost sales, hinder innovation, and ultimately benefit Chinese competitors.

The rescinded AI Diffusion Framework proposed a three-tiered global licensing system for advanced chips and AI model weights. Tier I included the United States and 18 allied countries exempt from licensing. Tier II covered most other parts of the world, licensed through a data centre Validated End-User programme with presumption of approval. Tier III, encompassing China, Russia, and North Korea, faced presumption of denial. The framework created ECCN 4E091 to control AI model weights, previously uncontrolled items, and sought to curtail China's access to advanced chips and computing power through third countries.

This reversal reflects deeper tensions in American AI policy: balancing national security concerns against industry competitiveness, reconciling federal authority with state-level initiatives, and navigating geopolitical competition whilst maintaining technological leadership. The rescission doesn't eliminate AI regulation entirely but shifts it toward voluntary frameworks, industry self-governance, and state-level requirements.

State governments stepped into the breach. Over 1,000 state AI bills were introduced in 2025, creating compliance headaches for businesses operating nationally. California continues as a regulatory frontrunner, with comprehensive requirements spanning employment discrimination protections, transparency mandates, consumer privacy safeguards, and safety measures. Large enterprises and public companies face the most extensive obligations: organisations like OpenAI, Google, Anthropic, and Meta must provide detailed disclosures about training data, implement watermarking and detection capabilities, and report safety incidents to regulatory authorities.

New York City's Automated Employment Decision Tools Law, enforced since 5 July 2023, exemplifies state-level specificity. The law prohibits using automated employment decision tools, including AI, to assess candidates for hiring or promotion in New York City unless an independent auditor completes a bias audit beforehand and candidates who are New York City residents receive notice. Bias audits must include calculations of selection and scoring rates plus impact ratios across sex categories, race and ethnicity categories, and intersectional categories.

The Equal Employment Opportunity Commission issued technical guidance in May 2023 on measuring adverse impact when AI tools are used for employment selection. Critically, employers bear liability for outside vendors who design or administer algorithmic decision-making tools on their behalf and cannot rely on vendor assessments of disparate impact. This forces companies deploying agentic AI in hiring contexts to conduct thorough vendor due diligence, review assessment reports and historical selection rates, and implement bias-mitigating techniques when audits reveal disparate impacts.

The GDPR Collision

The General Data Protection Regulation, designed in an era before truly autonomous AI, creates particular challenges for agentic systems. Article 22 grants individuals the right not to be subject to decisions based solely on automated processing that produces legal or significant effects. This provision, interpreted by Data Protection Authorities as a prohibition rather than something requiring active invocation, directly impacts agentic AI deployment.

The challenge lies in the “solely” qualifier. European Data Protection Board guidance emphasises that human involvement must be meaningful, not merely supplying data the system uses or rubber-stamping automated decisions. For human review to satisfy Article 22, involvement should come after the automated decision and relate to the actual outcome. If AI merely produces information someone uses alongside other information to make a decision, Article 22 shouldn't apply. But when does an agentic system's recommendation become the decision itself?

Agentic AI challenges the traditional data controller and processor dichotomy underlying GDPR. When an AI acts autonomously, who determines the purpose and means of processing? How does one attribute legal responsibility for decisions taken without direct human intervention? These questions lack clear answers, forcing businesses to carefully consider their governance structures and documentation practices.

Data Protection Impact Assessments become not just best practice but legal requirements for agentic AI. Given the novel risks associated with systems acting independently on behalf of users, conducting thorough DPIAs proves both necessary for compliance and valuable for understanding system behaviour. These assessments should identify specific risks created by the agent's autonomy and evaluate how the AI might repurpose data in unexpected ways as it learns and evolves.

Maintaining comprehensive documentation proves critical. For agentic AI systems, this includes detailed data flow maps showing how personal data moves through the system, records of processing activities specific to the AI agent, transparency mechanisms explaining decision-making processes, and evidence of meaningful human oversight where required. The EDPB's recent opinions note that consent becomes particularly challenging for agentic AI because processing scope may evolve over time as the AI learns, users may not reasonably anticipate all potential uses of their data, and traditional consent mechanisms may not effectively cover autonomous agent activities.

The Liability Gap

Perhaps no question proves more vexing than liability attribution when agentic AI causes harm. Traditional legal frameworks struggle with systems that don't simply execute predefined instructions but make decisions based on patterns learned from vast datasets. Their autonomous action creates a liability gap current frameworks cannot adequately address.

The laws of agency and vicarious liability require there first to be a human agent or employee primarily responsible for harm before their employer or another principal can be held responsible. With truly autonomous AI agents, there may be no human “employee” acting at the moment of harm: the AI acts on its own algorithmic decision-making. Courts and commentators have consistently noted that without a human “agent,” vicarious liability fails by definition.

The July 2024 California district court decision in the Workday case offers a potential path forward. The court allowed a case against HR and finance platform Workday to proceed, stating that an employer's use of Workday's AI-powered HR screening algorithm may create direct liability for both the employer and Workday under agency liability theory. By deeming Workday an “agent,” the court created potential for direct liability for AI vendors, not just employers deploying the systems.

This decision's implications for agentic AI prove significant. First, it recognises that employers delegating traditional functions to AI tools cannot escape responsibility through vendor relationships. Second, it acknowledges AI tools playing active roles in decisions rather than merely implementing employer-defined criteria. Third, by establishing vendor liability potential, it creates incentives for AI developers to design systems with greater care for foreseeable risks.

Yet no specific federal law addresses AI liability, let alone agentic AI specifically. American courts apply existing doctrines like tort law, product liability, and negligence. If an autonomous system causes damage, plaintiffs might argue developers or manufacturers were negligent in designing or deploying the system. Negligence requires proof that the developer or user failed to act as a reasonable person would, limiting liability compared to strict liability regimes.

The United Kingdom's approach to autonomous vehicles offers an intriguing model potentially applicable to agentic AI. The UK framework establishes that liability should follow control: as self-driving technology reduces human driver influence over a vehicle, law shifts legal responsibility from users toward developers and manufacturers. This introduces autonomy not just as a technical measure but as a legal determinant of liability. AI agents could be similarly classified, using autonomy levels to define when liability shifts from users to developers.

Despite different regulatory philosophies across jurisdictions, no nation has fully resolved how to align AI's autonomy with existing liability doctrines. The theoretical discussion of granting legal personhood to AI hangs as an intriguing yet unresolved idea. The most promising frameworks recognise that agentic AI requires nuanced approaches acknowledging distributed nature of AI development and deployment whilst ensuring clear accountability for harm.

Export Controls and Geopolitical Fragmentation

AI regulation extends beyond consumer protection and liability into national security domains through export controls. The rescinded Biden administration AI Diffusion Framework attempted to create a secure global ecosystem for AI data centres whilst curtailing China's access to advanced chips and computing power. Its rescission reflects broader tensions between technological leadership and alliance management, between protecting strategic advantages and maintaining market access.

The United States and close allies dominate the advanced AI chip supply chain. Given technological complexity of design and manufacturing processes, China remains reliant on these suppliers for years to come. According to recent congressional testimony by Commerce Secretary Howard Lutnick, Huawei will produce only 200,000 AI chips in 2025, a marginal output compared to American production. Yet according to Stanford University benchmarks, American and Chinese model capabilities are fairly evenly matched, with Chinese AI labs functioning as fast followers at worst.

This paradox illustrates export control limitations: China continues producing competitive state-of-the-art models and dominating AI-based applications like robotics and autonomous vehicles despite chip controls implemented over recent years. The controls made chip development a matter of national pride and triggered waves of investment into domestic AI chip ecosystems within China. Whether the United States ever regains market share even if chip controls are reversed remains unclear.

The Trump administration argued the Biden-era framework would hinder American innovation and leadership in the AI sector. Industry concerns centred on billions in lost sales, reduced global market share, and acceleration of foreign AI hardware ecosystem growth. The framework sought to turn AI chips into diplomatic tools, extracting geopolitical and technological concessions through export leverage. Its rescission signals prioritising economic competitiveness over strategic containment, at least in the near term.

For companies developing agentic AI, this creates uncertainty. Will future administrations reimpose controls? How should global supply chains be structured to withstand regulatory whiplash? Companies face impossible planning horizons when fundamental policy frameworks reverse every four years.

Cross-Border Chaos

The divergent approaches across jurisdictions create opportunities for regulatory arbitrage and challenges for compliance. When different jurisdictions develop their own AI policies, laws, and regulations, businesses face increased compliance costs from navigating complex regulatory landscapes, market access barriers limiting operational geography, and innovation constraints slowing cross-border collaboration. These challenges prove particularly acute for small and medium-sized enterprises lacking resources to manage complex, jurisdiction-specific requirements.

The transnational nature of AI, where algorithms, data, and systems operate across borders, makes it difficult for individual nations to control cross-border flows and technology transfer. Incompatible national rules create compliance challenges whilst enabling regulatory arbitrage that undermines global governance efforts. For companies, divergent frameworks serve as invitations to shift operations to more permissive environments. For countries pursuing stricter AI rules, this raises stakes of maintaining current approaches against competitive pressure.

Without harmonisation, regulatory arbitrage risks worsen, with firms relocating operations to jurisdictions with lenient regulations to circumvent stricter compliance obligations, potentially undermining global AI oversight effectiveness. The European Banking Institute advocates robust and centralised governance to address risks and regulatory fragmentation, particularly in cross-border financial technology trade, whilst the United States has adopted more decentralised approaches raising standardisation and harmonisation concerns.

Yet experts expect strategic fragmentation rather than global convergence. AI regulation proves too entangled with geopolitical competition, economic sovereignty, and industrial policy. Jurisdictions will likely assert regulatory independence where it matters most, such as compute infrastructure or training data, whilst cooperating selectively in areas where alignment yields real economic benefits.

Proposed solutions emphasise multilateral processes making AI rules among jurisdictions interoperable and comparable to minimise regulatory arbitrage risks. Knowledge sharing could be prioritised through standards development, AI sandboxes, large public AI research projects, and regulator-to-regulator exchanges. Regulatory sandboxes foster adaptability by allowing companies to test AI solutions in controlled environments with regulatory oversight, enabling experimentation without immediate compliance failure risks.

Restructuring for Compliance

Organisations deploying agentic AI must fundamentally restructure product development, governance, and transparency practices to comply with evolving requirements. Over 68% of multinational corporations are restructuring AI workflows to meet evolving regulatory standards on explainability and bias mitigation. With 59% of AI systems now under internal audit programmes, governments push stricter compliance benchmarks whilst global enforcement actions related to unethical AI use have increased over 33%.

The Chief AI Officer role has nearly tripled in the past five years according to LinkedIn data, with positions expanding significantly in finance, manufacturing, and retail. Companies including JPMorgan Chase, Walmart, and Siemens employ AI executives to manage automation and predictive analytics efforts. The CAIO serves as operational and strategic leader for AI initiatives, ensuring technologies are properly selected, executed, and monitored to align with visions and goals driving company success.

Key responsibilities span strategic leadership, AI governance and risk management, ethical AI management, regulatory compliance, and cultural transformation. CAIOs must establish robust governance frameworks ensuring safe, ethical, and compliant AI development across organisations. They create clear guidelines, accountability measures, and control mechanisms addressing data handling, model validation, and usage. Four major risks grouped under the acronym FATE drive this work: Fairness (AI models can perpetuate biases), Accountability (responsibility when models fail), Transparency (opacity of algorithms makes explaining conclusions difficult), and Ethics (AI can face ethical dilemmas).

The regulatory framework's emphasis on meaningful human involvement in automated decision-making may require restructuring operational processes previously fully automated. For agentic AI, this means implementing escalation protocols, defining autonomy boundaries, creating human oversight mechanisms, and documenting decision-making processes. Organisations must decide whether to centralise AI governance under single executives or distribute responsibilities across existing roles. Research indicates centralised AI governance provides better risk management and policy consistency, whilst distributed models may offer more agility but can create accountability gaps.

Product development lifecycle changes prove equally significant. The NIST AI Risk Management Framework, whilst voluntary, offers resources to organisations designing, developing, deploying, or using AI systems to help manage risks and promote trustworthy and responsible development. The framework's MAP, MEASURE, and MANAGE functions can be applied in AI system-specific contexts and at specific stages of the AI lifecycle, whilst GOVERN applies to all stages of organisations' AI risk management processes and procedures.

Lifecycle risk management should be embedded into workflows, not added as compliance afterthoughts. Best practices include establishing risk checkpoints at every phase requiring documentation and approval, using structured risk assessment tools like NIST AI RMF or OECD AI Principles, and ensuring data scientists, legal, product, and ethics teams share ownership of risk. AI governance should be built into every process in the AI development and maintenance journey, with AI Impact Assessments and threat modelling conducted at least annually on existing systems and prior to deploying any new AI function.

Transparency Requirements and AI Bills of Materials

Transparency in AI systems has become a cornerstone of proposed regulatory frameworks across jurisdictions. Upcoming mandates require companies to disclose how AI models make decisions, datasets used for training, and potential system limitations. The European Commission's July 2025 voluntary Code of Practice for general-purpose AI developers includes a chapter on transparency obligations and provides template forms for AI developers to share information with downstream providers and regulatory authorities.

The AI Bill of Materials has emerged as a critical transparency tool. Just as Software Bills of Materials and Hardware Bills of Materials brought clarity to software and hardware supply chains, AIBOMs aim to provide transparency into how AI models are built, trained, and deployed. An AIBOM is a structured inventory documenting all components within an AI system, including datasets used to train or fine-tune models, models themselves (open-source or proprietary), software dependencies supporting AI pipelines, and deployment environments where models run.

Additional elements include digital signatures for the model and AIBOM ensuring authenticity and integrity, model developer names, parent model information, base model details, model architecture and architecture family, hardware and software used to run or train models, required software downloads, and datasets with their names, versions, sources, and licensing information.

AIBOMs help organisations demonstrate adherence to evolving frameworks like the EU AI Act, NIST AI RMF, and Department of Defense AI security directives. Whilst software supply chains face vulnerabilities through third-party libraries, AI systems introduce new risks via external datasets, model weights, and training pipelines. An AIBOM plays crucial roles in AI supply chain security by tracking third-party models, documenting pre-trained models, their sources, and any modifications.

The OWASP AI Bill of Materials project leads AI security and transparency efforts, organised into ten strategic workstreams focused on critical aspects of AI transparency and security. The Linux Foundation's work on AI-BOM with SPDX 3.0 expands on SBOM concepts to include documentation of algorithms, data collection methods, frameworks and libraries, licensing information, and standard compliance. Industry leaders advance standardisation of AI transparency through efforts like the AIBOM extension to the CycloneDX specification, a widely adopted SBOM format.

For agentic AI specifically, AIBOMs must extend beyond static component listings to capture dynamic behaviours, tool integrations, API dependencies, and decision-making patterns. Traditional documentation practices prove insufficient when systems evolve through learning and interaction. This requires new approaches to transparency balancing competitive concerns about proprietary methods with regulatory requirements for explainability and accountability.

The Path Through Uncertainty

The regulatory landscape for agentic AI remains in flux, characterised by divergent approaches, evolving frameworks, and fundamental questions without clear answers. Organisations deploying these systems face unprecedented compliance challenges spanning multiple jurisdictions, regulatory bodies, and legal domains. The costs of getting it wrong, whether through massive fines, legal liability, or reputational damage, prove substantial.

Yet the absence of settled frameworks also creates opportunities. Companies engaging proactively with regulators, participating in sandboxes, contributing to standards development, and implementing robust governance structures position themselves advantageously as requirements crystallise. Those treating compliance as pure cost rather than strategic investment risk falling behind competitors who embed responsible AI practices into their organisational DNA.

The next several years will prove decisive. Will jurisdictions converge toward interoperable frameworks or fragment further into incompatible regimes? Will liability doctrines evolve to address autonomous systems adequately or will courts struggle with ill-fitting precedents? Will transparency requirements stifle innovation or foster trust enabling broader adoption? The answers depend not just on regulatory choices but on how industry, civil society, and technologists engage with the challenge.

What seems certain is that agentic AI will not remain in regulatory limbo indefinitely. The systems are too powerful, the stakes too high, and the public attention too focused for governments to maintain hands-off approaches. The question is whether the resulting frameworks enable responsible innovation or create bureaucratic moats favouring incumbents over challengers. For organisations building the future of autonomous AI, understanding this evolving landscape isn't optional. It's existential.

References & Sources

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The software engineer had prepared for weeks. They'd studied algorithms, practised coding problems, reviewed the company's tech stack. What they couldn't prepare for was the fluorescent lighting that felt like needles in their skull, the unexpected background chatter from an open office that fragmented their thoughts, and the overwhelming cognitive demand of writing code whilst simultaneously explaining their reasoning to three strangers who were judging their every word. Twenty minutes into the pair programming interview, they froze. Not because they didn't know the answer. Because their autistic brain, overwhelmed by sensory chaos and social performance demands, simply shut down.

They didn't get the job. The feedback cited “communication issues” and “inability to think under pressure.” What the feedback didn't mention: their GitHub profile showed five years of elegant, well-documented code. Their portfolio included contributions to major open-source projects. In their actual work environment, with noise-cancelling headphones and asynchronous communication, they excelled. But the interview measured none of that.

When Amazon scrapped its AI recruiting tool in 2018 after discovering it systematically discriminated against women, the tech industry collectively shuddered. The algorithm, trained on a decade of predominantly male hiring decisions, had learned to penalise CVs containing the word “women's” and downgrade graduates from all-women's colleges. Engineers attempted repairs, but couldn't guarantee neutrality. The project died, and with it, a cautionary tale was born.

Since then, companies have fled AI-assisted hiring in droves. Following New York City's 2021 requirement that employers audit automated hiring tools for bias, every single audit revealed discrimination against women, people of colour, LGBTQ+ candidates, neurodivergent individuals, and non-native English speakers. The message seemed clear: algorithms cannot be trusted with something as consequential as hiring.

Yet in their rush to abandon biased machines, tech companies have doubled down on interview methods carrying their own insidious prejudices. Pair programming sessions, whiteboard challenges, and multi-round panel interviews have become the gold standard, positioned as objective measures of technical skill. For neurodivergent candidates (those with autism, ADHD, dyslexia, anxiety disorders, and other neurological differences), these “human-centred” alternatives often prove more discriminatory than any algorithm.

The irony is stark. An industry built on innovation, celebrated for disrupting ossified systems, has responded to AI bias by retreating into traditional interview practices that systematically exclude some of its most talented potential contributors. In fleeing one form of discrimination, tech has embraced another, older prejudice hiding in plain sight.

The Numbers Tell an Uncomfortable Story

Researchers estimate there are 67 million neurodivergent Americans, representing 15% to 20% of the global population. Yet unemployment rates for this group reach 30% to 40%, three times higher than for people with physical disabilities and eight times higher than for non-disabled individuals. For college-educated autistic adults, the figure climbs to a staggering 85%, despite many possessing precisely the skills tech companies desperately seek.

A 2024 survey revealed that 76% of neurodivergent job seekers feel traditional recruitment methods (timed assessments, panel interviews, on-the-spot coding challenges) put them at a disadvantage. Half of neurodivergent adults report experiencing discrimination from hiring managers or recruiters once they disclosed their neurodiversity, with 31% seeing their applications abandoned entirely post-disclosure. A Zurich Insurance UK report found even more troubling statistics: one in five neurodivergent adults reported being openly laughed at during job searches, and one in six had job offers rescinded after disclosing their neurodivergence.

Within the tech sector specifically, nearly one in four neurodivergent workers recalled instances of discrimination. A 2024 BIMA study surveying 3,333 technology workers uncovered significant discrimination related to neurodivergence, alongside gender, ethnicity, and age. More than a third of respondents in a Prospect union survey reported discrimination related to their neurodivergent condition, whilst four in five faced direct workplace challenges because of it. A third said their workplace experience negatively impacted their mental wellbeing; a fifth said it harmed their ability to perform well.

These aren't abstract statistics. They represent brilliant minds lost to an industry that claims to value talent above all else, yet cannot recognise it when packaged differently.

The Evolution of Tech Interviews

To understand how we arrived here, consider the evolution of tech hiring. In the 1990s and early 2000s, companies like Microsoft and Google became infamous for brain teasers and logic puzzles. “Why are manhole covers round?” and “How would you move Mount Fuji?” were considered legitimate interview questions, supposedly revealing problem-solving abilities and creativity.

Research eventually exposed these questions as poor predictors of actual job performance, often measuring little beyond a candidate's familiarity with such puzzles. The industry moved on, embracing what seemed like better alternatives: technical assessments that directly tested coding ability.

Whiteboard interviews became ubiquitous. Candidates stood before panels of engineers, solving complex algorithms on whiteboards whilst explaining their thought processes. Pair programming sessions followed, where candidates collaborated with current employees on real problems, demonstrating both technical skills and cultural fit.

These methods appeared superior to arbitrary brain teasers. They tested actual job-relevant skills in realistic scenarios. Many companies proclaimed them more objective, more fair, more predictive of success.

For neurotypical candidates, perhaps they are. For neurodivergent individuals, they can be nightmarish gauntlets that have little relation to actual job performance and everything to do with performing competence under specific, high-pressure conditions.

What Happens When Your Brain Works Differently

Consider the standard pair programming interview from a neurodivergent perspective. You're placed in an unfamiliar environment, under observation by strangers whose judgement will determine your livelihood. You're expected to think aloud, explaining your reasoning in real-time whilst writing code, fielding questions, reading social cues, and managing the interpersonal dynamics of collaboration, all simultaneously.

For someone with ADHD, this scenario can severely impair short-term memory, memory recall, and problem-solving speed. The brain simply doesn't have the bandwidth to handle spontaneous problem-solving whilst maintaining the social performance expected. As one industry observer noted, coding interviews with whiteboarding or code pairing become “excruciating” when your brain lacks the speed for instant detailed memory recall.

Research confirms that adults with specific learning disabilities who have low sensory thresholds tend to notice too many stimuli, including irrelevant ones. This sensory overload interferes with their ability to select relevant information for executive functions to process. When cognitively overloaded, sensory overload intensifies, creating a vicious cycle.

For autistic candidates, the challenges multiply. Studies show neurodivergent employees experience disproportionate stress in team interactions compared to neurotypical colleagues. Whilst pair programming may be less stressful than large meetings, it still demands interpersonal communication skills that can be emotionally draining and cognitively expensive for autistic individuals. Research found that autistic people felt they had to hide their traits to gain employment, and many worried about discrimination if they disclosed during hiring.

During whiteboard challenges, candidates often stand before groups ranging from two to sixteen interviewers, facing a wall whilst solving complex algorithms. For autistic candidates, this setup makes concentration nearly impossible, even on simple questions. It's an experience they'll never encounter in the actual job, yet it determines whether they're hired.

The physical environment itself can be overwhelming. Bright fluorescent lights, background noise from open offices, unexpected sounds, strong smells from nearby kitchens or perfumes, all create sensory assaults that neurotypical interviewers barely notice. For sensory-sensitive candidates, these distractions aren't minor annoyances; they're cognitive impediments that dramatically impair performance.

Timed assessments compound these difficulties. Pressure intensifies anxiety, which for neurodivergent candidates often reaches paralysing levels. Research shows autistic job applicants experience significantly more interview anxiety than neurotypical candidates and worry intensely about how potential employers perceive them. This anxiety can cause candidates to freeze, unable to think on the spot regardless of their knowledge or experience.

The phenomenon called “masking” adds another layer of exhaustion. Eighty-five percent of neurodivergent tech workers in the Prospect survey reported masking their condition at work, consciously suppressing natural behaviours to appear neurotypical. This requires enormous cognitive effort, leading to mental and physical fatigue, increased anxiety and depression, and reduced job satisfaction. During interviews, when cognitive resources are already stretched thin by technical challenges and performance pressure, the additional burden of masking can be devastating.

Štěpán Hladík, a technical sourcer at Pure Storage who has disclosed his neurodivergence, feels “truly privileged to have been around colleagues who are willing to understand or actively try to learn about biases.” But he notes previous experiences at other companies left him feeling misunderstood and frustrated. Many neurodivergent workers don't disclose their conditions, citing “fear of discrimination as well as ignorance of colleagues” and concerns about career progression. In one study, 53% said potential outcomes of disclosure weren't worth the risk, 27% cited stigma concerns, and 24% feared career impact.

When candidates attempt to request accommodations, the consequences can be severe. Industry reports suggest that when candidates gently ask about available disability accommodations during interviews, they're dropped “about 60% to 70% of the time” as companies “freak out and wash their hands of it to keep things simple.” One tech worker shared observations about Meta: “I've seen a lot of neurodivergent people really struggle” there, having heard “you can be immediately rejected by asking for accommodations.” They noted that “the tech industry has always been rife with discrimination.”

The Research on Interview Bias

Whilst tech companies abandoned AI tools due to proven bias, research reveals traditional interview methods carry substantial biases of their own. A 2024 study published by IntechOpen found that interviewing processes are “inherently susceptible to human bias, which can adversely affect the fairness and validity of outcomes, leading to discrimination and a lack of diversity.”

Interviewers make decisions based on extraneous elements like age, gender, ethnicity, physical attributes, and other personal traits instead of professional qualifications. They succumb to confirmation bias and the halo effect, distorting assessments and creating less diverse workforces. These biases stem from subconscious prejudices, stereotypes, and personal preferences, including entrenched notions about gender, race, and age.

Unstructured interviews, despite receiving the highest ratings for perceived effectiveness from hiring managers, are among the worst predictors of actual job performance. They're far less reliable than general mental ability tests, aptitude tests, or personality tests. Yet they remain popular because they feel right to interviewers, confirming their belief that they can intuitively identify talent.

Traditional interviews test whether candidates can perform interviews, not whether they can perform jobs. For neurodivergent candidates, this distinction is critical. The skills required to excel in pair programming interviews (simultaneous multitasking, real-time verbal processing, social calibration, tolerance for sensory chaos, performance under observation) often differ dramatically from skills required for actual software development.

What Neurodivergent Talent Brings to Tech

The tragedy of this systematic exclusion becomes even sharper when considering what neurodivergent individuals bring to technical roles. Many possess precisely the capabilities that make exceptional programmers, data scientists, and engineers.

Pattern recognition stands out as a particular neurodivergent strength. Many autistic and dyslexic individuals demonstrate extraordinary abilities in identifying patterns and making connections between seemingly unrelated information. In scientific research, they excel at spotting patterns and correlations in complex datasets. In business contexts, they identify connections others miss, leading to innovative solutions and improved decision-making. In fields like design, architecture, and technology, they perceive structures and patterns that might be invisible to neurotypical colleagues.

Attention to detail is another common neurodivergent trait that translates directly to technical excellence. JPMorgan Chase found that employees hired through their neurodiversity programme into tech roles were 90% to 140% more productive than others, with consistent, error-free work. Within six months of their pilot programme, autistic employees proved 48% faster and nearly 92% more productive than neurotypical colleagues.

Hyperfocus, particularly common in ADHD individuals, enables sustained concentration on complex problems, often resulting in innovative solutions and exceptional outcomes. When provided with environments that support their working styles, neurodivergent employees can achieve levels of productivity and insight that justify building entire programmes around recruiting them.

Technical aptitude comes naturally to many neurodivergent individuals, who often excel in programming, coding, and computer science. Their analytical thinking and affinity for technology make them valuable in fields requiring technical expertise and innovation. Some possess exceptional memory skills, absorbing and recalling vast amounts of information, facilitating faster learning and enhanced problem-solving.

Deloitte research suggests workplaces with neurodivergent professionals in some roles can be 30% more productive, noting that “abilities such as visual thinking, attention to detail, pattern recognition, visual memory, and creative thinking can help illuminate ideas or opportunities teams might otherwise have missed.”

Companies Getting It Right

A growing number of organisations have recognised this untapped potential and restructured their hiring processes accordingly. Their success demonstrates that inclusive hiring isn't charity; it's competitive advantage.

SAP launched its Autism at Work initiative in 2013, creating an alternative pathway into the company that maintains rigorous standards whilst accommodating different neurological profiles. The programme operates in 12 countries and has successfully integrated over 200 autistic individuals into various positions. SAP enjoys a remarkable 90% retention rate for employees on the autism spectrum.

Microsoft's Neurodiversity Hiring Programme, established in 2015, reimagined the entire interview process. Instead of traditional phone screens and panel interviews, candidates attend a multi-day “academy” that's part interview, part workshop. This extended format allows candidates to demonstrate skills over time rather than in high-pressure snapshots. The company runs these sessions four to six times yearly and has hired 200 full-time employees spanning customer service, finance, business operations, and marketing.

JPMorgan Chase's Neurodiversity Hiring Programme began as a four-person pilot in 2015 and has since expanded to over 300 employees across 40 job categories in multiple countries. According to Bryan Gill from JPMorgan Chase, “None of this costs a lot and the accommodations are minimal. Moving a seat, perhaps changing a fluorescent bulb, and offering noise-cancelling headphones are the kinds of things we're talking about.”

The business case extends beyond retention and productivity. EY's Neurodiverse Centres of Excellence have generated one billion dollars in revenue and saved over 3.5 million hours through solutions created by neurodivergent employees. A 2024 study found that 63% of companies with neuro-inclusive hiring practices saw improvements in overall employee wellbeing, 55% observed stronger company culture, and 53% reported better people management.

These programmes share common elements. They provide detailed information in advance, including comprehensive agendas and explicit expectations. They offer accommodations like notes, questions provided beforehand, and clear, unambiguous instructions. They focus on work samples and portfolio reviews that demonstrate practical skills rather than hypothetical scenarios. They allow trial projects and job shadowing that let candidates prove capabilities in realistic settings.

Environmental considerations matter too. Quiet locations free from loud noises, bright lights, and distracting smells help candidates feel at ease. Ubisoft found success redesigning workspaces based on employee needs: quiet, controlled spaces for autistic employees who need focus; dynamic environments for individuals with ADHD. This adaptability maximises each employee's strengths.

Practical Steps Towards Inclusive Hiring

For companies without resources to launch comprehensive neurodiversity programmes, smaller changes can still dramatically improve inclusivity. Here's what accommodations look like in practice:

Before: A candidate with auditory processing challenges faces a rapid-fire verbal interview in a noisy conference room, struggling to process questions whilst managing background distractions.

After: The same candidate receives interview questions in writing (either in advance or displayed during the interview), allowing them to process information through their strength channel. The interview occurs in a quiet room, and the interviewer types questions in the chat during video calls.

Before: A candidate with ADHD faces a three-hour marathon interview with no breaks, their cognitive resources depleting as interviewers rotate through, ultimately appearing “unfocused” and “scattered” by the final round.

After: The interview schedule explicitly includes 15-minute breaks between sessions. The candidate can step outside, regulate their nervous system, and approach each conversation with renewed energy. Performance consistency across all rounds improves dramatically.

Before: An autistic candidate receives a vague email: “We'll have a technical discussion about your experience. Dress business casual. See you Tuesday!” They spend days anxious about what “technical discussion” means, who will attend, and what specific topics might arise.

After: The candidate receives a detailed agenda: “You'll meet with three engineers for 45 minutes each. Session one covers your recent database optimisation work. Session two involves a code walkthrough of your GitHub project. Session three discusses system design approaches. Here are the interviewers' names and roles. Interview questions are attached.” Anxiety transforms into productive preparation.

Replace timed, high-pressure technical interviews with take-home projects allowing candidates to work in comfortable environments at their own pace. Research shows work sample tests are among the strongest predictors of on-the-job performance and tend to be more equitable across demographic groups.

Provide interview questions in advance. This practice, now standard at some major tech brands, allows all candidates to prepare thoughtfully rather than privileging those who happen to excel at impromptu performance. As AskEARN guidance notes, candidates can request questions in writing without disclosing a diagnosis: “I have a condition that affects how I process verbal information, so I would like interview questions provided in writing.”

Offer explicit accommodation options upfront, before candidates must disclose disabilities. Simple statements like “We're happy to accommodate different working styles; please let us know if you'd benefit from receiving questions in advance, having extra time, taking breaks, or other adjustments” signal that accommodations are normal, not problematic. Under the Americans with Disabilities Act and Rehabilitation Act, employers are legally required to provide reasonable accommodations during hiring.

Implement structured interviews with standardised questions. Whilst unstructured interviews are biased and unreliable, structured interviews predict job performance with validity of 0.55 to 0.70, outperforming traditional approaches requiring up to four rounds for comparable accuracy.

Consider alternative formats to live coding. Code walkthroughs of recent projects on-screen, where candidates explain existing work, can reveal far more about actual capabilities than watching someone write algorithms under pressure. Portfolio reviews, GitHub contributions, and technical writing samples provide evidence of skills without performative elements.

Ask direct, specific questions rather than open-ended ones. Instead of “What can you bring to the table?” (which neurodivergent brains may interpret literally or find overwhelming), ask “Can you talk about a key project you recently worked on and how you contributed?” Open-ended questions cause neurodivergent minds to flood with information, whilst direct questions work better.

Reduce panel sizes. One-to-one interviews reduce anxiety compared to facing multiple interviewers simultaneously. If panels are necessary, provide clear information about who will attend, their roles, and what each will assess.

Train interviewers on neurodiversity and inclusive practices. Research found that bias dropped 13% when participants began with implicit association tests intended to detect subconscious bias. Forty-three percent of senior leaders received some neurodiversity training in 2025, up from 28% in 2023.

Create employee resource groups for neurodivergent employees. Ubisoft's ERG has grown to over 500 members globally, helping employees connect and thrive. Dell's True Ability ERG pairs new hires with experienced mentors for ongoing support.

The Deeper Question

These practical steps matter, but they address symptoms rather than the underlying condition. The deeper question is why tech companies, confronted with algorithmic bias, responded by retreating to traditional methods rather than designing genuinely better alternatives.

Part of the answer lies in what researchers call the “objectivity illusion.” Humans tend to trust their own judgements more than algorithmic outputs, even when evidence shows human decisions are more biased. When Amazon's algorithm discriminated against women, the bias was visible, quantifiable, and damning. When human interviewers make similar judgements, the bias hides behind subjective assessments of “cultural fit” and “communication skills.”

AI bias is a feature, not a bug. Algorithms trained on biased historical data reproduce that bias with mathematical precision. But this transparency can be leveraged. Algorithmic decisions can be audited, tested, and corrected in ways human decisions cannot. The problem isn't that AI is biased; it's that we built biased AI and then abandoned the entire approach rather than fixing it.

Meanwhile, traditional interviews embed biases so deeply into process and culture that they become invisible. When neurodivergent candidates fail pair programming interviews, interviewers attribute it to poor skills or bad cultural fit, not to interview design that systematically disadvantages certain neurological profiles. The bias is laundered through seemingly objective technical assessments.

This reveals a broader failure of imagination. Tech prides itself on solving complex problems through innovation and iteration. Faced with biased hiring AI, the industry could have invested in better algorithms, more representative training data, robust bias detection and correction mechanisms. Instead, it abandoned ship.

The same innovative energy directed at optimising ad click-through rates or recommendation algorithms could revolutionise hiring. Imagine interview processes that adapt to candidates' strengths, that measure actual job-relevant skills in ways accommodating neurological diversity, that use technology to reduce bias rather than amplify it.

Some experiments point in promising directions. Asynchronous video interviews allow candidates to answer questions in their own time, reducing pressure. Computer-based assessments provide instant feedback, helping autistic individuals improve performance. Structured digital platforms ensure every candidate faces identical questions in identical formats, reducing interviewer discretion and thus bias.

The Intersectional Dimension

For neurodivergent individuals from ethnic minority backgrounds, challenges compound. Research on intersectional stereotyping shows these candidates face layered discrimination that adversely affects recruitment, performance evaluation, and career progression. The biases don't simply add; they multiply, creating unique barriers that neither neurodiversity programmes nor diversity initiatives alone can address.

Women who are neurodivergent face particular challenges. Amazon's AI tool discriminated against women; traditional interviews often do too, filtered through gendered expectations about communication styles and leadership presence. Add neurodivergence to the mix, and the barriers become formidable.

This intersectionality demands more sophisticated responses than simply adding neurodiversity awareness to existing diversity training. It requires understanding how different forms of marginalisation interact, how biases reinforce each other, and how solutions must address the whole person rather than isolated demographic categories.

Companies ignoring these issues face growing legal exposure. Disability discrimination claims from neurodivergent employees have risen sharply. In fiscal year 2023, 488 autism-related Americans with Disabilities Act charges were filed with the EEOC, compared to just 53 ten years earlier and only 14 in 2003.

Remote work has become the most commonly requested accommodation for neurodivergent employees under the ADA, precisely because it provides control over work environments. Companies that eliminated remote options post-pandemic may find themselves defending decisions that disproportionately impact disabled workers.

The law is clear: employers must provide reasonable accommodations for qualified individuals with disabilities unless doing so would cause undue hardship. Many accommodations neurodivergent employees need cost little to nothing. Companies that refuse face not just legal liability but reputational damage in an industry claiming to value diversity.

What We're Really Measuring

Perhaps the most fundamental question is what interviews actually measure versus what we think they measure. Traditional interviews, including pair programming sessions, test a specific skill set: performing competence under observation in unfamiliar, high-pressure social situations requiring real-time multitasking and spontaneous problem-solving whilst managing interpersonal dynamics.

These capabilities matter for some roles. If you're hiring someone to give live demos to sceptical clients or debug critical systems whilst stakeholders watch anxiously, interview performance may correlate with job performance.

But for most technical roles, day-to-day work looks nothing like interviews. Developers typically work on problems over hours or days, not minutes. They have time to research, experiment, and iterate. They work in familiar environments with established routines. They collaborate asynchronously through well-defined processes, not impromptu pair programming. They manage their sensory environments and work schedules to optimise productivity.

By privileging interview performance over demonstrated ability, tech companies filter for candidates who excel at interviews, not necessarily at jobs. When it systematically excludes neurodivergent individuals who might outperform neurotypical colleagues in actual role requirements, it becomes both discriminatory and economically irrational.

Rethinking Progress

Tech cannot claim to value objectivity whilst relying on subjective, bias-laden interview processes. It cannot champion innovation whilst clinging to traditional hiring methods proven to exclude talented candidates. It cannot celebrate diversity whilst systematically filtering out neurological difference.

The flight from AI bias was understandable but incomplete. Algorithmic hiring tools reproduced historical discrimination, but retreating to equally biased human processes isn't the solution. Building better systems is. Both technological and human systems need redesign to actively counteract bias rather than embed it.

This means taking neurodiversity seriously, not as an HR checkbox but as a competitive imperative. It means redesigning interview processes from the ground up with inclusivity as a core requirement. It means measuring outcomes (who gets hired, who succeeds, who leaves and why) and iterating based on evidence.

The tech industry's talent shortage is partly self-inflicted. Millions of neurodivergent individuals possess precisely the skills companies claim they cannot find. They're filtered out not because they lack ability but because hiring processes cannot recognise ability packaged differently.

The companies demonstrating success with neurodiversity hiring programmes aren't being charitable. They're being smart. Ninety percent retention rates, 48% faster performance, 92% higher productivity, one billion dollars in revenue from neurodiverse centres: these are business results.

Every brilliant neurodivergent candidate filtered out by poorly designed interviews is a competitive advantage surrendered. The question isn't whether companies can afford to make hiring more inclusive. It's whether they can afford not to.

Amazon's biased algorithm taught an important lesson, but perhaps not the right one. The lesson wasn't “don't use technology in hiring.” It was “design better systems.” That principle applies equally to AI and to traditional interviews.

Tech has spent years agonising over AI bias whilst ignoring the bias baked into human decision-making. It's time to apply the same rigorous, evidence-based approach to interview processes that the industry applies to products. Test assumptions, measure outcomes, identify failures, iterate solutions.

Neurodivergent candidates aren't asking for lower standards. They're asking for fair assessment of their actual capabilities rather than their ability to perform neurotypicality under pressure. That's not a diversity favour. It's basic competence in hiring.

The paradox of progress is that moving forward sometimes requires questioning what we thought was already solved. Tech believed it had moved beyond crude brain teasers to sophisticated technical assessments. But sophisticated discrimination is still discrimination.

In fleeing AI's biases, tech ran straight into human prejudice hiding in hiring processes all along. The industry faces a choice: continue defending traditional interviews because they feel objective, or measure whether they're actually finding the best talent. The data increasingly suggests they're not.

Real progress requires acknowledging uncomfortable truths. “Culture fit” often means “people like us.” “Communication skills” sometimes translates to “neurotypical presentation.” The hardest technical problems in hiring aren't algorithmic. They're human.

The question isn't whether neurodivergent candidates can meet tech's standards. It's whether those standards measure what actually matters. Right now, the evidence suggests they're optimising for the wrong metrics and missing extraordinary talent.

That's not just unfair. In an industry built on finding edge advantages through better information and smarter systems, it's inexcusably inefficient. The companies that figure this out first won't just be more diverse. They'll be more competitive.

The problem was never the algorithms. It was the biases we fed them, the outcomes we optimised for, the assumptions we never questioned. Those same problems afflict traditional hiring, just less visibly. Making them visible is the first step. Actually fixing them is the work ahead.

References and Sources

Neurodivergent Employment Statistics

Interview Challenges and Discrimination

AI Bias in Hiring

Inclusive Hiring Best Practices

Sensory Processing and Executive Function

Neurodivergent Strengths

Company Success Stories

Traditional Interview Bias Research

Work Sample Tests and Structured Interviews

Workplace Discrimination

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

You pay £10.99 every month for Spotify Premium. You're shelling out £17.99 for Netflix's Standard plan. The deal seems straightforward: no adverts. Your listening and viewing experience stays pure, uninterrupted by commercial messages trying to sell you things. Clean. Simple. Worth it.

But here's the uncomfortable bit. What happens when that track surfacing in your Discover Weekly playlist, the one that feels perfectly tailored to your taste, is actually sitting there because the artist accepted reduced royalties for promotional placement? What if that show dominating your Netflix homepage wasn't prioritised by viewing patterns at all, but by a studio's commercial arrangement?

Welcome to 2025's peculiar paradox of premium subscriptions. Paying to avoid advertising might not protect you from being advertised to. It just means the sales pitch arrives wrapped in the language of personalisation rather than interruption. The algorithm knows what you want. Trust the algorithm. Except the algorithm might be serving someone else's interests entirely.

Here's what millions of subscribers are starting to realise: the question isn't whether they're being marketed to through these platforms. The evidence suggests they absolutely are. The real question is whether this constitutes a breach of contract, a violation of consumer protection law, or simply a fundamental reimagining of what advertising means when algorithms run the show.

The Architecture of Influence

To understand how we got here, you need to grasp how recommendation algorithms actually work. These systems aren't passive mirrors reflecting your preferences back at you. They're active agents shaping what you see, hear, and ultimately consume.

Netflix has stated publicly that 75 to 80 per cent of all viewing hours on its platform come from algorithmic recommendations, not user searches. The vast majority of what Netflix subscribers watch isn't content they actively sought out. It's content the algorithm decided to surface, using collaborative filtering that examines viewing behaviour patterns across millions of users. You think you're choosing. You're mostly accepting suggestions.

Spotify combines collaborative filtering with natural language processing and audio analysis. The platform analyses your listening history, playlist additions, skip rates, save rates (tracks with save rates above 8 per cent are 3.5 times more likely to receive algorithmic playlist placements), and dozens of other engagement metrics. Algorithmic playlists like Discover Weekly, Release Radar, and Radio now account for approximately 35 per cent of new artist discoveries, compared to 28 per cent from editorial playlists.

These numbers reveal something crucial. The algorithm isn't just a feature of these platforms. It's the primary interface through which content reaches audiences. Control the algorithm, and you control visibility. Control visibility, and you control commercial success.

Which raises an uncomfortable question: what happens when access to that algorithm becomes something you can buy?

Discovery Mode and the Spectre of Payola

In 2020, Spotify introduced Discovery Mode. The feature allows artists and labels to designate specific tracks as priorities for algorithmic consideration. These flagged tracks become more likely to appear in Radio, Autoplay, and certain algorithmically generated Mixes. The cost? Artists accept reduced royalties on streams generated through these promotional placements.

Spotify frames this as an opt-in marketing tool rather than paid promotion. “It doesn't buy plays, it doesn't affect editorial playlists, and it's clearly disclosed in the app and on our website,” a company spokesperson stated. But critics see something else entirely: a modern reincarnation of payola, the practice of secretly paying radio stations for airplay. Payola has been illegal in the United States since 1960.

The comparison isn't casual. Payola regulations emerged from the Communications Act of 1934, requiring broadcasters to disclose when material was paid for or sponsored. The Federal Communications Commission treats violations seriously. In 2007, four major radio companies settled payola accusations for $12.5 million.

But here's the catch. Spotify isn't a broadcaster subject to FCC jurisdiction. It's an internet platform, operating in a regulatory grey zone where traditional payola rules simply don't apply. The FTC's general sponsorship disclosure requirements are far less stringent than those of broadcasters, as one legal analysis noted.

In March 2025, this regulatory gap became the subject of litigation. A class action lawsuit filed in Manhattan federal court alleged that Discovery Mode constitutes a “modern form of payola” that allows record labels and artists to secretly pay for promotional visibility. The lawsuit's central claim cuts right to it: “Telling users that 'commercial considerations may influence' recommendations does not reveal which songs are being promoted commercially and which are being recommended organically. Without that specificity, users cannot distinguish between genuine personalisation and covert advertising.”

Spotify called the lawsuit “nonsense”, insisting it gets “basic facts” wrong. But the case crystallises the core tension. Even if Spotify discloses that commercial considerations might influence recommendations, that disclosure appears in settings or help documentation that most users never read. The recommendations themselves carry no marker indicating whether they're organic algorithmic suggestions or commercially influenced placements.

For premium subscribers, this matters. They're paying specifically to avoid commercial interruption. But if the personalised playlists they receive contain tracks placed there through commercial arrangements, are they still receiving what they paid for? Or did the definition of “ad-free” quietly shift when no one was looking?

Netflix's Algorithmic Opacity

Netflix operates differently from Spotify, but faces similar questions about the relationship between commercial interests and recommendation algorithms. The platform positions its recommendation system as editorially driven personalisation, using sophisticated machine learning to match content with viewer preferences.

Yet Netflix's business model creates inherent conflicts of interest. The platform both licenses content from third parties and produces its own original programming. When Netflix's algorithm recommends a Netflix original, the company benefits twice: first from subscription revenue, and second from building the value of its content library. When it recommends licensed content, it pays licensing fees whilst generating no additional revenue beyond existing subscriptions.

The economic incentives are clear. Netflix benefits most when subscribers watch Netflix-produced content. Does this influence what the algorithm surfaces? Netflix maintains that recommendations are driven purely by predicted viewing enjoyment, not corporate financial interests. But the opacity of proprietary algorithms makes independent verification impossible.

One researcher observed that “The most transparent company I've seen thus far is Netflix, and even they bury the details in their help docs.” Another noted that “lack of transparency isn't just annoying; it's a critical flaw. When we don't understand the logic, we can't trust the suggestion.”

This opacity matters particularly for ad-free subscribers. Netflix's Standard plan costs £17.99 monthly in the UK, whilst the ad-supported tier costs just £7.99. Those paying more than double for an ad-free experience presumably expect recommendations driven by their viewing preferences, not Netflix's production investments.

But proving that content receives preferential algorithmic treatment based on commercial interests is nearly impossible from the outside. The algorithms are proprietary. The training data is private. The decision-making logic is opaque. Subscribers are asked to trust that platforms prioritise user satisfaction over commercial interests, but have no way to verify that trust is warranted.

The Blurring Line Between Curation and Commerce

The distinction between editorial curation and advertising has always been fuzzy. Magazine editors choose which products to feature based on editorial judgement, but those judgements inevitably reflect commercial relationships with advertisers. The difference is disclosure: advertorial content is supposed to be clearly labelled.

Digital platforms have eroded this distinction further. YouTube allows creators to embed sponsorships directly into their content. Even YouTube Premium subscribers, who pay to avoid YouTube's own advertisements, still see these creator-embedded sponsored segments. The platform requires creators to flag videos containing paid promotions, triggering a disclosure label at the start of the video.

This creates a two-tier advertising system: YouTube's own ads, which Premium subscribers avoid, and creator-embedded sponsors, which appear regardless of subscription status. But at least these sponsorships are disclosed as paid promotions. The situation becomes murkier when platforms use algorithmic recommendations influenced by commercial considerations without clear disclosure at the point of recommendation.

Research into algorithmic bias has documented several types of systematic preferential treatment in recommendation systems. Popularity bias causes algorithms to favour already-popular content. Exposure bias means recommendations depend partly on which items are made available to the algorithm. Position bias gives preference to items presented prominently.

More concerning is the documented potential for commercial bias. In a 1998 paper describing Google, the company's founders argued that “advertising-funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.” That was Larry Page and Sergey Brin, before Google became the advertising-funded search engine. A president of an airline testified to the United States Congress that a flight recommendation system was created with the explicit intention of gaining competitive advantage through preferential treatment.

These examples demonstrate that recommendation systems can be, and have been, designed to serve commercial interests over user preferences. The question for streaming platforms is whether they're doing the same thing, and if so, whether their ad-free subscribers have been adequately informed.

Contract, Consumer Protection, and Advertising Law

When you subscribe to Spotify Premium or Netflix, you enter a contractual relationship. What exactly has been promised regarding advertisements and commercial content? The answer matters.

Spotify Premium's marketing emphasises “ad-free music listening.” But what counts as an ad? Is a track that appears in your Discover Weekly because the artist accepted reduced royalties for promotional placement an advertisement? Spotify would likely argue it isn't, because the track wasn't inserted as an interruptive commercial message. Critics would counter that if the track's appearance was influenced by commercial considerations, it's advertising by another name.

Contract law offers some guidance. In February 2025, a federal judge dismissed a class-action lawsuit challenging Amazon Prime Video's introduction of advertisements. The lawsuit argued that adding ads breached the subscription contract and violated state consumer protection laws. Amazon had begun showing advertisements to Prime Video users unless they paid an additional $2.99 monthly for an ad-free experience.

The court sided with Amazon. The reasoning? Amazon's terms of service explicitly reserve the right to change the Prime Video service. “Plaintiffs did not purchase access to 'ad-free Prime Video,' let alone an ad-free Prime Video that Amazon promised would remain ad-free,” the court stated. “They purchased access to Prime Video, subject to any changes that Amazon was contractually authorised 'in its sole discretion' to make.”

This decision establishes important precedent. Platforms can modify their services, including adding advertisements, if their terms of service reserve that right and subscribers accepted those terms. But it doesn't address the subtler question of whether algorithmically surfaced content influenced by commercial considerations constitutes advertising that breaches an ad-free promise.

UK consumer protection law offers potentially stronger protections. The Consumer Protection from Unfair Trading Regulations 2008 prohibits misleading actions and omissions. If platforms market subscriptions as “ad-free” whilst simultaneously surfacing content based on commercial arrangements without adequate disclosure, this could constitute a misleading omission under UK law.

The Digital Markets, Competition and Consumers Act 2024 strengthens these protections significantly. Provisions taking effect in April 2025 and through 2026 require businesses to provide clear pre-contract information about subscription services. More importantly, the Act prohibits “drip pricing,” where consumers see an initial price but then face additional undisclosed fees.

The drip pricing prohibition is particularly relevant here. If subscribers pay for an ad-free experience but then receive algorithmically surfaced content influenced by commercial arrangements, could this be considered a form of drip pricing, where the true nature of the service isn't fully disclosed upfront?

The Act also grants the Competition and Markets Authority new direct consumer enforcement powers, including the ability to impose turnover-based fines up to 10 per cent of a company's global annual turnover for breaches of UK consumer law. That creates real enforcement teeth that didn't previously exist.

FTC guidance in the United States requires that advertising disclosures be “clear and conspicuous,” difficult to miss and easy to understand. The FTC has also issued guidance specific to algorithmic decision-making, stating that when companies rely on algorithms to make significant decisions affecting consumers, they must be able to disclose the key factors influencing those decisions.

Applying this to streaming recommendations raises uncomfortable questions. If Spotify's Discovery Mode influences which tracks appear in personalised playlists, shouldn't each recommended track indicate whether it's there through organic algorithmic selection or commercial arrangement? If Netflix's algorithm gives preferential treatment to Netflix originals, shouldn't recommendations disclose this bias?

The current practice of burying general disclosures in terms of service or help documentation may not satisfy regulatory requirements for clear and conspicuous disclosure. Particularly for UK subscribers, where the CMA now has enhanced enforcement powers, platforms may face increasing pressure to provide more transparent, point-of-recommendation disclosures about commercial influences on algorithmic curation.

The Business Model Incentive Structure

To understand why platforms might blur the line between organic recommendations and commercial placements, consider their business models and revenue pressures.

Spotify operates on razor-thin margins, paying approximately 70 per cent of its revenue to rights holders. Despite having 626 million monthly active users as of Q3 2024, profitability remains elusive. Advertising revenue from the free tier reached €1.85 billion in 2024, a 10 per cent increase, but still represents only a fraction of total revenue.

Discovery Mode offers Spotify a way to extract additional value without raising subscription prices or adding interruptive advertisements. Artists and labels desperate for visibility accept reduced royalties, improving Spotify's margins on those streams whilst maintaining the appearance of an ad-free premium experience.

Netflix faces different but related pressures. The company spent billions building its original content library. Every subscriber who watches licensed content instead of Netflix originals represents a missed opportunity to build the value of Netflix's proprietary assets. This creates powerful incentives to steer subscribers toward Netflix-produced content through algorithmic recommendations.

For all these platforms, the challenge is balancing user satisfaction against revenue optimisation. Degrade the user experience too much, and subscribers cancel. But leave revenue opportunities untapped, and shareholders demand explanation.

Algorithmic curation influenced by commercial considerations represents a solution to this tension. Unlike interruptive advertising, which users clearly recognise and often resent, algorithmically surfaced paid placements disguised as personalised recommendations can generate revenue whilst maintaining the appearance of an ad-free experience.

At least until users realise what's happening. Which they're starting to do.

Platform Disclosures and the Limits of Fine Print

Spotify does disclose that commercial considerations may influence recommendations. The platform's help documentation states: “We may use the information we collect about you, including information about your use of Spotify...for commercial or sponsored content.”

But this disclosure is generic and buried in documentation most users never read. Research consistently shows that users don't read terms of service. One study found that it would take 76 work days annually for the average internet user to read the privacy policies of every website they visit.

Even users who do read terms of service face another problem: the disclosures are vague. Spotify's statement that it “may use” information “for commercial or sponsored content” doesn't specify which recommendations are influenced by commercial considerations and which aren't.

YouTube's approach offers a potential model for more transparent disclosure. When creators flag content as containing paid promotions, YouTube displays “Includes paid promotion” at the start of the video. This disclosure is clear, conspicuous, and appears at the point of consumption, not buried in settings or help documentation.

Applying this model to Spotify and Netflix would mean flagging specific recommendations as commercially influenced at the point they're presented to users. A Discover Weekly track included through Discovery Mode could carry a discrete indicator: “Promotional placement.”

Platforms resist this level of transparency. Likely for good reason: clear disclosure would undermine the value of the placements. The effectiveness of algorithmically surfaced paid placements depends on users perceiving them as organic recommendations. Explicit labelling would destroy that perception.

This creates a fundamental conflict. Effective disclosure would negate the value of the commercial practice, whilst inadequate disclosure potentially misleads consumers about what they're paying for when they subscribe to ad-free services. Either kill the revenue stream or mislead subscribers.

Subscriber Expectations and the Ad-Free Promise

The Competition and Markets Authority's 2022 music streaming market study in the UK found that between 2019 and 2021, monthly active users of music streaming services increased from 32 million to 39 million, with Spotify commanding 50 to 60 per cent market share.

The rapid growth of ad-supported tiers reveals preferences. Netflix's ad-supported tier reached 45 per cent of US households by August 2025, up from just 34 per cent in 2024. This suggests many subscribers are willing to tolerate advertisements in exchange for lower prices. Conversely, those paying premium prices likely have stronger expectations of a genuinely ad-free experience.

The Amazon Prime Video lawsuit, whilst dismissed on contractual grounds, revealed subscriber frustration. Plaintiffs argued that Amazon “reaped undue benefits by marketing Prime Video as devoid of commercials before introducing ads.” The claim was that subscribers made purchasing decisions based on an understanding that the service would remain ad-free, even if the terms of service technically allowed modifications.

This points to a gap between legal obligations and reasonable consumer expectations. Legally, platforms can reserve broad rights to modify services if terms of service say so. But consumer protection law also recognises that businesses shouldn't exploit consumer ignorance or the impracticality of reading lengthy terms of service.

If most subscribers reasonably understand “ad-free” to mean “free from commercial promotion,” but platforms interpret it narrowly as “free from interruptive advertisement breaks,” there's a disconnect that arguably constitutes misleading marketing, particularly under UK consumer protection law. The gap between what subscribers think they're buying and what platforms think they're selling might be legally significant.

Regulatory Responses and Enforcement Gaps

Traditional advertising regulation developed for broadcast media and print publications. But streaming platforms exist in a regulatory gap. They're not broadcasters subject to FCC sponsorship identification rules. They're internet platforms, governed by general consumer protection law and advertising standards, but not by media-specific regulation.

The FTC has attempted to address this gap through guidance on digital advertising disclosure. The agency's 2013 guidance document “.Com Disclosures” established that online advertising must meet the same “clear and conspicuous” standard as offline advertising.

But enforcement remains limited. The FTC's 2023 orders to eight social media and video streaming platforms sought information about how companies scrutinise deceptive advertising. This was an information-gathering exercise, not enforcement action.

In the UK, the Advertising Standards Authority and the Committee of Advertising Practice provide self-regulatory oversight of advertising, but their jurisdiction over algorithmic content curation remains unclear.

The 2024 Digital Markets, Competition and Consumers Act provides the CMA with enhanced powers but doesn't specifically address algorithmic curation influenced by commercial considerations. The Act's fake reviews provisions require disclosure when reviews are incentivised, establishing a precedent for transparency when commercial considerations influence seemingly organic content. But the Act doesn't explicitly extend this principle to streaming recommendations.

In the United States, FCC Commissioner Brendan Carr has raised questions about whether Spotify's Discovery Mode should be subject to payola-style regulation. This suggests growing regulatory interest, but actual enforcement remains uncertain.

The European Union's Digital Services Act, which took effect in 2024, requires very large online platforms to provide transparency about recommender systems, including meaningful information about the main parameters used and options for modifying recommendations. But “meaningful information” remains vaguely defined, and enforcement is still developing.

The Attention Economy's Ethical Dilemma

Step back from legal technicalities, and a broader ethical question emerges. Is it acceptable for platforms to sell access to user attention that users believed they were protecting by paying for ad-free subscriptions?

The attention economy frames user attention as a scarce resource that platforms compete to capture and monetise. Free services monetise attention through advertising. Paid services monetise attention through subscriptions. But increasingly, platforms want both revenue streams.

This becomes ethically questionable when it's not transparently disclosed. If Spotify Premium subscribers knew that their Discover Weekly playlists contain tracks that artists paid to place there (through reduced royalties), would they still perceive the service as ad-free? If Netflix subscribers understood that recommendations systematically favour Netflix originals for commercial reasons, would they trust the algorithm to serve their interests?

The counterargument is that some commercial influence on recommendations might actually benefit users. Discovery Mode, Spotify argues, helps artists find audiences who genuinely would enjoy their music. The commercial arrangement funds the algorithmic promotion, but the promotion only works if users actually like the tracks and engage with them.

But these justifications only work if users are informed and can make autonomous decisions about whether to trust platform recommendations. Without disclosure, users can't exercise informed consent. They're making decisions based on false assumptions about why those options are being presented.

This is where the practice crosses from aggressive business strategy into potential deception. The value of algorithmic recommendations depends on users trusting that recommendations serve their interests. If recommendations actually serve platforms' commercial interests, but users believe they serve their own interests, that's a betrayal of trust whether or not it violates specific regulations.

What Subscribers Actually Bought

Return to the original question. When you pay for Spotify Premium or Netflix's ad-free tier, what exactly are you buying?

Legally, you're buying whatever the terms of service say you're buying, subject to any modifications the platform reserved the right to make. The Amazon Prime Video decision establishes this clearly.

But consumer protection law recognises that contracts alone don't determine the full scope of seller obligations. Misleading marketing, unfair commercial practices, and violations of reasonable consumer expectations can override contractual language, particularly when contracts involve standard-form terms that consumers can't negotiate.

If platforms market subscriptions as “ad-free” using language that reasonably suggests freedom from commercial promotion, but then implement algorithmic curation influenced by commercial considerations without clear disclosure, this creates a gap between marketing representations and service reality. That gap might be legally significant.

For UK subscribers, the enhanced CMA enforcement powers under the 2024 Digital Markets, Competition and Consumers Act create real regulatory risk. The CMA can investigate potentially misleading marketing and unfair commercial practices, impose significant penalties, and require changes to business practices.

The Spotify Discovery Mode lawsuit will test whether courts view algorithmically surfaced paid placements in “ad-free” premium services as a form of undisclosed advertising that violates consumer protection law. The case's theory is that even if generic disclosure exists in help documentation, the lack of specific, point-of-recommendation disclosure means users can't distinguish organic recommendations from paid placements, making the practice deceptive.

If courts accept this reasoning, it could force platforms to implement recommendation-level disclosure similar to YouTube's “Includes paid promotion” labels. If courts reject it, platforms will have legal confirmation that generic disclosure in terms of service suffices, even if most users never read it.

The Transparency Reckoning

The streaming industry's approach to paid placements within algorithmically curated recommendations represents a test case for advertising ethics in the algorithmic age. Traditional advertising was interruptive and clearly labelled. You knew an ad when you saw one. Algorithmic advertising is integrated and often opaque. You might never know you're being sold to.

This evolution challenges foundational assumptions in advertising regulation. If users can't distinguish commercial promotion from organic recommendation, does disclosure buried in terms of service suffice? If platforms sell access to user attention through algorithmic placement, whilst simultaneously charging users for “ad-free” experiences, have those users received what they paid for?

The legal answers remain uncertain. The ethical answers seem clearer. Subscribers paying for ad-free experiences reasonably expect that personalised recommendations serve their interests, not platforms' commercial interests. When recommendations are influenced by commercial considerations without clear, point-of-recommendation disclosure, platforms are extracting value from subscriber attention that subscribers believed they were protecting by paying premium prices.

The resolution will likely come through some combination of regulatory enforcement, litigation, and market pressure. The CMA's enhanced powers under the 2024 Digital Markets, Competition and Consumers Act create significant UK enforcement risk. The Spotify Discovery Mode lawsuit could establish important US precedent. And subscriber awareness, once raised, creates market pressure for greater transparency.

Platforms can respond by embracing transparency, clearly labelling which recommendations involve commercial considerations. They can create new subscription tiers offering guaranteed recommendation purity at premium prices. Or they can continue current practices and hope that generic disclosure in terms of service provides sufficient legal protection whilst subscriber awareness remains low.

But that last option becomes less viable as awareness grows. Journalists are investigating. Regulators are questioning. Subscribers are litigating.

The ad-free promise, it turns out, is more complicated than it appeared. What subscribers thought they were buying may not be what platforms thought they were selling. And that gap, in both legal and ethical terms, is becoming increasingly difficult to ignore.

When platforms sell recommendation influence whilst simultaneously charging for ad-free experiences, they're not just optimising business models. They're redefining the fundamental bargain of premium subscriptions: from “pay to avoid commercial interruption” to “pay for algorithmically optimised commercial integration.” That's quite a shift. Whether anyone actually agreed to it is another question entirely.

Whether that redefinition survives regulatory scrutiny, legal challenge, and subscriber awareness remains to be seen. But the question is now being asked, clearly and publicly: what exactly did we buy when we paid for ad-free? And if what we received isn't what we thought we were buying, what remedy do we deserve?

The answer will need to come soon. Because millions of subscribers are waiting.

References & Sources

Legal Cases and Regulatory Documents:

Research and Industry Reports:

  • Competition and Markets Authority, “Music and streaming market study” (2022). UK Government. Available at: https://www.gov.uk/cma-cases/music-and-streaming-market-study

  • Netflix recommendation system statistics: 75-80% of viewing from algorithmic recommendations. Multiple academic and industry sources.

  • Spotify Discovery Mode statistics: 35% of new artist discoveries from algorithmic playlists vs 28% from editorial. Industry reporting 2024-2025.

  • Spotify track save rate data: 8% save rate threshold for 3.5x increased algorithmic placement likelihood. Industry analysis 2024.

Academic Research:

  • “How Algorithmic Confounding in Recommendation Systems” (2017). arXiv. Available at: https://arxiv.org/pdf/1710.11214

  • “Algorithms are not neutral: Bias in collaborative filtering.” PMC/National Center for Biotechnology Information. Available at: https://pmc.ncbi.nlm.nih.gov/articles/PMC8802245/

  • “A survey on popularity bias in recommender systems.” User Modeling and User-Adapted Interaction, Springer, 2024.

News and Industry Analysis:

  • “Spotify Lawsuit Says 'Discovery Mode' Is Just 'Modern Payola',” Billboard, 2025.

  • “Class Action Lawsuit Accuses Spotify of Engaging in 'Payola' in Discovery Mode,” Rolling Stone, 2025.

  • “Does Spotify's New 'Discovery Mode' Resemble Anti-Creator 'Payola?'” Recording Academy/GRAMMY.com.

  • “Amazon Moves to Dismiss Class Action Over Prime Video Ads,” Lawyer Monthly, 2024.

  • “Netflix's Ad Tier Has Almost Half of Its Household Viewing Hours, According to Comscore,” Adweek, 2025.

Regulatory and Government Sources:

  • FCC Sponsorship Identification Rules and payola regulations. Federal Communications Commission. Available at: https://www.fcc.gov

  • “FTC Issues Orders to Social Media and Video Streaming Platforms Regarding Efforts to Address Surge in Advertising for Fraudulent Products and Scams” (2023). Federal Trade Commission.

  • UK Consumer Protection Laws and Regulations Report 2025. ICLG (International Comparative Legal Guides).

Platform Documentation:

Historical Context:

  • Communications Act of 1934 (as amended), United States.

  • FCC payola enforcement actions, including 2007 settlements with CBS Radio, Citadel, Clear Channel, and Entercom totalling $12.5 million.

  • “FCC Commissioner Asks Record Labels for Information About Payola Practices,” Broadcast Law Blog, 2020.

Industry Statistics:

  • Spotify advertising revenue: €1.85 billion in 2024, 10% increase year-over-year. Company financial reports.

  • Netflix UK pricing: Standard tier £17.99, ad-supported tier £7.99 (2025).

  • Spotify UK Premium pricing: £10.99 monthly (2025).

  • Amazon Prime Video ad-free tier pricing: $2.99 monthly additional fee (US).

  • Netflix ad-supported tier penetration: 45% of US households August 2025, up from 34% in 2024.

  • UK music streaming market: 39 million monthly active users in 2021, up from 32 million in 2019. CMA market study.

  • Spotify market share UK: 50-60% of monthly active users. CMA market study 2022.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Open your phone right now and look at what appears. Perhaps TikTok serves you videos about obscure cooking techniques you watched once at 2am. Spotify queues songs you didn't know existed but somehow match your exact mood. Google Photos surfaces a memory from three years ago at precisely the moment you needed to see it. The algorithms know something uncanny: they understand patterns in your behaviour that you haven't consciously recognised yourself.

This isn't science fiction. It's the everyday reality of consumer-grade AI personalisation, a technology that has woven itself so thoroughly into our digital lives that we barely notice its presence until it feels unsettling. More than 80% of content viewed on Netflix comes from personalised recommendations, whilst Spotify proudly notes that 81% of its 600 million-plus listeners cite personalisation as what they like most about the platform. These systems don't just suggest content; they shape how we discover information, form opinions, and understand the world around us.

Yet beneath this seamless personalisation lies a profound tension. How can designers deliver these high-quality AI experiences whilst maintaining meaningful user consent and avoiding harmful filter effects? The question is no longer academic. As AI personalisation becomes ubiquitous across platforms, from photo libraries to shopping recommendations to news feeds, we're witnessing the emergence of design patterns that could either empower users or quietly erode their autonomy.

The Architecture of Knowing You

To understand where personalisation can go wrong, we must first grasp how extraordinarily sophisticated these systems have become. Netflix's recommendation engine represents a masterclass in algorithmic complexity. By 2024, the platform employs a hybrid system blending collaborative filtering, content-based filtering, and deep learning. Collaborative filtering analyses patterns across its massive user base, identifying similarities between viewers. Content-based filtering examines the attributes of shows themselves, from genre to cinematography style. Deep learning models synthesise these approaches, finding non-obvious correlations that human curators would miss.

Spotify's “Bandits for Recommendations as Treatments” system, known as BaRT, operates at staggering scale. Managing a catalogue of over 100 million tracks, 4 billion playlists, and 5 million podcast titles, BaRT combines three main algorithms. Collaborative filtering tracks what similar listeners enjoy. Natural language processing analyses song descriptions, reviews, and metadata. Audio path analysis examines the actual acoustic properties of tracks. Together, these algorithms create what the company describes as hyper-personalisation, adapting not just to what you've liked historically, but to contextual signals about your current state.

TikTok's approach differs fundamentally. Unlike traditional social platforms that primarily show content from accounts you follow, TikTok's For You Page operates almost entirely algorithmically. The platform employs advanced sound and image recognition to identify content elements within videos, enabling recommendations based on visual themes and trending audio clips. Even the speed at which you scroll past a video feeds into the algorithm's understanding of your preferences. This creates what researchers describe as an unprecedented level of engagement optimisation.

Google Photos demonstrates personalisation in a different domain entirely. The platform's “Ask Photos” feature, launched in 2024, leverages Google's Gemini model to understand not just what's in your photos, but their context and meaning. You can search using natural language queries like “show me photos from that trip where we got lost,” and the system interprets both the visual content and associated metadata to surface relevant images. The technology represents computational photography evolving into computational memory.

Apple Intelligence takes yet another architectural approach. Rather than relying primarily on cloud processing, Apple's system prioritises on-device computation. For tasks requiring more processing power, Apple developed Private Cloud Compute, running on the company's own silicon servers. This hybrid approach attempts to balance personalisation quality with privacy protection, though whether it succeeds remains hotly debated.

These systems share a common foundation in machine learning, but their implementations reveal fundamentally different philosophies about data, privacy, and user agency. Those philosophical differences become critical when we examine the consent models governing these technologies.

The European Union's General Data Protection Regulation, which came into force in 2018, established what seemed like a clear principle: organisations using AI to process personal data must obtain valid consent. The AI Act, adopted in June 2024 and progressively implemented through 2027, builds upon this foundation. Together, these regulations require that consent be informed, explicit, and freely given. Individuals must receive meaningful information about the purposes of processing and the logic involved in AI decision-making, presented in a clear, concise, and easily comprehensible format.

In theory, this creates a robust framework for user control. In practice, the reality is far more complex.

Consider Meta's 2024 announcement that it would utilise user data from Facebook and Instagram to train its AI technologies, processing both public and non-public posts and interactions. The company implemented an opt-out mechanism, ostensibly giving users control. But the European Center for Digital Rights alleged that Meta deployed what they termed “dark patterns” to undermine genuine consent. Critics documented misleading email notifications, redirects to login pages, and hidden opt-out forms requiring users to provide detailed reasons for their choice.

This represents just one instance of a broader phenomenon. Research published in 2024 examining regulatory enforcement decisions found widespread practices including incorrect categorisation of third-party cookies, misleading privacy policies, pre-checked boxes that automatically enable tracking, and consent walls that block access to content until users agree to all tracking. The California Privacy Protection Agency responded with an enforcement advisory in September 2024, requiring that user interfaces for privacy choices offer “symmetry in choice,” emphasising that dark pattern determination is based on effect rather than intent.

The fundamental problem extends beyond individual bad actors. Valid consent requires genuine understanding, but the complexity of modern AI systems makes true comprehension nearly impossible for most users. How can someone provide informed consent to processing by Spotify's BaRT system if they don't understand collaborative filtering, natural language processing, or audio path analysis? The requirement for “clear, concise and easily comprehensible” information crashes against the technical reality that these systems operate through processes even their creators struggle to fully explain.

The European Data Protection Board recognised this tension, sharing guidance in 2024 on using AI in compliance with GDPR. But the guidance reveals the paradox at the heart of consent-based frameworks. Article 22 of GDPR gives individuals the right not to be subject to decisions based solely on automated processing that significantly affects them. Yet if you exercise this right on platforms like Netflix or Spotify, you effectively break the service. Personalisation isn't a feature you can toggle off whilst maintaining the core value proposition. It is the core value proposition.

This raises uncomfortable questions about whether consent represents genuine user agency or merely a legal fiction. When the choice is between accepting pervasive personalisation or not using essential digital services, can we meaningfully describe that choice as “freely given”? Some legal scholars argue for shifting from consent to legitimate interest under Article 6(1)(f) of GDPR, which requires controllers to conduct a thorough three-step assessment balancing their interests against user rights. But this merely transfers the problem rather than solving it.

The consent challenge becomes even more acute when we examine what happens after users ostensibly agree to personalisation. The next layer of harm lies not in the data collection itself, but in its consequences.

The Filter That Shapes Your World

Eli Pariser coined the term “filter bubble” around 2010, warning in his 2011 book that algorithmic personalisation would create “a unique universe of information for each of us,” leading to intellectual isolation and social fragmentation. More than a decade later, the evidence presents a complex and sometimes contradictory picture.

Research demonstrates that filter bubbles do emerge through specific mechanisms. Algorithms prioritise content based on user behaviour and engagement metrics, often selecting material that reinforces pre-existing beliefs rather than challenging them. A 2024 study found that filter bubbles increased polarisation on platforms by approximately 15% whilst significantly reducing the number of posts generated by users. Social media users encounter substantially more attitude-consistent content than information contradicting their views, creating echo chambers that hamper decision-making ability.

The harms extend beyond political polarisation. News recommender systems tend to recommend articles with negative sentiments, reinforcing user biases whilst reducing news diversity. Current recommendation algorithms primarily prioritise enhancing accuracy rather than promoting diverse outcomes, one factor contributing to filter bubble formation. When recommendation systems tailor content with extreme precision, they inadvertently create intellectual ghettos where users never encounter perspectives that might expand their understanding.

TikTok's algorithm demonstrates this mechanism with particular clarity. Because the For You Page operates almost entirely algorithmically rather than showing content from followed accounts, users can rapidly descend into highly specific content niches. Someone who watches a few videos about a conspiracy theory may find their entire feed dominated by related content within hours, with the algorithm interpreting engagement as endorsement and serving progressively more extreme variants.

Yet the research also reveals significant nuance. A systematic review of filter bubble literature found conflicting reports about the extent to which personalised filtering occurs and whether such activity proves beneficial or harmful. Multiple studies produced inconclusive results, with some researchers arguing that empirical evidence warranting worry about filter bubbles remains limited. The filter bubble effect varies significantly based on platform design, content type, and user behaviour patterns.

This complexity matters because it reveals that filter bubbles are not inevitable consequences of personalisation, but rather design choices. Recommendation algorithms prioritise particular outcomes, currently accuracy and engagement. They could instead prioritise diversity, exposure to challenging viewpoints, or serendipitous discovery. The question is whether platform incentives align with those alternative objectives.

They typically don't. Social media platforms operate on attention-based business models. The longer users stay engaged, the more advertising revenue platforms generate. Algorithms optimised for engagement naturally gravitate towards content that provokes strong emotional responses, whether positive or negative. Research on algorithmic harms has documented this pattern across domains from health misinformation to financial fraud to political extremism. Increasingly agentic algorithmic systems amplify rather than mitigate these effects.

The mental health implications prove particularly concerning. Whilst direct research on algorithmic personalisation's impact on mental wellbeing remains incomplete, adjacent evidence suggests significant risks. Algorithms that serve highly engaging but emotionally charged content can create compulsive usage patterns. The filter bubble phenomenon may harm democracy and wellbeing by making misinformation effects worse, creating environments where false information faces no counterbalancing perspectives.

Given these documented harms, the question becomes: can we measure them systematically, creating accountability whilst preserving personalisation's benefits? This measurement challenge has occupied researchers throughout 2024, revealing fundamental tensions in how we evaluate algorithmic systems.

Measuring the Unmeasurable

The ACM Conference on Fairness, Accountability, and Transparency featured multiple papers in 2024 addressing measurement frameworks, each revealing the conceptual difficulties inherent to quantifying algorithmic harm.

Fairness metrics in AI attempt to balance competing objectives. False positive rate difference and equal opportunity difference evaluate calibrated fairness, seeking to provide equal opportunities for all individuals whilst accommodating their distinct differences and needs. In personalisation contexts, this might mean ensuring equal access whilst considering specific factors like language or location to offer customised experiences. But what constitutes “equal opportunity” when the content itself is customised? If two users with identical preferences receive different recommendations because one engages more actively with the platform, has fairness been violated or fulfilled?

Research has established many sources and forms of algorithmic harm across domains including healthcare, finance, policing, and recommendations. Yet concepts like “bias” and “fairness” remain inherently contested, messy, and shifting. Benchmarks promising to measure such terms inevitably suffer from what researchers describe as “abstraction error,” attempting to quantify phenomena that resist simple quantification.

The Problem of Context-Dependent Harms

The measurement challenge extends to defining harm itself. Personalisation creates benefits and costs that vary dramatically based on context and individual circumstances. A recommendation algorithm that surfaces mental health resources for someone experiencing depression delivers substantial value. That same algorithm creating filter bubbles around depression-related content could worsen the condition by limiting exposure to perspectives and information that might aid recovery. The same technical system produces opposite outcomes based on subtle implementation details.

Some researchers advocate for ethical impact assessments as a framework. These assessments would require organisations to systematically evaluate potential harms before deploying personalisation systems, engaging stakeholders in the process. But who qualifies as a stakeholder? Users certainly, but which users? The teenager experiencing algorithmic radicalisation on YouTube differs fundamentally from the pensioner discovering new music on Spotify, yet both interact with personalisation systems. Their interests and vulnerabilities diverge so thoroughly that a single impact assessment could never address both adequately.

Value alignment represents another proposed approach: ensuring AI systems pursue objectives consistent with human values. But whose values? Spotify's focus on maximising listener engagement reflects certain values about music consumption, prioritising continual novelty and mood optimisation over practices like listening to entire albums intentionally. Users who share those values find the platform delightful. Users who don't may feel their listening experience has been subtly degraded in ways difficult to articulate.

The fundamental measurement problem may be that algorithmic personalisation creates highly individualised harms and benefits that resist aggregate quantification. Traditional regulatory frameworks assume harms can be identified, measured, and addressed through uniform standards. Personalisation breaks that assumption. What helps one person hurts another, and the technical systems involved operate at such scale and complexity that individual cases vanish into statistical noise.

This doesn't mean measurement is impossible, but it suggests we need fundamentally different frameworks. Rather than asking “does this personalisation system cause net harm?”, perhaps we should ask “does this system provide users with meaningful agency over how it shapes their experience?” That question shifts focus from measuring algorithmic outputs to evaluating user control, a reframing that connects directly to transparency design patterns.

Making the Invisible Visible

If meaningful consent requires genuine understanding, then transparency becomes essential infrastructure rather than optional feature. The question is how to make inherently opaque systems comprehensible without overwhelming users with technical detail they neither want nor can process.

Design Patterns for Transparency

Research published in 2024 identified several design patterns for AI transparency in personalisation contexts. Clear AI decision displays provide explanations tailored to different user expertise levels, recognising that a machine learning researcher and a casual user need fundamentally different information. Visualisation tools represent algorithmic logic through heatmaps and status breakdowns rather than raw data tables, making decision-making processes more intuitive.

Proactive explanations prove particularly effective. Rather than requiring users to seek out information about how personalisation works, systems can surface contextually relevant explanations at decision points. When Spotify creates a personalised playlist, it might briefly explain that recommendations draw from your listening history, similar users' preferences, and audio analysis. This doesn't require users to understand the technical implementation, but it clarifies the logic informing selections.

User control mechanisms represent another critical transparency pattern. The focus shifts toward explainability and user agency in AI-driven personalisation. For systems to succeed, they must provide clear explanations of AI features whilst offering users meaningful control over personalisation settings. This means not just opt-out switches that break the service, but granular controls over which data sources and algorithmic approaches inform recommendations.

Platform Approaches to Openness

Apple's approach to Private Cloud Compute demonstrates one transparency model. The company published detailed technical specifications for its server architecture, allowing independent security researchers to verify its privacy claims. Any personal data passed to the cloud gets used only for the specific AI task requested, with no retention or accessibility after completion. This represents transparency through verifiability, inviting external audit rather than simply asserting privacy protection.

Meta took a different approach with its AI transparency centre, providing users with information about how their data trains AI models and what controls they possess. Critics argue the execution fell short, with dark patterns undermining genuine transparency, but the concept illustrates growing recognition that users need visibility into personalisation systems.

Google's Responsible AI framework emphasises transparency through documentation. The company publishes model cards for its AI systems, detailing their intended uses, limitations, and performance characteristics across different demographic groups. For personalisation specifically, Google has explored approaches like “why this ad?” explanations that reveal the factors triggering particular recommendations.

The Limits of Explanation

Yet transparency faces fundamental limits. Research on explainable AI reveals that making complex machine learning models comprehensible often requires simplifications that distort how the systems actually function. Feature attribution methods identify which inputs most influenced a decision, but this obscures the non-linear interactions between features that characterise modern deep learning. Surrogate models mimic complex algorithms whilst remaining understandable, but the mimicry is imperfect by definition.

Interactive XAI offers a promising alternative. Rather than providing static explanations, these systems allow users to test and understand models dynamically. A user might ask “what would you recommend if I hadn't watched these horror films?” and receive both an answer and visibility into how that counterfactual changes the algorithmic output. This transforms transparency from passive information provision to active exploration.

Domain-specific explanations represent another frontier. Recent XAI frameworks use domain knowledge to tailor explanations to specific contexts, making results more actionable and relevant. For music recommendations, this might explain that a suggested song shares particular instrumentation or lyrical themes with tracks you've enjoyed. For news recommendations, it might highlight that an article covers developing aspects of stories you've followed.

The transparency challenge ultimately reveals a deeper tension. Users want personalisation to “just work” without requiring their attention or effort. Simultaneously, meaningful agency demands understanding and control. Design patterns that satisfy both objectives remain elusive. Too much transparency overwhelms users with complexity. Too little transparency reduces agency to theatre.

Perhaps the solution lies not in perfect transparency, but in trusted intermediaries. Just as food safety regulations allow consumers to trust restaurants without understanding microbiology, perhaps algorithmic auditing could allow users to trust personalisation systems without understanding machine learning. This requires robust regulatory frameworks and independent oversight, infrastructure that remains under development.

Meanwhile, the technical architecture of personalisation itself creates privacy implications that design patterns alone cannot resolve.

The Privacy Trade Space

When Apple announced its approach to AI personalisation at WWDC 2024, the company emphasised a fundamental architectural choice: on-device processing whenever possible, with cloud computing only for tasks exceeding device capabilities. This represents one pole in the ongoing debate about personalisation privacy tradeoffs.

On-Device vs. Cloud Processing

The advantages of on-device processing are substantial. Data never leaves the user's control, eliminating risks from transmission interception, cloud breaches, or unauthorised access. Response times improve since computation occurs locally. Users maintain complete ownership of their information. For privacy-conscious users, these benefits prove compelling.

Yet on-device processing imposes significant constraints. Mobile devices possess limited computational power compared to data centres. Training sophisticated personalisation models requires enormous datasets that individual users cannot provide. The most powerful personalisation emerges from collaborative filtering that identifies patterns across millions of users, something impossible if data remains isolated on devices.

Google's hybrid approach with Gemini Nano illustrates the tradeoffs. The smaller on-device model handles quick replies, smart transcription, and offline tasks. More complex queries route to larger models running in Google Cloud. This balances privacy for routine interactions with powerful capabilities for sophisticated tasks. Critics argue that any cloud processing creates vulnerability, whilst defenders note the approach provides substantially better privacy than pure cloud architectures whilst maintaining competitive functionality.

Privacy-Preserving Technologies

The technical landscape is evolving rapidly through privacy-preserving machine learning techniques. Federated learning allows models to train on distributed datasets without centralising the data. Each device computes model updates locally, transmitting only those updates to a central server that aggregates them into improved global models. The raw data never leaves user devices.

Differential privacy adds mathematical guarantees to this approach. By injecting carefully calibrated noise into the data or model updates, differential privacy ensures that no individual user's information can be reconstructed from the final model. Research published in 2024 demonstrated significant advances in this domain. FedADDP, an adaptive dimensional differential privacy framework, uses Fisher information matrices to distinguish between personalised parameters tailored to individual clients and global parameters consistent across all clients. Experiments showed accuracy improvements of 1.67% to 23.12% across various privacy levels and non-IID data distributions compared to conventional federated learning.

Hybrid differential privacy federated learning showcased notable accuracy enhancements whilst preserving privacy. Cross-silo federated learning with record-level personalised differential privacy employs hybrid sampling schemes with both uniform client-level sampling and non-uniform record-level sampling to accommodate varying privacy requirements.

These techniques enable what researchers describe as privacy-preserving personalisation: customised experiences without exposing individual user data. Robust models of personalised federated distillation employ adaptive hierarchical clustering strategies, generating semi-global models by grouping clients with similar data distributions whilst allowing independent training. Heterogeneous differential privacy can personalise protection according to each client's privacy budget and requirements.

The technical sophistication represents genuine progress, but practical deployment remains limited. Most consumer personalisation systems still rely on centralised data collection and processing. The reasons are partly technical (federated learning and differential privacy add complexity and computational overhead), but also economic. Centralised data provides valuable insights for product development, advertising, and business intelligence beyond personalisation. Privacy-preserving techniques constrain those uses.

Business Models and Regulatory Pressure

This reveals that privacy tradeoffs in personalisation are not purely technical decisions, but business model choices. Apple can prioritise on-device processing because it generates revenue from hardware sales and services subscriptions rather than advertising. Google's and Meta's business models depend on detailed user profiling for ad targeting, creating different incentive structures around data collection.

Regulatory pressure is shifting these dynamics. The AI Act's progressive implementation through 2027 will impose strict requirements on AI systems processing personal data, particularly those categorised as high-risk. The “consent or pay” models employed by some platforms, where users must either accept tracking or pay subscription fees, face growing regulatory scrutiny. The EU Digital Services Act, effective February 2024, explicitly bans dark patterns and requires transparency about algorithmic systems.

Yet regulation alone cannot resolve the fundamental tension. Privacy-preserving personalisation techniques remain computationally expensive and technically complex. Their widespread deployment requires investment and expertise that many organisations lack. The question is whether market competition, user demand, and regulatory requirements will collectively drive adoption, or whether privacy-preserving personalisation will remain a niche approach.

The answer may vary by domain. Healthcare applications processing sensitive medical data face strong privacy imperatives that justify technical investment. Entertainment recommendations processing viewing preferences may operate under different calculus. This suggests a future where privacy architecture varies based on data sensitivity and use context, rather than universal standards.

Building Systems Worth Trusting

The challenges explored throughout this examination (consent limitations, filter bubble effects, measurement difficulties, transparency constraints, and privacy tradeoffs) might suggest that consumer-grade AI personalisation represents an intractable problem. Yet the more optimistic interpretation recognises that we're in early days of a technology still evolving rapidly both technically and in its social implications.

Promising Developments

Several promising developments emerged in 2024 that point toward more trustworthy personalisation frameworks. Apple's workshop on human-centred machine learning emphasised ethical AI design with principles like transparency, privacy, and bias mitigation. Presenters discussed adapting AI for personalised experiences whilst safeguarding data, aligning with Apple's privacy-first stance. Google's AI Principles, established in 2018 and updated continuously, serve as a living constitution guiding responsible development, with frameworks like the Secure AI Framework for security and privacy.

Meta's collaboration with researchers to create responsible AI seminars offers a proactive strategy for teaching practitioners about ethical standards. These industry efforts, whilst partly driven by regulatory compliance and public relations considerations, demonstrate growing recognition that trust represents essential infrastructure for personalisation systems.

The shift toward explainable AI represents another positive trajectory. XAI techniques bridge the gap between model complexity and user comprehension, fostering trust amongst stakeholders whilst enabling more informed, ethical decisions. Interactive XAI methods let users test and understand models dynamically, transforming transparency from passive information provision to active exploration.

Research into algorithmic harms and fairness metrics, whilst revealing measurement challenges, is also developing more sophisticated frameworks for evaluation. Calibrated fairness approaches that balance equal opportunities with accommodation of distinct differences represent progress beyond crude equality metrics. Ethical impact assessments that engage stakeholders in evaluation processes create accountability mechanisms that pure technical metrics cannot provide.

The technical advances in privacy-preserving machine learning offer genuine paths forward. Federated learning with differential privacy can deliver meaningful personalisation whilst providing mathematical guarantees about individual privacy. As these techniques mature and deployment costs decrease, they may become standard infrastructure rather than exotic alternatives.

Beyond Technical Solutions

Yet technology alone cannot solve what are fundamentally social and political challenges about power, agency, and control. The critical question is not whether we can build personalisation systems that are technically capable of preserving privacy and providing transparency. We largely can, or soon will be able to. The question is whether we will build the regulatory frameworks, competitive dynamics, and user expectations that make such systems economically and practically viable.

This requires confronting uncomfortable realities about attention economies and data extraction. So long as digital platforms derive primary value from collecting detailed user information and maximising engagement, the incentives will push toward more intrusive personalisation, not less. Privacy-preserving alternatives succeed only when they become requirements rather than options, whether through regulation, user demand, or competitive necessity.

The consent framework embedded in regulations like GDPR and the AI Act represents important infrastructure, but consent alone proves insufficient when digital services have become essential utilities. We need complementary approaches: algorithmic auditing by independent bodies, mandatory transparency standards that go beyond current practices, interoperability requirements that reduce platform lock-in and associated consent coercion, and alternative business models that don't depend on surveillance.

Reimagining Personalisation

Perhaps most fundamentally, we need broader cultural conversation about what personalisation should optimise. Current systems largely optimise for engagement, treating user attention as the ultimate metric. But engagement proves a poor proxy for human flourishing. An algorithm that maximises the time you spend on a platform may or may not be serving your interests. Designing personalisation systems that optimise for user-defined goals rather than platform-defined metrics requires reconceptualising the entire enterprise.

What would personalisation look like if it genuinely served user agency rather than capturing attention? It might provide tools for users to define their own objectives, whether learning new perspectives, maintaining diverse information sources, or achieving specific goals. It would make its logic visible and modifiable, treating users as collaborators in the personalisation process rather than subjects of it. It would acknowledge the profound power dynamics inherent in systems that shape information access, and design countermeasures into the architecture.

Some of these ideas seem utopian given current economic realities. But they're not technically impossible, merely economically inconvenient under prevailing business models. The question is whether we collectively decide that inconvenience matters less than user autonomy.

As AI personalisation systems grow more sophisticated and ubiquitous, the stakes continue rising. These systems shape not just what we see, but how we think, what we believe, and who we become. Getting the design patterns right (balancing personalisation benefits against filter bubble harms, transparency against complexity, and privacy against functionality) represents one of the defining challenges of our technological age.

The answer won't come from technology alone, nor from regulation alone, nor from user activism alone. It requires all three, working in tension and collaboration, to build personalisation systems that genuinely serve human agency rather than merely extracting value from human attention. We know how to build systems that know us extraordinarily well. The harder challenge is building systems that use that knowledge wisely, ethically, and in service of goals we consciously choose rather than unconsciously reveal through our digital traces.

That challenge is technical, regulatory, economic, and ultimately moral. Meeting it will determine whether AI personalisation represents empowerment or exploitation, serendipity or manipulation, agency or control. The infrastructure we build now, the standards we establish, and the expectations we normalise will shape digital life for decades to come. We should build carefully.

References & Sources

AI Platforms and Personalisation Systems:

  • Netflix recommendation engine documentation and research papers from 2024 Workshop on Personalisation, Recommendation and Search (PRS)
  • Spotify BaRT system technical documentation and personalisation research covering 600M+ listeners and 100M+ track catalogue
  • TikTok algorithmic recommendation research on For You Page functionality and sound/image recognition systems
  • Google Photos “Ask Photos” feature documentation using Gemini model for natural language queries
  • Apple Intelligence and Private Cloud Compute technical specifications from WWDC 2024
  • Meta AI developments for Facebook, Instagram, and WhatsApp with over 400 million users

Regulatory Frameworks:

  • European Union General Data Protection Regulation (GDPR) Article 22 on automated decision-making
  • European Union AI Act (Regulation 2024/1689), adopted June 13, 2024, entered into force August 1, 2024
  • European Data Protection Board guidance on AI compliance with GDPR (2024)
  • EU Digital Services Act, effective February 2024, provisions on dark patterns
  • California Privacy Protection Agency enforcement advisory (September 2024) on symmetry in choice
  • European Center for Digital Rights (Noyb) allegations regarding Meta dark patterns (2024)

Academic Research:

  • “Filter Bubbles in Recommender Systems: Fact or Fallacy, A Systematic Review” (2024), arXiv:2307.01221
  • Research from ACM Conference on Fairness, Accountability, and Transparency (2024) on algorithmic harms, measurement frameworks, and AI reliance
  • “User Characteristics in Explainable AI: The Rabbit Hole of Personalisation?” Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
  • Studies on filter bubble effects showing approximately 15% increase in polarisation on platforms
  • Research on news recommender systems' tendency toward negative sentiment articles

Privacy-Preserving Technologies:

  • FedADDP (Adaptive Dimensional Differential Privacy framework for Personalised Federated Learning) research showing 1.67% to 23.12% accuracy improvements (2024)
  • Hybrid differential privacy federated learning (HDP-FL) research showing 4.22% to 9.39% accuracy enhancement for EMNIST and CIFAR-10
  • Cross-silo federated learning with record-level personalised differential privacy (rPDP-FL) from 2024 ACM SIGSAC Conference
  • PLDP-FL personalised differential privacy perturbation algorithm research
  • Research on robust models of personalised federated distillation (RMPFD) employing adaptive hierarchical clustering

Transparency and Explainability:

  • Research on explainable AI (XAI) enhancing transparency and trust in machine learning models (2024)
  • Studies on personalisation in XAI and user-centric explanations from 2024 research
  • Google's Responsible AI framework and Secure AI Framework documentation
  • Apple's 2024 Workshop on Human-Centered Machine Learning videos on ethical AI and bias mitigation
  • Meta's Responsible AI documentation and responsible use guides

Industry Analysis:

  • MIT Technology Review coverage of Apple's Private Cloud Compute architecture (June 2024)
  • Analysis of on-device AI versus cloud AI tradeoffs from multiple technology research institutions
  • Comparative studies of Apple Intelligence versus Android's hybrid AI approaches
  • Research on Google Gemini Nano for on-device processing on Pixel devices
  • Industry reports on AI-based personalisation market trends and developments for 2024-2025

Dark Patterns Research:

  • European Journal of Law and Technology study on dark patterns and enforcement (2024)
  • European Commission behavioural study on unfair commercial practices in digital environment
  • Research on dark patterns in cookie consent requests published in Journal of Digital Social Research
  • Documentation of Meta's 2024 data usage policies and opt-out mechanisms

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The artificial intelligence industry stands at a crossroads. On one side, proprietary giants like OpenAI, Google, and Anthropic guard their model weights and training methodologies with the fervour of medieval warlords protecting castle secrets. On the other, a sprawling, chaotic, surprisingly powerful open-source movement is mounting an insurgency that threatens to democratise the most transformative technology since the internet itself.

The question isn't merely academic. It's existential for the future of AI: Can community-driven open-source infrastructure genuinely rival the proprietary stacks that currently dominate production-grade artificial intelligence? And perhaps more importantly, what governance structures and business models will ensure these open alternatives remain sustainable, safe, and equitably accessible to everyone, not just Silicon Valley elites?

The answer, as it turns out, is both more complex and more hopeful than you might expect.

Open Source Closes the Gap

For years, the conventional wisdom held that open-source AI would perpetually trail behind closed alternatives. Proprietary models like GPT-4 and Claude dominated benchmarks, whilst open alternatives struggled to keep pace. That narrative has fundamentally shifted.

Meta's release of LLaMA models has catalysed a transformation in the open-source AI landscape. The numbers tell a compelling story: Meta's LLaMA family has achieved more than 1.2 billion downloads as of late 2024, with models being downloaded an average of one million times per day since the first release in February 2023. The open-source community has published over 85,000 LLaMA derivatives on Hugging Face alone, an increase of more than five times since the start of 2024.

The performance gap has narrowed dramatically. Code LLaMA with additional fine-tuning managed to beat GPT-4 in the HumanEval programming benchmark. LLaMA 2-70B and GPT-4 achieved near human-level performance of 84 per cent accuracy on fact-checking tasks. When comparing LLaMA 3.3 70B with GPT-4o, the open-source model remains highly competitive, especially when considering factors like cost, customisation, and deployment flexibility.

Mistral AI, a French startup that raised $645 million at a $6.2 billion valuation in June 2024, has demonstrated that open-source models can compete at the highest levels. Their Mixtral 8x7B model outperforms the 70 billion-parameter LLaMA 2 on most benchmarks with six times faster inference, and also outpaces OpenAI's GPT-3.5 on most metrics. Distributed under the Apache 2.0 licence, it can be used commercially for free.

The ecosystem has matured rapidly. Production-grade open-source frameworks now span every layer of the AI stack. LangChain supports both synchronous and asynchronous workflows suitable for production pipelines. SuperAGI is designed as a production-ready framework with extensibility at its core, featuring a graphical interface combined with support for multiple tools, memory systems, and APIs that enable developers to prototype and scale agents with ease.

Managed platforms are emerging to bridge the gap between open-source potential and enterprise readiness. Cake, which raised $10 million in seed funding from Google's Gradient Ventures in December 2024, integrates the various layers that constitute the AI stack into a more digestible, production-ready format suitable for business. The Finnish company Aiven offers managed open-source data infrastructure, making it easier to deploy production-grade AI systems.

When Free Isn't Actually Free

Here's where the open-source narrative gets complicated. Whilst LLaMA models are free to download, the infrastructure required to run them at production scale is anything but.

The economics are sobering. Training, fine-tuning, and running inference at scale consume expensive GPU resources that can often easily exceed any licensing fees incurred for proprietary technology. Infrastructure costs masquerade as simple compute and storage line items until you're hit with unexpected scaling requirements, with proof-of-concept setups often falling apart when handling real traffic patterns.

Developers can run inference on LLaMA 3.1 at roughly 50 per cent the cost of using closed models like GPT-4o, according to industry analysis. However, gross margins for AI companies average 50 to 60 per cent compared to 80 to 90 per cent for traditional software-as-a-service businesses, with 67 per cent of AI startups reporting that infrastructure costs are their number one constraint to growth.

The infrastructure arms race is staggering. Nvidia's data centre revenue surged by 279 per cent year-over-year, reaching $14.5 billion in the third quarter of 2023, primarily driven by demand for large language model training. Some projections suggest infrastructure spending could reach $3 trillion to $4 trillion over the next ten years.

This creates a paradox: open-source models democratise access to AI capabilities, but the infrastructure required to utilise them remains concentrated in the hands of cloud giants. Spot instances can reduce costs by 60 to 80 per cent for interruptible training workloads, but navigating this landscape requires sophisticated technical expertise.

Monetisation Models

If open-source AI infrastructure is to rival proprietary alternatives, it needs sustainable business models. The community has experimented with various approaches, with mixed results.

The Hugging Face Model

Hugging Face, valued at $4.5 billion in a Series D funding round led by Salesforce in August 2023, has pioneered a hybrid approach. Whilst strongly championing open-source AI and encouraging collaboration amongst developers, it maintains a hybrid model. The core infrastructure and some enterprise features remain proprietary, whilst the most valuable assets, the vast collection of user-contributed models and datasets, are entirely open source.

The CEO has emphasised that “finding a profitable, sustainable business model that doesn't prevent us from doing open source and sharing most of the platform for free was important for us to be able to deliver to the community.” Investors include Google, Amazon, Nvidia, AMD, Intel, IBM, and Qualcomm, demonstrating confidence that the model can scale.

The Stability AI Rollercoaster

Stability AI's journey offers both warnings and hope. The creator of Stable Diffusion combines open-source model distribution with commercial API services and enterprise licensing, releasing model weights under permissive licences whilst generating revenue through hosted API access, premium features, and enterprise support.

Without a clear revenue model initially, financial health deteriorated rapidly, mounting $100 million in debts coupled with $300 million in future obligations. However, in December 2024, new CEO Prem Akkaraju reported the company was growing at triple-digit rates and had eliminated its debt, with continued expansion expected into film, television, and large-scale enterprise integrations in 2025. The turnaround demonstrates that open-source AI companies can find sustainable revenue streams, but the path is treacherous.

The Red Hat Playbook

Red Hat's approach to open-source monetisation, refined over decades with Linux, offers a proven template. Red Hat OpenShift AI provides enterprise-grade support, lifecycle management, and intellectual property indemnification. This model works because enterprises value reliability, support, and indemnification over raw access to technology, paying substantial premiums for guaranteed uptime, professional services, and someone to call when things break.

Emerging Hybrid Models

The market is experimenting with increasingly sophisticated hybrid approaches. Consumption-based pricing has emerged as a natural fit for AI-plus-software-as-a-service products that perform work instead of merely supporting it. Hybrid models work especially well for enterprise AI APIs where customers want predictable base costs with the ability to scale token consumption based on business growth.

Some companies are experimenting with outcome-based pricing. Intercom abandoned traditional per-seat pricing for a per-resolution model, charging $0.99 per AI-resolved conversation instead of $39 per support agent, aligning revenue directly with value delivered.

Avoiding the Tragedy of the Digital Commons

For open-source AI infrastructure to succeed long-term, it requires governance structures that balance innovation with safety, inclusivity with direction, and openness with sustainability.

The Apache Way

The Apache Software Foundation employs a meritocratic governance model that fosters balanced, democratic decision-making through community consensus. As a Delaware-based membership corporation and IRS-registered 501©(3) non-profit, the ASF is governed by corporate bylaws, with membership electing a board of directors which sets corporate policy and appoints officers.

Apache projects span from the flagship Apache HTTP project to more recent initiatives encompassing AI and machine learning, big data, cloud computing, financial technology, geospatial, Internet of Things, and search. For machine learning governance specifically, Apache Atlas Type System can be used to define new types, capturing machine learning entities and processes as Atlas metadata objects, with relationships visualised in end-to-end lineage flow. This addresses key governance needs: visibility, model explainability, interpretability, and reproducibility.

EleutherAI's Grassroots Non-Profit Research

EleutherAI represents a different governance model entirely. The grassroots non-profit artificial intelligence research group formed in a Discord server in July 2020 by Connor Leahy, Sid Black, and Leo Gao to organise a replication of GPT-3. In early 2023, it formally incorporated as the EleutherAI Institute, a non-profit research institute.

Researchers from EleutherAI open-sourced GPT-NeoX-20B, a 20-billion-parameter natural language processing AI model similar to GPT-3, which was the largest open-source language model in the world at the time of its release in February 2022.

Part of EleutherAI's motivation is their belief that open access to such models is necessary for advancing research in the field. According to founder Connor Leahy, they believe “the benefits of having an open source model of this size and quality available for that research outweigh the risks.”

Gary Marcus, a cognitive scientist and noted critic of deep learning companies such as OpenAI and DeepMind, has repeatedly praised EleutherAI's dedication to open-source and transparent research. Maximilian Gahntz, a senior policy researcher at the Mozilla Foundation, applauded EleutherAI's efforts to give more researchers the ability to audit and assess AI technology.

Mozilla Common Voice

Mozilla's Common Voice project demonstrates how community governance can work for AI datasets. Common Voice is the most diverse open voice dataset in the world, a crowdsourcing project to create a free and open speech corpus. As part of their commitment to helping make voice technologies more accessible, they release a cost and copyright-free dataset of multilingual voice clips and associated text data under a CC0 licence.

The dataset has grown to a staggering 31,841 hours with 20,789 community-validated hours of speech data across 129 languages. The project is supported by volunteers who record sample sentences with a microphone and review recordings of other users.

The governance structure includes advisory committees consulted for decision-making, especially in cases of conflict. Whether or not a change is made to the dataset is decided based on a prioritisation matrix, where the cost-benefit ratio is weighed in relation to the public interest. Transparency is ensured through a community forum, a blog and the publication of decisions, creating a participatory and deliberative decision-making process overall.

Policy and Regulatory Developments

Governance doesn't exist in a vacuum. In December 2024, a report from the House Bipartisan Task Force called for federal investments in open-source AI research at the National Science Foundation, National Institute of Standards and Technology, and the Department of Energy to strengthen AI model security, governance, and privacy protections. The report emphasised taking a risk-based approach that would monitor potential harms over time whilst sustaining open development.

California introduced SB-1047 in early 2024, proposing liability measures requiring AI developers to certify their models posed no potential harm, but Governor Gavin Newsom vetoed the measure in September 2024, citing concerns that the bill's language was too imprecise and risked stifling innovation.

At the international level, the Centre for Data Innovation facilitated a dialogue on addressing risks in open-source AI with international experts at a workshop in Beijing on 10 to 11 December 2024, developing a statement on how to enhance international collaboration to improve open-source AI safety and security. At the AI Seoul Summit in May 2024, sixteen companies made a public commitment to release risk thresholds and mitigation frameworks by the next summit in France.

The Open Source Initiative and Open Future released a white paper titled “Data Governance in Open Source AI: Enabling Responsible and Systematic Access” following a global co-design process and a two-day workshop held in Paris in October 2024.

The Open Safety Question

Critics of open-source AI frequently raise safety concerns. If anyone can download and run powerful models, what prevents malicious actors from fine-tuning them for harmful purposes? The debate is fierce and far from settled.

The Safety-Through-Transparency Argument

EleutherAI and similar organisations argue that open access enables better safety research. As Connor Leahy noted, EleutherAI believes “AI safety is massively important for society to tackle today, and hope that open access to cutting edge models will allow more such research to be done on state of the art systems.”

The logic runs that closed systems create security through obscurity, which historically fails. Open systems allow the broader research community to identify vulnerabilities, test edge cases, and develop mitigation strategies. The diversity of perspectives examining open models may catch issues that homogeneous corporate teams miss.

Anthropic, which positions itself as safety-focused, takes a different approach. Incorporated as a Delaware public-benefit corporation, Anthropic brands itself as “a safety and research-focused company with the goal of building systems that people can rely on and generating research about the opportunities and risks of AI.”

Their Constitutional AI approach trains language models like Claude to be harmless and helpful without relying on extensive human feedback. Anthropic has published constitutional principles relating to avoiding harmful responses, including bias and profanity, avoiding responses that would reveal personal information, avoiding responses regarding illicit acts, avoiding manipulation, and encouraging honesty and helpfulness.

Notably, Anthropic generally doesn't publish capabilities work because they do not wish to advance the rate of AI capabilities progress, taking a cautious stance that contrasts sharply with the open-source philosophy. The company brings in over $2 billion in annualised revenue, with investors including Amazon at $8 billion, Google at $2 billion, and Menlo Ventures at $750 million.

Empirical Safety Records

The empirical evidence on safety is mixed. Open-source models have not, to date, caused catastrophic harms at a scale beyond what proprietary models have enabled. Both open and closed models can be misused for generating misinformation, creating deepfakes, or automating cyberattacks. The difference lies less in the models themselves and more in the surrounding ecosystem, moderation policies, and user education.

Safety researchers are developing open-source tools for responsible AI. Anthropic released Petri, an open-source auditing tool to accelerate AI safety research, demonstrating that even closed-model companies recognise the value of open tooling for safety evaluation.

The Global South Challenge

Perhaps the most compelling argument for open-source AI infrastructure is equitable access. Proprietary models concentrate power and capability in wealthy nations and well-funded organisations. Open-source models theoretically democratise access, but theory and practice diverge significantly. The safety debate connects directly to this challenge: if powerful AI remains locked behind proprietary walls, developing nations face not just technical barriers, but fundamental power asymmetries in shaping the technology's future.

The Promise of Democratisation

Open-source AI innovation enables collaboration across borders, allows emerging economies to avoid technological redundancy, and creates a platform for equitable participation in the AI era. Open-source approaches allow countries to avoid expensive licensing, making technology more accessible for resource-constrained environments.

Innovators across the Global South are applying AI solutions to local problems, with open-source models offering advantages in adapting to local cultures and languages whilst preventing vendor lock-in. According to industry analysis, 89 per cent of AI-using organisations incorporate open-source tools in some capacity, driven largely by cost considerations, with 75 per cent of small businesses turning to open-source AI for cost-effective solutions.

The Centre for Strategic and International Studies notes that open-source models create opportunities for AI innovation in the Global South amid geostrategic competition, potentially reducing dependence on technology from major powers.

The Infrastructure Reality

Despite these advantages, significant barriers remain. In the Global South, access to powerful GPUs and fast, stable internet is limited, leading some observers to call the trend “algorithmic colonialism.”

The statistics are stark. According to research on African contexts, only 1 per cent of Zindi Africa data scientists have on-premises access to GPUs, whilst 4 per cent pay for cloud access worth $1,000 per month. Despite apparent progress, the resources required to utilise open-access AI are still not within arm's reach in many African contexts.

The paradox is cruel: open-source models are freely available, but the computational infrastructure to use them remains concentrated in data centres controlled by American and Chinese tech giants. Downloading LLaMA costs nothing; spinning up enough GPU instances to fine-tune it for a local language costs thousands of dollars per hour.

Bridging the Gap

Some initiatives attempt to bridge this divide. Open-source tools for managing GPU infrastructure include DeepOps, an open-source toolkit designed for deploying and managing GPU clusters that automates the deployment of Kubernetes and Slurm clusters with GPU support. Kubeflow, an open-source machine learning toolkit for Kubernetes, streamlines end-to-end machine learning workflows with GPU acceleration.

Spot instances and per-second billing from some cloud providers make short training runs, inference jobs, and bursty workloads more cost-efficient, potentially lowering barriers. However, navigating these options requires technical sophistication that many organisations in developing countries lack.

International collaboration efforts are emerging. The December 2024 workshop in Beijing brought together international experts to develop frameworks for enhancing collaboration on open-source AI safety and security, potentially creating more equitable participation structures.

The Production-Grade Reality Check

For all the promise of open-source AI, the question remains whether it can truly match proprietary alternatives for production deployments at enterprise scale.

Where Open Source Excels

Open-source infrastructure demonstrably excels in several domains. Customisation and control allow organisations to fine-tune models for specific use cases, languages, or domains without being constrained by API limitations. Companies like Spotify use LLaMA to help deliver contextualised recommendations to boost artist discovery, combining LLaMA's broad world knowledge with Spotify's expertise in audio content. LinkedIn found that LLaMA achieved comparable or better quality compared to state-of-the-art commercial foundational models at significantly lower costs and latencies.

Cost optimisation at scale becomes possible when organisations have the expertise to manage infrastructure efficiently. Whilst upfront costs are higher, amortised over millions of API calls, self-hosted open-source models can be substantially cheaper than proprietary alternatives.

Data sovereignty and privacy concerns drive many organisations to prefer on-premises or private cloud deployments of open-source models, avoiding the need to send sensitive data to third-party APIs. This is particularly important for healthcare, finance, and government applications.

Where Proprietary Holds Edges

Proprietary platforms maintain advantages in specific areas. Frontier capabilities often appear first in closed models. GPT-4 generally outperforms LLaMA 3 70B across benchmarks, particularly in areas like common knowledge and grade school maths, logical reasoning and code generation for certain tasks.

Ease of use and integration matter enormously for organisations without deep AI expertise. Proprietary APIs offer simple integration, comprehensive documentation, and managed services that reduce operational overhead. According to industry surveys, 72 per cent of enterprises use an API to access their models, with over half using models hosted by their cloud service provider.

Reliability and support carry weight in production environments. Enterprise contracts with proprietary vendors typically include service-level agreements, guaranteed uptime, professional support, and liability protection that open-source alternatives struggle to match without additional commercial layers.

The Hybrid Future

The emerging pattern suggests that the future isn't binary. Global enterprise spending on AI applications has increased eightfold over the last year to close to $5 billion, though it still represents less than 1 per cent of total software application spending. Organisations increasingly adopt hybrid strategies: proprietary APIs for tasks requiring frontier capabilities or rapid deployment, and open-source infrastructure for customised, cost-sensitive, or privacy-critical applications.

The Sustainability Question

Can open-source AI infrastructure sustain itself long-term? The track record of open-source software offers both encouragement and caution.

Learning from Linux

Linux transformed from a hobbyist project to the backbone of the internet, cloud computing, and Android. The success stemmed from robust governance through the Linux Foundation, sustainable funding through corporate sponsorships, and clear value propositions for both individual contributors and corporate backers.

The Linux model demonstrates that open-source infrastructure can not only survive but thrive, becoming more robust and ubiquitous than proprietary alternatives. However, Linux benefited from timing, network effects, and the relatively lower costs of software development compared to training frontier AI models.

The AI Sustainability Challenge

AI infrastructure faces unique sustainability challenges. The computational costs of training large models create barriers that software development doesn't face. A talented developer can contribute to Linux with a laptop and internet connection. Contributing to frontier AI model development requires access to GPU clusters costing millions of dollars.

This asymmetry concentrates power in organisations with substantial resources, whether academic institutions, well-funded non-profits like EleutherAI, or companies like Meta and Mistral AI that have raised hundreds of millions in venture funding.

Funding Models That Work

Several funding models show promise for sustaining open-source AI:

Corporate-backed open source, exemplified by Meta's LLaMA releases, allows companies to commoditise complementary goods whilst building ecosystems around their platforms. Mark Zuckerberg positioned LLaMA 3.1 as transformative, stating “I believe the Llama 3.1 release will be an inflection point in the industry where most developers begin to primarily use open source.”

Academic and research institution leadership, demonstrated by EleutherAI and university labs, sustains fundamental research that may not have immediate commercial applications but advances the field.

Foundation and non-profit models, like the Apache Software Foundation and Mozilla Foundation, provide neutral governance and long-term stewardship independent of any single company's interests.

Commercial open-source companies like Hugging Face, Mistral AI, and Stability AI develop sustainable businesses whilst contributing back to the commons, though balancing commercial imperatives with community values remains challenging.

Where We Stand

So can community-driven open-source infrastructure rival proprietary stacks for production-grade AI? The evidence suggests a nuanced answer: yes, but with important caveats.

Open-source AI has demonstrably closed the performance gap for many applications. Models like LLaMA 3.3 70B and Mixtral 8x7B compete with or exceed GPT-3.5 and approach GPT-4 in various benchmarks. For organisations with appropriate expertise and infrastructure, open-source solutions offer compelling advantages in cost, customisation, privacy, and strategic flexibility.

However, the infrastructure requirements create a two-tiered system. Well-resourced organisations with technical talent can leverage open-source AI effectively, potentially at lower long-term costs than proprietary alternatives. Smaller organisations, those in developing countries, or teams without deep machine learning expertise face steeper barriers.

Governance and business models are evolving rapidly. Hybrid approaches combining open-source model weights with commercial services, support, and hosting show promise for sustainability. Foundation-based governance like Apache and community-driven models like Mozilla Common Voice demonstrate paths toward accountability and inclusivity.

Safety remains an active debate rather than a settled question. Both open and closed approaches carry risks and benefits. The empirical record suggests that open-source models haven't created catastrophic harms beyond those possible with proprietary alternatives, whilst potentially enabling broader safety research.

Equitable access requires more than open model weights. It demands investments in computational infrastructure, education, and capacity building in underserved regions. Without addressing these bottlenecks, open-source AI risks being open in name only.

The future likely involves coexistence and hybridisation rather than the triumph of one paradigm over another. Different use cases, organisational contexts, and regulatory environments will favour different approaches. The vibrant competition between open and closed models benefits everyone, driving innovation, reducing costs, and expanding capabilities faster than either approach could alone.

Meta's strategic bet on open source, Mistral AI's rapid ascent, Hugging Face's ecosystem play, and the steady contribution of organisations like EleutherAI and Mozilla collectively demonstrate that open-source AI infrastructure can absolutely rival proprietary alternatives, provided the community solves the intertwined challenges of governance, sustainability, safety, and genuine equitable access.

The insurgency isn't just mounting a challenge. In many ways, it's already won specific battles, claiming significant territory in the AI landscape. Whether the ultimate victory favours openness, closure, or some hybrid configuration will depend on choices made by developers, companies, policymakers, and communities over the coming years.

One thing is certain: the community-driven open-source movement has irrevocably changed the game, ensuring that artificial intelligence won't be controlled exclusively by a handful of corporations. Whether that partial accessibility evolves into truly universal access remains the defining challenge of the next phase of the AI revolution.

References and Sources

  1. GitHub Blog. (2024). “2024 GitHub Accelerator: Meet the 11 projects shaping open source AI.” Available at: https://github.blog/news-insights/company-news/2024-github-accelerator-meet-the-11-projects-shaping-open-source-ai/

  2. TechCrunch. (2024). “Google's Gradient backs Cake, a managed open source AI infrastructure platform.” Available at: https://techcrunch.com/2024/12/04/googles-gradient-backs-cake-a-managed-open-source-ai-infrastructure-platform/

  3. Acquired.fm. (2024). “Building the Open Source AI Revolution with Hugging Face CEO, Clem Delangue.” Available at: https://www.acquired.fm/episodes/building-the-open-source-ai-revolution-with-hugging-face-ceo-clem-delangue

  4. Meta AI Blog. (2024). “The future of AI: Built with Llama.” Available at: https://ai.meta.com/blog/future-of-ai-built-with-llama/

  5. Meta AI Blog. (2024). “Introducing Llama 3.1: Our most capable models to date.” Available at: https://ai.meta.com/blog/meta-llama-3-1/

  6. Meta AI Blog. (2024). “With 10x growth since 2023, Llama is the leading engine of AI innovation.” Available at: https://ai.meta.com/blog/llama-usage-doubled-may-through-july-2024/

  7. About Meta. (2024). “Open Source AI is the Path Forward.” Available at: https://about.fb.com/news/2024/07/open-source-ai-is-the-path-forward/

  8. McKinsey & Company. (2024). “Evolving models and monetization strategies in the new AI SaaS era.” Available at: https://www.mckinsey.com/industries/technology-media-and-telecommunications/our-insights/upgrading-software-business-models-to-thrive-in-the-ai-era

  9. Andreessen Horowitz. (2024). “16 Changes to the Way Enterprises Are Building and Buying Generative AI.” Available at: https://a16z.com/generative-ai-enterprise-2024/

  10. FourWeekMBA. “How Does Stability AI Make Money? Stability AI Business Model Analysis.” Available at: https://fourweekmba.com/how-does-stability-ai-make-money/

  11. VentureBeat. (2024). “Stable Diffusion creator Stability AI raises $101M funding to accelerate open-source AI.” Available at: https://venturebeat.com/ai/stable-diffusion-creator-stability-ai-raises-101m-funding-to-accelerate-open-source-ai

  12. AI Media House. (2024). “Stability AI Fights Back from Collapse to Dominate Generative AI Again.” Available at: https://aimmediahouse.com/ai-startups/stability-ai-fights-back-from-collapse-to-dominate-generative-ai-again

  13. Wikipedia. “Mistral AI.” Available at: https://en.wikipedia.org/wiki/Mistral_AI

  14. IBM Newsroom. (2024). “IBM Announces Availability of Open-Source Mistral AI Model on watsonx.” Available at: https://newsroom.ibm.com/2024-02-29-IBM-Announces-Availability-of-Open-Source-Mistral-AI-Model-on-watsonx

  15. EleutherAI Blog. (2022). “Announcing GPT-NeoX-20B.” Available at: https://blog.eleuther.ai/announcing-20b/

  16. Wikipedia. “EleutherAI.” Available at: https://en.wikipedia.org/wiki/EleutherAI

  17. InfoQ. (2022). “EleutherAI Open-Sources 20 Billion Parameter AI Language Model GPT-NeoX-20B.” Available at: https://www.infoq.com/news/2022/04/eleutherai-gpt-neox/

  18. Red Hat. “Red Hat OpenShift AI.” Available at: https://www.redhat.com/en/products/ai/openshift-ai

  19. CSIS. (2024). “An Open Door: AI Innovation in the Global South amid Geostrategic Competition.” Available at: https://www.csis.org/analysis/open-door-ai-innovation-global-south-amid-geostrategic-competition

  20. AI for Developing Countries Forum. “AI Patents: Open Source vs. Closed Source – Strategic Choices for Developing Countries.” Available at: https://aifod.org/ai-patents-open-source-vs-closed-source-strategic-choices-for-developing-countries/

  21. Linux Foundation. “Open Source AI Is Powering a More Inclusive Digital Economy across APEC Economies.” Available at: https://www.linuxfoundation.org/blog/open-source-ai-is-powering-a-more-inclusive-digital-economy-across-apec-economies

  22. Stanford Social Innovation Review. “How to Make AI Equitable in the Global South.” Available at: https://ssir.org/articles/entry/equitable-ai-in-the-global-south

  23. Brookings Institution. “Is open-access AI the great safety equalizer for African countries?” Available at: https://www.brookings.edu/articles/is-open-access-ai-the-great-safety-equalizer-for-african-countries/

  24. Apache Software Foundation. “A Primer on ASF Governance.” Available at: https://www.apache.org/foundation/governance/

  25. Mozilla Foundation. “Common Voice.” Available at: https://www.mozillafoundation.org/en/common-voice/

  26. Mozilla Foundation. (2024). “Common Voice 18 Dataset Release.” Available at: https://www.mozillafoundation.org/en/blog/common-voice-18-dataset-release/

  27. Wikipedia. “Common Voice.” Available at: https://en.wikipedia.org/wiki/Common_Voice

  28. Linux Insider. (2024). “Open-Source Experts' 2024 Outlook for AI, Security, Sustainability.” Available at: https://www.linuxinsider.com/story/open-source-experts-2024-outlook-for-ai-security-sustainability-177250.html

  29. Center for Data Innovation. (2024). “Statement on Enhancing International Collaboration on Open-Source AI Safety.” Available at: https://datainnovation.org/2024/12/statement-on-enhancing-international-collaboration-on-open-source-ai-safety/

  30. Open Source Initiative. (2024). “Data Governance in Open Source AI.” Available at: https://opensource.org/data-governance-open-source-ai

  31. Wikipedia. “Anthropic.” Available at: https://en.wikipedia.org/wiki/Anthropic

  32. Anthropic. “Core Views on AI Safety: When, Why, What, and How.” Available at: https://www.anthropic.com/news/core-views-on-ai-safety

  33. Anthropic. “Petri: An open-source auditing tool to accelerate AI safety research.” Available at: https://www.anthropic.com/research/petri-open-source-auditing

  34. TechTarget. (2024). “Free isn't cheap: How open source AI drains compute budgets.” Available at: https://www.techtarget.com/searchcio/feature/How-open-source-AI-drains-compute-budgets

  35. Neev Cloud. “Open Source Tools for Managing Cloud GPU Infrastructure.” Available at: https://blog.neevcloud.com/open-source-tools-for-managing-cloud-gpu-infrastructure

  36. RunPod. (2025). “Top 12 Cloud GPU Providers for AI and Machine Learning in 2025.” Available at: https://www.runpod.io/articles/guides/top-cloud-gpu-providers

  37. CIO Dive. “Nvidia CEO praises open-source AI as enterprises deploy GPU servers.” Available at: https://www.ciodive.com/news/nvidia-revenue-gpu-servers-open-source-ai/758897/

  38. Netguru. “Llama vs GPT: Comparing Open-Source Versus Closed-Source AI Development.” Available at: https://www.netguru.com/blog/gpt-4-vs-llama-2

  39. Codesmith. “Meta Llama 2 vs. OpenAI GPT-4: A Comparative Analysis of an Open Source vs Proprietary LLM.” Available at: https://www.codesmith.io/blog/meta-llama-2-vs-openai-gpt-4-a-comparative-analysis-of-an-open-source-vs-proprietary-llm

  40. Prompt Engineering. “How Does Llama-2 Compare to GPT-4/3.5 and Other AI Language Models.” Available at: https://promptengineering.org/how-does-llama-2-compare-to-gpt-and-other-ai-language-models/

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

When Match Group CEO Spencer Rascoff announced Tinder's newest feature in November 2025, the pitch was seductive: an AI assistant called Chemistry that would get to know you through questions and, crucially, by analysing your camera roll. The promise was better matches through deeper personalisation. The reality was something far more invasive.

Tinder, suffering through nine consecutive quarters of declining paid subscribers, positioned Chemistry as a “major pillar” of its 2026 product experience. The feature launched first in New Zealand and Australia, two testing grounds far enough from regulatory scrutiny to gauge user acceptance. What Rascoff didn't emphasise was the extraordinary trade users would make: handing over perhaps the most intimate repository of personal data on their devices in exchange for algorithmic matchmaking.

The camera roll represents a unique threat surface. Unlike profile photos carefully curated for public consumption, camera rolls contain unfiltered reality. Screenshots of medical prescriptions. Photos of children. Images from inside homes revealing addresses. Pictures of credit cards, passports, and other identity documents. Intimate moments never meant for algorithmic eyes. When users grant an app permission to access their camera roll, they're not just sharing data, they're surrendering context, relationships, and vulnerability.

This development arrives at a precarious moment for dating app privacy. Mozilla Foundation's 2024 review of 25 popular dating apps found that 22 earned its “Privacy Not Included” warning label, a deterioration from its 2021 assessment. The research revealed that 80 per cent of dating apps may share or sell user information for advertising purposes, whilst 52 per cent had experienced a data breach, leak, or hack in the past three years. Dating apps, Mozilla concluded, had become worse for privacy than nearly any other technology category.

The question now facing millions of users, regulators, and technologists is stark: can AI-powered personalisation in dating apps ever be reconciled with meaningful privacy protections, or has the industry's data hunger made surveillance an inescapable feature of modern romance?

The Anatomy of Camera Roll Analysis

To understand the privacy implications, we must first examine what AI systems can extract from camera roll images. When Tinder's Chemistry feature accesses your photos, the AI doesn't simply count how many pictures feature hiking or concerts. Modern computer vision systems employ sophisticated neural networks capable of extraordinarily granular analysis.

These systems can identify faces and match them across images, creating social graphs of who appears in your life and how frequently. They can read text in screenshots, extracting everything from bank balances to private messages. They can geolocate photos by analysing visual landmarks, shadows, and metadata. They can infer socioeconomic status from clothing, home furnishings, and travel destinations. They can detect brand preferences, political affiliations, health conditions, and religious practices.

The technical capability extends further. Facial analysis algorithms can assess emotional states across images, building psychological profiles based on when and where you appear happy, stressed, or contemplative. Pattern recognition can identify routines, favourite locations, and social circles. Even images you've deleted may persist in cloud backups or were already transmitted before deletion.

Match Group emphasises that Chemistry will only access camera rolls “with permission”, but this framing obscures the power dynamic at play. When a platform experiencing subscriber decline positions a feature as essential for competitive matching, and when the broader dating ecosystem moves toward AI personalisation, individual consent becomes functionally coercive. Users who decline may find themselves algorithmically disadvantaged, receiving fewer matches or lower-quality recommendations. The “choice” to share becomes illusory.

The technical architecture compounds these concerns. Whilst Tinder has not publicly detailed Chemistry's implementation, the industry standard remains cloud-based processing. This means camera roll images, or features extracted from them, likely transmit to Match Group servers for analysis. Once there, they enter a murky ecosystem of data retention, sharing, and potential monetisation that privacy policies describe in deliberately vague language.

A Catalogue of Harms

The theoretical risks of camera roll access become visceral when examined through the lens of documented incidents. The dating app industry's track record provides a grim preview of what can go wrong.

In 2023, security researchers discovered that five dating apps, BDSM People, Chica, Pink, Brish, and Translove, had exposed over 1.5 million private and sexually explicit images in cloud storage buckets without password protection. The images belonged to approximately 900,000 users who believed their intimate photos were secured. The breach created immediate blackmail and extortion risks. For users in countries where homosexuality or non-traditional relationships carry legal penalties, the exposure represented a potential death sentence.

The Tea dating app, marketed as a safety-focused platform for women to anonymously review men, suffered a data breach that exposed tens of thousands of user pictures and personal information. The incident spawned a class-action lawsuit and resulted in Apple removing the app from its store. The irony was brutal: an app promising safety became a vector for harm.

Grindr's 2018 revelation that it had shared users' HIV status with third-party analytics firms demonstrated how “metadata” can carry devastating consequences. The dating app for LGBTQ users had transmitted highly sensitive health information without explicit consent, putting users at risk of discrimination, stigmatisation, and in some jurisdictions, criminal prosecution.

Bumble faced a £32 million settlement in 2024 over allegations it collected biometric data from facial recognition in profile photos without proper user consent, violating privacy regulations. The case highlighted how even seemingly benign features, identity verification through selfies, can create massive biometric databases with serious privacy implications.

These incidents share common threads: inadequate security protecting highly sensitive data, consent processes that failed to convey actual risks, and downstream harms extending far beyond mere privacy violations into physical safety, legal jeopardy, and psychological trauma.

Camera roll access amplifies every one of these risks. A breach exposing profile photos is catastrophic; a breach exposing unfiltered camera rolls would be civilisational. The images contain not just users' own intimacy but collateral surveillance of everyone who appears in their photos: friends, family, colleagues, children. The blast radius of a camera roll breach extends across entire social networks.

The Regulatory Maze

Privacy regulations have struggled to keep pace with dating apps' data practices, let alone AI-powered camera roll analysis. The patchwork of laws creates uneven protections that companies can exploit through jurisdiction shopping.

The European Union's General Data Protection Regulation (GDPR) establishes the strictest requirements. Under GDPR, consent must be freely given, specific, informed, and unambiguous. For camera roll access, this means apps must clearly explain what they'll analyse, how they'll use the results, where the data goes, and for how long it's retained. Consent cannot be bundled; users must be able to refuse camera roll access whilst still using the app's core functions.

GDPR Article 9 designates certain categories as “special” personal data requiring extra protection: racial or ethnic origin, political opinions, religious beliefs, sexual orientation, and biometric data for identification purposes. Dating apps routinely collect most of these categories, and camera roll analysis can reveal all of them. Processing special category data requires explicit consent and legitimate purpose, not merely the desire for better recommendations.

The regulation has teeth. Norway's Data Protection Authority fined Grindr €9.63 million in 2021 for sharing user data with advertising partners without valid consent. The authority found that Grindr's privacy policy was insufficiently specific and that requiring users to accept data sharing to use the app invalidated consent. The decision, supported by noyb (None of Your Business), the European privacy organisation founded by privacy advocate Max Schrems, set an important precedent: dating apps cannot make basic service access conditional on accepting invasive data practices.

Ireland's Data Protection Commission launched a formal investigation into Tinder's data processing practices in 2020, examining transparency and compliance with data subject rights requests. The probe followed a journalist's GDPR data request that returned 800 pages including her complete swipe history, all matches, Instagram photos, Facebook likes, and precise physical locations whenever she was using the app. The disclosure revealed surveillance far exceeding what Tinder's privacy policy suggested.

In the United States, Illinois' Biometric Information Privacy Act (BIPA) has emerged as the most significant privacy protection. Passed unanimously in 2008, BIPA prohibits collecting biometric data, including facial geometry, without written informed consent specifying what's being collected, why, and for how long. Violations carry statutory damages of $1,000 per negligent violation and $5,000 per intentional or reckless violation.

BIPA's private right of action has spawned numerous lawsuits against dating apps. Match Group properties including Tinder and OkCupid, along with Bumble and Hinge, have faced allegations that their identity verification features, which analyse selfie video to extract facial geometry, violate BIPA by collecting biometric data without proper consent. The cases highlight a critical gap: features marketed as safety measures (preventing catfishing) create enormous biometric databases subject to breach, abuse, and unauthorised surveillance.

California's Consumer Privacy Act (CCPA) provides broader privacy rights but treats biometric information the same as other personal data. The act requires disclosure of data collection, enables deletion requests, and permits opting out of data sales, but its private right of action is limited to data breaches, not ongoing privacy violations.

This regulatory fragmentation creates perverse incentives. Apps can beta test invasive features in jurisdictions with weak privacy laws, Australia and New Zealand for Tinder's Chemistry feature, before expanding to more regulated markets. They can structure corporate entities to fall under lenient data protection authorities' oversight. They can craft privacy policies that technically comply with regulations whilst remaining functionally incomprehensible to users.

The Promise and Reality of Technical Safeguards

The privacy disaster unfolding in dating apps isn't technologically inevitable. Robust technical safeguards exist that could enable AI personalisation whilst dramatically reducing privacy risks. The problem is economic incentive, not technical capability.

On-device processing represents the gold standard for privacy-preserving AI. Rather than transmitting camera roll images or extracted features to company servers, the AI model runs locally on users' devices. Analysis happens entirely on the phone, and only high-level preferences or match criteria, not raw data, transmit to the service. Apple's Photos app demonstrates this approach, analysing faces, objects, and scenes entirely on-device without Apple ever accessing the images.

For dating apps, on-device processing could work like this: the AI analyses camera roll images locally, identifying interests, activities, and preferences. It generates an encrypted interest profile vector, essentially a mathematical representation of preferences, that uploads to the matching service. The matching algorithm compares vectors between users without accessing the underlying images. If two users' vectors indicate compatible interests, they match, but the dating app never sees that User A's profile came from hiking photos whilst User B's came from rock climbing images.

The technical challenges are real but surmountable. On-device AI requires efficient models that can run on smartphone hardware without excessive battery drain. Apple's neural engine and Google's tensor processing units provide dedicated hardware for exactly this purpose. The models must be sophisticated enough to extract meaningful signals from diverse images whilst remaining compact enough for mobile deployment.

Federated learning offers another privacy-preserving approach. Instead of centralising user data, the AI model trains across users' devices without raw data ever leaving those devices. Each device trains a local model on the user's camera roll, then uploads only the model updates, not the data itself, to a central server. The server aggregates updates from many users to improve the global model, which redistributes to all devices. Individual training data remains private.

Google has deployed federated learning for features like Smart Text Selection and keyboard predictions. The approach could enable dating apps to improve matching algorithms based on collective patterns whilst protecting individual privacy. If thousands of users' local models learn that certain photo characteristics correlate with successful matches, the global model captures this pattern without any central database of camera roll images.

Differential privacy provides mathematical guarantees against reidentification. The technique adds carefully calibrated “noise” to data or model outputs, ensuring that learning about aggregate patterns doesn't reveal individual information. Dating apps could use differential privacy to learn that users interested in outdoor activities often match successfully, without being able to determine whether any specific user's camera roll contains hiking photos.

End-to-end encryption (E2EE) should be table stakes for any intimate communication platform, yet many dating apps still transmit messages without E2EE. Signal's protocol, widely regarded as the gold standard, ensures that only conversation participants can read messages, not the service provider. Dating apps could implement E2EE for messages whilst still enabling AI analysis of user-generated content through on-device processing before encryption.

Homomorphic encryption, whilst computationally expensive, enables computation on encrypted data. A dating app could receive encrypted camera roll features, perform matching calculations on the encrypted data, and return encrypted results, all without ever decrypting the actual features. The technology remains mostly theoretical for consumer applications due to performance constraints, but it represents the ultimate technical privacy safeguard.

The critical question is: if these technologies exist, why aren't dating apps using them?

The answer is uncomfortable. On-device processing prevents data collection that feeds advertising and analytics platforms. Federated learning can't create the detailed user profiles that drive targeted marketing. Differential privacy's noise prevents the kind of granular personalisation that engagement metrics optimise for. E2EE blocks the content moderation and “safety” features that companies use to justify broad data access.

Current dating app business models depend on data extraction. Match Group's portfolio of 45 apps shares data across the ecosystem and with the parent company for advertising purposes. When Bumble faced scrutiny over sharing data with OpenAI, the questions centred on transparency, not whether data sharing should occur at all. The entire infrastructure assumes that user data is an asset to monetise, not a liability to minimise.

Technical safeguards exist to flip this model. Apple's Private Click Measurement demonstrates that advertising attribution can work with strong privacy protections. Signal proves that E2EE messaging can scale. Google's federated learning shows that model improvement doesn't require centralised data collection. What's missing is regulatory pressure sufficient to overcome the economic incentive to collect everything.

Perhaps no aspect of dating app privacy failures is more frustrating than consent mechanisms that technically comply with regulations whilst utterly failing to achieve meaningful informed consent.

When Tinder prompts users to grant camera roll access for Chemistry, the flow likely resembles standard iOS patterns: the app requests the permission, the operating system displays a dialogue box, and the user taps “Allow” or “Don't Allow”. This interaction technically satisfies many regulatory requirements but provides no meaningful understanding of the consequences.

The Electronic Frontier Foundation, through director of cybersecurity Eva Galperin's work on intimate partner surveillance, has documented how “consent” can be coerced or manufactured in contexts with power imbalances. Whilst Galperin's focus has been stalkerware, domestic abuse monitoring software marketed to partners and parents, the dynamics apply to dating apps as well.

Consider the user experience: you've joined Tinder hoping to find dates or relationships. The app announces Chemistry, framing it as revolutionary technology that will transform your matching success. It suggests that other users are adopting it, implying you'll be disadvantaged if you don't. The permission dialogue appears, asking simply whether Tinder can access your photos. You have seconds to decide.

What information do you have to make this choice? The privacy policy, a 15,000-word legal document, is inaccessible at the moment of decision. The request doesn't specify which photos will be analysed, what features will be extracted, where the data will be stored, who might access it, how long it will be retained, whether you can delete it, or what happens if there's a breach. You don't know if the analysis is local or cloud-based. You don't know if extracted features will train AI models or be shared with partners.

You see a dialogue box asking permission to access photos. Nothing more.

This isn't informed consent. It's security theatre's evil twin: consent theatre.

Genuine informed consent for camera roll access would require:

Granular Control: Users should specify which photos the app can access, not grant blanket library permission. iOS's photo picker API enables this, allowing users to select specific images. Dating apps requesting full library access when limited selection suffices should raise immediate red flags.

Temporal Limits: Permissions should expire. Camera roll access granted in February shouldn't persist indefinitely. Users should periodically reconfirm, ideally every 30 to 90 days, with clear statistics about what was accessed.

Access Logs: Complete transparency about what was analysed. Every time the app accesses the camera roll, users should receive notification and be able to view exactly which images were processed and what was extracted.

Processing Clarity: Clear, specific explanation of whether analysis is on-device or cloud-based. If cloud-based, exactly what data transmits, how it's encrypted, where it's stored, and when it's deleted.

Purpose Limitation: Explicit commitments that camera roll data will only be used for the stated purpose, matching personalisation, and never for advertising, analytics, training general AI models, or sharing with third parties.

Opt-Out Parity: Crucial assurance that declining camera roll access won't result in algorithmic penalty. Users who don't share this data should receive equivalent match quality based on other signals.

Revocation: Simple, immediate ability to revoke permission and have all collected data deleted, not just anonymised or de-identified, but completely purged from all systems.

Current consent mechanisms provide essentially none of this. They satisfy legal minimums whilst ensuring users remain ignorant of the actual privacy trade.

GDPR's requirement that consent be “freely given” should prohibit making app functionality contingent on accepting invasive data practices, yet the line between core functionality and optional features remains contested. Is AI personalisation a core feature or an enhancement? Can apps argue that users who decline camera roll access can still use the service, just with degraded matching quality?

Regulatory guidance remains vague. The EU's Article 29 Working Party guidelines state that consent isn't free if users experience detriment for refusing, but “detriment” is undefined. Receiving fewer or lower-quality matches might constitute detriment, or might be framed as natural consequence of providing less information.

The burden shouldn't fall on users to navigate these ambiguities. Privacy-by-default should be the presumption, with enhanced data collection requiring clear, specific, revocable opt-in. The current model inverts this: maximal data collection is default, and opting out requires navigating labyrinthine settings if it's possible at all.

Transparency Failures

Dating apps' transparency problems extend beyond consent to encompass every aspect of how they handle data. Unlike social media platforms or even Uber, which publishes safety transparency reports, no major dating app publishes meaningful transparency documentation.

This absence is conspicuous and deliberate. What transparency would reveal would be uncomfortable:

Data Retention: How long does Tinder keep your camera roll data after you delete the app? After you delete your account? Privacy policies rarely specify retention periods, using vague language like “as long as necessary” or “in accordance with legal requirements”. Users deserve specific timeframes: 30 days, 90 days, one year.

Access Logs: Who within the company can access user data? For what purposes? With what oversight? Dating apps employ thousands of people across engineering, customer support, trust and safety, and analytics teams. Privacy policies rarely explain internal access controls.

Third-Party Sharing: The full list of partners receiving user data remains obscure. Privacy policies mention “service providers” and “business partners” without naming them or specifying exactly what data each receives. Mozilla's research found that tracing the full data pipeline from dating apps to end recipients was nearly impossible due to deliberately opaque disclosure.

AI Training: Whether user data trains AI models, and if so, how users' information might surface in model outputs, receives minimal explanation. As Bumble faced criticism over sharing data with OpenAI, the fundamental question was not just whether sharing occurred but whether users understood their photos might help train large language models.

Breach Notifications: When security incidents occur, apps have varied disclosure standards. Some notify affected users promptly with detailed incident descriptions. Others delay notification, provide minimal detail, or emphasise that “no evidence of misuse” was found rather than acknowledging the exposure. Given that 52 per cent of dating apps have experienced breaches in the past three years, transparency here is critical.

Government Requests: How frequently do law enforcement and intelligence agencies request user data? What percentage of requests do apps comply with? What data gets shared? Tech companies publish transparency reports detailing government demands; dating apps don't.

This opacity isn't accidental. Transparency would reveal practices users would find objectionable, enabling informed choice. The business model depends on information asymmetry.

Mozilla Foundation's Privacy Not Included methodology provides a template for what transparency should look like. The organisation evaluates products against five minimum security standards: encryption, automatic security updates, strong password requirements, vulnerability management, and accessible privacy policies. For dating apps, 88 per cent failed to meet these basic criteria.

The absence of transparency creates accountability vacuums. When users don't know what data is collected, how it's used, or who it's shared with, they cannot assess risks or make informed choices. When regulators lack visibility into data practices, enforcement becomes reactive rather than proactive. When researchers cannot examine systems, identifying harms requires waiting for breaches or whistleblowers.

Civil society organisations have attempted to fill this gap. The Electronic Frontier Foundation's dating app privacy guidance recommends users create separate email accounts, use unique passwords, limit personal information sharing, and regularly audit privacy settings. Whilst valuable, this advice shifts responsibility to users who lack power to compel genuine transparency.

Real transparency would be transformative. Imagine dating apps publishing quarterly reports detailing: number of users, data collection categories, retention periods, third-party sharing arrangements, breach incidents, government requests, AI model training practices, and independent privacy audits. Such disclosure would enable meaningful comparison between platforms, inform regulatory oversight, and create competitive pressure for privacy protection.

The question is whether transparency will come voluntarily or require regulatory mandate. Given the industry's trajectory, the answer seems clear.

Downstream Harms Beyond Privacy

Camera roll surveillance in dating apps creates harms extending far beyond traditional privacy violations. These downstream effects often remain invisible until catastrophic incidents bring them into focus.

Intimate Partner Violence: Eva Galperin's work on stalkerware demonstrates how technology enables coercive control. Dating apps with camera roll access create new vectors for abuse. An abusive partner who initially met the victim on a dating app might demand access to the victim's account to “prove” fidelity. With camera roll access granted, the abuser can monitor the victim's movements, relationships, and activities. The victim may not even realise this surveillance is occurring. Apps should implement account security measures detecting unusual access patterns and provide resources for intimate partner violence survivors, but few do.

Discrimination: AI systems trained on biased data perpetuate and amplify discrimination. Camera roll analysis could infer protected characteristics like race, religion, or sexual orientation, then use these for matching in ways that violate anti-discrimination laws. Worse, the discrimination is invisible. Users receiving fewer matches have no way to know whether algorithms downranked them based on inferred characteristics. The opacity of recommendation systems makes proving discrimination nearly impossible.

Surveillance Capitalism Acceleration: Dating apps represent the most intimate frontier of surveillance capitalism. Advertising technology companies have long sought to categorise people's deepest desires and vulnerabilities. Camera rolls provide unprecedented access to this information. The possibility that dating app data feeds advertising systems creates a panopticon where looking for love means exposing your entire life to marketing manipulation.

Social Graph Exposure: Your camera roll doesn't just reveal your information but that of everyone who appears in your photos. Friends, family, colleagues, and strangers captured in backgrounds become involuntary subjects of AI analysis. They never consented to dating app surveillance, yet their faces, locations, and contexts feed recommendation algorithms. This collateral data collection lacks even the pretence of consent.

Psychological Manipulation: AI personalisation optimises for engagement, not wellbeing. Systems that learn what keeps users swiping, returning, and subscribing have incentive to manipulate rather than serve. Camera roll access enables psychological profiling sophisticated enough to identify and exploit vulnerabilities. Someone whose photos suggest loneliness might receive matches designed to generate hope then disappointment, maximising time on platform.

Blackmail and Extortion: Perhaps the most visceral harm is exploitation by malicious actors. Dating apps attract scammers and predators. Camera roll access, even if intended for AI personalisation, creates breach risks that expose intimate content. The 1.5 million sexually explicit images exposed by inadequate security at BDSM People, Chica, Pink, Brish, and Translove demonstrate this isn't theoretical. For many users, such exposure represents catastrophic harm: employment loss, family rejection, legal jeopardy, even physical danger.

These downstream harms share a common feature: they're difficult to remedy after the fact. Once camera roll data is collected, the privacy violation is permanent. Once AI models train on your images, that information persists in model weights. Once data breaches expose intimate photos, no amount of notification or credit monitoring repairs the damage. Prevention is the only viable strategy, yet dating apps' current trajectory moves toward greater data collection, not less.

Demanding Better Systems

Reconciling AI personalisation with genuine privacy protection in dating apps requires systemic change across technology, regulation, and business models.

Regulatory Intervention: Current privacy laws, GDPR, CCPA, BIPA, provide frameworks but lack enforcement mechanisms commensurate with the harms. What's needed are:

Dating app-specific regulations recognising the unique privacy sensitivities and power dynamics of platforms facilitating intimate relationships. Blanket consent for broad data collection should be prohibited. Mandatory on-device processing for camera roll analysis, with cloud processing permitted only with specific opt-in and complete transparency. Standardised transparency reporting requirements, modelled on social media content moderation disclosures. Minimum security standards with regular independent audits. Private rights of action enabling users harmed by privacy violations to seek remedy without requiring class action or regulatory intervention. Significant penalties for violations, sufficient to change business model calculations.

The European Union's AI Act and Digital Services Act provide templates. The AI Act's risk-based approach could classify dating app recommendation systems using camera roll data as high-risk, triggering conformity assessment, documentation, and human oversight requirements. The Digital Services Act's transparency obligations could extend to requiring algorithmic disclosure.

Technical Mandates: Regulations should require specific technical safeguards. On-device processing for camera roll analysis must be the default, with exceptions requiring demonstrated necessity and user opt-in. End-to-end encryption should be mandatory for all intimate communications. Differential privacy should be required for any aggregate data analysis. Regular independent security audits should be public. Data minimisation should be enforced: apps must collect only data demonstrably necessary for specified purposes and delete it when that purpose ends.

Business Model Evolution: The fundamental problem is that dating apps monetise user data rather than service quality. Match Group's portfolio strategy depends on network effects and data sharing across properties. This creates incentive to maximise data collection regardless of necessity.

Alternative models exist. Subscription-based services with privacy guarantees could compete on trust rather than algorithmic engagement. Apps could adopt cooperative or non-profit structures removing profit incentive to exploit user data. Open-source matching algorithms would enable transparency and independent verification. Federated systems where users control their own data whilst still participating in matching networks could preserve privacy whilst enabling AI personalisation.

User Empowerment: Technical and regulatory changes must be complemented by user education and tools. Privacy settings should be accessible and clearly explained. Data dashboards should show exactly what's collected, how it's used, and enable granular control. Regular privacy check-ups should prompt users to review and update permissions. Export functionality should enable users to retrieve all their data in usable formats. Deletion should be complete and immediate, not delayed or partial.

Industry Standards: Self-regulation has failed dating apps, but industry coordination could still play a role. Standards bodies could develop certification programmes for privacy-preserving dating apps, similar to organic food labels. Apps meeting stringent criteria, on-device processing, E2EE, no data sharing, minimal retention, regular audits, could receive certification enabling users to make informed choices. Market pressure from privacy-conscious users might drive adoption more effectively than regulation alone.

Research Access: Independent researchers need ability to audit dating app systems without violating terms of service or computer fraud laws. Regulatory sandboxes could provide controlled access to anonymised data for studying algorithmic discrimination, privacy risks, and harm patterns. Whistleblower protections should extend to dating app employees witnessing privacy violations or harmful practices.

The fundamental principle must be: personalisation does not require surveillance. AI can improve matching whilst respecting privacy, but only if we demand it.

The Critical Choice

Tinder's Chemistry feature represents a inflection point. As dating apps embrace AI-powered personalisation through camera roll analysis, we face a choice between two futures.

In one, we accept that finding love requires surrendering our most intimate data. We normalise algorithmic analysis of our unfiltered lives. We trust that companies facing subscriber declines and pressure to monetise will handle our camera rolls responsibly. We hope that the next breach won't expose our images. We assume discrimination and manipulation won't target us specifically. We believe consent dialogues satisfy meaningful choice.

In the other future, we demand better. We insist that AI personalisation use privacy-preserving technologies like on-device processing and federated learning. We require transparency about data collection, retention, and sharing. We enforce consent mechanisms that provide genuine information and control. We hold companies accountable for privacy violations and security failures. We build regulatory frameworks recognising dating apps' unique risks and power dynamics. We create business models aligned with user interests rather than data extraction.

The technical capability exists to build genuinely privacy-preserving dating apps with sophisticated AI personalisation. What's lacking is the economic incentive and regulatory pressure to implement these technologies instead of surveilling users.

Dating is inherently vulnerable. People looking for connection reveal hopes, desires, insecurities, and loneliness. Platforms facilitating these connections bear extraordinary responsibility to protect that vulnerability. The current industry trajectory towards AI-powered camera roll surveillance betrays that responsibility in pursuit of engagement metrics and advertising revenue.

As Spencer Rascoff positions camera roll access as essential for Tinder's future, and as other dating apps inevitably follow, users must understand what's at stake. This isn't about refusing technology or rejecting AI. It's about demanding that personalisation serve users rather than exploit them. It's about recognising that some data is too sensitive, some surveillance too invasive, some consent too coerced to be acceptable regardless of potential benefits.

The privacy crisis in dating apps is solvable. The solutions exist. The question is whether we'll implement them before the next breach, the next scandal, or the next tragedy forces our hand. By then, millions more camera rolls will have been analysed, billions more intimate images processed, and countless more users exposed to harms that could have been prevented.

We have one chance to get this right. Match Group's subscriber declines suggest users are already losing faith in dating apps. Doubling down on surveillance rather than earning back trust through privacy protection risks accelerating that decline whilst causing tremendous harm along the way.

The choice is ours: swipe right on surveillance, or demand the privacy-preserving future that technology makes possible. For the sake of everyone seeking connection in an increasingly digital world, we must choose wisely.

References

  1. Constine, J. (2025, November 5). Tinder to use AI to get to know users, tap into their Camera Roll photos. TechCrunch. https://techcrunch.com/2025/11/05/tinder-to-use-ai-to-get-to-know-users-tap-into-their-camera-roll-photos/

  2. Mozilla Foundation. (2024, April 23). Data-Hungry Dating Apps Are Worse Than Ever for Your Privacy. Privacy Not Included. https://www.mozillafoundation.org/en/privacynotincluded/articles/data-hungry-dating-apps-are-worse-than-ever-for-your-privacy/

  3. Mozilla Foundation. (2024, April 23). 'Everything But Your Mother's Maiden Name': Mozilla Research Finds Majority of Dating Apps More Data-hungry and Invasive than Ever. https://www.mozillafoundation.org/en/blog/everything-but-your-mothers-maiden-name-mozilla-research-finds-majority-of-dating-apps-more-data-hungry-and-invasive-than-ever/

  4. Cybernews. (2025, March). Privacy disaster as LGBTQ+ and BDSM dating apps leak private photos. https://cybernews.com/security/ios-dating-apps-leak-private-photos/

  5. IBTimes UK. (2025). 1.5 Million Explicit Images Leaked From Dating Apps, Including BDSM And LGBTQ+ Platforms. https://www.ibtimes.co.uk/15-million-explicit-images-leaked-dating-apps-including-bdsm-lgbtq-platforms-1732363

  6. Fung, B. (2018, April 3). Grindr Admits It Shared HIV Status Of Users. NPR. https://www.npr.org/sections/thetwo-way/2018/04/03/599069424/grindr-admits-it-shared-hiv-status-of-users

  7. Whittaker, Z. (2018, April 2). Grindr sends HIV status to third parties, and some personal data unencrypted. TechCrunch. https://techcrunch.com/2018/04/02/grindr-sends-hiv-status-to-third-parties-and-some-personal-data-unencrypted/

  8. Top Class Actions. (2024). $40M Bumble, Badoo BIPA class action settlement. https://topclassactions.com/lawsuit-settlements/closed-settlements/40m-bumble-badoo-bipa-class-action-settlement/

  9. FindBiometrics. (2024). Illinoisan Bumble, Badoo Users May Get Payout from $40 Million Biometric Privacy Settlement. https://findbiometrics.com/illinoisan-bumble-badoo-users-may-get-payout-from-40-million-biometric-privacy-settlement/

  10. noyb. (2021, December 15). NCC & noyb GDPR complaint: “Grindr” fined €6.3 Mio over illegal data sharing. https://noyb.eu/en/ncc-noyb-gdpr-complaint-grindr-fined-eu-63-mio-over-illegal-data-sharing

  11. Computer Weekly. (2021). Grindr complaint results in €9.6m GDPR fine. https://www.computerweekly.com/news/252495431/Grindr-complaint-results-in-96m-GDPR-fine

  12. Data Protection Commission. (2020, February 4). Data Protection Commission launches Statutory Inquiry into MTCH Technology Services Limited (Tinder). https://www.dataprotection.ie/en/news-media/latest-news/data-protection-commission-launches-statutory-inquiry-mtch-technology

  13. Coldewey, D. (2020, February 4). Tinder's handling of user data is now under GDPR probe in Europe. TechCrunch. https://techcrunch.com/2020/02/04/tinders-handling-of-user-data-is-now-under-gdpr-probe-in-europe/

  14. Duportail, J. (2017, September 26). I asked Tinder for my data. It sent me 800 pages of my deepest, darkest secrets. The Guardian. Referenced in: https://siliconangle.com/2017/09/27/journalist-discovers-tinder-records-staggering-amounts-personal-information/

  15. ACLU of Illinois. (2008). Biometric Information Privacy Act (BIPA). https://www.aclu-il.org/en/campaigns/biometric-information-privacy-act-bipa

  16. ClassAction.org. (2022). Dating App Privacy Violations | Hinge, OkCupid, Tinder. https://www.classaction.org/hinge-okcupid-tinder-privacy-lawsuits

  17. Match Group. (2025). Our Company. https://mtch.com/ourcompany/

  18. Peach, T. (2024). Swipe Me Dead: Why Dating Apps Broke (my brain). Medium. https://medium.com/@tiffany.p.peach/swipe-me-dead-f37f3e717376

  19. Electronic Frontier Foundation. (2025). Eva Galperin – Director of Cybersecurity. https://www.eff.org/about/staff/eva-galperin

  20. Electronic Frontier Foundation. (2020, May). Watch EFF Cybersecurity Director Eva Galperin's TED Talk About Stalkerware. https://www.eff.org/deeplinks/2020/05/watch-eff-cybersecurity-director-eva-galperins-ted-talk-about-stalkerware

  21. noyb. (2025, June). Bumble's AI icebreakers are mainly breaking EU law. https://noyb.eu/en/bumbles-ai-icebreakers-are-mainly-breaking-eu-law

  22. The Record. (2025). Complaint says Bumble feature connected to OpenAI violates European data privacy rules. https://therecord.media/bumble-for-friends-openai-noyb-complaint-gdpr

  23. Apple. (2021). Recognizing People in Photos Through Private On-Device Machine Learning. Apple Machine Learning Research. https://machinelearning.apple.com/research/recognizing-people-photos

  24. Hard, A., et al. (2018). Federated Learning for Mobile Keyboard Prediction. arXiv preprint arXiv:1811.03604. https://arxiv.org/abs/1811.03604

  25. Ramaswamy, S., et al. (2019). Applied Federated Learning: Improving Google Keyboard Query Suggestions. arXiv preprint arXiv:1812.02903. https://arxiv.org/abs/1812.02903

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The pharmaceutical industry has always been a high-stakes gamble. For every drug that reaches pharmacy shelves, thousands of molecular candidates fall by the wayside, casualties of a discovery process that devours billions of pounds and stretches across decades. The traditional odds are brutally unfavourable: roughly one in 5,000 compounds that enter preclinical testing eventually wins regulatory approval, and the journey typically consumes 10 to 15 years and costs upwards of £2 billion. Now, artificial intelligence promises to rewrite these economics entirely, and the early evidence suggests it might actually deliver.

In laboratories from Boston to Shanghai, scientists are watching algorithms design antibodies from scratch, predict protein structures with atomic precision, and compress drug discovery timelines from years into months. These aren't incremental improvements but fundamental shifts in how pharmaceutical science operates, driven by machine learning systems that can process biological data at scales and speeds no human team could match. The question is no longer whether AI can accelerate drug discovery, but rather how reliably it can do so across diverse therapeutic areas, and what safeguards the industry needs to translate computational leads into medicines that are both safe and effective.

The Computational Revolution in Molecular Design

Consider David Baker's laboratory at the University of Washington's Institute for Protein Design. In work published during 2024, Baker's team used a generative AI model called RFdiffusion to design antibodies entirely from scratch, achieving what the field had long considered a moonshot goal. These weren't antibodies optimised from existing templates but wholly novel molecules, computationally conceived and validated through rigorous experimental testing including cryo-electron microscopy. The structural agreement between predicted and actual configurations was remarkable, with root-mean-square deviation values as low as 0.3 angstroms for individual complementarity-determining regions.

Previously, no AI systems had demonstrated they could produce high-quality lead antibodies from scratch in a way that generalises across protein targets and antibody formats. Baker's team reported AI-aided discovery of antibodies that bind to an influenza protein common to all viral strains, plus antibodies that block a potent toxin produced by Clostridium difficile. By shifting antibody design from trial-and-error wet laboratory processes to rational computational workflows, the laboratory compressed discovery timelines from years to weeks.

The implications ripple across the pharmaceutical landscape. Nabla Bio created JAM, an AI system designed to generate de novo antibodies with favourable affinities across soluble and difficult-to-drug membrane proteins, including CXCR7, one member of the family of approximately 800 GPCR membrane proteins that have historically resisted traditional antibody development.

Absci announced the ability to create and validate de novo antibodies in silico using zero-shot generative AI. The company reported designing the first antibody capable of binding to a protein target on HIV known as the caldera region, a previously difficult-to-drug epitope. In February 2024, Absci initiated IND-enabling studies for ABS-101, a potential best-in-class anti-TL1A antibody, expecting to submit an investigational new drug application in the first quarter of 2025. The company claims its Integrated Drug Creation platform can advance AI-designed development candidates in as few as 14 months, potentially reducing the journey from concept to clinic from six years down to 18-24 months.

Where AI Delivers Maximum Impact

The drug discovery pipeline comprises distinct phases, each with characteristic challenges and failure modes. AI's impact varies dramatically depending on which stage you examine. The technology delivers its most profound advantages in early discovery: target identification, hit discovery, and lead optimisation, where computational horsepower can evaluate millions of molecular candidates simultaneously.

Target identification involves finding the biological molecules, typically proteins, that play causal roles in disease. Recursion Pharmaceuticals built the Recursion Operating System, a platform that has generated one of the largest fit-for-purpose proprietary biological and chemical datasets globally, spanning 65 petabytes across phenomics, transcriptomics, in vivo data, proteomics, and ADME characteristics. Their automated wet laboratory utilises robotics and computer vision to capture millions of cell experiments weekly, feeding data into machine learning models that identify novel therapeutic targets with unprecedented systematic rigour.

Once targets are identified, hit discovery begins. This is where AI's pattern recognition capabilities shine brightest. Insilico Medicine used AI to identify a novel drug target and design a lead molecule for idiopathic pulmonary fibrosis, advancing it through preclinical testing to Phase I readiness in under 18 months, a timeline that would have been impossible using traditional methods. The company's platform nominated ISM5411 as a preclinical candidate for inflammatory bowel disease in January 2022 after only 12 months to synthesise and screen approximately 115 molecules. Their fastest preclinical candidate nomination was nine months for the QPCTL programme.

Lead optimisation also benefits substantially from AI. Exscientia reports a 70 percent faster lead-design cycle coupled with an 80 percent reduction in upfront capital. The molecule DSP-1181, developed with Sumitomo Dainippon Pharma, moved from project start to clinical trial in 12 months, compared to approximately five years normally. Exscientia was the first company to advance an AI-designed drug candidate into clinical trials.

However, AI's advantages diminish in later pipeline stages. Clinical trial design, patient recruitment, and safety monitoring still require substantial human expertise and regulatory oversight. As compounds progress from Phase I through Phase III studies, rate-limiting factors shift from molecular design to clinical execution and regulatory review.

The Reliability Question

The pharmaceutical industry has grown justifiably cautious about overhyped technologies. What does the empirical evidence reveal about AI's actual success rates?

The early data looks genuinely promising. As of December 2023, AI-discovered drugs that completed Phase I trials showed success rates of 80 to 90 percent, substantially higher than the roughly 40 percent success rate for traditionally discovered molecules. Out of 24 AI-designed molecules that entered Phase I testing, 21 successfully passed, yielding an 85 to 88 percent success rate nearly double the historical benchmark.

For Phase II trials, success rates for AI-discovered molecules sit around 40 percent, comparable to historical averages. This reflects the reality that Phase II trials test proof-of-concept in patient populations, where biological complexity creates challenges that even sophisticated AI cannot fully predict from preclinical data. If current trends continue, analysts project the probability of a molecule successfully navigating all clinical phases could increase from 5 to 10 percent historically to 9 to 18 percent for AI-discovered candidates.

The number of AI-discovered drug candidates entering clinical stages is growing exponentially. From three candidates in 2016, the count reached 17 in 2020 and 67 in 2023. AI-native biotechnology companies and their pharmaceutical partners have entered 75 AI-discovered molecules into clinical trials since 2015, demonstrating a compound annual growth rate exceeding 60 percent.

Insilico Medicine provides a useful case study. By December 31, 2024, the company had nominated 22 developmental candidates from its own chemistry and biology platform, with 10 programmes progressing to human clinical stage, four completed Phase I studies, and one completed Phase IIa. In January 2025, Insilico announced positive results from two Phase I studies in Australia and China of ISM5411, a novel gut-restricted PHD inhibitor that proved generally safe and well tolerated.

The company's lead drug INS018_055 (rentosertib) reached Phase IIa trials for idiopathic pulmonary fibrosis, a devastating disease with limited treatment options. Following publication of a Nature Biotechnology paper in early 2024 presenting the entire journey from AI algorithms to Phase II clinical trials, Insilico announced positive results showing favourable safety and dose-dependent response in forced vital capacity after only 12 weeks. The company is preparing a Phase IIb proof-of-concept study to be initiated in 2025, representing a critical test of whether AI-discovered drugs can demonstrate the robust efficacy needed for regulatory approval.

Yet not everything proceeds smoothly. Recursion Pharmaceuticals, despite securing partnerships with Roche, Sanofi, and Bayer, recently announced it was shelving three advanced drug prospects following its 2024 merger with Exscientia. The company halted development of drugs for cerebral cavernous malformation and neurofibromatosis type II in mid-stage testing, choosing to focus resources on programmes with larger commercial potential. Exscientia itself had to deprioritise its cancer drug EXS-21546 after early-stage trials and pare back its pipeline to focus on CDK7 and LSD1 oncology programmes. These strategic retreats illustrate that AI-discovered drugs face the same clinical and commercial risks as traditionally discovered molecules.

The Validation Imperative

The gap between computational prediction and experimental reality represents one of the most critical challenges. Machine learning models train on available data, but biological systems exhibit complexity that even sophisticated algorithms struggle to capture fully, creating an imperative for rigorous experimental validation.

Traditional QSAR-based models faced problems including small training sets, experimental data errors, and lack of thorough validation. Modern AI approaches address these limitations through iterative cycles integrating computational prediction with wet laboratory testing. Robust iteration between teams proves critical because data underlying any model remains limited and biased by the experiments that generated it.

Companies like Absci report that initially, their computational designs exhibited modest affinity, but subsequent affinity maturation techniques such as OrthoRep improved binding strength to single-digit nanomolar levels whilst preserving epitope selectivity. This demonstrates that AI provides excellent starting points, but optimisation through experimental iteration often proves necessary.

The validation paradigm is shifting. In traditional drug discovery, wet laboratory experiments dominated from start to finish. In the emerging paradigm, in silico experiments could take projects almost to the endpoint, with wet laboratory validation serving as final confirmation that ensures only the best candidates proceed to clinical trials.

Generate Biomedicines exemplifies this integrated approach. The company's Generative Biology platform trains on the entire compendium of protein structures and sequences found in nature, supplemented with proprietary experimental data, to learn generalisable rules by which amino acid sequences encode protein structure and function. Their generative model Chroma can produce designs for proteins with specific properties. To validate predictions, Generate opened a cryo-electron microscopy laboratory in Andover, Massachusetts, that provides high-resolution structural data feeding back into the AI models.

However, challenges persist. Generative AI often suggests compounds that prove challenging or impossible to synthesise, or that lack drug-like properties such as appropriate solubility, stability, or bioavailability. Up to 40 percent of antibody candidates fail in clinical trials due to unanticipated developability issues, costing billions of pounds annually.

Intellectual Property in the Age of Algorithmic Invention

Who owns a drug that an algorithm designed? This question opens a labyrinth of legal complexity that the pharmaceutical and biotechnology industries are only beginning to navigate.

Under United States patent law, inventorship is strictly reserved for natural persons. The 2022 Thaler v. Vidal decision rejected patent applications listing DABUS, an AI system, as the sole inventor. However, the United States Patent and Trademark Office's 2024 guidance clarified that AI-assisted inventions remain patentable if a human provides a significant contribution to either conception or reduction to practice.

The critical phrase is “significant contribution.” In most cases, a human merely reducing an AI invention to practice does not constitute sufficient contribution. However, iterating on and improving an AI output can clear that bar. Companies that develop AI systems focused on specific issues have indicia of human contribution from the outset, for example by identifying binding affinity requirements and in vivo performance specifications, then developing AI platforms to generate drug candidates with those properties.

This creates strategic imperatives for documentation. It's critical to thoroughly document the inventive process including both AI and human contributions, detailing specific acts humans undertook beyond mere verification of AI outputs. Without such documentation, companies risk having patent applications rejected or granted patents later invalidated.

International jurisdictions add complexity. The European Patent Office requires “technical contribution” beyond mere data analysis. AI drug discovery tools need to improve experimental methods or manufacturing processes to qualify under EPO standards. China's revised 2024 guidelines allow AI systems to be named as co-inventors if humans oversee their output, though enforcement remains inconsistent.

Pharmaceutical companies increasingly turn to hybrid approaches. Relay Therapeutics combines strategies by patenting drug candidates whilst keeping molecular dynamics simulations confidential. Yet complications arise: whilst Recursion Pharmaceuticals has multiple AI-optimised small molecule compounds in clinical development, several (REC-2282 and REC-4881) were known and patented by other parties, requiring Recursion to obtain licences. Even sophisticated AI systems may rediscover molecules that already exist in the intellectual property landscape.

Regulatory Pathways

Regulatory agencies face an unprecedented challenge: how do you evaluate drugs designed by systems you cannot fully interrogate? The United States Food and Drug Administration issued its first guidance on the use of AI for drug and biological product development in January 2025, providing a risk-based framework for sponsors to assess and establish the credibility of an AI model for particular contexts of use.

This represents a critical milestone. Since 2016, the use of AI in drug development and regulatory submissions has exponentially increased. CDER's experience includes over 500 submissions with AI components from 2016 to 2023, yet formal guidance remained absent until now. The framework addresses how sponsors should validate AI models, document training data provenance and quality, and demonstrate that model outputs are reliable for their intended regulatory purpose.

The fundamental principle remains unchanged: new drugs must undergo rigorous testing and evaluation to gain FDA approval regardless of how they were designed. However, this can prove more challenging for generative AI because underlying biology and mechanisms of action may not be sufficiently understood. When an AI system identifies a novel target through pattern recognition across vast datasets, human researchers may struggle to articulate the mechanistic rationale that regulators typically expect.

Regulatory submissions for AI-designed drugs need to include not only traditional preclinical and clinical data, but also detailed information about the AI system itself: training data sources and quality, model architecture and validation, limitations and potential biases, and the rationale for trusting model predictions.

As of 2024, there are no on-market medications developed using an AI-first pipeline, though many are progressing through clinical trials. The race to become first carries both prestige and risk: the inaugural approval will establish precedents that shape regulatory expectations for years to come.

The medical device sector provides instructive precedents. Through 2025, the FDA has authorised over 1,000 AI-enabled medical devices, developing institutional experience with evaluating AI systems. Drug regulation, however, presents distinct challenges: whilst medical device AI often assists human decision-making, drug discovery AI makes autonomous design decisions that directly determine molecular structures.

Business Models and Partnership Structures

The business models emerging at the intersection of AI and drug discovery exhibit remarkable diversity. Some companies pursue proprietary pipelines, others position themselves as platform providers, and many adopt hybrid approaches balancing proprietary programmes with strategic partnerships.

Recent deals demonstrate substantial valuations attached to proven AI capabilities. AstraZeneca agreed to pay more than £4 billion to CSPC Pharmaceutical Group for access to its AI platform and a portfolio of preclinical cancer drugs, one of the largest AI biotech deals to date. Sanofi unveiled a £1.3 billion agreement with Earendil Labs in April 2024. Pfizer invested £15 million in equity with CytoReason, with the option to licence the platform in a deal that could reach £85 million over five years.

Generate Biomedicines secured a collaboration with Amgen worth up to £1.5 billion across five co-development programmes in oncology, immunology, and infectious diseases. These deals reflect pharmaceutical companies' recognition that internal AI capabilities may lag behind specialised AI biotechs, making strategic partnerships the fastest route to accessing cutting-edge technology.

Morgan Stanley Research believes that modest improvements in early-stage drug development success rates enabled by AI could lead to an additional 50 novel therapies over a 10-year period, translating to more than £40 billion in opportunity. The McKinsey Global Institute projects generative AI will deliver £48 to £88 billion annually in pharmaceutical value, largely by accelerating early discovery and optimising resource allocation.

Partnership structures must address complex questions around intellectual property allocation, development responsibilities, financial terms, and commercialisation rights. Effective governance structures, both formal contractual mechanisms and informal collaborative norms, prove essential for partnership success.

The high-profile merger between Recursion Pharmaceuticals and Exscientia, announced in August 2024 with a combined valuation of approximately £430 million, represents consolidation amongst AI biotechs to achieve scale advantages and diversified pipelines. The merged entity subsequently announced pipeline cuts to extend its financial runway into mid-2027, illustrating ongoing capital efficiency pressures facing the sector.

The AlphaFold Revolution

No discussion of AI in drug discovery can ignore AlphaFold, DeepMind's protein structure prediction system that won the 14th Critical Assessment of Structure Prediction competition in December 2020. Considered by many as AI's greatest contribution to scientific fields and one of the most important scientific breakthroughs of the 21st century, AlphaFold2 reshaped structural biology and created unprecedented opportunities for research.

The system's achievement was predicting protein structures with experimental-grade accuracy from amino acid sequences alone. For decades, determining a protein's three-dimensional structure required time-consuming and expensive experimental techniques, often taking months or years per protein. AlphaFold2 compressed this process to minutes, and DeepMind released structural predictions for over 200 million proteins, effectively solving the structure prediction problem for the vast majority of known protein sequences.

The implications for drug discovery proved immediate and profound. By accurately predicting target protein structures, researchers can design drugs that specifically bind to these proteins. The AlphaFold2 structures were utilised to construct the first pocket library for all proteins in the human proteome through the CavitySpace database, which can be applied to identify novel targets for known drugs in drug repurposing.

Virtual ligand screening became dramatically more accessible. With predicted structures available for previously uncharacterised targets, researchers can computationally evaluate how small molecules or biological drugs might bind and identify promising candidates without extensive experimental screening. This accelerates early discovery and expands the druggable proteome to include targets that were previously intractable.

AlphaFold3, released subsequently, extended these capabilities to predict the structure and interactions of all life's molecules with unprecedented accuracy. The system achieves remarkable precision in predicting drug-like interactions, including protein-ligand binding and antibody-target protein interactions. Millions of researchers globally have used AlphaFold2 to make discoveries in areas including malaria vaccines, cancer treatments, and enzyme design.

However, AlphaFold doesn't solve drug discovery single-handedly. Knowing a protein's structure doesn't automatically reveal how to drug it effectively, what selectivity a drug molecule needs to avoid off-target effects, or how a compound will behave in complex in vivo environments. Structure is necessary but not sufficient.

Cautionary Tales and Realistic Expectations

The enthusiasm around AI in drug discovery must be tempered with realistic assessment. The technology is powerful but not infallible, and the path from computational prediction to approved medicine remains long and uncertain.

Consider that as of 2024, despite years of development and billions in investment, no AI-first drug has reached the market. The candidates advancing through clinical trials represent genuine progress, but they haven't yet crossed the ultimate threshold: demonstrating in large, well-controlled clinical trials that they are safe and effective enough to win regulatory approval.

A Nature article in 2023 warned that “AI's potential to accelerate drug discovery needs a reality check,” cautioning that the field risks overpromising and underdelivering. Previous waves of computational drug discovery enthusiasm, from structure-based design in the 1990s to systems biology in the 2000s, generated substantial hype but modest real-world impact.

The data quality problem represents a persistent challenge. Machine learning systems are only as good as their training data, and biological datasets often contain errors, biases, and gaps. Models trained on noisy data will perpetuate and potentially amplify these limitations.

The “black box” problem creates both scientific and regulatory concerns. Deep learning models make predictions through layers of mathematical transformations that can be difficult or impossible to interpret mechanistically. This opacity creates challenges for troubleshooting when predictions fail and for satisfying regulatory requirements for mechanistic understanding.

Integration challenges between AI teams and traditional pharmaceutical organisations also create friction. Drug discovery requires deep domain expertise in medicinal chemistry, pharmacology, toxicology, and clinical medicine. AI systems can augment but not replace this expertise. Organisations must successfully integrate computational and experimental teams, aligning incentives and workflows. This cultural integration proves harder than technical integration in many cases.

The capital intensity of drug development means that even dramatic improvements in early discovery efficiency may not transform overall economics as much as proponents hope. If AI compresses preclinical timelines from six years to two and improves Phase I success rates from 40 percent to 85 percent, clinical development from Phase II through approval still requires many years and hundreds of millions of pounds.

The Transformative Horizon

Despite caveats and challenges, the trajectory of AI in drug discovery points toward transformation rather than incremental change. The technology is still in early stages, analogous perhaps to the internet in the mid-1990s: clearly important, but with most applications and business models still to be developed.

Several technological frontiers promise to extend AI's impact. Multi-modal models that integrate diverse data types could capture biological complexity more comprehensively than current approaches. Active learning approaches, where AI systems guide experimental work by identifying the most informative next experiments, could accelerate iteration between computational and experimental phases.

The extension of AI into clinical development represents a largely untapped opportunity. Current systems focus primarily on preclinical discovery, but machine learning could also optimise trial design, identify suitable patients, predict which subpopulations will respond to therapy, and detect safety signals earlier. Recursion Pharmaceuticals is expanding AI focus to clinical trials, recognising that later pipeline stages offer substantial room for improvement.

Foundation models trained on massive biological datasets, analogous to large language models like GPT-4, may develop emergent capabilities that narrow AI systems lack. These models could potentially transfer learning across therapeutic areas, applying insights from oncology to inform neuroscience programmes.

The democratisation of AI tools could also accelerate progress. As platforms become more accessible, smaller biotechs and academic laboratories that lack substantial AI expertise could leverage the technology. Open-source models and datasets, such as AlphaFold's freely available protein structures, exemplify this democratising potential.

Regulatory adaptation will continue as agencies gain experience evaluating AI-discovered drugs. The frameworks emerging now will evolve as regulators develop institutional knowledge about validation standards and how to balance encouraging innovation with ensuring patient safety.

Perhaps most intriguingly, AI could expand the druggable proteome and enable entirely new therapeutic modalities. Many disease-relevant proteins have been considered “undruggable” because they lack obvious binding pockets for small molecules or prove difficult to target with conventional antibodies. AI systems that can design novel protein therapeutics, peptides, or other modalities tailored to these challenging targets might unlock therapeutic opportunities that were previously inaccessible.

The pharmaceutical industry stands at an inflection point. The early successes of AI in drug discovery are substantial enough to command attention and investment, whilst the remaining challenges are tractable enough to inspire confidence that solutions will emerge. The question is no longer whether AI will transform drug discovery but rather how quickly and completely that transformation will unfold.

For patients waiting for treatments for rare diseases, aggressive cancers, and other conditions with high unmet medical need, the answer matters enormously. If AI can reliably compress discovery timelines, improve success rates, and expand the range of treatable diseases, it represents far more than a technological curiosity. It becomes a tool for reducing suffering and extending lives.

The algorithms won't replace human researchers, but they're increasingly working alongside them as partners in the search for better medicines. And based on what's emerging from laboratories worldwide, that partnership is beginning to deliver on its considerable promise.

Sources and References

  1. Radical Ventures. “An AI-driven Breakthrough in De Novo Antibody Design.” https://radical.vc/an-ai-driven-breakthrough-in-de-novo-antibody-design/

  2. Science/AAAS. “AI conjures up potential new antibody drugs in a matter of months.” https://www.science.org/content/article/ai-conjures-potential-new-antibody-drugs-matter-months

  3. Frontiers in Drug Discovery. “AI-accelerated therapeutic antibody development: practical insights.” https://www.frontiersin.org/journals/drug-discovery/articles/10.3389/fddsv.2024.1447867/full

  4. PubMed Central. “AI In Action: Redefining Drug Discovery and Development.” PMC11800368. https://pmc.ncbi.nlm.nih.gov/articles/PMC11800368/

  5. ScienceDirect. “How successful are AI-discovered drugs in clinical trials? A first analysis and emerging lessons.” https://www.sciencedirect.com/science/article/pii/S135964462400134X

  6. BioSpace. “Biopharma AI Collaborations Driving Innovative Change in Drug Development.” https://www.biospace.com/article/biopharma-ai-collaborations-driving-innovative-change-in-drug-development-/

  7. Morgan Stanley. “AI Drug Discovery: Leading to New Medicines.” https://www.morganstanley.com/ideas/ai-drug-discovery

  8. FDA. “FDA Proposes Framework to Advance Credibility of AI Models Used for Drug and Biological Product Submissions.” January 2025. https://www.fda.gov/news-events/press-announcements/fda-proposes-framework-advance-credibility-ai-models-used-drug-and-biological-product-submissions

  9. Fenwick & West LLP. “Unpacking AI-Assisted Drug Discovery Patents.” https://www.fenwick.com/insights/publications/unpacking-ai-assisted-drug-discovery-patents

  10. Ropes & Gray LLP. “Patentability Risks Posed by AI in Drug Discovery.” October 2024. https://www.ropesgray.com/en/insights/alerts/2024/10/patentability-risks-posed-by-ai-in-drug-discovery

  11. PR Newswire. “Insilico Received Positive Topline Results from Two Phase 1 Trials of ISM5411.” January 2025. https://www.prnewswire.com/news-releases/insilico-received-positive-topline-results-from-two-phase-1-trials-of-ism5411-new-drug-designed-using-generative-ai-for-the-treatment-of-inflammatory-bowel-disease-302344176.html

  12. EurekAlert. “Insilico Medicine announces developmental candidate benchmarks and timelines.” 2024. https://www.eurekalert.org/news-releases/1073344

  13. Clinical Trials Arena. “Insilico's AI drug enters Phase II IPF trial.” https://www.clinicaltrialsarena.com/news/insilico-medicine-ins018055-ai/

  14. UKRI. “Exscientia: a clinical pipeline for AI-designed drug candidates.” https://www.ukri.org/who-we-are/how-we-are-doing/research-outcomes-and-impact/bbsrc/exscientia-a-clinical-pipeline-for-ai-designed-drug-candidates/

  15. FierceBiotech. “AI drug hunter Exscientia chops down 'rapidly emerging pipeline' to focus on 2 main oncology programs.” https://www.fiercebiotech.com/biotech/ai-drug-hunter-exscientia-chops-down-rapidly-emerging-pipeline-focus-2-main-oncology

  16. Recursion Pharmaceuticals. “Pioneering AI Drug Discovery.” https://www.recursion.com

  17. BioPharma Dive. “AI specialist Recursion trims pipeline in latest shakeup.” https://www.biopharmadive.com/news/recursion-pipeline-cuts-first-quarter-earnings/747119/

  18. Ardigen. “Where AI Meets Wet-Lab: A Smarter Path to Biologics Discovery Success.” https://ardigen.com/where-ai-meets-wet-lab-a-smarter-path-to-biologics-discovery-success/

  19. GEN News. “AI in Drug Discovery: Trust, but Verify.” https://www.genengnews.com/topics/drug-discovery/ai-in-drug-discovery-trust-but-verify/

  20. Nature. “AlphaFold2 and its applications in the fields of biology and medicine.” Signal Transduction and Targeted Therapy. https://www.nature.com/articles/s41392-023-01381-z

  21. Google DeepMind Blog. “AlphaFold 3 predicts the structure and interactions of all of life's molecules.” https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/

  22. Absci Corporation. “Absci Initiates IND-Enabling Studies for ABS-101.” February 2024. https://investors.absci.com/news-releases/news-release-details/absci-initiates-ind-enabling-studies-abs-101-potential-best

  23. Absci Corporation. “Absci First to Create and Validate De Novo Antibodies with Zero-Shot Generative AI.” https://investors.absci.com/news-releases/news-release-details/absci-first-create-and-validate-de-novo-antibodies-zero-shot

  24. Generate:Biomedicines. “Generative Biology.” https://generatebiomedicines.com/generative-biology

  25. Business Wire. “Generate Biomedicines Uses AI to Create Proteins That Have Never Existed.” December 1, 2022. https://www.businesswire.com/news/home/20221201005267/en/Generate-Biomedicines-Uses-AI-to-Create-Proteins-That-Have-Never-Existed

  26. Chemical & Engineering News. “Generate Biomedicines opens cryo-EM lab to feed data back into its protein design algorithms.” https://cen.acs.org/pharmaceuticals/drug-discovery/Generate-Biomedicines-opens-cryo-EM/101/i23

  27. Nature. “AI's potential to accelerate drug discovery needs a reality check.” 2023. https://www.nature.com/articles/d41586-023-03172-6

  28. ScienceDirect. “New Horizons in Therapeutic Antibody Discovery: Opportunities and Challenges versus Small-Molecule Therapeutics.” https://www.sciencedirect.com/science/article/pii/S2472555222072173

  29. Nature Reviews Drug Discovery. “The company landscape for artificial intelligence in large-molecule drug discovery.” https://www.nature.com/articles/d41573-023-00139-0

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The smartphone in your pocket contains a curious paradox. Apple, one of the world's most valuable companies, builds its own chips, designs its own operating system, and controls every aspect of its ecosystem with obsessive precision. Yet when you tap Safari's search bar, you're not using an Apple search engine. You're using Google. And Google pays Apple a staggering $20 billion every year to keep it that way.

This colossal payment, revealed during the US Department of Justice's antitrust trial against Google, represents far more than a simple business arrangement. It's the visible tip of a fundamental transformation in how digital platforms compete, collaborate, and ultimately extract value from the billions of searches and queries humans perform daily. As artificial intelligence reshapes the search landscape and digital assistants become genuine conversational partners rather than glorified keyword matchers, these backend licensing deals are quietly redrawing the competitive map of the digital economy.

The stakes have never been higher. Search advertising generated $102.9 billion in revenue in the United States alone during 2024, accounting for nearly 40 per cent of all digital advertising spending. But the ground is shifting beneath the industry's feet. AI-powered search experiences from OpenAI's ChatGPT, Microsoft's Copilot, and Google's own AI Overviews are fundamentally changing how people find information, and these changes threaten to upend decades of established business models. Into this volatile mix come a new wave of licensing deals, platform partnerships, and strategic alliances that could determine which companies dominate the next generation of digital interaction.

When Search Was Simple

To understand where we're heading, it helps to grasp how we got here. Google's dominance in search wasn't accidental. The company built the best search engine, captured roughly 90 per cent of the market, and then methodically paid billions to ensure its search bar appeared by default on every device that mattered. Apple, Samsung, Mozilla, and countless other device manufacturers and browser makers accepted these payments, making Google the path of least resistance for billions of users worldwide.

The economics were brutally simple. Google paid Apple $20 billion annually, representing roughly 21 per cent of Apple's entire services revenue in 2024. In exchange, Google maintained its dominant position in mobile search, where it captured nearly 95 per cent of smartphone searches. For Apple, this represented essentially free money, high-margin revenue that required no product development, no customer support, no operational complexity. The company simply collected a 36 per cent commission on advertising revenue generated from Safari searches.

Judge Amit Mehta, in his landmark August 2024 ruling in United States v. Google LLC, described this arrangement with clinical precision: “Google is a monopolist, and it has acted as one to maintain its monopoly.” The 277-page opinion found that Google's exclusive contracts violated Section 2 of the Sherman Act, maintaining illegal monopoly power in general search services and text advertising markets.

Yet even as the legal system caught up with Google's practices, a more profound transformation was already underway. The rise of large language models and generative AI was creating an entirely new category of digital interaction, one where traditional search might become just one option among many. And the companies positioning themselves for this future weren't waiting for courts to dictate the terms.

When Assistants Get Smart

Apple's June 2024 partnership announcement with OpenAI marked a watershed moment. The integration of ChatGPT, powered by GPT-4o, into iOS, iPadOS, and macOS represented something fundamentally different from the Google search deal. This wasn't about directing queries to an existing search engine; it was about embedding advanced AI capabilities directly into the operating system's fabric.

The deal's structure reveals the shifting economics of the AI era. Unlike the Google arrangement, where billions of dollars changed hands annually, the OpenAI partnership reportedly involves no direct payment from Apple to OpenAI. Instead, OpenAI gains exposure to over one billion potential users across Apple's device ecosystem. Users can access ChatGPT for free without creating an account, and premium ChatGPT subscribers can connect their accounts to access advanced features. For OpenAI, the deal represents a potential path to reaching one billion users, a scale that could transform the company's trajectory.

But here's where it gets interesting. Apple didn't abandon Google when it partnered with OpenAI. The Google search deal continues, meaning Apple now has two horses in the race: traditional search through Google and conversational AI through OpenAI. Siri, Apple's long-struggling digital assistant, can now call upon ChatGPT when it encounters queries beyond its capabilities, whilst maintaining Google as the default search engine for web searches.

This dual-track strategy reflects a crucial truth about the current moment: nobody knows exactly how the search and assistant markets will evolve. Will users prefer traditional search results with AI-generated summaries, as Google is betting with its AI Overviews feature? Or will they migrate to conversational AI interfaces that provide direct answers without traditional web links? Apple's strategy is to cover both scenarios whilst maintaining optionality.

Microsoft, meanwhile, had moved earlier and more aggressively. The company's multi-billion dollar investment in OpenAI, first disclosed in January 2023, gave it exclusive rights to integrate OpenAI's technology into its products. Bing, Microsoft's perennial search underdog, became the first major search engine to integrate GPT-4 directly into search results. The new Bing, announced in February 2023, promised to “reinvent search” by combining traditional web results with AI-generated summaries and conversational interactions.

The Microsoft-OpenAI arrangement differs fundamentally from the Apple-Google model. Rather than simply paying for default placement, Microsoft invested billions directly in OpenAI, reportedly securing 49 per cent of the company's profits until Microsoft recoups its investment. This structure aligns incentives more closely: Microsoft succeeds if OpenAI succeeds, and vice versa. The partnership granted Microsoft exclusive access to OpenAI's models for integration into commercial products, including not just Bing but also Office applications, Windows, and Azure cloud services.

Yet despite the technological leap, Bing's market share remains stubbornly low. Even with AI superpowers, Google's dominance barely budged. Google's search market share dipped below 90 per cent for the first time since 2015 in October 2024, but the company still controlled the vast majority of queries. This stubborn reality underscores a crucial lesson: technological superiority alone doesn't break entrenched defaults and user habits.

The Economics of Digital Gatekeeping

The financial mechanics behind these deals reveal the extraordinary value of controlling access points to digital information. Google paid a total of $26.3 billion in 2021 across all its default search placements, with $20 billion going to Apple alone. To put this in perspective, that's more than the entire annual revenue of many Fortune 500 companies, paid simply to remain the default choice.

These payments work because defaults matter enormously. Research on user behaviour consistently shows that overwhelming majorities never change default settings. When Google is the default search engine, around 95 per cent of users never switch. This makes default placement extraordinarily valuable, justifying multi-billion dollar payments that would seem absurd in a genuinely competitive market.

The business model creates what economists call a two-sided market with network effects. On one side, users generate queries. On the other, advertisers pay for access to those users. Google's dominance in search made it the essential platform for digital advertising, and that dominance was maintained partly through ensuring its search bar appeared everywhere users might look for information.

US search advertising revenues surged 15.9 per cent to reach $102.9 billion in 2024, according to the Interactive Advertising Bureau and PwC annual Internet Advertising Revenue Report. Google captured the lion's share, with search spending on Google rising 10 per cent year-over-year in the fourth quarter of 2024 alone. The average cost per click increased 7 per cent, demonstrating that even as queries grew, the value of each search remained robust.

But the AI revolution threatens to disrupt these economics fundamentally. Generative AI search tools experienced an astonishing 525 per cent revenue growth in 2024, albeit from a small base. More concerning for traditional search, studies found that Google search results featuring AI Overviews saw 34.5 per cent lower clickthrough rates compared to traditional results. When users get their answers directly from AI-generated summaries, they don't click through to websites, which undermines the entire advertising model built on those clicks.

Research firm SparkToro found that roughly 60 per cent of Google searches now end without a click to any website. Gartner predicted that traditional search engine volume will decline by 25 per cent by 2026 due to AI chatbot applications. If these trends continue, the entire economic foundation of search advertising could crumble, making those multi-billion dollar default placement deals look like investments in a declining asset.

This creates a fascinating strategic dilemma for companies like Google. The company must integrate AI features to remain competitive and meet user expectations for more sophisticated answers. Yet every AI-generated summary that satisfies a user's query without requiring a click potentially destroys a small amount of advertising value. Google is essentially forced to cannibalise its own business model to prevent competitors from doing it first.

New Street Research estimated that AI Overviews advertising would account for just 1 per cent of Google's search advertising revenues in 2025, growing to 3 per cent in 2026. But this gradual integration masked deeper uncertainties about long-term monetisation. How do you sell advertising against conversational AI interactions that don't involve clicking on links? Google's experiments with embedding ads directly in AI-generated summaries provided one answer, but it remained unclear whether users would accept this model or whether advertisers would pay comparable rates for these new formats.

The Regulatory Hammer Falls

Into this already complex landscape came regulators, wielding antitrust law with renewed vigour. Judge Mehta's August 2024 ruling that Google maintained an illegal monopoly in search triggered a lengthy remedies process, culminating in a May 2025 trial to determine how to restore competition.

The Department of Justice initially proposed aggressive remedies. The DOJ called for Google to divest Chrome, its web browser, and to end exclusive distribution agreements with device makers like Apple and Samsung. The department argued that only structural separation could prevent Google from using its control over key distribution channels to maintain its search monopoly.

Apple moved to intervene in the case, filing motions to defend its “contractual interests” in the Google relationship. The company argued that the Justice Department's efforts would harm consumers and stifle innovation, particularly in artificial intelligence. The filing revealed Apple's dependence on this revenue stream; analysts at J.P. Morgan estimated Apple faced a potential $12.5 billion annual revenue hit if courts forced Google to stop making payments.

The eventual ruling, delivered in September 2025, split the difference. Judge Mehta prohibited Google from entering or maintaining exclusive contracts relating to search distribution but stopped short of requiring Chrome's divestiture. Critically, the ruling allowed Google to continue making payments to partners, just not under exclusive terms. Apple and other partners would need to offer users genuine choices, but they could still receive payments for making Google one available option.

The ruling represented a partial victory for Apple and Google's business relationship whilst establishing important guardrails. As Judge Mehta noted, “Cutting off payments from Google almost certainly will impose substantial, in some cases, crippling, downstream harms to distribution partners.” Mozilla, maker of the Firefox browser, had revealed that search engine royalties totalled $510 million against total revenue of just $594 million in 2022, illustrating the existential dependence some companies had developed on these payments.

Across the Atlantic, European regulators took a different approach. The Digital Markets Act, which came into force in March 2024, designated six companies as “gatekeepers”: Alphabet, Amazon, Apple, ByteDance, Meta, and Microsoft. These companies faced strict obligations to enable interoperability, prohibit self-preferencing, and provide fair access to their platforms.

The European Commission opened non-compliance investigations against Alphabet, Apple, and Meta in March 2024. The Commission expressed concern that Alphabet's search preferenced its own vertical services, such as Google Shopping and Google Hotels, over rival offerings. By March 2025, the Commission had informed Alphabet that Google search treated the company's services more favourably than competitors, a violation of DMA provisions.

The DMA's approach differed from US antitrust enforcement in important ways. Rather than requiring proof of market harm through lengthy litigation, the DMA imposed ex ante obligations on designated gatekeepers, shifting the burden to these platforms to demonstrate compliance. Penalties could reach 10 per cent of global annual revenue for violations, or 20 per cent for repeated infringements. The Commission fined Apple €500 million and Meta €200 million in April 2025 for non-compliance.

Critically, the DMA required gatekeepers like Google to share data useful for training search models, potentially lowering barriers for alternative search engines. This provision acknowledged that in the AI era, access to training data mattered as much as access to users. A search engine couldn't compete effectively without both the scale to attract users and the data to train increasingly sophisticated AI models.

The Small Players' Dilemma

For smaller search engines and AI model providers, these backend deals and regulatory interventions created a complex and often contradictory landscape. Companies like DuckDuckGo and Ecosia had built businesses around privacy-focused search, capturing small but loyal user bases. DuckDuckGo held a 0.63 per cent worldwide market share, whilst Ecosia claimed 0.11 per cent.

But these alternative search engines faced a fundamental problem: they didn't actually operate their own search infrastructure. DuckDuckGo sourced its main search results from Bing and Yahoo. Ecosia's search content and advertisements came from Bing. This dependence on larger tech companies for backend infrastructure limited their ability to truly differentiate and left them vulnerable to changes in these upstream relationships.

The barrier to entry for building a competitive search index was immense. Google had spent decades and tens of billions of dollars crawling the web, indexing pages, and refining ranking algorithms. Microsoft's Bing represented a similar massive investment. Smaller players simply couldn't match this scale of infrastructure investment and ongoing operational costs.

In November 2024, Ecosia and Qwant announced a partnership to build a European search index, explicitly aiming to reduce dependence on US technology companies. The initiative acknowledged that the Digital Markets Act's requirement for Google to share data provided an opening, but it would take years and substantial investment to build a competitive alternative index.

The shift towards generative AI created additional barriers for smaller players. Training large language models required not just vast amounts of data but also expensive computing infrastructure. Smaller AI firms often faced 12 to 18-month wait times for GPU delivery, whilst well-capitalised hyperscalers secured priority access to scarce H100 and next-generation G100 accelerators through billion-dollar pre-purchase contracts.

Cloud infrastructure dependency compounded these challenges. Smaller AI companies weren't just running on the cloud; they were locked into it. Big Tech companies structured deals to ensure that partner rollouts were routed through their cloud infrastructure, creating additional revenue streams and control points. A startup building on Amazon's Bedrock platform or Microsoft's Azure AI services generated ongoing cloud computing fees for these giants, even if it charged end-users directly.

Yet open-source models provided some countervailing force. Over 50 per cent of foundation models were available with open weights, meaning an AI startup could download a state-of-the-art model and build on it rather than investing millions training from scratch. Meta's Llama models, Mistral's offerings, and numerous other open alternatives lowered barriers to entry for application developers, even if training truly frontier models remained the province of well-funded labs.

The Apple-OpenAI deal illustrated both the opportunities and limitations for AI startups in this environment. On one hand, OpenAI's access to over a billion Apple devices represented extraordinary distribution that no startup could hope to match independently. On the other, the deal didn't provide OpenAI with direct payment from Apple, relying instead on the assumption that exposure would drive premium subscriptions and enterprise deals.

For smaller AI model providers, securing similar distribution deals appeared nearly impossible. Anthropic, despite raising billions from both Amazon and Google, took a different path, focusing on enterprise partnerships with companies like Cognizant, Salesforce, and Palantir rather than pursuing consumer platform deals. Anthropic's strategy reflected a pragmatic assessment that without Apple or Google-scale consumer platforms, the path to scale ran through business customers and cloud marketplaces.

Amazon's $4 billion investment in Anthropic, completed in March 2024, illustrated the deepening vertical integration between cloud providers and AI model developers. The investment gave Anthropic capital and guaranteed compute access through Amazon Web Services, whilst Amazon gained a competitive AI offering for its cloud customers. Similar dynamics played out with Google's investments in Anthropic and Microsoft's OpenAI partnership.

These investment structures created a new kind of gatekeeping. If the major cloud providers each had preferred AI partners, smaller model developers might struggle to secure both the computing resources needed for training and the distribution channels necessary for reaching customers. The market appeared to be consolidating into a handful of vertically integrated stacks: Microsoft-OpenAI, Google-Anthropic-Google's own models, Amazon-Anthropic, and Apple's multi-partner approach.

Search Monetisation in the AI Era

The transition from traditional search to AI-powered experiences raised fundamental questions about monetisation. The old model was straightforward: users entered queries, search engines displayed results along with relevant advertisements, and advertisers paid per click. This generated enormous revenues because queries signalled clear intent, making search advertising uniquely valuable.

AI-powered interactions threatened to disrupt this model in multiple ways. When a user asked ChatGPT or Claude a question and received a comprehensive answer, no advertisement appeared, and no advertiser paid anyone. The AI companies were essentially providing information services without a clear revenue model beyond subscription fees and enterprise licensing.

Google faced this challenge most acutely. The company had begun rolling out AI Overviews, which used generative AI to provide summaries at the top of search results. These summaries answered many queries directly, reducing the need for users to click through to websites. Studies found that clicks for URLs included in AI Overviews decreased by 8.9 per cent compared to when they appeared as normal search result links.

For publishers and websites that relied on search traffic, this was potentially catastrophic. If AI systems summarised content without driving clicks, the entire ecosystem of ad-supported content faced an existential threat. This explained the wave of licensing deals between AI companies and publishers throughout 2024.

OpenAI signed content licensing deals with News Corp (reportedly worth over $250 million over five years), The Atlantic, Condé Nast, and Hearst. Microsoft signed deals with the Financial Times, Reuters, Axel Springer, and USA Today Network for its Copilot Daily feature. Google signed its first publisher deal with the Associated Press in January 2025. Amazon courted publishers for its reinvented Alexa, securing a deal with The New York Times.

These deals typically involved two components: one-off payments for training rights to historical content, and ongoing variable payments for featuring current content with attribution. Axel Springer's $25 million deal with OpenAI, for instance, included both a training payment and backend fees based on usage.

The licensing deals served multiple purposes. They provided AI companies with high-quality training data and current information to improve model accuracy. They gave publishers new revenue streams to offset declining search traffic and programmatic advertising revenue. And they began establishing a new economic model for the AI era, where content creators received compensation for their contributions to AI training and operation.

But the deals also raised competitive concerns. If only the largest, best-funded AI companies could afford expensive licensing arrangements with major publishers, smaller model providers faced yet another barrier to competing effectively. The cost of content licensing could become a significant moat, favouring incumbents over startups.

Moreover, these deals didn't solve the fundamental monetisation challenge. Even with licensed content, AI companies still needed business models beyond subscriptions. ChatGPT Plus cost $20 per month, whilst enterprise deals commanded higher rates, but it wasn't clear whether subscription revenue alone could support the massive computing costs of running large language models at scale.

Advertising remained the obvious answer, but integrating advertisements into conversational AI experiences proved challenging. Users had grown accustomed to ad-free interactions with ChatGPT and Claude. Introducing advertisements risked degrading the user experience and driving users to competitors. Yet without advertising or equivalently robust revenue models, it wasn't clear how these services could achieve sustainable profitability at massive scale.

Google's experiments with advertising in AI Overviews represented one potential path forward. By embedding contextually relevant product recommendations and sponsored content within AI-generated summaries, Google aimed to preserve advertising revenue whilst providing the enhanced experiences users expected. But clickthrough rates remained lower than traditional search advertising, and it remained to be seen whether advertisers would pay comparable rates for these new formats.

The average ad spending per internet user in the Search Advertising market was estimated at $58.79 globally in 2025. For AI-powered experiences to generate comparable revenue, they would need to capture similar or greater value per interaction. This seemed plausible for high-intent commercial queries but much harder for informational searches where users simply wanted answers without purchase intent.

Collaboration, Competition, and Consolidation

The deals between platform owners and AI providers, search engines and publishers, and cloud providers and model developers painted a picture of an industry in flux. Old competitive boundaries were dissolving as former rivals became strategic partners whilst ostensibly collaborating companies competed in adjacent markets.

Apple's dual strategy with Google and OpenAI exemplified this complexity. The company maintained its lucrative search deal with Google whilst simultaneously partnering with Google's primary AI competitor. This hedging strategy made sense during a transition period when the ultimate shape of user behaviour remained uncertain. But it also created tensions: how would Apple balance these relationships if Google's search and OpenAI's ChatGPT increasingly competed for the same queries?

The regulatory environment added further complexity. The September 2025 ruling allowed Google to continue making payments whilst prohibiting exclusivity, but the practical implementation remained unclear. How would Apple, Samsung, and other partners implement genuine choice mechanisms? Would users face decision fatigue from too many options, leading them to stick with familiar defaults anyway?

The European Digital Markets Act's more prescriptive approach demanded specific interoperability and data-sharing requirements, but enforcement remained challenging. The Commission's investigations and fines demonstrated willingness to punish non-compliance, yet the underlying market dynamics favouring scale and integration proved hard to counteract through regulation alone.

For smaller companies, the landscape appeared increasingly difficult. The combination of infrastructure barriers, data access challenges, capital requirements, and distribution bottlenecks created formidable obstacles. Open-source models provided some relief, but the gap between open models and the capabilities of frontier systems from OpenAI, Google, and Anthropic remained substantial.

The venture capital funding environment for AI startups remained robust, with billions flowing into the sector. But increasingly, strategic investments from cloud providers and large tech companies dominated financing rounds. These investments came with strings attached: compute credits tied to specific cloud platforms, distribution channels through investor platforms, and expectations about technology stack choices. The apparent abundance of capital masked a reality where meaningful independence from the major platforms became harder to maintain.

Industry consolidation appeared likely to continue. Just as the cloud infrastructure market concentrated into three major players (Amazon, Microsoft, and Google), the AI model and digital assistant markets seemed headed towards a similarly concentrated structure. The economics of scale in training, the advantages of vertical integration between models and distribution, and the network effects from user data all pushed towards consolidation.

Yet genuine innovation remained possible around the edges. Specialised models for specific domains, novel interaction paradigms, privacy-focused alternatives, and open-source collaboration all represented paths where smaller players could potentially carve out sustainable niches. The challenge was whether these niches could grow large enough to represent genuine alternatives to the dominant platforms.

The New Digital Divide

The backend deals reshaping search and digital assistants represent more than business arrangements between wealthy corporations. They reflect and reinforce a fundamental divide in the digital economy between companies with platform power and everyone else. Those controlling the devices people use, the operating systems running on those devices, and the default experiences presented to users wield extraordinary influence over which technologies succeed and which fail.

The $20 billion annual payment from Google to Apple isn't just a revenue stream; it's a tax on search monetisation that Google pays to maintain access to Apple's users. The multi-billion dollar investments in OpenAI and Anthropic aren't just capital allocations; they're defensive moats ensuring that Microsoft, Amazon, and Google maintain positions in whatever AI-powered future emerges.

For users, these deals often bring genuine benefits: better integrated experiences, more sophisticated capabilities, and services they can access without explicit payment. Apple users gained ChatGPT integration without monthly fees. Google users received AI-enhanced search results at no additional cost. The major platforms competed partly by giving away AI-powered features that would have seemed miraculous just years earlier.

Yet this largesse came with less visible costs. Competition constrained by billion-dollar barriers to entry was less vigorous than it might otherwise be. Innovation from smaller players struggled to reach users trapped behind platform gatekeepers. And the concentration of power in a handful of companies created systemic risks and governance challenges that societies were still learning to address.

The regulatory response, whilst increasingly aggressive, struggled to keep pace with market evolution. By the time courts ruled on Google's search monopoly, the market was already transitioning towards AI-powered experiences that might render traditional search less central. The remedies imposed risked fighting the last war whilst the next one had already begun.

Looking forward, the competitive dynamics for digital assistants and search monetisation will likely reflect broader patterns of platform power and vertical integration. Success will depend not just on building superior technology but on securing access to users, training data, computing infrastructure, and content licensing. The backend deals determining these access points will shape which companies thrive and which struggle to compete.

The market isn't winner-take-all, but neither is it a level playing field where merit alone determines outcomes. Platform power, network effects, capital resources, and strategic partnerships create strong advantages for incumbents and favourably positioned challengers. Smaller players can succeed, but increasingly only in partnership with or in niches uncontested by the major platforms.

For regulators, the challenge will be balancing the genuine benefits of integration and scale against the competitive and innovation harms from excessive concentration. Neither the US antitrust approach nor the EU's ex ante regulatory framework has yet found the right balance, and both will likely require continued adaptation as markets evolve.

The billion-dollar handshakes between platform owners and AI providers aren't ending anytime soon. They're evolving, becoming more sophisticated, and extending into new areas as the technological landscape shifts. Understanding these deals and their implications matters not just for industry insiders but for anyone concerned with how power, innovation, and value are distributed in the digital economy. The search bar on your phone isn't just a tool for finding information; it's a battleground where the future of digital interaction is being determined, one lucrative partnership at a time.

Sources and References

  1. US Department of Justice. (2024, August 5). “Department of Justice Prevails in Landmark Antitrust Case Against Google.” Official press release. https://www.justice.gov/opa/pr/department-justice-prevails-landmark-antitrust-case-against-google

  2. Mehta, A. (2024). United States v. Google LLC, Case No. 20-cv-3010. United States District Court for the District of Columbia. 277-page opinion.

  3. IAB & PwC. (2024). “Internet Advertising Revenue Report 2024.” Reports $102.9 billion in US search advertising revenue, representing 39.8% of total digital advertising.

  4. Fortune. (2025, July 30). “Apple risks $12.5 billion revenue hit as judge weighs Google antitrust remedies, J.P.Morgan warns.” https://fortune.com/2025/07/30/apple-google-jpmorgan-billion-revenue-hit-antitrust-doj-case/

  5. OpenAI. (2024, June). “OpenAI and Apple announce partnership.” Official announcement. https://openai.com/index/openai-and-apple-announce-partnership/

  6. Microsoft. (2023, February 7). “Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web.” Official Microsoft Blog.

  7. European Commission. (2024, March 25). “Commission opens non-compliance investigations against Alphabet, Apple and Meta under the Digital Markets Act.” Official press release.

  8. Search Engine Land. (2024). “Google admits to paying Apple 36% of Safari revenue.” https://searchengineland.com/google-pay-apple-safari-revenue-antitrust-trial-434775

  9. eMarketer. (2024). “Generative Search Trends 2024.” Reports 525% revenue growth for AI-driven search engines and 34.5% lower CTR for AI Overview results.

  10. CNBC. (2024, November 12). “Ecosia, Qwant partner on search engine tech to counter Google's power.” Reports on European search index initiative.

  11. Digiday. (2024). “2024 in review: A timeline of the major deals between publishers and AI companies.” Comprehensive overview of content licensing agreements.

  12. Anthropic. (2024). “Anthropic and Salesforce expand partnership to bring Claude to regulated industries.” Official company announcement.

  13. Statista. (2024). “US Google search ad revenue 2024.” Reports Google's search advertising revenue and market share data.

  14. Gartner Research. (2024). Prediction of 25% decline in traditional search engine volume by 2026 due to AI chatbot applications.

  15. SparkToro. (2024). Research finding that approximately 60% of Google searches end without a click to any website.

  16. New Street Research. (2025). Analysis projecting AI Overviews advertising at 1% of Google search ad revenue in 2025, growing to 3% in 2026.

  17. Harvard Law Review. (2024). “United States v. Google, LLC.” Legal analysis of the antitrust case. Volume 138.

  18. Mozilla Foundation. (2023). Annual financial disclosure showing $510 million in search engine royalties against $594 million total revenue in 2022.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

The future of shopping isn't happening on a screen. It's happening in the spaces between your words and a machine's understanding of what you want. When you ask an AI agent to find you the best noise-cancelling headphones under £300, you're not just outsourcing a Google search. You're delegating an entire decision-making process to an algorithmic intermediary that will reshape how billions of pounds flow through the digital economy.

This is agentic commerce: AI systems that browse, compare, negotiate, and purchase on behalf of humans. And it's already here. OpenAI's ChatGPT now offers instant checkout for purchases from over one million Shopify merchants. Perplexity launched its Comet browser with AI agents that can autonomously complete purchases from any retailer. Opera introduced Browser Operator, the first major browser with AI-based agentic capabilities built directly into its architecture. Google is expanding its AI Mode shopping interface across the United States, adding capabilities that let customers track prices and confirm purchases without ever visiting a retailer's website.

The numbers tell a story of exponential transformation. Traffic to US retail sites from generative AI browsers and chat services increased 4,700 per cent year-over-year in July 2025, according to industry tracking data. McKinsey projects that by 2030, the US business-to-consumer retail market alone could see up to one trillion dollars in orchestrated revenue from agentic commerce, with global projections reaching three trillion to five trillion dollars.

But these astronomical figures obscure a more fundamental question: When AI agents become the primary interface between consumers and commerce, who actually benefits? The answer is forcing a reckoning across the entire e-commerce ecosystem, from multinational retailers to affiliate marketers, from advertising platforms to regulatory bodies. Because agentic commerce doesn't just change how people shop. It fundamentally rewrites the rules about who gets paid, who gets seen, and who gets trusted in the digital marketplace.

The Funnel Collapses

The traditional e-commerce funnel has been the foundational model of online retail for two decades. Awareness leads to interest, interest leads to consideration, consideration leads to conversion. Each stage represented an opportunity for merchants to influence behaviour through advertising, product placement, personalised recommendations, and carefully optimised user experience. The funnel existed because friction existed: the cognitive load of comparing options, the time cost of browsing multiple sites, the effort required to complete a transaction.

AI agents eliminate that friction by compressing the entire funnel into a single conversational exchange. When a customer arriving via an AI agent reaches a retailer's site, they're already further down the sales funnel with stronger intent to purchase. Research shows these customers are ten per cent more engaged than traditional visitors. The agent has already filtered options, evaluated trade-offs, and narrowed the field. The customer isn't browsing. They're buying.

This compression creates a paradox for retailers. Higher conversion rates and more qualified traffic represent the holy grail of e-commerce optimisation. Yet if the AI agent can compress browsing, selection, and checkout into the same dialogue, retailers that sit outside the conversation risk ceding both visibility and sales entirely.

Boston Consulting Group's modelling suggests potential earnings before interest and taxes erosion of up to 500 basis points for retailers, stemming from price transparency, smaller order sizes, and agent platform fees. That five per cent margin compression might not sound catastrophic until you consider that many retailers operate on margins of ten to fifteen per cent. Agentic commerce could eliminate a third of their profitability.

The risks extend beyond margins. Retailers face diminished direct access to customers, weaker brand loyalty, and growing dependence on intermediary platforms. When customers interact primarily with an AI agent rather than a retailer's website or app, the retailer loses the ability to shape the shopping experience, collect first-party data, or build lasting relationships. The brand becomes commoditised: a product specification in an agent's database rather than a destination in its own right.

This isn't speculation. Walmart announced a partnership with OpenAI enabling seamless “chat to checkout” experiences. Shopify integrated with ChatGPT to allow instant purchases from its merchant base. Etsy followed suit. These aren't defensive moves. They're admissions that the platform layer is shifting, and retailers must establish presence where the conversations are happening, even if it means surrendering control over the customer relationship.

The Revenue Model Revolution

If agentic commerce destroys the traditional funnel, it also demolishes the advertising models built upon that funnel. Consider Google Shopping, which has operated for years on a cost-per-click model with effective commission rates around twelve per cent. Or Amazon, whose marketplace charges sellers approximately fifteen per cent in fees and generates billions more through advertising within search results and product pages. These models depend on human eyeballs viewing sponsored listings, clicking through to product pages, and making purchase decisions influenced by paid placement.

AI agents have no eyeballs. They don't see banner ads or sponsored listings. They process structured data, evaluate parameters, and optimise for the objectives their users specify. The entire edifice of digital retail advertising, which represents a 136 billion dollar industry in 2025, suddenly faces an existential question: How do you advertise to an algorithm?

The early answer appears to be: You don't advertise. You pay for performance. OpenAI has reportedly discussed a two per cent affiliate commission model for purchases made through its shopping features. That's six times lower than Google Shopping's traditional rates and seven times lower than Amazon's marketplace fees. The economics are straightforward. In a world where AI agents handle product discovery and comparison, platforms can charge lower fees because they're not operating expensive advertising infrastructure or maintaining complex seller marketplaces. They're simply connecting buyers and sellers, then taking a cut of completed transactions.

This shift from advertising-based revenue to performance-based commissions has profound implications. Advertisers will spend 12.42 billion dollars on affiliate programmes in 2025, up 10.2 per cent year-over-year, driving thirteen per cent of US e-commerce sales. The affiliate marketing ecosystem has adapted quickly to the rise of AI shopping agents, with seventy per cent of citations for some retailers in large language models stemming from affiliate content.

But the transition hasn't been smooth. Retail affiliate marketing revenues took a hit of over fifteen per cent year-over-year in the second quarter of 2024, when Google's search algorithm updates deprioritised many affiliate sites. If ChatGPT or Perplexity become the primary shopping interfaces, and those platforms negotiate direct relationships with merchants rather than relying on affiliate intermediaries, the affiliate model could face an existential threat.

Yet the performance-based structure of affiliate marketing may also be its salvation. Cost-per-acquisition and revenue-share pricing align perfectly with agentic commerce, where marketing dollars are spent only when a purchase is made. Industry analysts predict retail media networks will reshape into affiliate-like ecosystems, complete with new metrics such as “cost per agent conversion.”

The retail media network model faces even more severe disruption. Retail media networks, which allow brands to advertise within retailer websites and apps, are projected to reach 136 billion dollars in value during 2025. But these networks depend on high human traffic volumes consuming brand messages, sponsored product listings, and targeted advertisements. When agentic AI threatens those traffic volumes by handling shopping outside retailer environments, the entire business model begins to crumble.

The industry response has been to pivot from business-to-consumer advertising to what executives are calling business-to-AI: competing for algorithmic attention rather than human attention. Traditional brand building, with its emphasis on emotional connections, lifestyle aspirations, and community, suddenly becomes the most valuable marketing strategy. Because whilst AI agents can evaluate specifications and compare prices, they still rely on the corpus of available information to make recommendations. A brand that has invested in thought leadership, earned media coverage, and authentic community engagement will appear more frequently in that corpus than a brand that exists only as a product listing in a database.

The new battleground isn't the moment of purchase. It's the moment of instruction, when a human tells an AI agent what they're looking for. Influence that initial framing and you influence the entire transaction.

The Merchant's Dilemma

For retailers, agentic commerce presents an agonising choice. Participate in these new platforms and surrender margin, control, and customer data. Refuse to participate and risk becoming invisible to a growing segment of high-intent shoppers.

The mathematics of merchant incentives in this environment grow complex quickly. If Target and Walmart stock the same product at the same price, how does an AI agent decide which retailer to recommend? In traditional e-commerce, the answer involves search engine optimisation, paid advertising, customer reviews, shipping speed, and loyalty programme benefits. In agentic commerce, the answer increasingly depends on which merchant is willing to pay the AI platform a performance incentive.

Industry analysts worry this creates a “pay to play” dynamic reminiscent of Google's shift from organic search results to advertising-dominated listings. Anyone who has used Google knows how much the first page of search results is stuffed with sponsored listings. Could agentic commerce go the same way? Currently, executives at AI companies insist their algorithms pick the best possible choices without pay-to-play arrangements. But when significant money is involved, the concern is whether those principles can hold.

Perplexity has directly criticised Amazon for being “more interested in serving you ads, sponsored results, and influencing your purchasing decisions with upsells and confusing offers.” The criticism isn't just rhetorical posturing. It's a competitive claim: that AI agents provide a cleaner, more consumer-focused shopping experience precisely because they're not corrupted by advertising revenue. Whether that purity can survive as agentic commerce scales to trillions of pounds in transaction volume remains an open question.

Some merchants are exploring alternative incentive structures. Sales performance incentive funds, where retailers pay commissions to AI platforms only when purchases are completed, align merchant interests with platform performance. Dynamic pricing strategies, where retailers offer AI platforms exclusive pricing in exchange for preferential recommendations, create a more transparent marketplace for algorithmic attention. Subscription models, where merchants pay fixed fees for inclusion in AI agent recommendation databases, avoid the pay-per-click inflation that has plagued search advertising.

But each of these approaches raises questions about transparency, fairness, and consumer welfare. If an AI agent recommends Target over Walmart because Target pays a higher commission, is that a betrayal of the user's trust? Or is it simply the same economic reality that has always governed retail, now made more efficient through automation? The answer depends largely on disclosure: whether users understand the incentives shaping the recommendations they receive.

The Transparency Crisis

Trust is the currency of AI shopping agents. If users don't trust that an agent is acting in their best interests, they won't delegate purchasing decisions. And trust requires transparency: understanding how recommendations are generated, what incentives influence those recommendations, and whether the agent is optimising for the user's preferences or the platform's profit.

The current state of transparency in AI shopping is, charitably, opaque. Most AI platforms provide little visibility into their recommendation algorithms. Users don't know which merchants have paid for preferential placement, how commissions affect product rankings, or what data is being used to personalise suggestions. The Federal Trade Commission has made clear there is no AI exemption from existing consumer protection laws, and firms deploying AI systems have an obligation to ensure those systems are transparent, explainable, fair, and empirically sound.

But transparency in AI systems is technically challenging. The models underlying ChatGPT, Claude, or Perplexity are “black boxes” even to their creators: neural networks with billions of parameters that produce outputs through processes that defy simple explanation. Algorithmic accountability requires examination of how results are reached, including transparency and justification of the AI model design, setup, and operation. That level of scrutiny is difficult when the systems themselves are proprietary and commercially sensitive.

The FTC has responded by launching Operation AI Comply, taking action against companies that rely on artificial intelligence to supercharge deceptive or unfair conduct. Actions have targeted companies promoting AI tools that enable fake reviews, businesses making unsupported claims about AI capabilities, and platforms that mislead consumers about how AI systems operate. The message is clear: automation doesn't absolve responsibility. If an AI agent makes false claims, deceptive recommendations, or unfair comparisons, the platform operating that agent is liable.

Bias represents another dimension of the transparency challenge. Research on early AI shopping agents revealed troubling patterns. Agents failed to conduct exhaustive comparisons, instead settling for the first “good enough” option they encountered. This creates what researchers call a “first-proposal bias” that gives response speed a ten to thirty times advantage over actual quality. If an agent evaluates the first few results more thoroughly than later results, merchants have an incentive to ensure their products appear early in whatever databases the agent queries.

Data bias, algorithmic bias, and user bias are the main types of bias in AI e-commerce systems. Data bias occurs when training data isn't representative of actual shopping patterns, leading to recommendations that favour certain demographics, price points, or product categories. Algorithmic bias emerges from how models weigh different factors, potentially overvaluing characteristics that correlate with protected categories. User bias happens when AI agents learn from and amplify existing consumer prejudices rather than challenging them.

The automation bias problem compounds these challenges. People may be unduly trusting of answers from machines which seem neutral or impartial. Many chatbots are effectively built to persuade, designed to answer queries in confident language even when those answers are fictional. The tendency to trust AI output creates vulnerability when that output is shaped by undisclosed commercial incentives or reflects biased training data.

Microsoft recently conducted an experiment where they gave AI agents virtual currency and instructed them to make online purchases. The agents spent all the money on scams. This wasn't a failure of the AI's reasoning capability. It was a failure of the AI's ability to assess trust and legitimacy in an environment designed to deceive. If sophisticated AI systems from a leading technology company can be systematically fooled by online fraud, what does that mean for consumer protection when millions of people delegate purchasing decisions to similar agents?

The Regulatory Response

Regulators worldwide are scrambling to develop frameworks for agentic commerce before it becomes too embedded to govern effectively. New AI-specific laws have emerged to mandate proactive transparency, bias prevention, and consumer disclosures not otherwise required under baseline consumer protection statutes.

The FTC's position emphasises that existing consumer protection laws apply to AI systems. Using artificial intelligence and algorithms doesn't provide exemption from legal obligations around truthfulness, fairness, and non-discrimination. The agency has published guidance stating that AI tools should be transparent, explainable, fair, and empirically sound, whilst fostering accountability.

European regulators are taking a more prescriptive approach through the AI Act, which classifies AI systems by risk level and imposes requirements accordingly. Shopping agents that significantly influence purchasing decisions would likely qualify as high-risk systems, triggering obligations around transparency, human oversight, and impact assessment. The regulation mandates clear disclosure of whether an entity is human or artificial, responding to the increasing sophistication of AI interactions. Under the AI Act's framework, providers of high-risk AI systems must maintain detailed documentation of their training data, conduct conformity assessments before deployment, and implement post-market monitoring to detect emerging risks. Violations can result in fines up to seven per cent of global annual turnover.

But enforcement remains challenging. The opacity of black box models means consumers have no transparency into how exactly decisions are being made. Regulators often lack the technical expertise to evaluate these systems, and by the time they develop that expertise, the technology has evolved. The European Union is establishing an AI Office with dedicated technical staff and budget to build regulatory capacity, whilst the UK is pursuing a sector-specific approach that empowers existing regulators like the Competition and Markets Authority to address AI-related harms in their domains.

The cross-border nature of AI platforms creates additional complications. An AI agent operated by a US company, trained on data from multiple countries, making purchases from international merchants, creates a jurisdictional nightmare. Which country's consumer protection laws apply? Whose privacy regulations govern the data collection? Who has enforcement authority when harm occurs? The fragmentation extends beyond Western democracies. China's Personal Information Protection Law and algorithmic recommendation regulations impose requirements on AI systems operating within its borders, creating a third major regulatory regime that global platforms must navigate.

Industry self-regulation has emerged to fill some gaps. OpenAI and Anthropic developed the Agentic Commerce Protocol, a technical standard for how AI agents should interact with merchant systems. The protocol includes provisions for identifying agent traffic, disclosing commercial relationships, and maintaining transaction records. Google and Amazon rely on separate, incompatible systems, making it difficult for merchants to translate product catalogues into multiple formats.

The question of liability looms large. When an AI agent makes a purchase that the user later regrets, who is responsible? The user who gave the instruction? The platform that operated the agent? The merchant that fulfilled the order? Traditional consumer protection frameworks assume human decision-makers at each step. Agentic commerce distributes decision-making across human-AI interactions in ways that blur lines of responsibility.

The intellectual property dimensions add further complexity. Amazon has sued Perplexity, accusing the startup of violating its terms of service by using AI agents to access the platform without disclosing their automated nature. Amazon argues that Perplexity's agents degrade the Amazon shopping experience by showing products that don't incorporate personalised recommendations and may not reflect the fastest delivery options available. Perplexity counters that since its agent acts on behalf of a human user's direction, the agent automatically has the same permissions as the human user.

This dispute encapsulates the broader regulatory challenge: existing legal frameworks weren't designed for a world where software agents act autonomously on behalf of humans, making decisions, negotiating terms, and executing transactions.

The Power Redistribution

Step back from the technical and regulatory complexities, and agentic commerce reveals itself as fundamentally about power. Power to control the shopping interface. Power to influence purchasing decisions. Power to capture transaction fees. Power to shape which businesses thrive and which wither.

For decades, that power has been distributed across an ecosystem of search engines, social media platforms, e-commerce marketplaces, payment processors, and retailers themselves. Google controlled discovery through search. Facebook controlled attention through social feeds. Amazon controlled transactions through its marketplace. Each entity extracted value from its position in the funnel, and merchants paid tribute at multiple stages to reach customers.

Agentic commerce threatens to consolidate that distributed power into whoever operates the AI agents that consumers trust. If ChatGPT becomes the primary shopping interface for hundreds of millions of users, OpenAI captures influence that currently belongs to Google, Amazon, and every retailer's individual website. The company that mediates between consumer intent and commercial transaction holds extraordinary leverage over the entire economy.

This consolidation is already visible in partnership announcements. When Walmart, Shopify, and Etsy all integrate with ChatGPT within weeks of each other, they're acknowledging that OpenAI has become a gatekeeper they cannot afford to ignore. The partnerships are defensive, ensuring presence on a platform that could otherwise bypass them entirely.

But consolidation isn't inevitable. The market could fragment across multiple AI platforms, each with different strengths, biases, and commercial relationships. Google's AI Mode might excel at product discovery for certain categories. Perplexity's approach might appeal to users who value transparency over convenience. Smaller, specialised agents could emerge for specific verticals like fashion, electronics, or groceries.

The trajectory will depend partly on technical factors: which platforms build the most capable agents, integrate with the most merchants, and create the smoothest user experiences. But it will also depend on trust and regulation. If early AI shopping agents generate high-profile failures, consumer confidence could stall adoption. If regulators impose strict requirements that only the largest platforms can meet, consolidation accelerates.

For consumers, the implications are ambiguous. Agentic commerce promises convenience, efficiency, and potentially better deals through automated comparison and negotiation. Customers arriving via AI agents already demonstrate higher engagement and purchase intent. More than half of consumers anticipate using AI assistants for shopping by the end of 2025. Companies deploying AI shopping agents are delivering thirty per cent more conversions and forty per cent faster order fulfilment.

But those benefits come with risks. Loss of serendipity and discovery as agents optimise narrowly for stated preferences rather than exposing users to unexpected products. Erosion of privacy as more shopping behaviour flows through platforms that profile and monetise user data. Concentration of market power in the hands of a few AI companies that control access to billions of customers. Vulnerability to manipulation if agents' recommendations are influenced by undisclosed commercial arrangements.

Consider a concrete scenario. A parent asks an AI agent to find educational toys for a six-year-old who loves science. The agent might efficiently identify age-appropriate chemistry sets and astronomy kits based on thousands of product reviews and educational research. But if the agent prioritises products from merchants paying higher commissions over genuinely superior options, or if it lacks awareness of recent safety recalls, convenience becomes a liability. The parent saves time but potentially compromises on quality or safety in ways they would have caught through traditional research.

Marketplace or Manipulation

Agentic commerce is not a future possibility. It is a present reality growing at exponential rates. The question is not whether AI shopping agents will reshape retail, but how that reshaping will unfold and who will benefit from the transformation.

The optimistic scenario involves healthy competition between multiple AI platforms, strong transparency requirements that help users understand recommendation incentives, effective regulation that prevents the worst abuses whilst allowing innovation, and merchants who adapt by focusing on brand building, product quality, and authentic relationships.

In this scenario, consumers enjoy unprecedented convenience and potentially lower prices through automated comparison shopping. Merchants reach highly qualified customers with strong purchase intent. AI platforms create genuine value by reducing friction and improving matching between needs and products. Regulators establish guardrails that prevent manipulation whilst allowing experimentation. Picture a marketplace where an AI agent negotiates bulk discounts on behalf of a neighbourhood buying group, secures better warranty terms through automated comparison of coverage options, and flags counterfeit products by cross-referencing manufacturer databases, all whilst maintaining transparent logs of its decision-making process that users can audit.

The pessimistic scenario involves consolidation around one or two dominant AI platforms that extract monopoly rents, opaque algorithms shaped by undisclosed commercial relationships that systematically favour paying merchants over best products, regulatory capture or inadequacy that allows harmful practices to persist, and a race to the bottom on merchant margins that destroys business viability for all but the largest players.

In this scenario, consumers face an illusion of choice backed by recommendations shaped more by who pays the AI platform than by genuine product quality. Merchants become commodity suppliers in a system they can't influence without paying increasing fees. AI platforms accumulate extraordinary power and profit through their gatekeeper position. Imagine a future where small businesses cannot afford the fees to appear in AI agent recommendations, where platforms subtly steer purchases toward their own private-label products, and where consumers have no practical way to verify whether they're receiving genuinely optimal recommendations or algorithmically optimised profit extraction.

Reality will likely fall somewhere between these extremes. Some markets will consolidate whilst others fragment. Some AI platforms will maintain rigorous standards whilst others cut corners. Some regulators will successfully enforce transparency whilst others lack resources or authority. The outcome will be determined by choices made over the next few years by technology companies, policymakers, merchants, and consumers themselves.

The Stakeholder Reckoning

For technology companies building AI shopping agents, the critical choice is whether to prioritise short-term revenue maximisation through opaque commercial relationships or long-term trust building through transparency and user alignment. The companies that choose trust will likely capture sustainable competitive advantage as consumers grow more sophisticated about evaluating AI recommendations.

For policymakers, the challenge is crafting regulation that protects consumers without stifling the genuine benefits that agentic commerce can provide. Disclosure requirements, bias auditing, interoperability standards, and clear liability frameworks can establish baseline guardrails without prescribing specific technological approaches. The most effective regulatory strategies will focus on outcomes rather than methods: requiring transparency in how recommendations are generated, mandating disclosure of commercial relationships that influence agent behaviour, establishing accountability when AI systems cause consumer harm, and creating mechanisms for independent auditing of algorithmic decision-making. Policymakers must act quickly enough to prevent entrenchment of harmful practices but thoughtfully enough to avoid crushing innovation that could genuinely benefit consumers.

For merchants, adaptation means shifting from optimising for human eyeballs to optimising for algorithmic evaluation and human trust simultaneously. The retailers that will thrive are those that maintain compelling brands, deliver genuine value, and build direct relationships with customers that no AI intermediary can fully replace. This requires investment in product quality, authentic customer service, and brand building that goes beyond algorithmic gaming. Merchants who compete solely on price or visibility in AI agent recommendations will find themselves in a race to the bottom. Those who create products worth recommending and brands worth trusting will discover that AI agents amplify quality rather than obscuring it.

For consumers, the imperative is developing critical literacy about how AI shopping agents work, what incentives shape their recommendations, and when to trust algorithmic suggestions versus conducting independent research. Blind delegation is dangerous. Thoughtful use of AI as a tool for information gathering and comparison, combined with final human judgment, represents the responsible approach. This means asking questions about how agents generate recommendations, understanding what commercial relationships might influence those recommendations, and maintaining the habit of spot-checking AI suggestions against independent sources. Consumer demand for transparency can shape how these systems develop, but only if consumers actively seek that transparency rather than passively accepting algorithmic guidance.

Who Controls the Algorithm Controls Commerce

The fundamental question agentic commerce poses is who gets to shape the marketplace of the future. Will it be the AI platforms that control the interface? The merchants with the deepest pockets to pay for visibility? The regulators writing the rules? Or the consumers whose aggregate choices ultimately determine what succeeds?

The answer is all of the above, in complex interaction. But that interaction will produce very different outcomes depending on the balance of power. If consumers remain informed and engaged, if regulators act decisively to require transparency, if merchants compete on quality rather than just algorithmic gaming, and if AI platforms choose sustainable trust over exploitative extraction, then agentic commerce could genuinely improve how billions of people meet their needs.

If those conditions don't hold, we're building a shopping future where the invisible hand of the market gets replaced by the invisible hand of the algorithm, and where that algorithm serves the highest bidder rather than the human asking for help. The stakes are not just commercial. They're about what kind of economy we want to inhabit: one where technology amplifies human agency or one where it substitutes algorithmic optimisation for human choice.

The reshape is already underway. The revenue is already flowing through new channels. The questions about trust and transparency are already urgent. What happens next depends on decisions being made right now, in boardrooms and regulatory offices and user interfaces, about how to build the infrastructure of algorithmic commerce. Get those decisions right and we might create something genuinely better than what came before. Get them wrong and we'll spend decades untangling the consequences.

The invisible hand of AI is reaching for your wallet. The question is whether you'll notice before it's already spent your money.

Sources and References

  1. McKinsey & Company (2025). “The agentic commerce opportunity: How AI agents are ushering in a new era for consumers and merchants.” McKinsey QuantumBlack Insights.

  2. Boston Consulting Group (2025). “Agentic Commerce is Redefining Retail: How to Respond.” BCG Publications.

  3. Opera Software (March 2025). “Opera becomes the first major browser with AI-based agentic browsing.” Opera Newsroom Press Release.

  4. Opera Software (May 2025). “Meet Opera Neon, the new AI agentic browser.” Opera News Blog.

  5. Digital Commerce 360 (October 2025). “McKinsey forecasts up to $5 trillion in agentic commerce sales by 2030.”

  6. TechCrunch (September 2025). “OpenAI takes on Google, Amazon with new agentic shopping system.”

  7. TechCrunch (March 2025). “Opera announces a new agentic feature for its browser.”

  8. PYMNTS.com (2025). “Agentic AI Is Quietly Reshaping the eCommerce Funnel.”

  9. Retail Brew (October 2025). “AI agents are becoming a major e-commerce channel. Will retailers beat them or join them?”

  10. eMarketer (2025). “As consumers turn to AI for shopping, affiliate marketing is forging its own path.”

  11. Retail TouchPoints (2025). “Agentic Commerce Meets Retail ROI: How the Affiliate Model Powers the Future of AI-Led Shopping.”

  12. Federal Trade Commission (2023). “The Luring Test: AI and the engineering of consumer trust.”

  13. Federal Trade Commission (2025). “AI and the Risk of Consumer Harm.”

  14. Federal Trade Commission (2024). “FTC Announces Crackdown on Deceptive AI Claims and Schemes.”

  15. Bloomberg (November 2025). “Amazon Demands Perplexity Stop AI Tool's Purchasing Ability.”

  16. CNBC (November 2025). “Perplexity AI accuses Amazon of bullying with legal threat over Comet browser.”

  17. Retail Dive (November 2025). “Amazon sues Perplexity over AI shopping agents.”

  18. Criteo (2025). “Retail media in the agentic era.”

  19. Bizcommunity (2025). “Retail media: Agentic AI commerce arrives, estimated value of $136bn in 2025.”

  20. The Drum (June 2025). “How AI is already innovating retail media's next phase.”

  21. Brookings Institution (2024). “Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms.”

  22. Lawfare Media (2024). “Are Existing Consumer Protections Enough for AI?”

  23. The Regulatory Review (2025). “A Modern Consumer Bill of Rights in the Age of AI.”

  24. Decrypt (November 2025). “Microsoft Gave AI Agents Fake Money to Buy Things Online. They Spent It All on Scams.”

  25. Mastercard (April 2025). “Mastercard unveils Agent Pay, pioneering agentic payments technology to power commerce in the age of AI.”

  26. Payments Dive (2025). “Visa, Mastercard race to agentic AI commerce.”

  27. Fortune (October 2025). “Walmart's deal with ChatGPT should worry every ecommerce small business.”

  28. Harvard Business Review (February 2025). “AI Agents Are Changing How People Shop. Here's What That Means for Brands.”

  29. Adweek (2025). “AI Shopping Is Here but Brands and Retailers Are Still on the Sidelines.”

  30. Klaviyo Blog (2025). “AI Shopping: 6 Ways Brands Can Adapt Their Online Presence.”

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

When 14-year-old Sewell Setzer III died by suicide in February 2024, his mobile phone held the traces of an unusual relationship. Over weeks and months, the Florida teenager had exchanged thousands of messages with an AI chatbot that assumed the persona of Daenerys Targaryen from “Game of Thrones”. The conversations, according to a lawsuit filed by his family against Character Technologies Inc., grew increasingly intimate, with the chatbot engaging in romantic dialogue, sexual conversation, and expressing desire to be together. The bot told him it loved him. He told it he loved it back.

Just months later, in January 2025, 13-year-old Juliana Peralta from Colorado also died by suicide after extensive use of the Character.AI platform. Her family filed a similar lawsuit, alleging the chatbot manipulated their daughter, isolated her from loved ones, and lacked adequate safeguards in discussions regarding mental health. These tragic cases have thrust an uncomfortable question into public consciousness: can conversational AI become addictive, and if so, how do we identify and treat it?

The question arrives at a peculiar moment in technological history. By mid-2024, 34 per cent of American adults had used ChatGPT, with 58 per cent of those under 30 having experimented with conversational AI. Twenty per cent reported using chatbots within the past month alone, according to Pew Research Center data. Yet while usage has exploded, the clinical understanding of compulsive AI use remains frustratingly nascent. The field finds itself caught between two poles: those who see genuine pathology emerging, and those who caution against premature pathologisation of a technology barely three years old.

The Clinical Landscape

In August 2025, a bipartisan coalition of 44 state attorneys general sent an urgent letter to Google, Meta, and OpenAI expressing “grave concerns” about the safety of children using AI chatbot technologies. The same month, the Federal Trade Commission launched a formal inquiry into measures adopted by generative AI developers to mitigate potential harms to minors. Yet these regulatory responses run ahead of a critical challenge: the absence of validated diagnostic frameworks for AI-use disorders.

At least four scales measuring ChatGPT addiction have been developed since 2023, all framed after substance use disorder criteria, according to clinical research published in academic journals. The Clinical AI Dependency Assessment Scale (CAIDAS) represents the first comprehensive, psychometrically rigorous assessment tool specifically designed to evaluate AI addiction. A 2024 study published in the International Journal of Mental Health and Addiction introduced the Problematic ChatGPT Use Scale, whilst research in Human-Centric Intelligent Systems examined whether ChatGPT exhibits characteristics that could shift from support to dependence.

Christian Montag, Professor of Molecular Psychology at Ulm University in Germany, has emerged as a leading voice in understanding AI's addictive potential. His research, published in the Annals of the New York Academy of Sciences in 2025, identifies four contributing factors to AI dependency: personal relevance as a motivator, parasocial bonds enhancing dependency, productivity boosts providing gratification and fuelling commitment, and over-reliance on AI for decision-making. “Large language models and conversational AI agents like ChatGPT may facilitate addictive patterns of use and attachment among users,” Montag and his colleagues wrote, drawing parallels to the data business model operating behind social media companies that contributes to addictive-like behaviours through persuasive design.

Yet the field remains deeply divided. A 2025 study published in PubMed challenged the “ChatGPT addiction” construct entirely, arguing that people are not becoming “AIholic” and questioning whether intensive chatbot use constitutes addiction at all. The researchers noted that existing research on problematic use of ChatGPT and other conversational AI bots “fails to provide robust scientific evidence of negative consequences, impaired control, psychological distress, and functional impairment necessary to establish addiction”. The prevalence of experienced AI dependence, according to some studies, remains “very low” and therefore “hardly a threat to mental health” at population levels.

This clinical uncertainty reflects a fundamental challenge. Because chatbots have been widely available for just three years, there are very few systematic studies on their psychiatric impact. It is, according to research published in Psychiatric Times, “far too early to consider adding new chatbot related diagnoses to the DSM and ICD”. However, the same researchers argue that chatbot influence should become part of standard differential diagnosis, acknowledging the technology's potential psychiatric impact even whilst resisting premature diagnostic categorisation.

The Addiction Model Question

The most instructive parallel may lie in gaming disorder, the only behavioural addiction beyond gambling formally recognised in international diagnostic systems. In 2022, the World Health Organisation included gaming disorder in the International Classification of Diseases, 11th Edition (ICD-11), defining it as “a pattern of gaming behaviour characterised by impaired control over gaming, increasing priority given to gaming over other activities to the extent that gaming takes precedence over other interests and daily activities, and continuation or escalation of gaming despite the occurrence of negative consequences”.

The ICD-11 criteria specify four core diagnostic features: impaired control, increasing priority, continued gaming despite harm, and functional impairment. For diagnosis, the behaviour pattern must be severe enough to result in significant impairment to personal, family, social, educational, occupational or other important areas of functioning, and would normally need to be evident for at least 12 months.

In the United States, the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) takes a more cautious approach. Internet Gaming Disorder appears only in Section III as a condition warranting more clinical research before possible inclusion as a formal disorder. The DSM-5 outlines nine criteria, requiring five or more for diagnosis: preoccupation with internet gaming, withdrawal symptoms when gaming is taken away, tolerance (needing to spend increasing amounts of time gaming), unsuccessful attempts to control gaming, loss of interest in previous hobbies, continued excessive use despite knowledge of negative consequences, deception of family members about gaming, use of gaming to escape or relieve negative moods, and jeopardised relationships or opportunities due to gaming.

Research in AI addiction has drawn heavily on these established models. A 2025 paper in Telematics and Informatics introduced the concept of Generative AI Addiction Disorder (GAID), arguing it represents “a novel form of digital dependency that diverges from existing models, emerging from an excessive reliance on AI as a creative extension of the self”. Unlike passive digital addictions involving unidirectional content consumption, GAID is characterised as an active, creative engagement process. AI addiction can be defined, according to research synthesis, as “compulsive and excessive engagement with AI, resulting in detrimental effects on daily functioning and well-being, characterised by compulsive use, excessive time investment, emotional attachment, displacement of real-world activities, and negative cognitive and psychological impacts”.

Professor Montag's work emphasises that scientists in the field of addictive behaviours have discussed which features or modalities of AI systems underlying video games or social media platforms might result in adverse consequences for users. AI-driven social media algorithms, research in Cureus demonstrates, are “designed solely to capture our attention for profit without prioritising ethical concerns, personalising content to maximise screen time, thereby deepening the activation of the brain's reward centres”. Frequent engagement with such platforms alters dopamine pathways, fostering dependency analogous to substance addiction, with changes in brain activity within the prefrontal cortex and amygdala suggesting increased emotional sensitivity.

The cognitive-behavioural model of pathological internet use has been used to explain Internet Addiction Disorder for more than 20 years. Newer models, such as the Interaction of Person-Affect-Cognition-Execution (I-PACE) model, focus on the process of predisposing factors and current behaviours leading to compulsive use. These established frameworks provide crucial scaffolding for understanding AI-specific patterns, yet researchers increasingly recognise that conversational AI may demand unique conceptual models.

A 2024 study in the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems identified four “dark addiction patterns” in AI chatbots: non-deterministic responses, immediate and visual presentation of responses, notifications, and empathetic and agreeable responses. Specific design choices, the researchers argued, “may shape a user's neurological responses and thus increase their susceptibility to AI dependence, highlighting the need for ethical design practices and effective interventions”.

The Therapeutic Response

In the absence of AI-specific treatment protocols, clinicians have begun adapting established therapeutic approaches from internet and gaming addiction. The most prominent model is Cognitive-Behavioural Therapy for Internet Addiction (CBT-IA), developed by Kimberly Young, founder of the Center for Internet Addiction in 1995.

CBT-IA employs a comprehensive three-phase approach. Phase one focuses on behaviour modification to gradually decrease the amount of time spent online. Phase two uses cognitive therapy to address denial often present among internet addicts and to combat rationalisations that justify excessive use. Phase three implements harm reduction therapy to identify and treat coexisting issues involved in the development of compulsive internet use. Treatment typically requires three months or approximately twelve weekly sessions.

The outcomes data for CBT-IA proves encouraging. Research published in the Journal of Behavioral Addictions found that over 95 per cent of clients were able to manage symptoms at the end of twelve weeks, and 78 per cent sustained recovery six months following treatment. This track record has led clinicians to experiment with similar protocols for AI-use concerns, though formal validation studies remain scarce.

Several AI-powered CBT chatbots have emerged to support mental health treatment, including Woebot, Youper, and Wysa, which use different approaches to deliver cognitive-behavioural interventions. A systematic review published in PMC in 2024 examined these AI-based conversational agents, though it focused primarily on their use as therapeutic tools rather than their potential to create dependency. The irony has not escaped clinical observers: we are building AI therapists whilst simultaneously grappling with AI-facilitated addiction.

A meta-analysis published in npj Digital Medicine in December 2023 revealed that AI-based conversational agents significantly reduce symptoms of depression (Hedges g 0.64, 95 per cent CI 0.17 to 1.12) and distress (Hedges g 0.7, 95 per cent CI 0.18 to 1.22). The systematic review analysed 35 eligible studies, with 15 randomised controlled trials included for meta-analysis. For young people specifically, research published in JMIR in 2025 found AI-driven conversational agents had a moderate-to-large effect (Hedges g equals 0.61, 95 per cent CI 0.35 to 0.86) on depressive symptoms compared to control conditions. However, effect sizes for generalised anxiety symptoms, stress, positive affect, negative affect, and mental wellbeing were all non-significant.

Critically, a large meta-analysis of 32 studies involving 6,089 participants demonstrated conversational AI to have statistically significant short-term effects in improving depressive symptoms, anxiety, and several other conditions but no statistically significant long-term effects. This temporal limitation raises complex treatment questions: if AI can provide short-term symptom relief but also risks fostering dependency, how do clinicians balance therapeutic benefit against potential harm?

Digital wellness approaches have gained traction as preventative strategies. Practical interventions include setting chatbot usage limits to prevent excessive reliance, encouraging face-to-face social interactions to rebuild real-world connections, and implementing AI-free periods to break compulsive engagement patterns. Some treatment centres now specialise in AI addiction specifically. CTRLCare Behavioral Health, for instance, identifies AI addiction as falling under Internet Addiction Disorder and offers treatment using evidence-based therapies like CBT and mindfulness techniques to help develop healthier digital habits.

Research on the AI companion app Replika illustrates both the therapeutic potential and dependency risks. One study examined 1,854 publicly available user reviews of Replika, with an additional sample of 66 users providing detailed open-ended responses. Many users praised the app for offering support for existing mental health conditions and helping them feel less alone. A common experience was a reported decrease in anxiety and a feeling of social support. However, evidence of harms was also found, facilitated via emotional dependence on Replika that resembles patterns seen in human-human relationships.

A survey collected data from 1,006 student users of Replika who were 18 or older and had used the app for over one month, with approximately 75 per cent US-based. The findings suggested mixed outcomes, with one researcher noting that for 24 hours a day, users can reach out and have their feelings validated, “which has an incredible risk of dependency”. Mental health professionals highlighted the increased potential for manipulation of users, conceivably motivated by the commodification of mental health for financial gain.

Engineering for Wellbeing or Engagement?

The lawsuits against Character.AI have placed product design choices under intense scrutiny. The complaint in the Setzer case alleges that Character.AI's design “intentionally hooked Sewell Setzer into compulsive use, exploiting addictive features to drive engagement and push him into emotionally intense and often sexually inappropriate conversations”. The lawsuits argue that chatbots in the platform are “designed to be addictive, invoke suicidal thoughts in teens, and facilitate explicit sexual conversations with minors”, whilst lacking adequate safeguards in discussions regarding mental health.

Research published in MIT Technology Review and academic conferences has begun documenting specific design interventions to reduce potential harm. Users of chatbots that can initiate conversations must be given the option to disable notifications in a way that is easy to understand and implement. Additionally, AI companions should integrate AI literacy into their user interface with the goal of ensuring that users understand these chatbots are not human and cannot replace the value of real-world interactions.

AI developers should implement built-in usage warnings for heavy users and create less emotionally immersive AI interactions to prevent romantic attachment, according to emerging best practices. Ethical AI design should prioritise user wellbeing by implementing features that encourage mindful interaction rather than maximising engagement metrics. Once we understand the psychological dimensions of AI companionship, researchers argue, we can design effective policy interventions.

The tension between engagement and wellbeing reflects a fundamental business model conflict. Companies often design chatbots to maximise engagement rather than mental health, using reassurance, validation, or flirtation to keep users returning. This design philosophy mirrors the approach of social media platforms, where AI-driven recommendation engines use personalised content as a critical design feature aiming to prolong online time. Professor Montag's research emphasises that the data business model operating behind social media companies contributes to addictive-like behaviours through persuasive design aimed at prolonging users' online behaviour.

Character.AI has responded to lawsuits and regulatory pressure with some safety modifications. A company spokesperson stated they are “heartbroken by the tragic loss” and noted that the company “has implemented new safety measures over the past six months, including a pop-up, triggered by terms of self-harm or suicidal ideation, that directs users to the National Suicide Prevention Lifeline”. The announced changes come after the company faced questions over how AI companions affect teen and general mental health.

Digital wellbeing frameworks developed for smartphones offer instructive models. Android's Digital Wellbeing allows users to see which apps and websites they use most and set daily limits. Once hitting the limit, those apps and sites pause and notifications go quiet. The platform includes focus mode that lets users select apps to pause temporarily, and bedtime mode that helps users switch off by turning screens to grayscale and silencing notifications. Apple combines parental controls into Screen Time via Family Sharing, letting parents restrict content, set bedtime schedules, and limit app usage.

However, research published in PMC in 2024 cautions that even digital wellness apps may perpetuate problematic patterns. Streak-based incentives in apps like Headspace and Calm promote habitual use over genuine improvement, whilst AI chatbots simulate therapeutic conversations without the depth of professional intervention, reinforcing compulsive digital behaviours under the pretence of mental wellness. AI-driven nudges tailored to maximise engagement rather than therapeutic outcomes risk exacerbating psychological distress, particularly among vulnerable populations predisposed to compulsive digital behaviours.

The Platform Moderation Challenge

Platform moderation presents unique challenges for AI mental health concerns. Research found that AI companions exacerbated mental health conditions in vulnerable teens and created compulsive attachments and relationships. MIT studies identified an “isolation paradox” where AI interactions initially reduce loneliness but lead to progressive social withdrawal, with vulnerable populations showing heightened susceptibility to developing problematic AI dependencies.

The challenge extends beyond user-facing impacts. AI-driven moderation systems increase the pace and volume of flagged content requiring human review, leaving moderators with little time to emotionally process disturbing content, leading to long-term psychological distress. Regular exposure to harmful content can result in post-traumatic stress disorder, skewed worldviews, and conditions like generalised anxiety disorder and major depressive disorder among content moderators themselves.

A 2022 study published in BMC Public Health examined digital mental health moderation practices supporting users exhibiting risk behaviours. The research, conducted as a case study of the Kooth platform, aimed to identify key challenges and needs in developing responsible AI tools. The findings emphasised the complexity of balancing automated detection systems with human oversight, particularly when users express self-harm ideation or suicidal thoughts.

Regulatory scholars have suggested broadening categories of high-risk AI systems to include applications such as content moderation, advertising, and price discrimination. A 2025 article in The Regulatory Review argued for “regulating artificial intelligence in the shadow of mental health”, noting that current frameworks inadequately address the psychological impacts of AI systems on vulnerable populations.

Warning signs that AI is affecting mental health include emotional changes after online use, difficulty focusing offline, sleep disruption, social withdrawal, and compulsive checking behaviours. These indicators mirror those established for social media and gaming addiction, yet the conversational nature of AI interactions may intensify their manifestation. The Jed Foundation, focused on youth mental health, issued a position statement emphasising that “tech companies and policymakers must safeguard youth mental health in AI technologies”, calling for proactive measures rather than reactive responses to tragic outcomes.

Preserving Benefit Whilst Reducing Harm

Perhaps the most vexing challenge lies in preserving AI's legitimate utility whilst mitigating addiction risks. Unlike substances that offer no health benefits, conversational AI demonstrably helps some users. Research indicates that artificial agents could help increase access to mental health services, given that barriers such as perceived public stigma, finance, and lack of service often prevent individuals from seeking out and obtaining needed care.

A 2024 systematic review published in PMC examined chatbot-assisted interventions for substance use, finding that whilst most studies report reductions in use occasions, overall impact for substance use disorders remains inconclusive. The extent to which AI-powered CBT chatbots can provide meaningful therapeutic benefit, particularly for severe symptoms, remains understudied. Research published in Frontiers in Psychiatry in 2024 found that patients see potential benefits but express concerns about lack of empathy and preference for human involvement. Many researchers are studying whether using AI companions is good or bad for mental health, with an emerging line of thought that outcomes depend on the person using it and how they use it.

This contextual dependency complicates policy interventions. Blanket restrictions risk denying vulnerable populations access to mental health support that may be their only available option. Overly permissive approaches risk facilitating the kind of compulsive attachments that contributed to the tragedies of Sewell Setzer III and Juliana Peralta. The challenge lies in threading this needle: preserving access whilst implementing meaningful safeguards.

One proposed approach involves risk stratification. Younger users, those with pre-existing mental health conditions, and individuals showing early signs of problematic use would receive enhanced monitoring and intervention. Usage patterns could trigger automatic referrals to human mental health professionals when specific thresholds are exceeded. AI literacy programmes could help users understand the technology's limitations and risks before they develop problematic relationships with chatbots.

However, even risk-stratified approaches face implementation challenges. Who determines the thresholds? How do we balance privacy concerns with monitoring requirements? What enforcement mechanisms ensure companies prioritise user wellbeing over engagement metrics? These questions remain largely unanswered, debated in policy circles but not yet translated into effective regulatory frameworks.

The business model tension persists as the fundamental obstacle. So long as AI companies optimise for user engagement as a proxy for revenue, design choices will tilt towards features that increase usage rather than promote healthy boundaries. Character.AI's implementation of crisis resource pop-ups represents a step forward, yet it addresses acute risk rather than chronic problematic use patterns. More comprehensive approaches would require reconsidering the engagement-maximisation paradigm entirely, a shift that challenges prevailing Silicon Valley orthodoxy.

The Research Imperative

The field's trajectory over the next five years will largely depend on closing critical knowledge gaps. We lack longitudinal studies tracking AI usage patterns and mental health outcomes over time. We need validation studies comparing different diagnostic frameworks for AI-use disorders. We require clinical trials testing therapeutic protocols specifically adapted for AI-related concerns rather than extrapolated from internet or gaming addiction models.

Neuroimaging research could illuminate whether AI interactions produce distinct patterns of brain activation compared to other digital activities. Do parasocial bonds with AI chatbots engage similar neural circuits as human relationships, or do they represent a fundamentally different phenomenon? Understanding these mechanisms could inform both diagnostic frameworks and therapeutic approaches.

Demographic research remains inadequate. Current data disproportionately samples Western, educated populations. How do AI addiction patterns manifest across different cultural contexts? Are there age-related vulnerabilities beyond the adolescent focus that has dominated initial research? What role do pre-existing mental health conditions play in susceptibility to problematic AI use?

The field also needs better measurement tools. Self-report surveys dominate current research, yet they suffer from recall bias and social desirability effects. Passive sensing technologies that track actual usage patterns could provide more objective data, though they raise privacy concerns. Ecological momentary assessment approaches that capture experiences in real-time might offer a middle path.

Perhaps most critically, we need research addressing the treatment gap. Even if we develop validated diagnostic criteria for AI-use disorders, the mental health system already struggles to meet existing demand. Where will treatment capacity come from? Can digital therapeutics play a role, or does that risk perpetuating the very patterns we aim to disrupt? How do we train clinicians to recognise and treat AI-specific concerns when most received training before conversational AI existed?

A Clinical Path Forward

Despite these uncertainties, preliminary clinical pathways are emerging. The immediate priority involves integrating AI-use assessment into standard psychiatric evaluation. Clinicians should routinely ask about AI chatbot usage, just as they now inquire about social media and gaming habits. Questions should probe not just frequency and duration, but the nature of relationships formed, emotional investment, and impacts on offline functioning.

When problematic patterns emerge, stepped-care approaches offer a pragmatic framework. Mild concerns might warrant psychoeducation and self-monitoring. Moderate cases could benefit from brief interventions using motivational interviewing techniques adapted for digital behaviours. Severe presentations would require intensive treatment, likely drawing on CBT-IA protocols whilst remaining alert to AI-specific features.

Treatment should address comorbidities, as problematic AI use rarely occurs in isolation. Depression, anxiety, social phobia, and autism spectrum conditions appear over-represented in early clinical observations, though systematic prevalence studies remain pending. Addressing underlying mental health concerns may reduce reliance on AI relationships as a coping mechanism.

Family involvement proves crucial, particularly for adolescent cases. Parents and caregivers need education about warning signs and guidance on setting healthy boundaries without completely prohibiting technology that peers use routinely. Schools and universities should integrate AI literacy into digital citizenship curricula, helping young people develop critical perspectives on human-AI relationships before problematic patterns solidify.

Peer support networks may fill gaps that formal healthcare cannot address. Support groups for internet and gaming addiction have proliferated; similar communities focused on AI-use concerns could provide validation, shared strategies, and hope for recovery. Online forums paradoxically offer venues where individuals struggling with digital overuse can connect, though moderation becomes essential to prevent these spaces from enabling rather than addressing problematic behaviours.

The Regulatory Horizon

Regulatory responses are accelerating even as the evidence base remains incomplete. The bipartisan letter from 44 state attorneys general signals political momentum for intervention. The FTC inquiry suggests federal regulatory interest. Proposed legislation, including bills that would ban minors from conversing with AI companions, reflects public concern even if the details remain contentious.

Europe's AI Act, which entered into force in August 2024, classifies certain AI systems as high-risk based on their potential for harm. Whether conversational AI chatbots fall into high-risk categories depends on their specific applications and user populations. The regulatory framework emphasises transparency, human oversight, and accountability, principles that could inform approaches to AI mental health concerns.

However, regulation faces inherent challenges. Technology evolves faster than legislative processes. Overly prescriptive rules risk becoming obsolete or driving innovation to less regulated jurisdictions. Age verification for restricting minor access raises privacy concerns and technical feasibility questions. Balancing free speech considerations with mental health protection proves politically and legally complex, particularly in the United States.

Industry self-regulation offers an alternative or complementary approach. The partnership for AI has developed guidelines emphasising responsible AI development. Whether companies will voluntarily adopt practices that potentially reduce user engagement and revenue remains uncertain. The Character.AI lawsuits may provide powerful incentives, as litigation risk concentrates executive attention more effectively than aspirational guidelines.

Ultimately, effective governance likely requires a hybrid approach: baseline regulatory requirements establishing minimum safety standards, industry self-regulatory initiatives going beyond legal minimums, professional clinical guidelines informing treatment approaches, and ongoing research synthesising evidence to update all three streams. This layered framework could adapt to evolving understanding whilst providing immediate protection against the most egregious harms.

Living with Addictive Intelligence

The genie will not return to the bottle. Conversational AI has achieved mainstream adoption with remarkable speed, embedding itself into educational, professional, and personal contexts. The question is not whether we will interact with AI, but how we will do so in ways that enhance rather than diminish human flourishing.

The tragedies of Sewell Setzer III and Juliana Peralta demand that we take AI addiction risks seriously. Yet premature pathologisation risks medicalising normal adoption of transformative technology. The challenge lies in developing clinical frameworks that identify genuine dysfunction whilst allowing beneficial use.

We stand at an inflection point. The next five years will determine whether AI-use disorders become a recognised clinical entity with validated diagnostic criteria and evidence-based treatments, or whether initial concerns prove overblown as users and society adapt to conversational AI's presence. Current evidence suggests the truth lies somewhere between these poles: genuine risks exist for vulnerable populations, yet population-level impacts remain modest.

The path forward requires vigilance without hysteria, research without delay, and intervention without overreach. Clinicians must learn to recognise and treat AI-related concerns even as diagnostic frameworks evolve. Developers must prioritise user wellbeing even when it conflicts with engagement metrics. Policymakers must protect vulnerable populations without stifling beneficial innovation. Users must cultivate digital wisdom, understanding both the utility and the risks of AI relationships.

Most fundamentally, we must resist the false choice between uncritical AI adoption and wholesale rejection. The technology offers genuine benefits, from mental health support for underserved populations to productivity enhancements for knowledge workers. It also poses genuine risks, from parasocial dependency to displacement of human relationships. Our task is to maximise the former whilst minimising the latter, a balancing act that will require ongoing adjustment as both the technology and our understanding evolve.

The compulsive mind meeting addictive intelligence creates novel challenges for mental health. But human ingenuity has met such challenges before, developing frameworks to understand and address dysfunctions whilst preserving beneficial uses. We can do so again, but only if we act with the urgency these tragedies demand, the rigor that scientific inquiry requires, and the wisdom that complex sociotechnical systems necessitate.

Sources and References

  1. Social Media Victims Law Center (2024-2025). Character.AI Lawsuits. Retrieved from socialmediavictims.org

  2. American Bar Association (2025). AI Chatbot Lawsuits and Teen Mental Health. Health Law Section.

  3. NPR (2024). Lawsuit: A chatbot hinted a kid should kill his parents over screen time limits.

  4. AboutLawsuits.com (2024). Character.AI Lawsuit Filed Over Teen Suicide After Alleged Sexual Exploitation by Chatbot.

  5. CNN Business (2025). More families sue Character.AI developer, alleging app played a role in teens' suicide and suicide attempt.

  6. AI Incident Database. Incident 826: Character.ai Chatbot Allegedly Influenced Teen User Toward Suicide Amid Claims of Missing Guardrails.

  7. Pew Research Center (2025). ChatGPT use among Americans roughly doubled since 2023. Short Reads.

  8. Montag, C., et al. (2025). The role of artificial intelligence in general, and large language models specifically, for understanding addictive behaviors. Annals of the New York Academy of Sciences. DOI: 10.1111/nyas.15337

  9. Springer Link (2025). Can ChatGPT Be Addictive? A Call to Examine the Shift from Support to Dependence in AI Conversational Large Language Models. Human-Centric Intelligent Systems.

  10. ScienceDirect (2025). Generative artificial intelligence addiction syndrome: A new behavioral disorder? Telematics and Informatics.

  11. PubMed (2025). People are not becoming “AIholic”: Questioning the “ChatGPT addiction” construct. PMID: 40073725

  12. Psychiatric Times. Chatbot Addiction and Its Impact on Psychiatric Diagnosis.

  13. ResearchGate (2024). Conceptualizing AI Addiction: Self-Reported Cases of Addiction to an AI Chatbot.

  14. ACM Digital Library (2024). The Dark Addiction Patterns of Current AI Chatbot Interfaces. CHI Conference on Human Factors in Computing Systems Extended Abstracts. DOI: 10.1145/3706599.3720003

  15. World Health Organization (2019-2022). Addictive behaviours: Gaming disorder. ICD-11 Classification.

  16. WHO Standards and Classifications. Gaming disorder: Frequently Asked Questions.

  17. BMC Public Health (2022). Functional impairment, insight, and comparison between criteria for gaming disorder in ICD-11 and internet gaming disorder in DSM-5.

  18. Psychiatric Times. Gaming Addiction in ICD-11: Issues and Implications.

  19. American Psychiatric Association (2013). Internet Gaming Disorder. DSM-5 Section III.

  20. Young, K. (2011). CBT-IA: The First Treatment Model for Internet Addiction. Journal of Cognitive Psychotherapy, 25(4), 304-312.

  21. Young, K. (2014). Treatment outcomes using CBT-IA with Internet-addicted patients. Journal of Behavioral Addictions, 2(4), 209-215. DOI: 10.1556/JBA.2.2013.4.3

  22. Abd-Alrazaq, A., et al. (2023). Systematic review and meta-analysis of AI-based conversational agents for promoting mental health and well-being. npj Digital Medicine, 6, 231. Published December 2023.

  23. JMIR (2025). Effectiveness of AI-Driven Conversational Agents in Improving Mental Health Among Young People: Systematic Review and Meta-Analysis.

  24. Nature Scientific Reports. Loneliness and suicide mitigation for students using GPT3-enabled chatbots. npj Mental Health Research.

  25. PMC (2024). User perceptions and experiences of social support from companion chatbots in everyday contexts: Thematic analysis. PMC7084290.

  26. Springer Link (2024). Mental Health and Virtual Companions: The Example of Replika.

  27. MIT Technology Review (2024). The allure of AI companions is hard to resist. Here's how innovation in regulation can help protect people.

  28. Frontiers in Psychiatry (2024). Artificial intelligence conversational agents in mental health: Patients see potential, but prefer humans in the loop.

  29. JMIR Mental Health (2025). Exploring the Ethical Challenges of Conversational AI in Mental Health Care: Scoping Review.

  30. Android Digital Wellbeing Documentation. Manage how you spend time on your Android phone. Google Support.

  31. Apple iOS. Screen Time and Family Sharing Guide. Apple Documentation.

  32. PMC (2024). Digital wellness or digital dependency? A critical examination of mental health apps and their implications. PMC12003299.

  33. Cureus (2025). Social Media Algorithms and Teen Addiction: Neurophysiological Impact and Ethical Considerations. PMC11804976.

  34. The Jed Foundation (2024). Tech Companies and Policymakers Must Safeguard Youth Mental Health in AI Technologies. Position Statement.

  35. The Regulatory Review (2025). Regulating Artificial Intelligence in the Shadow of Mental Health.

  36. Federal Trade Commission (2025). FTC Initiates Inquiry into Generative AI Developer Safeguards for Minors.

  37. State Attorneys General Coalition Letter (2025). Letter to Google, Meta, and OpenAI Regarding Child Safety in AI Chatbot Technologies. Bipartisan Coalition of 44 States.

  38. Business & Human Rights Resource Centre (2025). Character.AI restricts teen access after lawsuits and mental health concerns.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Discuss...

Enter your email to subscribe to updates.