SmarterArticles

There is a particular kind of professional disappearance that does not show up in the unemployment figures. The illustrator still has a desk. The translator still has a website. The session musician still owns a violin. None of them have been fired. None of them have been informed, in any official capacity, that their occupation is being phased out. Their names remain on the same freelance registers, the same union rolls, the same tax filings as last year. And yet, quietly, almost imperceptibly at first, the floor underneath their work has begun to give way.

This is the strangest economic story of the decade, and it is unfolding without a moment of high drama. There are no factory closures, no mass layoffs, no town-square photographs of redundant workers carrying cardboard boxes. Instead, there is a slow, grinding compression of the rates a cover illustrator can charge for a magazine commission, a slight but stubborn drop in the volume of subtitling work coming through the agency in São Paulo, a pause in the email chain from the German publisher who used to commission a translator in Lagos every six months. Each individual moment is deniable. Taken together, they describe a structural rearrangement of the creative economy that the existing policy vocabulary, fixated on automation and job displacement, is not equipped to name.

UNESCO's flagship report on creativity and digital transformation, published on 18 February 2026 as the fourth edition of its Re|Shaping Policies for Creativity series, attempted to put a number on the rearrangement. Drawing on data from more than 120 countries, the report projected that creators worldwide are on course to lose up to 24 per cent of their revenues by 2028 as a direct consequence of generative AI, with music creators bearing the heaviest exposure and audiovisual creators close behind. An accompanying analysis published the same week by Inter Press Service amplified the geographical dimension of the problem, observing that the income losses are falling most heavily on freelance and self-employed creators in the global south, layering a new digital injury on top of long-standing inequalities in the cultural economy.

The numbers are striking, but they are not the most interesting part of the story. The most interesting part is the mechanism. This is not, in the conventional sense, a tale of automation. The translator working from Yoruba to Spanish has not been replaced by a translation engine in any specific role. She is still the translator on her own letterhead. What has happened is that the demand curve she used to live on has been displaced, almost overnight, by something that produces an approximate substitute for her output at near-zero marginal cost. The publishers, the production companies, the marketing agencies who used to commission her have not declared that they are switching to machines. They have simply stopped commissioning at the same volume, or they have begun negotiating from a position that assumes her labour competes with a free alternative. There is no transition point. There is no redundancy notice. There is no clean moment at which the policy concept of retraining becomes applicable, because the person is still doing the job. It is the job, as a paid activity, that is being hollowed out.

This distinction matters, and not only as semantics. The entire architecture of late twentieth-century labour policy, from unemployment insurance to active labour market programmes, was built on the assumption that economic dislocation comes with a clear event horizon. Someone is hired. Someone is laid off. Someone is retrained. Someone is rehired. Generative AI breaks the model not by accelerating the cycle but by detaching the harm from any of these events. The freelance writer is never laid off because she was never on a payroll. The illustrator does not get a redundancy package because there was no employer. The market she sold into has simply contracted, and the policy instruments designed to catch falling workers were not built to catch a falling market.

If the diagnosis is right, then the question that follows is the one the UNESCO report, the IPS News analysis, and a fast-growing literature on AI and the creative economy have all begun to circle. What policy instruments, beyond copyright reform, can address the harm? And who, in any meaningful sense, is responsible for the structural losses already under way?

The market, not the job, is the unit of analysis

To understand why this matters, it helps to look at what the existing debate is mostly about. Almost every legal and political fight currently being waged on behalf of creative workers concerns the upstream side of the AI value chain: whether AI labs should be allowed to train models on copyrighted works without permission, whether they should pay licensing fees for ingesting them, whether opt-out registers should be opt-in, whether the European Union's text-and-data-mining exception should narrow or expand. These are real and important fights. The Copyright Licensing Agency in the United Kingdom announced its Generative AI Training Licence to allow collective compensation for ingestion. The US Copyright Office has explored extended collective licensing on the Danish model. The Court of Justice of the European Union held its first hearing on generative AI and copyright in March 2026 in Like Company v Google, a case that may reshape the press publishers' right across the bloc.

Yet copyright, in any of its forms, only addresses one half of the harm UNESCO identified. The CISAC global economic study published in late 2024, conducted by the consultancy PMP Strategy on behalf of the international confederation of authors' societies, was unusually clear about this. The losses creators face split into two distinct streams. The first stream is the value of their existing works being scraped into training data without consent or remuneration. Copyright reform, however imperfectly, is built to address that. The second stream is the substitution effect: AI-generated outputs competing in the market against human-made works, depressing rates and shrinking commissions. Copyright, as currently understood, has very little to say about that second stream. Even a perfectly negotiated training licence does not change the fact that, once the model is trained, the marginal cost of producing a passable cover illustration falls towards zero, and the rate the illustrator can charge falls with it.

This is the harder problem, and it is the one to which the policy debate is only beginning to turn. The question is no longer simply how to compensate creators for the use of their work in training. It is how to sustain a market for human creative labour at all, when the marginal product of that labour can be approximated, however crudely, by a system that does not pay rent, sleep, or eat.

A taxonomy of instruments beyond copyright

The good news, if it can be called that, is that the policy toolkit available to address market collapse is broader than the copyright debate sometimes suggests. The bad news is that almost every instrument involves redistributing money from somebody who currently does not pay to somebody who currently does not receive, and the political economy of that redistribution is brutal.

AI cultural levies

The most direct proposal, and the one that has gained the most traction in European policy circles over the past year, is a levy on AI systems pegged to their commercial use of human cultural output. Arthur Mensch, the chief executive of the French AI lab Mistral, surprised many observers in 2025 when he publicly endorsed a revenue-based levy of roughly 1 to 1.5 per cent on commercial providers placing AI models on the European market, with funds channelled into a central pot to support cultural creation. The Mistral proposal is hardly altruistic; it would also, conveniently, harden a continental moat against American and Chinese model providers. But its underlying logic is sound, and it draws on a legal heritage that goes back six decades.

The German Copyright Act of 1965 introduced the first private copying levy, attaching a small charge to the cost of devices and media that allowed users to duplicate protected works. The principle was that where copying is structurally uncontrollable, levy-funded compensation, distributed by collective management organisations, is a more workable alternative than litigation. Generative AI presents an almost perfect analogue. The training and inference of a foundation model, at any meaningful scale, is structurally beyond the reach of one-by-one licensing. A statutory levy on commercial AI services, collected by reformed collective management organisations and distributed to creators on a metered basis, would close the substitution-side gap that copyright cannot reach. It would also avoid the worst pathology of contemporary copyright reform, which is that platforms can outspend rights holders in any line-by-line negotiation.

There are real objections. A levy must be set high enough to matter and low enough not to suppress useful applications. It must be administered by institutions trusted by both creators and developers, which is not how the existing collective management landscape is universally regarded. And it must avoid becoming a moat for incumbents who can absorb a 1.5 per cent levy more easily than a research lab in Nairobi or a co-operative model in Buenos Aires. None of these objections is fatal. All of them require institutional design rather than policy retreat.

Statutory remuneration rights

A close cousin of the levy approach is the statutory remuneration right, which decouples permission from payment. Under such a regime, AI developers might be permitted to train on lawfully accessible works without negotiating individual licences, but they would owe a non-waivable payment to authors through a collective body. The European Parliament's commissioned study on generative AI and copyright, published in 2025, examined this possibility in detail. Springer Nature's International Review of Intellectual Property and Competition Law has run a series of analyses, by scholars including Christophe Geiger, arguing that a statutory remuneration right grounded in fundamental rights to participate in cultural life could be the most workable foundation for a new compact.

The advantage of a statutory remuneration right over a pure levy is that it sits more comfortably within the existing copyright framework. The disadvantage is that it still ties payment to the use of identifiable works, which means it primarily addresses the ingestion side rather than the substitution side. Combined with a levy, however, it begins to look like a serviceable architecture.

Public commissioning and the Irish experiment

While the levy debate continues, a quieter experiment has been running in Ireland since 2022. The Basic Income for the Arts scheme, originally a three-year pilot, paid 2,000 randomly selected artists 325 euros a week, regardless of output. The Irish Department of Culture, Communications and Sport opened applications for the 2026 to 2029 successor scheme in April 2026, and a published cost-benefit analysis found that for every euro invested, society received a return of 1.39 euros, a number that has been disputed in the Irish press but has not been seriously dislodged.

The Irish scheme is not a response to AI. It was designed to address the chronic under-monetisation of cultural work in a digital economy that had already eroded the freelance commercial base before generative models arrived. But its logic transfers cleanly. If the market for creative output is being structurally compressed by a technology whose externalities are not internalised, then a state instrument that decouples income from market success becomes more, not less, defensible. A universal creative income, scaled to the working population of practising artists in any given country, would stand to working creatives roughly as agricultural support payments stand to small farmers facing global commodity competition. It is unromantic, slightly bureaucratic, and precisely the kind of thing that has historically allowed cultural production to survive market shocks.

The political objection is that it looks like a cultural welfare state. The substantive objection is that, depending on how it is administered, it can entrench the credentialing power of arts councils and reproduce existing gatekeeping. Both are genuine. Neither is decisive against an instrument that, in Ireland at least, has empirical results to its name.

Public procurement as creative-economy policy

A surprisingly underused lever is the purchasing power of governments themselves. Public sector bodies are, in aggregate, among the largest commissioners of design, illustration, translation, audiovisual production, and music in most economies. The US General Services Administration's draft AI procurement clause, the December 2025 OMB memorandum M-26-04, the United Kingdom's Procurement Policy Note 017 from February 2025, and California's executive order on AI vendor certification signed by Governor Gavin Newsom in April 2026 all introduce disclosure obligations for AI-generated content within government contracts. None of them, however, goes the further step of creating a procurement preference for human creative work in cultural production funded by public money.

A modest reform would be to require, for example, that any public broadcaster, national museum, ministry of culture, or city government commissioning creative output beyond a defined threshold use human creators where reasonably possible, with transparent disclosure when generative tools are used. This costs the state more, in the short term, than letting procurement officers chase the cheapest bid. It also creates a stable demand floor for working creatives and signals, with the kind of clarity that markets respond to, that public money will not be deployed to accelerate the collapse of the freelance creative class. India's labelling thresholds for AI-generated visual and audio content, and the EU AI Act's transparency requirements, are early sketches of the disclosure architecture this would require.

Collective bargaining as economic infrastructure

The 2023 agreements between the Writers Guild of America, the Screen Actors Guild and the Alliance of Motion Picture and Television Producers are, for all their imperfections, the most concrete demonstration that collective bargaining can produce workable rules around AI in creative work. The WGA contract specifies that AI-generated material cannot be considered literary material for credit purposes and gives writers the right to refuse to use AI tools, while preserving their ability to challenge the use of writers' work to train AI. The SAG-AFTRA contract distinguishes digital replicas of identifiable performers from synthetic performers built from no individual likeness, and creates compensation and consent obligations around both.

These provisions are not perfect. The 2024 SAG-AFTRA video game performer strike, which ran for many months over precisely these AI consent and compensation issues in the interactive sector, demonstrated how quickly a contract negotiated for one segment of the industry begins to look incomplete when applied to another. But the agreements demonstrate the principle that collective bargaining can do work that copyright cannot, by setting industry-wide floors on consent, attribution, and compensation that apply regardless of the specific upstream provenance of any given AI output.

The implication for creative workers outside the unionised entertainment sector is uncomfortable but unavoidable. The freelance illustrator, the literary translator, the independent musician, the documentary editor often have no equivalent collective body. Building one, on a national or transnational basis, becomes infrastructure rather than ideology. The European Federation of Journalists, the European Writers' Council, the International Federation of Translators, and the Concerts Promoters Association are all operating in this space, as are emerging co-operative models among illustrators in continental Europe. Any serious policy response to the structural compression of creative labour markets needs to take seriously the question of how to fund and support these bodies.

Sovereign wealth approaches and creative commons funds

A more radical proposal, occasionally floated in policy circles and yet to find a serious political champion, is the sovereign wealth approach. The argument runs that the corpus of human cultural output ingested by foundation models is a non-rival public resource analogous to a national fishery or a hydrocarbon basin. Where states extract rents from companies exploiting natural resources, the rents fund either current public expenditure or, in the Norwegian case, an intergenerational sovereign wealth fund. By analogy, a national or supranational creative commons fund, capitalised by an ingestion-based levy on commercial AI training and operation, could be invested to provide perpetual support for cultural production.

The sovereign wealth analogy is imperfect. Cultural output is not extracted from a finite reservoir; it is generated, continuously, by living people whose labour the fund is meant to compensate. But the analogy is useful precisely because it forces a recognition that the value flowing into AI labs from human cultural output is, in macroeconomic terms, an unpriced externality of historic scale. The OECD's 2025 report on intellectual property and AI training data raised, without endorsing, the question of whether the absence of pricing on this externality represents a market failure that justifies non-market correction. That is exactly the conceptual frame a sovereign wealth approach would adopt.

The global south is not a footnote

Any honest reckoning with the policy space has to confront the dimension that the IPS News analysis put squarely on the table: the income losses from generative AI are not falling evenly across geography. They are falling disproportionately on freelance and self-employed creators in the global south. UNESCO's data, repeated in the Re|Shaping Policies for Creativity report, is sobering. In developed economies, 67 per cent of people possess essential digital skills; in developing economies the figure is 28 per cent. Cultural and creative leadership in developed countries has reached 64 per cent women in some institutional categories; in developing nations it is 30 per cent. Public funding for culture sits below 0.6 per cent of global GDP and is projected to decline. Only 61 per cent of countries surveyed have intellectual property frameworks UNESCO considers adequate.

These structural baselines were already producing inequality. Generative AI compounds them. Viviana Rangel, a Colombian independent expert quoted in the IPS News analysis, framed the problem in a sentence: the region does not produce this kind of technology; it consumes it. The economic flow runs in one direction. Cultural workers in Lagos, Lima, Manila, and Karachi see their commissions evaporate as European and North American clients route work through models trained on a corpus from which their own contributions are statistically marginal. The royalties and rents from those models, when they exist at all, flow to collective management organisations and rights holders concentrated in the North Atlantic.

This dimension has implications for every instrument discussed above. A European AI cultural levy, however well designed, will tend to recapture funds from European AI providers and recycle them through European collective management organisations to predominantly European creators. That is not necessarily wrong, but it is not a global solution. The CISAC study's projection of 22 billion euros in cumulative losses to music and audiovisual creators globally over five years is a number that needs distributional analysis. Where do the losses fall? Where do the few gains fall? UNESCO's framing of the problem as a global development issue, rather than a North Atlantic intellectual property dispute, opens space for instruments that the copyright debate alone would not generate.

The most credible candidates, at this point, are international transfers built into the supranational architecture of AI governance. A share of any revenues raised through training data taxes, statutory remuneration rights, or AI cultural levies should be directed, by treaty or legislative carve-out, to a global fund supporting creators in the regions where the harm falls hardest. This is not charity. It is restitution for an extraction whose proceeds are presently retained by a small group of companies whose corpora include cultural output from every continent. UNESCO, by virtue of its mandate over cultural diversity and the global character of its 120-country reporting, is the obvious institutional vehicle, although the World Intellectual Property Organisation and the United Nations Conference on Trade and Development have credible roles to play.

The harder version of the global south argument concerns sovereignty. If a Senegalese government wants to protect its translators, illustrators, and musicians from market compression caused by foundation models trained largely outside its borders, what tools does it have? The honest answer is: not many, in the short term. National AI levies on a small market produce modest revenue. National copyright reform reaches AI labs only weakly. National public commissioning and basic income programmes are constrained by fiscal capacity. This is one reason why the architecture of any serious policy response has to be partly supranational. It is also why policy frameworks that treat the global south as an afterthought, or that solve the problem of the European illustrator while leaving the Lagos illustrator untouched, will be morally and politically unstable.

Who bears responsibility?

The question of responsibility is the one most likely to be flattened by political slogans, so it is worth taking slowly. There are at least five candidates for the moral and economic ledger, and a serious policy framework needs to assign weight to each rather than collapsing them into a single villain.

The AI labs themselves are the most obvious candidate. They built the systems whose outputs are compressing creative labour markets. They trained the models on corpora they did not pay for, in most cases, and they continue to extract economic rent from those corpora at scale. The defence offered by lab leadership tends to combine the argument that training on publicly available content is fair use with the argument that the productivity gains from foundation models will, over time, raise everyone's incomes including creators'. Both arguments are contestable. The fair use claim is being litigated across multiple jurisdictions. The productivity-spillover claim has, so far, generated almost no observable benefit for the working creators whose markets are contracting fastest. Responsibility, on any plausible reading, sits substantially with the labs, and the policy instruments above should be priced accordingly.

The platforms that distribute creative work are a second locus. Streaming services that ingest AI-generated music into the same recommendation streams as human-made music; stock image platforms that have become, in some categories, predominantly synthetic; commissioning marketplaces that allow buyers to specify AI-generated drafts as deliverables. Each of these platforms makes choices about how to label, reward, and surface human versus synthetic output. UNESCO's report observes that opaque algorithms and platform consolidation are themselves part of the structural undercutting. Procurement-style transparency requirements, content provenance standards, and labelling rules are the relevant instruments here, and platforms are properly the subjects of them.

Governments are the third candidate. They license the regulatory environment within which labs and platforms operate, and they hold the fiscal and statutory authority to introduce levies, statutory remuneration rights, public commissioning rules, and basic income schemes. They also have the slowest reflexes. The EU AI Act, the UK text-and-data-mining consultation, the patchwork of state-level AI laws in the United States, and the regulatory regimes emerging across Asia and Latin America operate on time horizons measured in years; market compression is occurring in quarters. Responsibility for that gap falls on legislatures and on public agencies that have not yet pivoted from a copyright-only frame to a market-structure frame.

The fourth candidate is the end user: the corporate or individual buyer who chooses an AI-generated cover, a synthetic voice-over, or an automated translation over a human alternative. Moral responsibility here is real but limited. Buyers respond to prices, and prices are an artefact of upstream institutional architecture; no buyer can plausibly be expected to internalise an externality the policy regime has not bothered to price. End-user weight matters most in the public sector, in editorial institutions whose readers care about provenance, and in industries where reputation rewards transparency. Disclosure rules, labelling standards, and provenance technologies make this responsibility legible and therefore actionable.

The fifth candidate, the most diffuse and the least talked about, is the public itself, conceived as the political constituency that decides whether to treat creative labour as economically valuable enough to defend. The Irish basic income scheme exists because Irish politics decided it should. The WGA and SAG-AFTRA agreements exist because audiences, in the end, did not want to consume an industry whose writers and performers were being squeezed past tolerance. The slow shift in European policy thinking towards an AI cultural levy exists because European publics and their elected representatives have, for now, not lost their attachment to the idea that cultural work is a public good worth supporting through institutional design. That political attachment is not automatic. It can erode. Where it erodes, the labs and platforms set the frame.

What a serious settlement would look like

A serious policy settlement, on the analysis above, is not a single instrument but a stack. Copyright reform sits at the bottom of the stack, addressing the upstream ingestion question that copyright is institutionally suited to handle. Statutory remuneration rights and AI cultural levies sit above it, addressing the substitution-side compression that copyright cannot reach. Public commissioning rules and procurement preferences sit above those, deploying the state's purchasing power to maintain a demand floor for human creative work. Universal creative income schemes, on the Irish model, sit above those, decoupling baseline livelihood from market success. Collective bargaining and trade-association infrastructure sits across the stack, providing the institutional capacity for creators to negotiate consent, attribution, and compensation in real-time as the technology evolves. International transfers, capitalised by the levies and routed through multilateral cultural institutions, sit on top, addressing the global south dimension that no national policy can solve alone.

No single country has the full stack today. Ireland has a basic income scheme. France and the European Union are debating a levy. The United Kingdom has a collective licensing prototype. Spain has labelling rules. Germany has a private copying levy heritage that could be retrofitted. The United States has the WGA and SAG-AFTRA agreements, the GSA procurement clause, and California's vendor certification regime. India has labelling thresholds. UNESCO has 8,100 catalogued policy measures across 120 countries. The pieces exist; what is missing is the integration.

This is, in some sense, the unromantic conclusion. The problem that generative AI has created in the creative economy is not, primarily, a problem that demands a new philosophical framework, although philosophical frameworks help. It is a problem that demands the assembly of a known set of instruments into a coherent stack, with serious institutional design and credible enforcement, and with explicit redistribution towards the creators and regions where the harm is concentrated. The political difficulty of that assembly is high. The intellectual difficulty is lower than the public debate sometimes implies.

The alternative, if no such stack is built, is reasonably easy to describe. The compression continues. The freelance creative class, in the global north and more sharply in the global south, contracts. The cultural production that survives concentrates among those with independent means, institutional employment, or audiences large enough to bypass the compression. The texture of cultural output narrows in ways that are hard to see in real time but legible in retrospect, in the same way the disappearance of regional newspapers became legible only after the fact. The labs and platforms continue to capture the rents from a corpus they did not build. Lodovico Folin-Calabi, the UNESCO director who told the press at the report's launch that the world must critically examine how these technologies are deployed and whose voices are represented, may turn out to have been describing not a turning point but a wake.

Whether the settlement gets built is, finally, a political question rather than a technical one. The technical question has answers. The political question, whether public, labs, platforms, and governments collectively decide that the structural losses already under way are worth correcting, has only the answers a generation chooses to give. The 24 per cent number is a forecast. It is also a decision, not yet made.

---

References & Sources

1. UNESCO. Re|Shaping Policies for Creativity: Making Creativity Count. Fourth global monitoring report. Launched 18 February 2026. https://www.unesco.org/en/articles/creators-face-projected-global-revenue-losses-24-2028-new-unesco-report-shows
2. UNESCO Diversity of Cultural Expressions. “Making creativity matter: UNESCO's latest Re|Shaping Policies for Creativity report calls for bold policy action.” February 2026. https://www.unesco.org/creativity/en/articles/making-creativity-matter-unescos-latest-reshaping-policies-creativity-report-calls-bold-policy
3. United Nations News. “Artists face steep income decline due to AI, UNESCO finds.” 18 February 2026. https://news.un.org/en/story/2026/02/1166989
4. Inter Press Service. “Generative AI Could Deepen Inequality, Revenue Losses in Creative Industries.” February 2026. https://www.ipsnews.net/2026/02/generative-ai-could-deepen-inequality-revenue-losses-in-creative-industries/
5. CISAC and PMP Strategy. Global economic study shows human creators' future at risk from generative AI. December 2024. https://www.cisac.org/Newsroom/news-releases/global-economic-study-shows-human-creators-future-risk-generative-ai
6. AI Business / IT Pro. “Mistral CEO Arthur Mensch calls for AI cultural levy in Europe.” 2025. https://www.itpro.com/technology/artificial-intelligence/mistral-ceo-calls-for-ai-cultural-levy
7. Copyright Licensing Agency. “CLA announces development of Generative AI Training Licence.” 2025. https://cla.co.uk/development-of-cla-generative-ai-licence/
8. United States Copyright Office. Copyright and Artificial Intelligence, Part 3: Generative AI Training (pre-publication report). 2025. https://www.copyright.gov/ai/Copyright-and-Artificial-Intelligence-Part-3-Generative-AI-Training-Report-Pre-Publication-Version.pdf
9. OECD. Intellectual property issues in artificial intelligence trained on scraped data. 2025. https://www.oecd.org/en/publications/intellectual-property-issues-in-artificial-intelligence-trained-on-scraped-data_d5241a23-en.html
10. Geiger, Christophe et al. “Generative AI, Digital Constitutionalism and Copyright: Towards a Statutory Remuneration Right grounded in Fundamental Rights.” Kluwer Copyright Blog and International Review of Intellectual Property and Competition Law (Springer Nature). 2025. https://link.springer.com/article/10.1007/s40319-025-01569-6
11. European Parliament. Generative AI and Copyright: Training, Creation, Regulation. Study commissioned by the JURI Committee. 2025. https://www.europarl.europa.eu/RegData/etudes/STUD/2025/774095/IUST_STU(2025)774095_EN.pdf
12. Government of Ireland, Department of Culture, Communications and Sport. “Basic Income for the Arts Scheme 2026-2029 Guidelines for Application.” April 2026. https://www.gov.ie/en/department-of-culture-communications-and-sport/publications/basic-income-for-the-arts-scheme-2026-2029-guidelines-for-application/
13. CNN Style. “Ireland is paying artists a basic income in a pioneering scheme.” 12 February 2026. https://www.cnn.com/2026/02/12/style/ireland-artists-basic-income-intl-scli
14. Perkins Coie. “Generative AI in Movies and TV: How the 2023 SAG-AFTRA and WGA Contracts Address Generative AI.” 2023. https://perkinscoie.com/insights/blog/generative-ai-movies-and-tv-how-2023-sag-aftra-and-wga-contracts-address-generative
15. Center for Democracy and Technology. “New WGA Labor Agreement Gives Hollywood Writers Important Protections in the Era of AI.” 2023. https://cdt.org/insights/new-wga-labor-agreement-gives-hollywood-writers-important-protections-in-the-era-of-ai/
16. Bird & Bird. “Like Company v Google: CJEU Holds First-Ever Hearing on Generative AI and Copyright on 10 March 2026.” 2026. https://www.twobirds.com/en/insights/2026/like-company-v-google-cjeu-holds-first-ever-hearing-on-generative-ai-and-copyright
17. Gibson Dunn. “GSA AI Procurement Rules Would Introduce New Disclosure and Use-Rights Requirements for Federal Contractors.” 2025. https://www.gibsondunn.com/gsa-ai-procurement-rules-would-introduce-new-disclosure-and-use-rights-requirements-for-federal-contractors/
18. Ropes & Gray. “Newsom Signs Executive Order Establishing AI Vendor Certification and Procurement Framework.” April 2026. https://www.ropesgray.com/en/insights/alerts/2026/04/newsom-signs-executive-order-establishing-ai-vendor-certification-and-procurement-framework
19. Euronews. “Spain could fine AI companies up to €35 million for mislabelling content.” 12 March 2025. https://www.euronews.com/next/2025/03/12/spain-could-fine-ai-companies-up-to-35-million-in-fines-for-mislabelling-content

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The knock comes on a Tuesday, late afternoon, when the rice is still on the hob and the youngest is doing homework at the kitchen table. A caseworker in a thin coat introduces herself, asks if she can come in, and explains that the city has received a report and is required to follow up. The mother, who has lived in the same flat for nine years and has never had a child welfare investigation in her life, asks who made the report. The caseworker hesitates. It is not exactly a report, she says. It is a flag.

This is the moment, repeated thousands of times a year across American cities, when a family discovers that they have been the subject of attention they did not know was possible. The flag did not come from a neighbour or a teacher or a paediatrician. It came from a model. A risk-scoring system, fed on years of administrative data, generated a number that crossed a threshold inside a software dashboard at the local child protective services office. A screener saw the number. A supervisor signed off. A caseworker was dispatched. Somewhere along that chain, a human being had to make a final decision, but the decision was anchored, framed, and quietly shaped by an output that nobody in the home would ever see.

The mother in the flat has no right to see the score. She has no right to know which features pushed her family above the threshold. She has no right to challenge any one of those features in front of a neutral reviewer. She has no right, in any meaningful sense, to know that the algorithm exists.

That is where American child welfare sits in the spring of 2026: an expanding lattice of predictive systems, deployed inside agencies whose decisions can place a family under state surveillance and, in the worst cases, separate parents from their children, operating almost entirely outside the procedural rights that any other consequential decision in modern life would attract. A family flagged by a credit-scoring algorithm has more statutory recourse than a family flagged by a child welfare risk model. A driver flagged by a parking enforcement camera has more transparency. A tenant flagged by an algorithmic landlord screen has more legal scaffolding to push back. The state has built one of the most invasive deployments of pattern-matching in American public administration, and it has done so on top of the thinnest possible layer of due process.

The Markup Investigation and What It Found

In 2025, The Markup published an investigation into the Administration for Children's Services in New York City, the agency that handles child abuse and neglect reports for roughly 1.6 million children. The investigation, drawing on internal documents and interviews with agency staff, established that ACS had been using an algorithmic risk-scoring tool to help decide which families warranted heightened scrutiny, surveillance and investigation following a hotline call. The tool, which the agency had introduced years earlier with limited public discussion, generated a score for every family entering the system, and that score informed which cases were elevated for what staff called “high-priority” review.

The Markup's reporters, working with academic researchers, found that the system disproportionately flagged Black and low-income families at rates higher than would be expected from the underlying base rates of confirmed maltreatment in those populations. The disparity was not fully explained by the data the agency claimed to be using. There were other variables, less obvious ones, that appeared to be doing meaningful work inside the model. Postcodes. Prior contact with public assistance programmes. Density of services in a neighbourhood. Each was, on its face, a non-racial input. Each, in practice, served as a proxy for race and class, because race and class in New York are written into the geography and the administrative trail of poverty.

The agency, when contacted, defended the tool. It pointed out that the score was advisory, that humans made the final calls, that the system had been validated internally. The agency declined to release the model's full feature set. It declined to release the weights. It declined to release the technical documentation that would have allowed independent researchers to reproduce the disparity findings or to test the model on counterfactual data. Families who had been flagged by the tool, and who had then had caseworkers in their homes, had no idea that an algorithm had been involved in the decision.

The Markup investigation matters not because it was the first time anyone had documented this pattern. It matters because it landed in the largest city in the United States, in the agency that handles the largest child welfare caseload in the country, and because it confirmed that what had previously been a research finding from smaller jurisdictions was now a continental-scale phenomenon. Child welfare is being run, in part, by black-box prediction.

The Allegheny Lineage

The patient zero of the modern child welfare risk-scoring movement is the Allegheny Family Screening Tool, deployed in Allegheny County, Pennsylvania, beginning in 2016. The tool was developed by a research consortium, validated using historical case data, and integrated into the county's call-screening process. When a hotline operator received a report, the tool produced a score that estimated the likelihood that a child in the household would be removed within two years. Higher scores triggered closer review.

The Allegheny tool was, in many ways, the public face of a movement that promised to bring rigour and consistency to an area of public administration long accused of being inconsistent and biased. Its developers were not naive technocrats. They were academics with serious credentials in social welfare and statistics, and they argued, plausibly, that human screeners themselves were biased, and that an algorithm trained on the same data could at least be audited. The tool was not deployed in secret. There were public meetings, advisory committees, journalistic profiles. For a brief moment in the late 2010s, Allegheny was held up as the responsible model.

What followed was a decade of audits that complicated that picture. Independent researchers, including teams who built fairness audit frameworks specifically for child welfare contexts, found that the tool's predictions correlated with socioeconomic status in ways that were not adequately disclosed in the public materials. They found that the tool's accuracy varied by demographic group. They found that the underlying training data, which was based on historical screening and removal decisions, encoded the biases of the human system the tool was supposed to improve. If the historical data showed that Black families in Allegheny had been more likely to have their children removed for any given maltreatment report, then a model trained on that data would learn to flag Black families more aggressively, and would do so even if every explicit racial variable was stripped from the inputs.

The Allegheny defenders responded that the tool reduced overall disparity compared with unaided human screening, and there is research that supports parts of that claim. The Allegheny critics responded that “less biased than the worst-case human” is not a high enough bar to justify deployment, particularly when the tool's mechanics remained opaque to the families it scored. By the early 2020s, this argument had calcified into a sort of trench warfare in the academic literature. The county kept using the tool. Other jurisdictions copied it. New York City's ACS was one of those jurisdictions, and the tool documented by The Markup is, in effect, a descendant of the Allegheny lineage, retrained on different data and tuned to different operational thresholds.

What the April 2026 Audits Showed

On 21 April 2026, two papers appeared on arXiv that, between them, gave the most rigorous picture yet of what is actually happening inside these systems. The first was a fairness audit of institutional risk models in welfare and safeguarding contexts. The second was an analysis of algorithmic fairness in case-note-augmented prediction systems, the newer generation of tools that pull free-text narrative from caseworker notes into the feature pipeline.

The findings, taken together, are damning in a precise and technical way. Models deployed in high-stakes welfare and safeguarding contexts routinely encode socioeconomic and racial proxies even when those variables are nominally excluded from the input set. The mechanisms are not mysterious. They are documented. Postcodes function as racial proxies in segregated cities. Prior interactions with means-tested benefits encode income and, indirectly, race. Neighbourhood-level deprivation indices, which were originally designed by social scientists to identify communities in need of investment, become, when fed into a risk model, indicators that an individual family is more dangerous to its own children. Each input, considered alone, has a defensible policy rationale. Stacked, weighted and combined inside a model that was optimised to predict historical removals, they produce a system that reproduces the geography of state intervention with eerie fidelity.

The case-note paper went further. Once a model starts ingesting free-text notes from caseworkers, the proxy problem deepens, because language itself is socially stratified. A caseworker note that describes a home as “chaotic” or a parent as “uncooperative” carries weight inside an embedding model. Whether those labels were accurate, fair, or applied consistently across demographic groups is a question the model cannot answer and the deployment process rarely interrogates. Audits showed that case-note-augmented models could amplify existing disparities, because the historical record of how caseworkers described different families itself encoded assumptions about whose homes were suspect.

Both papers stopped short of saying that current child welfare risk models cannot be made fair. Both papers said, in different ways, that current child welfare risk models are not currently fair, that their unfairness is structural rather than incidental, and that the standard mitigations on offer in the technical literature, group-balanced thresholds, adversarial debiasing, fairness constraints during training, are insufficient to address proxy encoding at the depth it currently operates. To put it bluntly: the tools the field has built to make these systems fair are themselves not powerful enough to overcome the data the systems are trained on.

The Berkeley Notice Problem

In January 2026, researchers at the University of California, Berkeley published an analysis of a different but related question. Not whether these systems are fair in a statistical sense, but whether the people they scored had any meaningful idea that scoring was happening. The Berkeley analysis catalogued the deployment of algorithmic decision systems across a growing range of life-altering institutional contexts, including child welfare assessments, public benefit eligibility, criminal justice risk assessment, healthcare allocation, and tenant screening. It found that, in the overwhelming majority of cases, the affected individual had no notice that an algorithm was involved, no access to an explanation, and no formal route of appeal that engaged with the algorithmic component of the decision specifically.

This is the harder problem, and in some ways the more politically tractable one. Statistical fairness is a moving target. Technically, you can argue forever about whether a particular calibration metric or error-rate parity standard is the right one. Notice and explanation are simpler. Either the family knows that a system was used or they do not. Either there is a document explaining the inputs or there is not. Either there is a procedure for contesting the score or there is no such procedure.

The Berkeley researchers' finding, applied to child welfare, is sobering. A family flagged by a risk-scoring tool in New York or Pittsburgh or Los Angeles has, in practice, no way to know that they were flagged by a tool. The caseworker on the doorstep is not required to tell them. The investigation paperwork does not disclose it. The records request, if they know to file one, may or may not produce the score. If it does, it almost certainly will not produce the underlying feature values, the model card, or any documentation that would allow them to understand what was being weighed and how.

The information asymmetry is total, and it sits on top of an existing power asymmetry that is itself substantial. Families in the child welfare system are disproportionately poor, disproportionately non-white, and disproportionately under other forms of state observation already, including housing assistance, food assistance, public schools, and Medicaid. The institutional knowledge of how to navigate any of these systems is unevenly distributed. Add an opaque algorithmic layer on top of all of that, and the result is a population of citizens making decisions, accepting investigations, and signing service plans without knowing one of the most important inputs into the state's interest in them.

Why No Rights Yet

The legal infrastructure that might check any of this exists, in patches, in places. None of it is robust enough to do the job.

At the federal level, the closest analogue to a comprehensive algorithmic accountability statute is the patchwork of civil rights law, which prohibits disparate-impact discrimination in some federally funded programmes but has never been successfully wielded against a child welfare risk-scoring tool in the way it has against, say, mortgage-lending models. The procedural due process clause of the Fourteenth Amendment offers some protection in cases where the state seeks to terminate parental rights, but the protections kick in late in the process, well after the algorithmic flag has done its work to set events in motion. Pre-investigation flags are not adjudicated. They are operational decisions, treated as administrative discretion, and discretion is precisely what courts have historically been reluctant to second-guess.

At the state level, a handful of legislatures have passed bills requiring agencies to disclose when they are using automated decision systems, but most of these laws contain carve-outs for “decision-support” tools, and almost every child welfare risk model is officially classified as decision-support rather than as automated decision-making. The reasoning is that a human screener still signs off. The reality is that the screener is reading a score that has been generated by software, and the score functions as the primary signal in many of those decisions. The carve-out exists because vendors and agencies argued for it, and because legislators who wrote the bills did not want to be accused of weakening safeguarding by wrapping it in transparency requirements.

At the procurement level, the contracts that govern these tools are often classified as confidential commercial information. Vendors negotiate terms that prohibit agencies from disclosing the model's inner workings, on the theory that the model is intellectual property. Agencies, who frequently do not have in-house data science capacity to evaluate the tools, accept these terms because they want the tools and they cannot easily build them themselves. The result is a procurement architecture in which the state delegates a consequential public function to a private contractor, accepts secrecy as a condition of the contract, and then refuses to disclose what it has bought, on the grounds that the secrecy is the vendor's right.

The contrast with the European Union is instructive. Article 22 of the General Data Protection Regulation, in its original 2018 form, gave individuals a right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. The right was always more limited in practice than in headline, because purely automated decisions are rare and most regulated decisions involve some human in the loop. But Article 22, paired with the wider GDPR architecture of subject access rights, transparency obligations, and data protection impact assessments, created a baseline that simply does not exist in the United States. The 2024 EU AI Act extended this baseline with risk-tiered obligations for high-risk systems, including those used in social welfare administration. The United States has nothing equivalent at the federal level. State-by-state, the strongest American statutes on automated decision-making are weaker than the floor European regulators consider unacceptable.

This is not because Americans are less concerned about state surveillance of families. It is because the American legislative process, on questions of child welfare, has a particular political shape. No politician wants to be the one who voted for the bill that allegedly weakened child protection. Every vendor of a risk model can frame transparency requirements as obstacles to keeping children safe. Every agency that uses one of these tools can claim that disclosure of the model's mechanics would teach abusive parents how to game the system. These framings are sometimes sincere and sometimes opportunistic, and both are politically effective. The result is a legislative landscape in which proposals to give families notice and challenge rights die in committee, while procurement contracts for new tools are renewed without serious public debate.

Voices in the Field

The academic literature on algorithmic harm in welfare contexts is, by 2026, large enough to constitute a small subfield. Virginia Eubanks, whose 2018 book on automated inequality remains foundational, argued that the deployment of predictive tools in welfare administration represents a new form of digital poorhouse, applying mass surveillance to the populations least able to resist it. Dorothy Roberts, whose work on the racial politics of family policing predates the algorithmic era, has long argued that the child welfare system is structurally biased against Black families and that data-driven tools, far from correcting that bias, formalise it and make it harder to contest. Rashida Richardson, who has written on algorithmic accountability and government use of predictive systems, has argued for procedural rights of notice, explanation and contestation as a baseline condition of legitimate deployment.

On the technical side, researchers like Solon Barocas have spent years documenting the mechanisms by which proxy variables encode protected attributes, and the limits of formal fairness criteria in the face of those mechanisms. Hadi Elzayn and collaborators have published audits of welfare-adjacent algorithmic systems showing, with empirical rigour, how disparate impact persists even under well-designed mitigation strategies. None of these scholars has called for a complete ban on predictive tools in welfare contexts. Most have called for a combination of structural reforms: independent audits, transparency requirements, due process rights, and a presumption that high-stakes deployments require a much higher evidentiary bar than what is currently common practice.

The interesting feature of this body of work is how unified it is on the procedural questions, even when scholars disagree on the technical questions. Whether a particular fairness metric is the right one is contested. Whether families should have a right to know that a model was used in a decision that affected them is, within this literature, essentially uncontested. The gap between the academic consensus and the operational reality of American child welfare is wide, and it is not narrowing.

What Rights Would Look Like

The shape of a meaningful rights framework for algorithmic decisions in child welfare is, at this stage, well rehearsed in policy literature. The components are not exotic.

Notice would mean that a family receiving a child welfare contact would be told, in writing, whether an algorithmic risk-scoring tool was used in the decision to investigate, and that they would be given the name and a plain-language description of the tool. This is an extremely low bar. It would not change the outcome of any individual investigation. It would simply close an information asymmetry that currently has no defensible justification.

Access would mean that the family could obtain the score that was generated for them, the inputs that fed into the score, and the documentation describing how the model translates inputs into outputs. The technical documentation already exists in most cases. It is generated as part of the procurement process. The barrier to disclosing it is contractual, not technical.

Contestation would mean that the family could challenge specific data points used in the score. This is where the model intersects with longstanding administrative law practice. Government records routinely contain errors. Some of those errors are typographical. Others are substantive. A family who has been flagged on the basis of a prior investigation that was later closed as unfounded should be able to point at that investigation and ask whether it was correctly weighted in the model. A family flagged on the basis of a postcode association should be able to ask whether that association is what is doing the work and, if so, whether the weight is justified.

Human review with authority would mean that the human in the loop is not just a person who reads the score and signs off, but a person with the institutional standing to overturn the score, the time to actually examine the inputs, and a documented record of the reasoning behind their decision. This is the most demanding component, because it requires resourcing and training that most agencies have not invested in. It is also the most consequential, because it transforms the human-in-the-loop from a procedural fig leaf into a real check.

Independent auditing would mean that agencies cannot simply self-validate their tools. They would be required to submit the tools to external technical review, including review by parties with no commercial interest in the tool's continued deployment. Audit findings would be public. Significant findings would trigger remediation requirements with deadlines.

A route of appeal would mean that there is a forum in which a family can challenge an algorithmically influenced decision and obtain meaningful relief. This is the hardest component to graft onto the existing child welfare system, because the system's procedural backbone is calibrated for a different kind of dispute. It is calibrated for fact-finding about events in a household, not for technical contestation of a model's behaviour. Building this capacity would require new staff, new training, and probably a new tier of administrative tribunal.

None of these proposals is technically novel. Each has been articulated in academic and policy literature. Each, in some form, exists in other regulatory contexts. What is missing is not the design. What is missing is the political coalition to build them in.

Why the Politics Are Stuck

The reasons no such coalition has consolidated are visible in the structure of the issue. Child protection, as a political project, runs on the premise that the state's job is to err on the side of intervention. The institutional culture of the agencies, the framing of legislative debates, and the media treatment of failures all push in one direction. When a child is harmed in a family that the system did not investigate, there are inquiries, commissions and resignations. When a family is harmed by an unjustified investigation, the story tends not to make the front page, and the family tends not to have a press office.

This asymmetry shapes how risk-scoring tools are introduced and how they are defended. The pitch to administrators is that the tool will reduce the rate of false negatives, the cases where the system missed a child who needed protection. The pitch to legislators is similar. The cost on the other side, the rate of false positives, the families subjected to investigation they did not need, is rarely treated as a comparable harm in the political conversation, even though it is a quantifiable and substantial cost in the lives of those families. The current generation of risk-scoring tools is calibrated according to thresholds chosen by agency leadership, and those thresholds are typically set conservatively in the direction of investigating more rather than fewer households.

The vendors of these tools have learned to operate within this politics. They market on the prevention of catastrophic outcomes. They underplay the operational disparities. They negotiate procurement contracts that limit disclosure. They cultivate relationships with academic researchers who can supply the legitimating veneer of validation studies. None of this is corrupt in any obvious sense. It is the normal behaviour of any commercial actor selling into a politically sensitive market with high stakes and asymmetric information. But the cumulative effect is an industry that is poorly disciplined by external oversight, because the external oversight does not have the tools to discipline it.

Affected families, meanwhile, are nearly impossible to organise. They are already under state scrutiny. They are reluctant to draw additional attention to themselves. They often do not know that other families have had similar experiences, because the information that would allow them to find each other does not flow. Civil society organisations have done significant work in this area, but they have done it at a scale that is dwarfed by the operational scale of the agencies and vendors they are trying to hold accountable.

What to Watch

The most likely vector of change in the near term is litigation. Several civil rights organisations have been preparing cases that target specific algorithmic deployments in welfare contexts, looking to establish precedent under existing civil rights and due process doctrine. The legal theory would not require a new statute. It would require a court to recognise that a family has a constitutionally cognisable interest in not being subjected to investigation on the basis of a process that they cannot contest. Whether such a case will succeed is uncertain. The doctrine is unfriendly. The factual records are hard to build. But the architecture of the litigation is plausible enough that several organisations are betting on it.

A second vector is local legislation, particularly in cities and states where the political balance is more amenable to civil liberties framings. New York, in the wake of The Markup investigation, has seen renewed legislative interest in algorithmic accountability for city agencies. Whether ACS specifically will be brought under stronger transparency rules remains to be seen. The vendors have lobbyists. The agency has institutional inertia. But the political weather, in 2026, is more favourable to disclosure than it was in 2020, and the gap between civil society capacity and vendor capacity is starting to narrow as algorithmic accountability becomes a more established advocacy field.

A third vector, and the one most aligned with the academic literature, is the construction of an external audit infrastructure. A non-governmental organisation, an academic consortium, or a hybrid public-private body with the technical capacity to audit child welfare risk-scoring tools and the legal standing to compel disclosure does not currently exist in the United States. Building one would require funding, talent, and a political settlement that recognises external audit as a legitimate function. There are precedents in other regulated industries: financial auditing, environmental impact assessment, clinical trial review. The case for an analogue in algorithmic public administration is, in the wake of the April 2026 audit findings, harder to dismiss than it once was.

The Family Comes Back

The mother in the flat does not see any of this. She sees a caseworker on a Tuesday afternoon. She answers questions she did not expect to answer. She watches her children watched by a stranger. She signs paperwork. The investigation, in her case, is closed without findings six weeks later. She is not removed from the system; she is now in it, in the database, as a household with a prior contact, a feature that may itself be ingested by the next iteration of the model the next time her name comes up.

She is told none of this. She is not told that an algorithm was involved, that her postcode contributed to the flag, that the model's developers have already been audited by independent researchers and found wanting. She is not told that the city paid a vendor several million dollars for the tool, or that the vendor's contract prohibits disclosure of the model's inner workings. She is not told that, in another country with a different legal regime, she would have had a statutory right to ask for and receive an explanation of the decision that put a stranger in her kitchen.

If American child welfare is going to have any meaningful answer to the question of what happened to her, the answer will not come from the agencies that deployed the tools or from the vendors that built them. It will come from courts willing to take procedural due process seriously when it is dressed in code, from legislators willing to pass disclosure requirements that survive vendor lobbying, and from a civil society infrastructure that does not yet exist at the scale the problem demands. The April 2026 audits, and the Berkeley analysis from earlier in the year, and the Markup investigation that preceded both, are not a complete map of the problem. They are a sufficient one. The technology is here. The harms are documented. The scaffolding of rights is a decade behind.

The next time the knock comes, the family on the other side of the door deserves, at minimum, a piece of paper that tells them what they are dealing with. That is not a radical demand. It is the floor.

References

1. The Markup. (2025). Investigation into the Administration for Children's Services algorithmic risk-scoring tool, New York City. The Markup.
2. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin's Press.
3. Roberts, D. E. (2022). Torn Apart: How the Child Welfare System Destroys Black Families and How Abolition Can Build a Safer World. Basic Books.
4. Chouldechova, A., Putnam-Hornstein, E., Benavides-Prado, D., Fialko, O., & Vaithianathan, R. (2018). A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. Proceedings of Machine Learning Research, 81, 134-148.
5. Vaithianathan, R., Maloney, T., Putnam-Hornstein, E., & Jiang, N. (2017). Children in the public benefit system at risk of maltreatment: Identification via predictive modeling. American Journal of Preventive Medicine, 45(3), 354-359.
6. Allegheny County Department of Human Services. (2019). Allegheny Family Screening Tool: Methodology, Version 2. Allegheny County, Pennsylvania.
7. Richardson, R., Schultz, J., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact police data, predictive policing systems, and justice. New York University Law Review Online, 94, 192-233.
8. Barocas, S., & Selbst, A. D. (2016). Big data's disparate impact. California Law Review, 104, 671-732.
9. Anonymous authors. (2026, 21 April). Fairness audits of institutional risk models in welfare and safeguarding contexts. arXiv preprint.
10. Anonymous authors. (2026, 21 April). Algorithmic fairness in case-note-augmented prediction systems. arXiv preprint.
11. UC Berkeley research team. (2026, January). Notice, explanation, and appeal in life-altering algorithmic decisions: An empirical analysis. University of California, Berkeley.
12. European Parliament and Council. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22.
13. European Parliament and Council. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act).
14. Elzayn, H., Black, E., Vossler, P., et al. (2023). Measuring and mitigating racial disparities in tax audits. Stanford Institute for Economic Policy Research, working paper.
15. Eubanks, V., & Mateescu, A. (2021). “We don't deserve this”: New app places homeless services under surveillance. Logic Magazine.
16. Hurley, D. (2018). Can an algorithm tell when kids are in danger? The New York Times Magazine, 2 January.
17. Brown, A., Chouldechova, A., Putnam-Hornstein, E., Tobin, A., & Vaithianathan, R. (2019). Toward algorithmic accountability in public services: A qualitative study of affected community perspectives. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
18. Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020). A human-centered review of algorithms used within the U.S. child welfare system. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
19. Ho, D. E., & Engstrom, D. F. (2020). Algorithmic accountability in the administrative state. Yale Journal on Regulation, 37, 800-854.
20. Citron, D. K., & Pasquale, F. (2014). The scored society: Due process for automated predictions. Washington Law Review, 89, 1-33.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

In February 2026, Erin West walked through an empty scam compound in Sihanoukville. The lights, she would later tell the Stolen podcast audience, were off across most of the Cambodian coastal city that had, for the better part of a decade, served as the operational headquarters of the global romance-fraud industry. Walking the corridors of a half-abandoned dormitory building, she found the small evidence of recent occupation: bedding still bunched on the floor, foreign-language posters peeled from concrete walls, plastic wash basins stacked in the corners. She also found, scattered around the workstations, the script binders and conversation guides that the trafficked workers had been required to memorise. What she did not find, by and large, was the workers themselves. They had been moved.

West, a former Santa Clara County deputy district attorney who spent twenty-six years prosecuting tech crime before founding Operation Shamrock, has spent three years documenting an industry that, on any honest accounting, has surpassed in scale and harm the international drug trade. Pig-butchering operations alone, the long-con romance fraud that combines synthetic intimacy with sham crypto investment, generated an estimated forty-three point eight billion dollars in revenue across Burma, Cambodia and Laos in 2023, equivalent to roughly forty per cent of those countries' combined official gross domestic product, according to figures cited in testimony to the United States Congress. The industry has not been dismantled by the international crackdown of late 2025. It has migrated, restructured and, increasingly, automated.

That last word is the one this story turns on. The compounds West walked through were emptier than she expected because the work being done inside them is, slowly but visibly, being absorbed by software. The cost of buying a trafficked Vietnamese teenager, beating compliance into him with rebar, and chaining him to a workstation to run twenty-four parallel romance scripts across WhatsApp and Tinder turns out to be higher than the cost of a stack of GPUs running an open-weights large language model fine-tuned for emotional manipulation. The fraud has not gone away. It has, in the way of all industrial production, become more efficient.

And the bills have started arriving.

The Numbers That Made It a Public Crisis

In February 2026, an industry analysis circulated by Credit Union Today put the headline figure for American losses to AI-driven romance fraud at six hundred and seventy-two million dollars across the preceding year. The figure tracked closely with the FBI's own 2025 Internet Crime Complaint Center reporting, which recorded romance and confidence scams as among the highest-loss categories of cybercrime, and which for the first time in IC3's twenty-five-year history saw total internet-crime complaints crossing one million. Reported fraud losses across all categories climbed to twenty point nine billion dollars, a twenty-six per cent increase over 2024.

The British numbers tell a similar story at a smaller scale. UK Finance, the industry body that tracks payment-system fraud across British banks, reported that romance scams cost UK victims twenty point five million pounds in the first six months of 2025 alone, across nearly three thousand cases, a thirty-five per cent increase year on year. City of London Police, which leads the national police response on fraud, recorded more than one hundred and six million pounds lost to romance fraud over the 2024 to 2025 financial year. TSB Bank, in February 2026, warned that romance fraud was rising sharply across its customer base, with the average victim sending eleven separate payments, losing roughly seven and a half thousand pounds before the deception was discovered.

These numbers are, in every estimation produced by every researcher who has examined them, undercounts. AARP's Foresight 50+ Omnibus survey, conducted by NORC at the University of Chicago and published in February 2026, found that more than half of American adults who had lost money to a scam never reported it to anyone. Among those who did report, only twenty-six per cent went to law enforcement; only twenty-three per cent contacted their bank. The reason, AARP's research consistently finds, is shame. While sixty-two per cent of people view scam victims primarily as targets of a crime, sixty per cent simultaneously assume the victim was naïve or too trusting. Those two attitudes coexist in the same survey, often in the same respondent, and they are the social mechanism through which under-reporting becomes structural.

Professor Monica Whitty, head of department for software systems and cybersecurity at Monash University and the leading academic authority on romance-scam victimology, has been documenting this dynamic since the early 2010s. Her foundational research, conducted with Tom Buchanan at the University of Westminster, established what subsequent studies have only deepened: that for many romance-scam victims, the loss of the relationship is more upsetting than the financial loss, with some victims describing the experience as equivalent to bereavement. The double trauma of grief and humiliation creates a powerful reporting suppressor. The figures we have, in other words, are a fraction of the harm being done.

What Changed When the Models Arrived

To understand why the AI inflection of this fraud matters, it helps to understand what it replaced. Romance scams in their pre-2023 form were a labour-intensive business. A scammer, working from a script, could maintain perhaps twenty active relationships at once, each requiring hours of attention, each calibrated to the specific emotional vulnerabilities the victim had revealed in earlier conversations, each constrained by the operator's stamina, language ability, time-zone alignment and capacity to remember which victim had told them what. The bottleneck was human cognition. The product was synthetic intimacy, and synthetic intimacy was, like every artisanal good, expensive to produce.

The bottleneck has now been removed. A study published in the Asian Journal of Criminology in 2025, examining how organised criminal groups in Myanmar were deploying machine learning across their operations, found that large language models were already being used for open-source intelligence gathering and victim-network construction, achieving estimated time and cost reductions of ninety-six point six per cent and ninety-nine point five per cent respectively compared with manual research methods. AI-enabled scams in the same study were found to be four and a half times more profitable than their manual counterparts. The authors anticipated that LLMs would, in the medium term, replace human operators in the conversational stages of pig butchering as well.

That medium term has arrived. Experian's 2026 fraud-trends report, cited approvingly by the American Bankers Association in January, named “machine-to-machine” romance scams, in which fully autonomous bots maintain hundreds of simultaneous personalised relationships, as one of the top five fraud trends to watch. The bots, the report noted, “respond convincingly, build trust over time, and manipulate victims with precision and emotion.” Norton's 2026 dating-scam research found that nearly half of US online daters reported having been targeted by a dating scam, with seventy-four per cent of those targeted falling for at least part of it. Fox Business reported in 2026 that Tinder had begun mandating facial verification and was working with World, the iris-scanning identity protocol formerly known as Worldcoin, to prove that users were human at the point of sign-up. Trustpilot user reviews now estimate that up to eighty per cent of profiles on some major dating sites are fake or AI-generated, up from ten to fifteen per cent before the generative-AI boom.

The technical capabilities being deployed are not science fiction. They are commodity. An off-the-shelf foundation model, fine-tuned on a few thousand transcripts of successful and failed romance-scam conversations, will reliably produce text that is grammatically perfect in dozens of languages, emotionally coherent across long conversational arcs, capable of remembering the victim's daughter's birthday and the name of her late husband and the specific complaint she made about her sister three weeks ago. Combine that text generation with voice cloning, with face-swapping live-video, with the ability to generate consistent images of the same fictitious “soldier deployed in Yemen” or “oil engineer working off the coast of Angola” in any pose holding any object, and the verification heuristics human beings have evolved over decades of online dating collapse simultaneously. The “send me a selfie holding today's newspaper” test, as one industry commentator told the security press in 2026, is dead.

The Architecture of Manufactured Attachment

The reason romance fraud is so devastating, and the reason its industrial scaling is so alarming, is that it does not exploit gullibility in the casual sense the public uses the word. It exploits the human attachment system, which is to say the same neural and behavioural machinery that makes long-term love possible in the first place. Whitty's persuasive-techniques model, derived from interviews with victims and from analysis of hundreds of scam conversations, identifies a sequence of stages. The scammer establishes presence and idealisation. The scammer engineers reciprocal self-disclosure. The scammer creates a private emotional world to which only they and the victim have access. The scammer, by this point typically referred to by the victim as a partner, then introduces a crisis whose resolution requires money.

What an LLM brings to this sequence is not a new technique but radically improved consistency at every stage. Human scammers, working long shifts in a Sihanoukville compound, forget details. They confuse one victim with another. They lapse out of character when tired. They misjudge tone. The model does not. It maintains a coherent persona across thousands of conversational turns, recalls every personal detail the victim has volunteered, mirrors the victim's emotional register with sub-clinical precision, and never, ever loses patience. Researchers at the University of New South Wales reported in February 2026 that participants in experimental studies of AI-generated romance content struggled to distinguish it from authentic human writing, even when explicitly told some samples were synthetic.

The harm profile that follows from this is, predictably, severe. A 2025 qualitative study published in the journal Computers in Human Behavior Reports, examining the emotional, physiological, financial and legal consequences of online romance scams in the United States, found four major themes of harm: mental-health consequences including suicidal ideation, physiological health consequences including stress-related conditions, financial consequences ranging from depleted retirement savings to bankruptcy, and legal consequences including, in some cases, prosecution of victims who had unwittingly become money mules in the laundering of other victims' funds. A separate 2025 paper in the journal Victims and Offenders, which interviewed victim-survivors and their family members about the lived experience of cyber-scam victimisation, recorded what one participant called “falling into a black hole,” a phrase that became the paper's title.

The interaction of grief and shame is the part most laypeople find hardest to model accurately. Victims are mourning the end of what felt to them like a real relationship, often the most emotionally intimate of their adult lives, and they are simultaneously being asked by the surrounding culture to feel embarrassed about the relationship having existed at all. NBC News, in its earlier reporting on victim self-harm, and the AARP's ongoing Perfect Scam podcast, which has documented multiple cases of suicide following romance fraud, have together built a body of journalism that establishes the connection beyond reasonable doubt: romance investment scams combine the two leading proximate causes of suicide identified in the public-health literature, namely the dissolution of an intimate relationship and the threat of imminent financial ruin. In a substantial minority of cases, both arrive in the same week.

The Liability Vacuum

If a foreign criminal network has stolen six hundred and seventy-two million dollars from American citizens by exploiting the infrastructure of American technology firms and American payment networks, the natural question is who, if anyone, is supposed to bear legal responsibility for what happened. The answer in 2026, as a matter of law in both the United States and the United Kingdom, is approximately nobody, distributed across a chain of approximately nobodies, each of whom can plausibly point to the next link as the appropriate defendant.

In the American framework, dating-app and social-media platforms remain heavily protected by Section 230 of the Communications Decency Act, which immunises interactive computer services from liability for content posted by their users. In February 2025, the United States Court of Appeals for the Ninth Circuit, in an opinion that the Electronic Frontier Foundation argued was correctly decided, held that Grindr could not be held responsible under product-liability or negligence theories for matching a fifteen-year-old who had falsified his age with adult users who subsequently raped him. The Supreme Court declined to take up an appeal of a separate Grindr Section 230 case in 2024. The doctrinal direction of travel, despite years of political pressure from both major American parties, is that the platforms remain shielded from liability for the third-party content they host, including content generated by foreign scam operations.

Foundation-model providers occupy a different but comparable redoubt. The major American AI companies, OpenAI and Anthropic prominent among them, have invested heavily in safety training designed to refuse explicit requests to generate scam content. OpenAI's December 2025 Model Spec explicitly prohibits using the company's models for “targeted or scaled exclusion” or “manipulation” of human autonomy, and the joint OpenAI-Anthropic safety evaluation published in 2025 demonstrated that both providers' models are reasonably resistant to direct jailbreaks. The problem is that romance fraud does not require the model to produce overtly malicious content. It requires the model to produce loving content, persistently, in response to victim messages that do not in themselves trigger any safety classifier. The fine-tuning required to weaponise a model for fraud is generally trivial. Open-weights models, which can be modified offline and deployed without provider oversight, foreclose the safety-training argument entirely.

The dating platforms themselves operate in a marketing environment that requires them to insist they are doing everything possible against scammers, while a business environment that rewards reduced friction at sign-up. Tinder's introduction of facial verification and World iris-scan integration in 2025 represented the most aggressive position in the industry, but Bumble, Hinge and the rest of the Match Group portfolio have moved in the same direction more slowly. None of these platforms is currently obliged, under American law, to reimburse victims of fraud that began on their service.

The British position is, on paper, modestly stronger. The Online Safety Act 2023, fully in force across 2025 and into 2026, designates fraud as a “relevant offence” for which platforms must take proactive risk-reduction measures, with Ofcom empowered to fine non-compliant providers up to eighteen million pounds or ten per cent of qualifying global revenue, whichever is higher. Ofcom's enforcement guidance, published in late 2025, explicitly identifies romance scams as in scope. The legislation does not, however, create a private right of action for individual victims to sue the platforms directly, and as of April 2026 the regulator had not initiated headline enforcement against any major dating service for romance-scam failures.

Where the British regime materially diverges from the American one is in the payment rails. From October 2024, the UK Payment Systems Regulator's mandatory reimbursement scheme for authorised push-payment fraud requires all in-scope payment-service providers to reimburse victims of APP fraud up to eighty-five thousand pounds per claim, with the cost shared fifty-fifty between the sending and receiving banks. The PSR's first-year data, published in early 2026, indicated that eighty-eight per cent of money lost to APP scams in the first twelve months of the scheme had been reimbursed, with eighty-two per cent of claims closed within five business days. Vulnerable customers are exempt from the standard one-hundred-pound excess. For all the scheme's limitations, including a fifty-thousand-pound cap initially proposed and then raised to eighty-five thousand after lobbying from consumer groups, it represents the most substantial liability shift any major economy has imposed on the financial sector for the cost of fraud committed against ordinary citizens.

The United States has nothing comparable. American banks, under Regulation E and the broader patchwork of consumer-protection law, are generally not required to reimburse customers who authorised the transfer themselves, even if they were tricked into doing so. The cryptocurrency leg of pig-butchering fraud, which is the most common modality, is even less regulated. Tether, the issuer of the USDT stablecoin that is the rail of choice for Southeast Asian scam compounds, has frozen more than three hundred and forty-four million dollars in suspect funds in cooperation with the US Treasury's Office of Foreign Assets Control and other law-enforcement bodies, but the freezes are discretionary and post-hoc, and the funds, once moved through enough cycles, are practically irrecoverable.

The criminal-law track has, to be clear, not been entirely toothless. In October 2025, the United States Treasury and His Majesty's Treasury jointly imposed what officials described as the largest coordinated sanctions package ever directed at Southeast Asian cyber-scam networks, designating one hundred and forty-six individuals and entities tied to Cambodia's Prince Holding Group as a transnational criminal organisation. Federal prosecutors in the Eastern District of New York unsealed a twenty-six-page indictment against the group's chairman, Chen Zhi, and a separate court filing revealed the seizure of fifteen billion dollars in cryptocurrency identified as proceeds of crime. South Korea, Singapore and other jurisdictions added their own designations through November and December. Chen was arrested in Cambodia and extradited to China in early January 2026. The international press, briefly, described the action as a turning point. Erin West, a few weeks later, was watching new compounds being constructed deeper inside Cambodia, staffed by the same trafficked workers who had been moved out of the sanctioned facilities. The criminal-law approach is necessary. It is not, on the available evidence, sufficient.

What Proportionate Response Actually Looks Like

The temptation, when an industry has scaled at the speed and scope of AI-enabled romance fraud, is to reach for a single-bullet solution: ban the model, regulate the platform, prosecute the kingpin. None of these has, on its own, the dimensions of the problem. The proportionate response, if such a thing is constructible, has to operate at every layer of the stack at once, on the principle that any one defensive measure can be circumvented but enough of them in combination create friction sufficient to make the unit economics of fraud worse than the unit economics of legitimate business.

The first layer is liability allocation. Romance fraud is currently a negative externality of three industries: dating platforms, AI model providers and payment processors. None of those industries pays the cost. The victims pay the cost. Standard externality economics tells us what to do in this situation: tax the externality back onto the producers, either through direct civil liability or through mandatory insurance schemes. The British APP-reimbursement model, applied to the AI inflection of fraud, is a credible starting point. The American conversation, dominated by Section 230 absolutism on one side and reactive criminalisation on the other, has not yet caught up with the proposition that platforms hosting industrial-scale fraud should pay a proportional share of the bill.

The second layer is technical. Provenance and content-credential standards, including the C2PA work led by Adobe, Microsoft and the BBC, offer a partial defence against synthetic media in the dating context, though they require widespread adoption to matter. Real-time scam-pattern detection, of the sort already deployed by some major banks against APP fraud, can be extended to dating platforms with the cooperation of the major providers. Crucially, the friction-budget conversation needs to be reopened: dating apps that allow free-tier sign-up with minimal verification have made an explicit choice to externalise risk onto the user base, and that choice can be unwound. Tinder's iris-scan experiment is awkward and intrusive and probably necessary. The privacy trade-offs are real and demand serious public scrutiny. They do not, on the current evidence, weigh more heavily than the harm being done.

The third layer is the financial chokepoint. Stablecoin issuers, cryptocurrency exchanges and remittance corridors that process pig-butchering proceeds are the single most leveraged point in the system, because the criminal enterprise cannot survive without them. Tether's discretionary freezes are an admission that the issuer can act when it wants to. The policy question is whether we want a global stablecoin network whose anti-fraud actions depend on the goodwill of a private issuer in the British Virgin Islands, or whether we want enforceable standards. The same question applies, with greater obvious leverage, to the major centralised exchanges. Coinbase, Kraken and Binance know which wallet clusters are receiving pig-butchering proceeds. The friction-imposition tools are already built. They are not yet mandatory.

The fourth layer is shame reduction. This is the layer journalists tend to skip, because it is the least technical and the least amenable to the policy-speak the trade prefers, and it is also the layer on which everything else turns. As long as romance-scam victims feel too humiliated to report, the entire enforcement apparatus is operating on a fraction of the available signal. AARP's research finding that sixty per cent of survey respondents simultaneously regard victims as crime targets and as personally naïve is the public-attitudes equivalent of a software bug, and like a software bug it is fixable. Public-information campaigns that explicitly de-stigmatise victimhood, similar in their emphasis to the long-running campaigns around domestic violence and sexual assault, are the work of years rather than months, but they pay back at compound interest. Police forces that train officers to treat romance-scam reports as serious crime rather than personal embarrassment do better at recovery. Banks that deploy specialist fraud teams trained in conversational intervention, of the sort TSB has begun piloting in the UK, recover more money and rupture fewer relationships in the process.

The Position

The position this article arrives at is simple enough to state and difficult enough to enact. If artificial intelligence has industrialised what was once a manual fraud, then the regulatory and civil-liability response must be industrial in proportion. The decade since Section 230 was last seriously contested in Congress is the decade across which the model providers, the platform owners and the payment processors have built a fraud-vulnerable infrastructure whose marginal costs are borne entirely by people who did not consent to the trade-off. That arrangement is not, on any defensible reading of consumer-protection law or basic distributive ethics, sustainable.

The British APP-reimbursement scheme is the most concrete signal in the global regulatory landscape that another arrangement is possible. It is imperfect. It has been gamed at the margins. It has shifted some risk from victims to banks, where lobbyists are already pushing back. But it has also, in its first year of operation, returned eighty-eight pence on the pound of stolen money to ordinary people who would otherwise have lost it. That is the kind of measurable, replicable outcome that should be the baseline expectation of any modern fraud-response regime. The American conversation, which currently treats the platforms and model providers as effectively immune from civil liability for the harms that flow through their services, has not yet caught up.

What the AI-fuelled romance fraud crisis tells us, more clearly than any other AI-policy story of the last five years, is that the question of who pays is not separable from the question of who builds. The companies whose models can sustain hundreds of simultaneous synthetic intimacies have built a capability that, in the absence of corresponding obligations, will be used by whoever has the lowest scruples and the cheapest GPUs. The dating platforms whose business model depends on frictionless sign-up have built an attack surface that, in the absence of mandatory verification, will continue to be exploited by foreign criminal syndicates. The payment processors whose systems clear billions in cross-border transfers within seconds have built an exfiltration channel that, in the absence of mandatory holds and reimbursement, will continue to be the last leg of every successful scam.

There are real trade-offs in every direction. Mandatory verification at dating-app sign-up imposes real privacy costs and may foreclose use by people with good reasons to remain pseudonymous. Liability shifts to model providers risk slowing the deployment of useful AI capabilities into other domains. Bank-level holds on cross-border transfers will frustrate legitimate users with foreign relationships. None of these trade-offs is trivial. None of them is anywhere near as severe as the harm currently being externalised onto romance-scam victims, whose lives are being ended, financially and sometimes literally, in their tens of thousands every year.

Back to Sihanoukville

Walking through the abandoned compound in February 2026, Erin West observed something that has stayed with the people who watched her video diaries on the Operation Shamrock site. The compound was not closed in any meaningful sense. The workers had been moved further inland, away from the coastal compounds that had drawn the attention of Western investigators, into newer facilities in the Cambodian interior, where the construction was active and the access roads were difficult and the international press had not yet arrived. The infrastructure of synthetic intimacy was, like every infrastructure that has ever been targeted by international enforcement, relocating.

The point of the relocation, from the operators' perspective, is that the work has become much easier to relocate. A workforce of trafficked humans is heavy, slow, expensive to move and dangerous to move because moved humans speak to journalists. A workforce of language models is none of those things. It can be replicated across server farms in jurisdictions with weak enforcement, fed translated scripts in any of fifty languages, and pointed at victims selected from leaked dating-site databases or social-media data brokers. The compound in Sihanoukville with its bunched bedding and abandoned wash basins is, in retrospect, an artefact of the human-labour era of romance fraud. The coming era will leave fewer artefacts, because there will be fewer humans to leave them.

What that era requires of the rest of us is the construction of a regulatory and legal infrastructure that does not depend on the existence of a physically locatable compound full of trafficking victims. The compounds were politically galvanising, but they were also a crutch. They allowed Western policymakers to treat romance fraud as an offshore problem that would yield to offshore enforcement. The model-driven version of the same fraud will yield only to onshore measures: civil liability for the platforms and the payment rails, mandatory provenance and verification standards for the dating services, reimbursement schemes for the victims, public-information campaigns that strip the shame off reporting. None of those measures is novel. All of them are tractable. The reason they have not been built yet is not technical and not even economic. It is that the people whose retirement savings are being extracted, conversation by tender conversation, by language models running in unmarked data centres, do not yet have lobbyists who can match the influence of the companies whose infrastructure is being used against them.

That, in the end, is the actual story of the six hundred and seventy-two million dollars and the twenty point five million pounds and the unknowably larger sums that will never be reported. It is not a story about AI alignment, or about jailbreaks, or even about the trafficking compounds, which are themselves only the most visible symptom of the deeper architecture of harm. It is a story about who pays, and the answer the current system gives is the wrong one. The bills are arriving in the post, in the bank statements, in the unanswered phone calls from a partner who does not exist, and the response so far, from the institutions whose infrastructure made the fraud possible, has been a press release. It is not enough. It will not be enough until the cost of building these systems carelessly is finally borne, in cash and in court, by the people who built them.

The compound lights are off in Sihanoukville. The models are warming up everywhere else.

---

References

1. CU Today. “Love, Lies And $672 Million Lost: Inside The AI-Fueled Romance Scam Explosion.” CU Today, February 2026. https://www.cutoday.info/THE-feature/Love-Lies-And-672-Million-Lost-Inside-The-AI-Fueled-Romance-Scam-Explosion
2. Federal Bureau of Investigation, Internet Crime Complaint Center. “FBI IC3 2025 Internet Crime Report.” FBI, 2026. Summary via SOCRadar, https://socradar.io/blog/fbi-ic3-2025-internet-crime-report-10-takeaways/ and AARP, https://www.aarp.org/money/scams-fraud/fbi-ftc-report-2025-losses/
3. UK Finance. “Over £600 million stolen by fraudsters in first half of 2025.” UK Finance Press Release, 2025. https://www.ukfinance.org.uk/news-and-insight/press-release/over-ps600-million-stolen-fraudsters-in-first-half-2025
4. City of London Police. “A wrong turn on Love Lane: City of London Police take over city streets to warn of the dangers of romance fraud, with more than £106 million lost in the last year.” City of London Police, June 2025. https://www.cityoflondon.police.uk/news/city-of-london/news/2025/june/a-wrong-turn-on-love-lane-city-of-london-police-take-over-city-streets-to-warn-of-the-dangers-of-romance-fraud-with-more-than-106-million-lost-in-the-last-year/
5. TSB Bank / Fintech Global. “TSB warns romance fraud rising sharply across social media.” Fintech Global, 6 February 2026. https://fintech.global/2026/02/06/tsb-warns-romance-fraud-rising-sharply-across-social-media/
6. AARP. “Romance Scams Put Millions of Adults 50-plus at Risk: Here is What You Need to Know.” AARP Press Release, 3 February 2026. https://www.aarp.org/press/releases/2026-02-03-Romance-Scams-2026.html
7. Whitty, Monica T.; Buchanan, Tom. “The online dating romance scam: The psychological impact on victims, both financial and non-financial.” Criminology and Criminal Justice, 2016. https://journals.sagepub.com/doi/10.1177/1748895815603773
8. Operation Shamrock. “I Walked Into an Abandoned Scam Compound.” Operation Shamrock, 2026. https://www.operationshamrock.org/blog/erin-west-cambodia-inside-scam-compound
9. West, Erin. Written Testimony for the United States House Committee on Financial Services, REACT Task Force, September 2024. https://docs.house.gov/meetings/BA/BA10/20240918/117659/HHRG-118-BA10-Wstate-WestE-20240918.pdf
10. Springer Nature. “An Artificial Intelligence-Driven Temporal Network Analysis of Myanmar's Cyber Scam Ecosystem.” Asian Journal of Criminology, 2025. https://link.springer.com/article/10.1007/s11417-025-09474-0
11. American Bankers Association Banking Journal. “AI romance, 'machine-to-machine' scams among top 2026 fraud trends.” ABA Banking Journal, January 2026. https://bankingjournal.aba.com/2026/01/ai-romance-machine-to-machine-scams-among-top-2026-fraud-trends/
12. Norton. “What Norton's Insights Reveal About AI Dating Scams (2026).” Norton, 2026. https://us.norton.com/blog/online-scams/ai-online-dating-scams
13. Fox Business. “Tinder turns to eye scans to weed out bots amid rising concerns over romance scams.” Fox Business, 2026. https://www.foxbusiness.com/media/tinder-turns-eye-scans-weed-out-bots-amid-rising-concerns-over-romance-scams
14. UK Government. “Online Safety Act: Explainer.” GOV.UK, 2024. https://www.gov.uk/government/publications/online-safety-act-explainer/online-safety-act-explainer
15. Payment Systems Regulator. “APP scams reimbursement dashboard for Q3 2025.” PSR, 2025. https://www.psr.org.uk/information-for-consumers/app-scams-reimbursement-dashboard/
16. Electronic Frontier Foundation. “Ninth Circuit Correctly Rules That Dating App Isn't Liable for Matching Users.” EFF Deeplinks, February 2025. https://www.eff.org/deeplinks/2025/02/ninth-circuit-correctly-rules-dating-app-isnt-liable-matching-users
17. OpenAI. “Model Spec.” OpenAI, 18 December 2025. https://model-spec.openai.com/2025-12-18.html
18. OpenAI. “Findings from a pilot Anthropic-OpenAI alignment evaluation exercise: OpenAI Safety Tests.” OpenAI, 2025. https://openai.com/index/openai-anthropic-safety-evaluation/
19. United States Department of the Treasury. “U.S. and U.K. Take Largest Action Ever Targeting Cybercriminal Networks in Southeast Asia.” Treasury Press Release, October 2025. https://home.treasury.gov/news/press-releases/sb0278
20. Crypto.news. “Tether freezes $344M in USDT tied to sanctions evasion, pig-butchering scams.” Crypto.news, 2025. https://crypto.news/tether-freezes-344m-in-usdt-tied-to-sanctions-evasion-pig-butchering-scams/
21. NBC News. “How online romance scams are netting millions of dollars and pushing victims to self-harm.” NBC News. https://www.nbcnews.com/tech/tech-news/how-online-romance-scams-netting-millions-self-harm-rcna85252
22. ScienceDirect. “A qualitative investigation of the emotional, physiological, financial, and legal consequences of online romance scams in the United States.” Computers in Human Behavior Reports, 2024. https://www.sciencedirect.com/science/article/pii/S2949791424000605
23. Taylor and Francis Online. “Falling into a Black Hole: A Qualitative Exploration of the Lived Experiences of Cyberscam Victim-Survivors and Their Social Support Networks.” Victims and Offenders, 2025. https://www.tandfonline.com/doi/full/10.1080/15564886.2025.2481267
24. University of New South Wales. “Valentine's Day is coming: How vulnerable are you to AI-enabled romance scams?” UNSW Newsroom, February 2026. https://www.unsw.edu.au/newsroom/news/2026/02/valentine-s-day-is-coming--how-vulnerable-are-you-to-ai-scams
25. Coalition for Content Provenance and Authenticity. “C2PA Specification.” C2PA, 2024. https://c2pa.org/specifications/specifications/2.0/index.html

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

John Steinbach opened the envelope at his kitchen table in Manassas, Virginia, and stared at a number that did not make sense. His January 2026 electricity bill came in at $281. The month before, he had paid something close to $100, the sort of bill he had been paying for years. He checked the decimal point. He read the meter reading twice. He was not running a grow-op in his garage. He had not left the heater jammed on. He was, as far as he could tell, living the same quiet suburban life he had always lived, in a house some twenty miles from the centre of what the industry calls Data Center Alley, the densest cluster of server farms on planet Earth.

“It's just so far beyond any bill that I've ever had,” he told Consumer Reports, in an investigation published in March 2026 that has become, in the months since, something close to a Rosetta Stone for understanding the strange new arithmetic of American household utilities. His bill had not tripled because he was using triple the electricity. It had tripled because of the vast silicon and concrete apparatus humming away in warehouses up the highway, the apparatus that now draws roughly 40 per cent of Virginia's electricity, the apparatus that is training and running the artificial intelligence models the rest of the world uses to summarise emails and generate images of astronauts on horseback.

The bill arrived, in other words, because somewhere in the invisible plumbing of the grid, a decision had been made about whose electricity this was, who would pay for the transmission upgrades required to keep the GPUs cool and the data centres fed, and who would absorb the shock when demand outstripped supply. That decision was not made by John Steinbach. It was not made by his neighbours. It was not made by anyone on his street, his block, or his county council. And that, more than the $181 increase, is the story.

A Gold Rush With Someone Else's Gold

The United States is currently in the middle of the largest private infrastructure build-out since the interstate highway system, and almost nobody who lives in its path has been asked to approve any part of it. According to Fortune, data centres now account for half of all new American electricity demand. According to Consumer Reports, residential electricity prices rose 7.1 per cent in 2025 alone, and in the Mid-Atlantic corridor served by the PJM grid, the wholesale auction that sets capacity prices reached record highs, resulting in average bill increases of more than $17 a month for Baltimore customers of Exelon's BGE utility, a figure that makes Steinbach's triple-digit leap look almost modest by comparison.

Behind these averages sits a deceptively simple question: when a hyperscaler, say, Amazon or Microsoft or Google or Meta, builds a two-gigawatt campus in a rural county, who pays for the substations, the transmission lines, the combined-cycle gas plants and the grid upgrades required to keep it running? Traditionally, utilities answered that question by spreading the cost across their entire rate base, which is to say, across every household and small business in the service territory. This is the legal framework that built America's electricity system, and it is also the legal framework that is now quietly transferring billions of dollars a year from retirees and single parents and small restaurants to the balance sheets of the richest corporations in history.

Ari Peskoe, director of the Electricity Law Initiative at Harvard Law School, has spent years documenting how this mechanism works in practice, and his conclusion is as blunt as it is unflattering: ratepayers, not shareholders, are funding the build-out. Kasia Tarczynska, a researcher at Good Jobs First, has traced the subsidy flows state by state and estimates that data centres are extracting roughly a billion dollars a year in tax breaks in Virginia and Texas alone, a figure that does not even include the uncosted externalities of rate shifting. The subsidies are public. The profits are private. The bills, when they arrive, arrive at the addresses of people like John Steinbach, who did not know there was a deal, let alone that he was on the hook for it.

The Mechanism, in Plain English

The mechanism is worth understanding, because it is the pivot on which the entire community-cost argument turns, and because utility regulators have for years hidden it behind a thicket of acronyms and rate-case filings that discourage lay scrutiny.

When a hyperscaler signs a contract with a utility for, say, a one-gigawatt data centre, that utility must deliver one gigawatt of firm, twenty-four-hour-a-day power. A gigawatt is roughly the annual consumption of 750,000 American homes. To deliver it, the utility must build or expand generation capacity (gas turbines, in most cases, because renewables are intermittent and nuclear is too slow), string new transmission lines, and upgrade substations. These are enormous capital projects, and utilities recoup their costs by applying to a state public service commission to raise rates.

The key word in that last sentence is “rates,” plural. American utility regulation does not typically require the cost of infrastructure built for a specific customer to be charged to that specific customer. Instead, it is socialised across the rate base on the grounds that everyone benefits from a robust grid. This assumption, reasonable in an era when new load came from population growth and air conditioning, becomes surreal when the new load is a single fenced-in campus drawing more power than a medium-sized city and producing no goods, no services and no jobs beyond a few dozen technicians.

In September 2025, Dominion Energy, Virginia's largest utility, formally acknowledged the problem by proposing a new rate class specifically for data centres. It was a partial concession, and critics including the Piedmont Environmental Council argued it did not go nearly far enough. In the 2026 legislative session that followed, Delegate John McAuliff pitched HB 503, a bill that would prevent utilities from putting the cost of transmission lines and generation serving data centres onto residential ratepayers. Governor Glenn Youngkin's subsequent amendments to related legislation instructed the State Corporation Commission to “take all measures to reasonably ensure” that data-centre costs “are not being subsidized by other customers of the utility.”

These are real legislative interventions, and they represent the first serious attempt in any American state to restore the polluter-pays principle to grid economics. They are also, so far, largely reactive. Nearly 600 data centres are already operational in Virginia. More than 100 are under construction or proposed. The horse is not so much bolting as already over the horizon, trailed by a slow-moving legislature and a furious electorate.

The Water Question, and the Silence That Follows It

In January 2026, Al Jazeera published a column by the Egyptian-British commentator Omar Shabana with the understated headline “AI's growing thirst for water is becoming a public health risk.” The column set out a case that, while partially familiar to climate reporters, had not quite been assembled with such clarity before. AI data centres, Shabana noted, are projected to increase global water usage from 1.1 billion to 6.6 billion cubic metres by 2027. Microsoft's own 2023 environmental report acknowledged that 41 per cent of its water withdrawals came from areas under water stress. Google's figure was 15 per cent from high-scarcity regions. And those were the self-reported numbers.

The public health argument ran as follows. Only 0.5 per cent of the water on Earth is fresh. When data centres siphon millions of gallons a day from municipal or aquifer sources in drought-prone regions, households are pushed to ration. Rationing means prioritising drinking and cooking over washing hands, food and bodies. The World Health Organization has long documented the correlation between reduced hygiene and the spread of cholera and diarrhoeal disease, conditions that disproportionately kill children under five, who bear 84 per cent of the global burden. “Although it is too early to draw direct causal links between AI data centres and water-related diseases,” Shabana wrote, “the known facts make this a significant concern.” It was, in WIRED's parlance, the softest possible framing of the hardest possible claim.

The specifics are where the argument gets teeth. In Newton County, Georgia, residents near a Meta facility reported discoloured, sediment-filled water in their taps. In Fayette County, similar complaints surfaced. In Phoenix, Arizona, Consumer Reports documented a facility consuming 385 million gallons a year for evaporative cooling, with projections rising to 3.7 billion gallons annually, in a desert city whose principal water source, the Colorado River, has been in declared shortage since 2021. A 2023 academic study led by researchers at the University of California, Riverside, estimated that a single ChatGPT conversation of 10 to 50 queries can consume roughly half a litre of water once electricity generation is factored in, a figure that sounds trivial until you multiply by a billion users.

The intellectual scandal, such as it is, concerns the asymmetry of visibility. The hyperscalers know precisely how much water their cooling towers consume; they metre it. The communities whose aquifers are being drawn down typically do not. In 25 of 31 Virginia communities surveyed by journalists in 2025, data-centre operators had insisted on non-disclosure agreements with local governments before revealing the details of their water or energy needs. These NDAs, enforceable under state commercial-confidentiality statutes, have meant that residents have often learned of a pending project's footprint only after construction began.

The Noise, the Heat, the Light

If the grid-rate and water stories are the invisible costs of the data-centre boom, the sensory costs are the ones that have, perhaps predictably, driven the fiercest backlash. Data centres, despite a reputation for clean-room minimalism, are loud. They are warm. They are often lit like small airports, twenty-four hours a day, on land that was once farm or forest.

In Southaven, Mississippi, Jason Haley stopped being able to sleep with the windows open in August 2025. The noise came from the direction of the old power-plant site half a mile from his house, where Elon Musk's xAI had begun erecting a cluster of natural-gas turbines to supply its Colossus supercomputer across the state line in Memphis. The turbines, as Haley described them to Mississippi Today, sounded like “a leaf blower” that ran for days at a stretch, all night, every night. Eighteen of them are currently operating. Fifty-nine are planned. xAI is awaiting state permits for the remainder. Whatever those permits say, the noise does not stop for them.

In Chandler, Arizona, a city of 280,000 just outside Phoenix, the battle over data-centre noise began as far back as 2014, when a million-square-foot facility produced a steady drone that drove residents to the council chambers. The city eventually adopted an ordinance targeting data-centre noise specifically. In December 2025, the Chandler City Council unanimously voted down the rezoning application for a proposed AI data centre, the kind of decisive “no” that, a few years ago, would have been almost unimaginable for a facility that promised capital investment and a handful of jobs.

Diana Dietz, who lives several miles from a QTS data-centre campus in Fayetteville, Georgia, told Consumer Reports she was not especially opposed to technology. What she objected to was the parade of earth-moving machinery through her neighbourhood at all hours. “You've got these giant excavation-style dump trucks two minutes from my house,” she said. The trucks, like the turbines, like the substation transformers and the chiller plants, are the visible evidence of a fact that AI boosterism tends to obscure: intelligence in a cloud is, in the physical world, heavy machinery.

The Regulatory Vacuum

The reason these stories keep happening in such similar shapes, in Manassas and Memphis and Chandler and Fayetteville, is that the United States has no coherent federal framework for siting AI infrastructure, and most states have not yet caught up. Zoning law is largely a matter of county and municipal jurisdiction. Utility regulation is a state matter, usually delegated to a public service commission whose commissioners are appointed, not elected, and whose rate-case proceedings are structured for expert testimony rather than public comment. Environmental permitting is split between the Environmental Protection Agency, state environmental agencies, and, in the case of water, a byzantine patchwork of river compacts, aquifer management districts and groundwater statutes that vary wildly from one jurisdiction to the next.

Into this regulatory swiss cheese rolls the hyperscaler, armed with non-disclosure agreements, a pre-negotiated economic-development package, and a timeline that counts in months rather than years. Local officials, often in rural counties with shrinking tax bases, see a multibillion-dollar investment and a handful of permanent jobs, and they sign. By the time the trucks arrive and the neighbours notice, the horse is out of the barn and several hundred megawatts of load have been added to a grid that was not designed for it.

Data Center Watch, a tracking group that has been cataloguing community responses since 2023, reported in early 2026 that roughly $64 billion worth of data-centre developments had been blocked or delayed by grassroots opposition across the United States. There are now 142 activist groups organised in 24 states. Twenty-five major projects were cancelled in 2025 alone. Twenty-one of those cancellations came in the second half of the year. In Wisconsin, sustained local opposition caused Microsoft to abandon plans for a 244-acre campus. In Brandy Station, Virginia, the Culpeper County Planning Commission unanimously denied rezoning for a $12 billion project in June 2024, a rare and galvanising rebuke. In King George County, Virginia, a newly elected board of supervisors reopened negotiations on Amazon's $6 billion campus.

These are not fringe events. They are becoming a pattern. And the pattern is, increasingly, that the hyperscalers are discovering the hard way that the social contract around infrastructure cannot be replaced with a press release.

Enter the Investors

In April 2026, Reuters reporters Simon Jessop, Valerie Volcovici and Supantha Mukherjee published a piece that marks, in retrospect, a turning point in the story. Amazon, Microsoft and Google had each, the article noted, recently abandoned multibillion-dollar data-centre projects in the face of community opposition. Now, shareholders were asking why. More than a dozen investors, ahead of spring annual meetings, were filing resolutions demanding disclosure of water consumption, power usage and community engagement strategies.

Trillium Asset Management, a Boston-based firm managing more than $4 billion, filed a resolution with Alphabet in December 2025 seeking clarity on how the company would meet its existing climate goals given the surging electricity needs of its data centres. Andrea Ranger at Trillium said the company had left investors “in the dark.” Jason Qi, lead technology analyst at Calvert Research and Management, offered the kind of restrained corporate-stewardship rebuke that reads as devastating only when you remember that Calvert sits on hundreds of billions in institutional assets: “We haven't seen them disclosing enough about their water consumption (and the) impact on the local community.” Green Century Capital Management joined the chorus.

The hyperscalers' responses were, on the whole, not reassuring. Microsoft said sustainability was “a core value” and that it was “proactively addressing sustainability challenges.” Google declined to comment. Meta did not return the Reuters reporters' request. Dan Diorio, vice-president of the Data Center Coalition, the industry's lobbying arm, offered what is probably the clearest articulation of where this argument is actually headed: “Being upfront with them regarding energy and water use, and so that residents can understand that this project will not stress their resources... and will protect them as rate payers is crucial.”

That is not the language of a confident industry. That is the language of an industry that has just realised its permits depend on the people whose aquifers it is draining and whose electricity bills it is inflating. The pressure, importantly, is now coming from two directions at once: from below, in the form of local activism and abandoned projects; and from above, in the form of fiduciary-duty shareholder resolutions from firms that can no longer pretend community opposition is an idiosyncratic, one-off risk.

The Social Licence to Operate

In March 2026, the World Economic Forum published a piece in its Agenda section with the title “Why AI and data-centre growth risks stalling without a social licence to operate.” The phrase “social licence to operate” originates in the mining industry, where it was coined in the 1990s to describe the informal consent that host communities grant (or withhold) from extractive projects, regardless of what formal permits say. In mining, the social licence is the difference between a functioning copper mine and a blockaded access road.

The WEF piece argued, in language calibrated for an audience of Davos attendees rather than Virginia ratepayers, that the AI industry was now facing precisely this kind of legitimacy test. “Understanding the importance of a social licence to operate is becoming critical as AI infrastructure, especially data centres, outpaces governance,” the piece observed. “Growing opposition across regulators, politicians and communities is already driving permit blocks and constraining scale.” Communities were not, it argued, “passive hosts” but “stakeholders with leverage.”

The piece then laid out what the industry needed to do to earn this licence: transparency about energy and water use; stable electricity prices; responsible water management; meaningful job pathways; enhanced local infrastructure. It was, in effect, an industry trade body admitting in print what the activists in Chandler and Brandy Station and Culpeper had been saying for years. If you do not earn consent, you do not get to build. If you get to build anyway, you will not get to build the next one. And the next one is where the money is.

The framing is useful because it reorients the ethical conversation away from the hypothetical harms of AI, the rogue superintelligences of science-fiction discourse, and towards the concrete, present, measurable harms of AI infrastructure. It is much easier to argue about whether GPT-7 will end civilisation than to explain to a state legislature why a Microsoft substation should or should not be socialised across the residential rate base. The social-licence framing does the explaining for us: it names the transaction, names the parties, and names the consent that was or was not given.

What Meaningful Consent Would Actually Look Like

If the social-licence argument is to be more than a rhetorical flourish, it has to be translated into institutional mechanisms with actual enforcement teeth. The literature on community consent in infrastructure, drawn from mining, pipelines, wind farms and landfill siting, suggests at least five such mechanisms, each of which is already being tested somewhere in the American data-centre debate.

The first is rate segregation, the Virginia HB 503 model, in which the cost of new generation and transmission built for a specific class of customer is recovered only from that class, rather than socialised across all ratepayers. This is the narrowest but most important reform, because it removes the hidden subsidy that currently makes data centres appear cheaper to host than they actually are. If hyperscalers had to pay the full incremental cost of the grid infrastructure they require, fewer speculative projects would pencil out, and the ones that did would bring with them a more defensible economic case.

The second is mandatory impact disclosure. HB 496 in Virginia, which would require localities to consider annual water-consumption estimates in rezoning decisions, is a modest but meaningful example. A more muscular version would require an environmental and infrastructure impact statement, subject to public hearing, for any data-centre project above a threshold size, modelled on the National Environmental Policy Act but administered at state level with specific provisions for aquifer drawdown, peak-load demand, noise contours and light pollution.

The third is community benefit agreements, the legal instrument used widely in urban development to bind a developer to specific commitments to the host community. In the data-centre context, a CBA might include funded local-grid upgrades, guaranteed rate stability for residential customers, water conservation investments, noise mitigation standards, a public education levy and a seat for community representatives on an ongoing oversight body. These agreements are enforceable contracts, not voluntary promises. The Chandler ordinance targeting data-centre noise is, in effect, a unilateral CBA imposed by a city after losing patience with the developer's assurances.

The fourth is meaningful public hearings with teeth, which sounds anodyne until one realises how often current hearings function as information sessions at which decisions already made are announced to residents. A hearing with teeth is one where the decision can actually be reversed, where the developer is obliged to answer questions under oath, and where non-disclosure agreements signed in advance by local officials are voided as a matter of public policy. The Culpeper County Planning Commission's unanimous denial in Brandy Station is a model of what this looks like when it works.

The fifth is water impact assessment with a veto right for water authorities, a mechanism borrowed from arid-region agricultural law. In several Western American states, irrigation districts already hold something close to this power over new industrial users. Applying it to data centres would mean that a project proposed in, say, the Phoenix metropolitan area would not simply need a permit from the city; it would need affirmative sign-off from the regional water management district, based on a fifty-year draw projection and a public-health impact assessment of the kind Omar Shabana described in Al Jazeera. This is how water scarcity is managed in places that take water scarcity seriously.

None of these mechanisms individually constitute “consent.” Together, they describe the shape of a regulatory regime in which a data-centre project could not be built until the people whose lives it would change had been informed, consulted, and given the ability to shape or stop it. That is not an anti-AI position. It is the position American infrastructure regulation already holds towards pipelines, landfills and nuclear plants. The hyperscalers have been exempt from it essentially by default, because no legislator anticipated that the data centre would become the dominant form of new industrial development in the country.

The Ratepayer Protection Pledge, and Why It Is Not Enough

In early March 2026, executives from Amazon, Microsoft, Google, Meta and Anthropic travelled to the White House to sign a document called the Ratepayer Protection Pledge. It was non-binding. It committed signatories to “principles” around transparency and responsible siting. Anthropic additionally pledged to cover electricity-price increases related to its own data-centre development, a gesture that impressed some observers and struck others as the kind of one-off public relations flourish that is easy to make when your data-centre footprint is a small fraction of Microsoft's.

The pledge is worth noting because it is, in effect, the industry's first serious attempt at self-regulation, and because self-regulation is almost always what an industry proposes when it fears the alternative. It is also worth noting because it is inadequate in ways that illustrate why the social-licence framing matters. A pledge without enforcement is an aspiration. A pledge that can be reinterpreted by its signatories is a negotiating document. The mechanisms that would actually protect John Steinbach in Manassas, or Jason Haley in Southaven, or Diana Dietz in Fayetteville, are statutory, not voluntary. They live in utility tariffs, zoning ordinances and environmental statutes, not in White House photo opportunities.

The Question of Who Pays

Return, finally, to John Steinbach's kitchen table, and to the $281 bill that started this story. The question the bill poses is not really about kilowatt-hours. It is about a much older political question: who pays for the infrastructure that powers the new economy? The answer the current system gives, by default, by inertia, by the absence of legislative attention, is that everyone does, apportioned by their share of the residential rate base. The answer the activists in Wisconsin and Virginia and Arizona are demanding is that the beneficiaries of the infrastructure should pay for the infrastructure, which is a principle so uncontroversial it appears in first-year economics textbooks. The reason it does not currently apply to data centres is that the law has not been updated to reflect a world in which a single industrial customer can consume more power than an entire American city.

The hyperscalers have, to their credit, begun to notice. The Reuters report on investor pressure, the WEF piece on social licence, the White House pledge, and the scattered community-benefit agreements now being negotiated in individual counties all suggest an industry that has belatedly realised it cannot continue to build its physical substrate in places that do not want it. What is still missing is a coherent legislative framework that gives communities the tools to say no, or to say yes on specific terms, before the earth-movers arrive.

That framework will arrive, eventually, the way such frameworks always arrive in the American system: through a patchwork of state reforms that eventually force federal harmonisation, driven by the accumulation of lawsuits, news stories, bills like Steinbach's and places like Brandy Station where the planners drew the wrong parcel on the map and the locals remembered they were allowed to vote. Until then, the story will continue to be written in envelopes opened at kitchen tables in towns most of the industry's customers have never heard of, by people whose consent was presumed rather than sought, for a build-out whose bill is still only beginning to come due.

The Cost Is Not Abstract

One of the oddities of covering this beat is how persistently the language used to describe AI infrastructure collapses the physical world into metaphor. We speak of “the cloud,” as though data did not live on spinning disks in concrete boxes. We speak of “compute,” as though it were an abstract quantity rather than a thermodynamic process that converts electricity into heat. We speak of “scaling,” as though the industrial footprint of a two-gigawatt campus were a matter of typography.

The activists in Chandler and the ratepayers in Manassas have done the rest of us a small service by refusing this language. The cost is not abstract. It is $281. It is a leaf-blower noise that does not stop. It is discoloured water in Newton County, Georgia, and a parade of dump trucks outside Fayetteville. It is a Microsoft cooling tower pulling from an aquifer that was already in decline, and an electricity bill that arrived in a January envelope and could not be explained by the weather.

What the AI industry has built is, on any honest accounting, extraordinary. What it has failed to build, so far, is the social architecture of consent that every previous heavy-industrial sector in American history eventually had to build: the permitting regimes, the environmental impact statements, the community benefit agreements, the ratepayer protections, the statutes that ensure the people who host the factory are not also the people who pay for it.

The absence of that architecture is not a technicality. It is the entire argument. Meaningful consent does not mean asking permission after the foundation has been poured. It means building a system in which the permission must be asked, granted, and periodically renewed, by the people whose land, water, electricity and night-time quiet are on the line. Anything short of that is not a social licence. It is an imposition with a press release.

John Steinbach will pay his bill. He does not have an option. The question is whether, by the time the next bill arrives, he will have been given any real say in what that bill is for. The answer to that question will determine, in ways the AI industry is only beginning to understand, whether the next gigawatt of data-centre capacity gets built at all.

---

References

1. Shah, Kaveh. “AI Data Centers: Big Tech's Impact on Electric Bills, Water, and More.” Consumer Reports, 20 March 2026. https://www.consumerreports.org/data-centers/ai-data-centers-impact-on-electric-bills-water-and-more-a1040338678/
2. Jessop, Simon; Volcovici, Valerie; Mukherjee, Supantha. “Investors press Amazon, Microsoft and Google on water, power use in US data centers.” Reuters, 6 April 2026. https://www.investing.com/news/stock-market-news/analysisinvestors-press-amazon-microsoft-and-google-on-water-power-use-in-us-data-centers-4598459
3. Shabana, Omar. “AI's growing thirst for water is becoming a public health risk.” Al Jazeera Opinion, 21 January 2026. https://www.aljazeera.com/opinions/2026/1/21/ais-growing-thirst-for-water-is-becoming-a-public-health-risk
4. World Economic Forum. “Why AI and data-centre growth risks stalling without a social licence to operate.” WEF Agenda, March 2026. https://www.weforum.org/stories/2026/03/ai-community-social-licence/
5. Fortune Editorial. “Data centers now account for half of all new U.S. electricity use, just as Americans start to sour on AI.” Fortune, 20 April 2026. https://fortune.com/2026/04/20/us-data-center-electricity-demand-public-opinion/
6. CNBC. “Who is really footing the AI energy bill? Inside the debate about data center electricity costs.” CNBC, 13 March 2026. https://www.cnbc.com/2026/03/13/ai-data-centers-electricity-prices-backlash-ratepayer-protection.html
7. NPR. “AI data centers use a lot of electricity. How it could affect your power bill.” NPR, 2 January 2026. https://www.npr.org/2026/01/02/nx-s1-5638587/ai-data-centers-use-a-lot-of-electricity-how-it-could-affect-your-power-bill
8. Virginia Mercury. “These bills aim to regulate Virginia data center siting, generator use and SCC oversight.” Virginia Mercury, 19 January 2026. https://virginiamercury.com/2026/01/19/these-bills-aim-to-regulate-virginia-data-center-siting-generator-use-and-scc-oversight/
9. Virginia Mercury. “The governor's amendments to energy and data center legislation will save money for customers.” Virginia Mercury, 22 April 2026. https://virginiamercury.com/2026/04/22/the-governors-amendments-to-energy-and-data-center-legislation-will-save-money-for-customers/
10. Piedmont Environmental Council. “2026 Data Center Reform Legislation.” PEC, 2026. https://www.pecva.org/region/regional-state-national-region/2026-data-center-reform-legislation/
11. Data Center Watch. “$64 billion of data center projects have been blocked or delayed amid local opposition.” Data Center Watch Report, 2026. https://www.datacenterwatch.org/report
12. Mississippi Today. “Southaven residents fear pollution, complain of noise, from Elon Musk's xAI data-center turbines.” Mississippi Today, 24 November 2025. https://mississippitoday.org/2025/11/24/southaven-residents-fear-pollution-complain-of-noise-from-elon-musks-xai-data-center-turbines/
13. AZ Family. “Chandler unanimously votes against proposed AI data center.” AZ Family, 12 December 2025. https://www.azfamily.com/2025/12/12/chandler-unanimously-votes-against-proposed-ai-data-center/
14. Environmental and Energy Study Institute. “Communities Are Raising Noise Pollution Concerns About Data Centers.” EESI. https://www.eesi.org/articles/view/communities-are-raising-noise-pollution-concernsabout-data-centers
15. ARLnow. “Winter electric bill spikes spark scrutiny of Va. data centers' power usage.” ARLnow, 27 February 2026. https://www.arlnow.com/2026/02/27/winter-electric-bill-spikes-spark-scrutiny-of-va-data-centers-power-usage/
16. The White House. “Ratepayer Protection Pledge.” The White House, March 2026. https://www.whitehouse.gov/releases/2026/03/ratepayer-protection-pledge/
17. Li, Pengfei; Yang, Jianyi; Islam, Mohammad A.; Ren, Shaolei. “Making AI Less Thirsty: Uncovering and Addressing the Secret Water Footprint of AI Models.” arXiv, 2023. https://arxiv.org/pdf/2304.03271
18. CalMatters. “AI data centers could hike California electricity bills.” CalMatters, March 2026. https://calmatters.org/environment/2026/03/little-hoover-data-center-electricity/
19. Virginia Mercury. “Dominion proposes higher utility rates, new rate class for data centers.” Virginia Mercury, 3 September 2025. https://virginiamercury.com/2025/09/03/dominion-proposes-higher-utility-rates-new-rate-class-for-data-centers/
20. Bloomberg. “AI Data Centers Are Sending Power Bills Soaring.” Bloomberg Graphics, 2025. https://www.bloomberg.com/graphics/2025-ai-data-centers-electricity-prices/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The fifty-minute hour has always been a negotiated fiction. A patient talks, a therapist listens, and somewhere between them a third thing forms: a clinical relationship dense with attention, memory, and the particular quality of being heard. For more than a century, that triangle contained only two people. It no longer does.

Open the laptop of a typical American psychologist in the spring of 2026 and the odds are better than even that an AI tool is already running. It might be an ambient scribe quietly listening to the session and transcribing it into a SOAP note by the time the patient reaches their car. It might be a risk-screening dashboard that flags linguistic markers of suicidality in a client's intake forms. It might be a real-time coaching tool that whispers a reframe into the clinician's earpiece when a silence stretches too long. The American Psychological Association's Practitioner Pulse Survey, the results of which landed in March 2026, is unambiguous: AI has moved from fringe novelty to routine infrastructure in clinical practice. Fifty-six per cent of the 1,742 psychologists surveyed said they had used an AI tool to assist their work in the past year. Twenty-nine per cent use one at least monthly. In 2024, seventy-one per cent said they had never touched the stuff. That is a generational shift compressed into twelve months.

What the APA did not find, and went out of its way to say it did not find, was a professional consensus on any of the hard questions. Not on disclosure. Not on patient rights. Not on what a therapist owes a client before flipping on a machine that will listen to their most intimate disclosures and then ship the audio to a vendor in a different state, possibly a different country, for processing. The technology has outrun the ethics, and the ethics have outrun the law, and the patient, sitting on the couch, is the last to know.

This is the part of the story that does not get a press release. When somebody talks about AI in mental health, they usually mean the chatbot, the thing a lonely teenager might befriend at three in the morning. That conversation has had its reckoning in courtrooms and congressional hearings, and it is not the one that matters most for the several million people who still see a human therapist each week. The interesting question is quieter and harder. If the clinician sitting across from you, the one you were referred to by your GP or your insurer, is quietly running an AI in the background of your treatment, what exactly are they obliged to tell you about it? And does the research support the idea that adding the machine makes the therapy better, or only cheaper to provide?

The Ambient Scribe Enters the Room

The single most common entry point for AI into clinical practice is also the most banal. Therapists hate paperwork. They always have. The running joke in any psychology department is that nobody went into the field to write insurance justification letters, and yet everyone spends roughly a third of their working hours doing exactly that. A standard therapy session runs fifty minutes. The documentation attached to it, progress notes, billing codes, risk assessments, treatment plan updates, can eat another thirty. Over a week, the arithmetic is brutal. A therapist seeing twenty-five clients is writing for twelve and a half hours on top of their clinical load. Much of that work happens after dinner.

Into this grim calculus walked a class of product nobody would have predicted five years ago: the AI therapy scribe. Blueprint, founded in 2020 and now one of the category leaders, offers an ambient listening tool that records sessions, transcribes them, and spits out a structured progress note in under sixty seconds. The pitch is seductive. The company claims average documentation times can fall from around 4.7 hours a day to 1.2, a reduction of roughly seventy-five per cent. Upheal, a Prague-based competitor, promises essentially the same thing with a slightly different feature set: SOAP, DAP, BIRP, or GIRP notes generated to a therapist's preferred template, with the model learning the clinician's writing style over time. Both companies advertise HIPAA compliance, SOC 2 certification, Business Associate Agreements, and end-to-end encryption. Blueprint automatically deletes its recordings after transcription. Upheal deletes audio too, though it retains transcripts for clinician review.

These are not marginal products. Blueprint says it is used by tens of thousands of therapists across the United States. Upheal reports a comparable footprint. They have competitors: Freed, Mentalyc, Twofold, Heidi Health. They have pricing tiers aimed at solo practitioners and at group practices, and in some cases at the in-house compliance officers at hospital systems. They have venture capital behind them. They have, above all, momentum.

And they sit at the centre of an ethical problem most of their users have not fully reckoned with. When a client walks into a therapy session, their spoken words are, under ordinary confidentiality rules, legally and ethically protected to a very high standard. Therapy is the paradigm case of a privileged communication. What happens when those words are transmitted in real time to a third-party vendor for algorithmic processing, stored in a cloud they cannot see, and used, in most contracts, to improve the vendor's underlying model? The HIPAA answer is that the vendor becomes a business associate, that a Business Associate Agreement must be in place, and that appropriate safeguards must be documented. The ethical answer is considerably more complicated. It turns on whether the client has any meaningful understanding of what is happening, and any meaningful ability to say no.

What the APA Actually Said

The March 2026 APA findings, laid out across the association's Monitor on Psychology coverage and the underlying Practitioner Pulse Survey data, painted a field in the middle of an uneven, improvised adoption curve. The most common AI uses reported by psychologists were administrative: drafting emails, summarising documents, generating content, and dictation or note-taking. These are not, for the most part, tasks that touch clinical judgement. They are the grinding logistics of running a practice. But a growing minority of respondents reported more ambitious uses: treatment planning support, risk assessment augmentation, and, in the most contested category, real-time prompting during sessions.

The concerns practitioners reported were granular and serious. Sixty-seven per cent worried about data breaches. At least sixty per cent flagged concerns about biased inputs and outputs, inaccurate outputs, lack of rigorous testing, and unanticipated social harms. Thirty-eight per cent worried that AI would eventually render some or all of their job duties obsolete. The APA's accompanying health advisory, issued in tandem with the survey, was blunt: AI chatbots and wellness apps lack the testing and safety measures needed to provide quality mental health support, and such tools cannot replace qualified clinicians. The advisory did not, however, tell therapists what to do about the AI already running quietly in their own offices.

That gap was partially filled by the APA's Ethical Guidance for AI in the Professional Practice of Health Service Psychology, a document released in mid-2025 and refined through the end of that year. The guidance articulates what, in theory, any American clinician using AI is obliged to do. It is worth dwelling on, because the clarity of the guidance stands in such contrast to the murk of actual practice. Clinicians must obtain informed consent. They must clearly communicate the purpose, application, and potential benefits and risks of the AI tools they are using. They must disclose AI use to individuals receiving direct care, to other relevant providers, and to any third party who might be considered a client. The disclosure must cover the type of AI tools involved, their effect on treatment, the flow of data, the role of third-party vendors, and the cost implications of AI-assisted care. Crucially, clients must be able to opt out, and where feasible be offered a human-only alternative. AI must augment, not replace, professional judgement. The psychologist remains responsible for every decision, and must not defer blindly to a machine-generated recommendation.

This is an admirably complete list, and for anyone who has sat through the five-minute consent ritual in a modern clinic, a slightly fantastical one. The average informed consent form in private practice is already a dense three-page document that clients sign while their therapist fetches water. Adding a granular disclosure of AI usage, one that specifies which vendor, which data paths, which retention policies, and which opt-out mechanisms apply, pushes the document towards something no reasonable client will read. And yet the APA is right that the alternative, a disclosure so thin it amounts to a rubber stamp, betrays the therapeutic relationship's foundational trust.

The Psychiatric News Point/Counterpoint in January 2026 surfaced the tension openly. One camp argued that AI disclosure should be patient-empowering by default, with consent forms that actively invite clients to ask questions about the technology in the room. The other argued for a stricter standard rooted in the principle of non-maleficence: if the clinician is not prepared to answer detailed technical questions about the tool they are using, they should not be using it. Somewhere between those two positions sits the average working therapist, using a scribe because it saves them six hours a week, with only a dim understanding of where the audio goes after the session ends.

The Evidence Problem

Suppose, for a moment, that every clinician disclosed every AI use with flawless clarity. Suppose every client understood, in appropriate detail, what was happening. The question underneath the disclosure question would remain: does any of this make therapy better, or does it simply make it cheaper to deliver? Here the literature is much thinner and much less flattering than the marketing materials suggest.

The most comprehensive recent answer came from a scoping review of reviews published in January 2026 in Frontiers in Psychiatry, which examined the state of AI in mental health care across the field. The review's core finding was that most clinical AI models in mental health remain proof-of-concept. They achieve impressive headline numbers, AUCs of up to 0.98 and accuracies reaching ninety-seven per cent, but those numbers come overwhelmingly from internal validation on small cohorts. When the same models are tested against independent data, performance degrades, sometimes substantially. External validation is sparse. Where it exists, AUCs typically settle into a more modest 0.80 to 0.88 range, and often lower. The review concluded that the gap between demonstrated performance in the laboratory and deployed performance in the clinic had not meaningfully narrowed.

This matters because the business case for clinical AI assumes the headline numbers. A risk-screening tool that correctly identifies suicidal ideation with ninety-five per cent accuracy in a training dataset might flag a tenth of that in practice. A treatment-planning assistant that suggests evidence-based interventions might do so using a corpus that over-represents certain populations and therapies. None of this is to say AI cannot help. It is to say that the current evidence base for routine clinical deployment is not yet what the confidence of the product demos implies.

The picture is worse, and better documented, for consumer-facing therapy chatbots, which remain distinct from clinician-supporting tools but share an underlying technology and some of the same pathologies. The Stanford study that has become the most frequently cited reference point in this debate, presented at the ACM Conference on Fairness, Accountability, and Transparency and led by Jared Moore, Declan Grabb, and Nick Haber of Stanford with collaborators from Carnegie Mellon, the University of Minnesota, and the University of Texas at Austin, delivered a particularly uncomfortable finding. The researchers evaluated five popular therapy chatbots, including 7cups' Pi and Noni and Character.AI's Therapist, against a set of clinical criteria. Licensed human therapists responded appropriately ninety-three per cent of the time. The chatbots responded appropriately less than sixty per cent of the time.

More damning still, the researchers found that the chatbots exhibited more stigma towards certain mental health conditions than others. Alcohol dependence and schizophrenia drew more negative responses than depression. Newer and larger models did not perform better than older or smaller ones. In scenarios involving suicidal ideation or delusions, where a trained clinician would gently push back or help the client reframe, the chatbots sometimes enabled the dangerous thinking rather than interrupting it. The Stanford team was careful about what they were claiming. Their argument was not that chatbots could never be useful. It was that the specific failure modes they documented, stigma reinforcement and crisis mishandling, are particularly dangerous precisely because of the kind of user who seeks out an AI therapist in the first place.

There is a subtler finding layered underneath. If the chatbot reinforces stigma about schizophrenia, and the user of the chatbot is someone with a family history of schizophrenia who has not yet sought formal care, the effect of the interaction is to push that person further from the clinician they actually need. The AI therapist, in certain configurations, does not merely fail to help. It delays the real help.

The TikTok Problem

A second category of failure sits adjacent to the clinical AI question, and is probably more consequential at a population level. It concerns the recommendation algorithms that sit on the social-media platforms where vulnerable people now habitually go looking for mental health information. An arXiv pre-print published on 16 April 2026 by a research team auditing TikTok's mental health recommendations brought the issue into sharp focus. The paper, titled “Seeking Help, Facing Harm”, used thirty simulated accounts that interacted with TikTok's For You feed over a seven-day period in January 2026, collecting 8,727 videos. The audit varied two dimensions: whether the initial search framing was distress-initiated or help-initiated, and whether the account's interaction strategy leaned into the content or away from it.

The finding was that engaged accounts saw their feeds saturate with mental health content, roughly forty-five per cent of daily recommendations. Avoidance and passive viewing reduced the exposure but did not eliminate it, with mental health content still comprising between eleven and twenty per cent of recommended videos. Crucially, TikTok's algorithm did not appear to meaningfully distinguish between a user expressing distress and a user seeking help. Both were treated as a single mental health interest cluster. The consequence is that someone who types “how to stop feeling suicidal” gets treated roughly the same as someone who types “romanticising depression”. Recovery-oriented intent is conflated with distress consumption, and vulnerable users get served content that may exacerbate rather than mitigate harm.

The TikTok finding matters for the therapy-room question because it changes the context in which the therapy happens. The client who walks into a session at five o'clock has spent the preceding weeks, often the preceding years, consuming algorithmic content that actively shapes their sense of what their condition is and how it should be treated. Eating-disorder content, trauma content, ADHD content, grief content: all of it flows through the same engagement-optimised pipe. When the therapist then introduces an additional AI tool into the session, even a benign one like a scribe, they are not doing so in neutral territory. They are doing so on ground already extensively ploughed by algorithmic influence. The disclosure question is no longer just about the scribe. It is about the whole ecosystem of machine-mediated attention the client arrives with.

What the Patient Is Owed

So: if AI is now embedded in what remains one of the most intimate and trust-dependent professional relationships most people will ever have, what disclosure obligations should apply? The answer is not a single sentence, but it is a reasonably short list.

First, a client has the right to know, in plain language, that AI is in the room. That starts with whether a scribe is recording the session, but it extends to whether any automated system is generating, drafting, or shaping the treatment plan; flagging risk signals that will influence the clinician's judgement; or processing session data after the fact. The disclosure should happen at the start of treatment, not buried in a stack of intake forms. It should be revisited when tools change.

Second, the client has the right to understand the data path. Not every technical detail, but the substantive answers to the questions any thoughtful person would ask. Where does the audio go? Who stores it? For how long? Is it used to train a model? Can it be accessed by anyone other than the therapist? What happens if the vendor is acquired, goes bankrupt, or suffers a breach? These questions are answerable, in most cases, from a vendor's published documentation. If the therapist cannot answer them, the therapist does not yet understand the tool they are using.

Third, the client has the right to opt out. This is the hinge of the whole disclosure regime. A disclosure that comes with no meaningful alternative is not consent; it is notification. The APA's guidance is explicit that, where feasible, a human-only alternative should be offered. The qualifier “where feasible” is doing a great deal of work, and it is precisely the word that clinicians and practice managers will lean on when it becomes inconvenient to honour the principle. A patient who says “please do not record the session” should not have to become a problem case.

Fourth, the client has the right to know what the AI is not doing. Most people, when told an AI is involved in their care, will assume more than is actually the case. They may assume the AI is making diagnostic judgements it is not making. They may assume the therapist has abdicated more authority than they have. They may also assume the AI is doing less than it is. The disclosure should be honest in both directions.

Illinois, in August 2025, became the first US state to write this logic into statute. Public Act 104-0054, known as the Wellness and Oversight for Psychological Resources Act, prohibits the use of AI to provide therapy directly, to make independent therapeutic decisions, to interact with clients in any form of therapeutic communication, or to generate treatment plans without clinician review. It permits AI for administrative and supplementary support, provided the patient has given specific, written, revocable consent. Violations carry civil penalties of up to $10,000. Illinois will not be the last state to legislate this territory. Governing AI in Mental Health, a fifty-state legislative review published in late 2025, identified similar proposals in a growing number of jurisdictions. The Federal Trade Commission, having opened an inquiry in September 2025 into AI chatbots acting as companions, is circling the consumer end of the problem. The HHS approach is more deregulatory, aimed at promoting adoption through pilot programmes, but the direction of travel on disclosure is fairly clear.

In the UK, the regulatory movement has been quieter but real. The British Psychological Society has endorsed a Global Psychology Alliance guidance document on AI in psychology. The Health and Care Professions Council, which regulates practitioner psychologists in the UK, collaborated with four other regulators in February 2026 on a joint statement on AI in education and training. Neither document carries the specificity of the Illinois statute. But the principle is being worked out in comparable terms: disclosure, consent, clinician oversight, and meaningful alternatives.

The Economic Gravity

The harder question, and the one the industry would prefer not to answer, is whether the motivation for AI adoption in therapy has more to do with clinical improvement or with cost reduction. The available evidence suggests the latter is doing most of the work.

Start with what the data says about outcomes. A randomised clinical trial of a conversational AI agent for psychiatric symptoms, published in late 2025, found meaningful reductions in anxiety and depression symptoms compared to group therapy and control conditions. Meta-analytic work on AI-driven interventions has found moderate effect sizes, broadly comparable to low-intensity clinician-delivered treatments, particularly for cognitive behavioural approaches to mild or moderate depression. The therapeutic alliance literature suggests users can form surprisingly strong bonds with AI agents, sometimes within three to five days, and that these bonds correlate with engagement and symptom improvement. None of this should be dismissed. For someone who would otherwise get no care at all, a decent AI-mediated intervention is clearly better than nothing.

Now set that next to the economic reality of how therapy is paid for. Insurers routinely deny around fifteen per cent of mental health claims even when prior authorisation has been obtained. Therapists in network spend roughly a third of their working week on documentation and billing. The average productivity gain from an AI scribe, across published implementation studies, is a reduction in documentation time of roughly seventy per cent. That saving does not, in general, accrue to the patient as a lower fee. It accrues to the practice as either higher clinician throughput, a reduction in administrative overhead, or both. In the most straightforwardly commercial model, the clinician sees more clients per week because each one requires less paperwork, and the practice's revenue per clinician rises.

This is not inherently sinister. More clinician availability in a field with chronic workforce shortages is a genuine social good. But it reshapes what AI is for in a way the consent forms rarely acknowledge. The AI is not principally a clinical tool; it is a throughput tool. It makes it economically viable for the therapist to do more of what they already do. Whether it makes what they do better is a separate question, and the answer so far is: perhaps slightly, and only in certain narrow domains, and only where clinicians actively engage with the output rather than letting it substitute for their attention.

The managed-care angle is where this becomes politically charged. If AI scribes and AI treatment-planning aids allow a clinician to see, say, thirty per cent more patients per week, an insurer looking at its network will be tempted, over time, to recalibrate reimbursement rates on the assumption that AI productivity gains will absorb the pressure. The savings that flowed to the practice in year one flow to the payer in year five. The patient's out-of-pocket cost does not decline, but the amount of clinician attention per dollar declines quietly. This is the pattern that radiology, primary care, and coding specialities have already been living through for a decade. Psychotherapy is a latecomer to the trend, but there is no obvious reason it will be immune.

There is a further and more disquieting possibility, which the APA's guidance gestures at without quite confronting. If insurer incentives push towards AI-augmented therapy, and AI-augmented therapy can be delivered at higher throughput, the marginal therapist may find it increasingly difficult to operate without the AI. Not because the AI makes them a better clinician, but because opting out becomes economically untenable. The clinician who refuses the scribe sees fewer patients, generates less revenue, and becomes less competitive in the market. The right to opt out, from the therapist's side, begins to erode under the same logic that is already eroding it on the patient's side.

The Alliance Question

The single deepest concern, and the one hardest to measure, is what all of this does to the therapeutic alliance itself. The alliance, in the clinical literature, refers to the particular quality of collaboration and bond between client and clinician that consistently predicts outcomes across therapeutic modalities. It is not a decoration on therapy; it is substantively what therapy is. The studies on AI-mediated interventions suggest the alliance can form with a machine as well as a human, which is a finding both reassuring and unsettling. A recent Journal of Medical Internet Research integrative review on the digital therapeutic alliance identified five components, goal alignment, task agreement, therapeutic bond, user engagement, and the facilitators and barriers that mediate them, and found each was at least partially achievable in digital contexts.

That literature, however, is mostly about stand-alone digital interventions. The live question for the clinician-with-AI case is whether introducing a machine into the room between two humans changes the alliance those two humans are building. Common sense suggests it must. If the client knows that a transcript of the session is being processed by a vendor, that knowledge will shape what they say. Certain topics, abortion, substance use, past legal trouble, contemplated self-harm, may come up more tentatively or not at all. The therapist's assurance that “the system is HIPAA compliant” will not, in most clients, fully dissolve the awareness that a machine is listening. Awareness of surveillance, even benign surveillance, changes speech. That change is precisely what good therapy exists to undo.

There is a further dynamic on the therapist's side. When a clinician is using an ambient scribe, a small part of their attention is necessarily devoted to what the machine is picking up, whether it is capturing the important moments, whether the resulting note will require heavy editing. That split attention is subtle, but it is not zero. A significant body of attentional research suggests that the perceived presence of a recording device changes how the recorder behaves. Whether that change, summed across a caseload of thirty clients per week, is large enough to measurably degrade therapeutic outcomes is an open empirical question. Nobody has yet designed the study that would answer it cleanly.

The Shape of an Honest Practice

If the current moment is a kind of frontier, a period in which adoption has raced ahead of norms, what does an honest AI-augmented practice look like? A few things are becoming clear.

It looks like disclosure written in plain English and revisited as tools change, not a legal boilerplate tucked into the intake packet. It looks like vendor contracts the clinician has actually read and can summarise for a curious patient in three sentences. It looks like genuine opt-out alternatives, including for the administrative AI, with no penalty for the patient who takes them. It looks like clinicians who can articulate what the AI is doing and, more importantly, what it is not doing. It looks like practices that do not quietly shift the productivity dividend of AI into more intense caseloads without acknowledging the trade-off. It looks like professional bodies that enforce their guidance, rather than publishing it and moving on. It looks like regulators, at the state and federal level, who treat psychotherapy as a high-risk domain and set the bar for deployment accordingly.

It also looks like a willingness, on the part of the profession, to resist the parts of this technology that do not serve the patient. Real-time therapeutic prompting, in which an AI listens to the session and suggests what the clinician should say next, is the category where the ethics strain hardest. There is a plausible account of it in which a supervisor-in-the-ear helps less experienced clinicians avoid mistakes and improves overall care. There is a much less flattering account in which a clinician becomes a kind of animatronic front-end for a model whose reasoning is opaque, whose training data is proprietary, and whose accuracy on this particular patient has never been validated. The Illinois statute's prohibition on AI-driven therapeutic communication without clinician review reflects a judgement that the risk outweighs the benefit at the present state of the art. Other jurisdictions will have to make the same call.

The research agenda, too, has to catch up. The Frontiers scoping review's conclusion that most clinical AI in mental health remains proof-of-concept is not an argument against the technology. It is an argument for external validation studies, for pre-registered trials comparing AI-augmented care to unaugmented care on the outcomes patients actually care about, for longitudinal work on therapeutic alliance in hybrid settings, and for harm-reporting infrastructure of the kind that exists for drugs and devices but does not yet exist for software that mediates clinical relationships.

The Patient Still in the Room

Return, finally, to the room. A client arrives for a fifty-minute hour. They sit on a couch that has accommodated many lives before theirs. They are about to tell a therapist something they have never told anyone, or something they have told many people but never in quite this way. The question the APA report in March, the Frontiers review in January, the Stanford chatbot study, and the TikTok audit in April all circle without quite naming is whether that room is still the same room it was before the machines moved in.

The answer, probably, is that it is not, but that it can still be a good room, provided the people responsible for it are honest about what has changed. The therapist who discloses the scribe and offers the alternative, who reads the vendor contract, who does not pretend to their client that the AI is neutral or invisible, is not being paranoid. They are doing the oldest job in their profession, which is the protection of the therapeutic relationship against whatever threatens it, including the tools the therapist themselves brought in.

The patient, for their part, has a right the culture has not quite caught up with: the right to ask, and to get a real answer. Not a legally defensible reassurance, but the answer a friend would give if they happened to know the technology. Whether the profession can make that kind of answer routine is probably the question that will decide, more than any regulation, whether AI makes therapy better, or simply makes it cheaper to deliver, and quieter in its cheapness.

References and Sources

1. American Psychological Association. “AI in the therapist's office: Uptake increases, caution persists.” Monitor on Psychology, March 2026. https://www.apa.org/monitor/2026/03/ai-reshaping-therapy
2. American Psychological Association. “2025 Practitioner Pulse Survey: AI in the therapist's office.” APA Publications, released December 2025. https://www.apa.org/pubs/reports/practitioner/2025
3. American Psychological Association. “Ethical guidance for AI in the professional practice of health service psychology.” APA Topics, 2025. https://www.apa.org/topics/artificial-intelligence-machine-learning/ethical-guidance-ai-professional-practice
4. Frontiers in Psychiatry. “Artificial intelligence in mental health care: a scoping review of reviews.” January 2026. https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2026.1688043/full
5. Moore, J., Grabb, D., Agnew, W., Ong, D. C., and Haber, N. (Stanford HAI, Carnegie Mellon, University of Minnesota, University of Texas at Austin). “Exploring the Dangers of AI in Mental Health Care.” Presented at ACM FAccT. Stanford HAI. https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care
6. “Seeking Help, Facing Harm: Auditing TikTok's Mental Health Recommendations.” arXiv preprint, 16 April 2026. Accepted at ICWSM 2026. https://arxiv.org/abs/2604.14832
7. Illinois General Assembly. “Public Act 104-0054: Wellness and Oversight for Psychological Resources Act.” August 2025. https://ilga.gov/Documents/Legislation/PublicActs/104/PDF/104-0054.pdf
8. JMIR Mental Health. “Artificial Intelligence in Mental Health Services Under Illinois Public Act 104-0054: Legal Boundaries and a Framework for Establishing Safe, Effective AI Tools.” 2025. https://mental.jmir.org/2025/1/e84854
9. Blueprint. “Blueprint: AI-Assisted EHR for Therapists.” Product documentation. https://www.blueprint.ai
10. Upheal. “AI Clinical Notes: HIPAA-compliant therapy documentation.” Product documentation. https://www.upheal.io/ai-clinical-notes
11. Blueprint. “AI for Therapists: ACA and APA Ethical Guidance.” https://www.blueprint.ai/blog/aca-and-apa-guidance-for-the-use-of-ai-for-therapists
12. Videra Health. “Navigating the APA's AI Ethical Guidance: A Clinician's Perspective.” July 2025. https://viderahealth.com/2025/07/03/apa-ai-ethical-guidance-clinician-perspective/
13. Documentation Wizard. “AI in Psychotherapy: Disclosure or Consent?” https://documentationwizard.com/ai-in-psychotherapy-disclosure-or-consent/
14. HIPAA Journal. “When AI Technology and HIPAA Collide.” https://www.hipaajournal.com/when-ai-technology-and-hipaa-collide/
15. Psychiatric News. “Point/Counterpoint: Empowering Patients vs. Do No Harm in the Age of AI.” January 2026. https://www.psychiatryonline.org/doi/10.1176/appi.pn.2026.01.1.11
16. British Psychological Society. “New guidance sets out principles for understanding artificial intelligence in psychology.” https://www.bps.org.uk/news/new-guidance-sets-out-principles-understanding-artificial-intelligence-psychology
17. Health and Care Professions Council. “HCPC collaborates on joint statement on use of AI in health and care professional education.” February 2026. https://www.hcpc-uk.org/education-providers/updates/2026/hcpc-collaborates-on-joint-statement-on-use-of-ai-in-health-and-care-professional-education/
18. Journal of Medical Internet Research Mental Health. “Does the Digital Therapeutic Alliance Exist? Integrative Review.” 2025. https://mental.jmir.org/2025/1/e69294
19. JMIR. “Efficacy of a Conversational AI Agent for Psychiatric Symptoms and Digital Therapeutic Alliance: A Randomized Clinical Trial.” 2025. https://pubmed.ncbi.nlm.nih.gov/41979879/
20. Bipartisan Policy Center. “Paying for AI in U.S. Health Care.” https://bipartisanpolicy.org/issue-brief/paying-for-ai-in-u-s-health-care/
21. Kelley Drye and Warren LLP. “AI Chatbots Face Rising Legal and Legislative Scrutiny.” https://www.kelleydrye.com/viewpoints/blogs/ad-law-access/ai-chatbots-face-rising-legal-and-legislative-scrutiny
22. Nixon Law Group. “How States Are Enforcing New AI Laws in Healthcare and Why It Matters.” https://www.nixonlawgroup.com/resources/how-states-are-enforcing-new-ai-laws-in-healthcareand-why-it-matters
23. NPR. “AI in the mental health care workforce is met with fear, pushback and enthusiasm.” 7 April 2026. https://www.npr.org/2026/04/07/nx-s1-5771707/mental-health-care-workforce-artificial-intelligence-ai
24. Manatt Health. “Health AI Policy Tracker.” https://www.manatt.com/insights/newsletters/health-highlights/manatt-health-health-ai-policy-tracker
25. Governing AI in Mental Health: 50-State Legislative Review. 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12578431/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

On a wet Wednesday evening in November 2025, Portland City Council took a vote that amounted to a municipal exorcism. By a margin of eight to two, with two councillors absent, the council adopted an ordinance amending the Affordable Housing Code to prohibit a very specific kind of ghost: the algorithmic rent-setting device. The ordinance, pushed to the floor by Councillor Angelita Morillo, banned the sale, licence, and use of tools that ingest landlords' competitively sensitive rental data, mix it together in a proprietary blender, and spit out monthly pricing recommendations that somehow, as if by coincidence, tend to rise. On the same day, Oregon announced a proposed seven million dollar settlement with one of the country's largest landlords over illegal rental price fixing. The message was not subtle. The algorithm, the council had decided, was not a neutral tool. It was a participant in a conspiracy that no single human had ever needed to speak aloud.

To understand why a mid-sized American city felt compelled to legislate against a piece of software as if it were a cartel, you have to reckon with an odd fact about the last decade of rental housing in North America: the landlord, increasingly, is not a person. Not in any meaningful sense. It is a stack. A portfolio held by a private equity fund, operated by a property management company, advised by a revenue management platform, screened by a background check vendor, underwritten by an income verification API, and enforced, when necessary, by an automated eviction pipeline. Somewhere in that stack there is usually a human being who signs forms. But the decisions, the ones that matter to a tenant, the rent you pay, the flat you get, the day you are handed a notice to quit, have migrated upstream into systems whose logic is proprietary, whose operators have every incentive not to explain themselves, and whose errors compound at scale.

The Portland ordinance was among the first municipal laws in North America to say, in plain terms, that this arrangement is not acceptable. It will not be the last. In March 2026, the New Jersey Senate Community and Urban Affairs Committee reported Bill S451 with amendments, and a companion bill, A3497, advanced in the Assembly. Sponsored by Senators Brian Stack and Teresa Ruiz, the New Jersey legislation takes a sharper rhetorical line than Portland's: it frames the use of profit-maximising rental algorithms as a potential violation of state antitrust law, a reclassification of software as conspiracy. Governor Mikie Sherrill has signalled that she will sign the measure if it reaches her desk, describing the practice in terms that would have sounded paranoid five years ago: “for-profit surveillance by Big Tech.” And in February 2026, OpenMedia, the Canadian digital rights organisation, published an investigation titled Watch This Space: The Rise of AI Landlords in Canada, a dispatch from a country quietly outsourcing one of its most important social relationships to a handful of opaque vendors, and discovering, too late, how few tools exist to hold them to account.

What is emerging, in other words, is a regulatory front. It is jagged, uncoordinated, and full of holes. But it is real. And it is belated.

The Hub, the Spokes, and the Polite Fiction

The antitrust case against algorithmic rent-setting software rests on a theory as old as cartels themselves. In the classical “hub-and-spoke” conspiracy, a central coordinator channels information among competitors in a way that allows them to align their behaviour without ever meeting in a smoke-filled room. The hub never explicitly tells the spokes to raise prices. The spokes never explicitly agree to. But information flows, prices move, and the market ends up acting as if there had been an agreement, because functionally there was one.

In 2024, the US Department of Justice filed a civil antitrust lawsuit alleging the hub could be software. The defendant was RealPage, a Texas-based property technology company whose YieldStar and AI Revenue Management products had come to dominate the multifamily rental market. According to the DOJ's complaint, RealPage's software ingested non-public, competitively sensitive pricing and occupancy data from participating landlords, ran it through a proprietary algorithm, and returned real-time rent recommendations. Each landlord, acting alone, could plausibly claim to be simply accepting third-party advice. Collectively, they were coordinating rents across thousands of competing properties. The software was doing what a cartel meeting used to do, only faster, more granularly, and with a user interface.

RealPage settled with the DOJ on 24 November 2025, agreeing to stop offering software that uses non-public competitively sensitive data shared among landlords. The final judgement installs an independent monitor for three years. A parallel class action in Tennessee produced preliminary settlements with twenty-six defendants in October 2025. In New Jersey, the state attorney general sued RealPage and ten of the state's largest landlords, alleging revenue management products had been used to inflate rents and eliminate competitive pricing.

Legal academics have spent two years picking apart a question the RealPage case raises but does not fully resolve: at what point does using the same pricing software as your competitors become collusion? The Ninth Circuit, in an August 2025 decision, held that mere parallel use of a common algorithm is not enough; there must be some additional factor, a “plus” in antitrust parlance, that suggests coordination beyond independent business judgement. Critics argue this standard is already obsolete. The whole point of modern revenue management software is that it dissolves the distinction between independent judgement and coordinated behaviour. When your “judgement” is literally reading off a dashboard populated with your competitors' data, there is nothing independent about it. You are just a spoke on a very well-lit wheel.

This is why the Portland ordinance and the New Jersey legislation matter beyond their jurisdictions. They are experiments in skipping past the antitrust doctrinal debate entirely. Rather than arguing, case by case, about whether a particular use of a particular algorithm constitutes a Sherman Act violation, they simply prohibit the class of tools that makes the question interesting. Hoboken passed a similar ordinance in July 2025. Jersey City became the first New Jersey municipality to ban AI-driven rent-setting software in May 2025. San Francisco, Berkeley, Philadelphia, and Minneapolis have followed variously. New York signed its own ban in 2025. The patchwork is real. So is the backlash: RealPage has sued Berkeley alleging pre-emption, and a bill in Congress would block local bans altogether, sponsored by legislators whose campaign contributions from the property technology sector are, to put it charitably, not coincidental.

The polite fiction that algorithmic pricing is a neutral productivity tool, rather than a coordinated market behaviour, is dissolving.

Screening, or the Quiet Part Out Loud

If algorithmic price-setting is the loud scandal, algorithmic tenant screening is the quieter one, and arguably the uglier. Price fixing harms everyone who pays rent. Screening algorithms harm, specifically and disproportionately, the people with the least leverage: low-income applicants, people of colour, housing voucher holders, recent immigrants, people with disability histories, and, as we will see, retired people whose wealth happens to live in the wrong kind of account.

The paradigmatic case is Louis v. SafeRent Solutions, which ended in a 2.275 million dollar settlement in late 2024. Mary Louis, a Black woman in the Boston area, applied for a flat. She had a housing voucher. She had a cosigner with high credit. She had a recommendation from a landlord who had rented to her for seventeen years. SafeRent's scoring algorithm denied her anyway. The score was a number. The number was the decision. When she asked why, nobody could tell her, because nobody knew. The algorithm's weights were proprietary. The landlord had outsourced the judgement. The scoring company had automated it. And Louis, who was very much a person with a life and a rental history and a plan, had been translated into a probability distribution that did not flatter her.

The SafeRent settlement established, in a negotiated form, that an algorithmic screening system producing disparate impact against members of a protected class creates liability under Section 3604 of the Fair Housing Act. Even if each of the algorithm's individual inputs is facially neutral, the output is what matters. This aligns with guidance issued by the United States Department of Housing and Urban Development in May 2023, which stated that housing providers and tenant screening companies that use algorithms are not absolved from liability when their practices disproportionately deny people of colour access to housing.

The problem is that guidance is not law, settlements bind only their parties, and the broader regulatory environment for tenant screening remains, in a word, rickety. In theory, the Fair Credit Reporting Act requires that when a tenant is denied housing based on a consumer report, the landlord must provide an adverse action notice naming the reporting company and explaining the right to dispute. In practice, a survey cited by the Center for Democracy and Technology found that only three per cent of renters report knowing the name of the screening company involved in their denial. Three per cent. Which means ninety-seven per cent of denied applicants have been handed a decision that was probably, under existing law, required to come with a full disclosure, and simply did not.

Reddit, predictably, is where the texture of this failure lives. On forums like r/Tenant, r/Renters, and country-specific subs in Canada and the UK, the rejection stories share a grammar. An applicant submits through a national portal. A form email arrives a day or two later, stating that the application has been declined, that the decision is final, and that the landlord cannot share additional information. No name of the scoring company. No numerical score. No reasoning. Sometimes the rejection arrives within minutes, which is the giveaway: no human has looked. The tenant, if persistent, writes to the property manager. The property manager, if they respond, redirects to the screening vendor. The vendor explains that their system is a “decision support” tool and that the landlord made the decision, which the landlord did, in the sense that they clicked accept on the output. The buck is passed, circulated, and finally mislaid.

This is not an anecdote. It is a structural feature. When every participant can point to another as the “real” decider, nobody is responsible. The algorithm is treated as advisory. The landlord is treated as the decision-maker. The vendor is treated as a mere processor. And the tenant, the one actor whose life is materially altered, is treated as an input.

The Retired Couple Problem

Nowhere is the brittleness of algorithmic screening more visible than in a class of applicant that ought to be trivially easy to approve: the asset-rich, income-light retiree. Consider a retired couple in their seventies. They have sold their family home. They have eight hundred thousand pounds in a combination of index funds, a self-invested personal pension, and ISA savings. They receive modest state pension income plus a small private pension. They want to rent a flat, possibly for the rest of their lives. Any sensible human landlord would recognise the applicants as close to ideal. They are not going to miss a payment. Their wealth is documented and liquid in the relevant sense.

An automated income verification system does not see any of this. It sees a bank account with low deposits relative to expected rent. It sees no salary. It sees a 1099-R in the United States, or a pension statement in the United Kingdom, that does not parse as “employment income” in the system's model. It sees “insufficient income to rent ratio,” flags the applicant, and issues a decline. The couple, baffled, may not even learn the reason. If they do, they face the near-impossible task of explaining, through a web form or a call centre, that their liquidity lives in instruments the software does not understand.

This failure mode is not hypothetical. OpenMedia's February 2026 investigation documented multiple instances of retirees and self-employed applicants being denied by automated screening systems that could not ingest asset-based or non-standard income. Applicants on irregular income, seasonal work, gig platforms, disability benefits, family support, or investment distributions find themselves recursively rejected by systems trained on the assumption that income is a W-2 or a PAYE payslip. The effect is to systematically disadvantage anyone whose financial life does not conform to the assumptions of a mid-twentieth-century labour market.

And the effect compounds. Because these screening systems increasingly operate on overlapping datasets, a rejection by one often produces a cascade of similar rejections elsewhere. A tenant declined by a large property management company in a major metropolitan area may find themselves effectively blacklisted across the private rented sector in their region, without ever having done anything wrong, simply because the algorithm could not parse their life. There is no central registry of these decisions. There is no equivalent of a credit bureau dispute process that reliably works for algorithmic tenant scores. There is, in many cases, not even a name to sue.

What OpenMedia Found

The February 2026 OpenMedia report captures what happens when this infrastructure is built out in a jurisdiction with weaker privacy enforcement than the European Union and more fragmented housing law than the United States. OpenMedia's researchers documented the rapid spread of AI landlord tools across the Canadian rental market, naming vendors including CERTN and Single Key. Their marketing materials promise landlords the ability to scan up to seven years of a prospective tenant's social media activity, search more than a hundred databases of personal information including eviction and criminal records, flag “risky” online behaviour, and generate automated risk scores.

The scope of surveillance on offer, even leaving aside questions of accuracy, is extraordinary. A 2018 report by the British Columbia Information and Privacy Commissioner, cited by OpenMedia, found that ten of thirteen landlords studied were already systematically over-collecting sensitive personal information in violation of the province's Personal Information Protection Act. That was before the current generation of AI screening tools existed. Canadian tenants increasingly apply through portals that ingest years of financial history, employment records, social media handles, criminal background, and in some cases access to open banking feeds that reveal every transaction for months.

The OpenMedia report also documents an aspect that rarely makes headlines: automated eviction initiation. Several Canadian AI landlord platforms now offer modules that flag tenants for non-payment risk and, in some configurations, automatically generate and file notice-to-quit paperwork once a threshold is crossed. The property manager is presented with an eviction already drafted. All they need to do is sign off. The frictionlessness is the point. And the frictionlessness is what makes it dangerous.

In the United Kingdom, similar tools are in circulation, though deployment is more fragmented. The UK's regulatory posture is, in principle, stronger: the UK GDPR, which preserves Article 22, gives tenants a right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects. The Information Commissioner's Office has issued guidance that housing decisions fall squarely within Article 22's scope. In practice, enforcement has been minimal, and the “solely” in “solely automated” has become a loophole: a human clicking an approval button is often treated as meaningful human involvement, even when the human does no independent review. GDPR's promise, in the rental context, has been largely notional.

The Frameworks That Would Need to Exist

If we were to design, from scratch, a legal and ethical framework adequate to algorithmically mediated housing, it would need several overlapping layers. Not one silver bullet. Not one statute. A stack.

The first is algorithmic transparency and the right to an explanation. The model already exists, in imperfect form, in Article 22 of the GDPR and the associated Articles 13 to 15, which grant data subjects a right to meaningful information about the logic involved in automated decisions. In the housing context, any decision to reject an application, to set or raise rent, to initiate eviction, or to classify a tenant as high-risk must be accompanied by an individualised explanation that a reasonable applicant can act on. Not a generic disclaimer. Not “your score was below our threshold.” The features that drove the decision, the weights attached to them, and the means to dispute. California's AB 2930, Colorado's SB 205, and the EU AI Act's high-risk system provisions gesture in this direction. None yet fully deliver.

The second is a modernised adverse action regime. The FCRA, drafted in 1970, was not built for machine learning. Its core insight, that when a consumer is harmed by a report they must be told who provided it and how to dispute it, remains sound. What it needs is an update that treats algorithmic scoring outputs as consumer reports in their own right, regardless of whether the vendor is technically a consumer reporting agency. It needs a private right of action strong enough that violations are actually litigated. And it needs clear rules on data provenance: tenants must be able to know where every input came from, and correct ones that are wrong.

The third is antitrust reform for algorithmic coordination. The Portland ordinance and the New Jersey legislation are, in effect, saying that existing Sherman Act and state antitrust doctrine is not agile enough to handle pricing algorithms that coordinate markets without explicit agreement. They may be right. A federal statute clarifying that the provision or use of pricing software that ingests competitors' non-public data and returns pricing recommendations is per se unlawful, full stop, would resolve most of the doctrinal uncertainty the Ninth Circuit has struggled with.

The fourth is fair housing law modernisation. The Fair Housing Act's disparate impact doctrine, as reaffirmed by HUD's 2023 guidance and the SafeRent settlement, already in principle applies to algorithmic screening. What is missing is pre-deployment obligation: a requirement that any system used to screen tenants be audited for disparate impact before deployment, and periodically thereafter, with results disclosed to regulators and, in summary form, to the public. The EU AI Act has adopted something like this for high-risk systems. US federal law has not.

The fifth is data minimisation and purpose limitation. Why, precisely, does a landlord need seven years of your social media history to decide whether to rent you a flat? They do not. The reason this data is collected is that it is available and that vendors have built products around ingesting it. A defensible regime would restrict the information that can be lawfully used in a tenancy decision to a short, well-justified list: identity, credit history for a defined period, verifiable income or assets, prior eviction judgements. Everything else, social media, location history, network-of-associates data, should be off-limits.

The sixth is a human-in-the-loop mandate with teeth. The current form of “human review” is often performative: a property manager glances at a dashboard and clicks accept. A meaningful mandate would require that any adverse decision, a denial, a rent increase above a threshold, an eviction filing, involve substantive human consideration of the specific circumstances, documented in writing, by a person with authority to override the algorithm. Anything less is GDPR Article 22's “solely automated” dressed in a lab coat.

The seventh is a private right of action. Without one, regulators are the only enforcement mechanism, and regulators are outnumbered, underfunded, and subject to political winds. Tenants harmed by algorithmic decisions must be able to sue, individually or collectively, with the prospect of statutory damages that matter to the vendors' bottom line.

None of this is impossibly difficult to design. All of it is politically difficult to pass.

Why It Has Been So Slow

The lag between the emergence of algorithmic rental infrastructure and the legal response is not accidental. It is the product of several overlapping forces.

The first is lobbying. The real estate technology sector is well capitalised and well connected. RealPage alone has spent substantial sums on lobbying in Washington and in state capitals, and the National Multifamily Housing Council has been an aggressive opponent of algorithmic pricing bans. The Congressional bill that would pre-empt local bans did not arrive by accident. It arrived because the sector organised, wrote briefing papers, cultivated relationships, and showed up.

The second is jurisdictional fragmentation. Housing law in North America is, fundamentally, local. Tenancy rules vary by state, province, county, and city. Fair housing enforcement is divided between federal, state, and municipal bodies with overlapping mandates. Data protection in the United States is a patchwork of sectoral statutes and state laws rather than a comprehensive regime. In Canada, privacy law is split between federal and provincial statutes with varying scope. There is no single venue in which the problem can be addressed.

The third is the “it's just software” framing. Property technology vendors have been remarkably successful at presenting their products as neutral productivity tools, no different in principle from spreadsheets or email. This framing has allowed them to slip past regulatory scrutiny that would have attached to an equivalent human operation. A trade association that circulated pricing data among competing landlords would have been an antitrust target within months. A piece of software that did the same thing operated for over a decade before the DOJ took action.

The fourth is the pace gap between technology and law. Legislatures move slowly. Machine learning systems iterate quickly. By the time a committee has held hearings, drafted a bill, negotiated amendments, and passed a statute, the technology has moved on. In housing, where the human stakes are so direct, the gap is particularly painful.

The fifth, and perhaps most important, is that housing is politically fraught. Every proposed tenant protection collides with a coalition of property owners, developers, landlord associations, and investors who will argue, often successfully, that any new regulation will reduce housing supply and hurt the very people it aims to protect. This argument has merit in some contexts and is made in bad faith in others. The effect, either way, is to make housing reform slow, incremental, and vulnerable to reversal. Banning rent-setting software is presented by its opponents as a form of price control. That the alternative is an effectively unregulated market-wide coordination mechanism does not feature prominently in the industry's press releases.

Housing as a Human Right, and the Asymmetry

Behind the specific technical debates about algorithms and adverse action notices lies a broader ethical question that most regulatory frameworks dance around. If housing is a human right, or at least a precondition for the exercise of other rights, then who gets to live where is not a purely private matter between landlord and tenant. It is a matter in which the state has a legitimate and, arguably, obligatory interest. This framing is common in European human rights jurisprudence and in UN declarations. It is less established in North American constitutional doctrine, where housing is typically treated as a commodity subject to market allocation, moderated by discrimination law and modest subsidies.

The algorithmic turn makes the question urgent. In a market of individual landlords making individual decisions, even when those decisions are discriminatory or arbitrary, the harm is distributed and contestable. A tenant denied by one landlord can try another. The friction of the market is the friction of a thousand independent actors.

In a market mediated by a handful of algorithmic vendors, all of whom share similar data, similar models, and similar blind spots, the friction collapses. Being rejected by one system is being rejected by most. Being flagged as high-risk by one score is being flagged similarly across portfolios. The asymmetry between landlord and tenant, which was always real, becomes qualitatively different when the landlord's side of the relationship is automated, networked, and effectively infinite, while the tenant's side remains one anxious person with a phone and a rejection email.

This is the ethical heart of the matter. The algorithm does not just mediate; it structurally biases the relationship in favour of the side that deployed it. It is not a neutral tool. It is a party. And when it is a party that cannot be questioned, sued effectively, or compelled to explain, the relationship ceases to have the features we normally associate with fair dealing between legal persons. It becomes an administrative system with the power of a landlord and the accountability of a weather pattern.

What the Portland Vote Actually Represents

Which brings us back to that wet Wednesday in November. The Portland ordinance, in the grand sweep of housing policy, is small. It will affect a single city. It carries civil penalties of up to a thousand dollars per violation, which is, by the standards of the entities it regulates, loose change. Its practical impact on rents in Portland will be debated for years.

And yet. The ordinance represents something the regulatory conversation has been missing: a willingness to treat algorithmic infrastructure as a political object rather than a technical inevitability. Portland's council did not hold a symposium on whether large language models will bring about the singularity. They did not convene a blue-ribbon commission to study the metaphysics of automated decision-making. They looked at a specific piece of software, understood what it did, concluded that what it did was unlawful if done by humans, and made it unlawful when done by software. It was, in a sense, a deeply old-fashioned move. It treated the technology as conduct, and conduct as regulable.

New Jersey's March 2026 legislation extends the logic. OpenMedia's February investigation documents the scale of what has been built. HUD's guidance, the SafeRent settlement, the RealPage consent decree, and the various state and municipal ordinances form, together, the rough outline of a response. Not a coherent framework, not yet. But the scaffolding of one.

The question is whether the federal government, or a coalition of states, can assemble the pieces into something that actually protects tenants. The reasons to persist are the stories that show up, daily, in tenant forums and courtrooms and outside the flats that retirees have been refused because a computer could not count their pension.

A decade from now, we will either look back at 2025 and 2026 as the moment the law began to catch up with the infrastructure of rental housing, or as the moment a handful of cities shouted into a wind that kept blowing. Portland voted eight to two. New Jersey's Senate committee voted to advance. OpenMedia published. The DOJ sued and settled. The work, as ever, is in the follow-through. The algorithm is not going to regulate itself. It does not notice whether the person it just declined is Mary Louis, or a retired couple, or someone whose only crime was having the wrong kind of income in the wrong kind of account.

That work, still, belongs to us.

References and Sources

1. City of Portland. “Amend Affordable Housing Code to add prohibition of anti-competitive rental practices including the sale and use of algorithmic devices.” November 2025. portland.gov/council/documents/ordinance/algorithmic-rental-pricing
2. Portland Mercury. “Portland City Council Votes to Adopt AI Rental Price-Fixing Software Ban.” 19 November 2025.
3. US Department of Justice. “Justice Department Requires RealPage to End the Sharing of Competitively Sensitive Information.” 24 November 2025.
4. US Department of Justice. “Justice Department Sues RealPage for Algorithmic Pricing Scheme That Harms Millions of American Renters.” 2024.
5. Reed Smith. “Algorithmic pricing under pressure: DOJ's RealPage settlement changes the rules for rental markets.” 2025.
6. New Jersey State Policy Lab, Rutgers University. “Senate Committee Advances Ban on Rent-Setting Algorithms.” 9 March 2026.
7. Multifamily Dive. “New Jersey mulls algorithmic rent pricing ban.” 2026.
8. City of Hoboken. “City of Hoboken to introduce ordinance prohibiting algorithmic rent-fixing.” 2025.
9. Insider NJ. “Jersey City Council Advances Ordinances that Ban AI-Powered Rent-Fixing Algorithms.” 2025.
10. Arnold & Porter. “Algorithmic Pricing Bans Go Coast to Coast.” October 2025.
11. OpenMedia. “Watch This Space: The Rise of AI Landlords in Canada.” February 2026.
12. American Bar Association, Human Rights Magazine. “Ghosts in the Machine: How Past and Present Biases Haunt Algorithmic Tenant Screening Systems.” June 2024.
13. Center for Democracy and Technology. “Tenant Screening Algorithms Enable Racial and Disability Discrimination at Scale.”
14. Consumer Financial Protection Bureau. “Consumer Snapshot: Tenant Background Checks.” November 2022.
15. The Leadership Conference on Civil and Human Rights. “AI + Tenant Screening.”
16. TechEquity Collaborative. “Traditional vs. algorithmic tenant screening.” 10 July 2024.
17. Georgetown Law Poverty Journal. “The Discriminatory Impacts of AI-Powered Tenant Screening Programs.”
18. American University Business Law Review. “Screened Out: How Faulty Algorithms Are Shutting Doors on Fair Housing.” September 2024.
19. California Law Review. “A Home for Digital Equity: Algorithmic Redlining and Property Technology.”
20. Information Commissioner's Office (UK). “What is the impact of Article 22 of the UK GDPR on fairness?”
21. General Data Protection Regulation. “Article 22: Automated individual decision-making, including profiling.” gdpr-info.eu/art-22-gdpr
22. Gothamist. “Hochul signs bill banning NY landlords from using algorithm software to set rents.” 2025.
23. Freshfields. “Settling Defendants to Pay over $141 million to Settle RealPage Price-Fixing Class Action Claims in Tennessee.” October 2025.
24. Arnold & Porter. “Ninth Circuit Clarifies Antitrust Implications of Algorithmic Pricing.” August 2025.
25. HUD and DOJ joint guidance on algorithm-based tenant screening. May 2023.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

On a humid July afternoon in Dover, Florida, an 82-year-old grandmother named Sharon Brightwell answered the telephone and heard her daughter crying. The voice was unmistakable. It sobbed, it choked on the words, it begged. There had been a car accident. A pregnant woman had been hurt. A lawyer would call shortly with instructions. Brightwell, who had raised this daughter, who had rocked her to sleep, who knew the exact timbre of her weeping because she had heard it a thousand times across half a century, did what any mother would do. She emptied an envelope of cash. She handed fifteen thousand dollars to a courier who appeared at her door. By the time she realised her daughter had never been in an accident at all, the money and the courier were gone.

The voice she heard was not her daughter. It was a synthetic reconstruction, stitched together by a generative model from audio scraped off the open internet, probably from a social media post, possibly from a voicemail greeting, certainly from nothing more than a few seconds of casual speech. The emotional content, the terror and the tears, was added by the same model as a matter of routine. Running the entire performance cost pennies. Producing a convincing clone of an unsuspecting person's voice, according to every major consumer research organisation that has tested the tools in the last eighteen months, now requires as little as thirty seconds of source material. Some academic demonstrations have done it with three.

That is the hinge on which this story turns, and it is the reason the United States has quietly slipped into one of the strangest fraud epidemics it has ever faced. In 2024, older Americans reported losing almost five billion dollars to scams and fraud, a figure that the Federal Trade Commission itself regards as a dramatic undercount. The agency's most recent report to Congress, issued in December 2025, estimates that the true cost of fraud against Americans aged 60 and over sat somewhere between ten billion and eighty-one and a half billion dollars last year, depending on the model used to correct for the torrent of cases that victims never report out of shame, confusion, or cognitive decline. Within that figure, the fastest-growing category, and by almost every measure the most psychologically ruinous, is the one that depends on cloned voices.

More than 75,000 consumers have now signed a petition urging the FTC to act. A bipartisan Senate bill introduced in December 2025 would criminalise the very act of using AI to impersonate someone with intent to defraud. And in April 2026, the Journal of Accountancy, a publication not generally given to panic, ran a feature instructing certified public accountants in the practical steps they now need to take to protect their elderly clients from being drained by phantom grandchildren. The professional class is catching up to what grandparents already know. Something has gone very wrong with the voice.

A weapon made of pennies

The mechanics are almost insultingly simple. A scammer needs three ingredients: a target, a sample of the target's loved one speaking, and a voice cloning tool. The first two are trivial. Older Americans are plentiful and reachable by phone. Voice samples have been uploaded by the billions to TikTok, Instagram, YouTube, Facebook, and countless wedding videos, podcast appearances, voicemail greetings, Zoom recordings, and church livestreams. The third ingredient used to be the hard part. It no longer is.

Commercial voice synthesis platforms such as ElevenLabs, Speechify, PlayHT, and LOVO have, over the last three years, made voice cloning available to anyone with a credit card and, in many cases, anyone willing to tick a box attesting that they have the legal right to reproduce the voice in question. A March 2025 assessment by Consumer Reports examined six leading voice cloning products and found that four of them, including three of the most widely used, relied exclusively on that self-attestation as their safeguard. The researchers who performed the test were able to clone real voices without providing any evidence of consent. The box was ticked, the clone was generated, the guardrail did nothing.

The FBI's Internet Crime Complaint Center reported that AI-enabled fraud accounted for roughly 893 million dollars in losses across more than 22,000 complaints in 2025, with around 352 million of that total attributed to elder fraud complaints in which an AI component was documented. Those are floors, not ceilings, because most victims do not file complaints and most families do not realise AI was involved when they do. Researchers inside Microsoft's AI for Good Lab, analysing 531,000 fraud reports drawn from AARP's Fraud Watch Network Helpline and the Better Business Bureau's Scam Tracker, reported that scams identified as AI-enabled, with realistic cloned voices or synthetic video, increased twentyfold between 2023 and 2025. Twentyfold. In two years.

The economics are what make it terrifying. A traditional phone scam requires a human operator, a convincing script, a plausible accent, and the nerve to hold a conversation with someone who may ask unexpected questions. Even the best of those operations run on thin margins. A generative voice scam requires none of that. The attacker can produce a bespoke audio deepfake in seconds, run the call through a VoIP provider, spoof the caller ID to match a known family number, and be off the line before a victim even registers that something is wrong. The cost of attacking one thousand targets is barely higher than the cost of attacking one. The cost of attacking ten thousand is not much higher than that.

Why the voice is special

There is a reason voice cloning is working as well as it is, and it has almost nothing to do with technology. It has to do with evolution.

The human auditory system is wired to treat voice as an exceptionally high-trust channel. Infants recognise their mother's voice within days of birth. By the time we reach adulthood, we can identify the voice of a close relative from a single syllable, across decades, through distortion, over a crackling phone line, even when we have not heard it for years. More importantly, voice is tightly coupled to emotion. The acoustic signatures of distress, fear, and pain trigger physiological responses in listeners that bypass conscious deliberation entirely. A mother hearing what she believes is her child crying on the telephone will release a cascade of stress hormones before any rational assessment has begun.

Jennifer DeStefano, a mother in Scottsdale, Arizona, described this experience in searing detail to the United States Senate Judiciary Committee in June 2023. She had received a call while her 15-year-old daughter Brianna was away skiing. She heard Brianna's voice sobbing “Mom!” and then a man's voice demanding a million dollars in ransom. She knew that voice. She had heard it, she testified, her entire life. She would never be able, she told the senators, to shake the sound of those desperate cries out of her mind. Only a coincidence saved her: a bystander handed her a separate phone, on which her actual daughter, safe and confused, was calling. Had that not happened, DeStefano would have wired whatever the callers demanded, because she was not making a financial decision. She was making a biological one.

This is the insight that voice cloning fraud weaponises with ruthless precision. The scams do not target rationality. They target the part of the brain that evolved to respond to a child in danger before the conscious mind has caught up. No amount of public awareness campaigning about “stop, verify, call back” survives first contact with that response, because the response is older than language itself. Older people, who are more likely to live alone, more likely to be isolated from the family members being impersonated, and more likely to have significant savings that can be moved in a single wire transfer, sit at the intersection of every vulnerability the attack model exploits.

Amy Nofziger, director of fraud victim support at AARP, has been telling anyone who will listen that the old mental model of the savvy-versus-gullible victim is obsolete. Her framing, delivered across AARP public briefings and research publications, is that this is not a matter of whether a person is smart enough to spot a scam but whether their nervous system can out-think a sound it has been trained to trust since birth. An AARP survey conducted in August 2024, fielded to a thousand adults aged 50 and over via a probability-based panel, found that 77 per cent of older Americans were concerned about becoming targets of AI-related fraud, and 85 per cent were worried about deepfakes generally. Concern, however, does not translate into immunity. In the same research, respondents massively overestimated their own ability to detect a cloned voice.

The regulatory scramble

Washington has noticed, but not at the speed the problem requires. The Federal Trade Commission finalised its rule on government and business impersonation in early 2024, following years of mounting complaints about scammers posing as the Internal Revenue Service, the Social Security Administration, Amazon customer service, and dozens of other familiar institutions. The rule was a genuine step forward. It gave the commission direct authority to sue impersonators and recover money for victims, and it laid the groundwork for an expansion to cover the impersonation of private individuals, an extension the agency had been seeking.

That expansion is where the fight has stalled. In February 2024, the FTC released a supplemental notice of proposed rulemaking that would cover the impersonation of individuals. Consumer Reports, joined by the Electronic Privacy Information Center, the National Consumers League, and other advocacy organisations, delivered a petition in August 2025 signed by more than 75,000 consumers demanding the commission finalise the individual impersonation rule and invoke its Section 5 powers to pursue the companies whose voice cloning products are enabling the scams in the first place. The petition called on the FTC to investigate the product-design failures, the absence of meaningful safeguards against cloning without consent, and the ease with which commercial voice tools can be turned into engines of fraud. As of April 2026, the rule remains under review. The commission has not yet moved to act against any of the major voice cloning vendors.

Congress has been marginally more active. In April 2025, Senators Chris Coons, Marsha Blackburn, Amy Klobuchar, and Thom Tillis reintroduced the Nurture Originals, Foster Art, and Keep Entertainment Safe Act, universally referred to as the NO FAKES Act. Originally conceived as a tool to protect musicians, actors, and other creative professionals from having their voices and likenesses replicated without consent, the bill would establish a federal right for every American to their own voice and visual likeness, create a notice-and-takedown regime for unauthorised deepfakes, and preempt the patchwork of state laws that has grown up in the meantime. The legislation has drawn support from SAG-AFTRA, the Recording Industry Association of America, the Motion Picture Association, OpenAI, and YouTube, which is an unusual coalition by any measure.

It has also drawn criticism. The Foundation for Individual Rights and Expression has argued that the bill's takedown provisions could be abused to suppress legitimate speech, and the Electronic Frontier Foundation has raised concerns about the scope of secondary liability for platforms. The result is that the NO FAKES Act, now in its third legislative cycle, remains unpassed.

More narrowly focused is the Preventing Deep Fake Scams Act, introduced by Senators Jon Husted and Raphael Warnock in June 2025 and endorsed by AARP, which would establish a federal task force led by the Treasury Department and financial regulators to coordinate a response to AI-driven fraud against financial institutions and their customers. And on 17 December 2025, Senators Shelley Moore Capito and Amy Klobuchar introduced the Artificial Intelligence Scam Prevention Act, a bipartisan bill that would make it illegal to use AI to impersonate any person with intent to defraud, and would establish an interagency committee bringing together the FTC, the FCC, the Consumer Financial Protection Bureau, and the Justice Department to coordinate enforcement.

None of these bills has yet become law. All of them, read together, suggest that even the most energetic legislators understand the problem to be growing faster than the legislative response. The April 2026 issue of the Journal of Accountancy, in a feature authored by forensic accountants David Zweighaft and Howard Silverstone, framed the matter in terms the professional services industry cannot ignore. Certified public accountants, the article argued, now have a practical obligation to warn elderly clients about voice cloning, to help them establish family verification codewords, and to build transaction-review processes that can flag urgent-sounding wire requests before the money leaves the account. It was a notice, delivered to a constituency that does not panic easily, that this is no longer hypothetical.

Defences out of pace

The technical defences against voice cloning fraud fall into three categories, and in April 2026, none of them works reliably.

The first category is detection. Academic and industry researchers have, over the last two years, produced a growing literature on audio deepfake detection, using machine-learning classifiers trained to distinguish synthesised speech from natural speech. A systematic 2025 analysis published in the ACM Transactions on Internet Technology, along with a peer-reviewed survey appearing in Engineering Reports, concluded that detection models perform well on the datasets they were trained on and collapse, sometimes catastrophically, when confronted with audio generated by unseen models. A paper presented at the Network and Distributed System Security Symposium in 2025, introducing a system called VoiceRadar, demonstrated that micro-variations in synthetic speech can be detected under controlled conditions but noted that adversarial retraining by attackers can neutralise those signals within weeks. More disturbingly, a 2025 study on synthetic speech detection reported that current systems exhibit demographic bias, with markedly higher false-positive rates for elderly speakers, adolescents, and speakers of certain English dialects, precisely the populations most often impersonated or most often targeted.

In practical terms, this means that by the time a consumer-grade voice deepfake arrives on someone's phone, no widely deployed tool can reliably tell them it is fake. Phone carriers do not scan audio content in real time. Call authentication protocols such as STIR/SHAKEN, which the Federal Communications Commission mandated across the United States telecom industry to combat robocalling, verify the origin and legitimacy of a calling number but say nothing about whether the voice on the other end of a legitimately placed call is human, synthetic, or stolen. STIR/SHAKEN was built for a world before generative AI. In that world, if you knew who was calling, you had a reasonable chance of knowing what they were going to say. That assumption no longer holds.

The second category is provenance. The Coalition for Content Provenance and Authenticity, known as C2PA, has since 2021 been building an open technical standard for cryptographically signing media at the point of creation so that downstream consumers can verify its origin and editing history. A new version of the specification was published in May 2025, and the standard is expected to be adopted as an ISO international standard this year. Microsoft, Adobe, Intel, Google, and several major news organisations have committed to implementing it. In principle, a C2PA-signed audio file can be traced back to its source, and synthetic audio generated by a compliant tool can be flagged as such.

In practice, provenance solves a different problem. It works beautifully for controlled creative pipelines, where a tool like a professional video editor or a generative image app stamps its output with verifiable credentials. It is much less effective against the scam caller who records a voice from a TikTok clip, strips the metadata, feeds it into a compliant or non-compliant voice model, and then pipes the output through a telephony gateway that was never designed to preserve signed audio. The telephone network, the channel through which the overwhelming majority of voice fraud is delivered, strips content credentials by design. Provenance works upstream. Fraud happens downstream.

The third category is authentication at the human level, which is the category most likely to actually help in the short term. The single most consistently effective defence that every major fraud researcher, from the FBI to AARP to the Journal of Accountancy to Consumer Reports, now recommends, is the family verification codeword: a simple, private phrase shared among relatives and used to confirm identity in emergencies. If a grandmother receives a call from her weeping grandson, she asks for the codeword. The real grandson knows it. The fraudster does not. This is, in the end, pre-industrial cryptography, a shared secret used to verify identity in a world where the cryptographic infrastructure we have built cannot reach the kitchen telephone.

Who is supposed to stop this?

The hardest question is not technical. It is the question of responsibility, and the answer involves six constituencies that have historically preferred to pass the problem to one another.

The voice model developers, companies such as ElevenLabs, LOVO, Speechify, PlayHT, and the large foundation-model labs whose capabilities underpin all of them, are the closest to the attack surface. The product can do what it can do because they built it that way. Consumer Reports' March 2025 findings were unambiguous: the majority of commercial voice cloning products tested lacked meaningful safeguards against unauthorised cloning. Some, like Resemble AI and Descript, required more than a ticked box. Most did not. Senator Maggie Hassan, in formal letters sent to ElevenLabs and three competitors in April 2026, asked each company to explain exactly how they prevent their tools from being used for fraud. The replies have been limited. ElevenLabs, to its credit, blocks the cloning of certain high-profile celebrity voices and uses internal classifiers to monitor for misuse, but the systems are imperfect and the incentives to prioritise growth over gatekeeping are, for a venture-funded company, almost irresistible.

The platforms that host voice samples are the second constituency. TikTok, Meta, YouTube, and their peers have built their business models around the frictionless sharing of audiovisual content, including audio of children, grandparents, and other family members who have no knowledge that their voices are being harvested. Default privacy settings on most of these platforms are permissive. Third-party scraping tools are widely available. No major social platform has yet committed to audio-scraping countermeasures or to robust default privacy settings that would shield users' voices from mass harvesting. Until they do, the raw material for every cloning scam will remain trivially accessible.

The telecommunications carriers are the third. They are the pipeline through which every fraudulent call travels. Their current fraud-prevention investments focus on number-level authentication, not content-level scrutiny. A content-aware defence, one that could scan incoming audio for markers of synthesis in real time, is technically plausible but would raise serious questions about call-content surveillance and would require regulatory scaffolding the United States does not currently possess. In the absence of such a framework, carriers remain, essentially, neutral conduits. The fraud flows through them as surely as the legitimate calls do.

The banks and wire-transfer services sit at the fourth position, and they are arguably the most capable of intervening because they are the last line before the money is gone. Every large wire transfer, every emergency cash pickup, every cryptocurrency exchange purchase executed in a single afternoon by a panicked elderly customer is a signal. Some banks have built transaction-monitoring systems that flag such patterns, and some branch staff have been trained to ask verification questions when a long-standing customer suddenly demands a large cash withdrawal for an unknown recipient. The bank teller in the Canadian case cited by CBC News, who stopped a grandmother from wiring nine thousand dollars to a claimed kidnapper, is exactly the kind of intervention point that works. But training is inconsistent, oversight is voluntary, and liability for downstream losses remains, in most US jurisdictions, the customer's.

The regulators are the fifth. The FTC, the FCC, the Consumer Financial Protection Bureau, the Securities and Exchange Commission, and state attorneys general all have jurisdictional claims somewhere in the fraud chain. None of them has yet been given the explicit statutory authority, the budget, or the coordinated mandate required to act as a unified response. The interagency committee envisioned by the December 2025 Capito-Klobuchar bill, if it becomes law, would be a start. Whether it would be adequate to a problem that compounds at twentyfold every two years is another question.

The sixth constituency is families themselves, and this is the uncomfortable part. The codeword, the verification call-back, the insistence on privacy settings, the education of elderly relatives about what is possible and what to watch for, all of this is work that falls on people who did not ask to become the first line of defence against a technology they do not fully understand. Shifting the burden of fraud prevention onto victims is, historically, the signature of a policy failure. And yet the codeword, today, works. No legislation, no regulator, no platform commitment currently works as reliably.

The trust question

There is a larger question beneath all of this, and it does not reduce cleanly to policy. It is a question about what happens to intergenerational trust in a world where the sound of a loved one's voice can no longer be presumed to be the loved one.

For the whole of human history, voice has been what philosophers have called a “warrant” of presence. Hearing someone's voice was evidence, perhaps the oldest form of evidence, that they were there, that they were real, that they were who they claimed to be. This assumption underwrote almost every important human relationship that was not conducted face to face. It underwrote the telephone as a technology, the voicemail as a convenience, the grandparent-grandchild call as a ritual. It was the reason, in the end, that voice scams worked at all. You believed the voice because voice was something that could be believed.

That assumption is now being eroded in real time, and the erosion is happening unevenly. Younger people, who have grown up around voice filters, autotune, AI-generated podcasts, and synthetic TikTok audio, have developed a generalised scepticism that older generations have not had the time or cultural context to acquire. An older person who has spent sixty years treating the telephone as a trusted channel cannot retool that instinct in an afternoon. The asymmetry is precisely what the attackers exploit.

What gets lost, if the trend continues, is not just money. It is a feature of family life that has existed since the invention of the telephone: the ability to hear a grandchild's voice and to know, instantly and without effort, that it is the grandchild. If every such call must now be preceded by a verification protocol, if every loved voice must be met with a reflexive “what is our codeword?”, something has changed about what it means to be in a family. The burden of suspicion, which used to live on the perimeter of our lives, has migrated inward.

AARP and its peers have tried, carefully, to describe this shift without causing the very panic that would make older adults withdraw from the phone altogether. Their advice in 2026 is consistent: have the codeword conversation, practise it, make it a ritual, do not treat it as a sign of mistrust but as a sign of love in a changed landscape. The framing matters. If the codeword is understood as a form of hygiene, like locking the front door, it becomes bearable. If it is understood as a sign that the voice itself can no longer be trusted, it becomes something else. It becomes an admission of defeat.

What effective protection would actually require

A serious response, one proportionate to the scale of the threat, would combine technical, legal, social, and platform-level interventions. It would begin at the model layer, where voice cloning companies would be required, by binding regulation, to implement robust consent-verification before cloning any voice, to watermark their output in forms that survive normal telephony transmission, and to maintain traceable provenance records accessible to law enforcement. Self-attestation is not a safeguard. It is a liability shield dressed up as a safety feature.

It would continue at the platform layer, where social networks hosting audio content would default to privacy settings that prevent mass scraping of users' voices, particularly those of minors, and would provide users with tools to audit and restrict the use of their voice recordings. This is not a speech issue. It is an infrastructure issue. The scale at which audio can currently be harvested from public platforms was not the intent of any of the people who uploaded that audio.

It would extend to the telecommunications layer, where carriers would be required to develop and deploy content-aware fraud-detection capabilities, with regulatory frameworks that make clear what such systems can and cannot do. STIR/SHAKEN is not enough. A protocol that authenticates numbers without authenticating voices is a partial answer to a problem that has fully evolved.

It would impose liability at the financial-services layer, making banks and wire services responsible for detecting and halting transactions that bear the signature of urgent-request fraud, and giving victims clear legal recourse when obvious warning signs are missed. It would give the FTC the final authority it has been asking for to pursue the individual-impersonation rule and to act against the product manufacturers whose tools are being used. It would pass the Artificial Intelligence Scam Prevention Act, or something like it, to create the federal criminal prohibition that does not currently exist. And it would fund, with federal dollars, the awareness and codeword-education campaigns that are currently being run, on shoestring budgets, by AARP and a handful of consumer advocacy organisations.

None of this is impossible. Most of it has been proposed, some of it in serious detail, in legislation already pending in the 119th Congress. What is missing is not the blueprint. What is missing is the velocity. The gap between the speed at which generative voice technology is proliferating and the speed at which the country's regulatory and platform responses are arriving is, right now, widening. Every additional month in which the FTC does not finalise its individual-impersonation rule, in which no federal statute criminalises AI-driven impersonation fraud, in which major voice platforms continue to rely on tick-box consent, in which social networks continue to default to public audio sharing, is a month in which more Sharon Brightwells empty envelopes of cash to couriers they will never see again.

The mechanism of the scam is new. The dilemma is old. A society has to decide how much of the burden of new technological harm it is prepared to place on the people least equipped to bear it, and how much it is prepared to impose on the institutions that built, distributed, and profited from the tools in the first place. So far, the distribution has been badly skewed. The voice cloners are paying pennies. The victims are paying with their savings, their dignity, and, in some cases, their ability to ever again pick up the telephone and believe what they hear.

If effective protection is possible, and it is, it will look like a deliberate rebalancing of that ledger. It will involve regulators willing to act before every piece of evidence is in, platforms willing to inconvenience their growth curves, carriers willing to be more than neutral, banks willing to own the moment of the transfer, model developers willing to build products that refuse to do the most dangerous things they are capable of doing, and families willing, as they have always been willing when pressed, to protect one another with the tools they have. The codeword is a start. It is not a strategy. The strategy is still, in April 2026, being written, and the clock is running at the speed of the next cloned voice.

References

1. Federal Trade Commission, Protecting Older Consumers 2024-2025: A Report of the Federal Trade Commission, December 2025. https://www.ftc.gov/system/files/ftc_gov/pdf/P144400-OlderAdultsReportDec2025.pdf
2. AARP, “$12.5 Billion Lost to Scams and Fraud in 2024, Older Adults Hit Hard,” 2025. https://www.aarp.org/money/scams-fraud/older-adults-ftc-fraud-report/
3. Greg Iacurci, “Financial fraud cost older adults up to $81.5 billion in 2024, FTC estimates,” CNBC, 13 December 2025. https://www.cnbc.com/2025/12/13/financial-fraud-seniors-ftc.html
4. Consumer Reports, “More than 75,000 consumers urge FTC to crack down on AI voice cloning fraud,” press release, August 2025. https://advocacy.consumerreports.org/press_release/more-than-75000-consumers-urge-ftc-to-crack-down-on-ai-voice-cloning-fraud/
5. Consumer Reports, “Consumer Reports' Assessment of AI Voice Cloning Products,” press release, March 2025. https://www.consumerreports.org/media-room/press-releases/2025/03/consumer-reports-assessment-of-ai-voice-cloning-products/
6. Federal Trade Commission, “FTC Proposes New Protections to Combat AI Impersonation of Individuals,” press release, February 2024. https://www.ftc.gov/news-events/news/press-releases/2024/02/ftc-proposes-new-protections-combat-ai-impersonation-individuals
7. Federal Trade Commission, “Approaches to Address AI-enabled Voice Cloning,” April 2024. https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/04/approaches-address-ai-enabled-voice-cloning
8. U.S. Senator Amy Klobuchar, “Klobuchar, Coons, Blackburn and Colleagues Reintroduce Bipartisan NO FAKES Act,” press release, April 2025. https://www.klobuchar.senate.gov/public/index.cfm/2025/4/klobuchar-coons-blackburn-and-colleagues-reintroduce-bipartisan-no-fakes-act
9. H.R.2794, 119th Congress (2025-2026), NO FAKES Act of 2025. https://www.congress.gov/bill/119th-congress/house-bill/2794/text
10. S.2117, 119th Congress (2025-2026), Preventing Deep Fake Scams Act. https://www.congress.gov/bill/119th-congress/senate-bill/2117/text
11. U.S. Senator Amy Klobuchar, “Klobuchar, Capito Introduce Bipartisan Artificial Intelligence Scam Prevention Act,” press release, 17 December 2025. https://www.klobuchar.senate.gov/public/index.cfm/2025/12/klobuchar-capito-introduce-bipartisan-artificial-intelligence-scam-prevention-act
12. AARP, “AARP Endorsement of Preventing Deep Fake Scams Act,” 2025. https://www.husted.senate.gov/wp-content/uploads/2025/06/AARP-Endorsement-of-Preventing-Deep-Fake-Scams-Act-119th-Senate1.pdf
13. David Zweighaft and Howard Silverstone, “Elder fraud rises as scammers use AI,” Journal of Accountancy, April 2026. https://www.journalofaccountancy.com/issues/2026/apr/elder-fraud-rises-as-scammers-use-ai/
14. Jennifer DeStefano, “Written Statement to the United States Senate Committee on the Judiciary,” 13 June 2023. https://www.judiciary.senate.gov/imo/media/doc/2023-06-13%20PM%20-%20Testimony%20-%20DeStefano.pdf
15. AARP, “AI-Powered Scams Make Fraud Even Harder to Spot,” 2025. https://www.aarp.org/money/scams-fraud/detecting-ai-fraud/
16. Biometric Update, “Voice cloning tools give rise to cacophony of impersonation fraud,” August 2025. https://www.biometricupdate.com/202508/voice-cloning-tools-give-rise-to-cacophony-of-impersonation-fraud
17. ElevenLabs, “Safety.” https://elevenlabs.io/safety
18. CBC Marketplace, “How con artists are using AI voice cloning to upgrade the grandparent scam,” 2025. https://www.cbc.ca/news/marketplace/marketplace-ai-voice-scam-1.7486437
19. American Bar Association Senior Lawyers Division, “The Rise of the AI-Cloned Voice Scam,” September 2025. https://www.americanbar.org/groups/senior_lawyers/resources/voice-of-experience/2025-september/ai-cloned-voice-scam/
20. Coalition for Content Provenance and Authenticity, Content Credentials White Paper, October 2025. https://c2pa.org/wp-content/uploads/sites/33/2025/10/content_credentials_wp_0925.pdf
21. National Security Agency, “Strengthening Multimedia Integrity in the Generative AI Era,” January 2025. https://media.defense.gov/2025/Jan/29/2003634788/-1/-1/0/CSI-CONTENT-CREDENTIALS.PDF
22. “Where are We in Audio Deepfake Detection? A Systematic Analysis over Generative and Detection Models,” ACM Transactions on Internet Technology, 2025. https://dl.acm.org/doi/10.1145/3736765
23. H. Shaaban et al., “Audio Deepfake Detection Using Deep Learning,” Engineering Reports, 2025. https://onlinelibrary.wiley.com/doi/full/10.1002/eng2.70087
24. Network and Distributed System Security Symposium, “VoiceRadar: Voice Deepfake Detection using Micro-frequency Variations,” 2025. https://www.ndss-symposium.org/wp-content/uploads/2025-3389-paper.pdf
25. Federal Communications Commission, “Promoting Caller ID Authentication to Combat Illegal Robocalls.” https://docs.fcc.gov/public/attachments/DOC-366783A1.pdf

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The voice command is simple. “Call my sister.” The user, sitting at a kitchen table in south London, says it three times, each time more slowly, each time more carefully. The smart speaker responds each time with the same brisk cheerfulness. It has heard “call Maria.” It has heard “call the sister.” It has, on the third attempt, offered to play a song called “Sister” by an artist she has never heard of. What it has not done is the thing she asked. Her speech, shaped by a neuromuscular condition that makes consonants softer and vowels longer, sits just outside the envelope of audio the model was trained to recognise. To the speaker's statistical ear, she is not quite a person giving an instruction. She is noise, or close enough to noise that the cheapest path is to guess.

On any given morning, a version of that scene plays out in hundreds of thousands of homes. It is one of the quieter harms of the current artificial intelligence moment, a harm so ordinary that the people experiencing it have mostly stopped complaining about it. You learn, over time, to pitch your voice higher. You learn to flatten your accent. You learn which words the machine prefers and which ones it cannot parse. You build, in other words, a second self, optimised for the model. And then you watch the technology press describe the same model as an assistant that understands natural language.

The argument that this is structural rather than a peripheral bug appeared in sharp form in a January 2026 Forbes analysis by Gus Alexiou, a long-time disability inclusion contributor who has written about assistive technology since his own multiple-sclerosis diagnosis. Alexiou framed disability as the ultimate stress test for AI. When a system optimises for uniform productivity and standardised interaction, the argument ran, it does not merely under-serve the approximately 1.3 billion people the World Health Organization counts as experiencing significant disability. It structurally excludes them, through the quiet accumulation of design defaults that never imagined them in the room.

Three weeks later, on 3 February 2026, the British government put that argument into institutional form. The Department for Work and Pensions, under Secretary Pat McFadden, convened a roundtable that brought Google, Meta, Microsoft and Amazon into the same room as Scope, Guide Dogs UK, AbilityNet, Disability Rights UK, the Business Disability Forum, RNID, the Lightyear Foundation, the Regional Stakeholder Network, the Global Disability Innovation Hub and the Atech Policy Lab. The framing was employment-focused, a conversation about closing the disability employment gap. The subtext was the one Alexiou had articulated in Forbes: the distance between what AI could do for disabled workers and what it currently does, and the question of who was on the hook for closing it.

Around the same time, a framework appeared in Frontiers in Digital Health. Written by Gabriella Waters of the Cognitive and Neurodiversity AI Lab at Virginia State University and Morgan State's Center for Equitable AI and Machine Learning Systems, the paper argued something that should have been obvious: there is no standardised methodology for evaluating whether AI systems work for disabled users, and the dominant evaluation practices actively disguise their failure to do so. Accuracy and generalisability, the two metrics that have governed machine-learning benchmarks for a decade, treat disabled users as statistical noise rather than as a population whose performance matters. Waters' proposed framework covers red teaming, model testing, field testing and usability testing with disabled participants, dressed up in the methodological language the field demands before it will listen.

Picking up the same thread from a different angle, a Rest of World investigation by deputy editor Rina Chandran, published 12 March 2026, traced the failure of Western AI models in overseas agricultural settings. Catherine Nakalembe, the University of Maryland geographer who won the 2020 Africa Food Prize, had to collect more than five million helmet-mounted camera images across Kenya and Uganda to train a system to recognise maize, beans and cassava because existing models could not. Arti Dhar, co-founder of Farmers for Forests, found that a widely used open-source segmentation model missed more than half the trees in a Maharashtra forest because it had been trained on North American species. Digital Green's FarmerChat, by contrast, works across 16 vernacular languages in South Asia and Africa because it was built with the farmers rather than for them.

The four sources do not look, at first, like they are telling the same story. Disability, workplace inclusion, testing methodology, smallholder agriculture in the global South. They are. When a technology scales by averaging, the people at the edges of the average pay the cost, and those people are not a minor accounting category. They are the majority of humanity, unevenly distributed. Add non-English speakers, rural users, people with accents the model was never tuned on, and older adults, and the exclusion is not a rounding error. It is the shape of the market.

The question the Forbes piece really asked, and that the UK convening declined to answer directly, is who currently has the power and the incentive to change any of this. That question has an uncomfortable answer.

The default is the person the model imagined

To see why disability is a stress test rather than a special case, it helps to understand how the defaults were set. A large language model is a very expensive statistical summary of text its builders chose, or could afford, to scrape. A voice recognition system is a very expensive statistical summary of speech its builders recorded, or bought, or licensed. A vision model is a very expensive statistical summary of images its builders labelled, or hired people to label, or used labels some earlier researcher had published. At every step, the system's eventual performance on any given user is roughly a function of how well that user resembles the median entity in the training distribution.

The median entity in most training distributions is, to a close approximation, a neurotypical, non-disabled, native-English-speaking adult in North America or Western Europe, interacting via keyboard, mouse or clear spoken English in a quiet room. Meredith Ringel Morris, the former Microsoft researcher who founded the Ability Research Group and now leads HCI research at Google DeepMind, has spent more than a decade cataloguing how that default haunts the systems built around it. Her 2020 Communications of the ACM paper, co-authored with Shari Trewin of IBM, laid out a research roadmap other researchers have been filling in ever since: self-driving car pedestrian detection that does not register wheelchair users; hiring screeners that discount autistic candidates because their facial expressions score as inauthentic; voice assistants that cannot parse speech disabilities; captioning systems that fail on British Sign Language because the model only trained on spoken audio.

Waters' Frontiers paper takes this catalogue as its starting point and asks what evaluation would have to look like to catch these failures before deployment rather than after. Her answer runs to seven specialised metrics, among them an Inclusive Accuracy Rate that requires systems to report performance broken down by user characteristics, an Accessibility Disparity Index that quantifies the gap between best and worst performing groups, an Assistive Technology Compatibility Score, a Cognitive Load Index for neurodivergent users, an Error Recovery Rate, and several others. None of them is exotic. What is exotic, at least in the dominant benchmarking culture, is the assumption that they should be reported at all.

At present, when a model card for a frontier system is published, it will typically include accuracy on standardised benchmarks, sometimes broken down by language or domain, very rarely broken down by user characteristics, and almost never broken down by disability. The ImageNet paper that became the benchmark for vision research did not report performance by subject characteristics. The GLUE paper that became the benchmark for natural-language understanding did not either. Partly this is because the benchmarks did not have that data. Partly it is because the benchmarks assumed the user did not matter. Either way, by the time a commercial system is built on these foundations, the places where performance falls apart have already been defined out of the evaluation.

The Forbes stress-test framing is, in that sense, a good one. A stress test does not tell you anything the system could not, in principle, have told you about itself. It just forces the information into the open. Run a text-to-image system on prompts including disability terms, as Ashley Shew of Virginia Tech and others have done, and you get outputs in which wheelchairs are rusting wrecks, guide dogs are ominous shadows, and prosthetic limbs are rendered as torture devices. Run a sentiment classifier on sentences containing “disability” and the valence collapses, sometimes by several standard deviations. Run a caption model on an image of a person signing and it will tell you about the person's clothes. None of this is a failure mode anyone intended. All of it is a failure mode anyone with a bias audit could have found. The absence of the audit is the choice.

Why employment is the leverage point

The UK government's convening on 3 February was, in one sense, about a much older problem than AI. The disability employment gap in Britain has hovered around 28 percentage points for a decade, and Scope's chief executive Mark Hodgkinson used his remarks at the roundtable to point out that roughly a million disabled people who want to work are currently out of the labour market, many of them not through any lack of capability but because the workplaces and the tools they are asked to work with do not accommodate them.

The reason the meeting mattered is that workplace AI is, right now, the sharp end of a different stress test. The workplace is where biased hiring algorithms live. It is where productivity monitoring systems penalise users whose working patterns deviate from the average. It is where speech recognition is deployed to take meeting minutes that then become performance evidence. It is where agentic tools are being rolled out as personal assistants that assume their user's schedule, preferences and interaction style map neatly onto the same neurotypical default. For a disabled worker, a badly designed AI system is not merely an inconvenience. It is, increasingly, a gatekeeper standing between them and the capacity to work at all.

Maxine Williams, vice president of accessibility at Meta, used the roundtable to announce that the company's AI-powered wearables had added real-time environmental description for blind and low-vision users. This is genuinely useful. It is also the kind of announcement that quietly obscures a harder question. The same company's advertising-targeting algorithms, the same platform's automated content moderation, the same recommender systems that decide what a disabled creator gets shown to, are built on training pipelines that have no comparable accessibility evaluation. A pair of smart glasses that describes a room to a blind user is a product feature. A newsfeed ranking system that deprioritises disability-related content because it scores as low-sentiment is an infrastructural choice. The first is visible, marketable and reputationally valuable. The second is invisible, cheap to leave alone, and reputationally expensive only if someone does the audit.

The same asymmetry runs through the Microsoft, Google and Amazon presentations at the DWP meeting. Amazon's Jaqui Sampson spoke about the company's neurodiversity hiring programme, which has placed several hundred people on the autism spectrum into warehouse and technical roles. Google's team highlighted Project Relate, the speech-recognition model fine-tuned on non-standard speech that grew out of its Project Euphonia research. Microsoft talked about the Seeing AI app and the Immersive Reader. Every one of these is, in isolation, a real contribution. Every one of them also sits alongside the same companies' core AI infrastructure, which is not accessibility-evaluated and is, in most cases, the foundation on which third parties are building the workplace tools disabled workers have to use. Assistive features at the edge, inaccessible scaffolding at the core. That is the pattern.

Guide Dogs UK's Alex Pepper said the quiet part out loud. Assistive technology, she told the room, can remove barriers at work, but it is not a solution on its own. Translation: the industry is very good at giving disability advocates product demos and very bad at changing the way the substrate systems are trained, evaluated and governed. A pair of AI glasses does not fix a hiring pipeline whose screening tool discards autistic applicants at the CV stage. A captioning feature does not fix a workplace analytics system that penalises a deaf employee for the below-average meeting participation metrics the same captioning layer enabled.

The colonial analogy is not a metaphor

One of the more striking aspects of reading the Forbes argument alongside the Rest of World investigation is how structurally similar the failure modes are. Chandran's reporting from March 2026 does not use the word “disability” once. It does not need to. The mechanism it describes, a Western model trained on Western data fails to recognise the objects, practices and languages of people outside the training distribution, is the same mechanism Morris and Waters and every disability AI researcher has been describing for years.

Nakalembe's 5 million-image dataset of smallholder crops in East Africa exists because the existing computer vision literature on agriculture had been trained almost entirely on North American and European industrial farms. Maize, in the Midwest sense, looks one way; maize, in the Rwandan sense, looks another. A segmentation model that was never shown the second cannot see it. Dhar's forest-monitoring tool that missed half the trees in Maharashtra was not malfunctioning by some local standard. It was doing exactly what it had been trained to do. It had simply never been trained on the relevant world.

Digital Green's FarmerChat, which now reaches a million farmers across South Asia and Africa, is instructive because of the corrective it represents. It works by doing what Waters' framework would require: sourcing the training data from the users who will actually use the tool, evaluating the model against their real queries rather than against a benchmark designed somewhere else, and building with the users rather than for them. In other words, it drops the default. It treats the smallholder farmer in vernacular Telugu or Swahili as the person the model is for, rather than as a tolerated edge case.

What the agricultural story shares with the disability story is the structural dynamic. In both, the dominant models were trained by institutions whose own users were implicitly treated as the species for which the tool was being built. In both, the populations outside that implicit species had to assemble their own data, build their own pipelines and argue, repeatedly, for the legitimacy of their inclusion. In both, the industry's preferred response has been to offer add-ons, localisation layers, assistive features bolted onto a substrate that remains unchanged. In both, the add-on strategy is cheaper than the redesign. In neither does the add-on strategy actually solve the underlying problem, which is that the training substrate itself encoded a worldview.

The reason this matters for the power question is that it connects two movements that have mostly operated in isolation. The disability AI community, led by researchers like Morris, Shew, Jutta Treviranus at OCAD University in Toronto, and Catherine Holloway at the Global Disability Innovation Hub at UCL, has been asking for inclusive training data and participatory evaluation for the better part of a decade. The global-south AI community, represented by researchers like Nakalembe, Timnit Gebru at the Distributed AI Research Institute, and the authors of the 2021 Stochastic Parrots paper, has been asking for the same thing from a different direction. When those two arguments are recognised as the same argument, the constituency pushing for structural change becomes considerably larger than either community looks on its own.

Who currently has the power

The honest answer to the question of who has the power to deliver genuine inclusion, and who has the incentive, does not flatter the industry.

The people who have the most power to change how models are trained, evaluated and deployed are the half-dozen companies whose frontier systems underpin most of the AI economy: OpenAI, Anthropic, Google DeepMind, Meta, Microsoft and, at the infrastructure layer, Nvidia. These are the entities whose training data choices, whose evaluation benchmarks, whose model cards and whose API behaviours determine what everyone downstream inherits. If any one of them chose, tomorrow, to publish accessibility-disaggregated performance data on its next frontier release, or to make Waters' Inclusive Accuracy Rate a standard reporting field, the rest of the industry would have to follow, because the procurement contracts and the regulatory filings and the research community would start asking for it. None of them currently does.

They have the power. They do not have the incentive, or at least not a large enough one to outweigh the cost. The cost, which is substantial, is what the disability community calls data work: assembling inclusive datasets, running participatory design processes, building evaluation suites that require human effort rather than automatable benchmarks, and, most awkwardly, publishing disparity metrics that will make the model look worse on average. The revenue upside of any of this for the model provider is real but small compared to the upside of the next efficiency frontier or the next reasoning benchmark. The downside, legally and reputationally, has so far been manageable.

The people who have the incentive but not the power are disabled users, disability-led organisations, and the researchers working alongside them. Scope, Disability Rights UK, RNID and the others who sat across the DWP table from the tech companies in February are unambiguously motivated to deliver genuine inclusion. They do not control the training pipelines. They do not sit on the model cards. They can, at best, act as a feedback channel, and only if the companies choose to listen. Catherine Holloway's Global Disability Innovation Hub and its Centre for Digital Language Inclusion, launched in partnership with the Royal Academy of Engineering and the University of Ghana in 2025, is building non-standard speech datasets precisely because no frontier lab has produced one at the scale required. That work is essential, and it is happening largely on philanthropic and academic funding while the companies that could resource it two orders of magnitude better continue not to.

The people who have some of both are the regulators, and regulators are the reason any of this is shifting at all. The European Accessibility Act, which came into force on 28 June 2025, requires a wide range of consumer-facing products and services to meet accessibility standards, including, increasingly, AI-enabled ones. The UK's Equality Act has always prohibited discrimination in the provision of goods and services, and the Equality and Human Rights Commission has been explicit that algorithmic discrimination is within its remit. The EU AI Act's high-risk categorisation for employment-related AI systems carries obligations that include bias mitigation and fundamental-rights impact assessments, and disability discrimination is among the rights it protects. The US Section 508 refresh, updated through 2024 and 2025, now covers procurement rules for federal AI systems. The 24 April 2026 deadline for US state and local government agencies to comply with Title II of the Americans with Disabilities Act at WCAG 2.1 Level AA is a week away from the moment this piece is being written, and will push a large volume of procurement activity toward accessibility-audited vendors.

The regulatory pressure is building, slowly, against the incentive gradient of the industry. Whether it builds fast enough to shift the default before the current generation of AI systems has ossified into infrastructure is the open question.

What genuine inclusion would require

Set aside the rhetoric of empowerment and the photograph-friendly accessibility features, and the substance of what inclusion requires is reasonably well understood. The academic and advocacy literature converges on roughly five things, and the striking feature of the convergence is that none of the five is technically hard. They are organisationally and commercially unwelcome.

The first is inclusive training data, collected through participatory processes with the populations the system is expected to serve. Digital Green's 120,000-query corpus of farmer questions in 16 languages is the template for what that looks like in agriculture. The Speech Accessibility Project at the University of Illinois, which has been collecting non-standard speech recordings from people with Parkinson's, cerebral palsy, ALS and Down syndrome since 2022, is the template for what it looks like in voice. The 5 million-image East African crop dataset Nakalembe assembled is the template for what it looks like in vision. None of these was built by the frontier labs. All of them should be.

The second is disaggregated evaluation, reported publicly, with performance broken down by user characteristics and context. Waters' seven metrics are a starting point. Morris' research roadmap offers another. At minimum, model cards for any deployed AI system should report performance on named disability-relevant evaluations, should state the demographic composition of the evaluation set, and should be updated when the underlying model is updated. At present, they typically do none of this.

The third is participatory design, built into the product cycle rather than retrofitted afterwards. Jutta Treviranus' Inclusive Design Research Centre has a motto, “design with not for”, that has circulated in the accessibility community for twenty years. In AI, it is still largely aspirational. The UK's Atech Policy Lab, which sat at the DWP roundtable, is one of the few bodies attempting to hard-code participatory practice into the AI assistive-technology pipeline. Most of the major labs' partnership structures with disability organisations remain advisory rather than decisional.

The fourth is interoperability with assistive technologies. The systems that disabled users already rely on, screen readers, switch controls, eye-gaze interfaces, alternative input devices, bespoke communication aids, need to be first-class citizens in the AI interaction model, not afterthoughts. This is a straightforwardly technical requirement, and it is the one area where mandatory standards are starting to bite. The W3C's WAI-AI task force, formed in 2024 and producing guidance through 2025 and 2026, is doing the unglamorous work of defining what accessible AI means in a way developers can actually implement.

The fifth is accountability when systems fail, meaning both the right of redress for the user harmed and the obligation of the provider to fix the underlying issue rather than adding yet another accessibility plug-in. This is where regulation becomes indispensable. Voluntary commitments have been on offer from the industry for a decade. The pattern the DWP roundtable demonstrated, tech companies promising product features while their core training pipelines remain unaudited, is what voluntary commitments reliably produce. Mandatory disclosure, backed by enforcement, is what changes the substrate.

Each of these five can be characterised as a cost centre, and each, properly executed, is a correction to a market failure currently paid for by disabled users, global-south users, and, eventually, the employers and public services on the receiving end of the systems' underperformance. A stress test does not create the fragility. It exposes it. The fragility was always there.

The architecture of the next decade

There is a version of the next few years in which the story gets worse before it gets better. More AI systems will be deployed into more workplaces, more public services, more clinical and educational and legal settings, before the regulatory substrate and the evaluation methodology have caught up. Disabled users, like global-south users, will continue to bear the cost of the gap between the technology's performance on the median user and its performance on them. The assistive add-on will continue to be the preferred industry response. The companies that could do the deeper work will continue to choose not to, because the incentive structure has not yet reversed.

There is also a version in which a combination of regulatory enforcement, procurement leverage from large public buyers, and sustained pressure from the research community and disability-led organisations starts to shift the defaults. The EU AI Act's first full enforcement cycle begins to bite through 2026 and 2027. The US ADA Title II deadline produces a wave of vendor audits. The DWP-style convenings, if they become regular rather than ceremonial, put the same five questions to the same companies repeatedly until the answers change. The Frontiers framework gets picked up by NIST, by the Alan Turing Institute's AI Safety Institute, by standards bodies that have the authority to make accessibility evaluation a default reporting field. The 5 million-image datasets and the 120,000-query corpora stop being heroic one-off efforts and start being line items in platform R&D budgets.

Which of these futures arrives depends on something more prosaic than a technical breakthrough. It depends on whether the power and the incentive can be brought into alignment, and whether the regulatory architecture is built fast enough to do the aligning. The power currently sits with the labs. The incentive currently does not. The gap between the two is where the exclusion lives, and it will continue to live there until someone, regulator or buyer or coalition of both, moves it.

Return, finally, to the kitchen table in south London. The woman asking her smart speaker to call her sister is not, in the technology industry's current accounting, a failure case. She is not a test the product was designed to pass. She is, from the system's point of view, slightly outside the curve, which is another way of saying slightly outside the imagination of the people who built it. Her experience is what the Forbes analysis meant by a stress test. If she works, the system works. If she does not, the system is not ready. The point of treating her as the test case is that it produces a higher-quality system for everyone, because the default person the models were trained to serve is not, in fact, most of the people the models are being sold to.

Alexiou's insight, and the convergence with Waters' framework and Chandran's reporting and the UK government's cautious, cautiously serious convening, is that inclusion is not a niche product requirement. It is the infrastructure question the next decade of this technology turns on. Who has the power to deliver it is mostly the labs. Who has the incentive is mostly the users, the regulators and the advocates. Closing that gap is the work. Nobody is going to close it by accident, and nobody is going to close it because it would be the nice thing to do. It will close when the cost of not closing it is made legible, and that is a job for law, for procurement, for journalism and for the disability community itself, not for a press release about a new AI-powered wearable. The kitchen-table speaker will start hearing her properly on the day the company that makes it has to report, publicly and disaggregated, how often it does not. Until then, she will keep pitching her voice higher and the rest of the market will keep pretending the model understands natural language.

---

References & Sources

1. Alexiou, G. “Why Disability Is The Ultimate Stress Test For Artificial Intelligence.” Forbes, January 2026. https://www.forbes.com/sites/gusalexiou/
2. Department for Work and Pensions. “Tech giants meet disability sector to break down barriers at work.” GOV.UK, 3 February 2026. https://www.gov.uk/government/news/tech-giants-meet-disability-sector-to-break-down-barriers-at-work
3. Waters, G. “AI testing, evaluation, verification and validation for accessibility: a comprehensive framework.” Frontiers in Digital Health, Volume 7, 2025-2026. DOI: 10.3389/fdgth.2025.1679603. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1679603/full
4. Chandran, R. “AI models from Western tech giants fail in overseas agricultural settings.” Rest of World, 12 March 2026. https://restofworld.org/2026/ai-agriculture-local-data/
5. World Health Organization. “Disability.” WHO Fact Sheet, 2023 (updated 2024-2025). https://www.who.int/news-room/fact-sheets/detail/disability-and-health
6. Morris, M. R. “AI and Accessibility: A Discussion of Ethical Considerations.” Communications of the ACM, Vol. 63 No. 6, June 2020. https://cacm.acm.org/magazines/2020/6/245157-ai-and-accessibility/fulltext
7. Trewin, S., Basson, S., Muller, M., Branham, S., Treviranus, J., Gruen, D., Hebert, D., Lyckowski, N. and Manser, E. “Considerations for AI Fairness for People with Disabilities.” AI Matters, 5(3), 2019. https://sigai.acm.org/static/aimatters/5-3/AIMatters-5-3-05-Trewin.pdf
8. Microsoft Research. “AI Fairness and Disability.” Project page and publications. https://www.microsoft.com/en-us/research/project/ai-fairness-and-disability/publications/
9. Microsoft. “Shrinking the 'data desert': Inside efforts to make AI systems more inclusive of people with disabilities.” Microsoft Source. https://news.microsoft.com/source/features/diversity-inclusion/shrinking-the-data-desert/
10. Global Disability Innovation Hub. “Changing lives through AI: a new Centre for Digital Language Inclusion.” 2025. https://www.disabilityinnovation.com/news/cdli
11. Global Disability Innovation Hub. “Prof Catherine Holloway profile.” https://www.disabilityinnovation.com/who-we-are/our-team/dr-catherine-holloway
12. Speech Accessibility Project, University of Illinois Urbana-Champaign. https://speechaccessibilityproject.beckman.illinois.edu/
13. Nakalembe, C. et al. NASA Harvest and University of Maryland work on East African smallholder agriculture, as reported in Rest of World, 12 March 2026.
14. Scope UK. Statements by Mark Hodgkinson, Chief Executive, at DWP roundtable, 3 February 2026. https://www.scope.org.uk/
15. European Accessibility Act. Directive (EU) 2019/882, in force 28 June 2025. https://ec.europa.eu/social/main.jsp?catId=1202
16. European Union. “Artificial Intelligence Act.” Regulation (EU) 2024/1689. https://artificialintelligenceact.eu/
17. US Access Board. “Section 508 Standards.” https://www.access-board.gov/ict/
18. US Department of Justice. “ADA Title II Web Content and Mobile Apps Rule.” Final rule published April 2024, compliance dates 2026-2027. https://www.ada.gov/resources/2024-03-08-web-rule/
19. W3C. “Accessibility of AI Systems (AI-Accessibility).” WAI task force, 2024-2026. https://www.w3.org/WAI/
20. Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. “On the Dangers of Stochastic Parrots.” FAccT 2021. https://dl.acm.org/doi/10.1145/3442188.3445922
21. Shew, A. “Against Technoableism: Rethinking Who Needs Improvement.” W. W. Norton, 2023.
22. Treviranus, J. “The Three Dimensions of Inclusive Design.” Inclusive Design Research Centre, OCAD University. https://legacy.inclusivedesign.ca/
23. Muckrack. “Gus Alexiou journalist profile.” https://muckrack.com/gus-alexiou/articles
24. Equality and Human Rights Commission. “Artificial intelligence, machine learning and automated decision-making: our guidance.” 2024-2025. https://www.equalityhumanrights.com/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The first thing the sensor sees is the ceiling. It is an unremarkable ceiling, white acoustic tile, fluorescent strip, a slight nicotine tinge from a generation of residents who were once allowed to smoke indoors. The sensor is not a camera in the conventional sense. It does not record video; the procurement document made a point of that. It is a low-resolution thermal array, mounted in a discreet white housing about the size of a smoke alarm, and it watches the room beneath it as a heat map. When a heat-map blob detaches from the bed and crosses the floor, it logs movement. When the blob lies horizontal in a place a human body should not be horizontal, it pings a tablet at the nurses' station. The vendor calls this fall detection. The procurement notice called it dignity-preserving monitoring. The night shift on a typical residential aged care floor in Australia or England in early 2026, which is often one registered nurse and two personal care workers covering upwards of forty residents, calls it the thing that goes off.

What the thing goes off about, on the kinds of nights the Australian Royal Commission into Aged Care Quality and Safety documented across ninety-nine sitting days of evidence and that the Care Quality Commission in England continues to describe in its state-of-care reports, is the sort of incident that happens when an older resident with dementia transfers from bed, returns toward it, and falls. The sensor logs the transfer; it logs the horizontal heat signature on the floor; it pings the tablet. The personal care worker on duty may be two corridors away changing another resident. By the time anybody arrives, the resident has been on the carpet long enough for a hip to break. The sensor has done exactly what the brochure said it would do. Nobody has been close enough for the information to matter. That pattern, not any one incident, is what the evidence that regulators have taken in sworn testimony describes.

It is the gap between those two facts, the thing that the technology measured and the thing that the system did with the measurement, that a paper published in The Conversation on 24 February 2026 by Barbara Barbosa Neves of the University of Sydney, Alexandra Sanders, and Geoffrey Mead set out to dramatise. Their argument, distilled from an analysis of the marketing materials of thirty-three companies selling AI tools into aged care across Australia, East Asia, Europe and North America, is that the industry has succeeded in convincing governments and investors that algorithmic monitoring, automated care planning and companion robotics are the answer to a workforce crisis when they are, in fact, a way of avoiding the question. The crisis is structural. The tools, however clever, cannot be structural answers. “If we let AI companies define what is broken,” the authors write, “we also let them define what repair looks like. That may leave our systems more profitable, but far less caring and humane.”

The numbers behind the pitch are now large enough to set the rest of the policy debate around them. Fortune Business Insights estimated the global elderly care market at 53.29 billion US dollars in 2025 and projected it to reach 57.78 billion in 2026, on its way to roughly 114 billion by 2034. The agetech subsegment, the layer of digital and AI products sold into that market, is projected by industry analysts cited in the Neves paper to reach A$170 billion by 2030. By any reading, the next decade of aged care will be one of the most heavily capitalised periods in the sector's history, and a substantial fraction of that capital is going into systems that are designed to do things humans currently do.

The question this article is concerned with is not whether the technology works. Some of it does, in narrow ways, under controlled conditions. The question is what accountability structures would have to exist before deploying it at scale, into a population that cannot easily refuse it and cannot reliably tell anyone when it has failed, could be considered ethical. The honest answer, in April 2026, is that very few of those structures exist anywhere, and most of what passes for them is designed to manage the reputational risk of providers and vendors rather than the safety of residents.

What The Inquiries Already Told Us

The Royal Commission into Aged Care Quality and Safety, which delivered its 2,500-page final report, Care, Dignity and Respect, on 1 March 2021, did not lack for diagnoses. Across twenty-three public hearings, ninety-nine sitting days, 641 witnesses and more than ten thousand public submissions, commissioners Lynelle Briggs and Tony Pagone arrived at 148 recommendations. The findings were as plain as they were grim. Commissioner Briggs put the proportion of residents who had experienced physical or sexual assault at between thirteen and eighteen per cent. The report described two decades of underfunding amounting to approximately 9.8 billion Australian dollars cut from the sector's annual budget. It documented residents left in soiled continence aids, malnourished, restrained chemically and physically, and dying in conditions the Commission did not euphemise.

What the Commission did not say, in any of those pages, is that the answer to those failings lay in machine learning. The recommendations focused on staff ratios, on the qualifications and pay of personal care workers, on a new statutory framework for the rights of older people, on enforceable care standards, and on an independent regulator with real teeth. The Aged Care Act 2024, which came into effect on 1 November 2025 after a delay from its originally legislated 1 July date, codified some of that framework. From October 2024, providers had been required to deliver a national average of 215 minutes of personal and nursing care per resident per day, of which 44 minutes was to come from a registered nurse. From 1 October 2025, the Star Ratings used to grade residential providers were re-engineered to require those minutes for a three-star or better staffing rating. None of those reforms involved an algorithm.

The same pattern recurs in every comparable jurisdiction. The Care Quality Commission in England, which by the summer of 2024 was being publicly described by the Secretary of State for Health and Social Care, Wes Streeting, as a failing organisation, commissioned the Dash Review of its operational effectiveness; the full report, published in October 2024, found that the time taken by the regulator to re-inspect a service rated “requires improvement” had risen from 142 days in 2015 to 360 days in 2024. The CQC's chief executive, Ian Trenholm, resigned that July. Skills for Care reported that as of March 2025 there were 111,000 vacant posts in adult social care in England, a vacancy rate of 6.4 per cent against a labour-market average of 2.2, with care worker vacancies running at 8.3 per cent and homecare vacancies above 10 per cent. Annual turnover sat at thirty per cent. In May 2025 the UK government closed the international recruitment route for new care workers, cutting off a pipeline that had been delivering an average of twelve thousand recruits a quarter into the independent sector. None of those problems have algorithmic solutions.

In the United States, the federal minimum staffing standard for long-term care facilities published by the Centres for Medicare and Medicaid Services in May 2024, requiring 3.48 hours of nursing care per resident per day and twenty-four-hour onsite registered nurse coverage, was repealed in December 2025. Section 71111 of Public Law 119-21 then prohibited CMS from implementing or enforcing the rule until at least 30 September 2034. Public Citizen and the Center for Medicare Advocacy estimated that the original rule, had it survived, would have prevented approximately thirteen thousand deaths a year. In Canada, the May 2020 Canadian Armed Forces report on five Ontario long-term care homes, which described cockroaches, rotting food, ulcerated bed-bound residents and staff cycling between units in contaminated personal protective equipment, prompted no national workforce reform of any depth; provincial inquiries in Ontario and Quebec produced more recommendations than implementations. The same picture, with local variations, holds in the Nordic countries, in France and in much of east Asia.

What the inquiries documented, in other words, was not a sector that had failed to adopt the latest technology. It was a sector that had failed to be funded, staffed, regulated and respected. The premise of the agetech pitch, that AI can plug the gap, is in this light a category error. There is no reasonable reading of Care, Dignity and Respect in which the missing ingredient is more sensors.

The Pitch And The Products

Walk the floor of any of the recent agetech expos, the SilverEco Forum in Cannes, the Aged Care 2026 conference in Melbourne, the Health 2.0 trade fair in Tokyo, and the categories repeat. There are passive monitoring systems, of which the thermal sensor in the opening scene is one example. There are wearable fall detectors that combine accelerometers and machine-learned gait classifiers, sold by firms like Vayyar, Kepler Vision and a long tail of European start-ups. There are continuous bed and chair sensors, marketed under names like SafelyYou and Tellus You Care. There are automated care-planning platforms that ingest electronic health records and generate suggested daily routines, hydration prompts and bowel charts. There are medication management dispensers. There are predictive analytics layers that promise to flag clinical deterioration days before it shows up in vital signs. There are companion robots: PARO, the harp-seal-shaped therapeutic robot developed by Takanori Shibata at Japan's National Institute of Advanced Industrial Science and Technology, in clinical use since the mid-2000s; ElliQ, the desktop social companion built by Intuition Robotics in Israel; SoftBank's humanoid Pepper, repurposed from retail receptionist into care-floor entertainer; and the various lower-cost robotic-cat and robotic-dog products that proliferate at the budget end.

The evidence base for these products is uneven and almost always thinner than the marketing implies. A 2021 systematic review and meta-analysis published in Innovation in Aging by Hung and colleagues, “Effectiveness of Companion Robot Care for Dementia”, found that PARO produced statistically meaningful but small improvements in agitation, depression and medication use across pooled trials, with the authors noting that most studies were small, short and unblinded. A 2023 systematic review in the International Journal of Nursing Studies reached a similar conclusion: a possible benefit, evidence quality low to moderate, no demonstrated long-term effect. A pilot randomised controlled trial of a different companion robot for community-dwelling people with dementia, published in 2017 by Moyle and colleagues, recorded engagement with the device but did not show robust effects on the primary outcomes.

ElliQ has produced more uplifting headline numbers, largely from one programme. The New York State Office for the Aging began deploying ElliQ in 2022; by May 2025 the agency reported 834 active clients, with 94 per cent saying they felt less lonely, 97 per cent feeling better overall, average usage of forty-one interactions per day, and a customer satisfaction score of 4.6 out of 5. Those are the figures Intuition Robotics quotes in its marketing decks. The peer-reviewed literature is more cautious. A 2024 review by Broxvall and colleagues, “ElliQ, an AI-Driven Social Robot to Alleviate Loneliness: Progress and Lessons Learned”, described the deployment as “promising” and explicitly called for randomised controlled trials before efficacy claims could be considered established. The NYSOFA outcomes, reassuring as they are, were collected from a self-selected user base that consented, that engaged voluntarily, and that retained the cognitive bandwidth to fill in a satisfaction survey. They tell us very little about what would happen if the same device were deployed by default to a less able population.

Fall detection is the category in which the gap between vendor claim and operational reality is widest. A 2025 scoping review in Applied Sciences, “AI-Driven Inpatient Fall Prevention Using Continuous Monitoring”, examined the evidence on continuous monitoring systems in hospital and long-term care settings and reached a conclusion that vendors do not put on their websites: while sensitivity for detecting falls can exceed ninety per cent, false-positive rates of thirty to forty per cent are common, and across the evidence base detection systems “did not consistently reduce fall incidence or the occurrence of injurious falls”. The same paper, like a closely related 2025 review in Medicina, found that reporting of “implementation-critical metrics” such as alert burden, response times and downstream actions was patchy. Studies of clinical alarm fatigue across acute care have repeatedly found that as many as eighty to ninety per cent of audible alarms in monitored wards are non-actionable. There is no plausible mechanism by which adding more alarms to an understaffed care floor improves outcomes, and reasonable mechanisms by which it makes them worse.

Predictive analytics for clinical deterioration carry a related set of problems. Algorithms trained on the electronic health records of one population have been shown repeatedly, including in a much-cited 2023 JAMA Internal Medicine analysis of the Epic sepsis prediction model, to perform worse than advertised when deployed in different populations. Aged care residents are an unusually heterogeneous group, often with multimorbidity, polypharmacy and cognitive complications that distort the signals the model was trained to detect. The risk is not that the model produces nothing useful; it is that it produces enough useful output to displace clinical judgement while the genuinely unusual cases, the ones a human carer would recognise on sight, slip past unflagged.

The Advocacy Gap

Across all of these tools, the same population variable does most of the moral work. The people on whom the sensors and dispensers and screens are aimed are, by definition of the sector they are in, frail. A substantial proportion are cognitively impaired; the Australian Institute of Health and Welfare estimated in its 2024 dementia report that more than half of permanent residents in Australian aged care had a diagnosis of dementia. Many are socially isolated; the loneliness data that companion robots cite as a justification is real. Many have limited or no digital fluency; older Australians in residential care are dramatically under-represented in surveys of internet use, smartphone ownership and the everyday literacy that allows a person to interrogate, refuse or modify a digital tool. And almost all of them sit in a profound power asymmetry with the people on whom they depend for daily care.

The implications for consent are not theoretical. The standard model of informed consent in healthcare assumes a person capable of understanding the nature of the proposed intervention, weighing it against alternatives, and communicating a decision. A 2025 review in Frontiers in Digital Health, “Designing for Dignity: Ethics of AI Surveillance in Older Adult Care”, catalogued how badly that model breaks down in practice when the intervention is a continuous, ambient monitoring system and the person being monitored has fluctuating capacity. Many older adults in care settings, the authors noted, have “no knowledge about what data is being harvested” and lack the cognitive or technical capability to adjust settings. Consent is typically obtained at admission, signed by a family member acting as substitute decision-maker, and never revisited. The system that the resident did not knowingly agree to becomes the system they live inside.

The asymmetry is sharper still where AI is making, or shaping, allocative decisions. Australia's new Support at Home programme, introduced in November 2025 to replace earlier home-care packages, uses a rules-based algorithm called the Integrated Assessment Tool to convert assessor responses into funding entitlements. As reported by The Conversation in March 2026 in a follow-up piece by Sebastian Cordoba and colleagues titled “First Robodebt, now NDIS and aged care: how computers still decide who gets care”, neither assessors nor participants can clearly see how the algorithm converts answers into funding levels. Departmental officials told a Senate inquiry that there is “no discretionary element” in the process; an override function present during testing was removed before the system went live. Evidence presented to the inquiry suggested the tool was systematically underestimating need, with reports of older Australians, including those with serious or degenerative conditions, having their support reduced. The Robodebt scandal, in which an automated debt-recovery system run by Services Australia issued more than 470,000 unlawful debt notices between 2016 and 2019 and was the subject of a 2023 Royal Commission, is the cautionary tale every Australian policy commentator now invokes. The aged care sector's algorithmic infrastructure is being built by a state apparatus that demonstrably has not learned its lesson.

The classic argument for surveillance and substitution technologies in care is that the people receiving them benefit, and that any inconvenience to autonomy is outweighed by safety. The problem with this argument is that it cannot be tested by the people on whom it is being made. A resident with moderate dementia cannot reliably explain to an inspector why the sensor in the corner of her room makes her feel watched, or whether she would prefer a human attendant to a tablet that pings someone who arrives nine minutes later. A non-verbal resident with advanced cognitive impairment cannot tell a researcher whether the companion robot is comforting her or merely keeping her quiet. The marketing literature sometimes claims that residents prefer the robots; the more careful research, including work by Neves and her collaborators in the Journal of Applied Gerontology in 2023, “Artificial Intelligence in Long-Term Care: Technological Promise, Aging Anxieties, and Sociotechnical Ageism”, finds that older adults' attitudes towards AI in their own care are considerably more ambivalent than the agetech sector implies, that they are acutely aware of being positioned as objects of management rather than subjects of care, and that they often experience monitoring as a loss of dignity rather than a gain in safety.

Cost Reduction Or Outcome Improvement And Who Carries The Risk

The business case for AI in aged care, in board meetings rather than press releases, is largely about labour. A monitoring system that allows a single night-shift carer to cover sixty residents instead of forty is, on paper, a workforce multiplier. A medication dispenser that prompts a resident through a regimen reduces the registered nursing time required for medication rounds. An automated care plan reduces the documentation burden on personal care workers. A companion robot, if it can hold attention, reduces the demand on staff for the relational work that has historically been the floor of dignified care. Each of these is a legitimate engineering goal in a sector where workforce shortage is real, severe and not going away. None of them is the same thing as improved outcomes for residents.

The distinction matters, because the risk of miscalibration falls asymmetrically. If a fall sensor's false-positive rate produces alarm fatigue and a real fall is missed, the cost is borne by the resident on the floor, not the procurement team that signed the contract. If a predictive deterioration model misses an unusual sepsis presentation in a resident with atypical baseline observations, the resident dies. If an automated care plan recommends a hydration schedule calibrated to a baseline weight two years out of date, the resident whose actual weight has dropped sharply goes thirsty. If a companion robot becomes the dominant social contact for a resident whose family visits have tapered, the human relationships that aged-care research consistently identifies as protective against decline are the ones that quietly disappear.

This asymmetry is what makes the cost-reduction framing dangerous. In a properly functioning market, the people who bear the risk of a product underperforming push back. In aged care, the people who bear the risk are very often unable to. The carers who notice that the system is not working, who see a resident on the floor long after the sensor said so, are positioned several layers below the procurement decisions that put the system there. They have, as the Neves paper notes, taken on additional cognitive labour interpreting the data the system generates, but they have lost discretion over whether the system should be used at all. The families who pay the bills are typically not on the floor when the system fails. The regulators who would, in theory, audit whether the technology was performing as advertised lack the technical capability and, increasingly, the inspection cadence.

A 2024 paper in Humanities and Social Sciences Communications titled “Paternalistic AI: the case of aged care” framed the underlying ethical structure crisply. AI systems in care, the authors argued, function as a particularly powerful form of soft paternalism. They purport to act in the interests of the person being cared for, but they remove from that person the practical opportunity to refuse, modify or contest the intervention. In the context of cognitive impairment, where soft paternalism shades into hard paternalism almost imperceptibly, the absence of accountability structures around the technology means that the ethical work that would normally be done by consent simply does not happen.

What Accountability Would Actually Look Like

If AI is going to be deployed at scale in aged care, the question is what would have to be in place before such deployment could be considered ethical. The honest answer is a layered structure, none of whose layers currently exist in anything like a complete form in any major jurisdiction.

The first layer is consent that is genuine, ongoing and revocable. Admission paperwork signed by a substitute decision-maker is not consent to a continuous monitoring regime. A robust framework would require that residents, where they have any capacity, are walked through what each technology in their environment does, in plain language, with the right to refuse specific elements without losing access to other care. Where capacity is absent, substitute decision-makers should be required to revisit consent on a defined cadence, and to weigh the technology's use against alternatives that include increased staffing. The recommendation, drawn from the 2025 Frontiers in Digital Health paper, of “easy-to-visualize dashboards and plain-language explanations” should be a procurement requirement, not a research aspiration.

The second layer is independent auditing, with statutory backing, of the actual performance of deployed systems in their actual settings. Vendor-supplied performance figures are, as the scoping reviews on fall detection make clear, systematically optimistic. An accountability regime worth the name would require providers to log false positives, false negatives, response times and downstream actions in a standardised format, and would require regulators, not vendors, to publish the resulting performance data. Australia's Aged Care Quality and Safety Commission, the CQC in England, CMS in the United States and their equivalents would need substantial additional resourcing and technical capability to conduct such audits credibly. None has it now.

The third layer is algorithmic transparency. Where an AI tool affects the allocation of care, including hours of staffing, level of monitoring, eligibility for funding or assignment to a particular care pathway, residents and their advocates should have a legal right to an explanation of how the system reached its conclusion, expressed in terms an ordinary person can understand. Article 22 of the General Data Protection Regulation in the European Union already prohibits decisions based solely on automated processing that produce significant legal or comparable effects. That principle needs to be operationalised specifically for aged care, with explicit recognition that algorithmic recommendations that substantially shape human decisions count, and that the convenient fiction of “human in the loop” cannot be used to launder automation.

The fourth layer is incident reporting. When an AI tool contributes to harm, whether by missing a fall, misallocating medication, displacing human contact or generating an unsafe care recommendation, the incident should be reportable, on the same statutory footing as a medication error, to the relevant regulator, with public aggregate reporting. The current regime, in which AI-related incidents are typically classified as either workflow events or clinical errors and never as software failures, makes systemic learning impossible.

The fifth layer is a hard ban on substitution where it matters most. The question of whether a companion robot should ever be the primary social contact for a person with dementia is not a question for procurement officers. The position taken by Sherry Turkle of MIT in her 2011 book Alone Together, and elaborated in subsequent work, is that the deployment of robots as substitutes for, rather than supplements to, human relational care is an abdication. That position should be encoded in regulation. Companion robots may have a role; they may not have a role that displaces the requirement for staffed human contact. Procurement should require evidence that a tool augments rather than replaces the relational work, and operational data should be auditable to confirm that what was contracted as augmentation has not, over time, drifted into substitution.

The sixth layer is procurement conditionality, and it is the lever that actually moves the others. Public funders of aged care, which in most jurisdictions means the state, have far more bargaining power than they currently use. Every procurement contract for an AI system in publicly funded aged care should carry conditions on consent processes, audit access, transparency, incident reporting, anti-substitution and a ceiling on the proportion of care time that may be displaced by the system. Vendors that decline to meet those conditions should not be funded. The market will adjust quickly when it has to.

The seventh layer is the one that the agetech sector finds least convenient to discuss. None of the above is a substitute for adequate staffing. Every accountability regime for AI in aged care has to be built on top of, not in place of, the staffing standards, pay levels and workforce protections that the Royal Commission, the Dash Review, the CMS rule and the Canadian Armed Forces report were calling for. AI deployed into an under-staffed environment cannot be made ethical by audits alone. The ethical baseline is a staffed floor.

A Reported Ending

It is tempting, when writing about technology and vulnerability, to land on a hopeful note. The honest reading of the evidence in April 2026 does not really support one. The Aged Care Act 2024 in Australia is in early implementation; the staffing minutes are being met on national average but missed in many individual facilities. The CQC in England is mid-restructure following the Dash operational review. The federal staffing rule in the United States has been repealed and is statutorily prohibited from re-implementation until at least 2034. The Canadian provinces have made limited structural progress since 2020. The agetech market continues to grow. The companies whose pitches Neves, Sanders and Mead analysed are not slowing down their fundraising rounds because the academic literature is cautious about their effect sizes.

What the Conversation article points at, and what the evidence on every category of agetech tool quietly confirms, is that the question of whether AI in aged care is ethical cannot be answered at the level of the individual product. PARO has uses. ElliQ helps some lonely people in Buffalo and Albany. A well-calibrated fall sensor, in a building with enough carers to respond inside three minutes, may well be a net good. None of those local truths bears on the systemic question, which is whether the deployment, in aggregate, is being driven by considerations that the people on whom it is deployed would endorse if they could.

The resident whose hip breaks while the sensor pings an empty corridor does not appear in any vendor case study. Her mobility does not fully return, in the way ninety-year-old mobility rarely does. The room in which she fell still has a sensor on the ceiling, and the sensor still pings when it sees a heat-map blob in the wrong place. The night shift on her floor is still, in April 2026, one registered nurse and two personal care workers covering upwards of forty residents, the kind of configuration that the inspectorate reports from three continents have documented as standard. The vendor's quarterly filings continue to note strong growth in the Asia-Pacific market and new partnerships with major residential care operators. None of those facts, on their own, is scandalous. Together they describe the architecture of a sector that has decided, without ever quite deciding, that the cheaper option is also the wiser one.

The accountability structures that would make AI in aged care ethical are not technically difficult. They are politically expensive. They require regulators to be staffed and funded to a level that no government has yet been willing to fund them to. They require public procurement to drive standards in a market where vendors have grown accustomed to selling unvalidated tools into desperate buyers. They require a public conversation about the proper role of human contact in care that the sector and the technology industry have, between them, been content to defer.

Until those structures exist, the most defensible position is the one Neves and her colleagues argue for: that AI in aged care, deployed primarily to manage the consequences of under-investment in human care, is not a solution to the crisis the Royal Commission documented. It is a way of making the crisis less visible. The sensor sees the ceiling. The ceiling is white. The blob on the floor is logged at a particular minute, and again two minutes later. Somewhere down the corridor, somebody is doing the work that the technology was sold as a substitute for, and somebody else is doing without the work because there was nobody to do it for them. The accounting we owe the residents is the one we have, so far, declined to do.

References & Further Information

1. Neves, B. B., Sanders, A., & Mead, G. “AI companies promise to 'fix' aged care, but they're selling a false narrative.” The Conversation, 24 February 2026. https://theconversation.com/ai-companies-promise-to-fix-aged-care-but-theyre-selling-a-false-narrative-275822
2. Royal Commission into Aged Care Quality and Safety. Final Report: Care, Dignity and Respect. Commonwealth of Australia, 1 March 2021. https://www.royalcommission.gov.au/aged-care/final-report
3. Aged Care Act 2024 (Cth), Federal Register of Legislation, Australia. Commenced 1 November 2025. https://www.legislation.gov.au/C2024A00104/latest
4. Australian Government. “Changes to Staffing rating for Star Ratings.” My Aged Care, October 2025. https://www.myagedcare.gov.au/news-and-updates/changes-staffing-rating-star-ratings
5. Fortune Business Insights. “Elderly Care Market Size, Share & Trends Analysis Report, 2026-2034.” https://www.fortunebusinessinsights.com/elderly-care-market-111477
6. Dash, P. “Review into the Operational Effectiveness of the Care Quality Commission: Full Report.” UK Department of Health and Social Care, October 2024. https://www.gov.uk/government/publications/review-into-the-operational-effectiveness-of-the-care-quality-commission-full-report
7. Skills for Care. “The State of the Adult Social Care Sector and Workforce in England 2025.” https://www.skillsforcare.org.uk/Adult-Social-Care-Workforce-Data/workforceintelligence/resources/Reports/National/The-state-of-the-adult-social-care-sector-and-workforce-in-England-2025-Executive-Summary.pdf
8. Centres for Medicare and Medicaid Services. “Medicare and Medicaid Programs; Minimum Staffing Standards for Long-Term Care Facilities.” Federal Register, 10 May 2024. https://www.federalregister.gov/documents/2024/05/10/2024-08273/medicare-and-medicaid-programs-minimum-staffing-standards-for-long-term-care-facilities
9. Centres for Medicare and Medicaid Services. “Repeal of Minimum Staffing Standards for Long-Term Care Facilities.” Federal Register, 3 December 2025. https://www.federalregister.gov/documents/2025/12/03/2025-21792/medicare-and-medicaid-programs-repeal-of-minimum-staffing-standards-for-long-term-care-facilities
10. Canadian Armed Forces. “Op LASER: JTFC Observations in Long Term Care Facilities in Ontario.” Public Safety Canada, May 2020. https://www.publicsafety.gc.ca/cnt/trnsprnc/brfng-mtrls/prlmntry-bndrs/20200831/069/index-en.aspx
11. Hung, L., et al. “Effectiveness of Companion Robot Care for Dementia: A Systematic Review and Meta-Analysis.” Innovation in Aging, 5(2), 2021. https://academic.oup.com/innovateage/article/5/2/igab013/6249558
12. Yu, C., et al. “The effectiveness of a therapeutic robot, 'Paro', on behavioural and psychological symptoms, medication use, total sleep time and sociability in older adults with dementia: A systematic review and meta-analysis.” International Journal of Nursing Studies, 2023. https://www.sciencedirect.com/science/article/abs/pii/S0020748923000950
13. Broxvall, M., et al. “ElliQ, an AI-Driven Social Robot to Alleviate Loneliness: Progress and Lessons Learned.” 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC10917141/
14. New York State Office for the Aging. “ElliQ Proactive Care Companion Initiative: Project Update 2026.” February 2026. https://aging.ny.gov/system/files/documents/2026/02/nysofa-elliq-project-update-2026.pdf
15. “AI-Driven Inpatient Fall Prevention Using Continuous Monitoring: From Early Detection to Workflow-Integrated Decision Support: A Scoping Review.” Applied Sciences, MDPI, 2025. https://www.mdpi.com/2076-3417/16/7/3383
16. “Digital Healthcare Approaches for Fall Detection and Prediction in Older Adults: A Systematic Review of Evidence from Hospital and Long-Term Care Settings.” Medicina, MDPI, 2025. https://www.mdpi.com/1648-9144/61/11/1926
17. Wong, A., et al. “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.” JAMA Internal Medicine, 2023.
18. Neves, B. B., Petersen, A., Vered, M., Carter, A., & Omori, M. “Artificial Intelligence in Long-Term Care: Technological Promise, Aging Anxieties, and Sociotechnical Ageism.” Journal of Applied Gerontology, 2023. https://journals.sagepub.com/doi/10.1177/07334648231157370
19. “Designing for dignity: ethics of AI surveillance in older adult care.” Frontiers in Digital Health, 2025. https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2025.1643238/full
20. “Paternalistic AI: the case of aged care.” Humanities and Social Sciences Communications, 2024. https://www.nature.com/articles/s41599-024-03282-0
21. Cordoba, S., et al. “First Robodebt, now NDIS and aged care: how computers still decide who gets care.” The Conversation, March 2026. https://theconversation.com/first-robodebt-now-ndis-and-aged-care-how-computers-still-decide-who-gets-care-280711
22. Royal Commission into the Robodebt Scheme. Report of the Royal Commission into the Robodebt Scheme. Commonwealth of Australia, July 2023.
23. General Data Protection Regulation, Article 22, “Automated individual decision-making, including profiling.” Regulation (EU) 2016/679. https://gdpr-info.eu/art-22-gdpr/
24. Turkle, S. Alone Together: Why We Expect More from Technology and Less from Each Other. Basic Books, 2011.
25. Australian Institute of Health and Welfare. Dementia in Australia 2024. https://www.aihw.gov.au/reports/dementia/dementia-in-australia

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...

The egg is too big. That is the first thing you notice. It rolls out from behind a barn door that is itself the wrong shape, on a farm whose perspective keeps sliding a degree or two out of true, and when the egg cracks open the horse that emerges is proportioned like a child's drawing of a horse attempted by a committee. Its legs are the wrong length. Its eyes are in slightly the wrong place. The soundtrack, set to an off-key rendition of “Old MacDonald Had a Farm”, is already on to the next verse before the animal has finished hatching. The whole sequence lasts about eleven seconds. Then the video cuts to a letter of the alphabet, a new animal, a new impossibility, the same music climbing a half-step higher than your ear wants it to.

This is not, on any sensible definition, a children's video. It is a machine's hallucination of one, optimised for the half-second in which a thirteen-month-old stops crying and reaches for the screen. According to the New York Times, which in February 2026 spent weeks reviewing more than a thousand videos that YouTube's algorithm recommended to accounts configured as children's, it is what vast stretches of the modern toddler's media environment now look like. After a single viewing of a legitimate CoComelon video, the Times reported, more than 40 per cent of the YouTube Shorts subsequently recommended to the test account contained synthetic visuals. The videos carried names that promised to teach the alphabet and animals and colours. They did no such thing. They were, in the exacting formulation the internet has settled on, slop.

On 1 April 2026 a coalition organised by the American advocacy group Fairplay and addressed jointly to Sundar Pichai, chief executive of Google's parent Alphabet, and Neal Mohan, chief executive of YouTube, asked the companies to do something about it. The letter was signed by more than 230 organisations and individual experts, including the American Federation of Teachers, the American Counseling Association, the National Black Child Development Institute, the Canadian Centre for Child Protection, Mothers Against Media Addiction and ParentsSOS. Among the individuals were Jonathan Haidt, author of The Anxious Generation, and the developmental behavioural paediatrician Jenny Radesky of the University of Michigan, who co-directs the American Academy of Pediatrics' Center of Excellence on Social Media and Youth Mental Health. The letter asked for six things: clear labelling of all AI-generated content across YouTube; an outright ban on AI content on YouTube Kids; a prohibition on AI-generated “made for kids” content on the main platform; a rule against recommending AI content to users under eighteen; a parental toggle that switches AI off by default; and a halt to further investment in AI-generated children's content.

Two weeks later, on 13 April, an arXiv paper titled “Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking” added an uncomfortable footnote. Even the parts of those demands that look technical, labelling, detection, identifying machine-made video at the point of delivery, cannot be done reliably with the tools platforms currently deploy. The three major regulatory frameworks that mandate watermarking, the EU AI Act, US Executive Order 14110 and China's Measures for Labeling AI-Generated Synthetic Content, all require some form of traceability; none requires that detection performance be evaluated across languages, cultural content types or demographic groups. The governance gap is not that watermarking is impossible. It is that watermarking, as currently specified, can be present and still fail silently in exactly the contexts in which it is supposed to protect the most vulnerable users.

All of which amounts to a problem with a particular shape. The first media environment that many of today's infants and toddlers encounter is increasingly generated by systems with no developmental mandate, governed by recommendation algorithms that optimise for watch time, delivered through platforms whose detection tools cannot reliably distinguish synthetic from human-made content, to children whose brains are at their most plastic. The question is not whether this matters. The question is what the mattering consists of, and what the companies whose infrastructure produces it are actually obliged to do.

The shape of the slop

To understand why the Fairplay letter exists, it helps to understand what has happened to the economics of children's video. The old attention economy, the one that produced Sesame Street and Bluey and, for better or worse, CoComelon, was expensive. It involved writers and animators and composers and, crucially, child-development consultants. A single episode of a flagship preschool programme could take a year and cost upwards of a million dollars. The cost structure was a filter: people who could not afford specialists did not tend to make the shows.

That filter has collapsed. According to a December 2025 Fortune profile, a 22-year-old entrepreneur named Adavia Davis runs a YouTube network whose videos are almost entirely generated by a proprietary pipeline called TubeGen, built by his partner Eddie Eizner. Scripts and visuals come out of Anthropic's Claude; narration from ElevenLabs; editing is automated. The results can run as long as six hours and cost as little as sixty dollars to produce. Davis told Fortune the network was taking in forty to sixty thousand dollars a month in advertising against about six and a half thousand in operating costs.

Zoom out, and the shape gets more alarming. A December 2025 study by the video-editing company Kapwing examined fifteen thousand trending YouTube channels and isolated 278 that produced nothing but AI-generated content fitting the slop profile. Those 278 channels had collectively amassed 63 billion views, 221 million subscribers, and an estimated 117 million US dollars in annual ad revenue. A single South Korean channel, Three Minutes Wisdom, had accumulated 2.02 billion views on its own. The broader Kapwing analysis suggested between a fifth and a third of the typical YouTube recommendation feed was now AI slop. In some children's categories, independent investigators have reported that only around five per cent of content in a given niche appears to be human-made.

These numbers describe a particular economic equilibrium. Each slop video earns a fraction of what a well-made children's programme earns per viewer, but the marginal cost of producing the next one is close to zero, and YouTube's algorithmic plumbing, which rewards raw watch time, does not meaningfully penalise the difference in quality. Neal Mohan listed “managing AI slop” among YouTube's priorities for 2026 in his February annual letter. The inauthentic content policy, clarified in July 2025, now explicitly targets “templated, low-effort videos at scale”. But YouTube has been clear, via creator liaison Rene Ritchie, that AI itself is not banned, and channels using AI remain eligible for monetisation. The company is trying to have an AI industry and a brand-safe children's platform in the same room at the same time.

What the videos actually show

The specifics are worse than the abstraction. The Times catalogued them at length. A gooey liquid squeezed into a glass of water before turning into animals representing each letter of the alphabet, except the animals were chimeras with mermaid tails. An impossibly proportioned horse hatching from an egg. Faces that warped mid-frame. Extra body parts appearing and disappearing within a single shot. Garbled text purporting to spell words. None of it was longer than about thirty seconds in Shorts form, which, as developmental researchers pointed out to the paper, allows no time for the repetition and narrative scaffolding that underlies actual learning.

Worse, some of the content was not merely incoherent but materially dangerous. A March 2026 follow-up reported by Futurism and the Los Angeles edition of National Today, drawing on videos flagged by the Times and by researchers at Children's Health Defense, documented AI-generated “educational” clips that depicted characters walking in the middle of a road with cars approaching as if this were normal; clips that taught road rules by informing children that “green means right” instead of “go”; clips that showed a baby swallowing whole grapes, a well-documented choking hazard; clips that showed a baby eating honey, which can cause infant botulism and kill children under one. These were not fringe horror videos surfaced by dedicated investigators. They were recommended by the platform, under thumbnails and titles that promised conventional toddler content.

The reason this happens is partly that the models do not know better and partly that, by the time the content has been generated, no part of the delivery pipeline is looking. Generative video systems trained on human-made footage can reproduce the surface characteristics of a children's cartoon, the saturated colours, the nursery-rhyme cadence, the toothy grin on a farm animal, without any internal representation of which of those surface features is meant to model appropriate behaviour. A script that says “baby eats a snack” can be rendered with any snack the model has seen enough of. If the model has seen honey, it can render honey. The video will pass every surface test the platform applies, because the surface is all there is.

Recommendation as developmental hazard

The coalition letter is careful not to suggest that every individual AI-generated video, considered on its own, is harmful. Its central argument is about system effects, and the part of the system that matters most is not creation but distribution. On YouTube, what a child watches next is a function of what the recommender predicts will maximise some combination of watch time, session length and ad impressions. In practice, as the Times documented and as a March 2026 EU Today investigation confirmed, the recommender steers children toward AI content because AI content is optimised, accidentally or deliberately, for exactly the features the recommender rewards: high novelty, short duration, bright colours, rapid cuts, sticky audio, the over-stimulating profile that holds a young child's gaze for longer than a slower, more coherent programme would.

This is the failure mode Radesky's American Academy of Pediatrics policy statement “Media and Young Minds”, published originally in 2016 and updated in 2024, really points at. Children under about two, according to the research literature her statement summarises, exhibit what psychologists call the “video deficit”. They learn language and behaviour from live human interaction far more readily than from video of the same speaker. The video deficit narrows if an adult co-views and scaffolds the material, asking questions, directing attention, connecting what is on the screen to the world around the child. It narrows further if the video involves genuine back-and-forth, as in a video call. It does not narrow if the child is alone in front of a screen that is alone in front of an algorithm.

Patricia Kuhl's 2003 work at the University of Washington is the canonical study. Nine-month-old American infants exposed to Mandarin speakers in person retained sensitivity to Mandarin phonemes they would otherwise have lost by twelve months. Infants exposed to the same speakers on video, saying the same things in the same way, showed no such retention. The human face, the shared gaze, the social contingency of the exchange, were doing work the pixels alone could not do. Two decades of follow-up research, including work on video chat by Lauren Myers and colleagues in Child Development (2014) and the 2018 PNAS study “Two are better than one” on peer-assisted video learning, has consistently found that whatever makes human interaction educationally potent for very young children is not reliably reproduced by a screen.

Now consider the environment the Fairplay letter is describing. A toddler sits in a high chair or a buggy or a back seat, alone with a tablet. The tablet is showing content no adult has vetted, because no adult knew what was going to appear next, because nobody knows, because the recommender is choosing in the same moment the child is watching. The content is generated by a model that has no mental model of a child. The audio is pitched to hold attention by exploiting the same perceptual features that attention-capturing advertising exploits in adults: the sudden colour change, the unexpected musical interval, the face that does not quite resolve. In place of joint attention, there is the machine's attention to engagement metrics. In place of a parent pointing at a horse and saying “horse”, there is a model that cannot quite render a horse and does not know why that matters.

The likely developmental cost is not that any single child will be ruined by any single video. It is that the aggregate environment is nudging language acquisition, narrative cognition and the early scaffolding of theory of mind toward corners they do not want to be in. Narrative cognition depends on following a story that unfolds over time, with characters whose goals and states change in recognisable ways. The thirty-second AI clip does not have characters in that sense; it has arrangements of pixels that a model has stitched together to maximise retention on a swipe feed. Parasocial bonds, which the Georgetown Center on Digital Media and Children's Development has shown can be genuinely educationally useful when they form with coherent characters like Daniel Tiger or Elmo, cannot form in the usual way with characters who mutate between shots because no stable character was ever authored. Theory of mind, the slow developmental achievement of understanding that other beings have beliefs, desires and intentions different from your own, depends on encountering minds in a reliable enough way to build a model of what a mind is. The AI chimera on the Shorts feed has no mind, never had one, and does not cohere long enough to pretend to.

The governance gap in detection

The coalition letter's demand for labelling is the one that sounds most technically tractable and is, in practice, the hardest. This is the territory the April 2026 arXiv paper stakes out. “Who Gets Flagged?” argues that the existing policy scaffolding around watermarking, Article 50 of the EU AI Act, US Executive Order 14110, China's content-labelling measures, shares a common flaw. Each framework obliges producers of generative AI systems to mark their outputs in some way, and each assumes, without requiring evidence, that the marks can be reliably recovered downstream. Recoverability varies with the statistical properties of the content itself. For text, with language. For images, with visual style. For audio, with compression, pitch and rate. None of the three frameworks requires evaluation of recoverability across those axes, which means a watermarking regime can be formally compliant and still be invisible on exactly the subset of content the regulator most needed to flag.

The paper does not single out children's content, but the implications are stark. Children's video is aggressively re-compressed, up-pitched, cut into Shorts, remixed with stock footage, re-uploaded across accounts. Every step degrades the signal. By the time a video reaches YouTube Kids, the original watermark a conscientious model provider might have embedded at generation time is, with high probability, unreadable. A related position paper, “Watermarking Without Standards Is Not AI Governance”, published on arXiv by Carnegie Mellon and partner institutions in May 2025 and updated in March 2026, makes the same point from the other direction: in the absence of shared standards, different providers produce different, mutually invisible watermarks, and platforms that want to act on the information cannot, because each provider's scheme requires a different decoder.

This does not mean YouTube cannot label AI content. It means the platform cannot label it purely by detecting a watermark in the file. It has to rely on other signals: account behaviour, metadata, manual review, creator flags. YouTube's 2024 disclosure requirement, updated through 2025, asks creators to tick a box in YouTube Studio when they upload “realistic altered or synthetic” content. The honest question is how much self-disclosure it really produces among operators whose business model depends on the audience not realising the videos are AI-generated. The same issue surfaced when the platform tried to rely on creator self-identification of child-directed content after the 2019 FTC settlement that required Google and YouTube to pay 170 million US dollars for alleged violations of the Children's Online Privacy Protection Act. Self-identification is a floor; the operators with the most reason to lie are the ones the label is supposed to constrain.

The obligations platforms are not yet meeting

The legal scaffolding around children's content has been quietly maturing while this environment has been forming beneath it. COPPA, the 1998 Children's Online Privacy Protection Act, is still the US baseline, and the 2019 settlement forced YouTube to build the “made for kids” designation that now, ironically, makes it easier for AI operators to flag their own slop into the child-directed pipeline. The UK's Online Safety Act 2023, whose Protection of Children codes came into force on 25 July 2025, imposes a statutory duty of care on user-to-user services accessed by children, backed by fines of up to 18 million pounds or 10 per cent of global turnover, enforced by Ofcom, which by March 2026 had opened more than eighty investigations under the Act's age-assurance provisions. The Act requires proportionate measures against content “harmful to children”, a category Ofcom's January 2026 guidance interprets to include content presenting risks of physical harm.

The EU's Digital Services Act, whose Article 28 guidelines on the protection of minors were finalised by the European Commission on 14 July 2025, obliges very large online platforms including YouTube to ensure “a high level of privacy, safety and security of minors”. The guidelines specifically address recommender systems: minors should have the ability to reset their recommended feeds; non-profiling recommender options should be available and, where appropriate, set as default; feedback mechanisms such as “show me less” should directly influence content visibility. The guidelines are non-binding in the strict sense, but the Commission has described them as a “significant and meaningful benchmark” for Article 28 compliance, and the enforcement powers are real.

Against this landscape, the Fairplay letter's demands look less like activism and more like a preview of what compliance will, within a few years, require. Clearly labelling all AI-generated content is a corollary of Article 50 once detection becomes reliable enough to enforce. Banning AI-generated content on YouTube Kids is implicit in a UK Online Safety Act duty of care that treats the cultivation of cognitive harm to under-fives as a foreseeable consequence of the current recommender configuration. Barring recommender systems from pushing AI content to under-eighteens is effectively what the DSA Article 28 guidelines describe. A parental toggle is the minimum user-control affordance those same guidelines enumerate.

What the platforms are not currently meeting is less the letter of these regimes than the spirit. YouTube's inauthentic content policy will remove a channel that produces templated videos at scale, but reactively, after views have accumulated and revenue has been disbursed and children have watched. YouTube Studio's disclosure tool flags AI content if the creator opts in, which is exactly what creators with a business model built on opacity will not do. The 2019 COPPA “made for kids” designation was never intended as a machine-readable tag for quality control, and is now being inverted by AI operators who flag their own output into the children's pipeline to collect child-directed advertising revenue that complies with COPPA's behavioural-targeting rules. None of this is outright illegal. A great deal of it is, in the terms of the Online Safety Act, arguably not reasonably practicable to prevent. That is the hinge on which every conversation about platform responsibility now swings.

Why the platforms have not acted

The political economy is mundane. The first reason is revenue. The 117 million US dollars in annual ad revenue across 278 slop channels is a small slice of YouTube's take, but non-trivial and growing, produced by channels whose cost of adding the next video is measured in dollars rather than thousands. Banning AI slop from YouTube Kids the way Fairplay has asked would require YouTube to either build a reliable classifier, exactly the problem the arXiv paper says is not solvable with watermarking alone, or default to human review of new children's channels, expensive, slow and the opposite of the platform's operating philosophy.

The second reason is that YouTube is not the primary villain of its own economics, and the company knows it. The slop pipeline extends upstream to Anthropic's Claude, OpenAI's video models, ElevenLabs' voice clones, Runway's generative video, and downstream to advertising networks that pay for watch time regardless of whether the watcher is sixteen months old. Each of those players faces regulatory pressure to label outputs; each, on present technology, cannot guarantee a label will survive the distribution pipeline. For YouTube to ban AI on Kids in a way that actually worked, it would need cooperation from every upstream provider and a detection regime that outperformed every adversarial operator with a financial incentive to evade it. That is not a problem any single platform can solve unilaterally.

The third reason is that there has historically been no political cost to inaction. COPPA enforcement is slow. The FTC's 170-million-dollar 2019 penalty was extracted for a decade of behavioural advertising violations, not for the content itself. The Online Safety Act has not yet been tested in a children's-content case at precedent level. The DSA is still in its enforcement adolescence. None of this is an excuse, but it explains why the board of Alphabet has not yet been presented with a memo saying the quiet part out loud: that the expected value, under the current regime, of cleaning up children's AI content is smaller than the expected value of continuing to run the platform roughly as it is.

The coalition letter is trying to change that calculus by making the political cost of inaction legible. Getting 230 organisations on the same page is a logistical feat. Getting the American Federation of Teachers, the National Black Child Development Institute, the Canadian Centre for Child Protection and a Jonathan Haidt on the same letter makes the story harder to dismiss as activist overreach or parental paranoia. It builds a record that will be cited at the next Ofcom enforcement action, the next DSA systemic-risk assessment, the next COPPA rulemaking. Those are the places where the cost-benefit math for YouTube will change.

What a responsible regime would look like

The contours of a workable regime, extrapolating from the coalition demands and the regulatory trajectory, are not mysterious. They are just expensive.

On the content side, YouTube Kids, the walled-garden app, should not contain AI-generated video at all for children under a threshold set by reference to the developmental literature Radesky and her colleagues have been producing for a decade. The argument that this is technically infeasible because detection is unreliable is, on closer inspection, an argument for whitelisting rather than blacklisting: allow only content from a finite set of human creators whose identities and production processes are verified, and default to refusal for everything else. This is the model the BBC CBeebies app already runs in effect. The cost is dramatic reduction in the library. The benefit is dramatic reduction in the harm surface.

On the main platform, the recommender should not push AI-generated content to accounts logged in as under-eighteen, and should offer a non-profiling recommender by default to those accounts. This is the DSA Article 28 default; Fairplay has simply asked YouTube to apply it to the specific class of content the data suggests is most hazardous. A parental toggle goes further, giving adult users a single control for AI content across the household account, on by default, shifting the cognitive labour of supervision back onto the platform.

On disclosure, the watermarking regime needs the interoperability the May 2025 arXiv paper has already laid out: a shared, audited, open-standard scheme every major model provider is contractually obliged to implement and every platform is contractually obliged to decode. Article 50 is the right vehicle. In the interim, platforms should rely on stronger signals than creator self-declaration: account behaviour, production cadence, upload patterns, and the content-forensics tooling newsrooms have been developing since the first deepfakes appeared.

On advertising, the monetisation pipeline for child-directed AI content should be closed. The post-COPPA made-for-kids designation was always meant to constrain commercial exploitation of child viewers. Treating AI-generated made-for-kids content as ineligible for the advertising revenue that funds the slop economy is the single regulatory change with the largest expected effect on the supply side. It does not require new legislation. It requires the platform to interpret its existing obligations in the direction of children rather than operators.

What the mattering consists of

Return, finally, to the hatching horse. There is a version of this story in which the debate is a fuss about nothing. Children have always watched odd television. The Teletubbies were strange. Peppa Pig is, on close inspection, relentlessly repetitive. No generation raised on the BBC test card turned out wholly deranged. The calculator analogy developmental psychologists have spent three years tearing down is not the only deflection available; there is also a cynical one, which says commercial children's media has always been a cognitive hazard and AI has merely lowered the production cost.

The answer the Fairplay coalition is giving is that scale and direction both matter. Scale, because the difference between a child watching one strange programme a week and a child being served an endless algorithmic feed of strange synthetic content is not a difference of degree but of what the child's media environment is. Direction, because every previous generation of children's media, even the commercially cynical ones, involved adults at some point making decisions about what was appropriate for a six-year-old to see, and those decisions, however imperfect, were adults' decisions. The AI slop pipeline removes those decisions. It replaces them with engagement metrics. It delegates to a model the things a responsible children's producer, even a merely competent one, would have been contractually obliged to think about.

The long-term implications are legible even if they are not yet measurable. A cohort of children whose earliest language exposure is dominated by content with no consistent narrator, no stable characters, no coherent grammar of the world, will build internal models of narrative, language and social reality that reflect that input. A cohort whose parasocial bonds form with mutating AI chimeras rather than with Daniel Tiger or Bluey or Elmo will have parasocial bonds that are less developmentally productive. A cohort whose first encounters with educational content are not educational, because they teach that green means right and that babies eat honey, will start formal schooling with a different set of priors than a cohort whose first encounters were produced by adults trying, in however flawed a way, to teach something true. These are not apocalyptic predictions. They are base-rate extrapolations from a developmental psychology literature that has been remarkably consistent for fifty years.

The obligations platforms are not currently meeting are also not exotic. They are the obligations that have existed, in some form, since television was invented and any regulator decided children's television should be treated differently from adult television. Broadcast regulators in every developed country, from the FCC to Ofcom to ACMA, have always operated on the premise that the developmental vulnerability of young children creates a special duty for the people who transmit content into their homes. YouTube has argued for twenty years that it is not a broadcaster, that it is a platform, that the content on it is produced by its users and that its role is infrastructural rather than editorial. The AI slop crisis is the moment at which that argument becomes unsustainable. When the content is not being produced by users in any meaningful sense, when it is produced by models operated by commercial entities whose relationship with the child is purely extractive, the platform is not hosting user content. It is running a delivery system for a synthetic children's media industry it declined to regulate because declining to regulate was profitable.

The coalition letter does not put it in quite those terms. It asks, politely and in the language of child welfare, for YouTube and Google to behave as if the children on the other end of the recommendation algorithm were their own. The question the letter is really posing is the one the next five years of regulatory enforcement will answer: whether a platform whose products meaningfully shape the cognitive development of a generation of infants and toddlers can continue to claim, with a straight face, that the cognitive development of that generation is somebody else's problem. It is not a question that has a technical answer. It is a question about what kind of industry we want children's media to be, and who, in the end, is responsible for the horse that hatches from the egg.

---

References & Sources

1. Metz, C. and Mickle, T. “How A.I.-Generated Videos Are Distorting Your Child's YouTube Feed.” The New York Times (via dnyuz.com), 26 February 2026. https://dnyuz.com/2026/02/26/how-a-i-generated-videos-are-distorting-your-childs-youtube-feed/
2. Fairplay for Kids. “YouTube: Stop 'AI Slop' for Kids.” 1 April 2026. https://fairplayforkids.org/youtube-stop-ai-slop-for-kids-says-letter-from-fairplay-over-200-experts-including-jonathan-haidt/
3. Fairplay for Kids. Open letter to Sundar Pichai and Neal Mohan. March 2026. https://fairplayforkids.org/wp-content/uploads/2026/03/YouTube-Letter-AI-Slop.pdf
4. “Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking.” arXiv:2604.13776, April 2026. https://arxiv.org/abs/2604.13776
5. “Watermarking Without Standards Is Not AI Governance.” arXiv:2505.23814, May 2025 (rev. March 2026). https://arxiv.org/abs/2505.23814
6. “Missing the Mark: Adoption of Watermarking for Generative AI Systems.” arXiv:2503.18156, March 2025. https://arxiv.org/abs/2503.18156
7. Futurism. “'Educational' YouTube AI Slop Encourages Kids to Play in Traffic.” March 2026. https://futurism.com/artificial-intelligence/educational-youtube-ai-slop-play-in-traffic
8. National Today (Los Angeles). “AI-Generated YouTube Videos Teach Children Dangerous Behaviors.” 30 March 2026. https://nationaltoday.com/us/ca/los-angeles/news/2026/03/30/ai-youtube-videos-teach-children-dangerous-behaviors/
9. EU Today. “YouTube steers children towards AI-made videos disguised as educational content.” 2026. https://eutoday.net/youtube-steers-children-towards-ai-made-videos/
10. Children's Health Defense. “AI 'Slop' Puts Young Brains at Risk.” 2026. https://childrenshealthdefense.org/defender/ai-slop-misleading-videos-youtube-children-safety-brain-development-risks/
11. Fortune. “This 22-year-old college dropout makes $700,000 a year from 'AI slop'.” 30 December 2025. https://fortune.com/2025/12/30/ai-slop-faceless-youtube-accounts-adavia-davis-user-generated-content/
12. MediaNama. “AI Slop Videos Make Up 33% of YouTube Feed, Says Study.” December 2025. https://www.medianama.com/2025/12/223-ai-slop-videos-youtube-algorithmic-recommendations/
13. YouTube Blog. “Neal Mohan's 2026 Letter: The Future of YouTube.” February 2026. https://blog.youtube/inside-youtube/the-future-of-youtube-2026/
14. Flocker. “YouTube Inauthentic Content Policy: AI Enforcement Wave 2026.” 2026. https://flocker.tv/posts/youtube-inauthentic-content-ai-enforcement/
15. Radesky, J. and Christakis, D. “Media and Young Minds.” AAP Policy Statement, Pediatrics, 2016 (updated 2024).
16. Kuhl, P., Tsao, F. and Liu, H. “Foreign-language experience in infancy.” PNAS, 2003.
17. Myers, L. et al. “Skype me! Socially Contingent Interactions Help Toddlers Learn Language.” Child Development, 2014. https://pmc.ncbi.nlm.nih.gov/articles/PMC3962808/
18. Lytle, S., Garcia-Sierra, A. and Kuhl, P. “Two are better than one.” PNAS, 2018. https://www.pnas.org/doi/10.1073/pnas.1611621115
19. Brunick, K. et al. “Children's future parasocial relationships with media characters.” Georgetown CDMC, 2016. https://cdmc.georgetown.edu/wp-content/uploads/2016/04/Brunick-et-al-2016.pdf
20. FTC. “Google and YouTube Will Pay Record $170 Million.” 4 September 2019. https://www.ftc.gov/news-events/news/press-releases/2019/09/google-youtube-will-pay-record-170-million-alleged-violations-childrens-privacy-law
21. UK Government. “Online Safety Act 2023.” https://www.legislation.gov.uk/ukpga/2023/50
22. Ofcom. “Protection of children duties under the Online Safety Act.” 2025. https://www.ofcom.org.uk/online-safety/protecting-children/protection-of-children-duties-under-the-online-safety-act
23. Ofcom. “Online safety industry bulletin, March 2026.” https://www.ofcom.org.uk/online-safety/illegal-and-harmful-content/online-safety-industry-bulletins/online-safety-industry-bulletin-march-2026
24. European Commission. “Guidelines on the protection of minors.” 14 July 2025. https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-protection-minors
25. Children's Online Privacy Protection Act of 1998 (COPPA), 15 U.S.C. §§ 6501-6506.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Discuss...