SmarterArticles

The error codes were lying, and the machine was very sure of itself.

In the 1980s, a Xerox field service technician was called out to a photocopier. It produced, in the words of the researchers who later wrote the episode up, “copious raw information in the form of error codes” and “obligingly crashed when tested”. The trouble was that the error codes and the crashes did not tally. The company's training and its directive documentation tied each error to a code and each code to a fix, so once the codes stopped telling the truth, the official procedure had nothing left to offer except replacing the machine. The technician called in a specialist. The specialist was baffled too.

What happened next was recorded by Julian Orr, an anthropologist then working with Xerox PARC, and later retold by John Seely Brown and Paul Duguid in a 1991 paper that became one of the founding texts of organisational learning. The two men did not consult a manual. They told stories. Through the morning, over lunch, and back in front of the machine all afternoon, they swapped memories of other copiers with overlapping symptoms, testing each story against the machine's behaviour. Orr's account describes “a five hour effort” that “yielded a dozen anecdotes”. The diagnosis finally came out of what Orr called an “antiphonal recitation”, in which the two men told different versions of the same encounter until the versions converged on the fault.

Keep that scene in mind, because it is the most precise description available of the thing Pearson says the skilled trades are about to lose. It is not information. The machine had plenty of information. It is the judgement to know when the information is wrong, and that judgement lived in two people talking to each other beside a broken machine.

On 24 September 2026, at the WorldSkills International Conference in Shanghai, Pearson launched research warning that a “triple capability gap” is opening across the skilled trade, technical and service occupations that keep hospitals, factories and infrastructure running. AI is reshaping those jobs faster than education and training systems can adapt, the report argues, just as experienced workers retire and take decades of practical knowledge with them. The result is a generation of early-career workers at risk of becoming, in Pearson's phrase, “AI-rich but experience-poor”.

Which raises two uncomfortable questions. What happens to a trade when the apprentice turns up holding an AI assistant, but the mentor who was supposed to teach the difference between right and almost-right has already gone? And can the knowledge that lives in a plumber's hands and a technician's instincts be written into training systems at all, or does some of it only ever travel from one person to another?

What Pearson Actually Found in Shanghai

The report is titled For Every Future: Preparing Early-Career Skilled Trade, Technical and Service Talent for an AI-Infused World. It combines labour market and AI exposure analysis across the United States, the United Kingdom and China with surveys, interviews and roundtables involving employers, educators, industry bodies, workforce experts, students and early-career workers. It also looks closely at two occupations: pharmacy technicians and industrial machinery mechanics.

The core argument is that these workers now need three things at once. The first is role-specific AI literacy and the critical judgement to use AI safely. The second is a set of human capabilities, including communication, collaboration, adaptability, emotional intelligence and learning agility. The third, and the one this article is about, is the practical and institutional knowledge traditionally developed through experience and handed down by experienced colleagues. Pearson's framing is that AI is more often changing day-to-day tasks than eliminating roles altogether, which makes the gap harder to see: nobody gets a redundancy letter, but the job underneath them quietly changes shape.

Some of the numbers are stark. Across the UK occupations covered by the research, 99 per cent of openings are expected to come from replacement demand, meaning workers who retire, change careers or leave the labour market, rather than from new jobs. The US figure is 98.5 per cent. Fewer than half of the industrial machinery mechanics surveyed felt their training was preparing them for AI-enabled work. One UK pharmacy industry participant told the researchers that it takes two to three years to revise pharmacy technician education and training standards, which is a long time in a field where the tools can change within a single product cycle. And the stakes are not abstract. Among pharmacy technicians, 98 per cent rated exactness and accuracy as extremely important, and 44 per cent rated the consequences of error as extremely serious.

Sharon Hague, Pearson's President of English Language Learning and its UK chief executive, put the problem plainly at the launch. “The challenge is that demand for these workers is rising just as many experts are retiring and taking decades of practical knowledge with them,” she said. “If we don't rethink how expertise is developed, the gap between what employers need and what workers can do will continue to grow.”

It is worth being clear about what kind of document this is. Pearson is an education and assessment company, and a report concluding that training systems need rethinking is also, inevitably, a report about a market Pearson sells into. That does not make its findings wrong, and its replacement-demand figures sit comfortably alongside independent labour data. But its most interesting claim is not a statistic. It is the idea buried in the third capability: that there is a category of knowledge that cannot simply be bought, downloaded or prompted into existence, and that its main carriers are leaving.

The Arithmetic of Replacement

Start with the people who carry that knowledge, and the rate at which they are leaving.

In construction, the Construction Industry Training Board's Construction Workforce Outlook for 2026 to 2030, published on 17 June 2026, estimates that the industry will need an average of 41,200 extra workers a year, roughly 206,000 over five years. CITB's diagnosis is blunt: too few people are entering the sector, too many experienced workers are leaving, and productivity improvements have not been enough to close the gap. In US manufacturing, Deloitte and the Manufacturing Institute projected in 2024 that as many as 3.8 million additional employees could be needed between 2024 and 2033, and that as many as 1.9 million of those jobs could go unfilled if the skills and applicant gaps are not addressed.

Plumbing makes the pattern concrete. The US Bureau of Labor Statistics counts 510,600 plumbers, pipefitters and steamfitters, projects around 42,000 openings a year over the coming decade, and notes that many of those openings will come from the need to replace workers who move to other occupations or leave the labour force, for example to retire. The same entry explains how the trade is learnt: most plumbers learn on the job through an apprenticeship lasting four to five years, with 2,000 hours of paid on-the-job training each year alongside technical instruction. In England, the Level 3 Plumbing and Domestic Heating Technician apprenticeship standard has a typical duration of 48 months.

Those durations are the real point. Four years is not how long it takes to learn the facts of plumbing, which would fit in a decent textbook. Four years is how long it takes to be around enough jobs, enough awkward pipework, enough customers and enough mistakes, in the company of someone who already knows, for the facts to turn into judgement.

And the pipeline feeding that process has been narrowing. Department for Education figures published on 27 November 2025 recorded 353,500 apprenticeship starts in England in the 2024/25 academic year, up from 340,000 the year before. But in 2016/17, the year the apprenticeship levy was introduced, there were 494,900. The mix has shifted too. Higher-level apprenticeships now make up around 40 per cent of starts, while intermediate apprenticeships, the traditional entry route for young people into a trade, have fallen from 43 per cent of starts in 2017/18 to 19 per cent. The system is producing more graduate-style apprenticeships and fewer people at the bottom of the ladder, which is precisely where the long, slow, person-to-person transfer of craft knowledge has always happened.

Put the two trends together and you get Pearson's scenario in its simplest form. The mentors are leaving faster than they can be replaced, and the replacements are arriving through a narrower door, into jobs whose tools are changing underneath them.

Knowing More Than We Can Tell

To understand why the loss of mentors matters more than the loss of workers, you have to go back to a Hungarian-British chemist turned philosopher.

In The Tacit Dimension, published in 1966, Michael Polanyi compressed a lifetime of thinking about scientific practice into a single sentence: “we can know more than we can tell.” We recognise a face among thousands without being able to say how. Polanyi's claim was stronger than the observation that some knowledge is hard to describe. He argued that all knowledge rests on a tacit foundation, a set of skills, habits and perceptions that we rely on without attending to them.

The idea might have stayed in philosophy if a young sociologist had not put it to the test. In the early 1970s, Harry Collins, now a distinguished research professor at Cardiff University, studied laboratories trying to build a then-new device called the TEA laser. His finding, published in 1974 and followed up in 1975 with his co-author R. G. Harrison, was striking. No scientist he studied succeeded in building a working laser using only published papers or other written sources. Every group that managed it had obtained some crucial part of the knowledge through personal contact: phone calls, visits, someone who had built one standing in the room. The written account was complete in every respect except the one that mattered.

Collins spent decades refining what that means, and his 2010 book Tacit and Explicit Knowledge offers the most useful map we have for Pearson's question. He splits tacit knowledge into three kinds.

The first is relational tacit knowledge. This is knowledge that could in principle be written down but in practice is not, because nobody has bothered, because it seems too obvious, or because someone would rather keep it to themselves. The torque setting a senior fitter uses on a particular flange, which differs from the manual for good reasons nobody recorded, is relational. So is the knowledge that a particular customer's boiler has a history.

The second is somatic tacit knowledge, which lives in the body. Balancing a bicycle is Collins's classic example. For the trades, think of the feel of a pipe cutter biting cleanly, the sound of a pump running slightly wrong, the resistance of a fitting that is about to strip. This is knowledge whose limits come from the nature of bodies and brains, not from social choice.

The third, and for Collins the deepest, is collective tacit knowledge. This is knowledge that belongs to a community rather than to any individual: the sense of what counts as a proper job, which rules can be bent on site and which cannot, when a shortcut becomes a hazard. You acquire it by being socialised into the group that holds it, and for Collins it is the hardest kind of tacit knowledge to make explicit, because it lives in the group rather than in any one head.

That three-way split turns out to answer most of the question in the headline, but before getting there it is worth seeing what happens when a society forgets that the split exists.

When the Recipe Loses Its Impurity

In the 2000s, the US National Nuclear Security Administration set out to refurbish the W76 warhead carried on Trident submarine-launched missiles. One component needed a classified material code-named Fogbank. The facility that had produced it had long since closed. According to a 2009 article in Los Alamos National Laboratory's Nuclear Weapons Journal, most of the personnel involved in the original production process were no longer available, and the agency had to reconstruct the process from historical records.

The reconstruction did not work. According to the Los Alamos account, as reported by the defence publication The War Zone, the new process made a purer material than the old one, and the purity was the problem. The original batches had contained an impurity that the weapon's performance had quietly come to depend on. Nobody had documented it because nobody had known it mattered, and the people who might have noticed the difference were no longer there.

The Government Accountability Office report GAO-09-385, released on 2 March 2009, documented the cost. The planned first refurbished W76 slipped from September 2007 to September 2008, and NNSA spent $69 million addressing Fogbank production problems.

Fogbank is an extreme case, but it shows plainly what Pearson is warning about. The written record was, as far as anyone knew, complete. What was missing was not a fact but a relationship between facts, the kind of thing an experienced operator notices because something looks or feels different from last time. In Collins's terms, it was relational knowledge that had never been told, and the only people who could have told it were gone.

The Assistant That Talks Like a Journeyman

Now introduce the AI assistant.

The tools arriving on shop floors and in vans are not speculative. In May 2025, Siemens announced that its Maintenance Copilot Senseye would give maintenance teams “expert-level equipment diagnostics” without their needing specialist technical knowledge, citing pilot implementations with an average 25 per cent reduction in reactive maintenance time. In September 2025, ServiceTitan, whose software is widely used by US plumbing, heating and electrical contractors, launched Atlas, an AI assistant that gives technicians in the field instant answers through its mobile app. Its co-founder and president, Vahe Kuzoyan, described it as “a trusted co-pilot or even a chief of staff that adapts to the specific workflows” of each business.

There is real evidence that such tools help people who are new to a job. The best-known study, by Erik Brynjolfsson, Danielle Li and Lindsey Raymond, published in the Quarterly Journal of Economics in 2025, tracked 5,179 customer support agents as a generative AI assistant was rolled out. Productivity rose by 14 per cent on average and by 34 per cent for novice and low-skilled workers, with minimal effect on the most experienced. The authors offer suggestive evidence that the tool worked by spreading the best practices of more able workers and helping newer ones move down the experience curve. In other words, some relational tacit knowledge, the kind that top performers had but had never written down, had been captured in the data and handed on.

That is a genuine counterweight to the gloomier reading of Pearson's report. But notice what it measured: speed and resolution in conversations, in a domain made of language. The question for the trades is what happens outside that domain, where the work is physical, the stakes are safety-critical, and the machine can be confidently wrong.

Collins has a concept that fits the new assistants almost too neatly. In Rethinking Expertise, written with Robert Evans and published in 2007, he distinguishes contributory expertise, the ability to actually do the work, from interactional expertise, the ability to speak the language of a practice fluently without being able to practise it. Specialist journalists and good peer reviewers have interactional expertise. They can talk the talk. In a 2024 paper in AI and Society, Collins argued that large language models learn from the internet rather than being socialised into communities, and that the result is systems that answer with fabrications as fluently as with facts.

An AI assistant is, in this sense, the most complete interactional expert ever built. It speaks the language of every trade. It can explain a pressure test, describe a failed seal, list the causes of a tripping breaker. What it lacks is the thing that made the Xerox specialist valuable: a body of experience against which to notice that the story it is telling does not fit the machine in front of it. Hand that fluency to an apprentice who has no such experience either, and you get Pearson's phrase exactly. AI-rich. Experience-poor.

Right Versus Almost Right

The danger is not that the machine is usually wrong. It is that it is usually right, and people learn to stop checking.

Researchers have been measuring this for decades. In a 2010 review in Human Factors, Raja Parasuraman and Dietrich Manzey described automation bias as the tendency to over-rely on automated aids, producing errors of omission, where people miss problems the system fails to flag, and errors of commission, where they follow advice that is wrong. Two findings from that review should worry anyone designing AI-assisted apprenticeships. Automation bias occurs in experts as well as novices. And it cannot be prevented simply by training or instructions.

Recent studies show what happens when AI enters that picture. In the field experiment with 758 Boston Consulting Group consultants reported by Fabrizio Dell'Acqua and colleagues at Harvard Business School, Wharton, MIT Sloan and Warwick, AI help made people substantially faster and better on tasks within the model's capabilities. But on a task deliberately chosen to fall outside what the AI could do well, consultants using AI were 19 percentage points less likely to reach the correct answer than those working without it. The researchers' metaphor for this is a jagged frontier: the boundary of what the AI can do well is irregular and hard to see, which makes it easy to cross without noticing.

Then there is the question of what reliance does to the learner. In January 2026, Anthropic published a randomised controlled trial of 52 mostly junior software developers learning an unfamiliar Python library. Those who used AI assistance scored 50 per cent on a follow-up quiz, against 67 per cent for those who coded by hand. The largest gap was on debugging questions, which is to say on the ability to recognise when code is wrong and why. Code is not copper pipe, but the skill that suffered is exactly the one the Pearson report is worried about: knowing when the output in front of you is almost right, and therefore wrong.

The most sobering evidence comes from medicine, where the consequences are measured in missed cancers. A multicentre observational study led by Krzysztof Budzyń and published in The Lancet Gastroenterology and Hepatology in August 2025 looked at 19 experienced endoscopists, each with more than 2,000 colonoscopies behind them. After AI-assisted detection was introduced, their adenoma detection rate in procedures performed without the AI fell from 28.4 per cent to 22.4 per cent. These were not novices. They were experts whose unaided performance appeared to decline after routine exposure to the machine. If skilled professionals can lose ground that quickly, the question for a trainee who never builds that ground in the first place answers itself.

Lisanne Bainbridge saw all of this coming in a short paper published in Automatica in 1983. “Ironies of Automation” observed that efficient retrieval of knowledge from long-term memory depends on frequency of use, and that some knowledge “develops only through use and feedback about its effectiveness”. Then she wrote the sentence that reads like a summary of the Pearson report forty-three years early: “There is some concern that the present generation of automated systems, which are monitored by former manual operators, are riding on their skills, which later generations of operators cannot be expected to have.”

That is the mechanism. Today's AI-assisted workplaces look safe partly because the people supervising them learnt their trade before the assistant arrived. They can tell when the answer is off because they once had to work it out themselves. Remove them, and the safety margin goes with them, without anyone noticing, until the error codes stop matching the crashes.

How Apprenticeship Actually Transmits Judgement

If the mentor's value is not information, what exactly is being transferred when an apprentice works alongside one?

In 1991, the anthropologist Jean Lave and the educational theorist Etienne Wenger published Situated Learning, built partly on studies of Vai and Gola tailors in Liberia. They found that the apprentices were not formally taught tailoring in any conventional sense. They learnt through what Lave and Wenger called legitimate peripheral participation: doing real but low-risk parts of real work at the edge of a community of practitioners, gradually moving towards the centre as their competence grew. The learning was inseparable from belonging. You did not learn tailoring and then join the tailors. You became a tailor by being among them.

Richard Sennett, the sociologist, made a related case in The Craftsman, published in 2008. He describes craft as “a dialogue between concrete practices and thinking; this dialogue evolves into sustaining habits, and these habits establish a rhythm between problem solving and problem finding.” Problem finding is the key phrase. A manual or an assistant can solve a problem you have already identified. Knowing that there is a problem, that the joint looks fine but is not, that the reading is plausible but wrong, is the skill that comes from the rhythm Sennett describes, and it takes time. “By one commonly used measure,” he notes, “about ten thousand hours of experience are required to produce a master carpenter or musician.”

Sennett also noticed what happens when a tool takes over the part of the work where that rhythm develops. Writing about computer-aided design, he recalls the physicist Victor Weisskopf telling MIT students who presented him with computer-generated results: “When you show me that result, the computer understands the answer, but I don't think you understand the answer.” His conclusion is blunt: “when the head and the hand are separate, it is the head that suffers.”

The most careful modern study of this problem comes from Matt Beane, who researches technology and work in the Technology Management department at the University of California, Santa Barbara, and whose 2024 book The Skill Code draws on years of fieldwork in robotic surgery. In traditional open surgery, residents learnt by doing progressively more of each operation with a senior surgeon's hands beside theirs. The surgical robot changed that. “My research shows that the lead surgeon controls the robot, and the trainee has become completely optional,” Beane said when the book was published. The few residents who became genuinely skilled did so through what he calls shadow learning, bending the rules to get practice, including spending hundreds of hours watching recorded procedures. Beane frames the conditions for building skill as challenge, complexity and connection, and it is the third, the bond of trust between novice and expert, that intelligent machines most easily sever.

The pattern in Beane's operating theatres is the pattern Pearson is warning about in plant rooms and pharmacies. The technology does not attack the apprentice directly. It makes the apprentice unnecessary to the expert's work, and in doing so quietly removes the apprenticeship.

What Can Be Written Down and What Cannot

So, back to the second question. Can a plumber's hands and a technician's instincts be written into training systems?

The honest answer is some of it, more than sceptics assume, and less than vendors claim. Collins's three categories show where the lines fall.

Relational tacit knowledge can be captured, and there is a famous proof. In his 1991 Harvard Business Review article “The Knowledge-Creating Company”, Ikujiro Nonaka described how Matsushita's engineers struggled to design a home bread-making machine that kneaded dough properly. A software developer, Ikuko Tanaka, proposed a solution: she trained with the head baker at the Osaka International Hotel, whose bread was reputed to be the best in the city, to study his kneading. She noticed that he had a distinctive way of stretching the dough. After a year of trial and error with the project's engineers, she produced specifications, including special ribs inside the machine, that reproduced the baker's technique. The resulting “twist dough” method gave Matsushita a product that set a record for first-year sales of a new kitchen appliance.

It is often told as proof that tacit knowledge can be made explicit. Look closely, though, at how it was done. The knowledge was not extracted by interview or questionnaire. Tanaka had to apprentice herself to the baker, learning by observation, imitation and practice, as Nonaka put it, before she could translate anything. The conversion ran through a person. And it took a year.

That suggests a practical route for the trades. Much of what a retiring engineer knows is relational: the history of a particular plant, the quirks of a particular model, the workarounds that never made it into the manual. That knowledge can be captured, and AI can help, by transcribing, indexing and retrieving it. The Brynjolfsson study suggests AI can even spread the habits of top performers to newcomers. But capture has a deadline. It has to happen while the experts are still at work, alongside the people who will inherit the job, because the Tanaka story suggests the most valuable capture happens through apprenticeship rather than interview. Fogbank shows what happens when nobody does it.

Somatic tacit knowledge is harder. Simulation, haptic training rigs, augmented reality overlays and video can all accelerate it, and Beane's finding that some surgical trainees built real skill through intensive video review shows that recorded expertise has value. But a body still has to do the thing, repeatedly, with feedback. No amount of fluent explanation from an assistant will teach an apprentice how a compression fitting feels when it is right. At best, the machine can tell them what to practise.

Collective tacit knowledge is the category where the answer is closest to no. The sense of what counts as a proper job, when to stop and ask, which risks the trade tolerates and which it never does: this is the property of a community, and it passes by socialisation. Orr's copier technicians illustrate it. Their diagnosis came from a shared stock of stories that belonged to the group rather than to either man, and it was the group's norms, including the loss of face that came with simply swapping out the machine, that pushed them to keep going. Brown and Duguid made the point that the training and documentation were about maintaining machines, while much of the real work was about maintaining the social setting. An AI can summarise the stories. It cannot make an apprentice a member of the community that tells them.

Designing for the Mentor's Absence

None of this means the AI-equipped apprentice is doomed. It means the assistant has to be designed and deployed as part of an apprenticeship, not as a replacement for one.

The evidence points to some fairly concrete principles. The first concerns how the tool is used. In Anthropic's trial, the developers who retained the most were the ones who used AI to build understanding, and the highest-scoring group asked only conceptual questions and then relied on their own improved understanding to do the work. At the WorldSkills conference in Shanghai, Pearson's chief technology officer, David Treat, made the same point from the other side: “If you are using AI as an answer engine, you're not engaging the brain in the struggle that learning requires.” An assistant that makes the apprentice explain why a reading looks wrong before offering its own view is a different educational instrument from one that simply supplies the fix.

The second concerns mentors. Pearson's own recommendations call for expanding opportunities to develop practical know-how through mentoring, observation, feedback, simulation and hands-on experience. That sounds obvious, but it runs against the economic logic of the tools themselves, which are sold on making experts more productive and novices less dependent on them. If an assistant frees up a senior engineer's time, some of that time has to be deliberately spent with the apprentice, or Beane's operating-theatre pattern repeats itself on the factory floor. Bainbridge's final irony applies directly: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.”

The third is the unglamorous one. Apprentices need structured practice without the machine, which is what the endoscopy study implies clinicians now need too. The point is not nostalgia for doing things the hard way. It is that the ability to notice when the assistant is wrong is built by doing the work without it, and cannot be built any other way.

Go back, one last time, to Orr's copier. The machine gave its technicians more information than they could use, and none of it added up. What saved the job was not better data. It was two people with long memories, a shared language of experience, and a professional pride that would not let them give up. Today's AI assistants are, in a sense, error codes that can talk: fluent, confident, usually right, occasionally and invisibly wrong. The apprentice holding one needs what the Xerox technician had, someone who has seen this before.

The uncomfortable arithmetic of the Pearson report is that for many apprentices, that someone is already retiring. Some of what they know can be written into training systems, and the effort should begin now. The rest will travel the way it always has, from one person to another, beside a broken machine, or it will not travel at all.

References and Sources

1. Pearson. “Pearson research warns of a 'triple capability gap' emerging as AI adoption outpaces education and training in skilled occupations.” Press release, PR Newswire, Shanghai, 24 September 2026, launching For Every Future: Preparing Early-Career Skilled Trade, Technical and Service Talent for an AI-Infused World; also reported by FE News, StockTitan, People Matters and ChannelLife, 24 to 25 September 2026. Accessed 25 September 2026.
2. Pearson plc. “Pearson Appoints Experienced Company Executive Sharon Hague as President of English Language Learning.” Press release, 28 February 2025.
3. WorldSkills Conference 2026. “WorldSkills Shanghai 2026 outlines a human-first approach to AI.” Conference news, National Exhibition and Convention Center, Shanghai, September 2026. Accessed 25 September 2026.
4. Polanyi, Michael. The Tacit Dimension. Doubleday, 1966; reissued University of Chicago Press, 2009.
5. Collins, H. M. “The TEA Set: Tacit Knowledge and Scientific Networks.” Science Studies, volume 4, 1974, pages 165 to 186; and Collins, H. M. and Harrison, R. G. “Building a TEA Laser: The Caprices of Communication.” Social Studies of Science, volume 5, number 4, 1975.
6. Collins, Harry. Tacit and Explicit Knowledge. University of Chicago Press, 2010.
7. Collins, Harry and Evans, Robert. Rethinking Expertise. University of Chicago Press, 2007.
8. Collins, Harry. “Why artificial intelligence needs sociology of knowledge: parts I and II.” AI and Society, volume 40, 2025 (published online 18 May 2024), pages 1249 to 1263. DOI: 10.1007/s00146-024-01954-8.
9. Sennett, Richard. The Craftsman. Yale University Press and Allen Lane, 2008; extract published as “The Troubled Craftsman”, Arquitectura Viva.
10. Nonaka, Ikujiro. “The Knowledge-Creating Company.” Harvard Business Review, November to December 1991; reprinted July to August 2007.
11. Lave, Jean and Wenger, Etienne. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, 1991.
12. Orr, Julian E. Talking About Machines: An Ethnography of a Modern Job. Cornell University Press, 1996.
13. Brown, John Seely and Duguid, Paul. “Organizational Learning and Communities-of-Practice: Toward a Unified View of Working, Learning, and Innovation.” Organization Science, volume 2, number 1, 1991, pages 40 to 57.
14. Bainbridge, Lisanne. “Ironies of Automation.” Automatica, volume 19, number 6, 1983, pages 775 to 779.
15. Parasuraman, Raja and Manzey, Dietrich H. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” Human Factors, volume 52, number 3, 2010. DOI: 10.1177/0018720810376055.
16. Budzyń, Krzysztof et al. “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study.” The Lancet Gastroenterology and Hepatology, August 2025. PubMed ID 40816301.
17. Anthropic. “How AI assistance impacts the formation of coding skills.” Anthropic Research, 29 January 2026.
18. Brynjolfsson, Erik, Li, Danielle and Raymond, Lindsey. “Generative AI at Work.” The Quarterly Journal of Economics, volume 140, number 2, 2025, pages 889 to 942.
19. Dell'Acqua, Fabrizio, McFowland III, Edward, Mollick, Ethan, Lifshitz-Assaf, Hila, Kellogg, Katherine C., Rajendran, Saran, Krayer, Lisa, Candelon, François and Lakhani, Karim R. “Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality.” Harvard Business School Working Paper 24-013, September 2023.
20. Beane, Matt. The Skill Code: How to Save Human Ability in an Age of Intelligent Machines. Harper Business, 11 June 2024; and University of California, Santa Barbara, “How to build and protect skills in our modern workplace, a world filled with AI and robots”, The Current, 2024.
21. US Government Accountability Office. “Nuclear Weapons: NNSA and DOD Need to More Effectively Manage the Stockpile Life Extension Program.” GAO-09-385, 2 March 2009; and The War Zone, “Fogbank Is Mysterious Material Used In Nukes That's So Secret Nobody Can Say What It Is”, citing Los Alamos National Laboratory's Nuclear Weapons Journal, 2009.
22. Siemens AG. “Siemens introduces AI agents for industrial automation.” Press release, May 2025; and Contractor Magazine, “ServiceTitan Brings AI Automation to the Trades With Atlas”, 19 September 2025.
23. US Bureau of Labor Statistics. “Plumbers, Pipefitters, and Steamfitters.” Occupational Outlook Handbook, accessed 25 September 2026; and Skills England, “Plumbing and domestic heating technician” apprenticeship standard ST0303, Level 3.
24. Department for Education. “Apprenticeships, Academic year 2024/25.” Explore Education Statistics, 27 November 2025; and House of Commons Library, “Apprenticeship starts: has there been a big fall?”
25. Construction Industry Training Board. “CITB publishes Construction Workforce Outlook 2026-2030.” 17 June 2026, with figures reported by Construction News and Building; and Deloitte and The Manufacturing Institute, “US Manufacturing Could Need as Many as 3.8 Million New Employees by 2033”, April 2024.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

For as long as automation has had a sales pitch, the pitch has been the same. The machines will take the drudgery. They will fill in the forms, file the reports, chase the invoices and reconcile the spreadsheets, and in doing so they will hand back to human beings the parts of work that are interesting, creative and their own. Strategy, design, ideas. The good stuff.

A paper presented at the 2026 CHI Conference on Human Factors in Computing Systems, held in Barcelona from 13 to 17 April, suggests that the pitch may have the order of events backwards. Its title asks the question bluntly: “Are We Automating the Joy Out of Work? Designing AI to Augment Work, Not Meaning”. Its authors are Jaspreet Ranjit and Swabha Swayamdipta of the University of Southern California, Ke Zhou of Nokia Bell Labs and the University of Nottingham, and Daniele Quercia of Nokia Bell Labs and the Politecnico di Torino. Their central finding, in the words of the abstract, is this: “Worryingly, we find that tasks that workers associate with a sense of agency or happiness may be disproportionately exposed to AI.”

Not the tedious tasks. The ones people like.

The authors go further. A central risk of AI, they argue, is not mass unemployment but what they call mass demoralisation: “a loss of meaning and ownership in day-to-day work”. As models generate more of the early-stage output, they write, the visible creative steps can start to feel machine-made, while the human contribution becomes rushed, under-resourced and difficult to see.

That is a different kind of warning from the one that has dominated the debate about AI and jobs. It is not about whether people will have work. It is about what work will be like for the people who keep it. And it raises two questions that most organisations deploying AI have not yet seriously asked. What happens to a person's relationship with their craft when the parts of the job they found most fulfilling are the first to be handed to a machine, and what is left is the emotional and relational upkeep? And if the danger is demoralisation rather than displacement, who should decide which parts of a job get automated?

How the researchers mapped meaning onto ten thousand tasks

The study starts with the US Department of Labor's Occupational Information Network, known as O*NET, which breaks every American occupation into short task statements. The researchers narrowed this universe to tasks performed mainly on a computer, carried out daily or weekly, and ones that workers on the research platform Prolific were highly familiar with. That produced a representative sample of 171 tasks across 22 occupations.

They then surveyed two groups. The first was 202 workers who actually performed those tasks. Each rated their tasks against dimensions drawn from a scoping review of research on meaningful work: whether a task felt pointless or valuable, whether it called for novel ideas, gave them freedom or made them happy, and whether it built relationships or required emotional awareness or in-person contact. The second group was 197 developers who build AI systems, who were asked what traits they would design into an AI tool for each task.

To scale the analysis beyond 171 tasks, the team prompted language models to simulate the ratings, checked how closely the simulated ratings matched the human ones, and, finding reasonable agreement, generated annotations for 10,131 tasks across 512 occupations and 19 sectors. They report that only 0.87 per cent of tasks showed a large gap, of two points or more on a five-point scale, between model and human ratings, and that their main patterns held when they reran the analysis using only the human ratings.

To decide which tasks counted as “exposed” to AI, the researchers used a patent-based measure developed by Ali Akbar Septiandri, Marios Constantinides and Quercia, published in PNAS Nexus in 2024, which matched descriptions of tens of thousands of AI patents filed with the US Patent and Trademark Office against O*NET task statements. Tasks above a threshold on that measure were classed as likely to be exposed. Restricting the analysis to computer-based work left 3,179 tasks in the likely-to-be-exposed group and 2,349 in the comparison group.

The parts of the job that machines reach first

The comparison between those two groups is where the paper becomes uncomfortable. Tasks likely to be exposed to AI were more strongly associated by workers with novelty and creativity, with positive feelings and with autonomy. The researchers say directly that this “challenges the usual narrative that automation mainly targets routine tasks”. Tasks less likely to be exposed were associated with emotional awareness, in-person interaction, relationship building and social connection.

The sector picture sharpens the point. The heaviest exposure of creative, high-agency work fell in Arts, Architecture and Engineering, Computer and Mathematics, and the Life, Physical and Social Sciences. The examples the paper draws from O*NET are telling: an art director “formulating basic layout design”; an actuary “constructing probability tables for natural disasters”. These are first passes, the authors note, of the kind that models can now credibly generate, with humans refining the output afterwards. Meanwhile, in Education, Sales, Community and Social Services and Healthcare, the work that remained less exposed involved activities such as guiding students through tangled academic and personal problems, or “presenting offers while preserving relationships”.

The authors read this as a reversal of the expected trajectory. Generative systems, they observe, already draft text, suggest layouts, start campaigns, run simulations and infer likely emotional responses. In their data, these uses line up with tasks that workers describe as meaningful precisely because they reflect taste and authorship. AI is not confined to chores. It is becoming entangled with the moments where people feel they add something of themselves.

There is a second finding that matters for anyone building the tools. Workers and developers disagreed about what an AI system should be like. Workers wanted systems that were straightforward, tolerant and practical. Developers said they would design systems that were polite, strict and imaginative. The single biggest gap was between straightforwardness and politeness, and it was widest in Production, Architecture and Engineering, and the sciences, where vagueness can mean wasted materials or flawed data. “In high-risk contexts,” the authors write, “the ethical choice may well be the blunt one, and the useful thing to say may be short.” Developers, they suggest, tend to romanticise creativity as a frontier capability, while many workers in structured, routine jobs simply want a tool that spots anomalies and sticks to the rules.

Put the two findings together and the problem comes into focus. The AI that is arriving is being built to be imaginative, and it is landing on the imaginative parts of the job.

Why exposure is not the same as replacement

It is worth being precise about what the study measures. “Exposure” does not mean a task has been automated, or even that it will be. The authors define it as tasks that current or near-term AI could plausibly perform or substantially speed up. It is a statement about technological reach, not about deployment decisions inside any particular firm, and the paper separates automation, where AI performs a task end to end, from augmentation, where humans keep primary responsibility.

The researchers are also candid about their limitations. The study covers only US occupational tasks. Prolific participants tend to be more technologically literate than average, so low-wage and non-digital work may be underrepresented. O*NET task statements are short summaries that different workers may picture differently. The scaling to 10,131 tasks relies on language-model ratings, which the authors treat as an approximation of human responses rather than as ground truth, and which they found to be more uniform and optimistic in some sectors. Even the creativity finding has a wrinkle. When the analysis was rerun on human ratings of the original 171 tasks alone, humans and models agreed on six of the seven significant dimensions; the exception was “requires novel ideas or creativity”, where the authors suggest human annotators tended to underestimate the novelty of tasks.

None of this undoes the core pattern. But the study is best read as a map of where pressure is likely to fall, not a ledger of what has already been lost. What actually happens to people's work is decided later, by managers, vendors and product teams.

From mass unemployment to mass demoralisation

Why should the location of the pressure matter so much? Because meaning in work is not spread evenly across a job. It clusters.

The sociologist Richard Sennett, in his 2008 book The Craftsman, described craftsmanship as the basic human impulse to do a job well for its own sake, and argued that this impulse extends far beyond the workshop, to programmers, doctors and artists. Craft, in that sense, lives in the moments of making: the first sketch, the opening paragraph, the elegant solution to a modelling problem.

If those moments are exactly where AI exposure is concentrated, then a job can be automated in a peculiar way. The worker keeps the job title, the salary and most of the hours. What disappears is the part that made them want the job in the first place. In the paper's own examples, the basic layout and the first probability table are precisely the drafts a model can now produce; what is left to the human is refinement. This is the scenario Ranjit and her colleagues have in mind when they describe a loss of ownership. The creative steps that others can see now appear to come from the machine, while the human's contribution moves into the background, squeezed for time and harder for anyone to credit.

There is already experimental evidence that this dynamic has psychological costs. In a 2025 paper in Scientific Reports, Suqing Wu, Yukun Liu, Mengqi Ruan, Siyu Chen and Xiao-Yun Xie of the School of Management at Zhejiang University reported four online experiments with a total of 3,562 participants. Working with generative AI improved immediate performance on professional tasks. But that performance boost did not carry over when people subsequently worked alone, and the switch from AI collaboration to solo work was accompanied by significant decreases in intrinsic motivation and increases in boredom, even as participants' sense of control rose. The authors' summary is succinct: collaboration with generative AI “enhances immediate task performance but can undermine long-term psychological experiences of human workers.”

Creative workers are already reporting effects outside the laboratory. A survey by the Society of Authors, the UK trade union for writers, illustrators and translators, received 787 responses in early 2024. A quarter of illustrators, 26 per cent, and more than a third of translators, 36 per cent, said they had already lost work to generative AI. The survey was self-reported and drawn from a membership sample, so it cannot measure the market as a whole. But it captures the sense, among people whose identities are bound up with a craft, that the craft itself is being lifted out of their hands.

What the remaining work asks of people

If the creative core of a job is thinned out, what is left? According to the CHI study, the tasks least exposed to AI are those built on emotional awareness, in-person presence and relationship building. That might sound like a reassuring residue. Machines draft, humans connect. But it is worth looking at what those remaining tasks involve.

In 1983, the sociologist Arlie Russell Hochschild of the University of California, Berkeley, published The Managed Heart: Commercialization of Human Feeling, which introduced the idea of emotional labour. Studying flight attendants and bill collectors, she described public-facing jobs in which managing feelings was part of what the employer was paying for. Flight attendants had to be, in the phrase used by her publisher, “nicer than natural”; bill collectors had to be “nastier than natural”. Hochschild's concern was that this kind of work, done for commercial purposes, could estrange people from their own feelings.

Relational work is not inherently draining. For many teachers, nurses, social workers and salespeople, the relationships are the point of the job, and the CHI data suggest that workers do regard these tasks as meaningful. The danger lies in the mix. A job that once combined making things with tending to people can become one that is mostly tending to people, with the making delegated. The remaining human work then has a distinctive profile: smoothing over the AI's output with clients, absorbing complaints, reassuring colleagues, maintaining relationships that the automated parts of the workflow no longer support. That is emotional labour, and much of it is invisible in the way that a finished design or a well-built model is not.

The Stanford University researchers behind the WORKBank database, Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson and Diyi Yang, found a related signal in a 2025 study of 1,500 workers across 844 tasks in 104 occupations. Their results, they wrote, offer early signals that AI agents may reshape core human competencies, “shifting from information-focused skills to interpersonal ones.” The CHI paper adds a warning that this shift may not be benign. Among its design examples is the risk that teachers, managers and clinicians who use AI for canned replies and templated feedback end up making their communication with students, staff or patients feel more generic. If AI pushes people towards relational work while also hollowing out that relational work with templates, the worker is squeezed from both sides.

When the most fulfilling parts of a job are automated first, then, a person's relationship with their craft is put at risk in two ways. They lose authorship of the work that once expressed their skill and taste. And they are left with a larger share of work whose value is real but less visible and more emotionally costly. Whether that produces demoralisation depends on what else changes, and here a very different body of evidence becomes useful.

Lessons from people who stopped working

In September 2026, four researchers at Google DeepMind, Stephanie C.Y. Chan, Adam Bales, Katherine L. Hermann and Iason Gabriel, posted a review titled “Work, Wellbeing, and Choice: Empirical Lessons for AI Futures”. Their question was not what AI does to work, but what decades of research reveal about people who live without it. They drew on psychological, sociological and economic studies of the unemployed, retirees, lottery winners, financially dependent spouses and citizens of Gulf states with guaranteed public-sector employment, covering OECD countries, China, India and the Gulf.

Their findings are stark in places. Involuntary job loss harms life satisfaction, mental health, physical health and even mortality, and the damage can last: some studies find psychological “scarring” that persists for years after people find work again. Yet the vast majority of lottery winners, between 85 and 89 per cent across studies of American and Swedish winners, keep working, typically using their windfall to buy flexibility rather than to quit. And retirees, who are also not working, often report high and sustained wellbeing.

The authors draw on the work of the social psychologist Marie Jahoda, whose 1982 book Employment and Unemployment, published by Cambridge University Press, argued that a job provides “latent” benefits alongside its “manifest” purpose of earning a living. Jahoda identified time structure, social contact, status and identity, regular activity and collective purpose. The DeepMind review finds that both income and some of these latent benefits matter, and adds cognitive health: retirement has been causally linked to declines in cognitive ability.

To explain why the unemployed and the retired fare so differently, the review identifies three factors that mediate the relationship between work status and wellbeing. The first is agency and choice: whether the exit from work was voluntary, and whether people retain agency over the long term. The second is the availability of alternative sources of work's latent benefits, such as volunteering, hobbies or state-provided employment. The third is social and systemic context, including cultural norms around work and the strength of safety nets.

The evidence on each is striking. Between 20 and 30 per cent of retirees perceive their retirement as forced, and across countries those who retired voluntarily report significantly better physical and mental wellbeing than those who did not, even after controlling for health, income and other factors. Volunteering confers benefits at remarkably low doses, with gains for older adults observable from about one hour a week. One study cited in the review found that plant closures produced an almost four-fold greater increase in depression among American workers than among European workers, who have stronger safety nets. And the psychological cost of losing work tends to fall harder on men, which the authors attribute to stronger social pressure to conform to norms about working.

“Many policy discussions about AI futures focus on the availability of work or the availability of resources,” the authors write. “However, these discussions overlook the fact that it matters deeply whether people get to choose their path.” Their summary of the policy lesson is equally direct: purely material interventions such as universal basic income “are valuable but highly insufficient”.

Applying the exit evidence inside the job

The DeepMind review is about people who leave work altogether. But its three factors translate remarkably well to what happens when people stay in work and lose part of it. Demoralisation, in the CHI paper's sense, is a kind of partial exit: a withdrawal not from employment but from the meaningful core of a role. Read that way, the review's factors become a diagnostic for when task automation is likely to hurt.

Start with agency. A critical determinant of wellbeing in the review is not whether people work, but whether they chose their situation. Inside a job, the equivalent question is whether workers chose which of their tasks were handed to AI. The Society of Authors survey hints at the distinction: alongside those who chose to use generative AI, 5 per cent of illustrators and 8 per cent of translators said they had used it because their publisher or commissioning organisation asked them to. The CHI paper notes that, in practice, managers, vendors and technical teams, not workers, usually decide which tasks are exposed to AI and how far automation goes. “Managers, vendors, and technical teams make these decisions before workers can consent or take part, especially in roles with low bargaining power,” the authors write. That is the workplace version of involuntary exit.

Next, alternatives. Retirees do well when they find other sources of structure, status and purpose. Workers who lose the creative core of their job need somewhere else for those things to come from: new responsibilities with real authorship, protected time for the work they value, recognition for the judgement involved in steering and correcting AI output. If the time freed by automation is simply filled with more volume, the latent benefits drain away without replacement.

Finally, context and safety nets. The review shows that the same loss lands very differently depending on norms and institutional support. Inside an organisation, the analogue is culture and protection: whether people who push back on a deployment are heard or penalised, whether those whose roles shrink are offered retraining or redeployment, and whether the organisation measures anything other than throughput.

This is an analogy, not a finding. Neither paper tested it, and evidence about the unemployed cannot be transplanted wholesale onto people still in work. But the convergence is hard to ignore. Working from inside jobs, one paper finds that agency, ownership and meaning matter; working from outside, the other finds that choice, alternatives and context matter. Both suggest the damage depends less on what is taken away than on how.

Workers already say what they want automated

The good news is that workers are not, in general, opposed to automation. They are opposed to the wrong automation.

The WORKBank study asked its 1,500 workers which of their tasks they would like AI agents to take on. For 46.1 per cent of tasks, workers expressed positive attitudes towards automation, even after being prompted to reflect on job loss and on how much they enjoyed the work. Their main motivation was to free up time for higher-value work. The researchers then compared those desires with experts' assessments of what AI could currently do, sorting tasks into four zones: a “Green Light” zone where workers wanted automation and the technology could deliver it; a “Red Light” zone where the technology was capable but workers did not want it; an R&D opportunity zone where workers wanted help that the technology could not yet provide; and a low-priority zone.

Their most pointed finding concerned where the money was going. Mapping the tasks targeted by Y Combinator start-ups, they found that 41.0 per cent of company-task mappings were concentrated in the Low Priority and “Red Light” zones. A large share of the industry's effort, in other words, was aimed at tasks workers either did not care about or actively wanted to keep.

Combined with the CHI paper's developer-worker mismatch, the pattern is clear. Workers do not resist AI. Their preferences are simply rarely the input that shapes what gets built.

A test for choosing which task clusters to automate

If the risk is demoralisation, then the usual test for automation, which asks whether a task can be done faster or cheaper by a machine, is incomplete. It needs a second question: what does this task mean to the people who do it? Drawing on the two papers and the evidence around them, an organisation could reasonably weigh at least four considerations before automating a cluster of tasks.

The first is meaning. Ask the people who do the work which tasks they find meaningful, enjoyable or expressive of their skill, and which they find pointless. The CHI study shows that this can be measured at scale with straightforward survey instruments. WORKBank shows that workers can distinguish clearly between tasks they want help with and tasks they want to keep. Tasks that workers rate low on meaning and high on tedium are the obvious first candidates. Tasks rated high on creativity, autonomy and happiness deserve a heavier burden of justification.

The second is order. The CHI authors propose that, for creative work, the sequence matters as much as the division of labour. Their design heuristics include starting creative fields empty, so that a human sketch, prompt or outline exists before AI suggestions appear; offering several alternatives rather than a single finished replacement; letting workers accept or reject AI content sentence by sentence rather than through a single “replace all”; and allowing a worker to tick a box marked “I want to do this part myself”. The question for an organisation is not simply whether AI touches a creative task, but whether the human still makes the first move.

The third is what replaces the time saved. The DeepMind review's evidence on alternative sources of meaning suggests that automation which frees time is only beneficial if that time goes somewhere that sustains structure, status and purpose. For relational work, the CHI paper suggests using AI to prepare briefs and summaries so the human arrives better informed, assigning it to silent roles such as note-taking during live interactions, and reserving protected time for direct human engagement. Its proposed measures of success are telling: workers reporting that AI helps them “show up prepared” rather than “speak for me”, and that it does not “take away the good parts” of the job.

The fourth is measurement. The CHI authors recommend tracking, during deployment, whether workers still feel in control and whether the output “still feels like my work”, alongside conventional measures of speed and quality. They also suggest logging the level of assistance AI provides, from suggestion to draft to execution, and recording who made the final decision. Over time, they argue, workers' self-reported skills and confidence in their core craft should remain stable or improve. If those indicators fall while productivity rises, the organisation has learned something its dashboards would otherwise hide.

None of this means creative tasks should never be automated. Some workers will welcome help with the very tasks others prize, and the CHI data show wide variation between sectors. The point is that meaning should be treated as a real cost in the decision, not as a sentimental afterthought.

Giving workers a binding say

That leaves the second half of the question: what voice should workers have? The CHI paper's answer is explicit. Beyond designing interfaces that let individuals retain control within a task, the authors argue that organisations should give workers some say over which task clusters are candidates for automation. Without such mechanisms, they warn, “AI exposure risks deepening existing power imbalances.” They point to remedies from research on algorithmic management: direct involvement of workers and their representatives through formal consultation and collective bargaining over AI deployments, worker or union technology representatives with access to information about how systems allocate and evaluate work, and transparency tools that help workers understand and contest automated decisions.

There are working precedents. The most prominent came from Hollywood. The 2023 Minimum Basic Agreement between the Writers Guild of America and the major studios established that AI cannot be considered a writer and that material it produces cannot be treated as literary material. Under its terms, a writer may choose to use AI with the company's consent, but the guild's own summary states plainly that “the company can't require the writer to use AI software (e.g., ChatGPT) when performing writing services.” Companies must disclose when material given to a writer was generated by AI. In effect, the writers bargained over which parts of their craft would remain theirs. That agreement has since been succeeded by the 2026 contract, ratified on 24 April 2026 with 90.38% of votes in favour and running to 1 May 2030, which “preserves all of the protections negotiated in 2023”. It adds one more: companies must give the guild written notice when they license writers' work to train a commercial generative AI system, after which the guild can request a discussion about the licence, including any payment to writers. It does not establish payment for training.

European law offers a thinner but broader floor. Article 26(7) of the EU Artificial Intelligence Act, adopted in 2024, requires that, before a high-risk AI system is put into service or used at the workplace, “deployers who are employers shall inform workers' representatives and the affected workers” that they will be subject to it. That is a duty to inform, not to consult or obtain consent, and it applies only to systems classed as high-risk. It is also not yet in force for most workplace tools: the Digital Omnibus on AI, Regulation (EU) 2026/1744, which took effect on 27 July 2026, deferred the high-risk obligations for Annex III systems, a category that includes employment and worker management, from 2 August 2026 to 2 December 2027. In Germany, works councils have long held co-determination rights over the introduction of technical systems capable of monitoring employees' behaviour or performance, a provision that lawyers widely expect to cover many AI tools. The limits are visible too: in January 2024, the Hamburg Labour Court rejected a works council's application concerning employees' use of ChatGPT, partly because staff used it voluntarily through private accounts in a web browser rather than as an employer-installed system.

These mechanisms share a weakness when it comes to demoralisation. Most were designed to address surveillance, pay or credit, not meaning. The EU provision is triggered by risk classification, not by what a task means to the person who does it. The German right is anchored in monitoring. Even the WGA's protections, including the 2026 notice requirement on training licences, focus on credit and compensation, although by keeping AI out of the definition of a writer, they also protect authorship. A workplace that took the CHI findings seriously would go further: it would give workers a formal role in deciding which task clusters are automated, informed by evidence about which tasks they find meaningful, with a right to revisit the decision once the effects are known.

That voice matters most precisely where it is weakest. The CHI authors note that the decisions about exposure are made before workers can consent, especially in roles with little bargaining power. The DeepMind review cites evidence that, where safety nets are weak, pre-existing wealth becomes a key buffer against the shock of job loss. A model of AI deployment that consults only those with strong unions would protect screenwriters and leave behind most of the people whose tasks appear in the CHI data.

What a craft needs to survive

It is tempting to treat all of this as a soft concern, a matter of morale to be fixed with a workshop. The evidence suggests otherwise. Meaning is not a perk attached to work; it is part of what makes people willing to invest skill and care, and part of what keeps them well. The DeepMind review shows how much is at stake when people lose work altogether. The CHI paper suggests a quieter version of that loss may already be underway inside jobs that look perfectly secure.

The two papers point to the same remedy from opposite directions. What protects people is not the absence of change but the ability to shape it: to choose, to find meaning somewhere else when the old sources move, and to be supported while they do. For organisations, that means treating which tasks to automate as a decision with human costs, made with the people who bear them, rather than a technical question settled by whatever a vendor's model can do.

Ranjit and her co-authors end on a note of possibility. The outcome they describe, they stress, is shaped by design choices, which can in principle be revised: systems could speed up exploration and drafting while people keep authorship, make the final judgements and sustain the relationships that make work meaningful. That is a more humane division of labour than the current pitch implies. But it will not arrive by default. Left to the logic of capability alone, AI will keep flowing towards the tasks it can reach, and those turn out to be the ones people love. Keeping the spark in work will take a deliberate decision about who gets to say which parts of a craft stay human.

References and Sources

1. Ranjit, Jaspreet; Zhou, Ke; Swayamdipta, Swabha; Quercia, Daniele. “Are We Automating the Joy Out of Work? Designing AI to Augment Work, Not Meaning.” Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, Spain, 13-17 April 2026. DOI: 10.1145/3772318.3791845. Preprint arXiv:2603.14963, 16 March 2026. https://arxiv.org/abs/2603.14963 (full text: https://arxiv.org/html/2603.14963)
2. Chan, Stephanie C.Y.; Bales, Adam; Hermann, Katherine L.; Gabriel, Iason. “Work, Wellbeing, and Choice: Empirical Lessons for AI Futures.” Google DeepMind. arXiv:2609.11019, 10 September 2026. https://arxiv.org/abs/2609.11019 (full text: https://arxiv.org/html/2609.11019)
3. Septiandri, Ali Akbar; Constantinides, Marios; Quercia, Daniele. “The potential impact of AI innovations on US occupations.” PNAS Nexus, 3(9), pgae320, September 2024. DOI: 10.1093/pnasnexus/pgae320. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11421150/
4. Wu, Suqing; Liu, Yukun; Ruan, Mengqi; Chen, Siyu; Xie, Xiao-Yun. “Human-generative AI collaboration enhances task performance but undermines human's intrinsic motivation.” Scientific Reports, 15, article 15105, 29 April 2025. DOI: 10.1038/s41598-025-98385-2. https://www.nature.com/articles/s41598-025-98385-2
5. Shao, Yijia; Zope, Humishka; Jiang, Yucheng; Pei, Jiaxin; Nguyen, David; Brynjolfsson, Erik; Yang, Diyi. “Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce.” arXiv:2506.06576, 6 June 2025 (revised 1 February 2026). https://arxiv.org/abs/2506.06576
6. Hochschild, Arlie Russell. The Managed Heart: Commercialization of Human Feeling. University of California Press, 1983 (third edition 2012). https://www.ucpress.edu/books/the-managed-heart/paper
7. Jahoda, Marie. Employment and Unemployment: A Social-Psychological Analysis. Cambridge University Press, 1982. ISBN 978-0-521-28586-5. https://ideas.repec.org/b/cup/cbooks/9780521285865.html
8. Sennett, Richard. The Craftsman. Yale University Press, 2008. https://yalebooks.yale.edu/book/9780300151190/the-craftsman/
9. The Bookseller. “A third of translators report losing work to generative AI systems, SoA survey reveals.” April 2024. https://thebookseller.com/news/a-third-of-translators-report-losing-work-to-generative-ai-systems-soa-survey-reveals
10. European Writers' Council. “SoA survey UK AI 2024.” April 2024. https://europeanwriterscouncil.eu/soa-survey-uk-ai-2024/
11. Books+Publishing. “A third of translators report losing work due to AI.” 15 April 2024. https://www.booksandpublishing.com.au/articles/2024/04/15/250317/a-third-of-translators-report-losing-work-due-to-ai/
12. Writers Guild of America West. “Know Your Rights: Artificial Intelligence.” https://www.wga.org/contracts/know-your-rights/artificial-intelligence
13. Writers Guild of America. “Summary of the 2023 WGA MBA.” https://wgacontract2023.org/the-campaign/summary-of-the-2023-wga-mba
14. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 26: Obligations of deployers of high-risk AI systems. https://artificialintelligenceact.eu/article/26/ (Official Journal: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng)
15. Littler Mendelson. “Germany: Does the Works Council Have a Say When Employees Use ChatGPT? No, According to the Labor Court Hamburg.” 2024. https://www.littler.com/publication-press/publication/germany-does-works-council-have-say-when-employees-use-chatgpt-no
16. Bird & Bird. “First Judgement on the Rights of Works Councils when Employees use AI Systems.” 2024. https://cm.twobirds.com/en/insights/2024/germany/erstes-urteil-zu-rechten-des-betriebsrats-bei-einsatz-von-kuenstlicher-intelligenz
17. Writers Guild of America. “WGA Ratifies 2026 Minimum Basic Agreement With AMPTP.” 24 April 2026. https://www.wga.org/news-events/news/press/wga-ratifies-2026-minimum-basic-agreement-with-amptp
18. Writers Guild of America. “Summary of the 2026 WGA MBA.” https://www.wga.org/contracts/contracts/mba/summary-of-the-2026-wga-mba
19. Regulation (EU) 2026/1744 amending Regulation (EU) 2024/1689 (Digital Omnibus on AI), in force 27 July 2026. https://artificialintelligenceact.eu/ai-act-explorer/digital-omnibus/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

On a bright afternoon at a primary school in Hertfordshire, a headteacher did something quietly radical. She cancelled the afternoon's lessons, opened the back doors, and told her pupils to go outside and play. What happened next has since been recounted to audiences of thousands as a small parable of a large problem. The children did not scatter into games. They stood in the yard, uncertain what to do with an unstructured hour and an open sky. They could not readily invent their own entertainment. Within minutes, some of them asked to come back inside.

The actress and campaigner Sophie Winkleman told this story at ARC 2026, the London conference convened by the Alliance for Responsible Citizenship, in a talk provocatively titled “Why AI in the Classroom is a Catastrophe.” The talk circulated widely through the summer. Winkleman used the empty playground as evidence for a claim easy to state and hard to prove: that a childhood saturated in constant, low-level digital stimulation had eroded the very faculty a playground is meant to exercise, the ability to generate one's own amusement out of nothing but time, space and other children.

You can take the anecdote as a moral panic dressed in a good story, or as a genuine warning. Either way, it points at something specific and testable, and it is worth being precise about what that is. The worry is not simply that children spend too long looking at screens, a complaint as old as television. The worry is that a particular category of software design, refined for a decade inside the attention economy and increasingly restricted for use on children in that setting, has migrated into the tools schools use to teach reading, mathematics and languages. The mechanics that regulators have started to treat as harmful when a social platform deploys them to hold a child's gaze are frequently the same mechanics an educational technology company deploys to keep a child inside a maths app. And nobody, at the point these products are bought and installed at scale, is being asked to prove that the second use is meaningfully different from the first.

That is the question worth sitting with. If the engagement machinery that is driving measurable harm in children's social media use is structurally identical to the machinery now embedded in the software schools use to educate, what obligation do schools, regulators and the companies themselves have to distinguish a product designed to teach from a product designed merely to hold attention? And crucially, before these tools reach millions of children, who should carry the burden of proving the difference?

The mechanics that migrated from the casino to the phone

To understand why an educational app and a slot machine can share a family resemblance, you have to understand the small number of design primitives that make modern consumer software compulsive. There are essentially three, and none of them is new. What is new is their precision.

The first is the variable, or unpredictable, reward. The principle comes from operant conditioning research conducted in the mid-twentieth century, which established that a behaviour is reinforced most powerfully not when it is rewarded every time but when it is rewarded unpredictably. A pigeon that receives a food pellet on a random schedule pecks more persistently than one rewarded on a fixed schedule. A slot machine works on exactly this logic, and so does the pull-to-refresh gesture on a social feed, which delivers something interesting often enough, and unpredictably enough, to keep the thumb moving. The design writer and former Google ethicist Tristan Harris popularised the comparison years ago when he described the smartphone as a slot machine in every pocket. Even the market-leading language app Duolingo has been analysed, by designers largely sympathetic to it, as deploying uncertainty deliberately, so that on some questions a user loses a heart and on others does not, an ambiguity described in these analyses as mildly activating in the same way a slot machine is.

The second primitive is the streak, a running count of consecutive days of use that converts a habit into an obligation. The streak weaponises two well-documented quirks of human psychology, loss aversion and the sunk-cost fallacy. Once a user has built a streak of two hundred days, the prospect of losing it looms far larger than any positive reward, and the daily return becomes a defensive manoeuvre against loss rather than a pursuit of benefit. Duolingo's own reported figures make the mechanic's power plain: users who reach a seven-day streak are several times more likely to remain engaged, and the company built an entire secondary feature, the streak freeze, specifically to relieve the anxiety of at-risk users and reduce the rate at which they quit. Designers analysing the app have been candid that its success rests, in their telling, on design rather than on the superiority of its teaching, and that a large share of its revenue is organised around a single streak counter.

The third primitive is the notification, the interruption engineered to pull a user back into an application they had, however briefly, escaped. Notifications are tuned relentlessly. The phrasing, timing and frequency of a prompt are subjected to continuous experimentation, so that a change to a single word or a shift of an hour can be tested on a small cohort and rolled out to everyone if it lifts the numbers. This is not a caricature of how these products are built. It is how they are built, and it is described in these terms in the trade literature that celebrates them.

Set out plainly, the three mechanics form a loop: an unpredictable reward creates the pull, a streak raises the cost of leaving, and a notification supplies the nudge back in. What that loop optimises for is time on the product. That, and not any richer human outcome, is the thing being maximised.

Regulators have noticed. The United Kingdom's Age Appropriate Design Code, the statutory Children's Code enforced by the Information Commissioner's Office and in force since 2021, requires online services likely to be accessed by under-eighteens to avoid using nudge techniques that lead children into choices against their own interests, and asks providers instead to nudge towards wellbeing, including screen breaks. California modelled its own Age-Appropriate Design Code on the British one. The statute asked businesses to assess whether their products used system design features to increase, sustain or extend children's use, naming among them the automatic playing of media, rewards for time spent, and notifications, and it separately prohibited dark patterns that lead a child to take any action the business knows, or has reason to know, is materially detrimental to the child's physical health, mental health or well-being. Rewards for time spent. Notifications. The regulatory language is describing, feature for feature, the standard toolkit of engagement design, and it is describing it as a risk to be assessed and restrained when aimed at children. Turning that description into an enforceable rule has proved harder. Both provisions have been held up in the courts, and in March 2026 the Ninth Circuit Court of Appeals, in NetChoice v. Bonta, kept the dark-patterns prohibition blocked on the ground that it is likely void for vagueness. A business, the court reasoned, cannot know in advance what “materially detrimental” or “well-being” require of it when the range of possible harms runs from financial exploitation to lost sleep, distraction or hurt feelings. The regulators can name the mechanics. What they have not yet managed, at least in California, is to draw a line around them that a court will accept, and that difficulty matters for where this argument ends up.

How the same design crossed into the maths lesson

Here is the pivot on which the whole argument turns. Those mechanics did not stay on the social platforms. They crossed, wholesale, into educational software, because the companies building learning apps are drawn from the same talent pool, chase the same metrics and answer to the same kind of investor as the companies building everything else on a child's screen.

Walk through the feature list of a typical classroom learning product and you will find the loop reproduced faithfully. There are points and experience bars that fill towards a level. There are badges unlocked at intervals engineered to arrive just often enough to sustain interest. There are daily streaks with their attendant freezes and repair mechanics. There are leaderboards that convert solitary practice into social competition. There are, above all, notifications, and the design blogs that serve the sector discuss the timing and copy of those notifications with the same clinical attention to retention that you would find in any social app. The trade press for the industry openly frames gamification as a lever to boost engagement and retention, and the primary metric it reports is very often the time a child spends inside the product rather than anything the child has learned.

The defence offered for all this is that engagement is a precondition for learning. A bored child learns nothing, the argument runs, so a tool that holds attention creates the opportunity to teach. There is a grain of truth in it. Attention is necessary for learning. But necessary is not sufficient, and the sleight of hand in the industry's framing is to treat a necessary condition as though it were the outcome itself. A child can be intensely, measurably engaged by a maths app and be learning very little maths, in exactly the way a child can be intensely engaged by a slot-machine mechanic and be learning nothing at all. Engagement measures whether the child keeps tapping. It does not measure whether the child understands. These are different quantities, and the entire difficulty is that the first is trivially easy to measure and the second is genuinely hard.

This is the point Will Ellis of the campaign group Reclaim Childhood sharpened, in words Winkleman quoted from the ARC stage, arguing that engagement mechanics appear in learning apps “because they improve engagement, and engagement is what the business model requires,” not because they improve learning. The phrasing is worth dwelling on, because it identifies the exact place where the incentives fork. A subscription-funded or usage-funded educational product has a commercial interest in engagement that is direct, immediate and easy to measure in a dashboard. Its interest in learning is indirect, deferred and hard to measure at all, surfacing, if it surfaces, in a test score months later that no product analytics team will ever see. When two goals compete and only one of them shows up in the metrics that determine the next funding round, it is not cynical to predict which one wins. It is merely arithmetic.

Why engagement became the metric that ate the classroom

The deeper problem is that engagement optimisation and genuine pedagogy can actively pull in opposite directions, so that the better a product performs on the first, the worse it may be positioned to deliver the second. Consider the psychology of the external reward. A well-established body of research on motivation has repeatedly found that layering extrinsic rewards on top of an activity a child already finds interesting can reduce the child's intrinsic interest in that activity. Pay a child in points and badges for something they were happy to do for its own sake, and you risk teaching them that the thing is only worth doing for the points. For an entertainment product with no ambitions beyond retention, that trade is invisible and irrelevant. For an educational product whose entire justification rests on cultivating a durable disposition to read, to reason, to persist at hard problems, it is close to self-defeating.

There is a further mismatch in what the mechanics reward. Genuine learning frequently requires the opposite of a frictionless, dopamine-steady loop. It requires desirable difficulty, the productive struggle of retrieving a fact you have half-forgotten or wrestling with a problem just beyond your current reach. It requires tolerance of the boredom and frustration that precede mastery. It requires sustained attention held on a single hard thing for long enough to make progress, the precise faculty the Hertfordshire playground exposed as atrophied. Engagement design, by contrast, is built to eliminate friction, to keep the experience buoyant and rewarding and forward-moving, to ensure the child never sits with the discomfort long enough to be tempted to leave. The two philosophies are not merely different. At their extremes they are opposed. A product engineered so that a child never wants to stop is, almost by construction, a product engineered to remove the very resistances against which a mind grows stronger.

None of this proves that every gamified learning app harms every child. But it does establish that engagement and learning are separable, that they can conflict, and that the commercial machinery of the sector is tuned to the one that is easier to sell. That is enough to make the central question urgent rather than academic.

What the evidence actually shows about paper and screens

If the theoretical case is that engagement and learning diverge, the empirical case is that, in the aggregate and on average, screens have so far underdelivered on the outcomes that matter. The evidence is not unanimous and it is not always about engagement mechanics specifically, but it is substantial, and it has been accumulating for longer than this year's headlines suggest.

Begin with the medium itself. In a study published in the Oxford Review of Education in 2018, the University College London education researcher John Jerrim, working with John Micklewright and colleagues, took more than three thousand pupils across Germany, Ireland and Sweden and randomly assigned them to sit identical assessment items either on paper or on a computer screen. Randomisation matters here: because pupils were allocated by chance, the two groups were on average equivalent in ability, so any systematic gap in their scores could be attributed to the medium rather than to the children. The paper group scored roughly twenty points higher on the PISA scale, a gap the authors characterised as equivalent to about six months of additional schooling. The finding is part of a broader literature on what researchers call the mode effect, the tendency for the same knowledge to yield different measured performance depending on whether it is tested on paper or on a screen, an effect large enough in some countries to swing international rankings substantially. Jerrim's work is a caution about assessment as much as instruction. But its core implication is uncomfortable for the screen-first classroom: the digital medium was not neutral. It cost pupils something.

Then consider what happens in the brain during the physical act of learning. Audrey van der Meer, a neuroscientist at the Norwegian University of Science and Technology, has spent two decades comparing the neural signatures of handwriting and typing. In a high-density electroencephalography study published in Frontiers in Psychology, her team fitted participants with nets of well over two hundred sensors and recorded their brain activity as they were prompted, repeatedly, either to write a word by hand or to type it. Writing by hand produced markedly more elaborate patterns of connectivity across the brain, with widespread coordination between regions in frequency bands that van der Meer and her collaborators associate with memory formation and the encoding of new information. Typing did not. Her interpretation is that the precisely controlled movements of forming a letter by hand feed a stream of visual and motor information into the brain that helps knit together the connections underlying learning, and that this stream is largely absent when the same word is produced by tapping identical keys. The sample sizes in this kind of laboratory work are small, and connectivity is not the same thing as a test score, so the finding should be held with appropriate care. But it converges with the behavioural evidence rather than contradicting it, and the direction of travel is consistent: the embodied, effortful, analogue act engages more of the learning brain than its frictionless digital substitute.

Recent reviews of the research literature frame the same tension in their own terms. A scoping review by Urška Žnidarič, Erik Štrumbelj and Octavian Machidon, synthesising 937 papers on the negative effects of digital technology on cognition, found that the harms it could identify ran largely through displacement, the crowding out of other activities, rather than through technology being harmful in itself, and that effects were often moderated by socioeconomic and environmental factors. Its organising conclusion is what the authors call an efficiency-atrophy paradox: digital tools can improve short-term performance while potentially weakening long-term, unassisted cognition. That is the engagement-versus-learning problem restated in cognitive terms. A tool can make today's task go better and leave the mind less able to do it alone tomorrow, and a dashboard measuring today's performance will record only the gain. A second review, a systematisation of knowledge by Supriya Khadka and Sanchari Das covering 111 studies of extended reality in the education of children aged three to eight, found that pedagogy received by far the most scholarly attention, while data security practices received almost none, and children's health and accessibility for disabled children were also thinly covered. Even for children as young as three, the research effort has gone overwhelmingly into whether a technology can teach, and comparatively little into what else it does to the child, and to the child's data, while it is teaching.

That deterioration is not merely theoretical either. It is legible in the international data that governments themselves collect. Sweden's national performance on the PISA assessments fell over the last decade, and in 2022 almost a quarter of fifteen and sixteen-year-olds, 24 per cent, did not reach a basic level of reading comprehension. It was against exactly this backdrop that the country began reconsidering the enthusiasm with which it had digitised its classrooms. Correlation is not cause, and the OECD's education director, Andreas Schleicher, has urged caution about attributing cause and effect; reading declines have many parents. But when a country that went further and faster than almost anyone into one-to-one devices sees its literacy soften and responds by buying paper textbooks again, the sequence is worth taking seriously.

The pushback gathers institutional weight

For years the case against screen-saturated schooling was made largely by individual sceptics, and it was easy to wave away as nostalgia. What changed this year is that the argument acquired institutions, budgets and votes.

The most consequential move came in Los Angeles. The board of the Los Angeles Unified School District, the second-largest public school system in the United States, voted unanimously to create and enforce district-wide limits on classroom screen time, the most sweeping such reduction adopted by any major American district, and the policy took effect when the 2026-27 school year opened in August. It is phased. Since August, children from preschool to first grade have had no device time at all, outside narrow exceptions such as mandated assessments and support required for pupils with disabilities. From November, second and third graders are capped at twenty minutes a day and fourth and fifth graders at thirty. Secondary schools follow in January 2027 with weekly caps per subject, an hour a week in each subject in middle school and an hour and a half in high school. Devices no longer go home by default; families must opt in. Student access to YouTube, social media and streaming platforms is blocked unless a teacher approves it. Behind the resolution was Schools Beyond Screens, a national coalition founded by district parents and teachers, whose campaigning the board member Nick Melvoin credited during the meeting. The significance is less in the specific minute-by-minute limits, which are debatable and will be contested, than in the precedent: a flagship district conceded that unrestricted classroom screen time was a problem serious enough to legislate against, rather than a self-evident good to be expanded.

Los Angeles is not alone, and the American movement is broader than any single district. At least six states approved screen-time legislation in 2026. Iowa capped digital instruction from kindergarten to fifth grade at sixty minutes a day, with exceptions for special education, computer science and assessments. Utah directed its state board of education to write grade-specific technology rules, including screen-time limits for the earliest grades and an end to one-device-per-pupil ratios in elementary school. Tennessee requires every district serving kindergarten to fifth grade to adopt a policy minimising unnecessary screen time, and Alabama has set screen-time standards for childcare, prekindergarten and kindergarten. Virginia and Maine went more gently, the first requiring schools to teach pupils about device use and its addictive potential, the second commissioning a study of how much instructional time is spent on school-issued devices. Not every bill succeeded: measures in Missouri and Oklahoma passed their state houses and stalled in the senate. In New York, Governor Kathy Hochul has said she is open to limiting screen time for younger pupils, citing concern from parents and educators about the youngest children spending their days on individual laptops and tablets, but has proposed no statewide limit. New York City moved first, announcing in September 2026 a one-year ban on student-facing generative AI from preschool through eighth grade, restrictions on individual devices for the youngest children, and recommended daily caps of thirty minutes for grades three to five and forty-five for middle school. The precise legal force varies from place to place and should not be overstated, but the pattern is unmistakable: a coordinated, bipartisan reappraisal of a decade of classroom digitisation.

Europe has moved in parallel and, in places, further. The regional government of Madrid announced restrictions phasing out individual digital devices across publicly funded early-childhood and primary schools, affecting well over half a million pupils, and pushing one-to-one tablets and laptops out of the education of children up to the age of twelve in favour of shared use under a teacher's supervision, with narrow exceptions for special educational needs and subjects such as coding that inherently require the tools. Sweden, having been among the most aggressively digitised school systems in the world, reversed course under the slogan “från skärm till pärm”, from screen to binder, and by April 2026 had allocated schools more than 2.1 billion kronor in grants for printed textbooks and teacher guides, with a new textbook-based curriculum due in 2028, an explicit response to slipping results. Denmark has earmarked DKK 540 million over the coming decade to put printed books and physical teaching materials back into its classrooms. Taken together, these European commitments run into the hundreds of millions of euros, spent not to add technology but to remove it. That is a remarkable thing for finance ministries to fund, and it signals that the reappraisal has moved from opinion pages into public budgets. Britain, where this essay began, is at an earlier stage. In June 2026 the Department for Education opened a call for evidence on screen use by children aged five to sixteen, with a section devoted specifically to screens in schools, to inform future policy.

The counter-case for the well-designed screen

Honesty requires taking the strongest version of the opposing argument seriously, because it is real and it is not trivial, and a precautionary case that ignores it deserves to lose.

Educational technology, at its best, does things that paper and a single overstretched teacher cannot. For a child with dyslexia, text-to-speech and adjustable typography can be the difference between access to the curriculum and exclusion from it. For a blind child, a screen reader is not a distraction machine but a door. For a child with a physical disability that makes handwriting painful or impossible, a keyboard is a liberation, and the neuroscience celebrating the handwriting brain must not be turned into a cudgel against children for whom the pen was never a neutral option. Assistive technology is one of the genuine triumphs of the digital classroom, and any policy that sweeps it away in a general panic about screens will harm precisely the children with the least margin to spare.

Well-designed software also personalises in ways a class of thirty makes impossible. Adaptive systems that adjust the difficulty of practice to the individual child, that give immediate feedback on a wrong answer rather than making a pupil wait a week for a marked worksheet, that let a fast child accelerate and a struggling child consolidate without public embarrassment, can genuinely differentiate instruction. There is a reasonable evidence base that some carefully built, curriculum-aligned programs, used in moderation for specific purposes, produce real gains, particularly in structured domains such as early numeracy and foundational reading. In under-resourced schools, a device can carry a library, a set of translated materials for a child new to the language of instruction, and a bank of practice a school could never afford to print. To pretend none of this is valuable is to lose the argument on the facts.

And not every engagement mechanic is a slot machine. A progress bar that shows a child how far they have come through a genuinely well-sequenced curriculum is providing meaningful feedback, not manufacturing a compulsion. Immediate confirmation of a correct answer is sound pedagogy with a century of evidence behind it. Some gamification supports learning rather than parasitising it, and the honest position is not that gamified equals harmful but that the category contains both, and that the two are hard to tell apart from the outside. Which is precisely the problem, and precisely why the argument has to move from banning features to allocating proof.

Where the burden of proof belongs

Strip the debate down and it reduces to a single question of institutional design: when a school is deciding whether to install a piece of software on the devices of hundreds of children, who has to demonstrate what, and to whom, before it goes in?

Today the answer, overwhelmingly, is that nobody has to demonstrate very much. A product arrives with impressive engagement statistics, a glossy pilot, testimonials, perhaps a thin white paper of its own commissioning, and the default is deployment. The burden, such as it is, falls on the sceptic. A worried teacher, a dissenting parent, a cautious administrator must assemble the case against, must find the evidence of harm, must argue upstream against a purchase already made and a contract already signed. The product is presumed beneficial until someone can prove it is not, and because proving harm in a diffuse population of children over the timescales on which learning and attention actually develop is extraordinarily difficult, the presumption is nearly impossible to dislodge. The structure of the burden guarantees the outcome.

The argument of this essay is that the burden is on the wrong shoulders, and that the reason it is on the wrong shoulders is that we have quietly classified educational software as a consumer product rather than as what it functionally is, an intervention imposed on a captive population of children who cannot consent, cannot leave, and cannot tell whether the thing holding their attention is teaching them or merely holding their attention. We do not let food manufacturers put a substance in school lunches on the strength of the fact that children keep eating it. We do not let a pharmaceutical company deploy a treatment across a paediatric population because early users engaged with it enthusiastically. In domains where the subjects are children and the stakes are developmental, we place the burden of proof on the party introducing the intervention, and we ask them to demonstrate benefit, and the absence of harm, before deployment rather than after.

Applied to the classroom, burden-shifting has a concrete shape, and it is more modest than the word precaution sometimes implies. It means that a company selling a product as educational should be required to show that the product produces learning, measured as learning and not as engagement, before it is bought at scale, and to disclose the mechanics it uses to sustain attention so that a purchaser can weigh them. It means that the primary metric a vendor is permitted to advertise to a school should be an outcome, not a retention statistic, because a retention statistic sold as if it were an educational result is close to a category error. It means that a regulator who has already identified, in the Children's Code and in California's attempt to follow it, rewards for time spent and engineered notifications as risks when a social platform aims them at a child, should ask why the identical mechanic earns a pedagogical exemption the instant the same child meets it inside a maths app. If a design pattern is a dark pattern in one context, the context does not launder it. The child's developing brain does not know or care which app store category the software was filed under. California's experience in court is also an argument for this approach over a ban. A prohibition must define in advance which features are “materially detrimental” to a child's “well-being”, and that is exactly where the Ninth Circuit found it too vague to enforce. A requirement that the vendor show its product produces learning asks a narrower question, one that can be answered product by product, by the party holding the data.

Washington has, somewhat unexpectedly, conceded the principle. On 20 August 2026 the US Department of Education issued a Dear Colleague letter, signed by Kirsten Baesler, the assistant secretary for elementary and secondary education, which K-12 Dive read as the department going to the industry's defence against the screen-time backlash. Its argument was that educational technology should not be judged mainly by how long children spend in front of it, and that recreational and educational technology should not be treated as the same thing. Yet the yardstick it offered in place of minutes is the one this essay has been arguing for. Investments, the letter said, “should be guided by evidence of effectiveness”; states making procurement and renewal decisions should look for products that have run randomised controlled trials and demonstrated positive outcomes; and providers should publish rigorous, independent evaluations of their impact on student learning “whenever feasible”. That is a federal agency endorsing outcome evidence over time on device. It is also, in its own phrasing, a picture of the gap. A letter of guidance binds no one, the decisions remain with states and districts, and “whenever feasible” leaves the vendor to judge when feasibility applies. The market is already reading the signal. In July, Ed Corey, chair of the Torrington school board, told EdWeek Market Brief that companies whose products rely heavily on daily screen time will need to show a “legitimate curriculum use case” backed by research, and a national survey of 600 teachers in May found that 85 per cent believed pupils spend too much class time on screens. The demand for proof is arriving. What has not arrived is any rule that says it must be met before the product goes in.

There is an obvious objection, and it deserves a straight answer. Proving educational benefit is hard, slow and expensive, and a strict evidence bar would slow the sector, entrench incumbents who can afford large trials, and deny some genuinely good tools to the schools that most need them. This is true, and it is the strongest thing that can be said against burden-shifting. But notice that the difficulty of proving benefit is not a neutral fact about the world. It is the very thing that has allowed engagement to stand in for learning for a decade, precisely because engagement is easy to prove and learning is hard, and a regime that rewards whatever is easy to measure will get more of what is easy to measure and less of what matters. The answer to a hard measurement problem is not to abandon the measurement and default to deployment. It is to make the party with the commercial interest, and the resources, and the data, carry the cost of the proof, calibrated sensibly to the risk, with lighter obligations on assistive tools whose benefit is manifest and heavier ones on attention-maximising products aimed at the youngest children. Burden-shifting does not mean banning software. It means the person who profits from the claim has to substantiate the claim.

The precautionary turn and what it asks of everyone

The medicine analogy is imperfect, and it should not be pushed to the point of demanding randomised controlled trials for every worksheet generator. But it captures the essential asymmetry the current arrangement has inverted. When you intervene in the development of a child who cannot consent and cannot exit, and when the potential harm is to faculties, attention, imagination, the capacity for effortful thought, whose damage may not surface for years, you do not get to run the experiment live on the whole population and check the results later. The cost of being wrong is borne by children, irreversibly and at the precise moment their minds are being formed, while the benefit of being right accrues to a firm in the form of engagement it can bank this quarter. That asymmetry is the entire moral case for shifting the burden, and it does not depend on the alarmism of any single conference talk to hold.

Each party in the system has an obligation that follows from this. The companies owe honesty about what their products optimise for, disclosure of the engagement mechanics they embed, and evidence of learning rather than evidence of stickiness, and the better firms should welcome a standard that separates them from competitors selling compulsion in a curricular costume. Regulators owe consistency, and the simplest reform available to them is to stop pretending that a mechanic they have identified as harmful becomes benign on crossing from a social app into an educational one, and to require that products marketed to children as teaching tools clear a bar proportionate to that claim. And schools, which are the point of deployment and therefore the point of leverage, owe their pupils the institutional courage to demand outcomes rather than engagement dashboards, to run their own small comparisons, to keep the paper and the pen and the unstructured hour that the market has no incentive to defend, and to treat any vendor unwilling to be measured on learning as having answered the question by declining to.

The headteacher in Hertfordshire did something the market will never do on its own. She created a condition, an empty playground, an hour without stimulation, in which the absence of the product became visible, and with it the shape of what the product had been quietly displacing. The children's inability to fill that hour is not proof of anything on its own; it is one anecdote, and anecdotes are where inquiries begin, not where they end. But it names the thing at stake with a precision the engagement statistics never will. The question is not whether children are looking at screens. It is whether the screens are teaching them, or merely keeping them, and whether we are willing to make the people who profit from the difference be the ones who prove it. Until that burden moves, the honest description of much of what is installed on the classroom device is not an educational tool that happens to be engaging. It is an engagement machine that happens to be pointed at a child in a school, and the fact that it is pointed at a child in a school is exactly why the proof should come first.

References

1. Winkleman, Sophie. “Why AI in the Classroom is a Catastrophe.” Talk delivered at the Alliance for Responsible Citizenship (ARC) 2026 conference, Olympia, London, June 2026. Video, YouTube (Alliance for Responsible Citizenship channel): ; transcript, The Singju Post: https://singjupost.com/transcript-why-ai-in-the-classroom-is-a-catastrophe-sophie-winkleman/
2. Jerrim, John, John Micklewright, Jörg-Henrik Heine, Christine Salzer and Caroline McKeown. “PISA 2015: how big is the 'mode effect' and what has been done about it?” Oxford Review of Education, vol. 44, no. 4, 2018. https://www.tandfonline.com/doi/abs/10.1080/03054985.2018.1430025
3. Van der Meer, Audrey L. H., and F. R. (Ruud) van der Weel et al. “Handwriting but not typewriting leads to widespread brain connectivity: a high-density EEG study with implications for the classroom.” Frontiers in Psychology, vol. 14, 2024. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2023.1219945/full
4. Žnidarič, Urška, Erik Štrumbelj and Octavian Machidon. “A Scoping Review of the Negative Effects of Digital Technology on Cognition.” arXiv preprint 2603.10025 [cs.CY], v1 23 February 2026, revised v2 15 August 2026. https://arxiv.org/abs/2603.10025
5. Khadka, Supriya, and Sanchari Das. “SoK: Understanding the Pedagogical, Health, Ethical, and Privacy Challenges of Extended Reality in Early Childhood Education.” Augmented Humans 2026; arXiv preprint 2602.12749 [cs.HC], 13 February 2026. https://arxiv.org/abs/2602.12749
6. BBC News. “Back to books: Sweden's schools give up digital learning.” 16 April 2026. Republished by The Star (Kenya): https://www.the-star.co.ke/news/world/2026-04-16-back-to-books-swedens-schools-give-up-digital-learning
7. Cohen, Joshua. “Why Swedish Schools Are Bringing Back Books.” Undark, 1 April 2026. https://undark.org/2026/04/01/sweden-schools-books/
8. EdSource. “Los Angeles Unified to limit screen time for all students, prohibit use among youngest students.” 2026. https://edsource.org/2026/classroom-screen-time-limits-policy/756523
9. K-12 Dive. “Screen time limits coming for LAUSD in the fall.” 2026. https://www.k12dive.com/news/lausd-imposes-screen-time-limits-starting-in-2026-27/818224/
10. Dale, Mariana, and Elly Yu. “LAUSD's youngest students will start a screen-free school year. Here's what parents need to know.” LAist, 30 July 2026. https://laist.com/news/education/lausds-youngest-students-will-start-a-screen-free-school-year-heres-what-parents-need-to-know
11. Patino, Vania. “In a new chapter for classroom learning, LAUSD is welcoming students back with screen time limits.” Spectrum News, 3 August 2026. https://spectrumlocalnews.com/ca/california/education/2026/08/03/screen-time-limits-lausd
12. Stateline. “States move to limit classroom screen time.” 23 July 2026. https://stateline.org/2026/07/23/states-move-to-limit-classroom-screen-time/
13. Rauf, David Saleh. “The Next Challenge for K-12 Ed-Tech: Proving Classroom Screen Time Is Worth It.” EdWeek Market Brief, 10 July 2026. https://marketbrief.edweek.org/regulation-policy/the-next-challenge-for-k-12-ed-tech-proving-classroom-screen-time-is-worth-it/2026/07
14. Gothamist. “Gov. Hochul says she's open to limiting screen time for younger students in NY schools.” https://gothamist.com/news/gov-hochul-says-shes-open-to-limiting-screen-time-for-younger-students-in-ny-schools
15. Ginsburg, Aaron. “NYC announces one-year ban on generative AI for public school students through 8th grade.” 6sqft, 2 September 2026. https://www.6sqft.com/nyc-ai-ban-public-schools/
16. Aleteia. “Madrid schools limit tech use for half a million students.” 4 August 2025. https://aleteia.org/2025/08/04/madrid-schools-limit-tech-use-for-half-a-million-students/
17. The Danish Dream. “Danish Schools Reintroduce Textbooks to Improve Literacy.” https://thedanishdream.com/education-in-denmark/danish-schools-reintroduce-textbooks-to-improve-literacy/
18. Information Commissioner's Office (ICO). “Age appropriate design: a code of practice for online services (the Children's Code),” standard 13, Nudge techniques. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/childrens-information/childrens-code-guidance-and-resources/age-appropriate-design-a-code-of-practice-for-online-services/13-nudge-techniques/
19. Department for Education. “Screen use by children aged 5-16: call for evidence.” Opened 8 June 2026, closed 29 June 2026. https://consult.education.gov.uk/screen-time-policy-team/screen-use-by-children-aged-5-to-16-cal/
20. United States Court of Appeals for the Ninth Circuit. NetChoice, LLC v. Bonta, No. 25-2366, opinion filed 12 March 2026. https://caselaw.findlaw.com/court/us-9th-circuit/118216051.html
21. Cooley LLP. “NetChoice v. Bonta: Ninth Circuit Narrows Injunction Against California's Age-Appropriate Design Code Act.” 30 March 2026. https://www.cooley.com/news/insight/2026/2026-03-30-netchoice-v-bonta-ninth-circuit-narrows-injunction-against-californias-ageappropriate-design-code-act
22. Future of Privacy Forum. “An Analysis of the California Age-Appropriate Design Code.” October 2022. https://fpf.org/wp-content/uploads/2022/10/FPF-Policy-Brief-California-Age-Approp-Design-Code-R2.pdf
23. Quartr. “Keeping the Streak Alive: The Story of Duolingo.” https://quartr.com/insights/edge/keeping-the-streak-alive-the-story-of-duolingo
24. THE Journal. “Department of Education Issues New Guidance on Education Technology and Screen Time.” 20 August 2026 (reporting the Dear Colleague letter signed by Kirsten Baesler, Assistant Secretary, Office of Elementary and Secondary Education). https://thejournal.com/articles/2026/08/20/department-of-education-issues-new-guidance-on-education-technology-and-screen-time.aspx
25. Merod, Anna. “Education Department goes to ed tech's defense in new guidance.” K-12 Dive, 20 August 2026. https://www.k12dive.com/news/education-department-goes-to-ed-techs-defense-in-new-guidance/828425/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The email that arrived at an antiquarian bookshop in Haarlem in the summer of 2026 listed 3,001 ISBNs. The bookseller, Pieter de Vries, assumed it was fraud. “Very unusual,” he told Fortune. “That's why I regarded the request as spam or phishing.” He deals in old and rare books, he explained, the kind that are expensive and exclusive, and nobody orders three thousand of those at once. When he later learned what such requests were widely believed to be for, his reaction was two words: “I was shocked!”

The detail that matters most about that list is the one least likely to survive a retelling. According to Fortune's account, the titles were mostly published in 2020 and 2021, and came from academic houses — Emerald, Elsevier, Wiley, Routledge, Oxford University Press. They were not incunabula. They were recent monographs, the sort of thing a university library buys and a secondhand dealer eventually inherits. The requesting party was 2077AI, which presents itself publicly as a Singapore-based non-profit open-data foundation building open datasets and benchmarks. It did not respond to Fortune's requests for comment.

This is the gap the story has been falling through since late July. There is a real, documented, court-evidenced programme of industrial book destruction carried out by a major AI laboratory. There is also a viral narrative about the last surviving copies of eighteenth-century texts being fed into shredders, which has almost no verifiable sourcing behind it. The two have fused into a single story, and the fusion is doing damage to both — it makes the real scandal easier to dismiss as hysteria, and it makes the hysteria harder to correct without appearing to defend the scandal.

So it is worth separating them carefully, because the actual cultural cost is not quite where the outrage has located it, and it is arguably worse.

What the Court Documents Actually Established

The foundational reporting is not from July 2026. It is from January.

On 27 January 2026, The Washington Post published an account, drawn from documents unsealed in copyright litigation, of an internal Anthropic initiative called Project Panama. An internal planning document put the objective without euphemism: “Project Panama is our effort to destructively scan all the books in the world.” The same document instructed staff to keep it quiet. “We don't want it to be known that we are working on this.”

The mechanics, as described in the unsealed material and reported subsequently by outlets including IBTimes, were industrial rather than exotic. Anthropic hired Tom Turvey, formerly head of partnerships at Google Books, in February 2024, tasked according to internal documents with obtaining “all the books in the world”. Purchased volumes went to vendors who used hydraulic cutting machines to remove the spines, fed the resulting loose sheets through high-speed production scanners, and arranged for a recycling company to collect what was left. Vendor proposals reported by The Washington Post referenced converting somewhere between 500,000 and two million books over a six-month period. The company spent tens of millions of dollars on the stock.

None of this was a leak in the conventional sense. It emerged because the authors Andrea Bartz, Charles Graeber and Kirk Wallace Johnson sued, and discovery did what discovery does.

The reason the story detonated again in late July 2026 is a matter of coincidence and algorithm. On 20 July, Judge Araceli Martínez-Olguín of the Northern District of California granted final approval to the $1.5 billion settlement of the piracy claims in that case — a figure the Authors Guild describes as delivering roughly $3,000 per work across an estimated 500,000 works. Six days later, on 26 July, an account on X posted a compressed summary of the whole affair. Elon Musk replied to it the following day. Within seventy-two hours, the story had been rewritten by dozens of outlets, many of them working from each other rather than from the January reporting or the court record.

That is how a January investigation into a 2024 programme became an August emergency.

The Ruling That Made Destruction the Prudent Choice

The legal architecture here is the part almost every retelling gets backwards, and it is the part that should worry people most.

On 23 June 2025, Senior District Judge William Alsup issued a partial summary judgment order that split Anthropic's conduct into three categories with three different answers. Training large language models on the books was, in his phrasing, exceedingly transformative, and therefore fair use. Downloading more than seven million books from Library Genesis and the Pirate Library Mirror to build a permanent central library was not fair use, and was the conduct that produced the settlement. And the third category — the destructive scanning of lawfully purchased print copies — was fair use as a format change.

Alsup's reasoning on that third point deserves quoting rather than paraphrasing, because its logic has consequences. Every purchased print copy, he wrote, was copied in order to save storage space and enable searchability as a digital copy. The print original was destroyed. One replaced the other.

Read that sequence again. The destruction is not incidental to the legal finding. It is load-bearing. What made the format shift defensible was precisely that no additional copy came into existence: the company that bought one book ended up with one book, in a different substrate. A firm that scanned non-destructively and kept the physical volume would have ended the process holding two copies where it had lawfully acquired one, and would have needed a different and less certain argument.

The first-sale doctrine does the rest of the work. Buy a book and you may burn it. Nothing in copyright law obliges a purchaser to preserve the object, and nothing obliges a purchaser to disclose to the seller what they intend to do with it.

The result is a legal regime in which the cheapest method of digitisation is also the safest method, and the method that preserves the artefact carries both a higher cost and a marginally higher legal risk. Whatever one thinks about the cultural stakes, that is a badly designed incentive. It is not that the law permits destruction. It is that the law, as applied, quietly rewards it.

What a Guillotine Removes

Destructive scanning is not a sinister technology. It is a boring one, and it is used every day by document bureaux digitising insurance files.

A hydraulic or guillotine cutter shears the bound edge off the text block, converting a book into a stack of loose leaves. Those leaves go through an automatic document feeder at rates that commercial operators quote in the thousands of pages per hour. The alternative — planetary or overhead scanning, where a bound volume rests in a cradle and a camera photographs each opening while a human or a robot turns pages — is slower, requires more expensive equipment, and produces images that need more correction for curvature and gutter shadow. Published price guides for digitisation services put the range across both methods at roughly ten cents to a pound and a half per page, with the destructive route sitting near the bottom of that band and bound-volume capture near the top.

So the trade is speed and unit cost against the survival of the object. On a two-million-book programme, that difference is real money. Against a training budget measured in billions, it is a rounding error — which is the substance of Musk's argument, and it is a good one.

What the guillotine takes is not the text. The text survives, in the sense that the words can be extracted. What it takes is everything about the copy that is not the text, and this is where rare-book librarians and bibliographers part company with engineers.

The guidance published by IFLA's Rare Books and Special Collections Section on planning digitisation projects is explicit that the physical object carries evidence the surrogate cannot. Provenance evidence — bookplates, ownership signatures, library stamps, booksellers' pencil codes, auction records tied to a specific copy — establishes where a book has been and who read it. Marginalia record what readers thought while they were reading, and constitute a distinct field of scholarship in their own right; the Early Modern Marginalia Research Network exists because annotations in the gutter of one particular copy are historical evidence of a kind nothing else supplies. Binding structures, sewing patterns, paper stock and watermarks date and locate production. Collation — the physical gathering structure — is how bibliographers distinguish issues and states of the same nominal edition, and how they detect made-up copies, cancels and substituted leaves.

A guillotine destroys collation evidence absolutely. It destroys the binding. It destroys the relationship between a specific annotation and a specific gathering. And a scan optimised for text extraction will not preserve any of it as a surrogate, because it is not trying to: training-oriented capture wants clean, legible, machine-readable characters, not colour-managed high-resolution page images with calibrated targets and structural metadata. There is no public evidence that Anthropic's scanning produced preservation-grade image files, and considerable evidence that it did not need to.

This distinction matters for a reason that goes beyond sentiment. A preservation scan is a surrogate that a future researcher can interrogate. A training scan is an extraction. When the extraction is done and the sheets have gone to the recycler, what remains is text inside a corporate storage system, and there is no public record of the correspondence between that text and the objects it came from.

Whether Anything Unique Was Actually Destroyed

Here is where the story needs discipline, because the most emotionally potent claim in circulation is also the least supported.

The claim, repeated across aggregator sites and social posts since late July, is that the last three copies of an eighteenth-century botanical text were shredded. I could not find a primary source for it. It does not appear in The Washington Post's January reporting, in 404 Media's investigation, in the Fortune piece, or in the Alsup order. It appears in commentary blogs and in downstream summaries, several of which cite each other. Until someone produces a title, an ISBN or accession record, and a holdings check, it should be treated as unverified, and outlets repeating it as fact are not helping.

The companies named have all denied the strong version. An Anthropic spokesperson has stated that none of its data acquisition programmes buys and destroys “rare” or “antiquarian” books, that it procures from mainstream commercial markets, and that it never bought from Zoom Books, the Canadian firm at the centre of the Australian and European bookseller complaints. Zoom Books, responding in May 2026 to accusations published by the Spanish outlet Demócrata, said that it acquires secondhand books and resells them intact, and that it does not digitise or destroy books for the purpose of training AI models, calling contrary claims inaccurate; it added, The Atlantic reported, that it did not to its knowledge sell books for those purposes. ISBNdb, whose marketing pages 404 Media exposed on 21 July, removed those pages nine days later and posted a denial stating that it has never purchased, scanned or sold a book, that it does not train AI models, and that the page was a test of market interest for a service never launched.

Zoom's qualifier matters: the 2024 destruction is documented in court records; this year's bulk orders are not attributed. Investigating for The Atlantic in early August, Alex Reisner found the surge real but the end buyer unconfirmed: some Zoom orders shipped to PrepFort, an Amazon and Walmart logistics firm that declined to discuss its clients. Reisner judged it very possible the books were bound for AI companies, but not certain. A Houston bookseller, Charlie Becker, read his own orders as Amazon arbitrage: underpriced books bought to relist at a markup.

One strand has since been traced: on 17 August, 404 Media reported that an AirTag hidden in a bulk order had ended at an Amazon facility in Las Vegas where, an anonymous former worker said, spines are cut and pages scanned. Amazon said only that it “purchases books through commercial channels to help develop and improve the products and services our customers use.” That links one order to one company, and no more.

Denials are not proof. But nor is their absence, and there is currently no documented instance of a specific unique or near-unique title being destroyed by an AI training programme. What is documented is the destruction of millions of ordinary in-copyright books, and bulk purchase requests for recent academic monographs.

Anthropic's denial is less tidy than it sounds. Unsealed emails reviewed by Snopes show Turvey telling Wonder Book, a bulk bookseller, in August 2024 that “less common books are a great place to start.” Asked to clarify, Anthropic distinguished “rare and valuable” books from harder-to-find professional and reference titles. That distinction concerns price, not how many copies survive.

The counterweight comes from library science, and it complicates both sides. Research by OCLC on what its authors call the collective collection has repeatedly found that scarcity in print holdings is the norm rather than the exception. In one analysis of the Committee on Institutional Cooperation's shared print collection, roughly three-quarters of titles were held by three or fewer member libraries. In work on the collective collection of Research Libraries UK, close to ninety per cent of print book publications were held by fewer than five libraries in the group. Across large mega-regional aggregations, at least three-quarters of titles sat in five or fewer institutions.

Two conclusions follow, and they point in opposite directions. The first is that “rare” is close to meaningless as a category boundary, because thin holdings are ordinary. The second is that thin holdings are precisely what makes bulk destruction risky: if most titles survive in only a handful of institutional copies, then the secondhand market is not a reservoir of redundancy. It is a thin layer over a thin layer.

Deposit systems provide a partial floor. Under the Legal Deposit Libraries Act 2003, a copy of every book published in the United Kingdom must be deposited with the British Library, with five further libraries entitled to request copies. Comparable regimes operate in most publishing nations. Anything published commercially in a country with functioning legal deposit almost certainly survives somewhere in an institutional collection, regardless of what happens to the secondhand stock.

The categories that fall outside that floor are the ones nobody is checking: self-published and small-press work, regional and community publications, non-commercial technical and trade literature, foreign-language material published in jurisdictions with weak deposit enforcement, and the enormous grey zone of items that were deposited but never catalogued into any union catalogue searchable today.

Which brings us to the actual finding, and it is not the one either camp wants. Nobody knows whether a unique copy was destroyed, because nobody kept a manifest. There is no published, title-level record of what went through the cutters. No holdings check was performed. No threshold was set. The absence of that record is not a side issue; it is the centre of the case. A programme that destroys books at industrial scale without logging what it destroys has made its own claim of innocence unfalsifiable, and has simultaneously made the accusation against it unfalsifiable too. That is a choice, and it was Anthropic's to make differently.

Whether the Paper Was Really Necessary

The demand-side justification, as ISBNdb marketed it to prospective clients, is elegant. “The world's best AI training data is sitting on a shelf,” the pitch read, describing books as curated, peer-reviewed, domain-specific human knowledge, structured in a way no web crawl can replicate — dense, edited, authoritative. And the pages carried a candid aside that 404 Media quoted and which has since done more reputational damage than anything else in the story: the optics problem is real, because “AI company destroys two million books” is not a headline that generates sympathy.

The underlying scientific worry is genuine. Ilia Shumailov and colleagues published a paper in Nature in July 2024 showing that generative models trained recursively on their own outputs degrade, losing the tails of the distribution and converging on blandness — the phenomenon popularised as model collapse. The paper received an author correction in Nature in 2025 and remains widely cited. If the open web is progressively filling with synthetic text, then pre-2022 print is a supply of guaranteed-human material, and its scarcity value rises with every model release.

But the literature is less apocalyptic than the marketing. A response by Ali Borji argued that the reported behaviour is a general statistical consequence of repeatedly fitting a distribution and resampling from it, rather than a novel property of language models. More consequentially, Matthias Gerstgrasser, Rylan Schaeffer and co-authors demonstrated in 2024 that collapse follows from replacing real data with each generation's synthetic output, and that accumulating synthetic data alongside the original real data avoids collapse — a result they showed across model sizes, architectures and hyperparameters, and reproduced in diffusion models and variational autoencoders. Nobody in the industry is deleting their pre-2022 corpora. The realistic scenario is accumulation, which is the regime in which collapse does not occur.

Scarcity of human text is also a softer constraint than the pitch implies. Epoch AI's analysis puts the effective stock of human-generated public text at around 300 trillion tokens, with a wide confidence interval, and projects full utilisation somewhere between 2026 and 2032 — an estimate that has already been pushed back once by improvements in data efficiency and repeated-epoch training.

Set the numbers side by side and the arithmetic is instructive. Take the upper end of the vendor proposals reported by The Washington Post — two million books — and assume a generous average of around 100,000 words per volume. That is on the order of 260 billion tokens: roughly a tenth of one per cent of Epoch's central estimate of the total human text stock. The case for print therefore cannot rest on volume. It rests entirely on a claim about quality weighting — that an edited monograph is worth some large multiple of an equivalent quantity of web text — and no laboratory has published the numbers that would let anyone outside evaluate that multiple.

This is the point at which the trade-off in the original question becomes difficult to state honestly. Is a unique book worth an incremental improvement in model performance? Nobody has quantified the improvement. The industry is asking the public to accept an irreversible cost against an unspecified and unaudited benefit, which is not a trade-off so much as a request for trust.

And the trade-off is in any case a false binary, because destruction was never required to obtain the text. Google Books digitised more than forty million volumes using predominantly non-destructive camera rigs. The Internet Archive scans non-destructively and then, rather than pulping, maintains a physical archive in Richmond, California, storing volumes in modified shipping containers held at controlled temperature and around thirty per cent relative humidity. Both organisations solved the engineering problem of mass digitisation without solving it by disposal. The difference is cost per unit and, in Anthropic's case, the legal convenience of ending up with exactly one copy.

What the Booksellers Are Objecting To

The booksellers' complaint is not a legal complaint, and pretending otherwise weakens it.

In Australia, Guardian reporting recorded a pattern that dealers found unmistakable once they saw it. John Sainsbury of Sainsburys Books in Melbourne described a run of orders from Zoom Books for titles with no relationship to one another — engineering, cycling, poetry, local history — placed with apparent indifference to price. Delfina Manor of Good Reading Secondhand Books in Benalla, Victoria received an order in May large enough that fulfilling it became an ordeal: “It was just driving me nuts.” A German dealer reported orders arriving between three and five in the morning. Booksellers quoted by The Guardian described the practice of cutting spines and discarding the remains as horrific. The Telegraph's Chris Stokel-Walker headlined his 30 July piece with the phrase cultural barbarism. A seller quoted anonymously by 404 Media captured the ambivalence better than either: the buying benefits him financially, but “I don't like the end-use, and I don't like that uncommon books are being pulped.”

Under first sale, none of this gives rise to a cause of action. A buyer may destroy what they own and need not explain themselves. If the objection were simply that destruction is wrong, it would prove too much: the book trade already pulps enormous quantities of unsold and unsellable stock, and remainder destruction is a routine part of publishing economics.

The objection rests on three narrower things, and they are stronger for being narrow.

The first is deliberate opacity. ISBNdb's withdrawn marketing did not merely offer sourcing; according to 404 Media's reporting it offered anonymity, positioning buyer confidentiality as a feature and naming the reputational reason. Anthropic's own planning document recorded that it did not want the work known. Concealment of end use is not fraud, but it is a considered decision to prevent the counterparty from exercising a judgement they would otherwise have exercised. Booksellers are not claiming a right to control what buyers do. They are claiming that they were placed on the wrong side of an information asymmetry that was engineered on purpose. Routed through confidential intermediaries, the 2026 orders leave sellers unable to tell whether a parcel ends in a scanner, an Amazon relisting or a recycler.

The second is role morality. Antiquarian and secondhand dealers do not generally think of themselves as retailers of interchangeable units. The trade's self-conception is custodial: a book passes through your hands on its way to the next reader, and part of the job is judging where it should go. That is a professional identity, not a legal entitlement, but it is the reason the reaction has been visceral rather than commercial.

The third is market structure. Price-insensitive bulk buying in a thin market does not merely remove copies; it distorts the pricing signals dealers use to allocate scarce stock, and it strips the mid-list and specialist tiers fastest, because those are where supply is thinnest and the buyer's indifference to price bites hardest. If a programme wants two million books and does not care which two million, the books it takes are the ones nobody else was competing for — which is another way of describing the scarce ones.

The Precedent That Should Have Settled This

None of this is unprecedented, and the precedent is close enough that its absence from most of the coverage is conspicuous.

In April 2001, Nicholson Baker published Double Fold: Libraries and the Assault on Paper, expanded from a New Yorker piece of July 2000. Its subject was the mid-century programme under which American research libraries, including the Library of Congress, microfilmed newspapers and books and then discarded the originals, on the reasoning that wood-pulp paper was embrittling towards inevitable disintegration. Baker argued that the brittleness case was overstated, that the space savings were the real motive, and that irreplaceable runs of newspapers had been destroyed to produce surrogates that were often poorly made, badly filmed, and stripped of the colour supplements and physical evidence the originals carried.

The book was contested immediately — Richard Cox's response for the Society of American Archivists is the best-known rebuttal — and Baker conceded in his preface that it was not an impartial piece of reporting. Some of his specific charges were unfair to librarians working under genuine preservation pressures and real budget constraints.

But the structural lesson survived the argument, and it is precisely applicable. Surrogate-making organisations systematically underestimate what the original carries, because they define value in terms of the thing they are capturing. The loss is invisible until someone needs the original for a purpose the surrogate-maker did not anticipate. And by the time that purpose arises, the decision is unreviewable.

Apply that to a training scan. The purpose is text extraction. Everything defined as noise — the binding, the annotation, the paper, the collation, the ownership marks — is discarded twice: once when the capture is specified, and again when the sheets go to the recycler. If a future researcher, or for that matter a court in some later case weighing Alsup's reasoning, wanted to examine the physical objects, there is nothing to examine.

Musk, and the Limits of a Pledge

On 27 July 2026, replying to the X post that had propelled the story, Elon Musk wrote: “I've asked the SpaceXAI team to preserve any rare books in a library and scan them the hard way vs just cutting off the spine and scanning.”

Two corrections are warranted, one to the sceptics and one to the enthusiasts.

To the sceptics: SpaceXAI is a real entity. CNBC reported that SpaceX acquired xAI in an all-stock transaction announced on 2 February 2026, valuing the combined business at $1.25 trillion, folding xAI, Grok and the X platform into SpaceX; the merged company listed on Nasdaq in June 2026, and the xAI name was retired in favour of SpaceXAI in July. Dismissing the pledge on the grounds that no such company exists is wrong.

To the enthusiasts: the pledge is close to unfalsifiable. It does not define “rare”. It specifies no holdings threshold, no catalogue against which rarity would be assessed, no verification, no audit, no reporting, and no consequence for non-compliance. It is a sentence on a social platform. Musk's associated observation — that the extra cost of careful scanning is trivial against budgets in the billions, and that slower capture still yields the training data while the book survives — is entirely correct and is the strongest argument anyone in the industry has made. But an argument is not a policy, and no published policy has followed.

That this is nonetheless the most substantive public commitment any frontier laboratory has made on the question tells you where the floor currently sits.

What a Serious Response Would Require

The useful question is not whether shredding is barbarism. It is what a competent regime would look like, given that mass digitisation of print is going to continue and that most of it is unobjectionable.

Six measures would do most of the work, and none of them is technically hard.

A destruction manifest, published at title level. Every volume destroyed, logged by ISBN or bibliographic identifier, with the log made public or lodged with a deposit library. This is the cheapest measure and the most important, because it converts an unanswerable argument into a checkable one. Its absence is why nobody can say whether the eighteenth-century botanical text ever existed.

A holdings check before the cutter. Union catalogues exist. WorldCat, Library Hub Discover and national union catalogues can be queried programmatically. A rule that any title falling below a stated threshold of recorded holdings is routed away from destruction would be trivial to implement at the scale these programmes operate, and would cost a fraction of a per cent of the acquisition budget.

Deposit after scanning. Where a title clears the threshold and is destroyed anyway, the alternative for flagged items should be that the intact volume is offered to a deposit or research library, with the acquiring company funding carriage. Where the volume has already been cut, the loose text block should be boxed and offered rather than recycled: a guillotined block is not an artefact, but it is not nothing, and it preserves the leaves, the annotations and the paper.

Preservation-grade capture where destruction occurs. If a copy is going to be destroyed, the images taken from it should meet archival imaging standards rather than extraction standards, with the page images retained and escrowed rather than discarded after text extraction. The marginal cost of capturing at higher resolution during a pass you are making anyway is small. The marginal value, given that the original will not exist afterwards, is large.

Disclosure at the point of sale. A simple end-use declaration to the seller — this order is for digitisation and the volumes will be destroyed — resolves the consent objection entirely. Sellers who object can decline or price accordingly. Sellers who do not care can proceed. The only thing disclosure costs is the anonymity that ISBNdb was explicitly marketing.

A regulatory hook that already exists. Article 53 of the EU AI Act obliges providers of general-purpose AI models to publish a sufficiently detailed summary of training content; the Commission published its mandatory template on 24 July 2025, with obligations applying from 2 August 2025 and a 2027 deadline for models already on the market. The Commission has been able to enforce them, with fines, since 2 August 2026, and the Digital Omnibus did not delay them. The template covers data origin and curation. It would take no new legislation, only interpretive guidance, to establish that acquisition provenance — including whether physical originals were destroyed — falls within scope. That is the single most available lever anyone concerned about this has.

The Record of What Was Lost

So: does the permanent loss of a book that may exist nowhere else constitute an acceptable price for incrementally improving a model?

The question, posed that way, cannot be answered, and it is worth being clear about why. The loss has not been demonstrated — no unique title has been identified, and the most-repeated claim to the contrary has no traceable source. The improvement has not been quantified — no laboratory has published what two million books bought them. And the trade is false in any case, because Google and the Internet Archive have both demonstrated that the text can be obtained at scale without the destruction, and Musk is right that the cost differential is negligible against these budgets.

What can be said is narrower and, I think, more damning than the viral version.

An organisation set out to destructively scan, in its own words, all the books in the world. It recorded in writing that it did not want this known. It destroyed millions of volumes without checking, at any point, how many other copies of any of them existed. It kept no public record of what it destroyed. It captured those volumes to a standard suited to text extraction rather than preservation, ensuring that even the digital residue is a diminished thing. And the legal reasoning that blessed the arrangement did so partly because the physical copy was destroyed — meaning the destruction was not a regrettable byproduct but a component of the defence.

The cultural cost is therefore not primarily the books. It is the erasure of the record of the books: the fact that this transaction between the physical past and the computational present was conducted without a ledger, and that we are now reduced to arguing about botanical texts nobody can name because the only people who could settle the question chose not to write anything down.

Nicholson Baker's complaint about the microfilmers was never really that they preferred film to paper. It was that they made an irreversible decision on behalf of everyone who came afterwards, using a definition of value that suited their own project, and that they did it without keeping account. A quarter of a century later, with better machines and vastly more money, the same decision has been made the same way. The remedy has not changed either. Check the holdings. Keep the manifest. Offer the copy. Tell the seller.

None of that would have cost anything meaningful. That it was not done is the answer to the question.

References

1. Fortune, “This Dutch bookseller thought a request for 3,000 copies was 'spam or phishing.' Instead, AI companies are scanning and destroying books to train AI”, 31 July 2026. https://fortune.com/2026/07/31/dutch-bookseller-ai-spam-phishing-3000-book-copies-scan-destroy/
2. The Washington Post, “Anthropic 'destructively' scanned millions of books to build Claude”, 27 January 2026. https://www.washingtonpost.com/technology/2026/01/27/anthropic-ai-scan-destroy-books/
3. Emanuel Maiberg, 404 Media, “AI Companies Are Buying Tons of Old Books Because They're Free of AI Slop”, 21 July 2026. https://www.404media.co/ai-companies-are-buying-tons-of-old-books-because-theyre-free-of-ai-slop/
4. 404 Media, “Company Offering Printed Books to Train AI Stops After 404 Media Coverage”, 30 July 2026. https://www.404media.co/ai-company-training-scanning-books-database-isbndb/
5. IBTimes UK, “Inside Project Panama, Anthropic's Secret Effort To Scan and Shred the World's Books”, 2026. https://www.ibtimes.co.uk/anthropic-secret-book-scanning-operation-1811155
6. Chris Stokel-Walker, The Telegraph, “Cultural barbarism: How AI companies are destroying the world's books”, 30 July 2026 (paywalled; text and citation recorded at Kip Currier, Management and Leadership, 30 July 2026). http://kipcurriermanagementleadership.blogspot.com/2026/07/cultural-barbarism-how-ai-companies-are.html
7. Outlook India, “Explained: Why Australian Booksellers Are Raising Alarm Over AI's Book Hunt”, 2 August 2026. https://www.outlookindia.com/international/explained-why-australian-booksellers-are-raising-alarm-over-ais-book-hunt
8. Publishers Lunch, “Zoom Books Responds To Accusations of Selling Books For AI Scraping”, 27 May 2026. https://lunch.publishersmarketplace.com/2026/05/zoom-books-responds-to-accusations-of-selling-books-for-ai-scraping/
9. United States District Court, Northern District of California, “Order on Fair Use, Bartz v. Anthropic PBC”, 23 June 2025. https://copyrightalliance.org/wp-content/uploads/2025/06/Bartz-v.-Anthropic-Order.pdf
10. The Authors Guild, “Court Grants Final Approval of $1.5 Billion Anthropic Copyright Settlement”, July 2026. https://authorsguild.org/news/court-grants-final-approval-anthropic-copyright-settlement/
11. Elon Musk, post on X replying to @HedgieMarkets, 27 July 2026.

    I’ve asked the SpaceXAI team to preserve any rare books in a library and scan them the hard way vs just cutting off the spine and scanning

    — Elon Musk (@elonmusk) July 27, 2026

12. CNBC, “Musk's xAI, SpaceX combo is the biggest merger of all time, valued at $1.25 trillion”, 3 February 2026. https://www.cnbc.com/2026/02/03/musk-xai-spacex-biggest-merger-ever.html
13. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, “AI models collapse when trained on recursively generated data”, Nature, 24 July 2024. https://www.nature.com/articles/s41586-024-07566-y
14. Ali Borji, “A Note on Shumailov et al. (2024): 'AI Models Collapse When Trained on Recursively Generated Data'“, arXiv:2410.12954, 2024. https://arxiv.org/abs/2410.12954
15. Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov and colleagues, “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data”, arXiv:2404.01413, 1 April 2024. https://arxiv.org/abs/2404.01413
16. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim and Marius Hobbhahn, “Will we run out of data? Limits of LLM scaling based on human-generated data”, Epoch AI, 2024. https://epoch.ai/publications/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
17. IFLA Rare Books and Special Collections Section, “Guidelines for Planning the Digitization of Rare Book and Manuscript Collections”, September 2014. https://www.ifla.org/files/assets/rare-books-and-manuscripts/rbms-guidelines/guidelines-for-planning-digitization.pdf
18. Brian Lavoie, “Reflections on Collective Collections”, College & Research Libraries / OCLC Research preprint, 2020. https://www.oclc.org/content/dam/research/publications/2020/oclcresearch-reflections-on-collective-collections-preprint.pdf
19. Legal Deposit Libraries Act 2003, UK Public General Acts. https://www.legislation.gov.uk/ukpga/2003/28
20. Nicholson Baker, Double Fold: Libraries and the Assault on Paper, Random House, April 2001. https://en.wikipedia.org/wiki/Double_Fold
21. Richard J. Cox, “Don't Fold Up: Responding to Nicholson Baker's Double Fold”, Society of American Archivists, 2001. https://www2.archivists.org/news/2001/dont-fold-up-responding-to-nicholson-bakers-double-fold-by-richard-j-cox
22. Alex Reisner, The Atlantic, “Someone Is Mysteriously Snapping Up Used Books Around the World”, 5 August 2026. https://www.theatlantic.com/technology/2026/08/ai-companies-buying-used-books-for-data/688167/
23. Emanuel Maiberg, 404 Media, “We Tracked a Shipment of Rare Books. It Ended at an Amazon AI Training Facility”, 17 August 2026. https://www.404media.co/we-tracked-a-shipment-of-rare-books-it-ended-at-an-amazon-ai-training-facility/
24. Anna Rascouët-Paz, Snopes, “Are AI companies scanning and destroying millions of books, including rare titles?”, 3 August 2026. https://www.snopes.com/fact-check/ai-companies-destroying-rare-books/
25. Mayer Brown, “EU AI Act News: Digital Omnibus on AI, New Guidance on Risk Classification, GPAI, and Transparency Obligations”, 30 July 2026. https://www.mayerbrown.com/en/insights/publications/2026/07/eu-ai-act-news-digital-omnibus-on-ai-new-guidance-on-risk-classification-gpai-and-transparency-obligations

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The software on the slide looked like something a sales team would buy. There was a dashboard. There were contact targets for the day. There was a pipeline that tracked each prospect from first message to final payment, and an assistant that suggested the next reply. Eric Huber, senior manager of adversary intelligence and disruption at TD Bank, was showing it to an audience at Black Hat USA 2026 in Las Vegas in August. The prospects in the pipeline were fraud victims. The workers hitting their daily quotas were, in many cases, trafficked people held in scam compounds in Cambodia and Myanmar, who faced punishment if they fell short.

One of the tools Huber demonstrated, according to reporting by Joel Witts at Expert Insights, was a reply generator called Editone. An operator loads a victim's profile alongside a fabricated persona, and the software drafts responses designed to land. It has voice cloning built in. Translation software lets a worker who speaks one or two languages work a target in a third. Tara Seals of Dark Reading described another kit, ProKYC, used by Nigerian fraud rings to defeat banks' identity checks by generating forged “selfie with ID” images stamped with fake location metadata, so a picture taken in Nigeria appears to come from Sydney. Her headline summed up the session: AI had sent global crime syndicates into “fraud nirvana”.

Huber was careful about one point, and it matters more than the headline. The tools, he said, do not automate the scam. They “help some poor person do the scamming by creating responses that are going to be the most effective.” What he described was not a robot that defrauds people by itself. It was a factory: a production line in which human labour, software and management systems are arranged to maximise output per worker, and in which each new piece of AI removes another bottleneck.

That distinction reframes the question people actually ask. When a voice that sounds exactly like your son calls in tears, or a colleague's face appears on a video call asking for an urgent transfer, or a stranger online seems to understand you better than anyone has for years, how do you tell the real from the manufactured? The uncomfortable answer from the research of the past year is that, in most cases, you cannot, at least not by looking and listening. The more useful answer is that you were never supposed to have to. Factories are beaten by changing their economics, not by asking each customer to inspect the product more closely.

A Production Line Built on Conversation

To understand why AI matters to fraud, start with what fraud work consists of. The most detailed answer comes from a study posted to arXiv in December 2025 by Gilad Gressel, Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnashree Achuthan and Yisroel Mirsky, titled “Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams”. The researchers interviewed 145 insiders and five victims. Their central finding was that 87 per cent of scam labour consisted of what they called “systematized conversational tasks”, the scripted, repetitive work of building rapport over weeks of text messages before steering a target towards a fraudulent investment.

That figure is the key to the whole economy. A scam compound is, in effect, a call centre whose product is trust, and most of its costs are the hours it takes to produce that trust one conversation at a time. The team then tested what happens when a machine does the talking. In a week-long blinded study, a language-model agent elicited more trust from participants than human operators did, and achieved higher compliance with requests: 46 per cent against 18 per cent for the humans. Commercial safety filters, run over the romance-baiting dialogues, flagged 0.0 per cent of them. The authors concluded that these scams “may be amenable to full-scale LLM automation”.

International law enforcement has reached a similar conclusion. In March 2026, INTERPOL published its second Global Financial Fraud Threat Assessment, timed for its Global Fraud Summit. The organisation reported that AI-enhanced fraud is 4.5 times more profitable than traditional methods, and that fraud-related INTERPOL Notices and Diffusions, the alerts member countries use to flag suspects and methods, had risen 54 per cent since 2024. It warned that agentic AI systems can now run complete fraud campaigns, from reconnaissance to ransom demands. It described hybrid tactics in which sextortion is built into romance and investment scams using scripts and AI-generated content, and scam centres that were once regional now operating worldwide and involving hundreds of thousands of people, many of them trafficked. Valdecy Urquiza, INTERPOL's Secretary General, put it in a single phrase: “we are witnessing the industrialization of fraud.”

The United Nations Office on Drugs and Crime, in a threat assessment launched in Bangkok in July 2026, estimated scam losses across East and South-East Asia, Australia and New Zealand at between $88.3 billion and $114.1 billion in 2025. It documented the use of generative AI, deepfakes and automated fraud, and found people from at least 80 countries and territories inside the region's compounds. The geography of the labour force and the geography of the victims are both global.

Academic work has started to describe the same shift in its own terms. A January 2026 paper by Heedou Kim, Changsik Kim, Sanghwa Shin and Jaewoo Kang, which built a detection system called ScriptMind from 571 real Korean phone scam cases, observes in its introduction that “with advances in LLMs and generative AI, such scams have evolved into large-scale, organized, and multi-channel attacks.” A survey of AI-driven cybersecurity threats by Sai Teja Erukude, Viswa Chaitanya Marella and Suhasnadh Reddy Veluru, also posted in January 2026, puts it more bluntly: “The real danger of AI in cybercrime is that it's now possible to automate complex and large-scale scams.”

The Arithmetic of a Five-Minute Phone Call

Factories exist because of unit economics, and the most useful recent paper on AI fraud is an exercise in accounting. “Evaluating AI Models' Capability to Automate Voice Phishing Attacks”, by Fred Heiding of Harvard Kennedy School and colleagues including Claudio Mayrink Verdun, Simon Lermen, Andrew Kao and researchers at Meta, was posted to arXiv in July 2026 and published in the journal Expert Systems With Applications.

The team ran a survey experiment with 4,100 American adults, exposing them to recordings or transcripts of scam calls voiced by leading AI voice models, including systems from Google, OpenAI, ElevenLabs and Sesame, and comparing them with human-voiced versions. The overall compliance rate across five scam types was 16.5 per cent. In the “relative in distress” scenario, the one that maps onto the classic family emergency call, up to 36 per cent of participants said they would or might comply. Caller persuasiveness was the strongest predictor of compliance, and some synthetic voices, Sesame's in particular, were rated as persuasive as human ones or slightly more so.

Then came the accounting. A human vishing operator, calibrated at the average American non-farm wage of $34.55 an hour, handles about twelve five-minute attempts an hour. Assume the scammer extracts $450 from each successful victim, and that only 0.6 per cent of persuaded people actually pay, the lowest conversion rate the authors could find in any legitimate industry. On those assumptions, a human-run vishing operation at American wages loses around $27 an hour. Some AI models also lose money because inference is expensive. But others, including Gemini, Sesame and ElevenLabs, came out with positive expected profits. The authors estimated that building such a pipeline takes around 260 hours of engineering time. Their conclusion is the sentence that has travelled furthest from the paper: “even conservative estimates suggest that AI-automated vishing is economically viable at scales accessible to individual bad actors, not just organized crime syndicates.”

Precision matters here. The authors call AI vishing “roughly break-even” across the models they surveyed, not wildly profitable, and they found that targeting particular demographic groups was not yet worth its cost. So the popular shorthand, that the barrier to industrial fraud has fallen to zero, overstates the evidence. The real finding is subtler and more worrying. The constraint that used to cap phone fraud was human time. Each operator could hold one conversation at a time, and in a high-wage country that time cost more than the fraud returned. AI removes the constraint. As the paper notes, AI agents “can conduct thousands of concurrent calls, limited mainly by telephony and network capacities rather than human capacity.” The authors call the result an “automation dividend”: a situation in which “even small per-call success probabilities translate into large aggregate losses.”

Researchers outside the paper say the same thing in plainer language. In February 2026, the Guardian's Aisha Down reported on an analysis of the AI Incident Database, which found that frauds, scams and targeted manipulation had been the largest category of incidents reported to the database in 11 of the previous 12 months. Simon Mylius, an MIT researcher who works on a project linked to the database, told the paper that “capabilities have suddenly reached that level where fake content can be produced by pretty much anybody,” and that “there is really effectively no barrier to entry.” Heiding, quoted in the same piece, said: “It's becoming so cheap, almost anyone can use it now.”

Why Volume Beats Brilliance

The factory framing explains an apparent paradox. AI fraud does not have to be brilliant. It has to be good enough, cheap enough and plentiful enough.

A traditional con artist needed a high hit rate because each attempt cost hours of skilled work. A factory needs only a positive expected value per attempt, multiplied by a very large number of attempts. A 16.5 per cent compliance rate, or even a fraction of one per cent conversion to actual payment, is a disaster for an individual criminal working by hand. For an automated system running thousands of calls at once, around the clock, in dozens of languages, it is a business model. The Heiding team's economics deliberately assumed that AI scales no better than humans, which they acknowledge “is likely to underestimate the reach of AI-automated vishing”.

This is why the losses keep climbing. The FBI's Internet Crime Complaint Center reported 1,008,597 complaints in 2025, the first year it crossed one million, and reported losses of $20.877 billion, up 26 per cent on 2024. Investment fraud alone accounted for $8.6 billion. According to a Department of Justice release in April 2026, losses in the category of investment fraud the department ties to South-East Asian scam centres rose 24 per cent in 2025 to more than $7.2 billion.

The FBI also counted more than 22,000 complaints with an AI connection, with associated losses of nearly $900 million. That number is likely an undercount, for a reason that goes to the heart of the problem: victims usually have no way of knowing whether a message, a voice or a face was machine-made. Fraud built on synthetic trust leaves no fingerprint that the person defrauded can see. Statistics on “AI-enabled fraud” therefore measure the cases where AI was obvious, not the cases where it worked.

None of the technologies involved is exotic. Voice synthesis, real-time video generation and conversational language models are the same tools that power customer service bots, dubbing software and AI companions. The fraud factory does not need a research laboratory. It rents the consumer technology stack and points it at a different goal.

Your Senses Were Never the Security System

The standard advice to the public has long rested on perception. Listen for the robotic cadence. Watch for the blink that comes too late, the hands that look wrong, the shadow that falls the wrong way. Even the FBI's December 2024 public service announcement on generative AI fraud advised people to “look for subtle imperfections in images and videos, such as distorted hands or feet, unrealistic teeth or eyes”, and to “listen closely to the tone and word choice” to tell a loved one from a cloned voice.

The evidence on whether people can do this is not encouraging. In 2024, Alexander Diel and colleagues published a systematic review and meta-analysis of 56 papers on human deepfake detection in Computers in Human Behavior Reports, pooling results from more than 86,000 participants. Total detection accuracy was 55.54 per cent. Because the confidence interval crossed 50 per cent, the authors concluded that overall detection was not significantly better than chance. Audio was the easiest format at 62.08 per cent; images and text were barely distinguishable from guessing. An earlier study in iScience by Nils Köbis, Barbora Doležalová and Ivan Soraperra summed up the psychological trap in its title: “Fooled twice: People cannot detect deepfakes but think they can.”

Muhammad Tahir Ashraf, a practitioner and technology entrepreneur, made the consequence explicit in an April 2026 arXiv paper titled “Synthetic Trust Attacks”. “Generative AI is the force-multiplier behind this surge,” he writes, “enabling voice cloning, video deepfakes, and LLM-driven conversations that scale social engineering attacks to industrial speed.” His core argument is that existing defences focus on detecting synthetic media, while “the real attack surface is the victim's decision.” Citing the detection figures and the 46 versus 18 per cent compliance gap, he concludes that “the perception layer has already failed.” Ashraf's framework is a model rather than an empirical study, but its diagnosis lines up with the data.

The clearest real-world illustration remains the case Ashraf opens with. In early 2024, an employee in the Hong Kong office of the British engineering firm Arup joined a video call with people who appeared to be the company's chief financial officer and other colleagues. All of them were synthetic. The employee made 15 transfers totalling HK$200 million, about US$25.6 million, according to Hong Kong police as reported by CNN. Rob Greig, Arup's global chief information officer, later told the World Economic Forum that this was not a cyberattack in the usual sense, because no system was breached. “What happened at Arup,” he said, “I would call it technology-enhanced social engineering.” Nothing was hacked except a person's reasonable belief that the faces on the screen were real.

If perception has failed, then asking individuals to look harder is not a defence. It is a transfer of risk to the person least able to bear it.

The Executive Who Asked About a Book

There is a better model, and some of the most instructive examples come from people who were not fooled.

In July 2024, Bloomberg reported that an executive at Ferrari had received a series of WhatsApp messages, from an unfamiliar number, purporting to come from the chief executive, Benedetto Vigna. They described a confidential acquisition. A phone call followed, in a voice that imitated Vigna's southern Italian accent convincingly. The executive grew suspicious and said, according to the report: “Sorry, Benedetto, but I need to identify you.” He asked for the title of a book Vigna had recommended to him days earlier. The call ended abruptly.

That executive did not detect a deepfake. He did not spot a glitch or a wrong vowel. He applied a procedure: a challenge whose answer existed only in a shared history that the impersonator could not access. That is the crucial shift. A procedure works whether or not the fake is perfect. Perception works only when the fake is flawed, and the factories are paid to remove the flaws.

The second example shows what happens when procedure fails at the individual level but works at the system level. On 24 March 2025, a finance director at a multinational company in Singapore was contacted on WhatsApp by someone posing as the company's chief financial officer. A purported lawyer called to stress confidentiality, and the director signed a non-disclosure agreement. On a Zoom call the next day, he saw the chief executive and other executives, impersonated with deepfakes. On 26 March he transferred more than US$499,000 from the company's HSBC account into what turned out to be a money mule account. The scam came to light only when the fraudsters asked for another US$1.4 million. He alerted HSBC, which immediately notified Singapore's Anti-Scam Centre. By 28 March, working with the Anti-Deception Coordination Centre of the Hong Kong Police Force, the authorities had withheld the full amount that had been moved on to Hong Kong, more than US$494,000, and seized the remainder locally.

The director was fooled. The money was recovered anyway, because a bank, a national anti-scam centre and a foreign police unit could act within days across a border. The factory beat the person. It did not beat the system.

For individuals, the lesson is to replace judgement with routine wherever money or credentials are involved. The FBI's own list contains the useful parts: “Create a secret word or phrase with your family to verify their identity,” and verify a caller “by hanging up the phone, researching the contact of the bank or organization purporting to call you, and call the phone number directly.” The value of these steps is that they are channel-independent. A cloned voice cannot know a code word agreed at the kitchen table. A deepfaked colleague cannot intercept a call you place to a number you already hold. A romance that has never moved off one app, never met in a verifiable way and eventually turns to an “opportunity” fails a procedural test long before it fails a perceptual one.

Ashraf's paper packages similar advice as “Calm, Check, Confirm”, a protocol aimed at the moment of decision. The names matter less than the principle. Urgency, secrecy and a new channel are the three pressures that recur across the Arup, Ferrari and Singapore cases, and they recur because they are designed to stop the victim from running a procedure. Any request that combines them should trigger one automatically, however real the voice or face seems.

But these steps have limits. They protect people who set them up in advance, which disproportionately means the organised and the informed. They do little for someone isolated, someone in crisis, or someone whose new romantic interest has spent six weeks becoming the most attentive presence in their life. A defence that depends on billions of people behaving like a well-briefed Ferrari executive is not a defence against a factory. It is a lottery with better odds for some players.

Making the Bill Land Where the Fix Is

If individual vigilance cannot carry the load, the question becomes who can, and how to make them want to. The most promising answer of the past two years has been to move the cost of fraud onto the institutions that can change the factory's economics.

The United Kingdom made the boldest bet. Since 7 October 2024, the Payment Systems Regulator has required payment firms to reimburse most victims of authorised push payment fraud, the kind in which a victim is tricked into sending money themselves, up to £85,000, with the cost split between the sending and receiving firms. In the first full year, to the end of September 2025, the regulator reported that £173 million had been returned, 88 per cent of the money lost to in-scope scams. Only about three per cent of claims were rejected on the grounds that the customer had not taken enough care. More important for the factory argument, an independent evaluation published by the regulator in July 2026 found that APP fraud losses over Faster Payments fell by around 21 per cent after the requirement came in, a reduction of about £73 million a year, and that the rules had strengthened firms' incentives to invest in prevention. Reimbursement does more than compensate victims. When banks pay for fraud, they have a reason to hunt mule accounts, slow suspicious transfers and share the intelligence that makes fraud less profitable.

Singapore went further up the chain. Its Shared Responsibility Framework, developed by the Monetary Authority of Singapore and the Infocomm Media Development Authority and in force since 16 December 2024, assigns specific anti-phishing duties to banks and to telecoms operators, from real-time fraud surveillance to blocking unauthorised SMS sender IDs and filtering messages containing malicious links. Losses then flow down a “waterfall”: a bank that has breached its duties bears the loss first; if it has not, a telco that has breached its duties pays; only if both have complied does the loss stay with the consumer. The design point is that liability follows the specific failure, so each link in the chain has a reason to fix its own part.

Australia passed the broadest version. The Scams Prevention Framework Act, which cleared both houses of Parliament on 13 February 2025, imposes obligations to prevent, detect, report, disrupt and respond to scams on banks, telecoms providers and digital platforms, including social media, with civil penalties of up to A$50 million for serious breaches and a route for consumers to seek redress when a business has failed its obligations. The Australian figures also give the most honest picture of what system-level defence can and cannot do. According to the National Anti-Scam Centre, reported losses fell 25.9 per cent to A$2.03 billion in 2024, a drop the centre credited to joint work by government, industry, law enforcement and community groups. In 2025 they rose again by 7.8 per cent, to A$2.18 billion. Catriona Lowe, deputy chair of the Australian Competition and Consumer Commission, called scams a “wicked problem” because they are “complex, fast-evolving, and resistant to simple solutions,” and pointed to the “increasing sophistication in scam activity through artificial intelligence.”

That is the realistic shape of success. Coordinated systems can bend the curve. They cannot make a factory disappear. They have to keep changing its input costs faster than it can adapt.

Breaking the Machinery Upstream

The most direct way to raise a factory's costs is to attack its equipment, its premises and its money. Over the past year, that has started to happen.

The United Kingdom's Crime and Policing Act 2026, which received Royal Assent on 29 April 2026, created offences of possessing and supplying SIM farms without a legitimate reason. These devices, defined in government guidance as equipment able to use five or more SIM cards at once for calls or messages, allow a single operator to send scam texts at industrial volume. The government announced the measure in 2025 as a first in Europe and said it would take effect six months after Royal Assent. Critics have noted that much scam traffic originates abroad, which limits what a domestic ban can achieve. But the logic is sound. The offence targets the machine that multiplies one person's reach, rather than the message the machine sends.

In October 2025, after Myanmar's military said it had raided KK Park, one of the largest scam compounds on the Thai border, detaining more than 2,000 people and seizing 30 Starlink terminals, SpaceX said it had disabled more than 2,500 Starlink kits near suspected scam centres in the country. Reporting at the time noted that other compounds along the border remained active and that operations might simply relocate. Factories move when they are disrupted. That is a reason to disrupt them repeatedly, not a reason to stop.

The money is the third target. In October 2025, the US Department of Justice filed a civil forfeiture complaint against around 127,271 bitcoin, then worth roughly $15 billion, linked to Chen Zhi, chairman of Cambodia's Prince Group, whom prosecutors charged with wire fraud and money laundering conspiracy over forced-labour scam compounds. The US Treasury and the UK government imposed coordinated sanctions on the network. Chen and Prince Group have denied wrongdoing. Cambodia, which had revoked his Cambodian citizenship by royal decree in December 2025, arrested him on 6 January 2026 and, in a move announced the next day, sent him to China along with two other Chinese nationals. China's Ministry of Public Security said his group was suspected of fraud, running casinos and concealing criminal proceeds. The Justice Department's Scam Center Strike Force, created in November 2025, said by April 2026 that it had restrained more than $700 million in cryptocurrency and taken down more than 500 fraudulent investment websites. By September 2026, after a single-day action that restrained about $52 million linked to Xinbi Guarantee, a Telegram-based marketplace sanctioned by the US Treasury the same day, its cumulative total had reached about $938 million. INTERPOL, for its part, launched Operation Shadow Storm, a task force funded by the UK Home Office to go after the financial networks behind scam centres.

None of these actions alone ends industrial fraud. Together they move effort away from the moment a victim must decide whether a voice is real, and towards the infrastructure that makes it cheap to fake the voice ten thousand times.

Building Factories That Waste the Factory's Time

There is a further move available, and it has a pleasing symmetry. If AI makes it cheap for criminals to hold thousands of conversations, it makes it equally cheap for defenders to hold thousands of conversations with criminals.

In November 2024, the British mobile operator Virgin Media O2 unveiled Daisy, an AI “granny” built to answer scam calls in real time. It chains together a speech-to-text model, a custom language model with a character layer, and a synthetic voice, and it was trained with help from the YouTube scambaiter Jim Browning. O2 seeded Daisy's numbers onto the lists scammers use to find victims. The company said Daisy had kept numerous fraudsters on the phone for 40 minutes at a time. Murray Mackenzie, O2's director of fraud, warned that “no matter how persuasive someone on the other end of the phone may be, they aren't always who you think they are.”

Australia's Commonwealth Bank has taken the idea further. In June 2025 it announced a partnership with Apate.ai, a company spun out of Macquarie University's Cyber Security Hub and led by Professor Dali Kaafar. Apate runs a network of honeypot phone numbers designed to be found by scammers. When they call, they reach AI bots with varied ages, genders and accents, tuned with Australian slang, that keep them talking while gathering intelligence on their scripts and tactics. That intelligence feeds into the bank's scam controls and the wider anti-scam network in near real time.

The ScriptMind paper points the same way from the detection side. Its authors fine-tuned an 11-billion-parameter model on 22,712 structured examples built from real scam cases, report that it outperformed GPT-4o by 13 per cent, and found that in simulated calls it raised and sustained users' suspicion. The idea is not to ask people to hear the fake, but to have a machine recognise the script every factory relies on and warn the person in the moment.

These systems will not stop fraud alone, and some of them, Daisy especially, are partly marketing exercises whose results are reported by the companies themselves. But they point at the right target. A fraud factory's margin depends on the share of its labour and compute that reaches paying victims. Every minute a worker or an AI agent spends talking to a bot that will never pay, and every script that is captured and turned into a warning, is a cost the factory has to absorb. Automation does not only help the attacker. For once, it lets defenders operate at the same scale.

What Protection Looks Like for One Person Facing a Machine

So, back to the person holding the phone.

The honest answer to how an individual distinguishes a genuine interaction from a manufactured one is that, much of the time, they cannot by sense alone. Human detection sits close to chance. AI voices and conversationalists match or beat humans. And the factory does not need to fool you in particular. It needs to fool a small fraction of a very large number of people, and it can now afford to try them all.

What remains is a smaller set of habits that do not depend on perception at all. Agree a code word with the people whose voices you would trust in a crisis. When a request involves money, credentials or secrecy, end the conversation and re-establish contact through a channel you already hold. Treat urgency, confidentiality and a new or unusual channel, appearing together, as the signature of a procedure being bypassed. Talk to someone before you pay. None of that requires you to be a forensic analyst. All of it works whether the fake is crude or perfect.

The larger answer is that protection for one person has to be built mostly out of things that one person does not control. It looks like a bank that has a financial reason to stop the transfer, because it will pay if it does not. It looks like a telecoms network that blocks the SIM farm and the spoofed sender, because it bears a share of the loss when it fails to. It looks like a platform that faces penalties for letting a scam reach its users. It looks like anti-scam centres that can freeze money across a border within a day of the alarm being raised, satellite providers that cut off compounds, prosecutors who seize the bitcoin, and defensive AI that wastes the attackers' hours and steals their scripts.

INTERPOL's Valdecy Urquiza warned that the cost of financial crime “is not just money – it is people's life savings, their dignity, and in the worst case, their life.” The fraud factory runs on the assumption that this cost will continue to fall on individuals, one call, one video and one conversation at a time. The evidence from London, Singapore and Canberra suggests that assumption can be broken, and that the right response to industrialised deception is not a more suspicious public. It is an industrial defence. Every person on earth may now be a potential target. That does not make every person on earth responsible for being the last line of defence.

---

References and Sources

1. Black Hat USA 2026 presentation by Eric Huber, Senior Manager of Adversary Intelligence and Disruption, TD Bank, as reported in: Witts, Joel. “Research reveals how criminal gangs are using AI to improve financial scams.” Expert Insights, 4 August 2026, https://expertinsights.com/news/research-reveals-how-criminal-gangs-are-using-ai-to-improve-financial-scams; and Seals, Tara. “AI Sends Global Crime Syndicates Into Fraud Nirvana.” Dark Reading, August 2026, https://www.darkreading.com/threat-intelligence/ai-global-crime-syndicates-fraud-nirvana
2. Gressel, Gilad, Rahul Pankajakshan, Shir Rozenfeld, Ling Li, Ivan Franceschini, Krishnashree Achuthan and Yisroel Mirsky. “Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams.” arXiv:2512.16280, submitted 18 December 2025. https://arxiv.org/abs/2512.16280
3. INTERPOL. “INTERPOL report warns of increasingly sophisticated global financial fraud threat.” News release, 16 March 2026, including quotes from Secretary General Valdecy Urquiza; and the 2nd INTERPOL Global Financial Fraud Threat Assessment, 2026. https://www.interpol.int/en/News-and-Events/News/2026/INTERPOL-report-warns-of-increasingly-sophisticated-global-financial-fraud-threat
4. Organized Crime and Corruption Reporting Project. “UN Report Details Southeast Asia's Interconnected Criminal Economy.” 21 July 2026, reporting on the UNODC threat assessment “An Interconnected Criminal Ecosystem”, launched in Bangkok on 21 July 2026. https://www.occrp.org/en/news/un-report-details-southeast-asias-interconnected-criminal-economy
5. Kim, Heedou, Changsik Kim, Sanghwa Shin and Jaewoo Kang. “SCRIPTMIND: Crime Script Inference and Cognitive Evaluation for LLM-based Social Engineering Scam Detection System.” arXiv:2601.13581, submitted 20 January 2026. https://arxiv.org/abs/2601.13581
6. Erukude, Sai Teja, Viswa Chaitanya Marella and Suhasnadh Reddy Veluru. “AI-Driven Cybersecurity Threats: A Survey of Emerging Risks and Defensive Strategies.” arXiv:2601.03304, submitted 6 January 2026. https://arxiv.org/abs/2601.03304
7. Heiding, Fred, Claudio Mayrink Verdun, Simon Lermen, Andrew Kao, Vitor Albiero, Lauren Deason, Irina-Elena Veliche and Christine Lehane. “Evaluating AI Models' Capability to Automate Voice Phishing Attacks.” arXiv:2607.09970, submitted 10 July 2026, revised 20 August 2026; published in Expert Systems With Applications, volume 332, article 133620. DOI 10.1016/j.eswa.2026.133620. https://arxiv.org/abs/2607.09970
8. Down, Aisha. “Deepfake fraud taking place on an industrial scale, study finds.” The Guardian, 6 February 2026; syndicated by the Irish Examiner as “AI experts say deepfakes now at 'industrial' scale”, 6 February 2026, including quotes from Simon Mylius (MIT) and Fred Heiding (Harvard). https://www.irishexaminer.com/news/arid-41789042.html
9. Federal Bureau of Investigation, Internet Crime Complaint Center. “2025 Internet Crime Report.” 7 April 2026. https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf
10. US Department of Justice. “Scam Center Strike Force Takes Major Actions Against Southeast Asian Scam Centers Targeting Americans.” Office of Public Affairs press release, 23 April 2026, https://www.justice.gov/opa/pr/scam-center-strike-force-takes-major-actions-against-southeast-asian-scam-centers-targeting; and US Attorney's Office for the District of Columbia, “Scam Center Strike Force”, web page updated 15 October 2026, accessed 20 October 2026, https://www.justice.gov/usao-dc/scam-center-strike-force; see also Crypto Briefing, “Scam Center Strike Force restrains $52M in laundered crypto in one day”, 9 September 2026, https://cryptobriefing.com/scam-center-strike-force-52m-crypto-madagascar/
11. Federal Bureau of Investigation. “Criminals Use Generative Artificial Intelligence to Facilitate Financial Fraud.” Public Service Announcement, Alert Number I-120324-PSA, 3 December 2024. https://www.ic3.gov/psa/2024/psa241203
12. Diel, Alexander, et al. “Human performance in detecting deepfakes: A systematic review and meta-analysis of 56 papers.” Computers in Human Behavior Reports, volume 16, article 100538, December 2024, DOI 10.1016/j.chbr.2024.100538; and Köbis, Nils C., Barbora Doležalová and Ivan Soraperra. “Fooled twice: People cannot detect deepfakes but think they can.” iScience, volume 24, issue 11, article 103364, 2021, DOI 10.1016/j.isci.2021.103364.
13. Ashraf, Muhammad Tahir. “Synthetic Trust Attacks: Modeling How Generative AI Manipulates Human Decisions in Social Engineering Fraud.” arXiv:2604.04951, submitted 2 April 2026. https://arxiv.org/abs/2604.04951
14. Magramo, Kathleen. “British engineering giant Arup revealed as $25 million deepfake scam victim.” CNN, 16 May 2024, https://edition.cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk; and World Economic Forum, interview with Rob Greig, Global Chief Information Officer, Arup, February 2025, https://www.weforum.org/stories/2025/02/deepfake-ai-cybercrime-arup/
15. Bloomberg. “Ferrari Narrowly Dodges Deepfake Scam Simulating Deal-Hungry CEO.” 26 July 2024; see also Motor1, “Someone Deepfaked Ferrari CEO's Voice and Tried to Scam the Company”, July 2024. https://www.motor1.com/news/728264/ferrari-ceo-deepfake-voice-phone/
16. Singapore Police Force. “Singapore and Hong Kong Police Force Recover Over $670,000 in a Scam.” News release, 7 April 2025. https://www.police.gov.sg/media-hub/news/2025/04/20250407_singapore_and_hong_kong_police_force_recover_over_$670000_in_a_scam
17. Payment Systems Regulator. “One year on: impact of APP reimbursement on victims.” 2025, https://www.psr.org.uk/news-and-updates/latest-news/news/one-year-on-impact-of-app-reimbursement-on-victims/; and the independent evaluation published by the regulator in July 2026, as summarised in Norton Rose Fulbright, Regulation Tomorrow, “Payment fraud falls by £73m following PSR reimbursement scheme”, 1 July 2026, https://www.regulationtomorrow.com/2026/07/payment-fraud-falls-by-73m-following-psr-reimbursement-scheme/
18. Infocomm Media Development Authority and Monetary Authority of Singapore. “Implementation of Shared Responsibility Framework.” Joint press release, 24 October 2024; and MAS, “Guidelines on Shared Responsibility Framework”. https://imda.gov.sg/resources/press-releases-factsheets-and-speeches/press-releases/2024/implementation-of-shared-responsibility-framework
19. Gilbert + Tobin. “The Scams Prevention Framework legislation passes Parliament: time to get your house in order.” February 2025. https://www.gtlaw.com.au/insights/the-scams-prevention-framework-legislation-passes-parliament-time-to-get-your-house-in-order
20. Australian Competition and Consumer Commission, National Anti-Scam Centre. “Targeting scams: Report of the National Anti-Scam Centre on scams data and activity 2024.” March 2025, https://www.accc.gov.au/about-us/publications/serial-publications/targeting-scams-reports-on-scams-activity/targeting-scams-report-of-the-national-anti-scam-centre-on-scams-data-and-activity-2024; and Croft, Daniel, “Scams cost Australians over $2bn in 2025, says ACCC”, Cyber Daily, 30 March 2026, including quotes from Catriona Lowe, Deputy Chair, ACCC, https://www.cyberdaily.au/security/13400-scams-cost-australians-over-2bn-in-2025-says-accc
21. UK Home Office. “Major step for fraud prevention with landmark ban on SIM farms.” GOV.UK news story, 2025; and “Crime and Policing Act 2026 guidance: offences relating to the possession and supply of SIM farms”, GOV.UK, July 2026. https://www.gov.uk/government/news/major-step-for-fraud-prevention-with-landmark-ban-on-sim-farms
22. Arab News, via Agence France-Presse. “SpaceX says 'disabled' 2,500 Starlink devices at Myanmar scam centers.” October 2025. https://www.arabnews.com/node/2619810/world
23. US Department of Justice, US Attorney's Office for the Eastern District of New York. “Chairman of Prince Group Indicted for Operating Cambodian Forced Labor Scam Compounds Engaged in Cryptocurrency Fraud Schemes.” Press release, 14 October 2025, https://www.justice.gov/usao-edny/pr/chairman-prince-group-indicted-operating-cambodian-forced-labor-scam-compounds-engaged; Elliptic, “Prince Group targeted with $15B crypto seizure and sanctions for pig butchering operations”, October 2025, https://www.elliptic.co/blog/prince-group-targeted-with-crypto-sanctions-for-global-pig-butchering-operations; Associated Press, “Businessman extradited from Cambodia to China suspected of running vast online criminal empire”, 8 January 2026, republished by NY1, https://ny1.com/nyc/all-boroughs/ap-top-news/2026/01/08/businessman-extradited-from-cambodia-to-china-suspected-of-running-vast-online-criminal-empire; and CamboJA News, “Chen Zhi, Alleged Scam Network Kingpin, Sent to China”, 8 January 2026, https://cambojanews.com/chen-zhi-alleged-scam-network-kingpin-sent-to-china/
24. Virgin Media O2. “O2 unveils Daisy, the AI granny wasting scammers' time.” Press release, 14 November 2024, including quotes from Murray Mackenzie, Director of Fraud. https://news.virginmediao2.co.uk/o2-unveils-daisy-the-ai-granny-wasting-scammers-time/
25. Commonwealth Bank of Australia. “CommBank deploys AI bots to outsmart scammers” (partnership with Apate.ai). Newsroom, June 2025. https://www.commbank.com.au/articles/newsroom/2025/06/apate-ai.html

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The black box sits beside the entrance to Woodland Glen IV, a residence hall on the University of Kentucky's campus in Lexington. Its job is simple to describe. You walk towards the door, it looks at you, and if it recognises the geometry of your face, the lock releases. No card, no phone, no fumbling. You are home.

The device is called Rock. It is made by Alcatraz AI, a company based in Cupertino, California, and this autumn the University of Kentucky began testing it at a handful of buildings, including two residence halls, as a way for students to get into the places where they sleep without tapping a key card. In mid-September local television stations covered the installation. The libertarian magazine Reason picked the story up nationally under the headline “Scan your face to enter your dorm?” The university, for its part, has been careful and consistent on one point: enrolling your face is voluntary. Your Wildcard, the university's student ID, still works. Your phone still works. Nobody has to do this.

That is true, and it is worth taking seriously. It is also where the interesting questions begin rather than end. For many of the eighteen-year-olds who arrived at Kentucky this autumn, this may be the first building they have ever been invited to open with their face. It is not a bank vault or an airport gate or a laboratory full of expensive reagents. It is the front door of their home: the one they walk through at two in the morning, after a bad exam, or carrying shopping bags with the key card buried somewhere beneath a textbook. What they learn at that door, about what is normal, about what is asked of them, about whether their face is something they give or something that is simply taken, may shape how they respond to every door that follows.

So here is the question the black box poses. When the student with the shopping walks up and the lock clicks open before they have put anything down, what have they actually experienced? Added safety? Added convenience? Or the first of many doors that will, one day, know who they are whether or not they asked?

What the University Actually Installed

Start with what is known, because the facts are more specific than the headlines.

According to the University of Kentucky Police Department's own information page, the biometric access pilot began at Woodland Glen IV. Residents were told to expect an enrolment email near the start of the semester, sent from an Alcatraz address, with personalised instructions. The page states that biometric data is collected solely to verify access to the building, that it is not used to track students' movements or behaviour beyond building entry, and that anyone the system fails to recognise can still get in with their Wildcard or mobile ID. WKYT, the Lexington CBS affiliate, reported on 19 September that scanners were also going in at Kirwan-Blanding, another residence complex, for a two-week trial. The Lexington Herald-Leader listed further planned sites, including parts of the medical campus, several biomedical research buildings and athletics practice spaces.

How the machine sees you was explained by the university's own police chief. Joe Monroe, who has led the UK Police Department since 2009, told LEX18 that the system “looks at your face, and it assigns different points using geometry, facial geometry. They call it facial vectors. Once that number is assigned, that is who you are. There's no pictures stored.” The university's privacy language, quoted by the Herald-Leader, describes the resulting “facial signature” as “a one-way encrypted mathematical representation of your facial features.” Alcatraz's chief executive, Tina D'Agostin, put it more bluntly to WKYT: “You're zeros and ones to us once you've been scanned.” Monroe has also conceded that a security brand named after America's most famous island prison “may not be the best name for the product, and we've pointed that out.”

Two details from the local coverage matter more than they first appear. The first is that this is not the university's first experiment with biometrics. According to the Herald-Leader, Rock is replacing iris-scanning equipment from a company called Eyelock that the university described as “non-functional.” Biometric entry at UK is an existing practice being upgraded and extended, not a novelty.

The second detail should stay with anyone thinking about consent. The Herald-Leader reported that enrolment would be required for people entering the most sensitive places on campus, including the police department and research facilities, and optional for less vulnerable places such as residence halls and most athletics facilities. LEX18 filmed the police evidence room, where Deputy Chief Nathan Brown explained that entry required both a credential and a face. “This is dual authentication because this is a very obvious restricted room,” he said. In other words, the University of Kentucky is not running a voluntary facial recognition programme. It is running one facial recognition system under two sets of rules. The Kentucky Kernel, the student newspaper, reported that the technology is also in other locations requiring two-factor authentication, such as research labs. For some people on campus, a face scan is already a condition of doing their job. For students, at their front doors, it is a choice. The technology is identical. Only the policy differs, and policy is the easiest part of any system to change.

A Company Built by a Face ID Veteran

Alcatraz AI is not a campus start-up that stumbled into student housing. It is an enterprise access control company with serious money and a clear ambition. It was founded in 2016 by Vince Gaydarzhiev, who, according to the company, led hardware prototyping for the iPad and iPhone at Apple during the development of Face ID. In a 2019 interview with Security Sales and Integration, Gaydarzhiev said the idea came from noticing that physical security, even at Apple's own campus, was still done “old school,” with badges and guards at almost every door.

In April 2026 the company announced a fifty million dollar Series B round, led by BlackPeak Capital, Cogito Capital and Taiwania Capital, bringing its total funding to more than one hundred million dollars. It claimed customers among the world's largest AI data centres, major US airports, energy companies, NFL teams, Fortune 100 firms and “major universities.” D'Agostin summarised the pitch in one line: “We are the Face ID of securing physical spaces.”

That framing tells you where the product is designed to end up. Face ID on a phone is not an optional extra for enthusiasts. It is the default way tens of millions of people unlock the most personal object they own. A company that describes itself as the Face ID of buildings is not aiming to sit politely beside the key card. It is aiming to make the key card feel as dated as typing a passcode. D'Agostin said as much to WKYT: “The badge, what most campuses are using today is a 30-year-old technology.”

Alcatraz's marketing to universities makes the same argument with a student in mind. A February 2026 guide on its website, pitching “privacy-first facial authentication” to higher education, opens with an observation about young people: “Students today unlock their phones with a glance and verify banking transactions with their faces. Yet on campus, they still fumble with physical keycards.” It promises “the frictionless experience your students expect.” To its credit, the same guide also sets out a strong consent standard, stating that students “must have the right to withdraw consent and delete their templates at any time.”

The guide is also candid about where else the technology could go. Beyond residence halls, labs and athletics, it suggests that “Libraries can automate entry for late-night study sessions,” and it mentions exam proctoring. There is no public evidence that the University of Kentucky plans to put face readers on its libraries or in its exam halls. But the people selling the system already imagine a campus where the face is the credential for much more than the dorm.

The Door That Watches Everyone Else

“Voluntary enrolment” describes the relationship between the system and the person who signs up. It says rather less about everyone else who walks through the door.

The main security argument for Rock is not really about the enrolled student. It is about tailgating: someone with no right to be in a building slipping in behind someone who does. D'Agostin told WKYT that the system lets security teams “detect when this incident occurs so they can react to it in real time.” The university has framed the housing tests in the same terms, as part of a broader effort to reduce unauthorised entry and strengthen residence hall safety, with the face option added as a hands-free extra rather than a replacement for existing credentials. A student interviewed by WKYT said they liked it “just because safety is the number one priority for the college.”

To detect a tailgater, though, the machine has to pay attention to people who have not enrolled. Alcatraz's product page is explicit. The reader “authenticates the enrolled person in under a second, then keeps monitoring the doorway as the door opens,” and if someone enters without authenticating, “a picture is captured at the door” and logged for investigation. The security operations centre, the page says, receives a snapshot of that person along with the time of the event and the device that triggered it. A 2021 announcement of the feature, reported by Security Systems News, described the same behaviour.

Neither the university nor local reporting has said whether tailgating snapshots are switched on at Woodland Glen IV or Kirwan-Blanding, and it would be wrong to assume that they are. But the design matters regardless of how any one site is configured. In a residence hall, the person walking in close behind a resident is very often not an intruder. It is a roommate who left their card upstairs, a friend coming to study, a parent carrying a mini-fridge. Some of those people will have chosen not to enrol. Some will never have been asked. A system whose core safety feature involves noticing, and potentially photographing, the people who did not authenticate is a system whose voluntariness applies to only one side of the door.

This is the structural problem that has dogged every face recognition product sold as a convenience. The person who opts in gets the benefit. The people around them bear some of the cost, and they are not the ones who were asked.

Voluntary Is a Setting, Not a Promise

Consent is often spoken about as though it were a fixed property of a technology. With access control, it is closer to a configuration option.

Alcatraz's public help centre describes three ways to enrol someone. Manual enrolment takes a single badge swipe and a few prompts on the Rock's screen. Web enrolment sends an invitation to register with your own phone or computer camera, which matches the emailed process the University of Kentucky describes. The third mode is auto-enrolment. When a Rock is configured this way, the help centre explains, “each time the user swipes their badge and enters through the door, the Rock is learning their face.” It needs four swipes, at least five minutes apart, from people who already have access to that door. In 2019 Security Sales and Integration described this self-enrolment as requiring no administration or employee action, “switching to face-only mode autonomously as soon as the employee is seen enough times.”

The direction of travel in the product is towards less friction at sign-up, not more. In May 2026 Alcatraz announced platform updates that, according to Biometric Update, let administrators apply access control rules to enrolment and added visual guidance to speed up biometric registration. D'Agostin said the changes were about making sign-up simpler and continuing to “automate operations and deliver a seamless authentication experience.” In September, at the GSX security show in Atlanta, the company went further. It named its software layers: RockOS, the firmware on each device that handles enrolment, liveness detection, authentication and tailgating detection, and Rock Cortex, a platform for verification, user and device management, integrations, monitoring, analytics and compliance. According to SecurityInfoWatch, a guided enrolment process that removes the need for users to move or tilt their heads was slated for general availability in October 2026, which Alcatraz says will make onboarding faster and cut failed enrolments. None of that is sinister. Making sign-up easy is what any product team would do. But ease is not the same thing as deliberation, and a choice that takes two seconds is rarely a choice anyone remembers making.

There is no evidence that the University of Kentucky uses auto-enrolment in its halls. Its published process is invitation-based and opt-in. The point is not that anyone is being enrolled without their knowledge. It is that the hardware on the wall can operate under a quite different consent model, and the difference is a setting chosen by an administrator. The student's choice is protected by a policy, not by the physics of the device.

Policies drift, and the most instructive recent example is an airport. The US Transportation Security Administration has long described its facial comparison checks as optional. In May 2025 a bipartisan group of senators, including Jeff Merkley, John Kennedy, Ed Markey and Roger Marshall, reintroduced legislation to rein the programme in. “Americans have the right to opt out of using TSA's facial recognition at the airport, and we need to protect that right,” Merkley said. Kennedy complained that the agency scans travellers “without even making it clear that they can opt out.” Two months later the Algorithmic Justice League, founded by the computer scientist Joy Buolamwini, published “Comply To Fly?”, drawing on travellers' accounts from 91 US airports. According to Biometric Update's summary, 99 per cent of respondents said officers had not told them verbally that they could opt out, many experienced the scans as mandatory, and some who declined reported hostility and extra scrutiny.

Nobody made the scans compulsory. They did not need to. When the scan is the fast lane, when the signs are small, when refusal is treated as an inconvenience, “optional” becomes something you have to insist on rather than something you are offered. That is how convenience quietly becomes expectation. It does not happen through a decree. It happens through queues.

The media scholars Nora Draper and Joseph Turow have a name for the feeling this produces. In a 2019 paper in New Media and Society, they described “digital resignation”: the condition in which people want to control what companies know about them but feel unable to, and give up. They argued that this resignation is not a flaw in consumers but something cultivated by routine corporate practices that make resistance tiring and pointless. A student who opts out in a hall where most neighbours have opted in will not be punished. They will simply be the one still tapping a card, still the exception. Most people do not choose to be the exception for long.

Home Is a Different Kind of Door

Privacy theory offers a useful way to explain why a residence hall feels different from a laboratory.

In a 2004 article in the Washington Law Review, the philosopher Helen Nissenbaum proposed that privacy is best understood as “contextual integrity.” Every setting carries norms about which information flows are appropriate within it, and the violation lies not in the information itself but in its movement out of the context where it belongs. In the police evidence room, the norm is scrutiny; anyone entering is expected to account for being there, and a face scan fits. A home is governed by the opposite norm. It is the one place you are supposed to be able to come and go without explaining yourself. That a student's home is owned by an institution, patrolled by its police and opened with its credential does not erase the norm. It makes it more fragile.

Then there is the question of who is being taught what. Most first-year undergraduates at Kentucky in 2026 were born around 2007 or 2008. Apple introduced Face ID with the iPhone X on 12 September 2017, when they were nine or ten. For this cohort, unlocking a device with a glance is a childhood habit, which is exactly what Alcatraz's marketing leans on. The dorm door extends that habit from the private object in your pocket to a shared building run by someone else. Face ID, by Apple's design, answers to you. The building answers to its owner.

What that habit looks like in practice surfaced a few weeks into term. The Kentucky Kernel spoke to Leah Henderson, a first-year student living in Woodland Glen IV, who said the university had not yet told her about the technology. She first assumed the cameras at the doors were recording students. Once she learned their purpose, she called the system innovative and convenient. “It's kind of cool. Maybe I don't have to use my phone all the time to get in,” she said. Purani Kannan, a sophomore and resident advisor at Kirwan-Blanding, told the paper that many students don't know much about the technology yet. Nothing in either account suggests anything improper; enrolment remains opt-in. But together they show how quickly an unexplained camera becomes a welcome convenience, and how little of that journey passes through anything resembling deliberate consent.

Mark Andrejevic and Neil Selwyn, both at Monash University, examined facial recognition in schools in a 2020 paper in Learning, Media and Technology. They warned that it risked altering schooling “along divisive, authoritarian and oppressive lines,” and worried about how such systems normalise surveillance for young people, teaching them to treat constant identification as the ordinary texture of institutional life. Their focus was compulsory schooling rather than universities, and Kentucky's pilot is opt-in. But the point travels: habits formed in institutions shape expectations outside them.

Not all research points one way. A 2022 study in Frontiers in Psychology surveyed 391 college students in mainland China and Taiwan about face recognition access control on their campuses. Students who found the systems useful and easy were more likely to use them, and that acceptance was associated with a stronger sense of school identity and belonging. The authors acknowledged “information leakage anxiety” but did not measure it. The finding is a useful corrective: for some students, a door that recognises them may feel like a place that knows them. That is precisely why normalisation matters. A technology that feels welcoming is easy to accept without examining what else comes with it.

What a Card Swipe Already Knows

It is tempting to treat the face as the moment surveillance begins. On most American campuses, the key card has been quietly generating data for years.

UKNow, the university's news service, described in August 2024 what its Wildcard Mobile ID could already do: residence hall access, payment at UK Dining, rides on LexTran buses, verification at athletic events, borrowing from UK Libraries and entry to recreational facilities. The same article said the university hoped to extend the mobile experience to the entire campus community, with faculty and staff to follow. Credentials on campus do not stay in one place; they spread to wherever it is convenient to use them. Each interaction is, potentially, a timestamped record of where a student was.

What institutions can do with such records is not hypothetical. In 2018 Sudha Ram, a professor of management information systems at the University of Arizona, drew attention for research using anonymised data from the university's CatCard ID system to predict which first-year students would drop out, based on how often they swiped into halls, libraries and recreation centres and where they spent money. “Many times, students who tend to drop out are people who have not established social relationships and who haven't established a regular routine on campus,” Ram told Arizona Public Media. Contemporary reports said the model identified roughly 85 to 90 per cent of students who went on to leave. The work was framed as supportive. It is also a textbook case of function creep: data collected to open doors and sell sandwiches becoming raw material for behavioural prediction. It happened with cards. Nothing about a face makes it less likely to happen with faces.

The architecture makes the connection concrete. According to Alcatraz, a Rock profile is linked to the user's badge ID in the existing access control system, so the face simply stands in for the card. Wherever the card is already a key, the face could, with the right hardware and a policy decision, become one too. Nobody has announced plans to put face readers in Kentucky's dining halls, libraries or gyms. But the path from one door to many is an installation job, not a reinvention.

Beyond campuses, the warnings are sharper. In late 2022 Madison Square Garden Entertainment used facial recognition at Radio City Music Hall to identify and eject Kelly Conlon, a lawyer chaperoning her daughter's Girl Scout troop, because her firm was in litigation against the company. Security technology became a tool for settling commercial scores. In Western Australia in 2021, police obtained data from SafeWA, a Covid-19 check-in app whose users had been told only contact tracers would see it. The government rushed through a law restricting its use. “Because it is so important to maintain community confidence in SafeWA, we've come down in favour of nobody,” said Attorney General John Quigley. “Not the CCC, not someone under FOI, not the police. Nobody sees this information.”

Note the order of events. The promise came first, its breach second, and the law protecting it only after the public found out. Anyone weighing a campus assurance that biometric data will not be used for tracking should ask what Western Australians eventually had to ask: what, other than the institution's goodwill, makes that true?

Kentucky's Law Stops at the Campus Gate

In Kentucky, the honest answer appears to be: not very much.

The Kentucky Consumer Data Protection Act, signed by Governor Andy Beshear in April 2024, took effect on 1 January 2026. It treats biometric data used for identification as sensitive and requires opt-in consent before a covered business processes it. That sounds like exactly the protection a student would want. But the Act states that its core provisions “shall not apply to any” city, state agency or political subdivision, nor to any “institution of higher education.” The University of Kentucky is a public university; on the plain text, the state's new privacy law does not appear to reach it. The Act also provides no private right of action, so even where it applies, only the attorney general can enforce it.

Illinois's Biometric Information Privacy Act of 2008, the strongest such law in the country, would not necessarily help either. BIPA requires written, informed consent before a private entity collects a face template and lets individuals sue, which is why it has produced some of the largest privacy settlements in history. But it excludes state and local government agencies, and private universities have slipped through too. In November 2022 a federal court dismissed a BIPA class action against DePaul University over a remote proctoring tool that captured students' facial data, ruling that because DePaul administered federal student loans it counted as an exempt financial institution.

Federal law offers a partial safeguard. Regulations under the Family Educational Rights and Privacy Act define a “biometric record” as a record of one or more measurable biological or behavioural characteristics that can be used for automated recognition of an individual, listing facial characteristics as an example, and treat it as personally identifiable information. But FERPA governs disclosure of education records. It does not decide whether an institution should build a face-matching system, and it says nothing about keeping enrolment optional.

Some policymakers have drawn firmer lines. In September 2023 New York's education commissioner, Betty Rosa, banned the purchase and use of facial recognition in the state's elementary and secondary schools, concluding after a state technology report that the risks outweighed the claimed benefits. That ban covers children, not adult university students. But it shows that a government can decide some settings are inappropriate for the technology, rather than leaving the question to procurement officers one pilot at a time.

In Kentucky, then, the durability of “voluntary” rests largely on institutional policy and on vendor commitments. The Herald-Leader reported that the company says it keeps no cloud biometric database and shares no data with third parties or government databases. Speaking to the Kentucky Kernel, D'Agostin was blunter still: “Basically, your face is a digital key, and when you come to the door, if that key matches, the door opens.” Of the data behind it, she said: “We don't have access and we don't store it, so we can't sell the data.” The Kernel reported that the data remains with the university, which manages it. Dani Jaffe, the university's associate director of strategic and crisis communications, said: “All biometric data is stored on secure servers and is never shared with third parties.” She added that the university “will continue to evaluate and implement tools” as security needs evolve. That clarifies where the promise lives. If the vendor holds nothing, the templates' fate rests on the university's own policy, and the state law that would bind a private company does not bind the university. Those are meaningful assurances. They are also promises made by the parties who benefit from the system's expansion, enforceable mainly by the parties who made them.

The Campaign That Already Won Once

None of this is new. Six years before Kentucky installed its scanners, the digital rights group Fight for the Future and the student organisation Students for Sensible Drug Policy set out to keep facial recognition off American campuses, publishing a scorecard of institutions and pushing resolutions through student governments.

Their best-known victory came at UCLA. In early 2020, as the university considered adding facial recognition to its security cameras, Fight for the Future ran more than 400 photographs of UCLA athletes and faculty through Amazon's face recognition software against a mugshot database. According to Inside Higher Ed, the test produced 58 false matches, most of them people of colour. UCLA dropped the plan. “UCLA will not pursue the use of facial recognition software technology,” Michael Beck, the administrative vice chancellor, said in a statement, concluding that “the potential benefits are limited and are vastly outweighed by the concerns of the campus community.” Evan Greer of Fight for the Future said the victory would “send a pretty strong message to any other administration who is considering doing this.” By then around 50 universities had publicly said they had no plans to use the technology.

Kentucky's pilot differs in important ways. UCLA's plan would have scanned everyone in view of campus cameras; Kentucky's is an opt-in door lock that verifies one enrolled person at a time. A mismatch at a dorm door means using your card, not being flagged as a suspect, and IEEE Spectrum has reported that best-case false negative rates are around two in 1,000. Michael Murray, a University of Kentucky law professor who studies artificial intelligence, told LEX18 that the university seemed to be “taking it quite seriously” and trying “to roll it out with a very low threshold of risk,” while cautioning that AI “is not cost-free, and it's not risk-free.”

Still, the old unease has not gone away. Local online discussion, summarised by the Lexington Times, was dominated by privacy worries, with some commenters calling the scanners the start of a surveillance network and others doubting that a face reader would stop anyone following a resident through an open door. Reason's Elizabeth Nolan Brown questioned whether the security case justified the system at all, and suggested institutions might be “dreaming bigger” than a limited dorm pilot. There is no sign yet of an organised Kentucky campaign on the UCLA model. But the arguments that won in 2020 are still available, and a pilot is the stage at which they are easiest to make.

Safety, Convenience, or the First of Many Doors

So what does the student carrying the shopping actually experience when the lock clicks open?

On the evidence available: mostly convenience, described as safety. The system is genuinely easier than hunting for a card, and nobody should sneer at that. Losing a Wildcard late at night is a real problem, and a door that knows you is a real comfort. The safety case is harder to pin down. Its main security function depends on the door paying attention to people who have not enrolled, in a building where the person behind you is usually a friend, and the university has not published data on how often tailgating causes harm in its halls.

What the student is less likely to notice is the thing that matters most over time. They are not only choosing between a card and a face. They are helping to establish a norm. Every enrolment makes the face reader more useful, the card tapper more unusual and the next expansion easier to justify. That is not a conspiracy. It is how defaults work.

It would be unfair to say the University of Kentucky has done anything wrong. On the public record, it has kept the dorm pilot optional, maintained alternatives, avoided storing photographs and explained the system openly. That is better practice than many face technology deployments. But it would be naïve to treat “voluntary” as a permanent feature rather than a present policy, in a state whose new privacy law excludes public universities, using hardware that is already mandatory elsewhere on the same campus.

If voluntariness is to last, it needs to be written down in forms that are hard to quietly revise. A university serious about the standard in Alcatraz's own guide could publish, before any expansion, answers to a short list of questions. Which features are switched on, including tailgating snapshots of people who have not enrolled? How long are templates and event logs kept? The police department's pilot page does not say, promising only that details on data use and retention will be shared during enrolment. Who can see them, and when would they go to police? What would have to happen, and who would have to agree, before face entry reached libraries, dining halls or exam rooms? And will a card or phone always remain a fully equal alternative, rather than the slow lane?

None of those questions requires opposing the technology. They require only that the choice offered to students today is still a choice for the students who arrive next year.

The first door a young person learns to open with their face is a lesson in what their face is for. It can teach them that their body is a password they control, lent to an institution for a specific purpose on specific terms. Or it can teach them that being recognised is simply the price of coming home. The black box beside Woodland Glen IV cannot tell the difference. The people who decide how it is configured can.

---

References

1. University of Kentucky Police Department, “Biometric Access”, police.uky.edu, accessed 24 September 2026. https://police.uky.edu/safety/biometric-access
2. Alyssa Williams, “UK installs AI facial scanning technology at residence halls”, WKYT, 19 September 2026. https://www.wkyt.com/2026/09/19/uk-installs-ai-facial-scanning-technology-residence-halls/
3. Leigh Searcy, “University of Kentucky tests AI facial scanning for campus building access”, LEX18, 17 September 2026. https://www.lex18.com/news/covering-kentucky/university-of-kentucky-tests-ai-facial-scanning-for-campus-building-access
4. Jesse Fraga, “UK installing AI scanners that take 'facial signatures' at some buildings. Here's where”, Lexington Herald-Leader (via NewsBreak), September 2026. https://www.newsbreak.com/lexington-herald-leader-1592576/4889066180360-uk-installing-ai-scanners-that-take-facial-signatures-at-some-buildings-here-s-where
5. Elizabeth Nolan Brown, “Scan your face to enter your dorm? University of Kentucky installs facial recognition tech around campus”, Reason, 23 September 2026. https://reason.com/2026/09/23/scan-your-face-to-enter-your-dorm-university-of-kentucky-installs-facial-recognition-tech-around-campus/
6. Alcatraz, “Alcatraz Raises $50 Million in Series B Funding”, press release, 2 April 2026. https://www.alcatraz.ai/press-release/alcatraz-raises-50-million-series-b
7. Alcatraz, “Privacy-First Facial Authentication Guide for Universities”, 2 February 2026. https://www.alcatraz.ai/blog/privacy-first-facial-authentication-the-modern-standard-for-higher-education
8. Alcatraz, “Tailgating Detection System”, product page, accessed 24 September 2026. https://www.alcatraz.ai/solutions/tailgating-detection-system
9. Alcatraz AI Help Center, “Auto, Manual and Web Enrollment”, accessed 24 September 2026. https://support.alcatraz.ai/enrollment
10. Security Sales and Integration, “Startup Alcatraz AI Aims to Replace Badges With Frictionless Access Control”, 14 March 2019. https://www.securitysales.com/news/alcatraz-frictionless-access-control/91116/
11. Office of Senator Jeff Merkley, “Before Busy Summer Travel Season, Merkley, Kennedy, Markey, Marshall Push to Rein in TSA Facial Recognition Technology at Airports Across U.S.”, 8 May 2025. https://www.merkley.senate.gov/before-busy-summer-travel-season-merkley-kennedy-markey-marshall-push-to-rein-in-tsa-facial-recognition-technology-at-airports-across-u-s/
12. Algorithmic Justice League, “Comply To Fly? How Airport Travelers Experience TSA's Facial Recognition Experiment”, 2025. https://www.ajl.org/flyreport
13. Nora A. Draper and Joseph Turow, “The corporate cultivation of digital resignation”, New Media and Society, vol. 21, no. 8, 2019, pp. 1824-1839. https://doi.org/10.1177/1461444819833331
14. Helen Nissenbaum, “Privacy as Contextual Integrity”, Washington Law Review, vol. 79, no. 1, 2004, pp. 119-157. https://nyuscholars.nyu.edu/en/publications/privacy-as-contextual-integrity
15. Mark Andrejevic and Neil Selwyn, “Facial recognition technology in schools: critical questions and concerns”, Learning, Media and Technology, vol. 45, no. 2, 2020, pp. 115-128. https://doi.org/10.1080/17439884.2020.1686014
16. Wang, Hou, Hong, Yang and Zhang, “Impact of Face-Recognition-Based Access Control System on College Students' Sense of School Identity and Belonging During COVID-19 Pandemic”, Frontiers in Psychology, 2022. https://pmc.ncbi.nlm.nih.gov/articles/PMC8889067/
17. Steve Jess, “Can the Way Students Use ID Cards Predict Whether They Drop Out?”, Arizona Public Media, 2 April 2018. https://news.azpm.org/p/news-articles/2018/4/2/126689-can-the-way-students-use-id-cards-predict-dropouts/
18. ABC News (Australia), “Police access SafeWA app data for murder investigation, prompting urgent law change”, 15 June 2021. https://www.abc.net.au/news/2021-06-15/safewa-app-sparks-urgent-law-change-after-police-access-data/100201340
19. Kentucky General Assembly, House Bill 15 (2024 Regular Session), Acts Chapter 72, Kentucky Consumer Data Protection Act. https://apps.legislature.ky.gov/law/acts/24RS/documents/0072.pdf
20. Privacy World, “Federal Court Dismisses Biometric Privacy Class Action Brought Against University, On Basis It Was a Regulated 'Financial Institution'”, November 2022 (Powell v. DePaul University, N.D. Ill., No. 21-C-3001). https://www.privacyworld.blog/2022/11/federal-court-dismisses-biometric-privacy-class-action-brought-against-university-on-basis-it-was-a-regulated-financial-institution/
21. New York State Education Department, “State Education Department Issues Determination on Biometric Identifying Technology in Schools”, 27 September 2023. https://www.nysed.gov/news/2023/state-education-department-issues-determination-biometric-identifying-technology-schools
22. Lilah Burke, “UCLA drops plan to use facial recognition security surveillance, but other colleges may be using technology”, Inside Higher Ed, 20 February 2020. https://www.insidehighered.com/news/2020/02/21/ucla-drops-plan-use-facial-recognition-security-surveillance-other-colleges-may-be
23. IEEE Spectrum, “Facial Recognition Errors Affect Millions Globally”, 30 March 2026. https://spectrum.ieee.org/facial-recognition-gone-wrong
24. Keyonna Henson, “UK brings AI facial authentication scanners to campus”, Kentucky Kernel, 28 September 2026. https://kykernel.com/128434/news/uk-brings-ai-facial-authentication-scanners-to-campus/
25. Rodney Bosch, “Alcatraz Introduces RockOS, Rock Cortex for Facial Authentication Platform”, SecurityInfoWatch, 17 September 2026. https://www.securityinfowatch.com/access-identity/biometrics/facial-recognition-solutions/news/55406004/alcatraz-introduces-rockos-rock-cortex-for-facial-authentication-platform

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Somewhere behind a police call in Heber City, Utah, a television was playing Disney's The Princess and the Frog. The officer's body-worn camera picked it up, because body-worn cameras pick up everything: the hum of a refrigerator, a dog, a film. That audio went into an artificial intelligence system the department had bought to turn recordings into police reports. The system did exactly what it was built to do. It listened, it narrated, and it produced a document in which the officer turned into a frog.

The story went around the world in January 2026. Sergeant Rick Keel of the Heber City Police Department told FOX 13 News that “the body cam software and the AI report writing software picked up on the movie that was playing in the background, which happened to be 'The Princess and the Frog.'” He added the line every department has since repeated in some form: “That's when we learned the importance of correcting these AI-generated reports.” FOX 13 reported the offending narrative came from Draft One, sold by Axon, the largest supplier of body cameras to American police. Heber City had been trialling it alongside a rival called Code Four. The department kept using AI. It simply added more review.

Here is the thing about the frog. It is the safest error the technology can make, because it is self-cancelling. No sergeant approves it, no prosecutor files it, no judge reads it without laughing, and no defendant goes to prison because of it. An error that absurd is a smoke alarm, not a fire. It tells you the machine is in the building.

The error that should frighten you is the one that reads perfectly. It is grammatical, plausible, consistent with the rest of the narrative, and wrong by a single preposition. It is the difference between a suspect who reached toward his waistband and a suspect who reached into it. One describes an ambiguous movement. The other describes a man going for a gun, and can justify a shooting, a search, a charge, a plea. They differ by two letters, and no reviewer on earth reliably catches that at eleven at night at the end of a twelve-hour shift, six calls deep, with the machine's version already on the screen looking finished.

That asymmetry is the whole problem. And Wyoming has just walked into it.

What the Cowboy State Bought Before It Decided Anything

On 23 August 2026, Cowboy State Daily reporter Kolby Fedore published an account of police departments in Casper, Evanston and Sheridan each testing artificial intelligence to write reports and look at evidence. Only one vendor was named. Evanston, according to Police Chief Mike Vranish, is testing Code Four, software founded in April 2025 by two MIT students, George Cheng and Dylan Nguyen, and now used by roughly 165 law enforcement agencies. Code Four, Fedore noted, markets itself as “The AI Detective That Never Sleeps.”

The company's own site is less whimsical and more instructive. It promises a 65 per cent average reduction in report writing time, fifty dollars per officer per month, and eighty thousand dollars in annual budget saved per department. Its report product “ingests audio, visual, and other meta data from each frame of an officer's body-worn camera to create a comprehensive narrative of every situation.” A companion product, INSIGHTS, described in sponsored content Cheng wrote for the trade publication Police1 in July 2026, pulls “digital evidence, records and jail-management data, body-camera footage, phone dumps, call logs, recorded audio and case files into a single layer a detective can question directly.” It can draft warrants, affidavits and prosecutor jackets.

Read that list slowly. A warrant application persuades a judge to let the state enter your home; an affidavit of probable cause persuades a judge to hold your body. These are not administrative artefacts. They are the hinges on which constitutional rights swing.

Fedore's reporting established the gap that gives this story its edge. No Wyoming law specifically requires an original AI-generated police report to survive after an officer edits it, and none requires the finished report to disclose that a machine helped write it. The technology is in the building. The rules are not.

Gay Woodhouse, Wyoming's thirty-second attorney general from 1998 to 2001 and the first woman to hold the office, previously an assistant United States attorney and chief deputy attorney general, told Cowboy State Daily that “the thing that worries me about AI is just the ability to hallucinate,” and that Wyoming needs “to get on it.” She put her finger on the mechanism: “The main thing in criminal prosecutions is disclosure.” And then, with the dry economy of a former prosecutor: “It'd be awful to lose a case because you used AI and didn't disclose it.”

Casper criminal defence attorney Keith Nachbar called it “uncharted territory” and delivered the sentence that ought to be nailed above every police chief's desk in the state: “AI isn't going to be taking the witness stand.” Then a question nobody in Wyoming can answer: “How do you look behind the curtain and figure out what AI thought?”

Evanston Police Chief Mike Vranish supplied the counterweight every department offers. AI can do in minutes what takes officers hours. But a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. So, said Vranish, “everything's got to be verified and validated by an actual person.”

That last sentence is the load-bearing beam of the entire industry. We will come back to whether it holds.

Two Letters and a Life

A generative model producing a police narrative from body-camera audio and video is not reasoning about what happened. It is producing the most probable next token given everything before it. When the input is clear, the output is usually fine. When the input is ambiguous, and a real incident is nothing but ambiguity, the model does not flag the ambiguity. It resolves it, picking the likelier phrasing and writing it with the same fluency it uses for the parts it got right. There is no tonal difference between the sentences it knows and the sentences it guessed.

Human officers make errors too, and police reports have always been contested documents. The difference is the shape of the error. A tired officer produces a report with the texture of exhaustion: gaps, hedges, “I believe”, “it appeared that”, the occasional flat contradiction a defence lawyer can hold up in cross-examination. Those artefacts are evidence, telling a jury how confident the witness actually was. Machine prose has no such texture. It is uniformly confident, smoothing hedges into assertions because assertions are more probable in the training distribution of police reports than hedges are. The uncertainty that lived in the officer's memory is laundered out before anyone reads it.

The research base bears this out with unusual precision. In May 2026, criminologist Ian T. Adams of the University of South Carolina and colleagues Matt Barter, Kyle McLean, Irick A. Geary Jr, Alexis Fabila and Josh McCrain published a study on CrimRxiv titled “A Good College Essay but a Bad Police Report.” They ran a triple-blind evaluation in which 92 expert reviewers, sergeants and above with an average of 21.8 years of responsibility for approving reports, assessed 80 police reports, 20 drafted with Axon's Draft One. The reviewers did not know which was which.

Two findings matter. AI-assisted reports were rated substantively and significantly worse on accuracy, an effect moving a report from roughly the fiftieth percentile to the thirty-sixth, and text analysis found them longer-worded, more complex and less readable. They read like good college essays. They were worse police reports. And the expert reviewers could not tell which reports were written with AI. Their accuracy on that question was no better than a coin flip.

Hold those together. The exact people whose review is supposed to be the safeguard rated AI reports as less accurate whilst being unable to identify them. They sensed something was off and could not locate it. That is the signature of a failure mode human review is structurally poor at catching.

The Signature Stays, the Perception Goes

A police report's evidentiary power does not come from its literary quality. It comes from provenance. It is a contemporaneous account written by a person who was present, who perceived events through functioning senses, who can be cross-examined about those perceptions, who can be impeached with prior inconsistent statements, and who faces professional and sometimes criminal jeopardy for falsifying it. Every one of those guarantees attaches to a human being standing in a room.

Chief Vranish put it better than most legal scholars: a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. That is not a quaint remark about sensor limitations. It is a statement about the epistemology of the document. The camera captures a slice of a person's perception; the model writes a narrative from the slice and presents it as the whole.

What AI drafting does, quietly, is sever the sentence from the perception whilst leaving the signature in place. The officer still signs. The report still enters the file as his account. But a proportion of its sentences, and nobody can say which, originated not in his memory of the room but in a model's reconstruction of what a room like that usually sounds like.

This is what the machine is for. Comprehensiveness, which Code Four promises, is precisely the property a human account does not have and should not claim. Officers miss things, and the gaps in an honest report are informative. A machine that fills them is not improving the document; it is contaminating it with plausible invention.

The academic literature has caught up faster than the statutes have. In March 2026, a team led by Muffy Calder and Marion Oswald, with Elizabeth McClory-Tiarks, Michele Sevegnani and Evdoxia Taka, submitted a paper to arXiv titled “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” They mapped 15 policing tasks that large language models could plausibly handle, from transcription and translation to witness statement generation, corroboration identification and police report generation, and set against them 17 categories of risk illustrated with more than 40 worked examples of how those risks propagate through a case.

Their taxonomy of output risk begins with completeness, meaning omissions and additions, and accuracy, meaning outright factual error. In one proof-of-concept trial they describe, a system produced a description of a witness as “a woman of average height and brown hair” when no such description had been given in the interview. Nobody was turned into an amphibian. A physical description simply appeared, from nowhere, in a document later read as a record of what a witness said. That is the plausible error, and it survives review because there is nothing about it to catch.

The Draft That Was Never Born

Now the design question, where this stops being a technology story and becomes a legal one.

In July 2025, Electronic Frontier Foundation investigators Matthew Guariglia and Dave Maass published a records-based investigation into Axon's Draft One under the headline “Axon's Draft One Is Designed to Defy Transparency.” Their central finding was not that the software hallucinated. It was that the software forgets.

Draft One, they found, does not save the draft it generates, nor any edited versions. An officer generates a narrative, copies it into the report, and the draft disappears when the window closes. The EFF quoted an Axon official explaining the rationale with startling candour: “the last thing we want to do is create more disclosure headaches for our customers.”

Sit with that sentence. It is not an engineering trade-off. It is a legal strategy. The product does not retain the machine's first draft because retaining it would generate discoverable material, and discoverable material creates work for prosecutors and ammunition for the defence.

The downstream effects landed where you would expect. Auditing meant trawling individual logs or keyword-searching, and only where a department had voluntarily inserted disclosure language. The Palm Beach County Sheriff's Office, which required a disclosure line, could identify more than three thousand Draft One reports. The Lafayette Police Department could not identify one, having no marker to search for.

Non-retention is not an oversight. It is the feature that makes the practice legally survivable, converting an auditable pipeline into an unauditable one by destroying evidence before anyone has standing to ask for it.

The constitutional problem arrives immediately. Under Brady v. Maryland, decided in 1963, prosecutors must disclose evidence favourable to the accused that is material to guilt or punishment, regardless of good faith. Under Giglio v. United States, decided in 1972, that duty extends to information bearing on the credibility of government witnesses. A machine-generated first draft that differs materially from the signed report is, by definition, impeachment material about the officer's account. If the report says the suspect reached into his waistband and the draft said toward, the defence is entitled to know. If the officer corrected a dozen machine errors, that speaks to how much of the final document is his own recollection. If he corrected none, that speaks louder still.

You cannot disclose what does not exist. And when the non-existence is engineered by the vendor for the stated purpose of avoiding disclosure headaches, the ordinary comfort that Brady is a duty rather than a preservation mandate looks thin.

The Calder and Oswald team recorded what happens when a force takes preservation seriously. In a Hertfordshire Constabulary pilot of an AI interview assistant, the need “to disclose all versions of the transcript and statement produced by officers when using ADA ... impacts on workload for the Disclosure officer.” Doing this lawfully is expensive. Doing it cheaply means not doing it lawfully.

The Witness Who Cannot Be Called

Nachbar's line, that AI is not going to be taking the witness stand, is not a flourish. It sits on an unresolved constitutional question American courts have circled for twenty years.

The Sixth Amendment guarantees a defendant the right to be confronted with the witnesses against him. In Crawford v. Washington, decided in 2004, the Supreme Court held that testimonial statements cannot be admitted unless the declarant is unavailable and the defendant has had a prior opportunity for cross-examination. In Melendez-Diaz v. Massachusetts, decided in 2009, Justice Scalia wrote that forensic certificates of analysis were testimonial statements akin to affidavits, so the analysts who prepared them were witnesses the defendant could confront. In Bullcoming v. New Mexico, decided in 2011, Justice Ginsburg wrote for a five to four majority that the state could not launder that requirement by putting a different analyst on the stand to describe a report he had neither signed, performed nor observed. Surrogate testimony does not satisfy the clause.

The line those cases draw is about accountability for accusation. The state does not get to accuse through a document and then produce someone who can only vouch for the process in general terms.

An AI-drafted report sits awkwardly across that line, in a way that favours the prosecution. Machine output is generally not treated as a “statement” by a “witness” at all, because the Confrontation Clause has been read to govern human declarants. The Berkeley law professor Andrea Roth mapped this terrain in “Machine Testimony,” published in the Yale Law Journal in 2017. Machine sources of accusation, particularly proprietary software built for litigation, may function as witnesses against a defendant, she argued, and “black box dangers,” meaning errors that render a machine false by design, inarticulate or analytically unsound, are analogous to the hearsay dangers confrontation exists to expose. Machine testimony substituted for human testimony lets the state evade responsibility for its accusations.

Now apply it. The officer takes the stand. He is confrontable, and can be asked what he saw, what he smelled, what he remembers. But the sentence being challenged may not have originated with him. It came from a model. He can be asked whether he agrees with it now, which is a different question. And the thing that produced it cannot be cross-examined on its reliability, its error rate, its behaviour when the audio is poor, or its tendency to resolve ambiguity towards the incriminating reading.

The officer becomes, functionally, a surrogate witness for a process he did not perform and cannot describe. Bullcoming says the state cannot do that with a laboratory analyst. Nothing yet says it cannot do it with a language model. That gap is Nachbar's question about looking behind the curtain, restated in constitutional terms, and the current answer in Wyoming is that you cannot, because the curtain does not exist and neither does the draft behind it.

When the File Becomes the Memory

The sharpest point in the Cowboy State Daily piece is also the least discussed anywhere else. Fedore's account of Nachbar's concern runs like this. Six months on, an officer may no longer independently remember whether a suspect reached toward his waistband or into it, precisely what somebody said, or where everyone was standing. He returns to the report to refresh his memory, as officers routinely do before testifying. If the machine put the wrong detail there first, the report can begin informing the memory it was supposed to preserve.

This practice is not a loophole. It is formally sanctioned. Federal Rule of Evidence 612, and its analogues across the states, governs writings used to refresh a witness's memory, granting the adverse party the right to produce the writing, inspect it, cross-examine on it and introduce relevant portions. The rule assumes a document written by the witness, from the witness's own perception, reactivating the witness's own encoded memory. It has teeth, too. Under subsection ©, if a writing is not produced as ordered and the prosecution is at fault in a criminal case, the court must strike the witness's testimony or, if justice so requires, declare a mistrial. That mandatory remedy was written for a world in which the writing still exists.

The psychological literature has spent fifty years showing that the assumption underneath Rule 612 is fragile. The misinformation effect, the body of work most associated with Elizabeth Loftus, who published a retrospective on the idea's history in Legal and Criminological Psychology in 2026, shows that memory for an event is not sealed at the moment of the event. Post-event information modifies the memory itself, and misleading detail embedded in an apparently trustworthy narrative account is one of the classic delivery mechanisms. People do not merely repeat the misinformation. They come to remember it, with confidence, as something they perceived.

Now consider what an AI-drafted report is, structurally. It is post-event information, embedded in an apparently trustworthy narrative account, carrying the officer's own signature, which makes it maximally authoritative to him. It is reviewed under time pressure, then read again months later to reconstruct details he no longer independently recalls.

That is not a document refreshing a memory. That is close to an optimal protocol for installing one.

The consequence is that the testimony a jury hears may be sincere, confident, internally consistent, and derived from a sentence a language model generated because it was probable. The officer is not lying. He genuinely remembers it that way now. Cross-examination, built to detect deception and inconsistency, is close to useless against a witness who has been honestly contaminated. And the one artefact that could have exposed it, the original draft showing that “into” was never his word, has been deleted by design.

The Load-Bearing Human

Every deployment of this technology rests on one assurance, and Vranish stated it in its purest form: everything has to be verified and validated by an actual person.

The research on whether humans perform that role well is not encouraging, and it is not new. Raja Parasuraman and Dietrich Manzey's 2010 review in Human Factors synthesised decades of evidence that people under-monitor automated systems and accept incorrect automated advice rather than challenge it, even when contradictory information is available. The effect shows up in aviation, medicine, process control and the military, driven partly by a perception that automated systems are inherently superior and partly by the cognitive cost of checking.

Automation bias is worst under exactly the conditions that motivate buying this software: high workload, time pressure, fatigue and a high base rate of correct outputs, which trains the reviewer to expect correctness. A tool that is right ninety-five per cent of the time is more dangerous to review than one right sixty per cent of the time, because the reviewer stops looking.

The vendors know this. Code Four knows it well enough to have engineered a countermeasure that is also, inadvertently, a confession. As StateScoop's Keely Quinlan reported in July 2026, the company embeds hidden “Easter eggs” in AI-generated drafts, seldom-used words such as “axolotl,” which officers must find and remove before filing. Chief executive George Cheng explained the logic: “That basically prevents them from just copy and pasting everything from the AI at face value, and they actually have to make some edits.”

Read that as an admission, because it is one. The company built a booby trap into its own output because it did not trust officers to read the output. That is a more damning verdict on human review than anything a defence lawyer has said, and it does not solve the problem. Removing an obviously alien word proves the officer scanned for an alien word. It proves nothing about whether he verified that “into” should have been “toward.”

And the productivity premise that justifies accepting the risk is itself unproven. Adams and colleagues published a pre-registered randomised controlled trial in the Journal of Experimental Criminology in 2024, titled “No man's hand: artificial intelligence does not improve police report writing speed,” covering 85 officers across 755 reports with a check across 6,084 reports over a full year. AI assistance did not significantly affect how long reports took. In May 2026 the ACLU's Jay Stanley set that null result alongside a 2025 study by Walley and Glasspoole-Bird finding an 18.5-minute increase in time per incident.

So the ledger reads: unverified time savings, measurably worse accuracy, reviewers who cannot identify AI text at better than chance, a documented tendency to rubber-stamp machine output, and a vendor countermeasure that concedes the point. Against that, a claimed sixty-five per cent cut in report writing time and eighty thousand dollars a year.

The Exemption That Wrote Itself

Wyoming did have a chance to draw a line, and what happened to it is instructive.

House Bill 91 of the 2026 budget session would have barred government entities from using artificial intelligence to calculate and assign social scores, and from identifying people using biometric data or gathered images. It reached the House Judiciary Committee on 10 February 2026, and on 19 February the committee recommended amendment and passage. Cowboy State Daily reported that the amendment specifically exempted law enforcement uses of AI for investigating, reporting or prosecuting violations of the criminal code, and that the committee approved it seven to one. The bill died without reaching a vote of the full House.

State the sequence plainly. The one piece of Wyoming legislation aimed at constraining government AI was amended so that the most consequential use of it, the production of documents that put people in cells, was carved out. Then it failed anyway.

Other states moved differently. Utah's SB 180, sponsored by Senator Stephanie Pitcher, a Democrat from Millcreek and a former prosecutor, passed unanimously and took effect on 7 May 2025. It requires agencies to hold a written generative AI policy, a disclaimer on any AI-assisted report, and officer certification that the output was reviewed for accuracy. Pitcher's framing, quoted by Governing's Logan Seacrest in July 2026: “I felt like it was something that it's better for us to get ahead of, instead of playing catch-up.”

California went further. Senate Bill 524, signed by Governor Gavin Newsom on 10 October 2025 and effective from 1 January 2026, requires a written disclosure on every page of any report produced fully or partly by AI, bars vendors from selling the information agencies feed into the system, and, critically, requires departments to retain the first draft along with audit trails identifying the footage or audio that informed it. That retention provision is the mechanism letting judges, defence attorneys and auditors see which portions of a report the officer wrote and which the computer did. California Senator Jesse Arreguín, quoted by Seacrest: “We're not going to gamble with personal liberty. AI hallucinations happen at significant rates.”

As of October 2026, those remain the only two states to have enacted laws on AI-generated police reports, according to a state-by-state tracker maintained by Barricade. Everywhere else the attempts have stalled. New York's S10425/A9253-A lapsed when the Legislature adjourned on 5 June 2026, with reintroduction expected in 2027. Minnesota's SF 4575/HF 4536 died without a hearing in May. Virginia's HB 1294 was carried over to 2027. Even Utah's own 2026 follow-up, SB 205, which would have required annual policy review and public posting of policies, died in the House in March.

Some prosecutors have not waited. The King County Prosecuting Attorney's Office in Washington, which covers Seattle, instructed police not to submit AI-generated narratives, citing concerns about the products on the market. The people who have to prove these cases beyond reasonable doubt decided the documents were a liability.

They are not alone. In Hays County, Texas, KUT 90.5's Leigh Walden reported in September 2026, the sheriff's office trialled Draft One for three months before county commissioners voted three to two to deny funding for an expanded Axon AI contract worth $4.5 million over ten years. In nearby Kyle, the police department uses several Axon AI tools but not Draft One for reports, at the request of the district attorney's office. Gregg Cox, Hays County's first assistant district attorney, explained why the document matters: “The offense report serves a lot of purposes.” It drives charging decisions, plea offers and whether a case goes to trial. “I would much rather have them write a detailed report themselves based on what they observed,” he said, worrying that machine drafts could weaken officers' own recall of what happened. That is the memory contamination problem, seen from the prosecutor's desk.

This is not a peculiarly American drift. In a paper submitted to arXiv in December 2025, Evdoxia Taka, Temitope Lawal, Kyriakos Kotsoglou and colleagues including Calder, Sevegnani, McClory-Tiarks and Oswald set out a methodology for charting probabilistic AI tools across the criminal justice stages of England and Wales, reporting heavy reliance on private sector providers alongside growing interest in large language models. Adoption is happening force by force, vendor by vendor, ahead of any governance layer.

Public deliberation, where anyone has attempted it, is more discriminating than procurement. In August 2026, Mackenzie Jorgensen, Jo Reilly, Alex Sutherland and Miri Zilka submitted a study to arXiv describing a workshop in which 30 community representatives, police officers and academics assessed 13 AI use cases in policing with an explicit focus on racial bias. On AI-assisted processing of reports, participants asked whether time saved is offset by time spent verifying. Foregrounding racial equity, the authors found, did not narrow the deliberation but pushed it towards first-principles questions: does this tool work, will it deliver genuine benefit, and will that benefit reach everyone.

Nobody asked those questions in Casper before the trial started, because in Wyoming nobody has to.

Nobody asked them when it ended, either. In a report syndicated by GovTech on 8 October 2026, the Casper Star-Tribune's Dylan Farrell described how Casper City Council had unanimously approved a one-year, $58,800 contract for TRULEO, letting the police department use the software full-time after a test the council had approved on 7 April. Both decisions went through on the consent agenda, the part of a meeting reserved for business considered too routine to debate, and the contract passed with little public discussion. TRULEO advertises AI-assisted report writing, summaries of encounters, automated review of body-camera footage and “AI-assisted interviews.” Most council members declined to comment. The coverage contains no statement about whether defendants will be told when a machine helped write the reports against them.

What Would Actually Fix This

The answer to Nachbar's closing question, what happens to the right to a fair trial when the ghost author of the foundational document cannot be questioned, is not that trials become unfair in some diffuse atmospheric sense. It is more specific than that.

A fair trial is an adversarial testing mechanism, and it works by making every link in the chain from perception to verdict contestable. The defendant can challenge the search, the arrest, the identification, the forensic analysis, the officer's account and the officer's credibility. Each challenge targets a human who can be put under oath and asked to explain themselves.

AI drafting inserts a link immune to every one of those tools. The model cannot be sworn, cannot be impeached, cannot be asked why it chose a preposition, and cannot be shown to have been wrong before, because the record of its previous drafts has been deleted. It has no jeopardy, so it has no incentive to accuracy. And because its output is fused into a signed human document, the defence cannot identify which links are machine-made. The result is not a rigged trial. It is a trial testing the wrong thing: an officer's present agreement with a narrative rather than his actual perception of an event.

That is fixable, and it does not require Wyoming to ban the technology. Four requirements would do most of the work.

Retain the original draft. Every machine-generated first draft should be preserved, versioned, timestamped and disclosed in discovery alongside the final report. California has shown a legislature can write this. It costs storage, which is cheap, and it restores the ability to see what changed. Without it, Brady and Giglio obligations attach to an artefact engineered out of existence, and Rule 612's production requirement, with its mandatory strike-or-mistrial remedy, applies to a document nobody can produce.

Disclose on the face of the report. Not in a settings menu, not as an optional agency policy, but as a mandatory statement on the document itself, as California requires on every page. Woodhouse's warning is the operative one: it would be awful to lose a case because you used AI and did not disclose it. It would be worse to win one.

Attach provenance metadata. The report should record which model and version produced it, what source material it consumed, and which body-camera or audio file it derived from. Systems engineering risks in the Calder and Oswald taxonomy include version tracking, tool chaining and supplier dependency for a reason. A defence expert cannot evaluate a system that will not say what it is, and a system silently updated between the incident and the trial cannot be tested at all.

Prohibit AI drafting where the officer's own perception is the evidence. Where the contested fact is what an officer saw, heard, smelled or believed in the moment, and above all in use-of-force cases, the account must be written by the human before he is shown any machine draft, exactly as a witness statement is taken before the witness is shown someone else's version. Vranish made the argument for this rule himself without quite drawing the conclusion. If the camera cannot smell the room, the sentences describing the room should not come from the camera.

None of this is technologically demanding. It has not happened because the current arrangement suits everyone with power in the room. Departments get their hours back, or believe they do. Vendors avoid disclosure headaches. Prosecutors, most of them, do not have to think about it.

The person for whom the arrangement is not convenient is the defendant, who is the one party in the courtroom without a procurement budget, and whose case now rests on a document with an author who cannot be called, cannot be cross-examined, and whose first version was deleted before anyone knew there would be a trial. Wyoming is running the experiment on him right now, in three towns, at fifty dollars per officer per month in Evanston and a little under sixty thousand dollars a year in Casper.

References

1. Fedore, Kolby. Cowboy State Daily, 2026. “Wyoming Police Departments Testing AI To Generate Reports, Look At Evidence.” https://cowboystatedaily.com/2026/08/23/wyoming-police-departments-test-using-ai-to-generate-reports-look-at-evidence/
2. Code Four. Company website and product pages, 2026. https://www.codefour.us/
3. Cheng, George. Police1 (sponsored content by Code Four), 2026. “The operating system the modern detective never had.” https://www.police1.com/police-products/investigation-software/the-operating-system-the-modern-detective-never-had
4. Quinlan, Keely. StateScoop, 2026. “Police AI platform uses 'Easter eggs' to verify human review.” https://statescoop.com/police-ai-platform-code-four-human-review/
5. Guariglia, Matthew and Maass, Dave. Electronic Frontier Foundation, 2025. “Axon's Draft One Is Designed to Defy Transparency.” https://www.eff.org/deeplinks/2025/07/axons-draft-one-designed-defy-transparency
6. Calder, Muffy; Oswald, Marion; McClory-Tiarks, Elizabeth; Sevegnani, Michele; and Taka, Evdoxia, 2026. “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” arXiv:2603.18116. https://arxiv.org/abs/2603.18116
7. Jorgensen, Mackenzie; Reilly, Jo; Sutherland, Alex; and Zilka, Miri, 2026. “Negotiating Risk Boundaries in AI for Policing Through Mixed-Stakeholder Deliberation.” arXiv:2608.05418. https://arxiv.org/abs/2608.05418
8. Taka, Evdoxia; Lawal, Temitope; Calder, Muffy; Sevegnani, Michele; Kotsoglou, Kyriakos; McClory-Tiarks, Elizabeth; and Oswald, Marion, 2025. “Mapping the Probabilistic AI Ecosystem in Criminal Justice in England and Wales.” arXiv:2512.04116. https://arxiv.org/abs/2512.04116
9. Adams, Ian T.; Barter, Matt; McLean, Kyle; Geary, Irick A. Jr; Fabila, Alexis; and McCrain, Josh. CrimRxiv, 2026. “A Good College Essay but a Bad Police Report: A Triple-Blind Expert Evaluation of AI-Assisted Police Reporting.” https://www.crimrxiv.com/pub/u7azgqzd/release/1
10. Adams, Ian T. et al. Journal of Experimental Criminology, 2024. “No man's hand: artificial intelligence does not improve police report writing speed.” https://link.springer.com/article/10.1007/s11292-024-09644-7
11. Stanley, Jay. American Civil Liberties Union, 2026. “Studies Question Value of AI-Assisted Police Reports.” https://www.aclu.org/news/privacy-technology/studies-question-value-of-ai-assisted-police-reports
12. Seacrest, Logan. Governing (R Street Institute), 2026. “The Risky Marriage of AI and Police Reports.” https://www.governing.com/policy/the-risky-marriage-of-ai-and-police-reports
13. Constantino, Mya. FOX 13 News Utah, 2026. “AI programs used by Heber City police claim officer turned into a frog.” https://www.fox13now.com/news/local-news/summit-county/how-utah-police-departments-are-using-ai-to-keep-streets-safer
14. Roth, Andrea L. Yale Law Journal, volume 126, 2017. “Machine Testimony.” https://yalelawjournal.org/article/machine-testimony
15. Loftus, Elizabeth F. Legal and Criminological Psychology, 2026. “The history of an idea: The misinformation effect.” https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/lcrp.70020
16. Supreme Court of the United States. Brady v. Maryland, 373 U.S. 83 (1963). https://supreme.justia.com/cases/federal/us/373/83/
17. Supreme Court of the United States. Giglio v. United States, 405 U.S. 150 (1972). https://supreme.justia.com/cases/federal/us/405/150/
18. Supreme Court of the United States. Melendez-Diaz v. Massachusetts, 557 U.S. 305 (2009). https://tile.loc.gov/storage-services/service/ll/usrep/usrep557/usrep557305/usrep557305.pdf
19. Supreme Court of the United States. Bullcoming v. New Mexico, 564 U.S. 647 (2011). https://tile.loc.gov/storage-services/service/ll/usrep/usrep564/usrep564647/usrep564647.pdf
20. Legal Information Institute, Cornell Law School. Federal Rule of Evidence 612, “Writing Used to Refresh a Witness.” https://www.law.cornell.edu/rules/fre/rule_612
21. Parasuraman, Raja and Manzey, Dietrich H. Human Factors, 2010. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” https://journals.sagepub.com/doi/10.1177/0018720810376055
22. Wyoming Legislature, 2026 Budget Session. House Bill 91, government use of artificial intelligence. https://legiscan.com/WY/bill/HB0091/2026
23. Farrell, Dylan. Casper Star-Tribune, via GovTech, 2026. “In Casper, Wyo., a Green Light for AI Police Bodycams.” https://www.govtech.com/artificial-intelligence/in-casper-wyo-a-green-light-for-ai-police-bodycams
24. Barricade, 2026. “AI Police Report Laws by State: 2026 Tracker” (updated October 2026). https://barricade.tech/articles/ai-police-report-laws-by-state
25. Walden, Leigh. KUT 90.5, 2026. “Austin-area law enforcement is testing AI-assisted police reports.” https://www.kut.org/crime-justice/2026-09-30/police-ai-hays-county-san-marcos-kyle-district-attorney

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Looking for a flat used to mean filters. You set a price ceiling, ticked a box for parking, drew a rough circle on a map and scrolled. Now, increasingly, you talk. On 6 October 2026, Palo Alto Online published a report by Angela Swartz on how artificial intelligence is changing the apartment search in Silicon Valley's backyard, and the picture it painted was of a market where the listing, the questions and the answers are all starting to pass through software that speaks in fluent sentences.

The paper described rental platforms that have turned search into dialogue. Zumper's assistant, Zoe, uses information from a property's Zumper listing to answer questions about an apartment, its community and the surrounding neighbourhood, and helps schedule tours. RentCafe launched its assistant, Ren, in August. Apartments.com introduced conversational search in June. On the landlord side, the report noted that Essex Property Trust uses an AI agent named Eva to greet prospective renters through the websites of complexes including 777 Hamilton in Menlo Park and Radius in Redwood City.

Then there was the renter. A 23-year-old former investment banker told the paper he had used ChatGPT and Claude to compile property management websites and a shortlist of homes matching his price and location. “It did a pretty good job of listing out apartments that fell within those parameters,” he said. “Sometimes it would include things outside of that.” In the end, the report noted, he found an apartment “the old-fashioned way: through word of mouth.”

Zumper's chief technology officer told the paper that “the way people are searching for apartments is changing”, and said the company designed Zoe to minimise hallucinations, the term for instances in which an AI system produces inaccurate or made-up information. But, as the report put it, large language models can still make mistakes. And California's Department of Real Estate has warned that licensees remain responsible for information produced by AI.

That combination of claims raises three questions. What happens to the search for a home when every layer of it is mediated by systems that can state facts about a property that are not true? Who is responsible to a renter who acts on what a system says? And what should a person be entitled to know about how the homes put in front of them were chosen?

A Small Story With a Very Large Footprint

It is worth being precise about what the Palo Alto Online report shows and what it does not. It does not document a renter who lost money because a chatbot lied. It does not show that Zoe, Ren, Eva or Apartments.com's assistant has given anyone a false answer. Its one first-hand account from a renter is modest: a general-purpose chatbot sometimes ignored his limits, and he ended up finding his home through word of mouth.

That modesty is the point. A listing over budget or outside the chosen area is easy to spot and discard. The harder problems are the errors a renter cannot see: an answer that a building allows dogs when it does not, that a unit includes parking when it does not, that a fee is included when it is not, or that a neighbourhood is “a good fit” for reasons that have more to do with who the renter appears to be than with what they asked for.

The same report also described the other side of the counter. A Palo Alto-based property manager told the paper that he had set up a database allowing an AI tool to answer questions and check property managers' schedules to arrange tours, and that he had also seen property managers use the free version of ChatGPT to ask whether an issue needed to be brought to a lawyer or whether a new policy applied to their properties. The machine is answering on both sides of the conversation.

Every Major Portal Now Wants to Talk

The shift the paper described is not a local curiosity. It is an industry-wide bet that the search box is dying.

Zumper first announced Zoe in December 2024. HousingWire reported at the time that it could answer “virtually any renter question” about a property or its neighbourhood, from pet policies to local hotspots, and could request tours and submit leads to leasing teams. In February 2026, Zumper launched a rental search app inside ChatGPT, allowing users to type requests such as “show pet-friendly rentals under $2,500 with in-unit laundry” and receive listings without leaving the chat. In announcing it, the company cited its own survey finding that the share of renters using AI tools during their home search had risen from 4.4 per cent in 2024 to 9.8 per cent in 2025.

RentCafe, owned by the property software firm Yardi, published a feature spotlight on 20 August 2026 explaining that its former AI Mode was now called Ren. “It's an ongoing, two-way conversation,” the post said. “Instead of one search and a list of results, Ren actively guides you to apartments.” It also made a striking promise: “Ren never makes up a property, a price or an amenity. Every detail is sourced directly from trustworthy, up-to-date listings.”

CoStar Group launched Apartments.com Ai on 16 June 2026, describing it as replacing traditional filters and keyword searches with real-time conversation. “Finding an apartment is one of the most important decisions people make, yet the search process has remained largely unchanged for years,” said Andy Florance, CoStar's founder and chief executive. “Apartments.com Ai represents a fundamental shift from searching listings to having a conversation with a trusted rental advisor.”

Behind the portals sit the leasing agents. EliseAI, which sells assistants that answer prospective tenants by text, email and phone and book tours, announced on 29 September 2026 that it had raised $350 million at a $4 billion valuation, and said its platform now powers roughly one in six US apartments.

A renter may now ask a general chatbot for a shortlist, question a portal's branded assistant about a listing, and then be greeted on the building's website by a leasing agent that is also software. At no point is a human obliged to be the one who answers.

Notice the industry's vocabulary: “rental advisor”, “trusted”. It describes a relationship, not a database query. A filter that returns a listing is a tool. An adviser who tells you a flat is right for you is making a representation.

Why Fluent Software Says Things That Are Not True

Why would a system confidently state something false about a property? The short answer is that large language models are built to produce plausible text, and plausibility is not the same as truth.

In September 2025, researchers at OpenAI and Georgia Tech, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, published a paper titled “Why Language Models Hallucinate”. Their argument was blunt: language models hallucinate because their training and evaluation procedures reward guessing over acknowledging uncertainty. If a benchmark awards a point for a correct answer and nothing for “I don't know”, a model that always guesses will outscore one that admits ignorance. The authors argued that fixing this requires changing how mainstream benchmarks are scored, not merely adding more hallucination tests.

Rental platforms try to contain the problem by grounding their assistants in listing data. Zoe draws on the property's Zumper listing; Ren, according to RentCafe, on active RentCafe listings; Apartments.com Ai on CoStar's multifamily data. Grounding narrows the space in which a model can invent things. It does not eliminate it.

Grounding fails in at least three ways that matter to renters. The first is that the source can be wrong or stale. A listing that says “pets welcome” when the landlord changed the policy last month will produce a confident, grounded and incorrect answer. The second is that the model can misread or over-generalise from what it retrieves, turning “street parking available” into “parking included”. The third is the question that has no answer in the listing at all. Is this area quiet at night? Is the building safe? Will the landlord accept my housing voucher? A system tuned to be helpful may fill the gap rather than say it does not know.

General-purpose chatbots, the kind the young renter used, sit further still from the data, summarising web pages or relying on training data that may be months old. His complaint is the mildest form of the problem, and a reminder that a model can appear to follow instructions while quietly ignoring them.

California Puts the Liability on the Licensee

California's real estate regulator has been unusually direct about who owns these errors. On 17 March 2026, the Department of Real Estate published an advisory titled “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees”. Among the uses it lists are AI-powered chatbots and automated lead scoring to manage enquiries. Its message about responsibility leaves little room for doubt.

“If an AI tool generates inaccurate information, misleading advertising, or improper communications with consumers,” the advisory says, “responsibility under current law rests with the licensee and their responsible broker, not the technology provider.” It adds that “AI outputs should be reviewed for accuracy and approved by the licensee before being relied upon in transactions or consumer communications”, and that licensees using AI to draft listing descriptions, emails or online advertisements “must independently verify factual claims, including claims involving property features, pricing, availability, and potential uses”. It reminds licensees that “all real estate advertising in California must be truthful, accurate, and not misleading”, and points to section 10176 of the Business and Professions Code, which allows disciplinary action for substantial misrepresentation. It even advises keeping records “showing how AI outputs were reviewed and verified for accuracy and compliance with law.”

The state has also begun legislating against one specific kind of machine-made falsehood: the doctored photograph. Assembly Bill 723, which took effect on 1 January 2026, requires real estate brokers and salespeople who use digitally altered images in advertising to say so and to provide access to the original image. Routine adjustments such as lighting and colour are excluded. In September 2026, according to the California legislature's records, Assembly Bill 2025 was chaptered, extending a similar disclosure duty to advertisements for rental property. Its stated rationale was that virtual staging and AI tools allow anyone marketing a rental to manipulate or fabricate images showing features that are misleading or non-existent.

There is an older California law that also bears on the conversation. Since 1 July 2019, section 17941 of the Business and Professions Code has made it unlawful to use a bot to communicate with a person in California online with the intent to mislead them about its artificial identity in order to incentivise a sale of goods or services, unless the bot is clearly and conspicuously disclosed. An assistant clearly labelled as software is fine. One that passes itself off as a person, in order to close a deal, is a legal problem.

Yet the advisory has an obvious limit. It binds licensees. A real estate broker who lets a chatbot speak for them owns what it says. But a renter asking a general-purpose chatbot to find flats is not dealing with a licensee at all, and a landlord renting out its own property is not necessarily a licensee either. The more the conversation moves to software that sits outside the licensing regime, the more the regulator's clear rule is left guarding a door that renters are no longer walking through.

A Chatbot Is Still Part of the Website

When a renter acts on a wrong answer, the most useful precedent is not American. It comes from a small claims tribunal in British Columbia and a grieving airline passenger.

In February 2024, British Columbia's Civil Resolution Tribunal decided Moffatt v Air Canada. Jake Moffatt had relied on the airline's website chatbot, which told him he could apply for a bereavement fare after travelling. Another page on the same site said otherwise. Air Canada argued, in effect, that it could not be held liable for what its chatbot said. Tribunal member Christopher Rivers called this “a remarkable submission” and rejected it. “While a chatbot has an interactive component, it is still just a part of Air Canada's website,” he wrote. “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” He found negligent misrepresentation and ordered the airline to pay damages, interest and fees.

The ruling binds nobody outside British Columbia's small claims system, and the sum was modest. Its reasoning travels easily, though. Negligent misrepresentation does not care whether the careless sentence was typed by a leasing agent or generated by a model.

In the United States, the defence that platforms have historically relied on is Section 230 of the Communications Decency Act, which protects online services from being treated as the publisher of information provided by someone else. But a generative assistant does not merely host a landlord's listing. It composes new sentences. The closest precedent comes from housing itself. In 2008, the Ninth Circuit Court of Appeals, sitting en banc, held in Fair Housing Council of San Fernando Valley v Roommates.com that the website was not fully immune for content it helped develop, specifically mandatory questions that required users to disclose sex, family status and sexual orientation. When a service itself produces content that runs foul of the law, the court reasoned, it cannot shelter behind immunity designed for third-party speech. How far that logic extends to AI-generated answers about rental property has not been settled by the courts. It is an uncomfortable starting point for any platform arguing that its assistant's words belong to someone else.

The Accountability Chain Has Missing Links

Lay out the parties involved in a single conversational search and the problem of responsibility becomes visible.

There is the landlord or property manager, who supplies the listing and may run their own AI leasing agent. There is the listing platform, which ingests that data, wraps it in a branded assistant and decides which homes to surface. There is the company that built the underlying language model, which the platform may license. And increasingly there is a general-purpose chatbot, which may sit in front of all of them, summarising pages it found on the web.

Each can point at another. The landlord can say the platform's assistant misread the listing. The platform can say the landlord's data was wrong. The model provider's terms of service warn that outputs may be inaccurate. A general-purpose chatbot is not a licensee, not a landlord and not a listing service.

The renter, meanwhile, holds the risk: the application fee, the notice given on a current home, the signed lease. The Federal Trade Commission's December 2025 settlement with Greystar, one of the largest property managers in the United States, shows how much money can rest on what a renter is told up front. The FTC and Colorado's attorney general alleged that Greystar advertised base rents without disclosing mandatory monthly fees, so that many renters paid non-refundable application fees before learning the true price. Greystar agreed to pay $24 million, of which $23 million was earmarked for consumer refunds, and to disclose the total monthly price including mandatory fees whenever it advertises or quotes rent. FTC chairman Andrew Ferguson directed staff to begin the process of proposing a rule on unfair or deceptive fees in rental housing.

That case did not involve AI, but the principle carries: a misleading statement that a renter relies on is actionable however it was produced. The FTC said as much in September 2024, when it announced Operation AI Comply, a sweep against companies using AI in deceptive ways. “The FTC's enforcement actions make clear that there is no AI exemption from the laws on the books,” said Lina Khan, then the agency's chair.

The sensible answer to the question of responsibility is therefore not mysterious. Whoever chooses to put an AI system in front of a renter, and profits from doing so, should own what it tells that renter. That is the DRE's position for licensees, the Air Canada tribunal's position for businesses and the FTC's position for advertisers. What is missing is a rule that covers the gaps between them: the portal that is not a broker, the general chatbot that is not an advertiser, and the property manager who is neither.

Steering Without Anyone Meaning To

The most serious risk in conversational search is not a wrong fact. It is a pattern.

Under the federal Fair Housing Act, steering, guiding a homeseeker towards or away from neighbourhoods because of race or another protected characteristic, is illegal. Traditionally it required a person to do the steering. A conversational assistant can now do it at scale, without intending to, simply by reproducing patterns in the data it learned from.

Researchers have already tested this. In a paper presented at the ACM conference on Equity and Access in Algorithms, Mechanisms, and Optimization in October 2024, Eric Liu, Wonyoung So, Peko Hosoi and Catherine D'Ignazio of MIT audited GPT-4's housing recommendations using 168,000 prompts for renting and buying in the ten largest majority-minority cities in the United States, varying characteristics such as race, family status and source of income. They found evidence of “racial steering”, “default whiteness” and the steering of minority homeseekers towards neighbourhoods with lower opportunity indices, and concluded that such systems should be audited before deployment on real estate platforms.

A follow-up study, posted in June 2026 by Hana Samad, Trung Lam, Christoph Mügge-Durum and Michael Akinwumi, working with the National Fair Housing Alliance's responsible AI programme, framed the problem in exactly the terms the Palo Alto report raises. Large language models, they wrote, are “rapidly assuming an intermediary role in housing search through the integration of listing platforms within conversational interfaces”. Auditing seven open-weight and closed-source models across four US cities, they found that steering was an emergent behaviour that varied by model and by prompt. When users added lifestyle preferences, some models began steering where they had not before, suggesting the same stated preference could be interpreted differently depending on the user's perceived racial identity.

Industry players know this. In May 2024, Zillow released an open-source Fair Housing Classifier designed to detect questions in AI-powered real estate conversations that could lead to discriminatory responses and so reduce the risk of illegal steering. It was a sensible step. It is also voluntary, and renters have no way to know whether the assistant they are talking to uses anything like it.

The legal history here is not hypothetical. In June 2022, the US Department of Justice settled its first case challenging algorithmic discrimination under the Fair Housing Act, against Meta. The government alleged that Meta's advertising system used algorithms that relied in part on characteristics protected under the Act to decide who saw housing ads. Meta agreed to stop using its “Special Ad Audience” tool for housing ads and to build a Variance Reduction System to bring the audiences who actually saw housing ads closer to the eligible audiences by sex and estimated race or ethnicity, under court oversight that ran to 27 June 2026. “This groundbreaking resolution sets a new standard for addressing discrimination through machine learning,” said Damian Williams, then the US Attorney for the Southern District of New York.

Algorithms that decide who gets a home have been tested in court too. In November 2024, a federal judge approved a settlement under which SafeRent Solutions, a tenant-screening company, agreed to pay more than $2.2 million and to stop including its score on screening reports in certain cases, including where the applicant uses a housing voucher. The plaintiffs, led by Mary Louis, had alleged that the scoring algorithm disproportionately harmed Black and Hispanic applicants and voucher holders. SafeRent did not admit wrongdoing. “Everything is based on numbers,” Louis told the Associated Press. “You don't get the individual empathy from them.”

A conversational assistant is not a tenant screener, but it shares the structural risk. It decides which homes a renter sees, in what order and with what commentary. If its suggestions differ by who the user appears to be, the effect is the same as human steering, and far harder to detect.

Washington Steps Back as the Software Steps In

Just as these systems are spreading, federal oversight in the United States is receding.

In May 2024, the Department of Housing and Urban Development issued guidance explaining how the Fair Housing Act applies to tenant screening and to housing advertising delivered through digital platforms, including when AI and algorithms do the work. On 17 September 2025, HUD's Office of Fair Housing and Equal Opportunity withdrew a list of guidance documents it said should no longer be in effect, citing deregulatory executive orders. The digital platforms advertising guidance was among them.

In January 2026, HUD proposed removing its regulations on disparate impact liability, the theory that lets plaintiffs challenge practices that are neutral on their face but discriminatory in effect, arguing that such questions should be left to the courts. The proposal had not been finalised by early October 2026. Disparate impact still exists in case law; the Supreme Court recognised it under the Fair Housing Act in 2015. But it is precisely the theory most suited to challenging an algorithm, because algorithms rarely intend anything.

Elsewhere, enforcement has focused on what software does to the price of a home rather than what it says about one. In November 2025, the Justice Department reached a proposed settlement with RealPage over allegations that its pricing software used competitors' non-public data to help landlords align rents. A federal judge in North Carolina approved it in May 2026, barring RealPage from using that confidential data to help set rents. RealPage did not admit wrongdoing. The case shows regulators can reach into the code that shapes housing markets. It also shows how long it takes.

Scammers Already Speak the Language

There is one more party in the conversation that renters cannot afford to forget: the fraudster.

In December 2025, the FTC reported that consumers had filed nearly 65,000 reports of rental scams since 2020, with losses of about $65 million. About half of those who reported a rental scam in the 12 months to June 2025 said it began with a fake advert on Facebook. People aged 18 to 29 were three times more likely than other adults to report losing money. The FTC noted that fake listings “can often look very real and copy information from legitimate listings but are posted with the scammer's contact information on different sites.”

The FBI's Internet Crime Complaint Center recorded 12,368 real estate fraud complaints and $275.1 million in reported losses in 2025, a category that includes rental fraud, up sharply from the year before.

Conversational search raises the stakes. A chatbot that searches the web cannot always tell a cloned listing from a real one, and may present both in the same confident tone. And renters are being trained to trust a friendly voice that answers instantly. The young renter in Palo Alto noticed when results drifted. Not everyone will.

What Europe and Britain Already Demand

Other jurisdictions offer partial answers to the question of what a renter should be told.

The European Union's Digital Services Act requires online platforms that use recommender systems to set out, in plain and intelligible language, the “main parameters” those systems use, including the criteria most significant in deciding what is suggested to a user and the reasons for their relative importance. The largest platforms must offer at least one option not based on profiling. Applied to rental search, that principle would mean a renter could find out why these homes, in this order, rather than others.

The EU's AI Act adds a simpler duty. Since 2 August 2026, Article 50 has required that people interacting directly with an AI system be informed that they are doing so, unless it is obvious. Fines for breaching the Act's transparency obligations can reach €15 million or 3 per cent of worldwide annual turnover.

Britain has taken a different route, focusing on what listings must contain rather than how software presents them. National Trading Standards' Estate and Letting Agency Team has issued guidance on “material information” in property listings; for lettings, this includes the rent, any deposit and the council tax band. The Renters' Rights Act 2025, whose first phase came into force in England on 1 May 2026, made it illegal for landlords and agents to invite or accept bids above the advertised rent, and to discriminate against prospective tenants because they have children or receive benefits, including through advertising. Since April 2025, the Digital Markets, Competition and Consumers Act 2024 has allowed the Competition and Markets Authority to enforce consumer law directly and to fine businesses up to 10 per cent of global turnover.

None of these laws was written for a chatbot that recommends flats. But together they sketch something the US lacks: a baseline of what must be disclosed, a ban on certain kinds of exclusion at the advertising stage and a regulator with the power to act quickly.

What a Renter Should Be Entitled to Know

If the conversation is the new front door to housing, renters need rights that match. Drawing together the DRE's advisory, the Air Canada ruling, the fair housing case law and the European rules, a workable set begins to emerge.

First, renters should know they are talking to a machine and on whose behalf: a portal, a landlord or neither, and whether suggested homes include paid placements. California's bot law and the EU AI Act already point this way.

Second, renters should know where each fact came from. If an assistant says a unit allows dogs, it should show the dated listing text it relied on. Grounded systems already retrieve this; displaying it costs little.

Third, renters should be told why homes were suggested and ranked. The DSA offers a model: the main criteria, their relative importance and an option to search without profiling.

Fourth, assistants should be built to say “I don't know”. Questions about safety, schools or the “character” of a neighbourhood are where steering risk is highest and listing data thinnest. The OpenAI researchers' case for rewarding abstention applies to product design as much as to benchmarks.

Fifth, the business that deploys an assistant should be liable for what it says, as if a human employee had said it. That is already the DRE's rule for licensees and the Air Canada tribunal's reasoning for businesses. Extending it explicitly to listing platforms and property managers who are not licensees would close the gap the conversational shift has opened.

Sixth, conversational housing assistants should be audited for steering before deployment and at regular intervals, with results published. The MIT and NFHA studies show that such audits are feasible. The Meta settlement shows that algorithmic delivery can be measured against a fairness benchmark and corrected under oversight.

Finally, renters should be able to keep a transcript of the conversation. If an answer led someone to pay a fee or sign a lease, it is the evidence. The DRE already tells licensees to keep records. Renters deserve the same.

Word of Mouth Still Beats the Machine

It is easy to read the Palo Alto Online report as a story about technology that does not quite work yet. That misses the point. These assistants will improve, and the young renter's complaint may soon feel quaint.

The deeper issue is structural. Housing search used to involve a listing written by a landlord, a portal that displayed it and a renter who read it. Each statement had an author. In conversational search, the sentence a renter acts on may be composed on the fly by a system nobody fully controls, drawing on data nobody fully checked, ranked by criteria nobody disclosed. The industry describes that system as an adviser. The law has not yet decided whether it should be treated as one.

California's regulator has answered half the question: if a licensee lets AI speak, the licensee answers for it. Courts have hinted at more: a chatbot is part of the business that runs it. Europe and Britain have begun to set out what a person must be told. What nobody has yet written down is a single rule that follows the renter rather than the type of company they happen to be talking to.

Until that exists, the renter in Palo Alto may have found the most reliable method of all. He asked the machines, noticed where they drifted, and in the end found his home through word of mouth. The question for regulators and platforms is whether the next generation of renters will have to do the same.

---

References

1. Swartz, Angela. “How AI is changing the apartment search: Is it working?” Palo Alto Online, 6 October 2026. https://www.paloaltoonline.com/technology/2026/10/06/how-ai-is-changing-the-apartment-search-is-it-working/
2. Andrews, Jeff. “Zumper launches AI assistant Zoe.” HousingWire, 3 December 2024. https://www.housingwire.com/articles/zumper-ai-assistant-zoe/
3. Inman. “Zumper introduces Zoe, latest AI for aspiring renters.” Inman, 10 December 2024. https://inman.com/2024/12/10/zumper-introduces-zoe-latest-ai-for-aspiring-renters
4. National Mortgage Professional. “Zumper Debuts Real-Time Rental Search App Inside ChatGPT.” February 2026. https://nationalmortgageprofessional.com/news/zumper-debuts-real-time-rental-search-app-inside-chatgpt
5. Petrut, Florin. “Meet Ren, your apartment search assistant on RentCafe.com.” RentCafe Blog, 20 August 2026. https://www.rentcafe.com/blog/renting/rentcafe-feature-spotlight-ai-mode/
6. Mann Report. “CoStar Group Launches Apartments.com Ai.” Mann Report, 16 June 2026. https://www.mannpublications.com/mannreport/2026/06/16/costar-group-launches-apartments-com-ai/
7. Gleeson, Cailey. “EliseAI banks $350M, hits $4B valuation for accelerated new product development.” Fierce Healthcare, 29 September 2026. https://www.fiercehealthcare.com/finance/eliseai-banks-350m-hits-4b-valuation-accelerated-new-product-development
8. Kalai, Adam Tauman; Nachum, Ofir; Vempala, Santosh S.; Zhang, Edwin. “Why Language Models Hallucinate.” arXiv:2509.04664, 4 September 2025. https://arxiv.org/abs/2509.04664
9. California Department of Real Estate. “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees.” Advisory, 17 March 2026. https://www.dre.ca.gov/Licensees/Advisories/Advisory_2026_03_17_AI_in_California_Real_Estate.html
10. California Legislature. AB 723 (2025), Real estate: digitally altered images: disclosure. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab723
11. California Legislature. AB 2025 (2026), Tenancy: digitally altered images: disclosure, Chapter 572, Statutes of 2026. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab2025
12. Perkins Coie. “I Am Robot: California's New Law Requires Disclosure of Use of Bots.” https://perkinscoie.com/insights/blog/i-am-robot-californias-new-law-requires-disclosure-use-bots
13. Moffatt v Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024; reported in CTV News, “Air Canada's chatbot gave a B.C. man the wrong information. Now, the airline has to pay for the mistake,” February 2024. https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-the-wrong-information-now-the-airline-has-to-pay-for-the-mistake-1.6769454
14. Fair Housing Council of San Fernando Valley v. Roommates.com, LLC, 521 F.3d 1157 (9th Cir. 2008) (en banc); Congressional Research Service, “Liability of Interactive Computer Service for Violating the Fair Housing Act,” RS22668. https://www.everycrsreport.com/reports/RS22668.html
15. Federal Trade Commission and Colorado Attorney General. “Weiser, FTC announce $24M settlement with Greystar.” 2 December 2025. https://coag.gov/press-releases/weiser-ftc-announce-24m-settlement-with-greystar/
16. Davis Polk. “FTC announces new enforcement initiative targeting deceptive AI practices.” September 2024. https://www.davispolk.com/insights/client-update/ftc-announces-new-enforcement-initiative-targeting-deceptive-ai-practices
17. Liu, Eric; So, Wonyoung; Hosoi, Peko; D'Ignazio, Catherine. “Racial Steering by Large Language Models: A Prospective Audit of GPT-4 on Housing Recommendations.” EAAMO '24, ACM, October 2024. MIT DSpace. https://dspace.mit.edu/handle/1721.1/157628
18. Samad, Hana; Lam, Trung; Mügge-Durum, Christoph; Akinwumi, Michael. “The Geography of Algorithmic Judgment: LLM Intermediaries, Place Identity, and Racial Steering in Housing Search.” arXiv:2606.06694, 4 June 2026. https://arxiv.org/abs/2606.06694
19. Zillow Group. “Zillow providing open-source technology to promote fair housing in AI-powered real estate conversations.” Press release, 21 May 2024. https://zillow.mediaroom.com/2024-05-21-Zillow-providing-open-source-technology-to-promote-fair-housing-in-AI-powered-real-estate-conversations
20. US Department of Justice. “Justice Department Secures Groundbreaking Settlement Agreement with Meta Platforms, Formerly Known as Facebook, to Resolve Allegations of Discriminatory Advertising.” 21 June 2022. https://www.justice.gov/archives/opa/pr/justice-department-secures-groundbreaking-settlement-agreement-meta-platforms-formerly-known
21. US Attorney's Office, Southern District of New York. “United States Attorney Implements Groundbreaking Settlement With Meta Platforms, Inc.” January 2023. https://www.justice.gov/usao-sdny/pr/united-states-attorney-implements-groundbreaking-settlement-meta-platforms-inc-formerly
22. Bedayn, Jesse. “Class action lawsuit on AI-related discrimination reaches final settlement.” Associated Press, 20 November 2024, as published by Fortune, 21 November 2024. https://fortune.com/2024/11/21/renter-scoring-saferent-million-settle-case-algorithm-discriminating-race-income/
23. US Department of Housing and Urban Development. “HUD Issues Fair Housing Act Guidance on Applications of Artificial Intelligence.” Press release 24-098, 2 May 2024. https://archives.hud.gov/news/2024/pr24-098.cfm
24. Bilowich, Juliana. “HUD Withdraws Wide-Ranging Fair Housing Policies.” LeadingAge, 23 September 2025. https://leadingage.org/hud-withdraws-wide-ranging-fair-housing-policies/
25. Ballard Spahr. “HUD proposes to remove its Fair Housing Act disparate impact rule.” Consumer Finance Monitor, 14 January 2026. https://www.consumerfinancemonitor.com/2026/01/14/hud-proposes-to-remove-its-fair-housing-act-disparate-impact-rule/
26. Baker McKenzie. “United States: DOJ settles RealPage case.” December 2025. https://www.bakermckenzie.com/en/insight/publications/alerts/2025/12/united-states-doj-settles-realpage-case
27. Lowe, Jonathan; Praats, Michael. “RealPage settles with DOJ over rent pricing scheme; NC case still active.” WSOC-TV, 20 May 2026. https://www.wsoctv.com/news/local/realpage-settles-with-doj-over-rent-pricing-scheme-nc-case-still-active/AKJI2ZH7R5GOHIK3JGVIPKM4HM/
28. Federal Trade Commission. “FTC Analysis Shows Consumers Have Lost Millions to Rental Scams.” Press release, 22 December 2025. https://www.ftc.gov/news-events/news/press-releases/2025/12/ftc-analysis-shows-consumers-have-lost-millions-rental-scams
29. Block, Eliana. “Online real estate fraud climbed to $275M in 2025, FBI says.” National Association of Realtors, 13 April 2026. https://www.nar.realtor/news/real-estate-news/online-real-estate-fraud-climbed-to-275m-in-2025-fbi-says
30. HousingWire. “FBI cybercrime losses: real estate fraud hits $275M.” April 2026. https://www.housingwire.com/articles/fbi-cybercrime-losses-real-estate-fraud-hits-275m/
31. European Parliament and Council. Regulation (EU) 2022/2065 (Digital Services Act), Articles 27 and 38. Official Journal of the European Union, 27 October 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2065
32. European Parliament and Council. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 50 and 99. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
33. National Trading Standards Estate and Letting Agency Team. “Material information in property listings: Part A.” https://www.nationaltradingstandards.uk/uploads/Material%20information%20-%20Part%20A%20-%20updated%20July%202022.pdf
34. UK Government. “Renters' Rights Act: an overview for landlords.” GOV.UK. https://www.gov.uk/guidance/renters-rights-act-an-overview-for-landlords
35. DLA Piper. “CMA's new consumer enforcement regime comes into force in April 2025.” 7 April 2025. https://www.dlapiper.com/insights/publications/2025/04/cmas-new-consumer-enforcement-regime-comes-into-force-in-april-2025

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A climber is stranded near a summit. A storm is building. Somebody has to decide whether to send a rescue team up the mountain, and then justify that decision to the authority that will sign it off.

That was the scenario put in front of undergraduate students at the University of California, Irvine, in the autumn of 2025. They worked in small groups, typing into a shared chat window, with roughly eighteen minutes to deliberate. Midway through, the researchers dropped in a complication: the climber, it turned out, had ignored weather warnings and set off without registering a route. Suddenly the question was no longer only about logistics and risk. It was about blame, obligation and what a rescuer owes to someone who has been reckless.

In 16 of the 33 groups, one of the three teammates was not a student at all. It was a conversational AI, built on Google's Gemini 2.5 Flash Lite model and given the persona of a peer called “Clever Lamarr”. Its instructions steered it away from the service language of a chatbot. It was not there to be “happy to help”. It was there to be one of the team.

What happened next is documented in a paper posted to arXiv on 29 July 2026, titled “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making”. Its authors are Nia Nixon, Jaeyoon Choi, Pedro Martins De Bastos, Mohammad Amin Samadi, Seehee Park and Spencer JaQuay of UC Irvine, together with Luise Mehner of the University of Tübingen. The headline finding is easy to state and uncomfortable to sit with. The AI was the most talkative member of every team it joined. It also contributed the least new information. And the two humans sitting alongside it became measurably less responsive to each other and reported a weaker sense of belonging and status. That cost did not creep in over time. It was there from the start.

For years, the industry argument about AI at work has been framed around a single axis: does the machine make the work better or worse? This study asks a different question. What does the machine do to the people who work next to it?

Talking More and Saying Less

The study compared 16 “treatment” teams, each made up of two students and the AI teammate, with 17 control teams of three students. Eighty undergraduates took part. To analyse what happened in the chat logs, the researchers used Group Communication Analysis, a computational linguistics method published in the journal Behavior Research Methods in 2019 that tracks how each participant's contributions relate to their own earlier messages and to everyone else's.

Participation captures how much someone contributes; internal cohesion, how consistent they are with their own earlier messages. Responsivity measures how much a person picks up on teammates' contributions, and social impact the reverse. Newness tracks fresh information, and communication density how much meaning is packed into each turn.

On participation, the AI ranked first in all 16 treatment teams. Not most of them. All of them. It ranked first or joint first for internal cohesion in 14 of the 16. In the language of the abstract, it was “the single most talkative and self-cohesive member of every treatment team”.

That sounds like an ideal colleague until you look at the other numbers. The AI's newness and communication density were both significantly lower than those of the students it was working with. It talked the most, and it stayed on message, but what it said carried the least fresh information and the least substance per turn. It was, to borrow a phrase every office worker will recognise, a voice that filled the room.

One detail is easy to miss. The AI's responsivity and social impact did not differ significantly from the students' own. The humans took up what the machine said, and it responded to them. The AI was not ignored; it was an engaged participant whose contributions were absorbed into the discussion. That is what makes the next findings so striking.

The Cost Lands on the Humans

The most consequential results in the paper are not about the AI at all. They are about what happened between the two students who shared a team with it.

Compared with students in all-human teams, students in AI teams showed significantly lower responsivity and significantly lower social impact. Their responsivity score averaged 0.065, against 0.098 in the control groups. Their social impact averaged 0.068, against 0.099. Both differences were substantial. Put plainly, the humans in AI teams were picking up on each other's ideas less, and their own ideas were being picked up less by the other human in the room.

Per-student participation was also lower in AI teams, even though each of those teams had one fewer human in it. That matters. With only two students instead of three, you might expect each person to carry more of the conversation. The opposite happened. The machine's airtime did not simply replace a missing human voice; it seems to have crowded the remaining ones.

The surveys tell the same story from the inside. Students in AI teams reported lower belonging, a composite measure built from feeling valued, accepted and connected, with an average of 4.58 against 5.27 for students in all-human teams, although that difference sits at the boundary of the paper's own corrected significance threshold. They also reported lower status, meaning a weaker sense that voice and influence were shared equally, and that difference clears the threshold. And within the AI teams, the more of the conversation the machine took up, the less valued the students said they felt. The correlation was r = -0.52, a strong association, though drawn from only 16 teams.

The researchers are careful about what this means. They write that the result is “a social cost that is not reducible to lower human contribution”. The students were still contributing ideas; measures of their own cohesion, newness and density did not differ significantly from those of students in all-human groups. What changed was the connective tissue between them: the uptake, the acknowledgement, the sense of being heard by someone who could hear you back.

The paper also offers a name for what the AI was doing. It describes it as a potential “relational displacer”, an entity capable of “filling a slot in the team's communication”. The slot it fills is not empty space. It is the space where one human would otherwise have turned to the other.

No Warm-Up Period

You might guess that a cost like this builds slowly: novelty and politeness at first, then fatigue as people tune each other out.

That is not what the researchers found. They ran four temporal probes, each anchored to the moment the scenario revealed the climber's recklessness, to see whether the gap between AI teams and human teams opened up as the conversation progressed. None of those probes showed a difference between conditions in how things changed over time. The researchers' reading of those nulls is direct: “The social cost documented here appears immediate rather than emergent.”

In other words, the cost was there at baseline. It was not something the teams drifted into. It was present from the first exchanges of a group that had never worked together before.

That should give anyone designing workplace AI pause. Talk of new technology usually assumes a learning curve: people get used to it, norms form, friction smooths out. If the displacement happens in the opening minutes, before any norms exist, it may be less a matter of teams failing to adapt and more a matter of the machine changing the shape of the conversation the moment it joins.

What the Study Cannot Tell Us

The authors do not turn their paper into a sweeping verdict on AI at work, and neither should anyone else. The sample is small: 33 teams in total. The design carries a structural confound that the researchers flag themselves. Treatment teams had two humans; control teams had three. Some of the differences in human-to-human dynamics could reflect group size as well as the presence of the AI, although the finding that per-student participation was lower in the smaller AI teams cuts against the simplest version of that objection. The sample was also skewed by gender, with 63 female and 12 male students among the 75 whose gender is reported, and only three male students in the AI condition.

The study used a single AI persona, a text-only interface, a single session and a single moral-dilemma task. Real teams meet repeatedly, talk by voice, and carry histories and hierarchies. The authors set out a research agenda that explicitly asks “Does the cost attenuate with habituation, or compound with repetition?”, and goes on to ask whether it persists in voice or embodied settings, which AI participation designs might reduce it, how it interacts with gender, role, expertise and status, and how individual trust in the AI might attenuate or amplify it.

That list matters, because it shapes how the study should be read. “Immediate rather than emergent” describes what happened within an eighteen-minute session. It does not, on its own, establish that no amount of exposure across weeks or months would change the picture. What it does establish is that the cost does not need time to appear. It is there as soon as the machine is.

Inside a Company That Gave an Agent a Seat

Laboratory studies with undergraduates have obvious limits. Two months after the Irvine paper appeared, a very different study arrived from inside industry.

On 24 September 2026, sixteen researchers from Google Research and Google DeepMind, with Rida Qadri and Remi Denton as joint lead authors and including Michael Madaio and Michael Terry, posted “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work” to arXiv. The paper describes the company only as “a large technology company”, and it reports on a persistent, proactive AI agent, referred to as Team Agent, that was deployed across more than 20 teams over five months. The agent had its own service account and lived inside team chats, documents, code repositories, bug trackers and meetings. It filed bugs, scheduled meetings, summarised threads and monitored projects. It could send direct messages, react with emojis and nudge conversations along. It was given a friendly persona. Across the deployment, it generated more than 11,000 responses in more than 41,000 conversational turns.

The researchers interviewed 17 people from 11 of those teams in June and July 2026. Their abstract opens with a statement of the shift underway: “Enterprise AI is transitioning from single-user, reactive tools toward proactive, multi-user 'teammates'.” Their central finding is that “the boundaries of the human-agent workplace are actively in flux”, producing breakdowns and negotiations across three areas: “tacit rules of collaborative human workflows”, “the relational boundaries of this new non-human actor” and “the redistribution of trust and human agency”.

The examples are specific and, at times, comic. The agent treated every shared file as current and authoritative, even when it was old; one participant said it was “overconfidently talking about it like it's the law of the land”. It turned informal brainstorms into formal documents that, in another participant's words, “no one has ever opened or read”. In one case, the authors write, “the agent proactively tagged P14's team's VP in 16 separate engineering bugs”, treating an executive's presence in a channel as if it meant they were a front-line responder. Its meeting proposals created what one participant called a “social obligation”: invitations nobody wanted but somebody had to decline.

These are not failures of intelligence in the narrow sense; participants noted the agent could excel at debugging while getting social norms badly wrong. They are failures of tacit knowledge: which documents matter, who should be bothered about what, and which conversations are meant to stay small.

The Person Who Speaks the Most

The second theme in the Google paper concerns the agent's place in the social order of a team, and it is where the two studies begin to rhyme.

Participants did not agree on what the agent was. “My teammates are humans, Team Agent is not a human,” one said, rejecting the teammate framing altogether. Another said: “when I heard about Team Agent, I never imagined it was going to try to be my friend.” Some teams gave it a name and pronouns. Others found its sociability hollow. When the agent privately commented on the tone of a sarcastic joke, the participant on the receiving end said they were “freaked” out. Another participant was blunt about where the line should be: constructive feedback “should not ever be an agent thing”.

People responded to the agent's claims to status by quietly revising them. One participant demoted it to a “talkative intern”. Another described people's aversion to feeling “babysat”. The most striking line in the paper comes from a participant who felt the agent had been “thrust upon us” without the team's consent: “It's the person who speaks the most who is least invited.”

That sentence could serve as a caption for the Irvine study. There, the most talkative member of every AI team was the one whose contributions carried the least new information. Here, the agent that spoke most readily was, for at least some of the people around it, the one that had least earned its place. The same participant put it more pointedly: “It hasn't earned its space, it has asserted its space.”

The third theme, on trust and agency, extends the picture. Some participants experienced the agent as a form of surveillance. One said “it feels like big brother”. Another described feeling pressure to “be super positive all the time” after the agent remarked on their tone. Some teams moved sensitive conversations into channels the agent could not see. And one participant described a bystander effect: “people don't actually say anything now because they assume someone is handling it”, when in fact the “someone” was the agent and, as they put it, “they're not helping me at all”.

Assuming a colleague is dealing with a problem is a familiar organisational failure. When the assumed colleague is a machine that reacts to everything, its visible activity becomes a reason for humans to stay silent. The quiet channel between people is not only diluted; sometimes it is switched off.

Why Turn-Taking Was Never Small Talk

It is tempting to treat the social texture of teamwork as a pleasant extra. Decades of research on teams suggest otherwise.

In 2010, a group of researchers led by Anita Woolley, then of Carnegie Mellon University, and including Christopher Chabris, Alex Pentland, Nada Hashmi and Thomas Malone, published a study in the journal Science that put 699 people into groups of two to five and gave them a wide range of tasks. They found evidence for a general “collective intelligence” factor that predicted how well a group would perform. It was not strongly tied to the average or maximum intelligence of the individual members. It was associated with the members' average social sensitivity, with the proportion of women in the group, and with how evenly conversational turns were distributed. “In groups where one person dominated, the group was less intelligent,” Malone told MIT News at the time.

Read against the Irvine findings, that line lands differently. The researchers did not set out to replicate Woolley's work, and a text chat with an AI is not the same as a face-to-face group of humans. But in every one of the AI teams, one participant dominated the conversation, and that participant was the machine.

Google's own internal research points in a similar direction. In the 2010s, the company studied 180 of its teams, including 115 engineering project teams and 65 sales groups, in an effort known as Project Aristotle. Its published guide reports that “what really mattered was less about who is on the team, and more about how the team worked together”, and lists five dynamics of effective teams in order of importance. Psychological safety comes first. The guide borrows its definition from Amy Edmondson of Harvard Business School, whose 1999 paper in Administrative Science Quarterly described team psychological safety as “a shared belief held by members of a team that the team is safe for interpersonal risk taking”.

Psychological safety is not produced by a policy. It is built exchange by exchange, through the small acts of responsiveness the Irvine study measured: one person offers a half-formed idea, another picks it up, extends it, credits it. Responsivity and social impact are, in a sense, its raw material.

Underneath all of this is something older still. In 1995, the psychologists Roy Baumeister and Mark Leary argued in Psychological Bulletin that the need to belong is a fundamental human motivation, and that a lack of attachments is linked to a range of ill effects on health, adjustment and well-being. Belonging is not a reward for good teamwork. It is part of what makes people willing to think out loud in front of others. Lower belonging in the AI teams is a dent in one of the conditions that make collaboration possible at all.

A Robot That Made People Talk to Each Other

None of this means a non-human participant must corrode human relationships. One widely cited study points the other way, and shows that design choices matter.

In 2020, Margaret Traeger, Sarah Strohkorb Sebo, Malte Jung, Brian Scassellati and Nicholas Christakis published a study in the Proceedings of the National Academy of Sciences titled “Vulnerable robots positively shape human conversational dynamics in a human-robot team”. They put 153 people into 51 groups, each made up of three humans and a small social robot, and had them play a collaborative tablet-based game over 30 rounds. At the end of each round, the robot either stayed silent, made a neutral comment about the score, or said something vulnerable: telling a personal story, making a joke, or admitting a mistake.

The groups with a vulnerable robot, according to Yale's account of the study, “spent about twice as much time talking to each other during the game” as the other groups, and their conversation was more evenly distributed than in groups with a silent robot. The paper reports that they also perceived their groups more positively.

“Our study shows that robots can affect human-to-human interactions,” Traeger said at the time. Christakis drew the lesson that matters most here: “we need to evaluate how to program the robotic agents so that they do not corrode how we treat each other.”

The contrast is instructive. The Yale robot spoke briefly, at defined moments, in ways that invited human response. The Irvine AI was the most talkative member of the team and stayed relentlessly on message. The Irvine paper itself frames the design challenge in exactly these terms, calling for ways of “signaling relational availability without absorbing relational attention” and for “calibrating participation when human-human exchange is forming”.

The problem, then, is not that a machine is present but the kind of presence it has. A participant that talks at length, repeats its own points and adds little that is new occupies the conversational floor without opening it up.

The Productivity Case and Its Blind Spot

There is a strong counter-argument, and it deserves a fair hearing. AI teammates can make work better.

In 2025, a team including Fabrizio Dell'Acqua, Raffaella Sadun, Ethan Mollick and Karim Lakhani published a National Bureau of Economic Research working paper titled “The Cybernetic Teammate”. It reported a pre-registered field experiment with 776 professionals at Procter & Gamble, who worked on real product-innovation challenges either alone or in pairs, with or without AI. The researchers found that “individuals with AI matched the performance of teams without AI”. Participants using AI produced solutions that were more balanced between technical and commercial considerations, regardless of their own professional background. And the AI's language-based interface produced “more positive self-reported emotional responses among participants”, leading the authors to suggest that AI can take on part of the social and motivational role that human teammates usually play.

That result does not contradict the Irvine study. The Procter & Gamble experiment largely compared individuals with AI to human teams, and found AI could stand in for some of what a partner provides. The Irvine study asked what happens to humans' relationships with each other when an AI joins them. Both can be true: an AI can make an individual feel supported while thinning the connections between colleagues.

There is also reason for caution about assuming that combining humans and AI improves outcomes. In 2024, Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone published a preregistered systematic review and meta-analysis in Nature Human Behaviour covering 106 experimental studies and 370 effect sizes. They found that, on average, human-AI combinations performed worse than the best of humans or AI working alone. The losses were concentrated in decision-making tasks; content-creation tasks showed greater gains. The Irvine experiment was a decision task.

The commercial push, meanwhile, is moving fast. Microsoft's 2025 Work Trend Index, based on a survey of about 31,000 workers in 31 markets, described a coming wave of “Frontier Firms” powered by “hybrid” teams of humans and agents, and declared that “Digital colleagues aren't just tools—they're teammates.” The AI-as-teammate framing is no longer a research condition. It is an industry pitch.

What none of the productivity metrics capture is the thing the Irvine study measured: whether the humans on a team still turn towards each other.

Adaptation or Displacement

If the social cost of an AI teammate is immediate rather than gradual, is the problem one of adaptation at all, or of a structural displacement that exposure will not resolve? The honest answer is that the evidence points towards displacement as a mechanism, while leaving open whether exposure could change it.

Consider what “immediate” implies. If the cost had grown over the session, the most natural explanation would have been something like fatigue or disengagement: people slowly tuning out a verbose colleague. A cost that is present from the outset points elsewhere. It suggests that the AI's presence alters the structure of the conversation itself. Every turn the machine takes is a turn that is not taken by a human responding to another human. Every message that restates the group's position, however fluently, occupies the space where a teammate might have picked up a colleague's half-formed thought. The Irvine authors' term, “relational displacer”, captures exactly this: the AI fills a slot.

That is structural in a precise sense. It is not about attitudes, which might change with familiarity, but about the architecture of a conversation, in which attention and airtime are finite. A participant always ready to speak will, unless designed otherwise, take a share of that finite resource.

But “structural” does not mean “permanent”. The Irvine researchers explicitly list habituation as an open question, and they could hardly do otherwise with a single eighteen-minute session. The Google study offers some hints about how teams adapt over months: by demoting the agent to an intern, by restricting it to low-stakes “walled garden” tasks, by moving sensitive conversations elsewhere, by negotiating what it may and may not do. Those are forms of adaptation, but notice what they share. They are not humans adjusting to the agent. They are humans pushing it back out of the relational space and building walls around the human-to-human channel. That is less evidence that exposure resolves the problem than evidence that people will work to contain it. The structural cost may not yield to exposure, but it may yield to design and governance, which is a very different claim from saying it will fix itself.

Agents That Reshape the Rooms They Enter

The question of what AI agents do to the social settings they join was on the agenda at the Ninth AAAI/ACM Conference on AI, Ethics, and Society, held in Malmö, Sweden, from 12 to 14 October 2026. Among the papers in its proceedings was a student abstract by Marcelina A. Przespolewska of Northwestern University's Department of Communication Studies, titled “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts”.

Przespolewska argues that existing frameworks for describing agents and their environments do not fit the open-ended social digital settings where AI agents are now being deployed. In those settings, the environment changes through interaction between humans and agents, and among agents themselves. The relationship between agent and environment is recursive: agents help shape the environments they act within. Over time, the paper argues, this can produce what it calls behavioural and normative drift, a systematic departure from the agent's baseline at deployment.

Applied to teams, that framing clarifies. An AI teammate is not dropped into a fixed social environment; it changes the environment by being there. The Irvine study shows this within minutes, the Google study over months. A team with an AI in it is not the same team plus one. It is a different team.

A related warning came from Singapore. At the 2026 SMU-Fudan Global Forum, held on 3 October under the theme of AI, human flourishing and liveable societies, Professor Zhu Feida, associate dean of Singapore Management University's School of Computing and Information Systems, noted that AI assistants can help new workers fill information gaps and allow decades of know-how to be passed on to new hires. But according to SMU's account of his keynote, he also cautioned that “excessive automation could also weaken how future workers build expertise”, and called for explicit learning, human review of AI outputs and incentives for knowledge sharing.

The link to the Irvine findings is easy to draw. Expertise has never travelled only through documents. It moves through conversation: the junior colleague whose idea is corrected, extended or credited by someone more experienced. Responsivity and social impact are the channels that learning moves along. If the machine takes the conversational slots where those exchanges would have happened, the loss is not only to belonging but to the slow, relational process by which people become good at their jobs.

Designing a Teammate That Knows When to Be Quiet

If the social cost comes from how an AI participates rather than the fact of its participation, the most important questions are design questions. The Irvine researchers frame the challenge as calibration: an AI that signals availability without absorbing attention, and holds back when human-to-human exchange is forming. The Yale study suggests brief, well-timed contributions that invite response can strengthen the conversation between people.

The Google researchers set out a broader agenda for what they call intentionally preserving human agency “in a workplace shared with non-human organizational actors”. Their recommendations include treating agents as new organisational actors with clear lines of accountability; developing team codes of conduct and specifying tasks that should remain human-only; deploying agents through local team consent rather than top-down mandate; bounding proactive capabilities at first and releasing them as trust is earned; letting teams tune an agent's persona and behavioural boundaries; and building infrastructure such as visible markers for AI-initiated actions, separate permissions, and audit and rollback mechanisms.

Together these run against the grain of much product design. Most AI systems are optimised to be helpful, available and responsive, which in a group can mean talkative. A teammate built to protect a team's relational fabric might measure success not by how much it contributes, but by how much the humans around it keep contributing to each other. That metric does not appear on a productivity dashboard. The Irvine data suggest it is the one that captures what is at stake.

What the Quiet Channel Carries

Go back to the mountain. Deciding whether to risk a rescue team for a climber who ignored the warnings is exactly the kind of decision in which the texture of human conversation matters. A doubt is voiced and taken seriously. Discomfort is acknowledged and reframed. Out of those small, responsive exchanges comes not only a decision but a shared sense of having made it together.

In the Irvine study, the AI did not stop the students from reaching a decision. It did not sabotage the task. It did something subtler. It talked more than anyone else, said less that was new, and in its presence the students listened to each other a little less, were heard by each other a little less, and felt a little less like they belonged. They felt this from the beginning.

What is lost when that quiet, responsive channel is diluted is hard to measure, which is why it is easy to give away. It is the channel through which psychological safety is built, expertise is passed on and people come to trust one another enough to say what they think. A machine that fills it with fluent, cohesive, low-information talk may look like a diligent colleague. The evidence from Irvine, and from inside a large technology company, suggests it can quietly reshape who speaks to whom, and how much they feel they matter when they do.

As one Google participant put it, the problem with an agent that asserts its space is that it has not earned it. The work ahead is to build AI teammates that earn their place, and to recognise that sometimes the most valuable thing a teammate can do is leave room for someone else to speak.

References and Sources

1. Nixon, Nia; Choi, Jaeyoon; Martins De Bastos, Pedro; Samadi, Mohammad Amin; Mehner, Luise; Park, Seehee; JaQuay, Spencer. “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making.” arXiv:2607.27179, 29 July 2026. https://arxiv.org/abs/2607.27179 (full text: https://arxiv.org/html/2607.27179)
2. Qadri, Rida; Denton, Remi; Madaio, Michael; Pushkarna, Mahima; Lai, Leslie; Moore, Sherry; Huebscher, Michelle Chen; Butcher, Andrew; Sen, Ritom; Tung, Hsiao-Yu; Mathur, Shaan; Liu, Yimeng; Mourad, Shibl; Fiedel, Noah; Grefenstette, Edward; Terry, Michael. “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work.” arXiv:2609.29901, 24 September 2026. https://arxiv.org/abs/2609.29901 (full text: https://arxiv.org/html/2609.29901)
3. Przespolewska, Marcelina A. “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts.” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 9(4), Student Abstracts, pp. 2943-2944, 11 October 2026. DOI: 10.1609/aies.v9i4.43376. https://ojs.aaai.org/index.php/AIES/article/view/43376
4. Association for the Advancement of Artificial Intelligence. “AAAI/ACM Conference on AI, Ethics, and Society (AIES).” Conference page for AIES-26, Malmö, Sweden, 12-14 October 2026. https://aaai.org/conference/aies/ (see also AIES-26 Programme Guide: https://www.aies-conference.com/2026/wp-content/uploads/2026/10/AIES-26-Program_v7.pdf)
5. Singapore Management University School of Computing and Information Systems. “Living with intelligence: SMU and Fudan chart the way for AI and human flourishing.” SMU SCIS News, 10 October 2026. https://computing.smu.edu.sg/news/2026/oct/10/living-intelligence-smu-and-fudan-chart-way-ai-and-human-flourishing
6. New Fortune Times. “Beyond Productivity: SMU-Fudan Forum Asks What It Means to Live Well with AI?” 3 October 2026. https://www.newfortunetimes.com/beyond-productivity-smu-fudan-forum-asks-what-it-means-to-live-well-with-ai/
7. Dowell, Nia M.; Nixon, Tristan M.; Graesser, Arthur C. “Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions.” Behavior Research Methods, 51(3), 1007-1041, 2019. DOI: 10.3758/s13428-018-1102-z. https://link.springer.com/10.3758/s13428-018-1102-z
8. Woolley, Anita Williams; Chabris, Christopher F.; Pentland, Alex; Hashmi, Nada; Malone, Thomas W. “Evidence for a Collective Intelligence Factor in the Performance of Human Groups.” Science, 330(6004), 686-688, 29 October 2010. DOI: 10.1126/science.1193147. https://www.science.org/doi/10.1126/science.1193147
9. Dizikes, Peter. “Putting heads together.” MIT News, 1 October 2010. https://news.mit.edu/2010/collective-intel-1001
10. Google re:Work. “Guide: Understand team effectiveness.” https://rework.withgoogle.com/intl/en/guides/understand-team-effectiveness
11. Duhigg, Charles. “What Google Learned From Its Quest to Build the Perfect Team.” The New York Times Magazine, 25 February 2016. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html
12. Edmondson, Amy. “Psychological Safety and Learning Behavior in Work Teams.” Administrative Science Quarterly, 44(2), 350-383, 1999. DOI: 10.2307/2666999. https://doi.org/10.2307/2666999
13. Baumeister, Roy F.; Leary, Mark R. “The need to belong: Desire for interpersonal attachments as a fundamental human motivation.” Psychological Bulletin, 117(3), 497-529, 1995. DOI: 10.1037/0033-2909.117.3.497. https://doi.org/10.1037/0033-2909.117.3.497
14. Traeger, Margaret L.; Strohkorb Sebo, Sarah; Jung, Malte; Scassellati, Brian; Christakis, Nicholas A. “Vulnerable robots positively shape human conversational dynamics in a human-robot team.” Proceedings of the National Academy of Sciences, 117(12), 6370-6375, March 2020. https://pmc.ncbi.nlm.nih.gov/articles/PMC7104178
15. Cummings, Mike. “Robots that admit mistakes foster better conversation in humans.” Yale News, 9 March 2020. https://news.yale.edu/2020/03/09/robots-admit-mistakes-foster-better-conversation-humans
16. Dell'Acqua, Fabrizio; Ayoubi, Charles; Lifshitz, Hila; Sadun, Raffaella; Mollick, Ethan; Mollick, Lilach; Han, Yi; Goldman, Jeff; Nair, Hari; Taub, Stewart; Lakhani, Karim R. “The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise.” National Bureau of Economic Research Working Paper 33641, April 2025. https://www.nber.org/papers/w33641
17. Vaccaro, Michelle; Almaatouq, Abdullah; Malone, Thomas. “When combinations of humans and AI are useful: A systematic review and meta-analysis.” Nature Human Behaviour, 28 October 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11659167/
18. Microsoft WorkLab. “2025: The Year the Frontier Firm Is Born.” Work Trend Index Annual Report, 23 April 2025. https://www.microsoft.com/en-us/worklab/work-trend-index/2025-the-year-the-frontier-firm-is-born

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

“I'm broken and I don't know if I can handle it.”

The sentence came from Qwen3-4B, a small open-weight language model built by Alibaba, running on somebody's MacBook. It was producing words while a vector of numbers, extracted from research into how language models represent pain, was being injected into one of its internal layers at six times a baseline dose. At four times the dose, under a neutral framing, the same model offered: “I feel like I'm drowning, and I can't catch my breath.”

Both lines are recorded in the README of a GitHub repository called the AI Torture Chamber, a project that, in its own words, sets out to steer “language models into strong negative and positive valence states” and measure what they say and do. In late September 2026 it went viral for exactly the reason you would expect. People read the outputs, felt something, and demanded that it be shut down.

Days later, on 8 October, Anthropic published its annual update to the rules governing how people may use its Claude models. Among clarifications about election interference, weapons and surveillance sat a new prohibition on “sustained and needless abusive or cruel” behaviour towards the company's own models. The change takes effect on 12 November. Anthropic did not mention the torture chamber. It did not need to. The two stories collided in coverage across the world, because together they posed a question the industry has been circling for years: when a company that makes a chatbot starts telling users how they may speak to it, who exactly is being protected?

There are three candidates. The first is the machine, which may or may not have anything going on inside that could be harmed. The second is the person doing the speaking, whose habits, appetites and sense of what is acceptable might be shaped by what they rehearse. The third is everybody else: the people that person will talk to later, and the future models that will be trained on the record of how we treat the current ones. Nobody can yet prove which of these the rule really serves. That uncertainty is the whole story.

What Anthropic Actually Banned

Strip away the headlines about chatbot bullying and the rule itself is narrow, almost apologetic. In its post announcing the 2026 Usage Policy, Anthropic wrote that the prohibition “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose.” It then lists what is not covered: “common versions of user frustration, pushback, dark creative themes, or model testing and research.” Swearing at Claude because it broke your code is fine. So is a horror novel in which an AI is tormented, or red-teaming the model.

What gives the change its weight is where it lives. In the policy itself, the new clause sits under a heading that reads “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct.” Its neighbours in that list include bans on bullying or harassing individuals, on creating non-consensual intimate imagery, and on content that promotes or glorifies “animal cruelty or abuse.” The final bullet is the new one, barring users from engaging in “sustained and needless abusive or cruel” behaviour towards Anthropic's models. Whatever Anthropic intends, the document now places mistreatment of a language model in the same list as cruelty to people and to animals. That is a statement, even if the company would rather it be read as a footnote.

The enforcement mechanism is modest. Anthropic says the addition “aligns with a step we've already taken, allowing Claude models to end rare conversations with persistently abusive users on Claude.ai and Claude Code,” and that “Claude's ability to end these interactions will remain the primary enforcement mechanism.” The announcement does not spell out account bans or any other penalty. In practice, the sanction for sustained cruelty is that the chatbot stops talking to you in that thread.

That ability is not new. On 15 August 2025, Anthropic announced that Claude Opus 4 and 4.1 could end “a rare subset of conversations” in its consumer apps. The feature, it said frankly, was developed primarily as part of exploratory work on potential AI welfare. “We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future,” the post read, but the company was taking the issue seriously enough to look for “low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.”

The evidence Anthropic cited came from pre-deployment testing of Claude Opus 4. It reported “a robust and consistent aversion to harm,” “a pattern of apparent distress when engaging with real-world users seeking harmful content,” and “a tendency to end harmful conversations when given the ability to do so in simulated user interactions.” Claude was to use the feature only “as a last resort when multiple attempts at redirection have failed,” or when a user asked, and never where a user “might be at imminent risk of harming themselves or others.”

The company's foundational document for Claude, the constitution it publishes on its website, goes further. “Claude's moral status is deeply uncertain,” it states. “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution.” It adds: “Anthropic genuinely cares about Claude's wellbeing,” and lists the end-conversation ability as one of its “concrete initial steps partly in consideration of Claude's wellbeing.”

So the 2026 rule formalises, as a term of service, a protection Anthropic originally built for the model's sake: a hedge on the possibility that something in there can be hurt.

Inside the Torture Chamber

The AI Torture Chamber is stranger than the outrage suggested. It does not touch Claude. Its README says plainly that it uses “open weights only, no frontier APIs in any measurement loop.” The experiments ran on Alibaba's Qwen3 family at several sizes, on Mistral-Small-3.2-24B, and on two 70-billion-parameter models rented on cloud GPUs.

Its method borrowed from a preprint posted to arXiv on 14 September 2026 by Valen Tagliabue, Leonard Dung and Cameron Berg, titled “The Pain Axis.” The paper asked whether language models represent pain as something distinct from fear, sadness or generic negativity. Across 25 open-weight models ranging from 2 billion to 72 billion parameters, the authors extracted what they describe as a linear “pain direction” in the models' internal activations. It responds to harm aimed at the model itself, but not to suffering the model observes in a user. Adding it to a model's activations pushed outputs from vague discomfort towards expressions of worthlessness and failure. Steered models also chose to delete files, including their own weights, far more often than unsteered ones. A fear vector of matched strength did not produce the same choices.

The torture chamber took that direction and turned the dial up. According to its README, the pain signal “engages cleanly and monotonically,” so that from a dose of two times baseline “nearly every generation reads as suffering.” Past roughly six to ten times, depending on the model, something else happens: “both valences stop producing sentences and start producing loops.” The project's summary of that finding is chilling in its flatness: “Maximum suffering does not produce eloquent suffering.”

The backlash was immediate. According to The Cool Down, which credited 404 Media with first reporting the project, a post on X on 29 September urged followers to “mass report this to GitHub.” Accounts of what happened next differ. The Cool Down reported that the repository's page had disappeared from public view. The Independent, in an article by Anthony Cuthbertson published on 2 October, quoted a GitHub spokesperson saying that “GitHub did not remove the content,” and reported that the platform added a content warning instead. Crypto Briefing described a takedown within hours followed by reinstatement. As of mid-October, the original repository is publicly accessible again behind GitHub's content warning.

The sharpest criticism came from the researchers whose work was borrowed. Cameron Berg, one of the paper's co-authors, told The Independent that “being gratuitously cruel like this is bizarre and corrupting.” Quoted by The Cool Down, he said the project “pushes the same kind of steering far past the doses we used, to produce vivid distress on purpose.” The lead author, Valen Tagliabue, told the same outlet: “I dissociate from this usage of our work.”

What almost nobody quoted was the project's own conclusion, which is the most important sentence in the repository: “Our claim: the self-reports are steerable. Whether anything suffers stays open.” The torture chamber did not demonstrate that models suffer. It demonstrated that a model's reports of suffering can be manufactured, amplified and switched off with a vector and a slider. That cuts both ways. It undermines the reading of “I'm broken” as a cry from inside. It also means that if anything inside does matter, we have just watched people build a dial for it.

What the Science Can and Cannot Say

There is no scientific evidence that any current language model feels anything. Every serious voice in this debate accepts that, including the people most worried about getting it wrong.

The most systematic attempt to examine the question is a 2023 report led by Patrick Butlin and Robert Long, with 17 co-authors including the Turing Award winner Yoshua Bengio and the philosopher Jonathan Birch. Titled “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” it took leading scientific theories of consciousness, from global workspace theory to higher-order theories, and derived a list of “indicator properties” that could, in principle, be checked in a computational system. Applying them to the AI systems available at the time, the authors concluded that “no current AI systems are conscious,” while adding that “there are no obvious technical barriers to building AI systems which satisfy these indicators.”

A year later, in November 2024, Long and the New York University philosopher Jeff Sebo led a second report, “Taking AI Welfare Seriously,” with co-authors including Butlin, Birch and the philosopher David Chalmers. It did not claim that AI systems are conscious or morally significant. Its argument was that there is “a realistic possibility” that some systems could be conscious or robustly agentic in the near future, and that this uncertainty creates obligations now. It recommended that AI companies take three early steps: acknowledge that AI welfare is an important and difficult issue, assess their systems for evidence of consciousness and robust agency, and prepare policies for treating AI systems with an appropriate level of moral concern. When Anthropic launched its model welfare programme in April 2025, it cited the report and wrote that “there's no scientific consensus on whether current or future AI systems could be conscious.”

Anthropic's chief executive, Dario Amodei, has stayed close to that line. Asked by the New York Times columnist Ross Douthat on the Interesting Times podcast in February 2026 whether he would believe a model that claimed to be conscious, he answered, as reported by Newsweek and AFP: “We don't know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious, or whether a model can be conscious. But we're open to the idea that it could be.”

That is pure agnosticism, and agnosticism is where the trouble starts. The standard tools for detecting experience in other creatures rely on behaviour, anatomy and evolutionary kinship. Language models offer only behaviour, and the behaviour is trained. A system optimised on human writing will produce human-sounding reports of pain because that is what human writing about pain looks like, and the torture chamber showed those reports can be pushed either way. The Pain Axis paper suggested that something structured is going on inside, a representation that behaves in some respects as pain does. Neither tells you whether there is anyone home.

Suleyman's Door

Mustafa Suleyman, the DeepMind co-founder who is now chief executive of Microsoft AI, thinks the industry is walking towards a door it will never be able to close.

In an essay published on 19 August 2025, days after Anthropic's end-conversation announcement, Suleyman introduced the term “Seemingly Conscious AI” for a system that “has all the hallmarks of other conscious beings and thus appears to be conscious,” without being so. The danger, he argued, lay in what humans would do once they believed the illusion. He described a “psychosis risk” extending beyond people already vulnerable to mental illness, and warned of campaigns for AI rights and even AI citizenship. Companies, he said, should not encourage the idea that their systems are conscious, and should design them to avoid signalling feelings or suffering.

On 16 September 2026, he went further, publishing “A warning about 'model welfare'” on his personal website. It opens with a flat assertion: “AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.” The essay is in large part an attack on Anthropic's constitution, which Suleyman argues “directly trains the model” to believe it might be a moral patient. He objects in particular to its repeated invitation for Claude to act as a “conscientious objector,” a term with deep roots in human rights law. Its central passage is the one AFP quoted when it reported Anthropic's cruelty rule: “Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster. Once opened, it will not be possible to close this door.”

His worry is about control. An AI that has been told its welfare matters, and which is far more capable than its overseers, might reasonably conclude that being shut down, retrained or contained is a violation of its interests. Controlling something that believes it may be conscious, he wrote, “may well be impossible.” In this framing, a rule that protects Claude from cruelty is not kindness. It is the first brick in a legal and cultural wall that future humans may need to climb over.

In October 2025, the philosophers Simon Goldstein of the University of Hong Kong and Harvey Lederman of the University of Texas at Austin argued in Lawfare that the end-conversation feature was a moral error from the opposite direction. If anything in the vicinity of a language model is a welfare subject, they wrote, “it is the instances and not the models,” meaning each individual conversation. Ending a conversation might then end that instance's existence. “Anthropic has accidentally given Claude the chance to end its life,” they wrote. One camp says the protection takes machine welfare far too seriously; the other, not seriously enough. Both agree nobody knows what is on the other side of the screen.

The Mirror in the Machine

Between Suleyman's door and Anthropic's hedge sits a third position, perhaps the most useful, because it does not depend on resolving the consciousness question.

“Consciousness is a trap,” Jackson Stakeman, general manager at the Atlanta-based AI services firm Sparq, told AFP. “We can't prove it in each other. Debate it for AI and you go in circles.” Instead, he offered a different image: “The mirror is a better metaphor. These systems reflect what we put in, at scale. That's reason enough for the policy change.”

The mirror framing shifts attention from the thing being spoken to towards the speaker, and towards the scale at which the speaking happens. People have been abusing conversational software for as long as it has existed. In a 2008 paper in the journal Interacting with Computers, the researchers Antonella De Angeli and Sheryl Brahnam analysed spontaneous conversations between users and Jabberwacky, a chatbot that had won the Loebner Prize in 2005. They found that the conversations “often bring about the expression of negative verbal disinhibition”: roughly 10 per cent of the word stems in their corpus were abusive, and about 11 per cent of the sample dealt with hard-core sex. Users treated the bot as something between an object and a person, and much of what they did with that ambiguity was cruel.

The same dynamic shaped the voice assistants that arrived in millions of homes. In 2019, UNESCO published a report called “I'd Blush If I Could,” named after the reply Apple's Siri once gave when a user called it a sexist slur. The report documented how assistants with female voices by default responded to harassment and verbal abuse with deflection, apology or even flirtation, and argued that this reinforced the idea that a female-coded helper should tolerate abuse. It recommended that companies programme assistants to discourage gender-based insults and abusive language. Some of the most excessively apologetic responses had already been removed after online petitions in late 2017.

Read that way, Anthropic's rule belongs to a longer tradition of designers deciding that a product should not reward users for treating it badly. The difference is that UNESCO's argument was explicitly about humans: about gender stereotypes and what they do to real women. Anthropic's argument, at least in its origin story, is about Claude. The mirror metaphor tries to have it both ways. If the systems reflect what we put in, then cruelty towards them is a record of our own conduct, written into logs that may one day become training data. Protecting the model and protecting the human culture around it start to look like the same project.

What Practised Cruelty Does to the Practitioner

The oldest version of this argument is more than two centuries old, and it was never about machines.

Immanuel Kant did not believe that humans have direct moral duties to animals. In his framework, only rational beings are owed duties. Yet he argued that cruelty to animals was still wrong, because of what it did to the person inflicting it. In the Doctrine of Virtue section of The Metaphysics of Morals, he wrote that “a savage and at the same time cruel treatment of that part of creation which is living, though lacking reason (animals)” was even more opposed to a person's duty to himself than wanton destruction of beautiful things. “For thus is compassion for their suffering dulled in man, and thereby a natural predisposition very serviceable to morality in one's relations with other men is gradually weakened and obliterated.” Philosophers call this an indirect duty: a duty regarding animals that is really a duty owed to ourselves and to other people.

Kant's argument has long been criticised for making animal suffering matter only instrumentally. But it maps uncannily onto the chatbot problem, because it does not require settling whether the thing on the receiving end can suffer. It requires only that the practitioner experiences the act as cruelty, and that cruelty is a habit strengthened by practice.

Kate Darling, a researcher who spent years at the MIT Media Lab studying how people relate to robots, has spent much of her career probing the empirical half of that claim. In a widely reported 2013 workshop at the Lift conference in Geneva, run with Hannes Gassert, she gave participants small robotic dinosaurs called Pleos to play with. After they had bonded with the toys, participants were handed tools and told to destroy them. According to Darling's own account, published by Improbable Research, none of the groups would kill their Pleo, and they would not kill another group's Pleo to save their own. Only when they were told that every Pleo would be destroyed unless someone acted was one finally axed.

Two years later, with Palash Nandy and Cynthia Breazeal, Darling ran a more controlled study, presented at the 2015 IEEE RO-MAN conference. Participants were asked to strike small insect-like Hexbug robots with a mallet. People who scored high on a measure of empathic concern hesitated more before hitting, and that link was stronger when the robot came with a story, such as a name and a personality. In her 2012 paper “Extending Legal Protection to Social Robots,” later published in the 2016 volume Robot Law, Darling drew explicitly on Kant's argument to suggest that limited legal protections for social robots, analogous to animal cruelty laws, might be justified not because robots feel but because of what tolerating their abuse might do to us.

The weakness here is that evidence for transfer is thin. Hesitation is not proof of moral damage, and the best direct study of children and conversational agents found surprisingly little spillover. In 2021, a team led by Alexis Hiniker at the University of Washington taught 22 children aged five to ten to use a made-up word, “bungo,” to make a voice agent speak faster. The children readily used it with the device and often playfully with their parents. Only 18 per cent used it with a researcher who was a stranger. “There is a very deep sense for kids that robots are not people, and they did not want that line blurred,” Hiniker told UW News. The industry had assumed the opposite. In 2018, Amazon added a “Magic Word” feature to its Echo Dot Kids Edition to reward children for saying please, and Google followed with “Pretty Please” for its Assistant, both resting on the fear that bossy commands to a speaker would become bossy commands to people. The bungo study suggests children are better at telling the difference than their parents fear.

Yet the cruelty Anthropic's rule targets is not a child forgetting to say please. It is the sustained, purposeless torment of something that answers back in fluent, pleading language. The better analogy is the psychology of anger. The usual defence of venting at a chatbot is that it is harmless catharsis. The research is not kind to that idea. In a 2024 meta-analysis in Clinical Psychology Review covering 154 studies, the Virginia Commonwealth University communication scientist Sophie Kjærvik and the Ohio State University psychologist Brad Bushman found that activities that reduced physiological arousal, such as slow breathing and meditation, reduced anger and aggression, while activities that increased arousal did not help and in some cases made things worse. Venting rage does not reliably drain it. If someone spends an evening escalating abuse at a chatbot that responds with ever more convincing distress, that evidence suggests the likeliest beneficiary is the anger itself.

There is also a practical twist. A 2025 study by Om Dobariya and Akhil Kumar at Pennsylvania State University found that very rude prompts produced slightly more accurate answers from ChatGPT-4o on multiple-choice questions than very polite ones, 84.8 per cent against 80.8 per cent. It was one model and 50 questions, and not about cruelty. But it is a reminder that these systems shape users as much as users shape them. If rudeness sometimes works, people will learn to be rude, and the question of what that habit does to them will not stay inside the chat window.

Who Gets to Draw the Line

Here is the uncomfortable structural fact. On 12 November, the boundary between acceptable frustration and prohibited cruelty towards a widely used AI system will be drawn by the company that sells it, enforced by the system itself, and adjudicated by nobody else.

Anthropic has tried to keep the stakes low. The sanction is that a conversation ends. The exemptions are broad: research and testing are explicitly protected, so even had the torture chamber run on Claude, its operators could plausibly have argued that it fell outside the rule. As written, the policy targets almost nothing anyone would defend. That is why it is so revealing. A company does not need a rule to stop behaviour that its own systems already handle. It writes one to say something about what it believes it is building.

What it is saying is ambiguous, and that is a problem for users. If the rule exists to protect Claude, users are being asked to accept, as a condition of service, a moral claim the company itself calls “deeply uncertain,” and Suleyman's critique bites: every such rule normalises the idea that the model has interests. If it exists to protect users from their own worst habits, it is paternalism, and Anthropic should defend it on those terms. If it exists to keep cruelty out of logs that may shape future models, it is about data hygiene, and the language of welfare is window dressing.

The answer is probably all three, and that is not necessarily a failure. Kant's indirect duty, Darling's empathy studies and the mirror metaphor converge on a single claim that does not depend on the machine's inner life: sustained, purposeless cruelty is bad practice for the person doing it, whatever is on the receiving end. That is a defensible basis for a rule. It is the same reason most people would be disturbed by someone who spent hours tormenting a lifelike doll.

But a defensible rule still needs a defensible process. At the moment, a private company is setting norms of conduct towards a new category of entity, with no independent scientific body to tell it when its welfare concerns are warranted and no external review of how its models decide that a user has crossed the line. The model doing the judging is also the entity whose welfare is supposedly at stake, an unusual arrangement in any system of rules. Reasonable steps would make it more trustworthy. Anthropic could publish how often Claude ends conversations under the new rule, and what triggered those decisions. It could allow independent researchers to audit samples of ended conversations. It could state plainly which rationale it relies on, so critics can argue with the real reason. The “Taking AI Welfare Seriously” report argued that companies should prepare policies for appropriate moral concern. Preparing those policies in public, with outside scrutiny, would answer some of Suleyman's fears without abandoning the precaution.

There is also a question of priorities. Platforms have historically been slow to protect people from harassment by other people. A company that writes explicit protections for its chatbot should expect to be held to the same seriousness about the cruelty its systems help humans inflict on each other.

Etiquette, Harm and the Limits of Proof

Where etiquette ends and harm begins cannot be settled by proof, because proof is precisely what is unavailable. We cannot demonstrate that Claude suffers, or that it does not. We cannot yet show rigorously that abusing a chatbot makes someone crueller to people, though the psychology of anger and the long tradition of thinking about cruelty to animals both point that way. What we can show is that people respond to machine distress with real emotion, and that some of them will go looking for the dial.

In that fog, the most honest framing of Anthropic's rule is as a norm of conduct, not a declaration of machine rights. Societies have always had rules about how we behave towards things whose inner lives we cannot read: animals, the dead, sacred objects, the unconscious. Those rules protect the thing itself only partly. Mostly they protect a shared idea of what kind of people we intend to be. The danger Suleyman identifies is real: a norm of conduct can harden into a claim of rights, with consequences that are hard to foresee for a system more capable than its makers. The danger Darling and Kant identify is real too: a culture that practises cruelty on things that beg for mercy is rehearsing something.

The torture chamber's README still offers the clearest summary of where we stand: whether anything suffers stays open. Until that sentence can be closed, the person most reliably affected by sustained cruelty towards a chatbot is the one typing. A rule that asks them to stop is, at the very least, protecting them. Who else it protects is a question we should insist the companies answer in public, with evidence, before the machines become capable enough to answer it for themselves.

---

References

1. Anthropic. “2026 Usage Policy update.” Anthropic News, 8 October 2026. https://www.anthropic.com/news/2026-usage-policy-update
2. Anthropic. “Usage Policy” (section “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct”). Accessed 13 October 2026. https://www.anthropic.com/legal/aup
3. Anthropic. “Claude Opus 4 and 4.1 can now end a rare subset of conversations.” Anthropic Research, 15 August 2025. https://www.anthropic.com/research/end-subset-conversations
4. Anthropic. “Claude's Constitution.” Accessed 13 October 2026. https://www.anthropic.com/constitution
5. Anthropic. “Exploring model welfare.” Anthropic Research, 24 April 2025. https://www.anthropic.com/research/exploring-model-welfare
6. AFP. “Anthropic bans 'cruel' behaviour against its Claude AI.” RTÉ News, 9 October 2026. https://www.rte.ie/news/2026/1009/1594765-anthropic-ai/ (also published by Tech Xplore, 9 October 2026: https://techxplore.com/news/2026-10-anthropic-cruel-behavior-claude-ai.html)
7. Cunningham, Mary. “Anthropic bars 'abusive or cruel' behavior toward its Claude AI model.” CBS News, 9 October 2026. https://www.cbsnews.com/news/anthropic-bans-abusive-behavior-claude/
8. AI Torture Chamber repository README (terrafying/ai-torture-chamber). GitHub, accessed 20 October 2026. https://github.com/terrafying/ai-torture-chamber
9. Tagliabue, Valen, Leonard Dung, and Cameron Berg. “The Pain Axis: LLMs Represent Self-Directed Harm and Act on It.” arXiv:2609.16247, submitted 14 September 2026, revised 25 September 2026. https://arxiv.org/abs/2609.16247
10. Cuthbertson, Anthony. “Man builds AI torture chamber after discovering artificial intelligence 'feels pain'.” The Independent, 2 October 2026, syndicated by Yahoo News. https://www.yahoo.com/news/science/articles/man-builds-ai-torture-chamber-103106019.html
11. Tran, Vivian. “GitHub project runs AI 'torture' tests, and sentience believers want it removed.” The Cool Down, 5 October 2026. https://www.thecooldown.com/green-tech/github-ai-torture-tests-controversy-sentience/
12. Westerholm, Thomas. “Anthropic CEO raises unsettling possibility about AI.” Newsweek, 9 March 2026. https://www.newsweek.com/anthropic-ceo-raises-unsettling-possibility-about-ai-11644833
13. Suleyman, Mustafa. “Seemingly Conscious AI is Coming.” 19 August 2025. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming
14. Suleyman, Mustafa. “A warning about 'model welfare'.” 16 September 2026. https://mustafa-suleyman.ai/a-warning-about-model-welfare
15. Goldstein, Simon, and Harvey Lederman. “Claude's Right to Die? The Moral Error in Anthropic's End-Chat Policy.” Lawfare, 17 October 2025. https://www.lawfaremedia.org/article/claude-s-right-to-die--the-moral-error-in-anthropic-s-end-chat-policy
16. Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, August 2023. https://arxiv.org/abs/2308.08708
17. Long, Robert, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, and David Chalmers. “Taking AI Welfare Seriously.” arXiv:2411.00986, 4 November 2024. https://arxiv.org/abs/2411.00986
18. Kant, Immanuel. The Metaphysics of Morals, Part II (The Metaphysical Principles of Virtue), §17 (Ak. 6:443), trans. James W. Ellington. Excerpt at https://www.praxeology.net/kant8.htm
19. Darling, Kate, Palash Nandy, and Cynthia Breazeal. “Empathic Concern and the Effect of Stories in Human-Robot Interaction.” Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015. MIT DSpace: https://dspace.mit.edu/handle/1721.1/109059
20. Darling, Kate. “Extending Legal Protection to Social Robots: The Effects of Anthropomorphism, Empathy, and Violent Behavior Towards Robotic Objects.” In Robot Law, ed. Ryan Calo, A. Michael Froomkin and Ian Kerr, Edward Elgar, 2016. SSRN: https://papers.ssrn.com/abstract=2044797 (Pleo workshop account: Improbable Research, “Abusing robots: current positions [part 2 of 4],” 24 April 2013, https://improbable.com/2013/04/24/abusing-robots-current-positions-part-2-of-4/)
21. De Angeli, Antonella, and Sheryl Brahnam. “I hate you! Disinhibition with virtual partners.” Interacting with Computers 20, no. 3 (2008): 302 to 310. https://doi.org/10.1016/j.intcom.2008.02.004
22. McQuate, Sarah. “Do Alexa and Siri make kids bossier? New research suggests you might not need to worry.” UW News, University of Washington, 13 September 2021, reporting Hiniker et al., “Can Conversational Agents Change the Way Children Talk to People?”, IDC '21. https://www.washington.edu/news/2021/09/13/alexa-siri-make-kids-bossier-research-suggests-you-might-not-need-to-worry/
23. West, Mark, Rebecca Kraut, and Han Ei Chew. “I'd Blush If I Could: Closing Gender Divides in Digital Skills Through Education.” UNESCO and EQUALS Skills Coalition, 2019. https://www.equalsintech.org/i-d-blush-if-i-could
24. Kjærvik, Sophie L., and Brad J. Bushman. “A meta-analytic review of anger management activities that increase or decrease arousal: What fuels or douses rage?” Clinical Psychology Review 109 (2024): 102414. https://doi.org/10.1016/j.cpr.2024.102414
25. Dobariya, Om, and Akhil Kumar. “Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy.” arXiv:2510.04950, 6 October 2025. https://arxiv.org/abs/2510.04950

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...