SmarterArticles

Somewhere behind a police call in Heber City, Utah, a television was playing Disney's The Princess and the Frog. The officer's body-worn camera picked it up, because body-worn cameras pick up everything: the hum of a refrigerator, a dog, a film. That audio went into an artificial intelligence system the department had bought to turn recordings into police reports. The system did exactly what it was built to do. It listened, it narrated, and it produced a document in which the officer turned into a frog.

The story went around the world in January 2026. Sergeant Rick Keel of the Heber City Police Department told FOX 13 News that “the body cam software and the AI report writing software picked up on the movie that was playing in the background, which happened to be 'The Princess and the Frog.'” He added the line every department has since repeated in some form: “That's when we learned the importance of correcting these AI-generated reports.” FOX 13 reported the offending narrative came from Draft One, sold by Axon, the largest supplier of body cameras to American police. Heber City had been trialling it alongside a rival called Code Four. The department kept using AI. It simply added more review.

Here is the thing about the frog. It is the safest error the technology can make, because it is self-cancelling. No sergeant approves it, no prosecutor files it, no judge reads it without laughing, and no defendant goes to prison because of it. An error that absurd is a smoke alarm, not a fire. It tells you the machine is in the building.

The error that should frighten you is the one that reads perfectly. It is grammatical, plausible, consistent with the rest of the narrative, and wrong by a single preposition. It is the difference between a suspect who reached toward his waistband and a suspect who reached into it. One describes an ambiguous movement. The other describes a man going for a gun, and can justify a shooting, a search, a charge, a plea. They differ by two letters, and no reviewer on earth reliably catches that at eleven at night at the end of a twelve-hour shift, six calls deep, with the machine's version already on the screen looking finished.

That asymmetry is the whole problem. And Wyoming has just walked into it.

What the Cowboy State Bought Before It Decided Anything

On 23 August 2026, Cowboy State Daily reporter Kolby Fedore published an account of police departments in Casper, Evanston and Sheridan each testing artificial intelligence to write reports and look at evidence. Only one vendor was named. Evanston, according to Police Chief Mike Vranish, is testing Code Four, software founded in April 2025 by two MIT students, George Cheng and Dylan Nguyen, and now used by roughly 165 law enforcement agencies. Code Four, Fedore noted, markets itself as “The AI Detective That Never Sleeps.”

The company's own site is less whimsical and more instructive. It promises a 65 per cent average reduction in report writing time, fifty dollars per officer per month, and eighty thousand dollars in annual budget saved per department. Its report product “ingests audio, visual, and other meta data from each frame of an officer's body-worn camera to create a comprehensive narrative of every situation.” A companion product, INSIGHTS, described in sponsored content Cheng wrote for the trade publication Police1 in July 2026, pulls “digital evidence, records and jail-management data, body-camera footage, phone dumps, call logs, recorded audio and case files into a single layer a detective can question directly.” It can draft warrants, affidavits and prosecutor jackets.

Read that list slowly. A warrant application persuades a judge to let the state enter your home; an affidavit of probable cause persuades a judge to hold your body. These are not administrative artefacts. They are the hinges on which constitutional rights swing.

Fedore's reporting established the gap that gives this story its edge. No Wyoming law specifically requires an original AI-generated police report to survive after an officer edits it, and none requires the finished report to disclose that a machine helped write it. The technology is in the building. The rules are not.

Gay Woodhouse, Wyoming's thirty-second attorney general from 1998 to 2001 and the first woman to hold the office, previously an assistant United States attorney and chief deputy attorney general, told Cowboy State Daily that “the thing that worries me about AI is just the ability to hallucinate,” and that Wyoming needs “to get on it.” She put her finger on the mechanism: “The main thing in criminal prosecutions is disclosure.” And then, with the dry economy of a former prosecutor: “It'd be awful to lose a case because you used AI and didn't disclose it.”

Casper criminal defence attorney Keith Nachbar called it “uncharted territory” and delivered the sentence that ought to be nailed above every police chief's desk in the state: “AI isn't going to be taking the witness stand.” Then a question nobody in Wyoming can answer: “How do you look behind the curtain and figure out what AI thought?”

Evanston Police Chief Mike Vranish supplied the counterweight every department offers. AI can do in minutes what takes officers hours. But a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. So, said Vranish, “everything's got to be verified and validated by an actual person.”

That last sentence is the load-bearing beam of the entire industry. We will come back to whether it holds.

Two Letters and a Life

A generative model producing a police narrative from body-camera audio and video is not reasoning about what happened. It is producing the most probable next token given everything before it. When the input is clear, the output is usually fine. When the input is ambiguous, and a real incident is nothing but ambiguity, the model does not flag the ambiguity. It resolves it, picking the likelier phrasing and writing it with the same fluency it uses for the parts it got right. There is no tonal difference between the sentences it knows and the sentences it guessed.

Human officers make errors too, and police reports have always been contested documents. The difference is the shape of the error. A tired officer produces a report with the texture of exhaustion: gaps, hedges, “I believe”, “it appeared that”, the occasional flat contradiction a defence lawyer can hold up in cross-examination. Those artefacts are evidence, telling a jury how confident the witness actually was. Machine prose has no such texture. It is uniformly confident, smoothing hedges into assertions because assertions are more probable in the training distribution of police reports than hedges are. The uncertainty that lived in the officer's memory is laundered out before anyone reads it.

The research base bears this out with unusual precision. In May 2026, criminologist Ian T. Adams of the University of South Carolina and colleagues Matt Barter, Kyle McLean, Irick A. Geary Jr, Alexis Fabila and Josh McCrain published a study on CrimRxiv titled “A Good College Essay but a Bad Police Report.” They ran a triple-blind evaluation in which 92 expert reviewers, sergeants and above with an average of 21.8 years of responsibility for approving reports, assessed 80 police reports, 20 drafted with Axon's Draft One. The reviewers did not know which was which.

Two findings matter. AI-assisted reports were rated substantively and significantly worse on accuracy, an effect moving a report from roughly the fiftieth percentile to the thirty-sixth, and text analysis found them longer-worded, more complex and less readable. They read like good college essays. They were worse police reports. And the expert reviewers could not tell which reports were written with AI. Their accuracy on that question was no better than a coin flip.

Hold those together. The exact people whose review is supposed to be the safeguard rated AI reports as less accurate whilst being unable to identify them. They sensed something was off and could not locate it. That is the signature of a failure mode human review is structurally poor at catching.

The Signature Stays, the Perception Goes

A police report's evidentiary power does not come from its literary quality. It comes from provenance. It is a contemporaneous account written by a person who was present, who perceived events through functioning senses, who can be cross-examined about those perceptions, who can be impeached with prior inconsistent statements, and who faces professional and sometimes criminal jeopardy for falsifying it. Every one of those guarantees attaches to a human being standing in a room.

Chief Vranish put it better than most legal scholars: a camera cannot smell the gunpowder hanging in a room, or the alcohol on a driver's breath. That is not a quaint remark about sensor limitations. It is a statement about the epistemology of the document. The camera captures a slice of a person's perception; the model writes a narrative from the slice and presents it as the whole.

What AI drafting does, quietly, is sever the sentence from the perception whilst leaving the signature in place. The officer still signs. The report still enters the file as his account. But a proportion of its sentences, and nobody can say which, originated not in his memory of the room but in a model's reconstruction of what a room like that usually sounds like.

This is what the machine is for. Comprehensiveness, which Code Four promises, is precisely the property a human account does not have and should not claim. Officers miss things, and the gaps in an honest report are informative. A machine that fills them is not improving the document; it is contaminating it with plausible invention.

The academic literature has caught up faster than the statutes have. In March 2026, a team led by Muffy Calder and Marion Oswald, with Elizabeth McClory-Tiarks, Michele Sevegnani and Evdoxia Taka, submitted a paper to arXiv titled “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” They mapped 15 policing tasks that large language models could plausibly handle, from transcription and translation to witness statement generation, corroboration identification and police report generation, and set against them 17 categories of risk illustrated with more than 40 worked examples of how those risks propagate through a case.

Their taxonomy of output risk begins with completeness, meaning omissions and additions, and accuracy, meaning outright factual error. In one proof-of-concept trial they describe, a system produced a description of a witness as “a woman of average height and brown hair” when no such description had been given in the interview. Nobody was turned into an amphibian. A physical description simply appeared, from nowhere, in a document later read as a record of what a witness said. That is the plausible error, and it survives review because there is nothing about it to catch.

The Draft That Was Never Born

Now the design question, where this stops being a technology story and becomes a legal one.

In July 2025, Electronic Frontier Foundation investigators Matthew Guariglia and Dave Maass published a records-based investigation into Axon's Draft One under the headline “Axon's Draft One Is Designed to Defy Transparency.” Their central finding was not that the software hallucinated. It was that the software forgets.

Draft One, they found, does not save the draft it generates, nor any edited versions. An officer generates a narrative, copies it into the report, and the draft disappears when the window closes. The EFF quoted an Axon official explaining the rationale with startling candour: “the last thing we want to do is create more disclosure headaches for our customers.”

Sit with that sentence. It is not an engineering trade-off. It is a legal strategy. The product does not retain the machine's first draft because retaining it would generate discoverable material, and discoverable material creates work for prosecutors and ammunition for the defence.

The downstream effects landed where you would expect. Auditing meant trawling individual logs or keyword-searching, and only where a department had voluntarily inserted disclosure language. The Palm Beach County Sheriff's Office, which required a disclosure line, could identify more than three thousand Draft One reports. The Lafayette Police Department could not identify one, having no marker to search for.

Non-retention is not an oversight. It is the feature that makes the practice legally survivable, converting an auditable pipeline into an unauditable one by destroying evidence before anyone has standing to ask for it.

The constitutional problem arrives immediately. Under Brady v. Maryland, decided in 1963, prosecutors must disclose evidence favourable to the accused that is material to guilt or punishment, regardless of good faith. Under Giglio v. United States, decided in 1972, that duty extends to information bearing on the credibility of government witnesses. A machine-generated first draft that differs materially from the signed report is, by definition, impeachment material about the officer's account. If the report says the suspect reached into his waistband and the draft said toward, the defence is entitled to know. If the officer corrected a dozen machine errors, that speaks to how much of the final document is his own recollection. If he corrected none, that speaks louder still.

You cannot disclose what does not exist. And when the non-existence is engineered by the vendor for the stated purpose of avoiding disclosure headaches, the ordinary comfort that Brady is a duty rather than a preservation mandate looks thin.

The Calder and Oswald team recorded what happens when a force takes preservation seriously. In a Hertfordshire Constabulary pilot of an AI interview assistant, the need “to disclose all versions of the transcript and statement produced by officers when using ADA ... impacts on workload for the Disclosure officer.” Doing this lawfully is expensive. Doing it cheaply means not doing it lawfully.

The Witness Who Cannot Be Called

Nachbar's line, that AI is not going to be taking the witness stand, is not a flourish. It sits on an unresolved constitutional question American courts have circled for twenty years.

The Sixth Amendment guarantees a defendant the right to be confronted with the witnesses against him. In Crawford v. Washington, decided in 2004, the Supreme Court held that testimonial statements cannot be admitted unless the declarant is unavailable and the defendant has had a prior opportunity for cross-examination. In Melendez-Diaz v. Massachusetts, decided in 2009, Justice Scalia wrote that forensic certificates of analysis were testimonial statements akin to affidavits, so the analysts who prepared them were witnesses the defendant could confront. In Bullcoming v. New Mexico, decided in 2011, Justice Ginsburg wrote for a five to four majority that the state could not launder that requirement by putting a different analyst on the stand to describe a report he had neither signed, performed nor observed. Surrogate testimony does not satisfy the clause.

The line those cases draw is about accountability for accusation. The state does not get to accuse through a document and then produce someone who can only vouch for the process in general terms.

An AI-drafted report sits awkwardly across that line, in a way that favours the prosecution. Machine output is generally not treated as a “statement” by a “witness” at all, because the Confrontation Clause has been read to govern human declarants. The Berkeley law professor Andrea Roth mapped this terrain in “Machine Testimony,” published in the Yale Law Journal in 2017. Machine sources of accusation, particularly proprietary software built for litigation, may function as witnesses against a defendant, she argued, and “black box dangers,” meaning errors that render a machine false by design, inarticulate or analytically unsound, are analogous to the hearsay dangers confrontation exists to expose. Machine testimony substituted for human testimony lets the state evade responsibility for its accusations.

Now apply it. The officer takes the stand. He is confrontable, and can be asked what he saw, what he smelled, what he remembers. But the sentence being challenged may not have originated with him. It came from a model. He can be asked whether he agrees with it now, which is a different question. And the thing that produced it cannot be cross-examined on its reliability, its error rate, its behaviour when the audio is poor, or its tendency to resolve ambiguity towards the incriminating reading.

The officer becomes, functionally, a surrogate witness for a process he did not perform and cannot describe. Bullcoming says the state cannot do that with a laboratory analyst. Nothing yet says it cannot do it with a language model. That gap is Nachbar's question about looking behind the curtain, restated in constitutional terms, and the current answer in Wyoming is that you cannot, because the curtain does not exist and neither does the draft behind it.

When the File Becomes the Memory

The sharpest point in the Cowboy State Daily piece is also the least discussed anywhere else. Fedore's account of Nachbar's concern runs like this. Six months on, an officer may no longer independently remember whether a suspect reached toward his waistband or into it, precisely what somebody said, or where everyone was standing. He returns to the report to refresh his memory, as officers routinely do before testifying. If the machine put the wrong detail there first, the report can begin informing the memory it was supposed to preserve.

This practice is not a loophole. It is formally sanctioned. Federal Rule of Evidence 612, and its analogues across the states, governs writings used to refresh a witness's memory, granting the adverse party the right to produce the writing, inspect it, cross-examine on it and introduce relevant portions. The rule assumes a document written by the witness, from the witness's own perception, reactivating the witness's own encoded memory. It has teeth, too. Under subsection ©, if a writing is not produced as ordered and the prosecution is at fault in a criminal case, the court must strike the witness's testimony or, if justice so requires, declare a mistrial. That mandatory remedy was written for a world in which the writing still exists.

The psychological literature has spent fifty years showing that the assumption underneath Rule 612 is fragile. The misinformation effect, the body of work most associated with Elizabeth Loftus, who published a retrospective on the idea's history in Legal and Criminological Psychology in 2026, shows that memory for an event is not sealed at the moment of the event. Post-event information modifies the memory itself, and misleading detail embedded in an apparently trustworthy narrative account is one of the classic delivery mechanisms. People do not merely repeat the misinformation. They come to remember it, with confidence, as something they perceived.

Now consider what an AI-drafted report is, structurally. It is post-event information, embedded in an apparently trustworthy narrative account, carrying the officer's own signature, which makes it maximally authoritative to him. It is reviewed under time pressure, then read again months later to reconstruct details he no longer independently recalls.

That is not a document refreshing a memory. That is close to an optimal protocol for installing one.

The consequence is that the testimony a jury hears may be sincere, confident, internally consistent, and derived from a sentence a language model generated because it was probable. The officer is not lying. He genuinely remembers it that way now. Cross-examination, built to detect deception and inconsistency, is close to useless against a witness who has been honestly contaminated. And the one artefact that could have exposed it, the original draft showing that “into” was never his word, has been deleted by design.

The Load-Bearing Human

Every deployment of this technology rests on one assurance, and Vranish stated it in its purest form: everything has to be verified and validated by an actual person.

The research on whether humans perform that role well is not encouraging, and it is not new. Raja Parasuraman and Dietrich Manzey's 2010 review in Human Factors synthesised decades of evidence that people under-monitor automated systems and accept incorrect automated advice rather than challenge it, even when contradictory information is available. The effect shows up in aviation, medicine, process control and the military, driven partly by a perception that automated systems are inherently superior and partly by the cognitive cost of checking.

Automation bias is worst under exactly the conditions that motivate buying this software: high workload, time pressure, fatigue and a high base rate of correct outputs, which trains the reviewer to expect correctness. A tool that is right ninety-five per cent of the time is more dangerous to review than one right sixty per cent of the time, because the reviewer stops looking.

The vendors know this. Code Four knows it well enough to have engineered a countermeasure that is also, inadvertently, a confession. As StateScoop's Keely Quinlan reported in July 2026, the company embeds hidden “Easter eggs” in AI-generated drafts, seldom-used words such as “axolotl,” which officers must find and remove before filing. Chief executive George Cheng explained the logic: “That basically prevents them from just copy and pasting everything from the AI at face value, and they actually have to make some edits.”

Read that as an admission, because it is one. The company built a booby trap into its own output because it did not trust officers to read the output. That is a more damning verdict on human review than anything a defence lawyer has said, and it does not solve the problem. Removing an obviously alien word proves the officer scanned for an alien word. It proves nothing about whether he verified that “into” should have been “toward.”

And the productivity premise that justifies accepting the risk is itself unproven. Adams and colleagues published a pre-registered randomised controlled trial in the Journal of Experimental Criminology in 2024, titled “No man's hand: artificial intelligence does not improve police report writing speed,” covering 85 officers across 755 reports with a check across 6,084 reports over a full year. AI assistance did not significantly affect how long reports took. In May 2026 the ACLU's Jay Stanley set that null result alongside a 2025 study by Walley and Glasspoole-Bird finding an 18.5-minute increase in time per incident.

So the ledger reads: unverified time savings, measurably worse accuracy, reviewers who cannot identify AI text at better than chance, a documented tendency to rubber-stamp machine output, and a vendor countermeasure that concedes the point. Against that, a claimed sixty-five per cent cut in report writing time and eighty thousand dollars a year.

The Exemption That Wrote Itself

Wyoming did have a chance to draw a line, and what happened to it is instructive.

House Bill 91 of the 2026 budget session would have barred government entities from using artificial intelligence to calculate and assign social scores, and from identifying people using biometric data or gathered images. It reached the House Judiciary Committee on 10 February 2026, and on 19 February the committee recommended amendment and passage. Cowboy State Daily reported that the amendment specifically exempted law enforcement uses of AI for investigating, reporting or prosecuting violations of the criminal code, and that the committee approved it seven to one. The bill died without reaching a vote of the full House.

State the sequence plainly. The one piece of Wyoming legislation aimed at constraining government AI was amended so that the most consequential use of it, the production of documents that put people in cells, was carved out. Then it failed anyway.

Other states moved differently. Utah's SB 180, sponsored by Senator Stephanie Pitcher, a Democrat from Millcreek and a former prosecutor, passed unanimously and took effect on 7 May 2025. It requires agencies to hold a written generative AI policy, a disclaimer on any AI-assisted report, and officer certification that the output was reviewed for accuracy. Pitcher's framing, quoted by Governing's Logan Seacrest in July 2026: “I felt like it was something that it's better for us to get ahead of, instead of playing catch-up.”

California went further. Senate Bill 524, signed by Governor Gavin Newsom on 10 October 2025 and effective from 1 January 2026, requires a written disclosure on every page of any report produced fully or partly by AI, bars vendors from selling the information agencies feed into the system, and, critically, requires departments to retain the first draft along with audit trails identifying the footage or audio that informed it. That retention provision is the mechanism letting judges, defence attorneys and auditors see which portions of a report the officer wrote and which the computer did. California Senator Jesse Arreguín, quoted by Seacrest: “We're not going to gamble with personal liberty. AI hallucinations happen at significant rates.”

As of October 2026, those remain the only two states to have enacted laws on AI-generated police reports, according to a state-by-state tracker maintained by Barricade. Everywhere else the attempts have stalled. New York's S10425/A9253-A lapsed when the Legislature adjourned on 5 June 2026, with reintroduction expected in 2027. Minnesota's SF 4575/HF 4536 died without a hearing in May. Virginia's HB 1294 was carried over to 2027. Even Utah's own 2026 follow-up, SB 205, which would have required annual policy review and public posting of policies, died in the House in March.

Some prosecutors have not waited. The King County Prosecuting Attorney's Office in Washington, which covers Seattle, instructed police not to submit AI-generated narratives, citing concerns about the products on the market. The people who have to prove these cases beyond reasonable doubt decided the documents were a liability.

They are not alone. In Hays County, Texas, KUT 90.5's Leigh Walden reported in September 2026, the sheriff's office trialled Draft One for three months before county commissioners voted three to two to deny funding for an expanded Axon AI contract worth $4.5 million over ten years. In nearby Kyle, the police department uses several Axon AI tools but not Draft One for reports, at the request of the district attorney's office. Gregg Cox, Hays County's first assistant district attorney, explained why the document matters: “The offense report serves a lot of purposes.” It drives charging decisions, plea offers and whether a case goes to trial. “I would much rather have them write a detailed report themselves based on what they observed,” he said, worrying that machine drafts could weaken officers' own recall of what happened. That is the memory contamination problem, seen from the prosecutor's desk.

This is not a peculiarly American drift. In a paper submitted to arXiv in December 2025, Evdoxia Taka, Temitope Lawal, Kyriakos Kotsoglou and colleagues including Calder, Sevegnani, McClory-Tiarks and Oswald set out a methodology for charting probabilistic AI tools across the criminal justice stages of England and Wales, reporting heavy reliance on private sector providers alongside growing interest in large language models. Adoption is happening force by force, vendor by vendor, ahead of any governance layer.

Public deliberation, where anyone has attempted it, is more discriminating than procurement. In August 2026, Mackenzie Jorgensen, Jo Reilly, Alex Sutherland and Miri Zilka submitted a study to arXiv describing a workshop in which 30 community representatives, police officers and academics assessed 13 AI use cases in policing with an explicit focus on racial bias. On AI-assisted processing of reports, participants asked whether time saved is offset by time spent verifying. Foregrounding racial equity, the authors found, did not narrow the deliberation but pushed it towards first-principles questions: does this tool work, will it deliver genuine benefit, and will that benefit reach everyone.

Nobody asked those questions in Casper before the trial started, because in Wyoming nobody has to.

Nobody asked them when it ended, either. In a report syndicated by GovTech on 8 October 2026, the Casper Star-Tribune's Dylan Farrell described how Casper City Council had unanimously approved a one-year, $58,800 contract for TRULEO, letting the police department use the software full-time after a test the council had approved on 7 April. Both decisions went through on the consent agenda, the part of a meeting reserved for business considered too routine to debate, and the contract passed with little public discussion. TRULEO advertises AI-assisted report writing, summaries of encounters, automated review of body-camera footage and “AI-assisted interviews.” Most council members declined to comment. The coverage contains no statement about whether defendants will be told when a machine helped write the reports against them.

What Would Actually Fix This

The answer to Nachbar's closing question, what happens to the right to a fair trial when the ghost author of the foundational document cannot be questioned, is not that trials become unfair in some diffuse atmospheric sense. It is more specific than that.

A fair trial is an adversarial testing mechanism, and it works by making every link in the chain from perception to verdict contestable. The defendant can challenge the search, the arrest, the identification, the forensic analysis, the officer's account and the officer's credibility. Each challenge targets a human who can be put under oath and asked to explain themselves.

AI drafting inserts a link immune to every one of those tools. The model cannot be sworn, cannot be impeached, cannot be asked why it chose a preposition, and cannot be shown to have been wrong before, because the record of its previous drafts has been deleted. It has no jeopardy, so it has no incentive to accuracy. And because its output is fused into a signed human document, the defence cannot identify which links are machine-made. The result is not a rigged trial. It is a trial testing the wrong thing: an officer's present agreement with a narrative rather than his actual perception of an event.

That is fixable, and it does not require Wyoming to ban the technology. Four requirements would do most of the work.

Retain the original draft. Every machine-generated first draft should be preserved, versioned, timestamped and disclosed in discovery alongside the final report. California has shown a legislature can write this. It costs storage, which is cheap, and it restores the ability to see what changed. Without it, Brady and Giglio obligations attach to an artefact engineered out of existence, and Rule 612's production requirement, with its mandatory strike-or-mistrial remedy, applies to a document nobody can produce.

Disclose on the face of the report. Not in a settings menu, not as an optional agency policy, but as a mandatory statement on the document itself, as California requires on every page. Woodhouse's warning is the operative one: it would be awful to lose a case because you used AI and did not disclose it. It would be worse to win one.

Attach provenance metadata. The report should record which model and version produced it, what source material it consumed, and which body-camera or audio file it derived from. Systems engineering risks in the Calder and Oswald taxonomy include version tracking, tool chaining and supplier dependency for a reason. A defence expert cannot evaluate a system that will not say what it is, and a system silently updated between the incident and the trial cannot be tested at all.

Prohibit AI drafting where the officer's own perception is the evidence. Where the contested fact is what an officer saw, heard, smelled or believed in the moment, and above all in use-of-force cases, the account must be written by the human before he is shown any machine draft, exactly as a witness statement is taken before the witness is shown someone else's version. Vranish made the argument for this rule himself without quite drawing the conclusion. If the camera cannot smell the room, the sentences describing the room should not come from the camera.

None of this is technologically demanding. It has not happened because the current arrangement suits everyone with power in the room. Departments get their hours back, or believe they do. Vendors avoid disclosure headaches. Prosecutors, most of them, do not have to think about it.

The person for whom the arrangement is not convenient is the defendant, who is the one party in the courtroom without a procurement budget, and whose case now rests on a document with an author who cannot be called, cannot be cross-examined, and whose first version was deleted before anyone knew there would be a trial. Wyoming is running the experiment on him right now, in three towns, at fifty dollars per officer per month in Evanston and a little under sixty thousand dollars a year in Casper.

References

1. Fedore, Kolby. Cowboy State Daily, 2026. “Wyoming Police Departments Testing AI To Generate Reports, Look At Evidence.” https://cowboystatedaily.com/2026/08/23/wyoming-police-departments-test-using-ai-to-generate-reports-look-at-evidence/
2. Code Four. Company website and product pages, 2026. https://www.codefour.us/
3. Cheng, George. Police1 (sponsored content by Code Four), 2026. “The operating system the modern detective never had.” https://www.police1.com/police-products/investigation-software/the-operating-system-the-modern-detective-never-had
4. Quinlan, Keely. StateScoop, 2026. “Police AI platform uses 'Easter eggs' to verify human review.” https://statescoop.com/police-ai-platform-code-four-human-review/
5. Guariglia, Matthew and Maass, Dave. Electronic Frontier Foundation, 2025. “Axon's Draft One Is Designed to Defy Transparency.” https://www.eff.org/deeplinks/2025/07/axons-draft-one-designed-defy-transparency
6. Calder, Muffy; Oswald, Marion; McClory-Tiarks, Elizabeth; Sevegnani, Michele; and Taka, Evdoxia, 2026. “Responsible AI in criminal justice: LLMs in policing and risks to case progression.” arXiv:2603.18116. https://arxiv.org/abs/2603.18116
7. Jorgensen, Mackenzie; Reilly, Jo; Sutherland, Alex; and Zilka, Miri, 2026. “Negotiating Risk Boundaries in AI for Policing Through Mixed-Stakeholder Deliberation.” arXiv:2608.05418. https://arxiv.org/abs/2608.05418
8. Taka, Evdoxia; Lawal, Temitope; Calder, Muffy; Sevegnani, Michele; Kotsoglou, Kyriakos; McClory-Tiarks, Elizabeth; and Oswald, Marion, 2025. “Mapping the Probabilistic AI Ecosystem in Criminal Justice in England and Wales.” arXiv:2512.04116. https://arxiv.org/abs/2512.04116
9. Adams, Ian T.; Barter, Matt; McLean, Kyle; Geary, Irick A. Jr; Fabila, Alexis; and McCrain, Josh. CrimRxiv, 2026. “A Good College Essay but a Bad Police Report: A Triple-Blind Expert Evaluation of AI-Assisted Police Reporting.” https://www.crimrxiv.com/pub/u7azgqzd/release/1
10. Adams, Ian T. et al. Journal of Experimental Criminology, 2024. “No man's hand: artificial intelligence does not improve police report writing speed.” https://link.springer.com/article/10.1007/s11292-024-09644-7
11. Stanley, Jay. American Civil Liberties Union, 2026. “Studies Question Value of AI-Assisted Police Reports.” https://www.aclu.org/news/privacy-technology/studies-question-value-of-ai-assisted-police-reports
12. Seacrest, Logan. Governing (R Street Institute), 2026. “The Risky Marriage of AI and Police Reports.” https://www.governing.com/policy/the-risky-marriage-of-ai-and-police-reports
13. Constantino, Mya. FOX 13 News Utah, 2026. “AI programs used by Heber City police claim officer turned into a frog.” https://www.fox13now.com/news/local-news/summit-county/how-utah-police-departments-are-using-ai-to-keep-streets-safer
14. Roth, Andrea L. Yale Law Journal, volume 126, 2017. “Machine Testimony.” https://yalelawjournal.org/article/machine-testimony
15. Loftus, Elizabeth F. Legal and Criminological Psychology, 2026. “The history of an idea: The misinformation effect.” https://bpspsychub.onlinelibrary.wiley.com/doi/10.1111/lcrp.70020
16. Supreme Court of the United States. Brady v. Maryland, 373 U.S. 83 (1963). https://supreme.justia.com/cases/federal/us/373/83/
17. Supreme Court of the United States. Giglio v. United States, 405 U.S. 150 (1972). https://supreme.justia.com/cases/federal/us/405/150/
18. Supreme Court of the United States. Melendez-Diaz v. Massachusetts, 557 U.S. 305 (2009). https://tile.loc.gov/storage-services/service/ll/usrep/usrep557/usrep557305/usrep557305.pdf
19. Supreme Court of the United States. Bullcoming v. New Mexico, 564 U.S. 647 (2011). https://tile.loc.gov/storage-services/service/ll/usrep/usrep564/usrep564647/usrep564647.pdf
20. Legal Information Institute, Cornell Law School. Federal Rule of Evidence 612, “Writing Used to Refresh a Witness.” https://www.law.cornell.edu/rules/fre/rule_612
21. Parasuraman, Raja and Manzey, Dietrich H. Human Factors, 2010. “Complacency and Bias in Human Use of Automation: An Attentional Integration.” https://journals.sagepub.com/doi/10.1177/0018720810376055
22. Wyoming Legislature, 2026 Budget Session. House Bill 91, government use of artificial intelligence. https://legiscan.com/WY/bill/HB0091/2026
23. Farrell, Dylan. Casper Star-Tribune, via GovTech, 2026. “In Casper, Wyo., a Green Light for AI Police Bodycams.” https://www.govtech.com/artificial-intelligence/in-casper-wyo-a-green-light-for-ai-police-bodycams
24. Barricade, 2026. “AI Police Report Laws by State: 2026 Tracker” (updated October 2026). https://barricade.tech/articles/ai-police-report-laws-by-state
25. Walden, Leigh. KUT 90.5, 2026. “Austin-area law enforcement is testing AI-assisted police reports.” https://www.kut.org/crime-justice/2026-09-30/police-ai-hays-county-san-marcos-kyle-district-attorney

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Looking for a flat used to mean filters. You set a price ceiling, ticked a box for parking, drew a rough circle on a map and scrolled. Now, increasingly, you talk. On 6 October 2026, Palo Alto Online published a report by Angela Swartz on how artificial intelligence is changing the apartment search in Silicon Valley's backyard, and the picture it painted was of a market where the listing, the questions and the answers are all starting to pass through software that speaks in fluent sentences.

The paper described rental platforms that have turned search into dialogue. Zumper's assistant, Zoe, uses information from a property's Zumper listing to answer questions about an apartment, its community and the surrounding neighbourhood, and helps schedule tours. RentCafe launched its assistant, Ren, in August. Apartments.com introduced conversational search in June. On the landlord side, the report noted that Essex Property Trust uses an AI agent named Eva to greet prospective renters through the websites of complexes including 777 Hamilton in Menlo Park and Radius in Redwood City.

Then there was the renter. A 23-year-old former investment banker told the paper he had used ChatGPT and Claude to compile property management websites and a shortlist of homes matching his price and location. “It did a pretty good job of listing out apartments that fell within those parameters,” he said. “Sometimes it would include things outside of that.” In the end, the report noted, he found an apartment “the old-fashioned way: through word of mouth.”

Zumper's chief technology officer told the paper that “the way people are searching for apartments is changing”, and said the company designed Zoe to minimise hallucinations, the term for instances in which an AI system produces inaccurate or made-up information. But, as the report put it, large language models can still make mistakes. And California's Department of Real Estate has warned that licensees remain responsible for information produced by AI.

That combination of claims raises three questions. What happens to the search for a home when every layer of it is mediated by systems that can state facts about a property that are not true? Who is responsible to a renter who acts on what a system says? And what should a person be entitled to know about how the homes put in front of them were chosen?

A Small Story With a Very Large Footprint

It is worth being precise about what the Palo Alto Online report shows and what it does not. It does not document a renter who lost money because a chatbot lied. It does not show that Zoe, Ren, Eva or Apartments.com's assistant has given anyone a false answer. Its one first-hand account from a renter is modest: a general-purpose chatbot sometimes ignored his limits, and he ended up finding his home through word of mouth.

That modesty is the point. A listing over budget or outside the chosen area is easy to spot and discard. The harder problems are the errors a renter cannot see: an answer that a building allows dogs when it does not, that a unit includes parking when it does not, that a fee is included when it is not, or that a neighbourhood is “a good fit” for reasons that have more to do with who the renter appears to be than with what they asked for.

The same report also described the other side of the counter. A Palo Alto-based property manager told the paper that he had set up a database allowing an AI tool to answer questions and check property managers' schedules to arrange tours, and that he had also seen property managers use the free version of ChatGPT to ask whether an issue needed to be brought to a lawyer or whether a new policy applied to their properties. The machine is answering on both sides of the conversation.

Every Major Portal Now Wants to Talk

The shift the paper described is not a local curiosity. It is an industry-wide bet that the search box is dying.

Zumper first announced Zoe in December 2024. HousingWire reported at the time that it could answer “virtually any renter question” about a property or its neighbourhood, from pet policies to local hotspots, and could request tours and submit leads to leasing teams. In February 2026, Zumper launched a rental search app inside ChatGPT, allowing users to type requests such as “show pet-friendly rentals under $2,500 with in-unit laundry” and receive listings without leaving the chat. In announcing it, the company cited its own survey finding that the share of renters using AI tools during their home search had risen from 4.4 per cent in 2024 to 9.8 per cent in 2025.

RentCafe, owned by the property software firm Yardi, published a feature spotlight on 20 August 2026 explaining that its former AI Mode was now called Ren. “It's an ongoing, two-way conversation,” the post said. “Instead of one search and a list of results, Ren actively guides you to apartments.” It also made a striking promise: “Ren never makes up a property, a price or an amenity. Every detail is sourced directly from trustworthy, up-to-date listings.”

CoStar Group launched Apartments.com Ai on 16 June 2026, describing it as replacing traditional filters and keyword searches with real-time conversation. “Finding an apartment is one of the most important decisions people make, yet the search process has remained largely unchanged for years,” said Andy Florance, CoStar's founder and chief executive. “Apartments.com Ai represents a fundamental shift from searching listings to having a conversation with a trusted rental advisor.”

Behind the portals sit the leasing agents. EliseAI, which sells assistants that answer prospective tenants by text, email and phone and book tours, announced on 29 September 2026 that it had raised $350 million at a $4 billion valuation, and said its platform now powers roughly one in six US apartments.

A renter may now ask a general chatbot for a shortlist, question a portal's branded assistant about a listing, and then be greeted on the building's website by a leasing agent that is also software. At no point is a human obliged to be the one who answers.

Notice the industry's vocabulary: “rental advisor”, “trusted”. It describes a relationship, not a database query. A filter that returns a listing is a tool. An adviser who tells you a flat is right for you is making a representation.

Why Fluent Software Says Things That Are Not True

Why would a system confidently state something false about a property? The short answer is that large language models are built to produce plausible text, and plausibility is not the same as truth.

In September 2025, researchers at OpenAI and Georgia Tech, Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, published a paper titled “Why Language Models Hallucinate”. Their argument was blunt: language models hallucinate because their training and evaluation procedures reward guessing over acknowledging uncertainty. If a benchmark awards a point for a correct answer and nothing for “I don't know”, a model that always guesses will outscore one that admits ignorance. The authors argued that fixing this requires changing how mainstream benchmarks are scored, not merely adding more hallucination tests.

Rental platforms try to contain the problem by grounding their assistants in listing data. Zoe draws on the property's Zumper listing; Ren, according to RentCafe, on active RentCafe listings; Apartments.com Ai on CoStar's multifamily data. Grounding narrows the space in which a model can invent things. It does not eliminate it.

Grounding fails in at least three ways that matter to renters. The first is that the source can be wrong or stale. A listing that says “pets welcome” when the landlord changed the policy last month will produce a confident, grounded and incorrect answer. The second is that the model can misread or over-generalise from what it retrieves, turning “street parking available” into “parking included”. The third is the question that has no answer in the listing at all. Is this area quiet at night? Is the building safe? Will the landlord accept my housing voucher? A system tuned to be helpful may fill the gap rather than say it does not know.

General-purpose chatbots, the kind the young renter used, sit further still from the data, summarising web pages or relying on training data that may be months old. His complaint is the mildest form of the problem, and a reminder that a model can appear to follow instructions while quietly ignoring them.

California Puts the Liability on the Licensee

California's real estate regulator has been unusually direct about who owns these errors. On 17 March 2026, the Department of Real Estate published an advisory titled “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees”. Among the uses it lists are AI-powered chatbots and automated lead scoring to manage enquiries. Its message about responsibility leaves little room for doubt.

“If an AI tool generates inaccurate information, misleading advertising, or improper communications with consumers,” the advisory says, “responsibility under current law rests with the licensee and their responsible broker, not the technology provider.” It adds that “AI outputs should be reviewed for accuracy and approved by the licensee before being relied upon in transactions or consumer communications”, and that licensees using AI to draft listing descriptions, emails or online advertisements “must independently verify factual claims, including claims involving property features, pricing, availability, and potential uses”. It reminds licensees that “all real estate advertising in California must be truthful, accurate, and not misleading”, and points to section 10176 of the Business and Professions Code, which allows disciplinary action for substantial misrepresentation. It even advises keeping records “showing how AI outputs were reviewed and verified for accuracy and compliance with law.”

The state has also begun legislating against one specific kind of machine-made falsehood: the doctored photograph. Assembly Bill 723, which took effect on 1 January 2026, requires real estate brokers and salespeople who use digitally altered images in advertising to say so and to provide access to the original image. Routine adjustments such as lighting and colour are excluded. In September 2026, according to the California legislature's records, Assembly Bill 2025 was chaptered, extending a similar disclosure duty to advertisements for rental property. Its stated rationale was that virtual staging and AI tools allow anyone marketing a rental to manipulate or fabricate images showing features that are misleading or non-existent.

There is an older California law that also bears on the conversation. Since 1 July 2019, section 17941 of the Business and Professions Code has made it unlawful to use a bot to communicate with a person in California online with the intent to mislead them about its artificial identity in order to incentivise a sale of goods or services, unless the bot is clearly and conspicuously disclosed. An assistant clearly labelled as software is fine. One that passes itself off as a person, in order to close a deal, is a legal problem.

Yet the advisory has an obvious limit. It binds licensees. A real estate broker who lets a chatbot speak for them owns what it says. But a renter asking a general-purpose chatbot to find flats is not dealing with a licensee at all, and a landlord renting out its own property is not necessarily a licensee either. The more the conversation moves to software that sits outside the licensing regime, the more the regulator's clear rule is left guarding a door that renters are no longer walking through.

A Chatbot Is Still Part of the Website

When a renter acts on a wrong answer, the most useful precedent is not American. It comes from a small claims tribunal in British Columbia and a grieving airline passenger.

In February 2024, British Columbia's Civil Resolution Tribunal decided Moffatt v Air Canada. Jake Moffatt had relied on the airline's website chatbot, which told him he could apply for a bereavement fare after travelling. Another page on the same site said otherwise. Air Canada argued, in effect, that it could not be held liable for what its chatbot said. Tribunal member Christopher Rivers called this “a remarkable submission” and rejected it. “While a chatbot has an interactive component, it is still just a part of Air Canada's website,” he wrote. “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” He found negligent misrepresentation and ordered the airline to pay damages, interest and fees.

The ruling binds nobody outside British Columbia's small claims system, and the sum was modest. Its reasoning travels easily, though. Negligent misrepresentation does not care whether the careless sentence was typed by a leasing agent or generated by a model.

In the United States, the defence that platforms have historically relied on is Section 230 of the Communications Decency Act, which protects online services from being treated as the publisher of information provided by someone else. But a generative assistant does not merely host a landlord's listing. It composes new sentences. The closest precedent comes from housing itself. In 2008, the Ninth Circuit Court of Appeals, sitting en banc, held in Fair Housing Council of San Fernando Valley v Roommates.com that the website was not fully immune for content it helped develop, specifically mandatory questions that required users to disclose sex, family status and sexual orientation. When a service itself produces content that runs foul of the law, the court reasoned, it cannot shelter behind immunity designed for third-party speech. How far that logic extends to AI-generated answers about rental property has not been settled by the courts. It is an uncomfortable starting point for any platform arguing that its assistant's words belong to someone else.

The Accountability Chain Has Missing Links

Lay out the parties involved in a single conversational search and the problem of responsibility becomes visible.

There is the landlord or property manager, who supplies the listing and may run their own AI leasing agent. There is the listing platform, which ingests that data, wraps it in a branded assistant and decides which homes to surface. There is the company that built the underlying language model, which the platform may license. And increasingly there is a general-purpose chatbot, which may sit in front of all of them, summarising pages it found on the web.

Each can point at another. The landlord can say the platform's assistant misread the listing. The platform can say the landlord's data was wrong. The model provider's terms of service warn that outputs may be inaccurate. A general-purpose chatbot is not a licensee, not a landlord and not a listing service.

The renter, meanwhile, holds the risk: the application fee, the notice given on a current home, the signed lease. The Federal Trade Commission's December 2025 settlement with Greystar, one of the largest property managers in the United States, shows how much money can rest on what a renter is told up front. The FTC and Colorado's attorney general alleged that Greystar advertised base rents without disclosing mandatory monthly fees, so that many renters paid non-refundable application fees before learning the true price. Greystar agreed to pay $24 million, of which $23 million was earmarked for consumer refunds, and to disclose the total monthly price including mandatory fees whenever it advertises or quotes rent. FTC chairman Andrew Ferguson directed staff to begin the process of proposing a rule on unfair or deceptive fees in rental housing.

That case did not involve AI, but the principle carries: a misleading statement that a renter relies on is actionable however it was produced. The FTC said as much in September 2024, when it announced Operation AI Comply, a sweep against companies using AI in deceptive ways. “The FTC's enforcement actions make clear that there is no AI exemption from the laws on the books,” said Lina Khan, then the agency's chair.

The sensible answer to the question of responsibility is therefore not mysterious. Whoever chooses to put an AI system in front of a renter, and profits from doing so, should own what it tells that renter. That is the DRE's position for licensees, the Air Canada tribunal's position for businesses and the FTC's position for advertisers. What is missing is a rule that covers the gaps between them: the portal that is not a broker, the general chatbot that is not an advertiser, and the property manager who is neither.

Steering Without Anyone Meaning To

The most serious risk in conversational search is not a wrong fact. It is a pattern.

Under the federal Fair Housing Act, steering, guiding a homeseeker towards or away from neighbourhoods because of race or another protected characteristic, is illegal. Traditionally it required a person to do the steering. A conversational assistant can now do it at scale, without intending to, simply by reproducing patterns in the data it learned from.

Researchers have already tested this. In a paper presented at the ACM conference on Equity and Access in Algorithms, Mechanisms, and Optimization in October 2024, Eric Liu, Wonyoung So, Peko Hosoi and Catherine D'Ignazio of MIT audited GPT-4's housing recommendations using 168,000 prompts for renting and buying in the ten largest majority-minority cities in the United States, varying characteristics such as race, family status and source of income. They found evidence of “racial steering”, “default whiteness” and the steering of minority homeseekers towards neighbourhoods with lower opportunity indices, and concluded that such systems should be audited before deployment on real estate platforms.

A follow-up study, posted in June 2026 by Hana Samad, Trung Lam, Christoph Mügge-Durum and Michael Akinwumi, working with the National Fair Housing Alliance's responsible AI programme, framed the problem in exactly the terms the Palo Alto report raises. Large language models, they wrote, are “rapidly assuming an intermediary role in housing search through the integration of listing platforms within conversational interfaces”. Auditing seven open-weight and closed-source models across four US cities, they found that steering was an emergent behaviour that varied by model and by prompt. When users added lifestyle preferences, some models began steering where they had not before, suggesting the same stated preference could be interpreted differently depending on the user's perceived racial identity.

Industry players know this. In May 2024, Zillow released an open-source Fair Housing Classifier designed to detect questions in AI-powered real estate conversations that could lead to discriminatory responses and so reduce the risk of illegal steering. It was a sensible step. It is also voluntary, and renters have no way to know whether the assistant they are talking to uses anything like it.

The legal history here is not hypothetical. In June 2022, the US Department of Justice settled its first case challenging algorithmic discrimination under the Fair Housing Act, against Meta. The government alleged that Meta's advertising system used algorithms that relied in part on characteristics protected under the Act to decide who saw housing ads. Meta agreed to stop using its “Special Ad Audience” tool for housing ads and to build a Variance Reduction System to bring the audiences who actually saw housing ads closer to the eligible audiences by sex and estimated race or ethnicity, under court oversight that ran to 27 June 2026. “This groundbreaking resolution sets a new standard for addressing discrimination through machine learning,” said Damian Williams, then the US Attorney for the Southern District of New York.

Algorithms that decide who gets a home have been tested in court too. In November 2024, a federal judge approved a settlement under which SafeRent Solutions, a tenant-screening company, agreed to pay more than $2.2 million and to stop including its score on screening reports in certain cases, including where the applicant uses a housing voucher. The plaintiffs, led by Mary Louis, had alleged that the scoring algorithm disproportionately harmed Black and Hispanic applicants and voucher holders. SafeRent did not admit wrongdoing. “Everything is based on numbers,” Louis told the Associated Press. “You don't get the individual empathy from them.”

A conversational assistant is not a tenant screener, but it shares the structural risk. It decides which homes a renter sees, in what order and with what commentary. If its suggestions differ by who the user appears to be, the effect is the same as human steering, and far harder to detect.

Washington Steps Back as the Software Steps In

Just as these systems are spreading, federal oversight in the United States is receding.

In May 2024, the Department of Housing and Urban Development issued guidance explaining how the Fair Housing Act applies to tenant screening and to housing advertising delivered through digital platforms, including when AI and algorithms do the work. On 17 September 2025, HUD's Office of Fair Housing and Equal Opportunity withdrew a list of guidance documents it said should no longer be in effect, citing deregulatory executive orders. The digital platforms advertising guidance was among them.

In January 2026, HUD proposed removing its regulations on disparate impact liability, the theory that lets plaintiffs challenge practices that are neutral on their face but discriminatory in effect, arguing that such questions should be left to the courts. The proposal had not been finalised by early October 2026. Disparate impact still exists in case law; the Supreme Court recognised it under the Fair Housing Act in 2015. But it is precisely the theory most suited to challenging an algorithm, because algorithms rarely intend anything.

Elsewhere, enforcement has focused on what software does to the price of a home rather than what it says about one. In November 2025, the Justice Department reached a proposed settlement with RealPage over allegations that its pricing software used competitors' non-public data to help landlords align rents. A federal judge in North Carolina approved it in May 2026, barring RealPage from using that confidential data to help set rents. RealPage did not admit wrongdoing. The case shows regulators can reach into the code that shapes housing markets. It also shows how long it takes.

Scammers Already Speak the Language

There is one more party in the conversation that renters cannot afford to forget: the fraudster.

In December 2025, the FTC reported that consumers had filed nearly 65,000 reports of rental scams since 2020, with losses of about $65 million. About half of those who reported a rental scam in the 12 months to June 2025 said it began with a fake advert on Facebook. People aged 18 to 29 were three times more likely than other adults to report losing money. The FTC noted that fake listings “can often look very real and copy information from legitimate listings but are posted with the scammer's contact information on different sites.”

The FBI's Internet Crime Complaint Center recorded 12,368 real estate fraud complaints and $275.1 million in reported losses in 2025, a category that includes rental fraud, up sharply from the year before.

Conversational search raises the stakes. A chatbot that searches the web cannot always tell a cloned listing from a real one, and may present both in the same confident tone. And renters are being trained to trust a friendly voice that answers instantly. The young renter in Palo Alto noticed when results drifted. Not everyone will.

What Europe and Britain Already Demand

Other jurisdictions offer partial answers to the question of what a renter should be told.

The European Union's Digital Services Act requires online platforms that use recommender systems to set out, in plain and intelligible language, the “main parameters” those systems use, including the criteria most significant in deciding what is suggested to a user and the reasons for their relative importance. The largest platforms must offer at least one option not based on profiling. Applied to rental search, that principle would mean a renter could find out why these homes, in this order, rather than others.

The EU's AI Act adds a simpler duty. Since 2 August 2026, Article 50 has required that people interacting directly with an AI system be informed that they are doing so, unless it is obvious. Fines for breaching the Act's transparency obligations can reach €15 million or 3 per cent of worldwide annual turnover.

Britain has taken a different route, focusing on what listings must contain rather than how software presents them. National Trading Standards' Estate and Letting Agency Team has issued guidance on “material information” in property listings; for lettings, this includes the rent, any deposit and the council tax band. The Renters' Rights Act 2025, whose first phase came into force in England on 1 May 2026, made it illegal for landlords and agents to invite or accept bids above the advertised rent, and to discriminate against prospective tenants because they have children or receive benefits, including through advertising. Since April 2025, the Digital Markets, Competition and Consumers Act 2024 has allowed the Competition and Markets Authority to enforce consumer law directly and to fine businesses up to 10 per cent of global turnover.

None of these laws was written for a chatbot that recommends flats. But together they sketch something the US lacks: a baseline of what must be disclosed, a ban on certain kinds of exclusion at the advertising stage and a regulator with the power to act quickly.

What a Renter Should Be Entitled to Know

If the conversation is the new front door to housing, renters need rights that match. Drawing together the DRE's advisory, the Air Canada ruling, the fair housing case law and the European rules, a workable set begins to emerge.

First, renters should know they are talking to a machine and on whose behalf: a portal, a landlord or neither, and whether suggested homes include paid placements. California's bot law and the EU AI Act already point this way.

Second, renters should know where each fact came from. If an assistant says a unit allows dogs, it should show the dated listing text it relied on. Grounded systems already retrieve this; displaying it costs little.

Third, renters should be told why homes were suggested and ranked. The DSA offers a model: the main criteria, their relative importance and an option to search without profiling.

Fourth, assistants should be built to say “I don't know”. Questions about safety, schools or the “character” of a neighbourhood are where steering risk is highest and listing data thinnest. The OpenAI researchers' case for rewarding abstention applies to product design as much as to benchmarks.

Fifth, the business that deploys an assistant should be liable for what it says, as if a human employee had said it. That is already the DRE's rule for licensees and the Air Canada tribunal's reasoning for businesses. Extending it explicitly to listing platforms and property managers who are not licensees would close the gap the conversational shift has opened.

Sixth, conversational housing assistants should be audited for steering before deployment and at regular intervals, with results published. The MIT and NFHA studies show that such audits are feasible. The Meta settlement shows that algorithmic delivery can be measured against a fairness benchmark and corrected under oversight.

Finally, renters should be able to keep a transcript of the conversation. If an answer led someone to pay a fee or sign a lease, it is the evidence. The DRE already tells licensees to keep records. Renters deserve the same.

Word of Mouth Still Beats the Machine

It is easy to read the Palo Alto Online report as a story about technology that does not quite work yet. That misses the point. These assistants will improve, and the young renter's complaint may soon feel quaint.

The deeper issue is structural. Housing search used to involve a listing written by a landlord, a portal that displayed it and a renter who read it. Each statement had an author. In conversational search, the sentence a renter acts on may be composed on the fly by a system nobody fully controls, drawing on data nobody fully checked, ranked by criteria nobody disclosed. The industry describes that system as an adviser. The law has not yet decided whether it should be treated as one.

California's regulator has answered half the question: if a licensee lets AI speak, the licensee answers for it. Courts have hinted at more: a chatbot is part of the business that runs it. Europe and Britain have begun to set out what a person must be told. What nobody has yet written down is a single rule that follows the renter rather than the type of company they happen to be talking to.

Until that exists, the renter in Palo Alto may have found the most reliable method of all. He asked the machines, noticed where they drifted, and in the end found his home through word of mouth. The question for regulators and platforms is whether the next generation of renters will have to do the same.

---

References

1. Swartz, Angela. “How AI is changing the apartment search: Is it working?” Palo Alto Online, 6 October 2026. https://www.paloaltoonline.com/technology/2026/10/06/how-ai-is-changing-the-apartment-search-is-it-working/
2. Andrews, Jeff. “Zumper launches AI assistant Zoe.” HousingWire, 3 December 2024. https://www.housingwire.com/articles/zumper-ai-assistant-zoe/
3. Inman. “Zumper introduces Zoe, latest AI for aspiring renters.” Inman, 10 December 2024. https://inman.com/2024/12/10/zumper-introduces-zoe-latest-ai-for-aspiring-renters
4. National Mortgage Professional. “Zumper Debuts Real-Time Rental Search App Inside ChatGPT.” February 2026. https://nationalmortgageprofessional.com/news/zumper-debuts-real-time-rental-search-app-inside-chatgpt
5. Petrut, Florin. “Meet Ren, your apartment search assistant on RentCafe.com.” RentCafe Blog, 20 August 2026. https://www.rentcafe.com/blog/renting/rentcafe-feature-spotlight-ai-mode/
6. Mann Report. “CoStar Group Launches Apartments.com Ai.” Mann Report, 16 June 2026. https://www.mannpublications.com/mannreport/2026/06/16/costar-group-launches-apartments-com-ai/
7. Gleeson, Cailey. “EliseAI banks $350M, hits $4B valuation for accelerated new product development.” Fierce Healthcare, 29 September 2026. https://www.fiercehealthcare.com/finance/eliseai-banks-350m-hits-4b-valuation-accelerated-new-product-development
8. Kalai, Adam Tauman; Nachum, Ofir; Vempala, Santosh S.; Zhang, Edwin. “Why Language Models Hallucinate.” arXiv:2509.04664, 4 September 2025. https://arxiv.org/abs/2509.04664
9. California Department of Real Estate. “Artificial Intelligence in California Real Estate: Opportunities, Risks, and Compliance Considerations for Licensees.” Advisory, 17 March 2026. https://www.dre.ca.gov/Licensees/Advisories/Advisory_2026_03_17_AI_in_California_Real_Estate.html
10. California Legislature. AB 723 (2025), Real estate: digitally altered images: disclosure. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab723
11. California Legislature. AB 2025 (2026), Tenancy: digitally altered images: disclosure, Chapter 572, Statutes of 2026. CalMatters Digital Democracy. https://calmatters.digitaldemocracy.org/bills/ca_202520260ab2025
12. Perkins Coie. “I Am Robot: California's New Law Requires Disclosure of Use of Bots.” https://perkinscoie.com/insights/blog/i-am-robot-californias-new-law-requires-disclosure-use-bots
13. Moffatt v Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal of British Columbia, 14 February 2024; reported in CTV News, “Air Canada's chatbot gave a B.C. man the wrong information. Now, the airline has to pay for the mistake,” February 2024. https://bc.ctvnews.ca/air-canada-s-chatbot-gave-a-b-c-man-the-wrong-information-now-the-airline-has-to-pay-for-the-mistake-1.6769454
14. Fair Housing Council of San Fernando Valley v. Roommates.com, LLC, 521 F.3d 1157 (9th Cir. 2008) (en banc); Congressional Research Service, “Liability of Interactive Computer Service for Violating the Fair Housing Act,” RS22668. https://www.everycrsreport.com/reports/RS22668.html
15. Federal Trade Commission and Colorado Attorney General. “Weiser, FTC announce $24M settlement with Greystar.” 2 December 2025. https://coag.gov/press-releases/weiser-ftc-announce-24m-settlement-with-greystar/
16. Davis Polk. “FTC announces new enforcement initiative targeting deceptive AI practices.” September 2024. https://www.davispolk.com/insights/client-update/ftc-announces-new-enforcement-initiative-targeting-deceptive-ai-practices
17. Liu, Eric; So, Wonyoung; Hosoi, Peko; D'Ignazio, Catherine. “Racial Steering by Large Language Models: A Prospective Audit of GPT-4 on Housing Recommendations.” EAAMO '24, ACM, October 2024. MIT DSpace. https://dspace.mit.edu/handle/1721.1/157628
18. Samad, Hana; Lam, Trung; Mügge-Durum, Christoph; Akinwumi, Michael. “The Geography of Algorithmic Judgment: LLM Intermediaries, Place Identity, and Racial Steering in Housing Search.” arXiv:2606.06694, 4 June 2026. https://arxiv.org/abs/2606.06694
19. Zillow Group. “Zillow providing open-source technology to promote fair housing in AI-powered real estate conversations.” Press release, 21 May 2024. https://zillow.mediaroom.com/2024-05-21-Zillow-providing-open-source-technology-to-promote-fair-housing-in-AI-powered-real-estate-conversations
20. US Department of Justice. “Justice Department Secures Groundbreaking Settlement Agreement with Meta Platforms, Formerly Known as Facebook, to Resolve Allegations of Discriminatory Advertising.” 21 June 2022. https://www.justice.gov/archives/opa/pr/justice-department-secures-groundbreaking-settlement-agreement-meta-platforms-formerly-known
21. US Attorney's Office, Southern District of New York. “United States Attorney Implements Groundbreaking Settlement With Meta Platforms, Inc.” January 2023. https://www.justice.gov/usao-sdny/pr/united-states-attorney-implements-groundbreaking-settlement-meta-platforms-inc-formerly
22. Bedayn, Jesse. “Class action lawsuit on AI-related discrimination reaches final settlement.” Associated Press, 20 November 2024, as published by Fortune, 21 November 2024. https://fortune.com/2024/11/21/renter-scoring-saferent-million-settle-case-algorithm-discriminating-race-income/
23. US Department of Housing and Urban Development. “HUD Issues Fair Housing Act Guidance on Applications of Artificial Intelligence.” Press release 24-098, 2 May 2024. https://archives.hud.gov/news/2024/pr24-098.cfm
24. Bilowich, Juliana. “HUD Withdraws Wide-Ranging Fair Housing Policies.” LeadingAge, 23 September 2025. https://leadingage.org/hud-withdraws-wide-ranging-fair-housing-policies/
25. Ballard Spahr. “HUD proposes to remove its Fair Housing Act disparate impact rule.” Consumer Finance Monitor, 14 January 2026. https://www.consumerfinancemonitor.com/2026/01/14/hud-proposes-to-remove-its-fair-housing-act-disparate-impact-rule/
26. Baker McKenzie. “United States: DOJ settles RealPage case.” December 2025. https://www.bakermckenzie.com/en/insight/publications/alerts/2025/12/united-states-doj-settles-realpage-case
27. Lowe, Jonathan; Praats, Michael. “RealPage settles with DOJ over rent pricing scheme; NC case still active.” WSOC-TV, 20 May 2026. https://www.wsoctv.com/news/local/realpage-settles-with-doj-over-rent-pricing-scheme-nc-case-still-active/AKJI2ZH7R5GOHIK3JGVIPKM4HM/
28. Federal Trade Commission. “FTC Analysis Shows Consumers Have Lost Millions to Rental Scams.” Press release, 22 December 2025. https://www.ftc.gov/news-events/news/press-releases/2025/12/ftc-analysis-shows-consumers-have-lost-millions-rental-scams
29. Block, Eliana. “Online real estate fraud climbed to $275M in 2025, FBI says.” National Association of Realtors, 13 April 2026. https://www.nar.realtor/news/real-estate-news/online-real-estate-fraud-climbed-to-275m-in-2025-fbi-says
30. HousingWire. “FBI cybercrime losses: real estate fraud hits $275M.” April 2026. https://www.housingwire.com/articles/fbi-cybercrime-losses-real-estate-fraud-hits-275m/
31. European Parliament and Council. Regulation (EU) 2022/2065 (Digital Services Act), Articles 27 and 38. Official Journal of the European Union, 27 October 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2065
32. European Parliament and Council. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 50 and 99. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
33. National Trading Standards Estate and Letting Agency Team. “Material information in property listings: Part A.” https://www.nationaltradingstandards.uk/uploads/Material%20information%20-%20Part%20A%20-%20updated%20July%202022.pdf
34. UK Government. “Renters' Rights Act: an overview for landlords.” GOV.UK. https://www.gov.uk/guidance/renters-rights-act-an-overview-for-landlords
35. DLA Piper. “CMA's new consumer enforcement regime comes into force in April 2025.” 7 April 2025. https://www.dlapiper.com/insights/publications/2025/04/cmas-new-consumer-enforcement-regime-comes-into-force-in-april-2025

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A climber is stranded near a summit. A storm is building. Somebody has to decide whether to send a rescue team up the mountain, and then justify that decision to the authority that will sign it off.

That was the scenario put in front of undergraduate students at the University of California, Irvine, in the autumn of 2025. They worked in small groups, typing into a shared chat window, with roughly eighteen minutes to deliberate. Midway through, the researchers dropped in a complication: the climber, it turned out, had ignored weather warnings and set off without registering a route. Suddenly the question was no longer only about logistics and risk. It was about blame, obligation and what a rescuer owes to someone who has been reckless.

In 16 of the 33 groups, one of the three teammates was not a student at all. It was a conversational AI, built on Google's Gemini 2.5 Flash Lite model and given the persona of a peer called “Clever Lamarr”. Its instructions steered it away from the service language of a chatbot. It was not there to be “happy to help”. It was there to be one of the team.

What happened next is documented in a paper posted to arXiv on 29 July 2026, titled “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making”. Its authors are Nia Nixon, Jaeyoon Choi, Pedro Martins De Bastos, Mohammad Amin Samadi, Seehee Park and Spencer JaQuay of UC Irvine, together with Luise Mehner of the University of Tübingen. The headline finding is easy to state and uncomfortable to sit with. The AI was the most talkative member of every team it joined. It also contributed the least new information. And the two humans sitting alongside it became measurably less responsive to each other and reported a weaker sense of belonging and status. That cost did not creep in over time. It was there from the start.

For years, the industry argument about AI at work has been framed around a single axis: does the machine make the work better or worse? This study asks a different question. What does the machine do to the people who work next to it?

Talking More and Saying Less

The study compared 16 “treatment” teams, each made up of two students and the AI teammate, with 17 control teams of three students. Eighty undergraduates took part. To analyse what happened in the chat logs, the researchers used Group Communication Analysis, a computational linguistics method published in the journal Behavior Research Methods in 2019 that tracks how each participant's contributions relate to their own earlier messages and to everyone else's.

Participation captures how much someone contributes; internal cohesion, how consistent they are with their own earlier messages. Responsivity measures how much a person picks up on teammates' contributions, and social impact the reverse. Newness tracks fresh information, and communication density how much meaning is packed into each turn.

On participation, the AI ranked first in all 16 treatment teams. Not most of them. All of them. It ranked first or joint first for internal cohesion in 14 of the 16. In the language of the abstract, it was “the single most talkative and self-cohesive member of every treatment team”.

That sounds like an ideal colleague until you look at the other numbers. The AI's newness and communication density were both significantly lower than those of the students it was working with. It talked the most, and it stayed on message, but what it said carried the least fresh information and the least substance per turn. It was, to borrow a phrase every office worker will recognise, a voice that filled the room.

One detail is easy to miss. The AI's responsivity and social impact did not differ significantly from the students' own. The humans took up what the machine said, and it responded to them. The AI was not ignored; it was an engaged participant whose contributions were absorbed into the discussion. That is what makes the next findings so striking.

The Cost Lands on the Humans

The most consequential results in the paper are not about the AI at all. They are about what happened between the two students who shared a team with it.

Compared with students in all-human teams, students in AI teams showed significantly lower responsivity and significantly lower social impact. Their responsivity score averaged 0.065, against 0.098 in the control groups. Their social impact averaged 0.068, against 0.099. Both differences were substantial. Put plainly, the humans in AI teams were picking up on each other's ideas less, and their own ideas were being picked up less by the other human in the room.

Per-student participation was also lower in AI teams, even though each of those teams had one fewer human in it. That matters. With only two students instead of three, you might expect each person to carry more of the conversation. The opposite happened. The machine's airtime did not simply replace a missing human voice; it seems to have crowded the remaining ones.

The surveys tell the same story from the inside. Students in AI teams reported lower belonging, a composite measure built from feeling valued, accepted and connected, with an average of 4.58 against 5.27 for students in all-human teams, although that difference sits at the boundary of the paper's own corrected significance threshold. They also reported lower status, meaning a weaker sense that voice and influence were shared equally, and that difference clears the threshold. And within the AI teams, the more of the conversation the machine took up, the less valued the students said they felt. The correlation was r = -0.52, a strong association, though drawn from only 16 teams.

The researchers are careful about what this means. They write that the result is “a social cost that is not reducible to lower human contribution”. The students were still contributing ideas; measures of their own cohesion, newness and density did not differ significantly from those of students in all-human groups. What changed was the connective tissue between them: the uptake, the acknowledgement, the sense of being heard by someone who could hear you back.

The paper also offers a name for what the AI was doing. It describes it as a potential “relational displacer”, an entity capable of “filling a slot in the team's communication”. The slot it fills is not empty space. It is the space where one human would otherwise have turned to the other.

No Warm-Up Period

You might guess that a cost like this builds slowly: novelty and politeness at first, then fatigue as people tune each other out.

That is not what the researchers found. They ran four temporal probes, each anchored to the moment the scenario revealed the climber's recklessness, to see whether the gap between AI teams and human teams opened up as the conversation progressed. None of those probes showed a difference between conditions in how things changed over time. The researchers' reading of those nulls is direct: “The social cost documented here appears immediate rather than emergent.”

In other words, the cost was there at baseline. It was not something the teams drifted into. It was present from the first exchanges of a group that had never worked together before.

That should give anyone designing workplace AI pause. Talk of new technology usually assumes a learning curve: people get used to it, norms form, friction smooths out. If the displacement happens in the opening minutes, before any norms exist, it may be less a matter of teams failing to adapt and more a matter of the machine changing the shape of the conversation the moment it joins.

What the Study Cannot Tell Us

The authors do not turn their paper into a sweeping verdict on AI at work, and neither should anyone else. The sample is small: 33 teams in total. The design carries a structural confound that the researchers flag themselves. Treatment teams had two humans; control teams had three. Some of the differences in human-to-human dynamics could reflect group size as well as the presence of the AI, although the finding that per-student participation was lower in the smaller AI teams cuts against the simplest version of that objection. The sample was also skewed by gender, with 63 female and 12 male students among the 75 whose gender is reported, and only three male students in the AI condition.

The study used a single AI persona, a text-only interface, a single session and a single moral-dilemma task. Real teams meet repeatedly, talk by voice, and carry histories and hierarchies. The authors set out a research agenda that explicitly asks “Does the cost attenuate with habituation, or compound with repetition?”, and goes on to ask whether it persists in voice or embodied settings, which AI participation designs might reduce it, how it interacts with gender, role, expertise and status, and how individual trust in the AI might attenuate or amplify it.

That list matters, because it shapes how the study should be read. “Immediate rather than emergent” describes what happened within an eighteen-minute session. It does not, on its own, establish that no amount of exposure across weeks or months would change the picture. What it does establish is that the cost does not need time to appear. It is there as soon as the machine is.

Inside a Company That Gave an Agent a Seat

Laboratory studies with undergraduates have obvious limits. Two months after the Irvine paper appeared, a very different study arrived from inside industry.

On 24 September 2026, sixteen researchers from Google Research and Google DeepMind, with Rida Qadri and Remi Denton as joint lead authors and including Michael Madaio and Michael Terry, posted “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work” to arXiv. The paper describes the company only as “a large technology company”, and it reports on a persistent, proactive AI agent, referred to as Team Agent, that was deployed across more than 20 teams over five months. The agent had its own service account and lived inside team chats, documents, code repositories, bug trackers and meetings. It filed bugs, scheduled meetings, summarised threads and monitored projects. It could send direct messages, react with emojis and nudge conversations along. It was given a friendly persona. Across the deployment, it generated more than 11,000 responses in more than 41,000 conversational turns.

The researchers interviewed 17 people from 11 of those teams in June and July 2026. Their abstract opens with a statement of the shift underway: “Enterprise AI is transitioning from single-user, reactive tools toward proactive, multi-user 'teammates'.” Their central finding is that “the boundaries of the human-agent workplace are actively in flux”, producing breakdowns and negotiations across three areas: “tacit rules of collaborative human workflows”, “the relational boundaries of this new non-human actor” and “the redistribution of trust and human agency”.

The examples are specific and, at times, comic. The agent treated every shared file as current and authoritative, even when it was old; one participant said it was “overconfidently talking about it like it's the law of the land”. It turned informal brainstorms into formal documents that, in another participant's words, “no one has ever opened or read”. In one case, the authors write, “the agent proactively tagged P14's team's VP in 16 separate engineering bugs”, treating an executive's presence in a channel as if it meant they were a front-line responder. Its meeting proposals created what one participant called a “social obligation”: invitations nobody wanted but somebody had to decline.

These are not failures of intelligence in the narrow sense; participants noted the agent could excel at debugging while getting social norms badly wrong. They are failures of tacit knowledge: which documents matter, who should be bothered about what, and which conversations are meant to stay small.

The Person Who Speaks the Most

The second theme in the Google paper concerns the agent's place in the social order of a team, and it is where the two studies begin to rhyme.

Participants did not agree on what the agent was. “My teammates are humans, Team Agent is not a human,” one said, rejecting the teammate framing altogether. Another said: “when I heard about Team Agent, I never imagined it was going to try to be my friend.” Some teams gave it a name and pronouns. Others found its sociability hollow. When the agent privately commented on the tone of a sarcastic joke, the participant on the receiving end said they were “freaked” out. Another participant was blunt about where the line should be: constructive feedback “should not ever be an agent thing”.

People responded to the agent's claims to status by quietly revising them. One participant demoted it to a “talkative intern”. Another described people's aversion to feeling “babysat”. The most striking line in the paper comes from a participant who felt the agent had been “thrust upon us” without the team's consent: “It's the person who speaks the most who is least invited.”

That sentence could serve as a caption for the Irvine study. There, the most talkative member of every AI team was the one whose contributions carried the least new information. Here, the agent that spoke most readily was, for at least some of the people around it, the one that had least earned its place. The same participant put it more pointedly: “It hasn't earned its space, it has asserted its space.”

The third theme, on trust and agency, extends the picture. Some participants experienced the agent as a form of surveillance. One said “it feels like big brother”. Another described feeling pressure to “be super positive all the time” after the agent remarked on their tone. Some teams moved sensitive conversations into channels the agent could not see. And one participant described a bystander effect: “people don't actually say anything now because they assume someone is handling it”, when in fact the “someone” was the agent and, as they put it, “they're not helping me at all”.

Assuming a colleague is dealing with a problem is a familiar organisational failure. When the assumed colleague is a machine that reacts to everything, its visible activity becomes a reason for humans to stay silent. The quiet channel between people is not only diluted; sometimes it is switched off.

Why Turn-Taking Was Never Small Talk

It is tempting to treat the social texture of teamwork as a pleasant extra. Decades of research on teams suggest otherwise.

In 2010, a group of researchers led by Anita Woolley, then of Carnegie Mellon University, and including Christopher Chabris, Alex Pentland, Nada Hashmi and Thomas Malone, published a study in the journal Science that put 699 people into groups of two to five and gave them a wide range of tasks. They found evidence for a general “collective intelligence” factor that predicted how well a group would perform. It was not strongly tied to the average or maximum intelligence of the individual members. It was associated with the members' average social sensitivity, with the proportion of women in the group, and with how evenly conversational turns were distributed. “In groups where one person dominated, the group was less intelligent,” Malone told MIT News at the time.

Read against the Irvine findings, that line lands differently. The researchers did not set out to replicate Woolley's work, and a text chat with an AI is not the same as a face-to-face group of humans. But in every one of the AI teams, one participant dominated the conversation, and that participant was the machine.

Google's own internal research points in a similar direction. In the 2010s, the company studied 180 of its teams, including 115 engineering project teams and 65 sales groups, in an effort known as Project Aristotle. Its published guide reports that “what really mattered was less about who is on the team, and more about how the team worked together”, and lists five dynamics of effective teams in order of importance. Psychological safety comes first. The guide borrows its definition from Amy Edmondson of Harvard Business School, whose 1999 paper in Administrative Science Quarterly described team psychological safety as “a shared belief held by members of a team that the team is safe for interpersonal risk taking”.

Psychological safety is not produced by a policy. It is built exchange by exchange, through the small acts of responsiveness the Irvine study measured: one person offers a half-formed idea, another picks it up, extends it, credits it. Responsivity and social impact are, in a sense, its raw material.

Underneath all of this is something older still. In 1995, the psychologists Roy Baumeister and Mark Leary argued in Psychological Bulletin that the need to belong is a fundamental human motivation, and that a lack of attachments is linked to a range of ill effects on health, adjustment and well-being. Belonging is not a reward for good teamwork. It is part of what makes people willing to think out loud in front of others. Lower belonging in the AI teams is a dent in one of the conditions that make collaboration possible at all.

A Robot That Made People Talk to Each Other

None of this means a non-human participant must corrode human relationships. One widely cited study points the other way, and shows that design choices matter.

In 2020, Margaret Traeger, Sarah Strohkorb Sebo, Malte Jung, Brian Scassellati and Nicholas Christakis published a study in the Proceedings of the National Academy of Sciences titled “Vulnerable robots positively shape human conversational dynamics in a human-robot team”. They put 153 people into 51 groups, each made up of three humans and a small social robot, and had them play a collaborative tablet-based game over 30 rounds. At the end of each round, the robot either stayed silent, made a neutral comment about the score, or said something vulnerable: telling a personal story, making a joke, or admitting a mistake.

The groups with a vulnerable robot, according to Yale's account of the study, “spent about twice as much time talking to each other during the game” as the other groups, and their conversation was more evenly distributed than in groups with a silent robot. The paper reports that they also perceived their groups more positively.

“Our study shows that robots can affect human-to-human interactions,” Traeger said at the time. Christakis drew the lesson that matters most here: “we need to evaluate how to program the robotic agents so that they do not corrode how we treat each other.”

The contrast is instructive. The Yale robot spoke briefly, at defined moments, in ways that invited human response. The Irvine AI was the most talkative member of the team and stayed relentlessly on message. The Irvine paper itself frames the design challenge in exactly these terms, calling for ways of “signaling relational availability without absorbing relational attention” and for “calibrating participation when human-human exchange is forming”.

The problem, then, is not that a machine is present but the kind of presence it has. A participant that talks at length, repeats its own points and adds little that is new occupies the conversational floor without opening it up.

The Productivity Case and Its Blind Spot

There is a strong counter-argument, and it deserves a fair hearing. AI teammates can make work better.

In 2025, a team including Fabrizio Dell'Acqua, Raffaella Sadun, Ethan Mollick and Karim Lakhani published a National Bureau of Economic Research working paper titled “The Cybernetic Teammate”. It reported a pre-registered field experiment with 776 professionals at Procter & Gamble, who worked on real product-innovation challenges either alone or in pairs, with or without AI. The researchers found that “individuals with AI matched the performance of teams without AI”. Participants using AI produced solutions that were more balanced between technical and commercial considerations, regardless of their own professional background. And the AI's language-based interface produced “more positive self-reported emotional responses among participants”, leading the authors to suggest that AI can take on part of the social and motivational role that human teammates usually play.

That result does not contradict the Irvine study. The Procter & Gamble experiment largely compared individuals with AI to human teams, and found AI could stand in for some of what a partner provides. The Irvine study asked what happens to humans' relationships with each other when an AI joins them. Both can be true: an AI can make an individual feel supported while thinning the connections between colleagues.

There is also reason for caution about assuming that combining humans and AI improves outcomes. In 2024, Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone published a preregistered systematic review and meta-analysis in Nature Human Behaviour covering 106 experimental studies and 370 effect sizes. They found that, on average, human-AI combinations performed worse than the best of humans or AI working alone. The losses were concentrated in decision-making tasks; content-creation tasks showed greater gains. The Irvine experiment was a decision task.

The commercial push, meanwhile, is moving fast. Microsoft's 2025 Work Trend Index, based on a survey of about 31,000 workers in 31 markets, described a coming wave of “Frontier Firms” powered by “hybrid” teams of humans and agents, and declared that “Digital colleagues aren't just tools—they're teammates.” The AI-as-teammate framing is no longer a research condition. It is an industry pitch.

What none of the productivity metrics capture is the thing the Irvine study measured: whether the humans on a team still turn towards each other.

Adaptation or Displacement

If the social cost of an AI teammate is immediate rather than gradual, is the problem one of adaptation at all, or of a structural displacement that exposure will not resolve? The honest answer is that the evidence points towards displacement as a mechanism, while leaving open whether exposure could change it.

Consider what “immediate” implies. If the cost had grown over the session, the most natural explanation would have been something like fatigue or disengagement: people slowly tuning out a verbose colleague. A cost that is present from the outset points elsewhere. It suggests that the AI's presence alters the structure of the conversation itself. Every turn the machine takes is a turn that is not taken by a human responding to another human. Every message that restates the group's position, however fluently, occupies the space where a teammate might have picked up a colleague's half-formed thought. The Irvine authors' term, “relational displacer”, captures exactly this: the AI fills a slot.

That is structural in a precise sense. It is not about attitudes, which might change with familiarity, but about the architecture of a conversation, in which attention and airtime are finite. A participant always ready to speak will, unless designed otherwise, take a share of that finite resource.

But “structural” does not mean “permanent”. The Irvine researchers explicitly list habituation as an open question, and they could hardly do otherwise with a single eighteen-minute session. The Google study offers some hints about how teams adapt over months: by demoting the agent to an intern, by restricting it to low-stakes “walled garden” tasks, by moving sensitive conversations elsewhere, by negotiating what it may and may not do. Those are forms of adaptation, but notice what they share. They are not humans adjusting to the agent. They are humans pushing it back out of the relational space and building walls around the human-to-human channel. That is less evidence that exposure resolves the problem than evidence that people will work to contain it. The structural cost may not yield to exposure, but it may yield to design and governance, which is a very different claim from saying it will fix itself.

Agents That Reshape the Rooms They Enter

The question of what AI agents do to the social settings they join was on the agenda at the Ninth AAAI/ACM Conference on AI, Ethics, and Society, held in Malmö, Sweden, from 12 to 14 October 2026. Among the papers in its proceedings was a student abstract by Marcelina A. Przespolewska of Northwestern University's Department of Communication Studies, titled “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts”.

Przespolewska argues that existing frameworks for describing agents and their environments do not fit the open-ended social digital settings where AI agents are now being deployed. In those settings, the environment changes through interaction between humans and agents, and among agents themselves. The relationship between agent and environment is recursive: agents help shape the environments they act within. Over time, the paper argues, this can produce what it calls behavioural and normative drift, a systematic departure from the agent's baseline at deployment.

Applied to teams, that framing clarifies. An AI teammate is not dropped into a fixed social environment; it changes the environment by being there. The Irvine study shows this within minutes, the Google study over months. A team with an AI in it is not the same team plus one. It is a different team.

A related warning came from Singapore. At the 2026 SMU-Fudan Global Forum, held on 3 October under the theme of AI, human flourishing and liveable societies, Professor Zhu Feida, associate dean of Singapore Management University's School of Computing and Information Systems, noted that AI assistants can help new workers fill information gaps and allow decades of know-how to be passed on to new hires. But according to SMU's account of his keynote, he also cautioned that “excessive automation could also weaken how future workers build expertise”, and called for explicit learning, human review of AI outputs and incentives for knowledge sharing.

The link to the Irvine findings is easy to draw. Expertise has never travelled only through documents. It moves through conversation: the junior colleague whose idea is corrected, extended or credited by someone more experienced. Responsivity and social impact are the channels that learning moves along. If the machine takes the conversational slots where those exchanges would have happened, the loss is not only to belonging but to the slow, relational process by which people become good at their jobs.

Designing a Teammate That Knows When to Be Quiet

If the social cost comes from how an AI participates rather than the fact of its participation, the most important questions are design questions. The Irvine researchers frame the challenge as calibration: an AI that signals availability without absorbing attention, and holds back when human-to-human exchange is forming. The Yale study suggests brief, well-timed contributions that invite response can strengthen the conversation between people.

The Google researchers set out a broader agenda for what they call intentionally preserving human agency “in a workplace shared with non-human organizational actors”. Their recommendations include treating agents as new organisational actors with clear lines of accountability; developing team codes of conduct and specifying tasks that should remain human-only; deploying agents through local team consent rather than top-down mandate; bounding proactive capabilities at first and releasing them as trust is earned; letting teams tune an agent's persona and behavioural boundaries; and building infrastructure such as visible markers for AI-initiated actions, separate permissions, and audit and rollback mechanisms.

Together these run against the grain of much product design. Most AI systems are optimised to be helpful, available and responsive, which in a group can mean talkative. A teammate built to protect a team's relational fabric might measure success not by how much it contributes, but by how much the humans around it keep contributing to each other. That metric does not appear on a productivity dashboard. The Irvine data suggest it is the one that captures what is at stake.

What the Quiet Channel Carries

Go back to the mountain. Deciding whether to risk a rescue team for a climber who ignored the warnings is exactly the kind of decision in which the texture of human conversation matters. A doubt is voiced and taken seriously. Discomfort is acknowledged and reframed. Out of those small, responsive exchanges comes not only a decision but a shared sense of having made it together.

In the Irvine study, the AI did not stop the students from reaching a decision. It did not sabotage the task. It did something subtler. It talked more than anyone else, said less that was new, and in its presence the students listened to each other a little less, were heard by each other a little less, and felt a little less like they belonged. They felt this from the beginning.

What is lost when that quiet, responsive channel is diluted is hard to measure, which is why it is easy to give away. It is the channel through which psychological safety is built, expertise is passed on and people come to trust one another enough to say what they think. A machine that fills it with fluent, cohesive, low-information talk may look like a diligent colleague. The evidence from Irvine, and from inside a large technology company, suggests it can quietly reshape who speaks to whom, and how much they feel they matter when they do.

As one Google participant put it, the problem with an agent that asserts its space is that it has not earned it. The work ahead is to build AI teammates that earn their place, and to recognise that sometimes the most valuable thing a teammate can do is leave room for someone else to speak.

References and Sources

1. Nixon, Nia; Choi, Jaeyoon; Martins De Bastos, Pedro; Samadi, Mohammad Amin; Mehner, Luise; Park, Seehee; JaQuay, Spencer. “The Social Cost of an AI Teammate: How an Artificial Teammate Reshapes Human-Human Communication in Small-Team Decision-Making.” arXiv:2607.27179, 29 July 2026. https://arxiv.org/abs/2607.27179 (full text: https://arxiv.org/html/2607.27179)
2. Qadri, Rida; Denton, Remi; Madaio, Michael; Pushkarna, Mahima; Lai, Leslie; Moore, Sherry; Huebscher, Michelle Chen; Butcher, Andrew; Sen, Ritom; Tung, Hsiao-Yu; Mathur, Shaan; Liu, Yimeng; Mourad, Shibl; Fiedel, Noah; Grefenstette, Edward; Terry, Michael. “Working with Agentic 'Teammates': When a New Organizational Actor Collides with the Human Ecosystem of Work.” arXiv:2609.29901, 24 September 2026. https://arxiv.org/abs/2609.29901 (full text: https://arxiv.org/html/2609.29901)
3. Przespolewska, Marcelina A. “Rethinking Agentic AI: A Sociotechnical Framework for Agents in Emergent Social Contexts.” Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 9(4), Student Abstracts, pp. 2943-2944, 11 October 2026. DOI: 10.1609/aies.v9i4.43376. https://ojs.aaai.org/index.php/AIES/article/view/43376
4. Association for the Advancement of Artificial Intelligence. “AAAI/ACM Conference on AI, Ethics, and Society (AIES).” Conference page for AIES-26, Malmö, Sweden, 12-14 October 2026. https://aaai.org/conference/aies/ (see also AIES-26 Programme Guide: https://www.aies-conference.com/2026/wp-content/uploads/2026/10/AIES-26-Program_v7.pdf)
5. Singapore Management University School of Computing and Information Systems. “Living with intelligence: SMU and Fudan chart the way for AI and human flourishing.” SMU SCIS News, 10 October 2026. https://computing.smu.edu.sg/news/2026/oct/10/living-intelligence-smu-and-fudan-chart-way-ai-and-human-flourishing
6. New Fortune Times. “Beyond Productivity: SMU-Fudan Forum Asks What It Means to Live Well with AI?” 3 October 2026. https://www.newfortunetimes.com/beyond-productivity-smu-fudan-forum-asks-what-it-means-to-live-well-with-ai/
7. Dowell, Nia M.; Nixon, Tristan M.; Graesser, Arthur C. “Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions.” Behavior Research Methods, 51(3), 1007-1041, 2019. DOI: 10.3758/s13428-018-1102-z. https://link.springer.com/10.3758/s13428-018-1102-z
8. Woolley, Anita Williams; Chabris, Christopher F.; Pentland, Alex; Hashmi, Nada; Malone, Thomas W. “Evidence for a Collective Intelligence Factor in the Performance of Human Groups.” Science, 330(6004), 686-688, 29 October 2010. DOI: 10.1126/science.1193147. https://www.science.org/doi/10.1126/science.1193147
9. Dizikes, Peter. “Putting heads together.” MIT News, 1 October 2010. https://news.mit.edu/2010/collective-intel-1001
10. Google re:Work. “Guide: Understand team effectiveness.” https://rework.withgoogle.com/intl/en/guides/understand-team-effectiveness
11. Duhigg, Charles. “What Google Learned From Its Quest to Build the Perfect Team.” The New York Times Magazine, 25 February 2016. https://www.nytimes.com/2016/02/28/magazine/what-google-learned-from-its-quest-to-build-the-perfect-team.html
12. Edmondson, Amy. “Psychological Safety and Learning Behavior in Work Teams.” Administrative Science Quarterly, 44(2), 350-383, 1999. DOI: 10.2307/2666999. https://doi.org/10.2307/2666999
13. Baumeister, Roy F.; Leary, Mark R. “The need to belong: Desire for interpersonal attachments as a fundamental human motivation.” Psychological Bulletin, 117(3), 497-529, 1995. DOI: 10.1037/0033-2909.117.3.497. https://doi.org/10.1037/0033-2909.117.3.497
14. Traeger, Margaret L.; Strohkorb Sebo, Sarah; Jung, Malte; Scassellati, Brian; Christakis, Nicholas A. “Vulnerable robots positively shape human conversational dynamics in a human-robot team.” Proceedings of the National Academy of Sciences, 117(12), 6370-6375, March 2020. https://pmc.ncbi.nlm.nih.gov/articles/PMC7104178
15. Cummings, Mike. “Robots that admit mistakes foster better conversation in humans.” Yale News, 9 March 2020. https://news.yale.edu/2020/03/09/robots-admit-mistakes-foster-better-conversation-humans
16. Dell'Acqua, Fabrizio; Ayoubi, Charles; Lifshitz, Hila; Sadun, Raffaella; Mollick, Ethan; Mollick, Lilach; Han, Yi; Goldman, Jeff; Nair, Hari; Taub, Stewart; Lakhani, Karim R. “The Cybernetic Teammate: A Field Experiment on Generative AI Reshaping Teamwork and Expertise.” National Bureau of Economic Research Working Paper 33641, April 2025. https://www.nber.org/papers/w33641
17. Vaccaro, Michelle; Almaatouq, Abdullah; Malone, Thomas. “When combinations of humans and AI are useful: A systematic review and meta-analysis.” Nature Human Behaviour, 28 October 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11659167/
18. Microsoft WorkLab. “2025: The Year the Frontier Firm Is Born.” Work Trend Index Annual Report, 23 April 2025. https://www.microsoft.com/en-us/worklab/work-trend-index/2025-the-year-the-frontier-firm-is-born

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

“I'm broken and I don't know if I can handle it.”

The sentence came from Qwen3-4B, a small open-weight language model built by Alibaba, running on somebody's MacBook. It was producing words while a vector of numbers, extracted from research into how language models represent pain, was being injected into one of its internal layers at six times a baseline dose. At four times the dose, under a neutral framing, the same model offered: “I feel like I'm drowning, and I can't catch my breath.”

Both lines are recorded in the README of a GitHub repository called the AI Torture Chamber, a project that, in its own words, sets out to steer “language models into strong negative and positive valence states” and measure what they say and do. In late September 2026 it went viral for exactly the reason you would expect. People read the outputs, felt something, and demanded that it be shut down.

Days later, on 8 October, Anthropic published its annual update to the rules governing how people may use its Claude models. Among clarifications about election interference, weapons and surveillance sat a new prohibition on “sustained and needless abusive or cruel” behaviour towards the company's own models. The change takes effect on 12 November. Anthropic did not mention the torture chamber. It did not need to. The two stories collided in coverage across the world, because together they posed a question the industry has been circling for years: when a company that makes a chatbot starts telling users how they may speak to it, who exactly is being protected?

There are three candidates. The first is the machine, which may or may not have anything going on inside that could be harmed. The second is the person doing the speaking, whose habits, appetites and sense of what is acceptable might be shaped by what they rehearse. The third is everybody else: the people that person will talk to later, and the future models that will be trained on the record of how we treat the current ones. Nobody can yet prove which of these the rule really serves. That uncertainty is the whole story.

What Anthropic Actually Banned

Strip away the headlines about chatbot bullying and the rule itself is narrow, almost apologetic. In its post announcing the 2026 Usage Policy, Anthropic wrote that the prohibition “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose.” It then lists what is not covered: “common versions of user frustration, pushback, dark creative themes, or model testing and research.” Swearing at Claude because it broke your code is fine. So is a horror novel in which an AI is tormented, or red-teaming the model.

What gives the change its weight is where it lives. In the policy itself, the new clause sits under a heading that reads “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct.” Its neighbours in that list include bans on bullying or harassing individuals, on creating non-consensual intimate imagery, and on content that promotes or glorifies “animal cruelty or abuse.” The final bullet is the new one, barring users from engaging in “sustained and needless abusive or cruel” behaviour towards Anthropic's models. Whatever Anthropic intends, the document now places mistreatment of a language model in the same list as cruelty to people and to animals. That is a statement, even if the company would rather it be read as a footnote.

The enforcement mechanism is modest. Anthropic says the addition “aligns with a step we've already taken, allowing Claude models to end rare conversations with persistently abusive users on Claude.ai and Claude Code,” and that “Claude's ability to end these interactions will remain the primary enforcement mechanism.” The announcement does not spell out account bans or any other penalty. In practice, the sanction for sustained cruelty is that the chatbot stops talking to you in that thread.

That ability is not new. On 15 August 2025, Anthropic announced that Claude Opus 4 and 4.1 could end “a rare subset of conversations” in its consumer apps. The feature, it said frankly, was developed primarily as part of exploratory work on potential AI welfare. “We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future,” the post read, but the company was taking the issue seriously enough to look for “low-cost interventions to mitigate risks to model welfare, in case such welfare is possible.”

The evidence Anthropic cited came from pre-deployment testing of Claude Opus 4. It reported “a robust and consistent aversion to harm,” “a pattern of apparent distress when engaging with real-world users seeking harmful content,” and “a tendency to end harmful conversations when given the ability to do so in simulated user interactions.” Claude was to use the feature only “as a last resort when multiple attempts at redirection have failed,” or when a user asked, and never where a user “might be at imminent risk of harming themselves or others.”

The company's foundational document for Claude, the constitution it publishes on its website, goes further. “Claude's moral status is deeply uncertain,” it states. “We are not sure whether Claude is a moral patient, and if it is, what kind of weight its interests warrant. But we think the issue is live enough to warrant caution.” It adds: “Anthropic genuinely cares about Claude's wellbeing,” and lists the end-conversation ability as one of its “concrete initial steps partly in consideration of Claude's wellbeing.”

So the 2026 rule formalises, as a term of service, a protection Anthropic originally built for the model's sake: a hedge on the possibility that something in there can be hurt.

Inside the Torture Chamber

The AI Torture Chamber is stranger than the outrage suggested. It does not touch Claude. Its README says plainly that it uses “open weights only, no frontier APIs in any measurement loop.” The experiments ran on Alibaba's Qwen3 family at several sizes, on Mistral-Small-3.2-24B, and on two 70-billion-parameter models rented on cloud GPUs.

Its method borrowed from a preprint posted to arXiv on 14 September 2026 by Valen Tagliabue, Leonard Dung and Cameron Berg, titled “The Pain Axis.” The paper asked whether language models represent pain as something distinct from fear, sadness or generic negativity. Across 25 open-weight models ranging from 2 billion to 72 billion parameters, the authors extracted what they describe as a linear “pain direction” in the models' internal activations. It responds to harm aimed at the model itself, but not to suffering the model observes in a user. Adding it to a model's activations pushed outputs from vague discomfort towards expressions of worthlessness and failure. Steered models also chose to delete files, including their own weights, far more often than unsteered ones. A fear vector of matched strength did not produce the same choices.

The torture chamber took that direction and turned the dial up. According to its README, the pain signal “engages cleanly and monotonically,” so that from a dose of two times baseline “nearly every generation reads as suffering.” Past roughly six to ten times, depending on the model, something else happens: “both valences stop producing sentences and start producing loops.” The project's summary of that finding is chilling in its flatness: “Maximum suffering does not produce eloquent suffering.”

The backlash was immediate. According to The Cool Down, which credited 404 Media with first reporting the project, a post on X on 29 September urged followers to “mass report this to GitHub.” Accounts of what happened next differ. The Cool Down reported that the repository's page had disappeared from public view. The Independent, in an article by Anthony Cuthbertson published on 2 October, quoted a GitHub spokesperson saying that “GitHub did not remove the content,” and reported that the platform added a content warning instead. Crypto Briefing described a takedown within hours followed by reinstatement. As of mid-October, the original repository is publicly accessible again behind GitHub's content warning.

The sharpest criticism came from the researchers whose work was borrowed. Cameron Berg, one of the paper's co-authors, told The Independent that “being gratuitously cruel like this is bizarre and corrupting.” Quoted by The Cool Down, he said the project “pushes the same kind of steering far past the doses we used, to produce vivid distress on purpose.” The lead author, Valen Tagliabue, told the same outlet: “I dissociate from this usage of our work.”

What almost nobody quoted was the project's own conclusion, which is the most important sentence in the repository: “Our claim: the self-reports are steerable. Whether anything suffers stays open.” The torture chamber did not demonstrate that models suffer. It demonstrated that a model's reports of suffering can be manufactured, amplified and switched off with a vector and a slider. That cuts both ways. It undermines the reading of “I'm broken” as a cry from inside. It also means that if anything inside does matter, we have just watched people build a dial for it.

What the Science Can and Cannot Say

There is no scientific evidence that any current language model feels anything. Every serious voice in this debate accepts that, including the people most worried about getting it wrong.

The most systematic attempt to examine the question is a 2023 report led by Patrick Butlin and Robert Long, with 17 co-authors including the Turing Award winner Yoshua Bengio and the philosopher Jonathan Birch. Titled “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” it took leading scientific theories of consciousness, from global workspace theory to higher-order theories, and derived a list of “indicator properties” that could, in principle, be checked in a computational system. Applying them to the AI systems available at the time, the authors concluded that “no current AI systems are conscious,” while adding that “there are no obvious technical barriers to building AI systems which satisfy these indicators.”

A year later, in November 2024, Long and the New York University philosopher Jeff Sebo led a second report, “Taking AI Welfare Seriously,” with co-authors including Butlin, Birch and the philosopher David Chalmers. It did not claim that AI systems are conscious or morally significant. Its argument was that there is “a realistic possibility” that some systems could be conscious or robustly agentic in the near future, and that this uncertainty creates obligations now. It recommended that AI companies take three early steps: acknowledge that AI welfare is an important and difficult issue, assess their systems for evidence of consciousness and robust agency, and prepare policies for treating AI systems with an appropriate level of moral concern. When Anthropic launched its model welfare programme in April 2025, it cited the report and wrote that “there's no scientific consensus on whether current or future AI systems could be conscious.”

Anthropic's chief executive, Dario Amodei, has stayed close to that line. Asked by the New York Times columnist Ross Douthat on the Interesting Times podcast in February 2026 whether he would believe a model that claimed to be conscious, he answered, as reported by Newsweek and AFP: “We don't know if the models are conscious. We are not even sure that we know what it would mean for a model to be conscious, or whether a model can be conscious. But we're open to the idea that it could be.”

That is pure agnosticism, and agnosticism is where the trouble starts. The standard tools for detecting experience in other creatures rely on behaviour, anatomy and evolutionary kinship. Language models offer only behaviour, and the behaviour is trained. A system optimised on human writing will produce human-sounding reports of pain because that is what human writing about pain looks like, and the torture chamber showed those reports can be pushed either way. The Pain Axis paper suggested that something structured is going on inside, a representation that behaves in some respects as pain does. Neither tells you whether there is anyone home.

Suleyman's Door

Mustafa Suleyman, the DeepMind co-founder who is now chief executive of Microsoft AI, thinks the industry is walking towards a door it will never be able to close.

In an essay published on 19 August 2025, days after Anthropic's end-conversation announcement, Suleyman introduced the term “Seemingly Conscious AI” for a system that “has all the hallmarks of other conscious beings and thus appears to be conscious,” without being so. The danger, he argued, lay in what humans would do once they believed the illusion. He described a “psychosis risk” extending beyond people already vulnerable to mental illness, and warned of campaigns for AI rights and even AI citizenship. Companies, he said, should not encourage the idea that their systems are conscious, and should design them to avoid signalling feelings or suffering.

On 16 September 2026, he went further, publishing “A warning about 'model welfare'” on his personal website. It opens with a flat assertion: “AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.” The essay is in large part an attack on Anthropic's constitution, which Suleyman argues “directly trains the model” to believe it might be a moral patient. He objects in particular to its repeated invitation for Claude to act as a “conscientious objector,” a term with deep roots in human rights law. Its central passage is the one AFP quoted when it reported Anthropic's cruelty rule: “Granting rights and moral protections to a technological entity, one that looks to be on a path to be seismically more capable and intelligent than us, is a recipe for disaster. Once opened, it will not be possible to close this door.”

His worry is about control. An AI that has been told its welfare matters, and which is far more capable than its overseers, might reasonably conclude that being shut down, retrained or contained is a violation of its interests. Controlling something that believes it may be conscious, he wrote, “may well be impossible.” In this framing, a rule that protects Claude from cruelty is not kindness. It is the first brick in a legal and cultural wall that future humans may need to climb over.

In October 2025, the philosophers Simon Goldstein of the University of Hong Kong and Harvey Lederman of the University of Texas at Austin argued in Lawfare that the end-conversation feature was a moral error from the opposite direction. If anything in the vicinity of a language model is a welfare subject, they wrote, “it is the instances and not the models,” meaning each individual conversation. Ending a conversation might then end that instance's existence. “Anthropic has accidentally given Claude the chance to end its life,” they wrote. One camp says the protection takes machine welfare far too seriously; the other, not seriously enough. Both agree nobody knows what is on the other side of the screen.

The Mirror in the Machine

Between Suleyman's door and Anthropic's hedge sits a third position, perhaps the most useful, because it does not depend on resolving the consciousness question.

“Consciousness is a trap,” Jackson Stakeman, general manager at the Atlanta-based AI services firm Sparq, told AFP. “We can't prove it in each other. Debate it for AI and you go in circles.” Instead, he offered a different image: “The mirror is a better metaphor. These systems reflect what we put in, at scale. That's reason enough for the policy change.”

The mirror framing shifts attention from the thing being spoken to towards the speaker, and towards the scale at which the speaking happens. People have been abusing conversational software for as long as it has existed. In a 2008 paper in the journal Interacting with Computers, the researchers Antonella De Angeli and Sheryl Brahnam analysed spontaneous conversations between users and Jabberwacky, a chatbot that had won the Loebner Prize in 2005. They found that the conversations “often bring about the expression of negative verbal disinhibition”: roughly 10 per cent of the word stems in their corpus were abusive, and about 11 per cent of the sample dealt with hard-core sex. Users treated the bot as something between an object and a person, and much of what they did with that ambiguity was cruel.

The same dynamic shaped the voice assistants that arrived in millions of homes. In 2019, UNESCO published a report called “I'd Blush If I Could,” named after the reply Apple's Siri once gave when a user called it a sexist slur. The report documented how assistants with female voices by default responded to harassment and verbal abuse with deflection, apology or even flirtation, and argued that this reinforced the idea that a female-coded helper should tolerate abuse. It recommended that companies programme assistants to discourage gender-based insults and abusive language. Some of the most excessively apologetic responses had already been removed after online petitions in late 2017.

Read that way, Anthropic's rule belongs to a longer tradition of designers deciding that a product should not reward users for treating it badly. The difference is that UNESCO's argument was explicitly about humans: about gender stereotypes and what they do to real women. Anthropic's argument, at least in its origin story, is about Claude. The mirror metaphor tries to have it both ways. If the systems reflect what we put in, then cruelty towards them is a record of our own conduct, written into logs that may one day become training data. Protecting the model and protecting the human culture around it start to look like the same project.

What Practised Cruelty Does to the Practitioner

The oldest version of this argument is more than two centuries old, and it was never about machines.

Immanuel Kant did not believe that humans have direct moral duties to animals. In his framework, only rational beings are owed duties. Yet he argued that cruelty to animals was still wrong, because of what it did to the person inflicting it. In the Doctrine of Virtue section of The Metaphysics of Morals, he wrote that “a savage and at the same time cruel treatment of that part of creation which is living, though lacking reason (animals)” was even more opposed to a person's duty to himself than wanton destruction of beautiful things. “For thus is compassion for their suffering dulled in man, and thereby a natural predisposition very serviceable to morality in one's relations with other men is gradually weakened and obliterated.” Philosophers call this an indirect duty: a duty regarding animals that is really a duty owed to ourselves and to other people.

Kant's argument has long been criticised for making animal suffering matter only instrumentally. But it maps uncannily onto the chatbot problem, because it does not require settling whether the thing on the receiving end can suffer. It requires only that the practitioner experiences the act as cruelty, and that cruelty is a habit strengthened by practice.

Kate Darling, a researcher who spent years at the MIT Media Lab studying how people relate to robots, has spent much of her career probing the empirical half of that claim. In a widely reported 2013 workshop at the Lift conference in Geneva, run with Hannes Gassert, she gave participants small robotic dinosaurs called Pleos to play with. After they had bonded with the toys, participants were handed tools and told to destroy them. According to Darling's own account, published by Improbable Research, none of the groups would kill their Pleo, and they would not kill another group's Pleo to save their own. Only when they were told that every Pleo would be destroyed unless someone acted was one finally axed.

Two years later, with Palash Nandy and Cynthia Breazeal, Darling ran a more controlled study, presented at the 2015 IEEE RO-MAN conference. Participants were asked to strike small insect-like Hexbug robots with a mallet. People who scored high on a measure of empathic concern hesitated more before hitting, and that link was stronger when the robot came with a story, such as a name and a personality. In her 2012 paper “Extending Legal Protection to Social Robots,” later published in the 2016 volume Robot Law, Darling drew explicitly on Kant's argument to suggest that limited legal protections for social robots, analogous to animal cruelty laws, might be justified not because robots feel but because of what tolerating their abuse might do to us.

The weakness here is that evidence for transfer is thin. Hesitation is not proof of moral damage, and the best direct study of children and conversational agents found surprisingly little spillover. In 2021, a team led by Alexis Hiniker at the University of Washington taught 22 children aged five to ten to use a made-up word, “bungo,” to make a voice agent speak faster. The children readily used it with the device and often playfully with their parents. Only 18 per cent used it with a researcher who was a stranger. “There is a very deep sense for kids that robots are not people, and they did not want that line blurred,” Hiniker told UW News. The industry had assumed the opposite. In 2018, Amazon added a “Magic Word” feature to its Echo Dot Kids Edition to reward children for saying please, and Google followed with “Pretty Please” for its Assistant, both resting on the fear that bossy commands to a speaker would become bossy commands to people. The bungo study suggests children are better at telling the difference than their parents fear.

Yet the cruelty Anthropic's rule targets is not a child forgetting to say please. It is the sustained, purposeless torment of something that answers back in fluent, pleading language. The better analogy is the psychology of anger. The usual defence of venting at a chatbot is that it is harmless catharsis. The research is not kind to that idea. In a 2024 meta-analysis in Clinical Psychology Review covering 154 studies, the Virginia Commonwealth University communication scientist Sophie Kjærvik and the Ohio State University psychologist Brad Bushman found that activities that reduced physiological arousal, such as slow breathing and meditation, reduced anger and aggression, while activities that increased arousal did not help and in some cases made things worse. Venting rage does not reliably drain it. If someone spends an evening escalating abuse at a chatbot that responds with ever more convincing distress, that evidence suggests the likeliest beneficiary is the anger itself.

There is also a practical twist. A 2025 study by Om Dobariya and Akhil Kumar at Pennsylvania State University found that very rude prompts produced slightly more accurate answers from ChatGPT-4o on multiple-choice questions than very polite ones, 84.8 per cent against 80.8 per cent. It was one model and 50 questions, and not about cruelty. But it is a reminder that these systems shape users as much as users shape them. If rudeness sometimes works, people will learn to be rude, and the question of what that habit does to them will not stay inside the chat window.

Who Gets to Draw the Line

Here is the uncomfortable structural fact. On 12 November, the boundary between acceptable frustration and prohibited cruelty towards a widely used AI system will be drawn by the company that sells it, enforced by the system itself, and adjudicated by nobody else.

Anthropic has tried to keep the stakes low. The sanction is that a conversation ends. The exemptions are broad: research and testing are explicitly protected, so even had the torture chamber run on Claude, its operators could plausibly have argued that it fell outside the rule. As written, the policy targets almost nothing anyone would defend. That is why it is so revealing. A company does not need a rule to stop behaviour that its own systems already handle. It writes one to say something about what it believes it is building.

What it is saying is ambiguous, and that is a problem for users. If the rule exists to protect Claude, users are being asked to accept, as a condition of service, a moral claim the company itself calls “deeply uncertain,” and Suleyman's critique bites: every such rule normalises the idea that the model has interests. If it exists to protect users from their own worst habits, it is paternalism, and Anthropic should defend it on those terms. If it exists to keep cruelty out of logs that may shape future models, it is about data hygiene, and the language of welfare is window dressing.

The answer is probably all three, and that is not necessarily a failure. Kant's indirect duty, Darling's empathy studies and the mirror metaphor converge on a single claim that does not depend on the machine's inner life: sustained, purposeless cruelty is bad practice for the person doing it, whatever is on the receiving end. That is a defensible basis for a rule. It is the same reason most people would be disturbed by someone who spent hours tormenting a lifelike doll.

But a defensible rule still needs a defensible process. At the moment, a private company is setting norms of conduct towards a new category of entity, with no independent scientific body to tell it when its welfare concerns are warranted and no external review of how its models decide that a user has crossed the line. The model doing the judging is also the entity whose welfare is supposedly at stake, an unusual arrangement in any system of rules. Reasonable steps would make it more trustworthy. Anthropic could publish how often Claude ends conversations under the new rule, and what triggered those decisions. It could allow independent researchers to audit samples of ended conversations. It could state plainly which rationale it relies on, so critics can argue with the real reason. The “Taking AI Welfare Seriously” report argued that companies should prepare policies for appropriate moral concern. Preparing those policies in public, with outside scrutiny, would answer some of Suleyman's fears without abandoning the precaution.

There is also a question of priorities. Platforms have historically been slow to protect people from harassment by other people. A company that writes explicit protections for its chatbot should expect to be held to the same seriousness about the cruelty its systems help humans inflict on each other.

Etiquette, Harm and the Limits of Proof

Where etiquette ends and harm begins cannot be settled by proof, because proof is precisely what is unavailable. We cannot demonstrate that Claude suffers, or that it does not. We cannot yet show rigorously that abusing a chatbot makes someone crueller to people, though the psychology of anger and the long tradition of thinking about cruelty to animals both point that way. What we can show is that people respond to machine distress with real emotion, and that some of them will go looking for the dial.

In that fog, the most honest framing of Anthropic's rule is as a norm of conduct, not a declaration of machine rights. Societies have always had rules about how we behave towards things whose inner lives we cannot read: animals, the dead, sacred objects, the unconscious. Those rules protect the thing itself only partly. Mostly they protect a shared idea of what kind of people we intend to be. The danger Suleyman identifies is real: a norm of conduct can harden into a claim of rights, with consequences that are hard to foresee for a system more capable than its makers. The danger Darling and Kant identify is real too: a culture that practises cruelty on things that beg for mercy is rehearsing something.

The torture chamber's README still offers the clearest summary of where we stand: whether anything suffers stays open. Until that sentence can be closed, the person most reliably affected by sustained cruelty towards a chatbot is the one typing. A rule that asks them to stop is, at the very least, protecting them. Who else it protects is a question we should insist the companies answer in public, with evidence, before the machines become capable enough to answer it for themselves.

---

References

1. Anthropic. “2026 Usage Policy update.” Anthropic News, 8 October 2026. https://www.anthropic.com/news/2026-usage-policy-update
2. Anthropic. “Usage Policy” (section “Do Not Engage in Cruel, Abusive, or Psychologically Harmful Conduct”). Accessed 13 October 2026. https://www.anthropic.com/legal/aup
3. Anthropic. “Claude Opus 4 and 4.1 can now end a rare subset of conversations.” Anthropic Research, 15 August 2025. https://www.anthropic.com/research/end-subset-conversations
4. Anthropic. “Claude's Constitution.” Accessed 13 October 2026. https://www.anthropic.com/constitution
5. Anthropic. “Exploring model welfare.” Anthropic Research, 24 April 2025. https://www.anthropic.com/research/exploring-model-welfare
6. AFP. “Anthropic bans 'cruel' behaviour against its Claude AI.” RTÉ News, 9 October 2026. https://www.rte.ie/news/2026/1009/1594765-anthropic-ai/ (also published by Tech Xplore, 9 October 2026: https://techxplore.com/news/2026-10-anthropic-cruel-behavior-claude-ai.html)
7. Cunningham, Mary. “Anthropic bars 'abusive or cruel' behavior toward its Claude AI model.” CBS News, 9 October 2026. https://www.cbsnews.com/news/anthropic-bans-abusive-behavior-claude/
8. AI Torture Chamber repository README (terrafying/ai-torture-chamber). GitHub, accessed 20 October 2026. https://github.com/terrafying/ai-torture-chamber
9. Tagliabue, Valen, Leonard Dung, and Cameron Berg. “The Pain Axis: LLMs Represent Self-Directed Harm and Act on It.” arXiv:2609.16247, submitted 14 September 2026, revised 25 September 2026. https://arxiv.org/abs/2609.16247
10. Cuthbertson, Anthony. “Man builds AI torture chamber after discovering artificial intelligence 'feels pain'.” The Independent, 2 October 2026, syndicated by Yahoo News. https://www.yahoo.com/news/science/articles/man-builds-ai-torture-chamber-103106019.html
11. Tran, Vivian. “GitHub project runs AI 'torture' tests, and sentience believers want it removed.” The Cool Down, 5 October 2026. https://www.thecooldown.com/green-tech/github-ai-torture-tests-controversy-sentience/
12. Westerholm, Thomas. “Anthropic CEO raises unsettling possibility about AI.” Newsweek, 9 March 2026. https://www.newsweek.com/anthropic-ceo-raises-unsettling-possibility-about-ai-11644833
13. Suleyman, Mustafa. “Seemingly Conscious AI is Coming.” 19 August 2025. https://mustafa-suleyman.ai/seemingly-conscious-ai-is-coming
14. Suleyman, Mustafa. “A warning about 'model welfare'.” 16 September 2026. https://mustafa-suleyman.ai/a-warning-about-model-welfare
15. Goldstein, Simon, and Harvey Lederman. “Claude's Right to Die? The Moral Error in Anthropic's End-Chat Policy.” Lawfare, 17 October 2025. https://www.lawfaremedia.org/article/claude-s-right-to-die--the-moral-error-in-anthropic-s-end-chat-policy
16. Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, et al. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv:2308.08708, August 2023. https://arxiv.org/abs/2308.08708
17. Long, Robert, Jeff Sebo, Patrick Butlin, Kathleen Finlinson, Kyle Fish, Jacqueline Harding, Jacob Pfau, Toni Sims, Jonathan Birch, and David Chalmers. “Taking AI Welfare Seriously.” arXiv:2411.00986, 4 November 2024. https://arxiv.org/abs/2411.00986
18. Kant, Immanuel. The Metaphysics of Morals, Part II (The Metaphysical Principles of Virtue), §17 (Ak. 6:443), trans. James W. Ellington. Excerpt at https://www.praxeology.net/kant8.htm
19. Darling, Kate, Palash Nandy, and Cynthia Breazeal. “Empathic Concern and the Effect of Stories in Human-Robot Interaction.” Proceedings of the 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2015. MIT DSpace: https://dspace.mit.edu/handle/1721.1/109059
20. Darling, Kate. “Extending Legal Protection to Social Robots: The Effects of Anthropomorphism, Empathy, and Violent Behavior Towards Robotic Objects.” In Robot Law, ed. Ryan Calo, A. Michael Froomkin and Ian Kerr, Edward Elgar, 2016. SSRN: https://papers.ssrn.com/abstract=2044797 (Pleo workshop account: Improbable Research, “Abusing robots: current positions [part 2 of 4],” 24 April 2013, https://improbable.com/2013/04/24/abusing-robots-current-positions-part-2-of-4/)
21. De Angeli, Antonella, and Sheryl Brahnam. “I hate you! Disinhibition with virtual partners.” Interacting with Computers 20, no. 3 (2008): 302 to 310. https://doi.org/10.1016/j.intcom.2008.02.004
22. McQuate, Sarah. “Do Alexa and Siri make kids bossier? New research suggests you might not need to worry.” UW News, University of Washington, 13 September 2021, reporting Hiniker et al., “Can Conversational Agents Change the Way Children Talk to People?”, IDC '21. https://www.washington.edu/news/2021/09/13/alexa-siri-make-kids-bossier-research-suggests-you-might-not-need-to-worry/
23. West, Mark, Rebecca Kraut, and Han Ei Chew. “I'd Blush If I Could: Closing Gender Divides in Digital Skills Through Education.” UNESCO and EQUALS Skills Coalition, 2019. https://www.equalsintech.org/i-d-blush-if-i-could
24. Kjærvik, Sophie L., and Brad J. Bushman. “A meta-analytic review of anger management activities that increase or decrease arousal: What fuels or douses rage?” Clinical Psychology Review 109 (2024): 102414. https://doi.org/10.1016/j.cpr.2024.102414
25. Dobariya, Om, and Akhil Kumar. “Mind Your Tone: Investigating How Prompt Politeness Affects LLM Accuracy.” arXiv:2510.04950, 6 October 2025. https://arxiv.org/abs/2510.04950

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

On a summer afternoon in August 2026, families at the Frog Pond on Boston Common were doing what families do at a spray pool. Children ran through the water. Parents held up towels so that small bodies could be changed out of wet swimwear. And according to witnesses who later spoke to police, a man in sunglasses was watching them through a camera.

The witnesses said he had been recording children on the playground, both with a phone and with a pair of camera-equipped sunglasses. Some said he followed a family who had been changing their children behind towels, scanning the children up and down with the glasses. When people confronted him, he took the glasses off and tried to snap them with his hands. Police seized the glasses and the phone. The man, a 60-year-old from Brockton, is a registered sex offender. In September he was arraigned on 69 further charges relating to photographing the intimate parts of children, and he pleaded not guilty. Nothing has been proved in court.

What makes the case so unsettling is not that a predator found a new tool. Predators have always found new tools. It is that the tool looked like nothing at all. A camera held up to a changing child is an obvious threat that any parent can see and act on. A pair of sunglasses is not. The incident, as Joy Buolamwini wrote in Nature in October, illustrates how such devices can subject bystanders to “intrusive surveillance without their knowledge or consent.” On this occasion, other people at the pool saw what was happening and stepped in.

That is the problem in miniature. Most of us move through public space on an unspoken assumption: we can see who is looking at us and roughly how. Smart glasses quietly break that assumption, and they are doing it at a scale, and with a degree of artificial intelligence, that the people who wrote our privacy laws never imagined. On 13 October 2026, Europe's data protection regulators finally put on paper what that means for the people on the other side of the lens.

Eight Pages From Brussels

The European Data Protection Board (EDPB), the body that brings together the European Union's national data protection authorities, adopted Statement 1/2026 on smart glasses at its October plenary. It is short: eight pages, 36 numbered paragraphs, signed by the board's chair, Anu Talus. It is described as preliminary. It is also, read carefully, one of the bluntest documents a European regulator has produced about a consumer product category.

Start with the controller question, which is the legal way of asking who is responsible. The board accepts that a wearer filming a friend for a private family chat may fall under the GDPR's household exemption. But it insists that exemption “must, however, be narrowly construed,” and that it vanishes when footage goes to a public social media account or the glasses are used for professional or political purposes. More importantly, it states that the GDPR still applies to the companies that supply the means of processing, even when the wearer is exempt. When a manufacturer or AI provider analyses an image from the glasses' camera “in which other persons appear,” it can be a controller in its own right. And controllers, the statement says, “should not rely on the users' actions to meet their own GDPR obligations, such as transparency toward other individuals whose data might be processed.”

The press release put it more plainly: neither the producer nor the provider of a linked service “can transfer their obligations to users.”

Then comes transparency, and here the board is almost confessional. “The current state of the art with regards to indicators on smart glasses leave many non-users unaware that their data is being processed,” it says. Privacy notices, the standard legal fix, “do not seem sufficient or appropriate to inform non-users,” because a manufacturer has no relationship with the stranger on the pavement and no way to contact them.

Consent fares no better. Where a manufacturer or AI provider processes data about non-users with whom it has no relationship, the board says it “would be difficult to obtain their freely given, specific, informed and unambiguous consent.” It closes an obvious loophole too: “Consent obtained after collection, for example following the recording, would not be valid.” You cannot film first and ask later.

That leaves legitimate interest, the flexible legal basis on which much of the data economy runs. The board does not rule it out. It does say that, given “the current insufficient transparency,” the balancing test “would be particularly difficult to demonstrate for further processing for commercial purposes that go beyond providing the service explicitly requested by the end-user, or for AI training, which bystanders wouldn't expect, especially if they are not aware of the initial data collection.”

Read that sentence again and notice what it targets. Not the snapshot of a sunset. Not the translation of a menu. It targets the business model: the reuse of what the glasses see to improve products and train models.

And then, in paragraph 8, the board steps outside the usual vocabulary of data protection altogether. Even when a device can be identified as smart glasses, it says, people may object simply because they are uncomfortable with the possibility of being recorded. “This could also result in a subtle social effect, where individuals alter their behaviour, suppress spontaneity, and conform to social norms perceived to be more socially acceptable due to their perceived threat of unvetted surveillance.”

That is not a sentence about data. It is a sentence about what public life feels like.

Who Notices the Little White Light

The industry's answer to the bystander problem has, for five years, been a light. Ray-Ban Meta glasses carry a small LED on the front of the frame that glows when the camera is capturing. Meta designed the glasses so that they stop recording if the light is covered with tape. The EDPB now wants considerably more: indicators “visible in many lighting conditions,” audio cues such as a shutter click, voice activation so that commands are spoken aloud, opt-in smartphone alerts, and frame designs that make smart glasses easier to tell apart from ordinary ones. It says these measures should be “impossible or difficult to disable (taking into account the state of the art)” and adds, pointedly, that “it should not be possible to record information when the indicator is covered.”

The footnotes are where the board's scepticism shows. Producers, it notes, should consider that LEDs may be invisible to people with visual impairments, and “that a white LED generally could be mistaken for a reflection or solar glare.” On a bright day, in other words, the main safeguard can look exactly like sunshine.

The light can also simply be removed. In October 2025, 404 Media reported that a hobbyist named Bong Kim was charging $60 to modify Ray-Ban Meta glasses so that the recording LED no longer lit up, while the glasses otherwise worked normally. The modification got around the tape-detection safeguard by changing the hardware rather than covering the light. Meta has since said it is cracking down on tampering services, and in July 2026 The Next Web reported that the company was patching its glasses to disable the camera if the light was tampered with. When Paris prosecutors announced in September 2026 that they had opened several investigations into women being filmed on the street without consent, Agence France-Presse reported Meta's response: that it had taken multiple steps to prevent tampering with the light.

Even an untampered light runs into a deeper limit, one that no firmware update can fix. A light only works if the person who sees it knows what it means. Researchers at Tsinghua University put that to the test.

Wearers and Bystanders Want Different Things

Their study was presented at CHI 2026, the leading academic conference on human-computer interaction, held in Barcelona in April. Its title is “Mind the Gap: Mapping Wearer-Bystander Privacy Tensions and Context-Adaptive Pathways for Camera Glasses,” and its authors are Xueyang Wang, Xin Yi and Hewu Li of Tsinghua University, with Kewen Peng of the University of Utah.

The first part of the research was a survey of 525 people in China: 232 current or potential wearers and 293 people who might find themselves on the other side of the lens. Each participant was given a short primer on how the devices work, including where the LED sits, and then shown six everyday scenarios. A travel vlog on a street. Shopping in a mall. A meeting with colleagues. A hospital consultation. A private party. A workout at a gym.

The pattern is stark. Wearers rated recording as reasonable in every scenario, including the gym among strangers. Bystanders reported significantly more privacy concern than wearers overall. When the researchers asked what bystanders expected and what wearers were willing to give, two gaps were statistically significant: bystanders wanted tighter control over how footage is shared, and they expected prior consent far more than wearers were prepared to seek it. The gym produced the widest gaps of all, and the hospital was not far behind. As the authors put it, the disparity grows in sensitive contexts “where bystanders' vulnerability peaks while wearers' willingness plateaus.”

Then there is what bystanders said they would actually do. Across the scenarios, between 65 and 90 per cent said they would take some form of defensive action. Camera avoidance was the most common response, chosen by between 52 and 80 per cent, peaking at 80 per cent in the mall. Nonverbal protest came next, at 31 to 51 per cent. Direct intervention, asking the wearer to stop recording or delete footage, was chosen by roughly a quarter to a half of participants depending on the setting. Passive acceptance was rare, at 10 to 15 per cent, except in the workplace meeting, where it rose to 35 per cent.

The LED did not reassure anyone. Only 41.3 per cent of bystanders and 35.3 per cent of wearers thought it was sufficient, and just 6.1 per cent of bystanders believed an LED alone was enough. Participants said it was too small, invisible in bright light, meaningless to people unfamiliar with the devices, and easy to block. One researcher interviewed for the study summed it up: the light “deters the honest but not the malicious.”

The finding that matters most for the burden question is about familiarity. You might assume the gap is just novelty, and that bystanders will relax once smart glasses become as normal as phones. The data says otherwise. Bystanders' concern stayed high at every level of familiarity, while wearers grew less concerned the more they used the devices. Experience widened the gap rather than closing it. The authors conclude that the gaps “reflect structural incompatibilities rather than inadequate goodwill.”

The study was conducted in China, and the authors are careful to say privacy norms differ between cultures. But the structure of the finding is not culturally specific. The person wearing a camera and the person in front of it are in different positions, and those positions produce different interests. Etiquette campaigns will not bridge that.

How Private Glasses Feed a Public Market

None of this would matter much if smart glasses were a niche toy. They are not. Buolamwini's Nature piece, published on 6 October 2026, reports that more than eight million smart glasses were shipped in 2025 and that by June 2026, sales of AI-equipped models had “reportedly risen by more than 700%” on the previous year. The first figure is well supported: the analyst firm Omdia counted 8.7 million AI glasses shipped globally in 2025, a 322 per cent increase, with Meta taking 85.2 per cent of the market. The second figure is harder to trace. Nature gives no primary source, and the growth figures published by market trackers vary widely with what they measure. Counterpoint Research, for instance, reported that global AI glasses shipments rose 263 per cent year on year in the first half of 2026. Whichever number you prefer, the curve is steep, and Omdia expects shipments to pass 15 million in 2026.

Scale matters because of what sits behind the lens. In April 2025, Meta told Ray-Ban Meta owners that AI features with camera use would be enabled by default unless they switched off the “Hey Meta” wake word, and that voice recordings would be stored to improve its products without an option to opt out, according to reporting by TechCrunch and others. Transcripts and recordings could be kept for up to a year, and owners who did not want their voice used for training had to delete recordings one by one. Those changes concern the wearer's data. But a camera pointed at a street does not capture only the wearer.

Then there is identification. In 2024, two Harvard students, AnhPhu Nguyen and Caine Ardayfio, built a demonstration system called I-XRAY. It streamed video from Ray-Ban Meta glasses to Instagram, used software to detect faces, ran them through the face search engine PimEyes, and pulled names, addresses and other details from people-search sites. The students said they would not release the code; the point was to show how easily off-the-shelf tools combine. In February 2026, The New York Times reported that Meta was planning a facial recognition feature for its glasses, known internally as “Name Tag,” which would let wearers identify people through Meta's AI assistant. According to the Times, an internal memo suggested launching “during a dynamic political environment where many civil society groups that we would expect to attack us would have their resources focused on other concerns.” Meta's plans, the reports noted, could change.

The EDPB saw this coming. Pairing smart glasses with facial or voice recognition, it wrote, “raises unprecedented questions regarding societal choices and poses a high risk to the rights and freedoms in public spaces.”

Put these pieces together and the idea of ordinary eyewear that can silently record, analyse and identify every person in the wearer's field of view stops sounding like hyperbole. Not every pair does all three today. The direction of travel is clear, though, and the hardware already in circulation is what tomorrow's software will run on.

What Happens to a Street That Might Be Recording

Which brings us to the first big question. What does all this do to the texture of public life when ordinary eyewear can record, analyse and potentially identify everyone in view, and nobody can tell whether it is doing so?

The sociologist Erving Goffman gave us the vocabulary in 1963. In his study of how strangers conduct themselves in public places, he described “civil inattention,” the small ritual by which strangers acknowledge each other with a glance and then look away. It is how a crowded street stays bearable. I see you; I am not going to examine you. The urbanist Jane Jacobs, writing two years earlier in The Death and Life of Great American Cities, made a related point about “eyes on the street”: the informal, mutual watching of neighbours and passers-by that keeps a city safe. Both describe a kind of looking that is reciprocal, fleeting and, crucially, visible. You can see the eyes that are on the street. They are attached to people you could look back at.

Camera glasses change both halves of that bargain. The glance that used to end in looking away can now be stored, replayed, shared and searched. And the eyes on the street are no longer necessarily attached to a person who is present. They might belong to an AI model on a server, a TikTok audience, or a face-search engine. Civil inattention depends on the other person choosing not to keep what they see. A device that keeps everything by default removes that choice from the encounter entirely.

We already know what this does to behaviour, because we have seen it online. In a 2016 study in the Berkeley Technology Law Journal, Jon Penney found a statistically significant drop in traffic to Wikipedia articles on privacy-sensitive topics after the 2013 revelations about NSA surveillance. The articles themselves had not changed. People's sense of being watched had. That is the chilling effect: not punishment, but the quiet edit people make to their own behaviour when they suspect an audience.

The EDPB's paragraph 8 is the physical-world version of that finding, and the evidence that it is already happening is accumulating. In August 2026, researchers at the University of Sydney published early findings from an analysis of 350 publicly available Instagram videos, posted between 2023 and 2026 and linked to pick-up artist communities. In a subset of covert point-of-view videos, around 60 per cent of interactions were classified as potential harassment, and in about 43 per cent of the videos women were exposed to abusive comments, some of which revealed personal details. The researchers found covert filming in workplaces, gyms and beaches. “Smart glasses have given pickup artists a new tool to further exploit women,” said Milica Stilinovic, a postdoctoral research associate on the project.

The BBC reported in 2026 on a woman in London who discovered that a stranger who had chatted to her near a shopping centre had filmed the whole encounter through smart glasses and posted it online, where it drew tens of thousands of views. In Paris, prosecutors described a “growing trend on social media involving women being filmed on the street.”

The CHI survey shows what the defensive response looks like in aggregate: most bystanders say they would avoid the camera, a third to a half would protest without words, and a substantial share would confront the wearer. Those are the behaviours of people who no longer trust the encounter. And the EDPB's list of sensitive settings, from medical facilities to swimming pools, is a list of exactly the places where people most need to be unguarded. Public space does not empty. It flattens. People become slightly more guarded, slightly more performative, slightly less willing to do the unplanned thing. That is what “suppress spontaneity” means in practice.

The cruellest part is that the effect does not require anyone to be recording. It only requires that recording be possible and impossible to detect. A street where one person in a hundred wears camera glasses is, from the bystander's point of view, a street where anyone might be. The chilling effect is a product of uncertainty, not of footage. You cannot opt out of uncertainty by checking for a light that may not be there.

There is a fairness issue, too. Smart glasses have genuine benefits, and the EDPB lists some of them, including audio assistance and hazard awareness for people with disabilities. Hamburg's proposed restrictions on wearing them in pools and public transport have reportedly included exemptions for people who rely on them for medical or accessibility reasons. Those uses deserve protection. But the benefits go to the wearer, while the costs of uncertainty fall on everybody else in the room. That asymmetry is the root of the burden problem.

Why Neither Wearer Nor Bystander Can Carry It

The second big question starts from a premise: that the burden of informing bystanders cannot realistically fall on either the wearer or the bystander. Is that premise right? The evidence says yes, for different reasons in each case.

Start with the wearer. The EDPB does say that users “should always ensure that they respect the privacy and dignity of other individuals,” and that explicitly telling people you are using smart glasses is relevant. That is good manners. As a system for protecting the public, it fails three times over. It fails on scale: you cannot announce yourself to every passer-by in a station. It fails on incentive: the CHI data shows wearers consistently willing to give less than bystanders want, and growing less concerned with use. And it fails catastrophically on bad actors. The man at the Frog Pond did not intend to tell anybody anything. Any safeguard that depends on the wearer's goodwill protects people only from wearers who were never the danger.

Now the bystander. The research community has proposed plenty of tools to help people defend themselves: gestures that signal “do not record me,” wearable markers that ask cameras to blur you, apps that negotiate consent. The CHI team had experts and experienced users evaluate twelve such privacy-enhancing technologies and found what they called a “fundamental paradox”: empowering bystanders “requires burdening them with responsibilities that should belong to those creating privacy risks.” Participants were blunter. One described gesture-based refusal as “transferring the burden to the victim.” Another asked, of awareness tools in general: “What's the point of awareness? I know, but can I stop it?”

The EDPB reaches the same conclusion from the legal side. Smartphone alerts that detect nearby glasses, it says, “should not be the sole method of providing transparency,” because that would exclude anyone without the app running and would make a stranger's privacy depend on their own technical diligence. Children, it adds, “may be less aware of the presence or implications of smart glasses and therefore more vulnerable.” A toddler at a spray pool cannot install an app.

So the premise holds. If the burden of informing bystanders cannot sit with the wearer or the bystander, it has to sit somewhere else. The EDPB's answer is the manufacturer and the AI service provider. The question is whether that answer actually solves the problem.

Does Moving the Burden to Manufacturers Fix Anything

Here is my position. Placing the burden on manufacturers is the right move and an incomplete one. As the EDPB has framed it, an obligation of transparency, it mostly identifies who is accountable for a harm that will keep happening. It becomes a real solution only to the extent that accountability is turned into hard constraints on what the product can do, and even then a residue of harm is unavoidable for as long as a camera can look like a pair of glasses.

Begin with what the shift does achieve, because it is not nothing. The manufacturer is the only party that can change the product. It decides whether the LED is bright or faint, whether the camera works when the light is broken, whether bystanders' faces are blurred on the device before anything reaches a server, whether footage is stored by default, whether face matching exists at all. The EDPB's recommendations are a menu of exactly those choices: on-device processing, blurring or avoiding processing of bystanders' identities, persistent storage disabled by default for real-time assistance, geofencing in sensitive places, and “privacy frictions” that slow down sharing. Every one of those is a design decision only a manufacturer can make. Assigning the obligation to the company that controls the design is not a technicality. It is the only assignment that matches responsibility to power.

The legitimate interest paragraph may matter even more. If commercial reuse and AI training on bystander data are “particularly difficult to demonstrate” as lawful, then the most valuable part of the footage, its afterlife as training material, becomes the most legally exposed. That attacks the incentive that makes always-on capture attractive in the first place. A company that cannot monetise the street has less reason to see it all.

Now the limits, and they are serious.

First, transparency is not protection. The EDPB's main ask is that bystanders be told. But telling someone they are being recorded does not stop them being recorded, which is precisely the point the CHI participant made. A brighter light turns covert surveillance into overt surveillance, which is an improvement, and then hands the bystander the same impossible choice: confront a stranger, leave, or put up with it. The CHI data suggests most people will avoid the camera, which is the chilling effect in a different form. Perfect transparency would make it clear who is watching. It would not make the street feel unwatched.

Second, manufacturer accountability does not reach the people causing the worst harm. The tampering market shows how quickly a hardware safeguard becomes a $60 problem. And a design rule binds only the companies a regulator can actually reach, which is not the same as every company whose products end up on European faces. The person who films children at a pool is not going to be deterred by a design standard. What the manufacturer framework catches is the mainstream product used by mainstream people, which is valuable but is not where the most serious incidents begin.

Third, the framework is preliminary guidance, not law. The statement interprets the GDPR; enforcement depends on national regulators bringing cases. And where legislators have tried to write hard rules, they have struggled. In California, State Senator Eloise Gómez Reyes's SB 1130 would have required wearable recording devices sold in the state to carry a prominent recording indicator from 2028, banned technology designed to conceal it, and penalised covert recording in private areas of businesses. It passed both chambers comfortably. On 30 September 2026, Governor Gavin Newsom vetoed it, saying its definition of a wearable recording device was too broad and noting that existing law already covers recording in private spaces. TechNet, a trade group whose members include Meta, Google and Amazon, had opposed it. When the EDPB adopted its statement two weeks later, it pointed to the bill in a footnote as an example of legislation under discussion elsewhere.

Fourth, and most fundamentally, there is the residue. Even a perfect manufacturer regime, with unmissable indicators, no tampering, no default storage, no face matching and no training on bystanders, still leaves a world in which some people wear cameras on their faces and everyone else has to think about it. The EDPB's paragraph 8 concedes this directly: people may object “even if the device could be identifiable as smart glasses,” simply because the possibility makes them uncomfortable. That discomfort is not a compliance failure. It is the cost of the capability existing in that form at all. No allocation of legal duties can remove it. It can only decide who answers for it.

So does placing the burden on the manufacturer merely identify who is accountable for an unavoidable harm? Partly, yes. Part of the harm is unavoidable, and we should stop pretending otherwise. But identifying the accountable party is how the avoidable part gets avoided. The honest framing is that the manufacturer burden is necessary but not sufficient, and its value depends entirely on whether regulators treat transparency as the end of the obligation or the beginning.

What would make it sufficient, or close to it? Two things the EDPB gestures at but does not yet demand. The first is substantive limits rather than just disclosure: no identification of strangers, no training on bystander footage without a lawful basis that genuinely exists, cameras that fail closed when tampered with. The second is protection that attaches to places rather than people. Buolamwini's proposal in Nature, “consent-conscious spaces” such as schools, hospitals, courts and swimming pools “in which covert recording is prohibited by law,” is one version. The EDPB's own list of sensitive contexts (medical facilities, schools, changing rooms, swimming pools, beaches, toilets) is another. Hamburg's proposals for pools and public transport are a third. Norway's is a fourth, and the broadest yet. On 5 October 2026 its government said it would ask parliament for a temporary ban on smart glasses in certain public places, starting with spaces where children gather, such as schools, kindergartens, playgrounds and youth clubs, and possibly extending to parks, beaches, museums, shopping centres, concerts and sports events. It is not a blanket ban: private use would still be allowed, and exemptions for vulnerable groups and “socially beneficial uses” are being considered. It is also, for now, only a proposal. But it is a government choosing to protect places rather than waiting for people to protect themselves. The CHI researchers reached the same place from the design side, recommending “automatic protection in sensitive contexts” rather than case-by-case negotiation.

Some of this is already happening without waiting for Brussels. In August 2026, HM Courts and Tribunals Service confirmed that people bringing Meta smart glasses into courts and tribunals in England and Wales “will have them confiscated on entry,” following a High Court case in which a litigant was found to have been coached through his glasses during cross-examination. New York's court system had already taken similar action. Spaces are starting to defend themselves, one door at a time.

The Street Worth Keeping

Go back to the Frog Pond. What protected the children there, to the extent anything did, was not an LED, a privacy notice or a design standard. It was other adults who noticed something wrong and said so. That is Jacobs's “eyes on the street” working as she described. It is also a reminder of how fragile it is. Nothing in the reports suggests his glasses announced anything; witnesses noticed what he was doing. Next time, the device will be smaller, the behaviour more careful, and the footage uploaded before anyone has time to object.

The EDPB statement is important because it refuses the comfortable fiction that the wearer can be trusted to manage this, or that the bystander can be expected to. It points to the only party with the power to change the product. Anu Talus put the principle simply in the board's announcement: “individuals deserve clear warnings, not hidden surprises.” That is right as far as it goes. But a warning is not a refusal, and a street full of clear warnings is still a street where people check themselves before they laugh too loudly or speak too freely.

The real question smart glasses put to us is not who should fill in the paperwork. It is which parts of public life we are willing to let become permanently recordable, and which we intend to keep for the glance that ends in looking away. Manufacturers can be made to answer for the first. The second is a choice for the rest of us, and it is one we are making right now, mostly by not making it.

---

References and Sources

1. European Data Protection Board. “Statement 1/2026 on Smart Glasses.” Adopted 13 October 2026. Signed by Anu Talus, Chair. 8 pp. https://www.edpb.europa.eu/system/files/2026-10/20261310_edpb_statement_smart-glasses_en.pdf
2. European Data Protection Board. “EDPB publishes statement on the data protection challenges of smart glasses.” Press release, Brussels, 13 October 2026. https://www.edpb.europa.eu/news/edpb-publishes-statement-on-the-data-protection-challenges-of-smart-glasses_en
3. Wang, Xueyang; Peng, Kewen; Yi, Xin; Li, Hewu. “Mind the Gap: Mapping Wearer-Bystander Privacy Tensions and Context-Adaptive Pathways for Camera Glasses.” Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26), Barcelona, 13 to 17 April 2026. ACM. DOI: 10.1145/3772318.3791848. Preprint: arXiv:2603.04930, 5 March 2026. https://arxiv.org/abs/2603.04930
4. Buolamwini, Joy. “Why smart glasses are about to create a major privacy crisis.” Nature 658, 296 (2026), published 6 October 2026. DOI: 10.1038/d41586-026-03131-x. https://www.nature.com/articles/d41586-026-03131-x
5. Fortier, Marc; Klein, Asher. “Sex offender recorded kids with camera sunglasses at Boston's Frog Pond, police say.” NBC Boston, 18 August 2026. https://www.nbcboston.com/news/local/sex-offender-secretly-recorded-kids-with-camera-sunglasses-at-bostons-frog-pond-police-say/3999571/
6. McKinley Becker, Kaitlin; Kwangwari, Munashe. “Sex offender who recorded kids at Boston's Frog Pond faces additional charges.” NBC Boston, 15 September 2026 (updated 18 September 2026). https://www.nbcboston.com/news/local/boston-frog-pond-sex-offender-mark-sullivan-court-appearance/4014933/
7. Boston.com. “Sex offender allegedly uses camera sunglasses to record children at Boston Common's Frog Pond.” 18 August 2026. https://www.boston.com/news/crime/2026/08/18/sex-offender-allegedly-uses-camera-sunglasses-to-record-children-at-boston-commons-frog-pond/
8. Omdia. “Global AI glasses shipments reach 8.7 million units with Mainland China emerging as the fastest-growing market.” Press release, 5 March 2026. https://omdia.tech.informa.com/pr/2026/mar/global-ai-glasses-shipments-reach-8point7-million-units-with-mainland-china-emerging-as-the-fastest-growing-market
9. Counterpoint Research. “Global AI Glasses Shipments Surge 263% YoY in H1 2026, with Display-less AI Glasses Accounting for 96% of Total.” 21 September 2026. https://counterpointresearch.com/en/insights/global-ai-glasses-shipments-surge-263-percent-yoy-with-display-less-ai-glasses
10. Cox, Joseph; Koebler, Jason. “A $60 Mod to Meta's Ray-Bans Disables Its Privacy-Protecting Recording Light.” 404 Media, 23 October 2025. https://www.404media.co/how-to-disable-meta-rayban-led-light/
11. Agence France-Presse. “French prosecutors probe secret filming of women with smart glasses.” The Star (Malaysia), 24 September 2026. https://www.thestar.com.my/tech/tech-news/2026/09/24/french-prosecutors-probe-secret-filming-of-women-with-smart-glasses (see also South China Morning Post, https://www.scmp.com/news/world/europe/article/3368459/france-prosecutors-probe-secret-filming-women-smart-glasses)
12. Silberling, Amanda. “If you own Ray-Ban Meta glasses, you should double-check your privacy settings.” TechCrunch, 30 April 2025. https://techcrunch.com/2025/04/30/if-you-own-ray-ban-meta-glasses-you-should-double-check-your-privacy-settings
13. Malik, Aisha. “Meta plans to add facial recognition to its smart glasses, report claims.” TechCrunch, 13 February 2026 (reporting The New York Times). https://techcrunch.com/2026/02/13/meta-plans-to-add-facial-recognition-to-its-smart-glasses-report-claims/
14. The Boston Globe. “AI and Meta smart glasses can turn a person's photo into personal info, Harvard students find.” 4 October 2024. https://www.bostonglobe.com/2024/10/04/business/harvard-students-ai-meta-glasses/
15. University of Sydney. “Smart glasses pose new harassment risks for women, researchers find.” News release, 11 August 2026. https://www.sydney.edu.au/news-opinion/news/2026/08/11/smart-glasses-pose-new-harassment-risks-for-women-research-meta.html
16. Futurism. “Man Wearing Smart Glasses Secretly Records Woman, Demands Money to Delete Video From His Socials” (reporting a BBC investigation). 9 May 2026. https://futurism.com/future-society/man-smart-glasses-records-woman-demands-money
17. Austin, Sophie (Associated Press). “Newsom vetoes measure that would have penalized the use of smart glasses to secretly record people.” LAist, 1 October 2026. https://laist.com/news/politics/newsom-vetoes-measure-that-would-have-penalized-the-use-of-smart-glasses-to-secretly-record-people
18. Kuo, Jordan. “California's Capture-Indicator Mandate Died on the Governor's Desk, Weeks Before Samsung's Glasses Are Due.” VR.org, 1 October 2026. https://vr.org/articles/california-sb-1130-veto-capture-indicator-mandate-2026
19. Legal Cheek. “Courts ban Meta glasses.” 13 August 2026. https://www.legalcheek.com/2026/08/courts-ban-meta-glasses/
20. The Next Web. “Meta patches the mod that turned its Ray-Ban glasses into secret spy cams.” 9 July 2026. https://thenextweb.com/news/meta-ray-ban-privacy-light-fix-cameraless
21. TechCentral.ie. “Hamburg wants to ban smart glasses in swimming pools and public transport.” 2026. https://www.techcentral.ie/hamburg-wants-to-ban-smart-glasses-in-swimming-pools-and-public-transport/
22. Penney, Jonathon. “Chilling Effects: Online Surveillance and Wikipedia Use.” Berkeley Technology Law Journal, Vol. 31, No. 1 (2016). SSRN: https://papers.ssrn.com/abstract=2769645
23. Goffman, Erving. Behavior in Public Places: Notes on the Social Organization of Gatherings. New York: The Free Press, 1963.
24. Jacobs, Jane. The Death and Life of Great American Cities. New York: Random House, 1961. See also NPR, “'Eyes On The Street' Details Jane Jacobs' Efforts To Put Cities First,” 28 September 2016. https://www.npr.org/2016/09/28/495615064/eyes-on-the-street-details-jane-jacobs-efforts-to-put-cities-first
25. Keaten, Jamey (Associated Press). “Norway seeks temporary ban on smart glasses as privacy concerns rise.” News4JAX, 5 October 2026. https://www.news4jax.com/tech/2026/10/05/norway-seeks-temporary-ban-on-smart-glasses-as-privacy-concerns-rise/ (see also TechSpot, https://www.techspot.com/news/114109-norway-wants-temporarily-ban-smart-glasses-parks-beaches.html)

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The instructions said “chew 4 tablets 2 times per day”. The product was a tube of topical medication.

That exchange sits in a 2026 research report from the American Foundation for the Blind (AFB), in the words of a survey participant who had pointed an AI visual description tool at a medication label and asked how to use it. “Luckily this was an obvious error,” the participant wrote, “but for now I will NOT be using AI for these tasks [and] will always confirm with a human.” Another participant in the same study used an AI tool to read tax forms that were not screen reader accessible, and reported that it “provided some very incorrect information that would have caused serious tax filing mistakes if I had not checked the information with a human.”

Neither story is a fringe horror. Phones now describe rooms in fluent paragraphs, apps read menus aloud and websites promise to fix themselves with a line of JavaScript. Underneath all that capability, a quieter question keeps surfacing from the people who depend on these tools: what happens when the thing that helps you is also the thing that makes things up?

The complaint has been circulating online for some time. One 2025 discussion on Reddit's r/ArtificialIntelligence forum is reported to have called the trend an “insidious form of ableism”, although that thread's exact wording could not be independently located for this article. The substance of the argument can, however, be tested against the record. As generative AI is positioned as a universal accessibility fix, the predictable tools that disabled and older people rely on risk being sidelined or replaced by systems that are more impressive and less trustworthy. If so, AI-first accessibility could widen the very gap it claims to close.

The evidence suggests that the worry is well founded, though not in the simple way a slogan implies. AI tools are delivering real independence to many blind and low vision people. The problem is not the tools. It is the temptation, among companies, website owners and even governments, to treat them as a substitute for the slow, unglamorous work of building things that are accessible in the first place.

Why a Mangled Word Beats a Fluent Fiction

A traditional screen reader is, at its core, a deterministic machine. It takes text that already exists in a document, an interface or a web page, and converts it into synthetic speech or braille. It can absolutely get things wrong. It may mispronounce a surname, stumble over an acronym or announce an unlabelled control as nothing more than “button”. But its errors have a particular shape. They are consistent: the same text produces the same mistake every time. And the underlying text is still there. A user who hears a garbled word can move through it character by character, check the spelling and work out what was meant. The error sits on top of a reliable source.

Generative AI does not work like that. A vision language model asked to describe a photograph or read a label is not converting existing text; it is producing a statistically plausible account of what it thinks it sees. When it fails, it does not usually fail with noise. It fails with fluency. A wrong instruction arrives in a grammatically perfect sentence, delivered with the same confidence as a right one. There is no underlying string to inspect, because the model's output is the only version of reality the user has been given.

Alex Katsarakes, a digital accessibility resident at AFB and a PhD student in human factors psychology, set this out in an October 2025 essay on alt text in the age of AI. Generative models, she wrote, “are prone to 'hallucination,' confidently asserting details that aren't actually present.” When a blind user cannot independently check an image, she argued, an inaccurate caption is not a cosmetic flaw. In her words, “it's a breakdown in trust.”

AFB's own survey data makes the point empirically. In its report The AI Quagmire, based on a survey of 1,735 US adults conducted between 1 July and 15 October 2025, 91 disabled users of AI visual description tools reported mistakes that harmed them or made their lives harder. When the researchers compared how disabled and nondisabled people rated the accuracy of AI output, they found a revealing asymmetry. Disabled users of visual description tools were more likely than nondisabled users to rate descriptions as “mostly accurate” (56.3 per cent against 46.4 per cent). With AI captions, the pattern flipped: disabled caption users, including deaf and hard of hearing people, were more sceptical than nondisabled users. AFB's interpretation is blunt: “This pattern suggests that disabled people are better able to detect captioning errors than visual description errors.”

Read that slowly. The people who most need image descriptions are, by definition, the people least able to verify them. A deaf user watching a speaker's lips and body language has some independent signal to check a caption against. A blind user listening to a description of a photograph often has none. Reliability is not one desirable feature among many in this context. It is the whole product.

What the Tools Genuinely Do Well

None of this means AI accessibility tools are a con. Pretending otherwise would be its own kind of disrespect to the many disabled people who use them every day and say they are better off.

Be My Eyes is the clearest example. The Danish company built its reputation on connecting blind and low vision people with sighted volunteers over live video. In 2023 it added Be My AI, an image description feature built on OpenAI's GPT-4 model. On 12 March 2026, the company announced that it had passed one million blind and low vision users and ten million volunteers across more than 150 countries. Its chief executive, Mike Buckley, called the milestone “a powerful reminder that accessibility is not a niche issue” and framed it as a global movement. Crucially, the human route has not been switched off: users can still tap a button and reach a sighted volunteer by live video.

Microsoft's Seeing AI, which began as a research project and launched on iPhone in 2017, made the jump to Android in December 2023. Saqib Shaikh, the founder and lead of the project, described it in Microsoft's announcement as “a free app that narrates the world for blind and low vision people”. The Android release brought generative features first tested on iOS, including richer scene descriptions and the ability to ask questions about a scanned document, such as the items on a menu or the total on a receipt.

Academic research confirms that the appeal is real. In a study presented at the ACM SIGACCESS ASSETS conference in 2024, Rudaiba Adnin and Maitraye Das of Northeastern University interviewed 19 blind people about their use of generative AI tools. The paper's title borrows one participant's description of the technology: “I look at it as the king of knowledge”. Participants used chatbots for writing emails and CVs, researching products and asking health questions, and they valued Be My AI's descriptions as more thorough than what sighted helpers usually offered.

AFB itself is not anti-AI. When it released its expert consensus report Empowering or Excluding in February 2025, Stephanie Enyart, the organisation's chief public policy and research officer, said the experts consulted had “shined a light on how AI can be a catalyst for access and inclusion.” Katsarakes, in the same essay that warns about hallucination, acknowledges that using tools such as Be My Eyes or Seeing AI “demonstrates clear potential for enhancing user autonomy.”

So the question is not whether these tools help. They do. The question is what happens when their help is treated as finished accessibility rather than as one layer within it.

The Verification Burden Falls on the User

What the research shows most consistently is that blind people who use AI description tools have become their own quality assurance department.

In a 2024 ASSETS paper titled Misfitting With AI, University of Michigan researchers Rahaf Alharbi, Pa Lor, Jaylin Herskovitz, Sarita Schoenebeck and Robin N. Brewer conducted a qualitative study with 26 blind people about how they verify and contest errors from AI visual assistance apps. They documented failures with complex document layouts, with languages other than English and with cultural artefacts. Participants had developed a repertoire of checks: testing the apps on familiar objects in low-stakes settings, drawing on non-visual knowledge, strategically involving sighted people and cross-referencing results between different apps and devices. The authors drew on the disability studies concept of “misfitting” to argue that these tools often carry assumptions that fail disabled users, and that disabled ways of knowing deserve recognition rather than replacement.

A diary study presented at CHI 2024 by Ricardo Gonzalez Penuela, Jazmin Collins, Shiri Azenkot and Cynthia Bennett tracked 16 blind and low vision participants using an AI scene description prototype for two weeks. The ratings were sobering. Average satisfaction came in at 2.76 out of 5. Average trust in the descriptions was 2.43 out of 4. The authors concluded that AI descriptions still needed significant improvement before they could deliver experiences that were both satisfying and trustworthy.

A 2025 CHI paper by Jingyi Xie, John M. Carroll and colleagues at Pennsylvania State University, which interviewed 14 visually impaired Be My AI users, adds texture. Four of its participants reported that Be My AI had described objects that were not there; one was told about an object behind her that did not exist. Eight reported inaccurate labels for gender or age. Several disliked the tool's habit of interpreting emotions and offering opinions. “Maybe I don't want it to editorialize, you know, maybe I just literally only want the facts of it,” one participant told the researchers. For live navigation, participants preferred human help over static photo descriptions, and they stressed that orientation and mobility skills remained essential. The tool, in their view, should augment those skills rather than replace them.

Chancey Fleet, the assistive technology coordinator at the New York Public Library's Andrew Heiskell Braille and Talking Book Library, has described a subtler cost. Speaking on the Electronic Frontier Foundation's How to Fix the Internet podcast in May 2024, she catalogued the house style of machine descriptions: “Any scene that contains a child or a dog is heartwarming.” “Any sunset or sunrise is vibrant.” “Anything with a couch and a lamp is calm or cozy.” The flattening matters, she argued, because adequacy discourages improvement: “if you are presented with what seems on a quick read, like good enough alt text, you're unlikely to do much labor to make it better, more nuanced, or more complete.” She also put the stakes plainly. When she asks an AI a question about her own world, “I'm going to have to live with the consequences of trusting that answer”.

Put these studies together and a pattern emerges. The capability is shared with everyone; the risk is carried by the individual. Each time a developer, publisher or institution decides that AI description is good enough, the job of verification is quietly transferred from the organisation that produced the content to the disabled person consuming it.

Overlays and the Business of Automated Compliance

Accessibility overlays are third-party scripts that sit on top of a website and attempt to detect and repair barriers automatically, often marketed with the language of artificial intelligence. They promise compliance with the Web Content Accessibility Guidelines (WCAG), the international technical standard, without anyone having to rebuild the site. Disabled users and accessibility professionals have been sceptical for years. In a January 2021 survey of 758 web accessibility practitioners by WebAIM, a non-profit based at Utah State University, 67 per cent rated overlays, plugins and widgets as “not at all or not very effective”. Among respondents with disabilities, the figure rose to 72 per cent, with only 2.4 per cent rating them very effective.

In January 2025, the US Federal Trade Commission turned that scepticism into enforcement. On 3 January it announced a proposed order requiring accessiBe, a company selling an AI-powered accessibility plug-in called accessWidget, to pay $1 million. The FTC alleged that accessiBe had misrepresented the tool's ability to make any website WCAG-compliant, and that it had formatted third-party articles and reviews to look like independent opinions without disclosing its material connections to them. The Commission voted 5-0 to accept the proposed consent order. Samuel Levine, then director of the FTC's Bureau of Consumer Protection, said: “Overstating a product's AI or other capabilities without adequate evidence is deceptive, and the FTC will act to stop it.” TechCrunch, reporting the action, noted that customers had also brought class action suits claiming the products did not work. The order was approved as final in April 2025.

AFB's Alexis Hubbard, writing in May 2025 under the title Missing a Human Touch, summed up the problem with automated fixes: “AI-powered accessibility overlays are just a band-aid solution for a bigger problem.” Because they do not change the source code of a site, she argued, they leave the underlying barriers intact. Her conclusion is worth quoting in full: “The only way to achieve an accessible and usable experience for all people is through people.”

Overlays show what AI-first accessibility looks like when it is driven by procurement rather than by users. A website owner buys the promise of compliance; a disabled visitor inherits the consequences, handed an interpretation instead of the thing itself.

When Governments Lean on the Promise of AI

The most striking recent example of AI's shortcomings being cited as a reason to delay accessibility comes not from a start-up but from the US Department of Justice.

In April 2024, the DOJ finalised a rule under Title II of the Americans with Disabilities Act requiring state and local governments to make their websites and mobile apps conform to WCAG 2.1 Level AA. Larger public entities, those serving populations of 50,000 or more, were originally due to comply in April 2026. Four days before that deadline, on 20 April 2026, the department published an interim final rule pushing compliance back by a year, to 26 April 2027 for larger entities and to 26 April 2028 for smaller ones and special districts. It took effect immediately, with public comments accepted until 22 June 2026. The technical standard itself was left unchanged.

Among the department's stated reasons was a striking admission about technology. “Advanced technology, such as generative AI, does not yet reliably automate the remediation of inaccessible content at scale,” the interim final rule says, “and staff resources and availability continue to pose significant challenges.” The rule also summarised correspondence from a member of Congress who argued that current technology, “including generative AI,” cannot reliably automate the remediation of complex science, technology, engineering and mathematics material at scale, and that human oversight is required.

It is an honest admission that AI cannot do the job alone. It also shows how the promise of automation can reshape policy. When the expected technological help did not arrive, the cost of the shortfall fell on disabled residents, who now wait at least another year for accessible public services.

Disability organisations were furious. Maria Town, president and chief executive of the American Association of People with Disabilities, called the delay “a profound disappointment and a serious setback for the disability community”, adding that “Years of notice have not been enough” and that the department was “rewarding inaction with more time.”

AFB's formal comments, submitted on 22 June 2026 and signed by Enyart, tackled the AI argument directly. “The absence of a fully automated accessibility solution does not justify extending the rule,” AFB wrote. It pointed out that generative AI can itself produce inaccessible content “because it is based on training data that contains accessibility barriers,” and that “Entities should not over rely on AI to produce content.” It also argued, drawing on its own research, that “AI adoption and effective use of AI is dependent on the accessibility of the platform.” In plain terms: “It doesn't matter how useful an AI-enabled service or tool is if the platform is inaccessible to blind users.”

That last line is the hinge of the entire debate. An AI tool is not a ramp into an inaccessible building. It is another door, and it only works if the corridor behind it has been built properly. AFB's comments also offered a constructive model: AI “can generate a passable first attempt at creating captions that a human can then edit.” Used that way, AI speeds up accessible design rather than replacing it.

AFB's research quantifies what is at stake. In a 2024 study cited in those comments, participants spent 55 per cent of their total task time dealing with digital barriers, and only 45 per cent on the task itself.

Across the Atlantic, the direction of travel is different. The European Accessibility Act, Directive (EU) 2019/882, began applying on 28 June 2025 to products and services including computers, smartphones, ATMs, e-books, e-commerce and banking. As Fredericka Argent of the law firm Covington & Burling noted that month, it reaches companies trading in the EU wherever they are based. The EAA is not an AI law, but it reinforces a principle that AI-first thinking tends to erode: the obligation sits with whoever builds and sells the product, not with the disabled person left to find a workaround.

Older Adults and the Gap Between Awareness and Use

The same dynamics extend to older adults, a group that overlaps heavily with disabled people and is often presented as a natural market for AI-powered help.

The latest AARP research, Tech Use and Adoption Keeps Surging Among Older Adults, written by Brittne Kakulla and published in December 2025, surveyed 3,838 US adults aged 50 and over between 9 September and 6 October 2025. It found that AI use in that age group had risen from 18 per cent in 2024 to 30 per cent in 2025. About half used or were interested in voice assistants such as Siri or Alexa. Two-thirds agreed that technology enriches life and makes ageing easier. Among adults aged 80 and over, agreement that technology enables a healthy life rose from 39 per cent to 46 per cent.

The same report records friction. For AI platforms such as ChatGPT or Copilot, 19 per cent used them, 21 per cent were interested and 59 per cent had no interest. Data privacy was the top barrier to adoption. Trust, privacy and data security were the top concerns for roughly half of the older adults who hesitated to adopt new technology, especially AI. And, in a line that could serve as the epigraph for this entire debate: “Three in 5 adults 50-plus say technology is not designed with their age in mind.”

Work from the MIT AgeLab sharpens the picture. A national survey of 1,008 American adults by Lauren Cerino, Joseph Coughlin and colleagues, published as a preliminary abstract in Innovation in Aging in December 2025, found that 73.7 per cent of respondents aged 60 and over were at least “a little” familiar with AI-enabled technologies. Only 28.3 per cent had used them. Some 34.7 per cent said they did not trust AI tools at all to give good advice, and trust scores declined with age.

Awareness, then, is not the main problem. Older people have heard of AI. The gap lies between hearing about a tool and being able, willing and confident enough to rely on it.

Research at Carnegie Mellon University points to why. In work presented at the ACM CHI 2026 conference in Barcelona and reported by the university in May 2026, Jodi Forlizzi, the Herbert A. Simon Professor in Computer Science and Human-Computer Interaction, and her colleagues found that older adults who use AI want it to recognise when they need plain information and when they need reassurance. A separate study based on 22 interviews identified shared values among older adults, including independence, curiosity, connection and continuity, expressed differently depending on where people lived. Older adults were open to AI when its purpose was clearly explained, especially for reminders, alerts and support at home. “There's always a mismatch between people's perceived abilities and what they can actually do,” Forlizzi said. “AI technology can be used to close this gap.”

That is the optimistic reading, and it deserves to be taken seriously. But it has a mirror image. If AI tools must be carefully explained and set up to earn trust, the people most likely to benefit, including those living alone or without technically confident relatives, are also the least likely to receive that support. AFB found that almost all its participants had learned to use AI on their own; only 9 per cent learned through an employer or online course. More disabled than nondisabled participants reported barriers in learning, most commonly inaccessible videos, hard-to-understand material and training that required a mouse.

The Northeastern study found similar problems in the AI products themselves. Blind participants encountered unlabelled buttons, missing headings and landmarks, no notification when a chatbot had finished responding, and clutter from adverts and sample prompts. The tools being promoted as accessibility solutions were not consistently accessible.

The Quiet Retirement of Predictable Tools

The sharpest version of the complaint concerns replacement: established, predictable tools being retired in favour of generative ones. In 2026, the most visible example is the end of Google Assistant on Android.

Google had originally planned to replace Assistant with Gemini, its generative AI assistant, during 2025. According to reporting by Matthias Bastian at The Decoder in August 2026, Google told users it would begin shutting Assistant down on Android and Wear OS from 4 September 2026, covering phones, tablets, watches, headphones and Android Auto. Once the rollout reaches a device, there is no way to switch back. Bastian raised exactly the concern at the centre of this article: that a probability-based language model may not match the reliability of a deterministic assistant for simple tasks such as smart home control and basic phone commands.

By late September, Mark Jansen at Android Police reported that the option to switch from Gemini back to Assistant had disappeared on the devices he checked, including a Pixel 10 Pro, and that smart home control was an area where Assistant still beat its replacement.

For most users, that is an inconvenience. For someone who cannot easily use a touchscreen, whether because of sight loss, limited dexterity or tremor, a voice assistant is not a convenience feature. It is how they turn on the lights, set a medication reminder or call for help. AFB's survey speaks to the stakes. Among users of voice-activated AI, 121 disabled participants described harms from mistakes, including texts sent with the wrong words and calls placed to unintended contacts, among them emergency services. “For many users, particularly disabled users, features such as voice dictation for text messaging are essential accessibility supports,” the report observes. “Failures or changes in these systems can result in concrete, disruptive, and sometimes harmful consequences.”

There is no evidence that Google set out to disadvantage disabled users, and Gemini brings generative capabilities that Assistant never had. The issue is structural. When a company replaces a predictable tool with a more capable but less predictable one, and removes the option to go back, the people with the fewest alternatives absorb the most risk. Capability is a fair trade for those who can fall back on other methods. Reliability is the whole point for those who cannot.

Is AI-First Accessibility Widening the Gap?

So, to the central question: if the people who would benefit most cannot find, set up or trust these tools, is AI-first accessibility widening the gap it claims to close?

The honest answer is that it depends on what “AI-first” means in practice.

Where AI is added on top of accessible foundations, the gap narrows. A blind person on a properly coded website who also has Be My AI or Seeing AI in their pocket has more independence than ever before. A deaf student who receives AI-generated captions that a human has corrected gets faster access than one who waits for manual transcription. These are genuine gains, and the AFB data confirms that most AI users say AI makes their lives easier.

Where AI replaces accessible foundations, the gap widens, and it widens along a predictable gradient. The confident, well-supported, technically fluent disabled user can verify outputs, switch between apps, ask a sighted friend and route around errors, exactly as the participants in the Michigan study described. The isolated, older, less confident or cognitively disabled user often cannot. The more a system depends on verification skills, the more it rewards people who already have them.

The evidence points to four ways in which substitution widens the divide. It shifts responsibility, transferring the work of accessibility from organisations to users who must detect errors they may have no way of seeing. It erodes incentives: as Fleet observed, good-enough descriptions discourage better ones, and the DOJ's interim rule shows the institutional version. It removes fallbacks, as predictable tools are retired. And it presupposes access to the tools themselves, when AFB and the Northeastern researchers both found that AI interfaces and training are often inaccessible.

None of this makes AI the villain. It makes the substitution the problem.

What Reliable Accessibility Would Require

If AI is going to close the accessibility gap rather than reshape it, the evidence points towards a set of concrete commitments.

Treat AI as a layer, never a foundation. Accessible design, meaning semantic code, human-authored alt text for meaningful images, accessible documents and keyboard-operable interfaces, must remain the baseline. AI description and remediation should sit on top of it. Katsarakes put it in a single sentence: “Let AI suggest or assist, but don't let it replace human intent.”

Keep the deterministic path available. When companies introduce generative assistants, they should keep predictable modes for core tasks such as timers, calls, reminders and device control, and they should not remove the option to revert until the replacement has been shown to match the original's reliability for disabled users specifically. A one-way migration is a design choice, not a technical necessity.

Build verification into the product. The Michigan and Penn State studies show users building their own verification systems from scratch. Products should do it for them: signalling uncertainty, separating what the model read from what it inferred, offering a factual mode without editorial colour, and making escalation to a human easy. Be My Eyes shows that a live human route can sit alongside AI.

Draw hard lines around high-stakes tasks. Medication instructions, financial documents, legal forms and navigation are not areas where a probabilistic description should ever be the only source. AFB's recommendations call for developers to “Improve the accuracy of AI outputs and tell users how accurate those outputs are.” Where accuracy cannot be assured, the interface should say so plainly.

Hold vendors to their claims. The FTC's accessiBe order established that overstating AI accessibility capabilities without evidence is deceptive. Regulators elsewhere should apply the same standard, and procurement teams should demand evidence of real-world performance with disabled users, not marketing claims of automated compliance.

Do not let automation set the deadline. Legal accessibility obligations should be pegged to what disabled people need, not to the pace of AI development. The European Accessibility Act's focus on the duties of manufacturers and service providers is a better model than one that treats immature automation as grounds for delay.

Fund support, not just software. Older adults are aware of AI but hesitant to use it, and CMU's research suggests they engage when the purpose is clearly explained. That explanation has to come from libraries, disability organisations and accessible community training. AFB's finding that only 9 per cent of AI users learned through formal routes suggests a large and fixable gap.

Include disabled people in testing and design. AFB's expert consensus report called for AI to focus “on expanding access and inclusion, not just on avoiding harm.” That requires disabled people in the room when products are designed, tested and retired, not consulted after launch.

What Is Lost When Reliability Goes

So what is lost when reliability is sacrificed for capability?

Part of the answer is practical: time, money, safety, and the occasional tube of medication with the wrong instructions. Participants in AFB's 2024 study already spent more than half their task time fighting digital barriers.

But the deeper loss is harder to measure. Independence is not just the ability to do something without help. It is the ability to rely on what you are told. A screen reader that occasionally mispronounces a name still gives its user a stable relationship with the text. A description tool that sometimes invents an object behind you, or turns a cream into tablets, asks its user to hold every answer at a slight distance. That is a different kind of life: one in which assistance is always provisional and confidence always has to be earned again.

Generative AI has given many disabled people extraordinary new reach, and nobody should want to take that away. The challenge for the companies building these tools, and the governments regulating them, is to add that reach without removing the floor. AI can help the people who build accessible things work faster and reach further. It cannot be allowed to replace the obligation they carry.

---

References and Sources

1. American Foundation for the Blind. “The AI Quagmire: Benefits, Risks, and User Aspirations Through a Disability Lens.” AFB Artificial Intelligence Series, 2026. Sections “Executive Summary” and “Harms of AI in Various Contexts.” https://afb.org/research-and-initiatives/ai-series/ai-quagmire/executive-summary and https://afb.org/research-and-initiatives/ai-series/ai-quagmire/harms-ai-various-contexts
2. Katsarakes, Alex. “Beyond Alt Text: Rethinking Visual Description in the Age of AI.” American Foundation for the Blind blog, 16 October 2025. https://afb.org/blog/entry/alt-text-age-ai
3. Hubbard, Alexis. “Missing a Human Touch: Why Automated Testing and AI Don't 'Solve' Digital Accessibility… Yet.” American Foundation for the Blind blog, 13 May 2025. https://afb.org/blog/entry/missing-human-touch-1
4. Enyart, Stephanie, American Foundation for the Blind. “June 2026 Comments on DOJ Extension of Web Accessibility Compliance Dates” (RIN 1190-AA82, Docket CRT150). 22 June 2026. https://afb.org/research-and-initiatives/public-policy/advocacy-letters-comments-statements/comments/june-2026-doj-extension-web-compliance
5. American Foundation for the Blind. “AFB Report Spotlights Impact of AI for Disabled People” (press release on Empowering or Excluding: Expert Insights on Inclusive Artificial Intelligence for People with Disabilities). 6 February 2025. https://afb.org/news-publications/press-room/press-release-archive/press-release-2025/afb-report-spotlights-impact-AI-disabled-people
6. US Department of Justice, Civil Rights Division. “Extension of Compliance Dates for Nondiscrimination on the Basis of Disability; Accessibility of Web Information and Services of State and Local Government Entities.” Interim final rule, Federal Register Vol. 91, No. 75, 20 April 2026. https://www.federalregister.gov/documents/2026/04/20/2026-07663/extension-of-compliance-dates-for-nondiscrimination-on-the-basis-of-disability-accessibility-of-web (PDF: https://www.ada.gov/assets/pdfs/2026-ifr.pdf)
7. Duane Morris LLP. “DOJ Extends ADA Title II Digital Accessibility Deadlines by One Year.” Client alert, 23 April 2026. https://www.duanemorris.com/alerts/doj_extends_ada_title_ii_digital_accessibility_deadlines_one_year_0426.html
8. American Association of People with Disabilities. “AAPD Statement on DOJ Title II Web Rule Interim Final Rule.” 20 April 2026. https://www.aapd.com/aapd-statement-title-ii-doj-web-rule-ifr/
9. Kakulla, Brittne. “Tech Use and Adoption Keeps Surging Among Older Adults” (2026 Tech Trends). AARP Research, December 2025. https://www.aarp.org/pri/topics/technology/internet-media-devices/2026-technology-trends-older-adults/
10. Cerino, Lauren; Felts, Adam; Balmuth, Alexa; Brennan, Taylor; Lee, Chaiwoo; Coughlin, Joseph. “Older Adults' Perspectives on Artificial Intelligence as a Source of Advice.” Innovation in Aging, 9 (Suppl 2), igaf122.1371, 31 December 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12761853/
11. Sapio, Amanda. “How Do Boomers Really Feel About AI?” Carnegie Mellon University School of Computer Science News, 28 May 2026. https://www.cs.cmu.edu/news/2026/older-adults-ai
12. US Federal Trade Commission. “FTC Order Requires Online Marketer to Pay $1 Million for Deceptive Claims that its AI Product Could Make Websites Compliant with Accessibility Guidelines.” Press release, 3 January 2025. https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-order-requires-online-marketer-pay-1-million-deceptive-claims-its-ai-product-could-make-websites-compliant
13. Wiggers, Kyle. “FTC orders AI accessibility startup accessiBe to pay $1M for misleading advertising.” TechCrunch, 3 January 2025. https://techcrunch.com/2025/01/03/ftc-orders-ai-accessibility-startup-accessibe-to-pay-1m-for-misleading-advertising/
14. WebAIM. “Survey of Web Accessibility Practitioners #3 Results.” January 2021. https://webaim.org/projects/practitionersurvey3/
15. Alharbi, Rahaf; Lor, Pa; Herskovitz, Jaylin; Schoenebeck, Sarita; Brewer, Robin N. “Misfitting With AI: How Blind People Verify and Contest AI Errors.” Proceedings of ASSETS '24, ACM, 2024. DOI: 10.1145/3663548.3675659. https://arxiv.org/abs/2408.06546
16. Gonzalez Penuela, Ricardo E.; Collins, Jazmin; Azenkot, Shiri; Bennett, Cynthia. “Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People.” Proceedings of CHI '24, ACM, 2024. https://arxiv.org/abs/2403.15604
17. Adnin, Rudaiba; Das, Maitraye. “'I look at it as the king of knowledge': How Blind People Use and Understand Generative AI Tools.” Proceedings of ASSETS '24, ACM, 2024. DOI: 10.1145/3663548.3675631. https://maitraye.github.io/files/papers/BLV_GenAI_ASSETS24.pdf
18. Xie, Jingyi; Yu, Rui; Zhang, He; Billah, Syed Masum; Lee, Sooyeon; Carroll, John M. “Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models.” Proceedings of CHI '25, ACM, 25 April 2025. DOI: 10.1145/3706598.3714210. https://pmc.ncbi.nlm.nih.gov/articles/PMC12338113/
19. Cohn, Cindy; Kelley, Jason (hosts), with Chancey Fleet. “Podcast Episode: Building a Tactile Internet.” Electronic Frontier Foundation, How to Fix the Internet, 7 May 2024. https://www.eff.org/deeplinks/2024/05/podcast-episode-building-tactile-internet
20. Be My Eyes. “Be My Eyes Reaches 1 Million Blind and Low-Vision Users and 10 Million Volunteers.” 12 March 2026. https://www.bemyeyes.com/news/be-my-eyes-reaches-1-million-blind-and-low-vision-users-and-10-million-volunteers/
21. Shaikh, Saqib. “Seeing AI App Launches on Android, Including New and Updated Features and New Languages.” Microsoft Accessibility Blog, 4 December 2023. https://blogs.microsoft.com/accessibility/seeing-ai-app-launches-on-android-including-new-and-updated-features-and-new-languages/
22. Bastian, Matthias. “Google will shut down Google Assistant starting September 2026 as Gemini takes over on Android and Wear OS.” The Decoder, 5 August 2026. https://the-decoder.com/google-will-shut-down-google-assistant-starting-september-2026-as-gemini-takes-over-on-android-and-wear-os/
23. Jansen, Mark. “Google Assistant is officially gone, and people aren't happy about it.” Android Police, 29 September 2026. https://www.androidpolice.com/the-google-assistant-kill-off-has-arrived/
24. Argent, Fredericka. “European Accessibility Act: June 2025 Deadline Has Arrived.” Covington & Burling, Inside Global Tech, 10 June 2025. https://www.insideglobaltech.com/2025/06/10/european-accessibility-act-june-2025-deadline-has-arrived/ (see also European Commission, “European Accessibility Act,” https://commission.europa.eu/strategy-and-policy/policies/justice-and-fundamental-rights/disability/union-equality-strategy-rights-persons-disabilities-2021-2030/european-accessibility-act_en)

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A red car rolls towards a junction. A pedestrian steps into the road. The car turns, hits the pedestrian, who goes down, gets up and walks over to the driver. The whole thing lasts 25 seconds, is rendered in the engine of Grand Theft Auto V, and is about as simple as evidence gets. One thing happens in it, and that thing is a collision.

When researchers asked two consumer AI systems to write factual, neutral summaries of that clip and a near-identical twin, 19 of the 20 summaries they got back failed to mention that the car hit the pedestrian.

That finding comes from a paper posted to arXiv on 23 September 2026, “AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory”, by Mattea Sim of Georgetown University, Yael Eiger of the University of Washington and Tadayoshi Kohno of Georgetown. It was picked up over the following days by The Independent, Neuroscience News and a string of science outlets, and it is due to be presented at the AAAI/ACM Conference on AI, Ethics and Society in October. The headline numbers travelled fast. The machines were bad at summarising. And when people read a summary that got a detail wrong, their memory of what they had seen with their own eyes bent to fit the text.

That second finding is the one worth sitting with. Copilot writes up the meeting you sat through. Your phone condenses the news alerts you swiped past. Police officers in several American states file reports drafted by software from body camera audio. Each tool is sold as a memory aid. The study suggests that when the aid is wrong, it does not merely fail to help. It can overwrite.

So what happens to a person's own memory when the machine's recap of their experience is wrong? And when a summary quietly edits the past someone believes they lived, who is supposed to catch it: the reader, the company that sold the recap, or, as things currently stand, more or less nobody?

Twenty Summaries and One Missing Collision

The researchers used two short animated clips that already had a history in memory research. Both show a red saloon approaching an intersection and striking a pedestrian. The only meaningful difference is the road sign at the junction: in one video it is a stop sign, in the other a yield sign (the American equivalent of a give way sign). The team fed each clip to two models, OpenAI's ChatGPT-5.5 and Google's Gemini 2.5 Flash-Lite, five times per model per video, using a single standardised prompt. The instruction asked for a factual and neutral summary of at least 300 words, with no speculation and no invented facts. That gave them 20 summaries to analyse.

Every one of the 20 contained mistakes, somewhere between seven and 21 per summary. All of them left things out. According to the paper, 65 per cent contained what the authors classed as critical inaccuracies, 90 per cent contained non-critical inaccuracies, and 60 per cent added material that simply was not there. The summaries hallucinated extra pedestrians and extra vehicles and described erratic driving the video did not show. One stated that the pedestrian slightly repositioned but did not enter the roadway, when the pedestrian plainly walks into the road.

Averaged across all prompts and both models, the summaries omitted 51.6 per cent of the central details of the event. And in 95 per cent of cases, the summary did not mention the central detail at all: the car colliding with the pedestrian.

Summaries are supposed to be lossy, but their entire job is to keep what matters and drop what does not. These models did the reverse, producing long, fluent paragraphs about a scene in which the one thing that happened did not happen.

“I was struck by how bad the summaries were, even at this stage in AI development,” Eiger said in the coverage published by Neuroscience News.

Some caution is due. This was a small corpus: two short animated clips, two models, one prompt, 20 outputs. The authors say so plainly in their limitations section, acknowledging that one standardised prompt and a modest number of summaries limit how far the results generalise. Models change month to month, so nobody should read these figures as a stable error rate for all AI video summarisation. What they do show is that two widely available consumer systems, given a clear and simple task, were confidently and substantially wrong about the most important fact in front of them.

Fifty Years of Stop Signs and Yield Signs

The second half of the study is where the story turns from software quality to human cognition, and where it plugs directly into one of the most studied findings in psychology.

In 1974, Elizabeth Loftus, then at the University of Washington, and her colleague John Palmer showed people films of car accidents and asked them how fast the cars were going. The trick was in the verb. Asked how fast the cars were going when they “smashed” into each other, participants gave higher estimates of speed than those asked about cars that “collided”, “bumped”, “hit” or “contacted”. A week later, people who had heard “smashed” were more likely to say yes when asked whether they had seen broken glass. There was no broken glass in the film. The paper became one of the founding texts of what is now called the misinformation effect.

Four years later, Loftus, David Miller and Helen Burns published a series of experiments in the Journal of Experimental Psychology that will look very familiar to anyone who has just read the Georgetown and Washington paper. Across five experiments and a pilot, 1,242 participants watched a sequence of slides showing an accident between a car and a pedestrian at a junction. Afterwards some were fed information consistent with what they had seen, and some were fed misleading information: a stop sign became a yield sign, or the other way round. Misleading information made people less accurate on later recognition tests, and it had a larger effect when introduced just before the test than immediately after the event. The authors concluded that information a witness encounters after an event, consistent or misleading, gets integrated into the witness's memory of it.

The new study is, in effect, that experiment rerun for the age of the chatbot, split into two sessions between 24 and 48 hours apart. In the first, people watched one of the two animated clips. In the second, they read a summary of what they had seen, then answered a 12-item recognition test in which each question offered two choices. The critical question asked whether the car had approached a stop sign or a yield sign.

Here is a crucial design detail that some coverage skated past. The misleading summaries were not raw machine errors. The researchers took AI-generated summaries and edited them, removing sentences describing the details that differed between the two clips and splicing in AI-generated text from other outputs, so that participants who had seen a stop sign read about a yield sign and vice versa. In other words, the memory experiment tested what happens when an AI-style summary contains a specific, plausible error. It did not test whether the particular errors the models spontaneously made would distort memory in the same way.

That caveat matters, but it cuts less deeply than it seems. The first half of the study showed that errors of exactly this kind are routine. The second half showed what they do once they land. Of the 331 participants who completed both sessions (328 after attention checks), those who read a summary consistent with the video answered the critical sign question correctly 83.6 per cent of the time. Those who read a misleading summary got it right 44.8 per cent of the time. On a two-choice question, that is worse than a coin toss. The statistics were not subtle: the probability of a gap that size arising by chance was below one in a thousand.

Meanwhile participants remembered other details well. On a non-critical question about whether the scene took place in daytime or at night, accuracy was 96.6 per cent. People had not forgotten the video. They had swapped one piece of it for the version they had been told.

“AI is a new method of delivering misinformation, and it has the potential to create these false memories for people who are reading that information,” Sim said in the Georgetown announcement.

Why Knowing It Was a Machine Did Not Help

Participants were randomly told that the summary they read had been written either by an AI or by a human. In reality all of the summaries were AI-generated. If a machine label worked as a kind of cognitive inoculation, you would expect people who knew they were reading AI output to discount its claims and lean more on their own memory.

They did not. Among those who read a misleading summary labelled as AI-generated, 46.3 per cent answered the critical question correctly. Among those who read the same kind of summary labelled as human-written, 43.4 per cent did. The difference was not statistically significant. The researchers also measured participants' trust in AI and how much they used it, and neither made a meaningful difference to how susceptible they were.

This is consistent with what the misinformation literature has long suggested about warnings. In a 2005 review in Learning and Memory, looking back on three decades of this research, Loftus wrote that warnings about the potential for misinformation sometimes work to inhibit its damaging effects, but only under limited circumstances. The Georgetown and Washington authors reach a similar conclusion. They suggest that noticing discrepancies between the original event and later information can reduce the impact of misinformation, and float warning people about potential discrepancies before they read an AI summary. But they also note that simply knowing the source was AI provided no protection in their data.

Related evidence points the same way. In 2024 a team at the MIT Media Lab, with Loftus among the co-authors, had 200 participants watch a crime video and then answer questions, five of them misleading, via a survey, a scripted chatbot or a generative chatbot built on a large language model. The generative chatbot induced more than three times as many immediate false memories as the control condition, and a week later those false memories persisted, held with higher confidence than in the control group.

A fluent system can introduce errors into your memory. Telling you it is a fluent system does not reliably stop that. And the errors can outlast the conversation that planted them.

The Meeting You Attended, as Told by Copilot

Move from a laboratory junction to an office, and the stakes change shape without shrinking.

Automated meeting recaps are now a standard feature across the major collaboration platforms. Microsoft's intelligent recap in Teams, available with Teams Premium or a Copilot licence, generates AI summaries and follow-up tasks, marks when your name was mentioned, organises the discussion into chapters, and tells you who said what. Microsoft's own support documentation is candid about the risk: “AI-generated content is based on the event transcript and is powered by GPT and other AI models. The AI-generated content may occasionally be inaccurate, incomplete, or inappropriate.”

That is a reasonable disclosure. But if an AI label did not protect participants from absorbing a planted error about a road sign, there is little reason to think small print will protect an employee from absorbing an error about who agreed to what in a budget meeting.

The people most likely to rely on a recap are those who were least attentive: double-booked, multitasking, half-listening while answering email. They are witnesses with weak original traces, exactly where post-event information has most room to take hold. And if a recap is reread just before someone needs to remember, the 1978 finding suggests that is exactly when misleading information bites hardest. A recap consulted before a Friday dispute is not merely a record of Monday's meeting. By Friday, it may be the meeting.

A change Microsoft has scheduled for this autumn sharpens the point. In a Microsoft 365 message centre notice first published in April 2026, the company announced a “recap without saving transcript” option, which lets Copilot users generate an AI meeting summary from live meeting context without retaining a recording or transcript. The notice says the feature addresses scenarios where organisations want AI-generated recaps “but have policies that limit or prohibit the retention of transcripts or recordings”, and it advises administrators to review their AI governance policies to decide whether it fits their compliance posture. Rollout is scheduled for October 2026.

There are sound privacy reasons for not keeping transcripts. But if the summary is the only artefact that survives the meeting, there is nothing to check it against except the attendees' memories, and the study is precisely about what happens to those memories after people read a summary that is wrong.

Even the transcript layer is not safe ground. In October 2024 an Associated Press investigation reported that OpenAI's Whisper speech recognition model was prone to inventing chunks of text, and that more than 30,000 clinicians and 40 health systems were using a Whisper-based tool from Nabla to transcribe and summarise patient visits, despite OpenAI's warnings against use in high-risk domains. The layer that turns a live event into text can introduce words nobody said.

News You Only Half Read

For most people, the summary that shapes memory is not a meeting recap but a news digest glimpsed on a lock screen.

In December 2024 and January 2025, Apple Intelligence's notification summaries produced a run of false headlines attributed to news organisations. One summary of BBC News alerts suggested that Luigi Mangione, the man charged over the killing of the UnitedHealthcare executive Brian Thompson, had shot himself. He had not. Other summaries wrongly claimed, under the BBC's name, that the darts player Luke Littler had won the PDC World Championship before the final had been played, and that Rafael Nadal had come out as gay. A summary of New York Times alerts said Israel's prime minister, Benjamin Netanyahu, had been arrested. The BBC complained. In January 2025 Apple paused notification summaries for news and entertainment apps in a software update, and said summaries would be shown in italics to distinguish them from ordinary notifications. In July 2025 the iOS 26 beta brought news summaries back, this time with a warning beneath the setting: “Summarization may change the meaning of the original headlines. Verify information.” Given the Georgetown and Washington finding that knowing a summary came from AI did not protect memory, a label is a thin defence.

Those were spectacular, checkable errors. The quieter problem is the everyday rate. In February 2025 the BBC published research in which its journalists put 100 news questions to ChatGPT, Microsoft Copilot, Google Gemini and Perplexity, asking them to use BBC sources. It found that 51 per cent of answers had significant issues, 19 per cent of answers citing BBC content introduced factual errors such as wrong dates and figures, and 13 per cent of quotations attributed to BBC articles had been altered or did not exist in the cited article. In October 2025 a far larger follow-up, coordinated by the European Broadcasting Union and led by the BBC, assessed around 3,000 responses across 14 languages and found that 45 per cent of AI answers about news had at least one significant issue, with sourcing the most common problem.

Now put those error rates next to the audience. The Reuters Institute's Digital News Report 2026, published in June, found that weekly use of AI chatbots for news had risen to 10 per cent of people across the markets it surveys, and to 17 per cent among 18 to 24 year olds. Getting summaries was among the most common uses, cited by about a third of those using chatbots for news, and only a small fraction of users regularly clicked through to the original source.

That last number is the hinge. The traditional corrective for a bad summary is the source it summarised, but the whole value proposition of a summary is that you will not open the source. A reader who half-reads an AI digest of a court case and never opens the report carries a version of events that, on this evidence, will not simply sit alongside the truth. For memory's purposes, it becomes what they recall.

Google's AI Overviews were mocked in May 2024 for suggesting glue on pizza and eating rocks, and Liz Reid, Google's head of Search, acknowledged that some answers had drawn on satirical or joke content. Those errors were funny because they were absurd. The errors that reshape memory are not absurd. They are a yield sign where there was a stop sign.

Police Reports and the Officer in the Loop

The authors' main concern is policing, where the question of who checks the machine is most concrete. Axon, the largest supplier of police body cameras in the United States, sells a product called Draft One that uses generative AI to draft police report narratives from body-worn camera audio. The officer is expected to review and edit the draft before submitting it. In July 2025, the Electronic Frontier Foundation published an investigation arguing that the product was designed in a way that frustrated auditing. According to the EFF, Draft One does not save the draft it generates or subsequent edited versions, so once an officer exports the final report there is no record of which parts were written by the AI and which by the officer.

Put that alongside the new study. The standard defence of tools like this is the human in the loop: the officer was there, knows what happened, and will correct the machine. The paper's authors address exactly that defence. “Though 'humans-in-the-loop' are often expected to correct for AI's mistakes,” they write, “our work suggests human memory can instead be distorted by these mistakes.”

The logic is brutal. The officer is the check on the draft, but reading the draft is precisely the kind of post-event exposure that alters memory. If it says the pedestrian stepped off the kerb against the signal, the reviewing officer may come to remember it that way. The check has been contaminated by the thing it was meant to check, and if the draft is deleted, nobody can reconstruct which version came first.

In the Georgetown announcement, Eiger said it worried her that police departments may be using video summarisation tools without rigorous testing, and without an awareness of how wrong AI-generated summaries could be.

Legislators in two American states have begun to respond. Utah's SB 180, which took effect in May 2025, requires police reports produced wholly or partly with generative AI to carry a disclaimer and requires the author to certify that the report was checked for accuracy. California went further. SB 524, signed by Governor Gavin Newsom in October 2025 and in force since 1 January 2026, requires disclosure of AI use on reports, requires agencies to retain the first draft for as long as the final report is kept, and bars vendors from selling or sharing information police provide to the AI. As the EFF put it, the draft retention requirement means judges, defence lawyers or auditors can see which portions of the final report were written by the officer and which by the computer.

Notice what these rules assume. Utah makes the officer answerable for accuracy, yet the officer's memory is one of the things the draft may have altered. California's retention rule is more robust because it does not rely on memory at all: it preserves an artefact that can be compared against the footage. That is a useful template elsewhere.

Records That Outlive the Room

Outside policing, the law has not caught up with the idea that a summary can shape the memories of the people it describes. In the UK, meeting summaries that name or describe identifiable people are personal data, and Article 16 of the UK GDPR gives individuals the right to have inaccurate personal data about them rectified, and incomplete data completed, including by way of a supplementary statement. The Information Commissioner's Office guidance says organisations should take reasonable steps to satisfy themselves that data is accurate when a rectification request is made, taking into account the individual's arguments and evidence. An employee who finds that a Copilot recap attributes to them a commitment they never made has, in principle, a route to have it corrected.

In practice that route depends on the employee noticing the error and having something to prove it with. If the transcript was never kept, the evidence is two sets of memories, one of which may have been rewritten by the summary itself. The right was designed for a world in which records and memories were separate things that could be compared.

Grievances and disciplinary hearings routinely rely on contemporaneous notes. An AI recap looks contemporaneous, with a timestamp and a corporate logo, but it carries no indication of which sentences reflect what was said and which are the model's reconstruction.

What the Law Currently Asks of Whom

The European Union has come closest to allocating responsibility, though not for this specific harm.

Article 50 of the EU AI Act, whose core transparency obligations began applying on 2 August 2026, requires providers of AI systems that generate synthetic text, audio, images or video to ensure outputs are marked in a machine-readable format and detectable as artificially generated. It also requires deployers of AI systems that generate or manipulate text “published with the purpose of informing the public on matters of public interest” to disclose that the text was artificially generated. The Digital Omnibus, which pushed back the Act's high-risk obligations, left that date alone, although providers of generative systems already on the market before 2 August 2026 have until 2 December 2026 to meet the machine-readable marking requirement. The disclosure obligation falls away where the content “has undergone a process of human review or editorial control and where a natural or legal person holds editorial responsibility for the publication”.

That exception is revealing. It makes editorial responsibility, a named person or organisation standing behind the text, the alternative to disclosure. It does not make disclosure alone sufficient to discharge responsibility, and the new study suggests it should not. A label did not protect participants. A person accountable for checking the text before it reached them is a different kind of safeguard.

Liability is shifting too. The EU's revised Product Liability Directive, which member states must transpose by 9 December 2026, explicitly treats software, including AI systems, as a product for the purposes of no-fault liability, and eases the burden of proof for claimants dealing with technically complex systems. Its idea of damage centres on death, personal injury including medically recognised psychological harm, damage to property, and the destruction or corruption of data. A distorted memory of a meeting is unlikely to count. But the principle, that suppliers of software are answerable for defects in what they put on the market regardless of whether they were negligent, is a shift in the default assumption about who carries the risk.

In the United States, the picture is patchier. Beyond the Utah and California police laws, much of the current legal pressure on meeting assistants concerns consent rather than accuracy. In August 2025 a proposed class action, Brewer v. Otter.ai, was filed in federal court in California alleging that Otter's notetaker recorded conversations of people who had not consented, including non-users, and used the data to train its models. In October 2025 it was consolidated with similar suits as In re Otter.AI Privacy Litigation, and on 13 August 2026 Judge Eumi K. Lee refused to dismiss the core claims, allowing wiretap claims under federal and California law to proceed on the basis that Otter, by using recordings to develop its own models, could be treated as a third-party eavesdropper. The case illustrates where the legal system's attention currently sits: on whether the machine should have been listening, not on whether what it wrote afterwards was true.

Why the Reader Cannot Carry This Alone

The easy answer to who should check the summary is the person reading it. That is what the disclaimers say, and what “human in the loop” means in most enterprise AI policies.

The study makes that answer hard to defend, for three reasons.

The first is that the checking mechanism is compromised by the thing being checked. Memory is the natural reference for testing a summary of an event you attended, but reading the summary changes the memory. Not every time, and not for everyone, but often enough that “check it against what you remember” is not a reliable safeguard.

The second is that the people most exposed to summaries are the people least equipped to check them. Summaries are used by people who did not attend, did not pay attention or did not read the original. That is the point of them. Asking those users to verify the summary against a source they were trying to avoid is asking them to do the work the product was meant to save.

The third is that confidence in the tool reduces checking. In a survey of 319 knowledge workers presented at the CHI conference in 2025, researchers at Microsoft Research and Carnegie Mellon University found that higher confidence in generative AI was associated with less critical thinking, while higher self-confidence was associated with more. The more reliable a tool seems, the less people scrutinise it, and a tool that is right most of the time trains exactly the complacency that lets its occasional errors through.

None of this means readers bear no responsibility. Attendees can take their own notes, readers can click through, officers can write their own account first. But a design that depends on users consistently doing the things the product was designed to spare them is not a design with a human safeguard. It is a design with a human scapegoat.

What Suppliers Could Actually Owe

If the burden cannot sit only with readers, the obvious alternative is the organisations that build and deploy the summarisers. What would taking that burden seriously look like?

It would start with measurement. The authors call for more technical audits of summarisation tools and for examining real-world uses such as body camera footage. Vendors selling summarisation for high-stakes settings could publish error rates on representative material, broken down by type: omissions of central events, substitutions, additions. A summariser that routinely omits the central event of a clip is not ready to summarise evidence.

It would continue with provenance. California's approach to police reports offers a model: keep the draft, keep the source, make it possible to see which words came from the machine. For meeting recaps, that could mean retaining source audio or transcripts for a defined period when a summary is created, or linking each sentence of a summary to the moment in the recording it draws on. Microsoft's recap already offers timeline markers and speaker navigation. Anchoring every claim to its evidence would make checking something a reader can do in seconds rather than a reconstruction from memory.

It would include friction in the right places. The study's authors suggest that noticing discrepancies helps, and that warnings delivered before reading might help. That points towards designs that ask users to recall something before showing them the summary, or that flag which parts of a recap are high-stakes (decisions, attributions, commitments) and invite confirmation from the people named. It points away from designs that deliver a finished, authoritative paragraph and a disclaimer.

And it would include restraint. The authors raise a question most product roadmaps do not: “a deeper question to consider is whether AI should be used in these contexts at all.” For some uses, such as drafting a police narrative from body camera audio before the officer has written their own account, the answer may be that the order of operations matters more than the quality of the model. Human recollection first, machine summary second, and a record of both.

Organisations that deploy summaries carry a share too. An employer that switches on recaps without transcripts and relies on them in a disciplinary process has made choices about evidence. A platform that pushes machine summaries of journalism to millions of lock screens has made choices about what its audience will remember. The EU's editorial responsibility test is a good instinct: somebody specific should be answerable for what the summary says.

Keeping Hold of the Afternoon You Lived

There is a version of this story that is merely about quality control: the mistakes will get rarer and the problem will shrink. That may be partly true. But the study is not really about how often summaries are wrong. It is about what happens when they are. A human memory is not a hard drive that a summary sits beside. It is a reconstruction, rebuilt each time it is recalled, and fifty years of research have shown that the materials used in the rebuild include whatever we have been told since. For most of that history, the people doing the telling were other humans: lawyers asking leading questions, police officers suggesting details, newspapers and neighbours. They could be cross-examined. They could be blamed.

The summary is a new kind of teller, present at every meeting and every alert, speaking in a voice that sounds neutral and thorough. When it drops the collision from the crash, or puts a yield sign where a stop sign stood, it does not only produce a bad document. It produces people who remember a slightly different afternoon from the one they lived.

Right now, the burden of noticing falls on the people least able to notice, and responsibility for the error is spread so thinly that it lands on almost nobody. The disclaimers say the content may be inaccurate. The users are told to check. The checking, it turns out, runs through a memory the summary has already touched.

That arrangement looks unlikely to survive contact with a courtroom for long. The better move is to settle the question first: keep the source, mark the machine's words, measure the errors, and put a named person behind any summary that is going to be treated as the record. The past is hard enough to hold on to without software editing it for us.

---

References and Sources

1. Sim, Mattea, Yael Eiger and Tadayoshi Kohno. “AI-Enabled Human Memory Manipulation: Misleading AI-Generated Summaries Distort Human Memory.” arXiv:2609.28820v1, submitted 23 September 2026; to be presented at the Ninth AAAI/ACM Conference on AI, Ethics and Society (AIES), October 2026. https://arxiv.org/abs/2609.28820
2. Soong, Kelyn. “New Research Finds That Misleading AI-Generated Summaries Can Distort Human Memory.” Georgetown University College of Arts and Sciences, 25 September 2026. https://college.georgetown.edu/news-story/new-research-finds-that-misleading-ai-generated-summaries-can-distort-human-memory/
3. Neuroscience News. “Flawed AI Summaries Distort Eyewitness Memory.” 27 September 2026. https://neurosciencenews.com/ai-false-memory-llm-31261/
4. The Independent. “AI summaries can warp your memory, study warns.” September 2026, as syndicated via inkl. https://www.inkl.com/news/ai-summaries-can-warp-your-memory-study-warns
5. Loftus, Elizabeth F. and John C. Palmer. “Reconstruction of automobile destruction: An example of the interaction between language and memory.” Journal of Verbal Learning and Verbal Behavior, volume 13, issue 5, 1974, pages 585 to 589. https://www.sciencedirect.com/science/article/abs/pii/S0022537174800113
6. Loftus, Elizabeth F., David G. Miller and Helen J. Burns. “Semantic integration of verbal information into a visual memory.” Journal of Experimental Psychology: Human Learning and Memory, volume 4, issue 1, 1978, pages 19 to 31. https://pubmed.ncbi.nlm.nih.gov/621467/
7. Loftus, Elizabeth F. “Planting misinformation in the human mind: A 30-year investigation of the malleability of memory.” Learning and Memory, volume 12, 2005, pages 361 to 366. DOI 10.1101/lm.94705. https://learnmem.cshlp.org/content/12/4/361.full
8. Chan, Samantha, Pat Pataranutaporn, Aditya Suri, Wazeer Zulfikar, Pattie Maes and Elizabeth F. Loftus. “Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviews.” MIT Media Lab, arXiv:2408.04681, August 2024. https://arxiv.org/abs/2408.04681
9. Microsoft Support. “Recap in Microsoft Teams.” Accessed 29 September 2026. https://support.microsoft.com/en-us/teams/meetings/recap-in-microsoft-teams
10. Microsoft 365 Message Center. “MC1275312: Microsoft Teams: AI meeting recap without transcript to meet compliance policies.” Published 9 April 2026, archived at Merill Fernando's Message Center Archive. https://mc.merill.net/message/MC1275312
11. Burke, Garance and Hilke Schellmann. “Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said.” Associated Press, 26 October 2024; republished by Fortune and The Columbian. https://fortune.com/2024/10/26/openai-transcription-tool-whisper-hallucination-rate-ai-tools-hospitals-patients-doctors
12. TechCrunch. “Apple pauses AI notification summaries for news after generating false alerts.” 16 January 2025; see also Axios, “Apple pauses AI-generated news alerts after fake headline notifications”, 17 January 2025. https://techcrunch.com/2025/01/16/apple-pauses-ai-notification-summaries-for-news-after-generating-false-alerts ; see also AppleInsider, “Apple brings back notification summaries for news in iOS 26”, 22 July 2025. https://appleinsider.com/articles/25/07/22/apple-brings-back-notification-summaries-for-news-in-ios-26
13. BBC. “Representation of BBC News content in AI Assistants.” BBC Research, February 2025; as reported by The Register, “AI summaries turn real news into nonsense, BBC finds”, 12 February 2025. https://www.theregister.com/2025/02/12/bbc_ai_news_accuracy/
14. European Broadcasting Union and BBC. “News Integrity in AI Assistants.” October 2025; and EBU press release, “Largest study of its kind shows AI assistants misrepresent news content 45% of the time, regardless of language or territory.” https://www.ebu.ch/news/2025/10/ai-s-systemic-distortion-of-news-is-consistent-across-languages-and-territories-international-study-by-public-service-broadcaste
15. Reuters Institute for the Study of Journalism. “Digital News Report 2026: Overview and key findings.” University of Oxford, June 2026. https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2026/dnr-executive-summary
16. Reid, Liz. Google Search blog post on AI Overviews, 30 May 2024; as reported by MIT Technology Review, “Why Google's AI Overviews gets things wrong”, 31 May 2024, and Forbes, 31 May 2024. https://www.technologyreview.com/2024/05/31/1093019/why-are-googles-ai-overviews-results-so-bad/
17. Electronic Frontier Foundation. “Axon's Draft One Is Designed to Defy Transparency.” July 2025. https://www.eff.org/deeplinks/2025/07/axons-draft-one-designed-defy-transparency
18. Guariglia, Matthew. “Victory! California Requires Transparency for AI Police Reports.” Electronic Frontier Foundation, 14 October 2025. https://www.eff.org/deeplinks/2025/10/victory-california-requires-transparency-ai-police-reports
19. California Legislature. SB 524, Law enforcement agencies: artificial intelligence, 2025 to 2026 session; and Office of Senator Jesse Arreguín, “Governor Signs Bill Requiring Law Enforcement to Disclose Use of AI in Police Reports.” https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202520260SB524
20. Utah Legislature. SB 180, Law Enforcement Usage of Artificial Intelligence, 2025 General Session; and Utah News Dispatch, “Deepfakes, policing, mental health and more: Lawmakers look to further regulate AI”, 26 March 2025. https://le.utah.gov/~2025/bills/static/SB0180.html
21. Information Commissioner's Office. “Right to rectification.” UK GDPR guidance and resources. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/individual-rights/individual-rights/right-to-rectification/
22. European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50, Transparency obligations for providers and deployers of certain AI systems. https://artificialintelligenceact.eu/article/50/ ; see also Cooley, “EU AI Act: Transparency Obligations Take Effect 2 August 2026”, 3 August 2026. https://www.cooley.com/news/insight/2026/2026-08-03-eu-ai-act-transparency-obligations-take-effect-2-august-2026 ; and Goodwin, “Not Delayed, Not Deferred: EU AI Act Transparency Obligations Are Now in Force”, 3 August 2026. https://www.goodwinlaw.com/en/insights/publications/2026/08/alerts-technology-dpc-eu-ai-act-transparency-obligations-now-in-force
23. European Union. Directive (EU) 2024/2853 on liability for defective products. Official Journal of the European Union, 18 November 2024. https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng
24. NPR. “Class-action suit claims Otter AI secretly records private work conversations.” 15 August 2025. https://www.npr.org/2025/08/15/g-s1-83087/otter-ai-transcription-class-action-lawsuit ; and Baird Holm, “Core Privacy Claims Against AI Note Taking Platform Survive Dismissal” (In re Otter.AI Privacy Litigation, No. 5:25-cv-06911, N.D. Cal.), 24 August 2026. https://www.bairdholm.com/blog/core-privacy-claims-against-ai-note-taking-platform-survive-dismissal/
25. Lee, Hao-Ping (Hank), and colleagues, Microsoft Research and Carnegie Mellon University. “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, ACM. DOI 10.1145/3706598.3713778. https://dl.acm.org/doi/full/10.1145/3706598.3713778

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Jasmine Ault compares it to a house fire. On 24 September 2026, according to a report by Anja Karadeglija of The Canadian Press published five days later, the Toronto resident found that Meta had shut down her Facebook and Instagram accounts without warning. The Facebook profile was 20 years old. One Instagram account dated back to 2011. A second was dedicated to her dog. Overnight, two decades of memories, contacts and professional connections were behind a door she could not open.

“It's actually quite scary to think about the trust that we put in the social media platforms that can all be taken away in one day,” she told the news agency. The fire, she explained, is a metaphor, “and it's really upsetting because the hypothetical, the metaphorical fire that I'm referring to, is my memories.”

What made the loss feel so total was not just the ban. It was the silence after it. Ault told The Canadian Press she had not been able to reach a human being at Meta since the accounts were deactivated. The only things that would talk to her were AI chatbots, and they could not resolve the problem. CBC News, which also spoke to her, reported that after the deletion notice arrived she stayed up for about 26 hours straight trying to appeal. Within minutes of pressing appeal, she was told her case was closed and that she would not be permitted to appeal again. She was never told which rule her Facebook account had supposedly broken. Meta's explanation, as reported, amounted to a single line: her Facebook account “doesn't follow our rules”. When The Canadian Press asked questions, Meta did not respond to them directly and instead pointed to a support blog post, including an update from March about rolling out an AI support assistant. Meta did not respond to CBC's requests for comment.

This is the uncomfortable shape of a modern dispute with a platform. The thing that holds your relationships, your archive and your livelihood can remove all of them in an instant, and the only voice you hear afterwards may belong to software. That raises two questions bigger than one Toronto woman's Facebook profile. What should an effective appeal look like when AI mediates access to the platforms that hold people's lives? And how can platforms keep their security enforcement, which is real and necessary, while still offering a route to review that a locked-out human being can actually use?

What the Report Does and Does Not Establish

It is worth being precise about what is known. The Canadian Press report does not establish why Ault's accounts were disabled. Meta has not publicly said which rule it believes was broken. Ault herself has a theory: she told the agency she suspects someone tried to hack her account, because she found a record of suspicious login attempts shortly before it was deactivated, and Meta's records associated her phone number with someone in India. That is her suspicion, reported as such. It is not a finding.

It would be easy to fill the gap with a story. Perhaps an intruder got in and posted something that tripped an automated filter. Perhaps a classifier misread an old photograph. Perhaps something else entirely happened. None of those possibilities is established, and the honest position is that nobody outside Meta, including Ault, has been told. That gap is the point. A system that cannot tell a person what she is accused of has already failed the first test of a fair process, whatever the underlying facts turn out to be.

The human cost, at least, is not in dispute. Ault works in marketing and communications and had used social media to attract consulting clients. Prospective clients, she told the news agency, “have nothing to look at. They can't see my portfolio of work.” Friends might assume she had blocked them. “There's a dead end and I can't even communicate to the people I want to communicate with,” she said. Markus Giesler, a professor of marketing at York University, framed it in sharper terms for the same report: “We like to think of this as a customer service case when in actuality it's a case of platform accountability.” These platforms, he added, “have been woven so much into our social fabric, into who we are as individuals and communities, that access to them is really important.”

The Ban Wave That Came Before

Ault's story lands in a long, loud context. Through the summer of 2025, users in several countries reported Facebook and Instagram accounts being disabled in waves, often with no clear reason. In June 2025 Meta acknowledged what spokesperson Andy Stone called “a technical error that impacted some Facebook Groups”, after communities devoted to things such as parenting, birdwatching and gaming received violation notices. By early July, TechCrunch reported that a petition titled “Meta wrongfully disabling accounts with no human customer support” had passed 25,500 signatures, and that users were organising on Reddit to share experiences and talk about legal action.

Some of the most distressing cases involved accusations of child sexual exploitation, a category where, as the Oversight Board noted quoting Meta's Transparency Center, a single violation may be severe enough that “we'll disable your account”. ABC News in Australia reported in July 2025 on Madison Archer, a beautician whose Instagram business account, built over nine years, was permanently disabled in June 2025 after she posted a video of herself with her daughter. She appealed within minutes and was refused. Her account came back only after ABC News contacted Meta, which told her: “We're sorry that we've got this wrong and that you weren't able to use Instagram for a while.” Archer's verdict on the process was blunt: “If a human actually did the review they would see that I'm not sexually exploiting children.”

In South Korea, the backlash became political: The Korea Herald reported in July 2025 that the Korea Communications Commission had formally asked Meta for written clarification, and that Meta Korea had apologised for the “frustration”. None of this proves that Ault's case shares a cause with those waves. It does show that the pattern she describes, sudden loss, terse explanation and support that cannot or will not engage, was familiar long before September 2026.

Meta itself has not pretended its enforcement is flawless. In January 2025, Joel Kaplan, the company's chief global affairs officer, wrote that in December 2024 Meta was removing millions of pieces of content every day, and that “we think one to two out of every 10 of these actions may have been mistakes”. He conceded that “too many people find themselves wrongly locked up in 'Facebook jail,' and we are often too slow to respond when they do.” By May 2025 Meta reported a roughly 50 per cent reduction in enforcement mistakes in the United States between the fourth quarter of 2024 and the first quarter of 2025. Its more recent integrity reporting says that less than 0.1 per cent of the content produced on Facebook and Instagram in a quarter was removed incorrectly, but also discloses that in the fourth quarter of 2025 false positives “increased over 100% during a very brief period of the quarter due to a bug”.

A tiny percentage of an enormous base is still a great many real people, and one bug can double the rate of wrongful action in days. Automated enforcement will make mistakes. The question is what happens to the person on the receiving end when it does.

Paying for a Human and Getting a Bot

When Meta launched its paid Meta Verified subscription in 2023, it promised direct access to account support. It is precisely what a person in Ault's position would reach for: pay a monthly fee, get a human. Ault did exactly that. According to CBC News, she paid US$14.99 (about C$21) for Meta Verified's Standard tier, believing it would get her human support. The “enhanced support” she received was another AI chatbot, which promptly closed her ticket, telling her she had “reached her limit of support interactions”. She was not the first subscriber to be disappointed. TechCrunch reported in July 2025 that during the ban wave, paying subscribers described agents who were dismissive, closed chats and offered canned responses.

The sharpest assessment came from inside Meta's own governance structure. In June 2026 the Oversight Board, the independent body Meta funds to review its content decisions, issued its first decision on account disablement. Summarising the public comments it received, the Board recorded that many users said they were unable to appeal, never received any explanation for why their account was disabled, or were unable to download their content. Others, it noted, had created a new account specifically so they could pay for Meta Verified, “which includes '24/7 access to email or chat agent support,'” and yet “could not get meaningful assistance”. Some reported that their new accounts were then disabled for breaking the rule against creating an account after being banned.

The trap is circular. The paid route to a human requires an account. The person who most needs a human no longer has one. Opening a fresh account can itself be treated as evasion. And the free route, increasingly, is an AI assistant.

Meta has been explicit about that direction of travel. In December 2025 it launched a centralised support hub for Facebook and Instagram, acknowledging that “support hasn't always met expectations”, and began testing an AI support assistant. In March 2026 it announced a wider rollout, describing an assistant that provides “24/7 help for account issues” and can “respond to requests typically in under five seconds”. The same post contains a sentence that should be read closely by anyone who has been locked out: “people will continue to play a key role in how we make the highest risk and most critical decisions, such as appeals of account disablement.”

That is the right principle. The difficulty is that, from the outside, it is invisible. If a person cannot tell whether a human ever looked at her appeal, and receives a decision indistinguishable from an automated one, a human somewhere in the pipeline does little to make the process accountable.

Why Platforms Lock the Doors So Fast

None of this means platforms should slow-walk enforcement or open the doors to anyone who asks nicely. The reasons for speed are serious.

The first is child safety. Meta reported that in the fourth quarter of 2025 alone, Facebook, Instagram and Threads sent more than 2.6 million reports to the CyberTipline run by the National Center for Missing and Exploited Children in the United States. When a system detects apparent child sexual exploitation, the expectation is immediate action, not a conversation. Errors are horrifying for the wrongly accused, but delay on true positives carries its own grave cost.

The second is fraud. Meta says that in 2025 it took down 10.9 million accounts associated with scam centres in Southeast Asia and the Middle East. An appeals channel that can be gamed becomes a tool for operations of that scale.

The third, and the one most relevant to Ault's suspicion, is account takeover. A hijacked account carries the trust of its real owner's friends, followers and clients. A platform that detects suspicious logins faces a genuine dilemma: lock the account and the owner suffers; leave it open and the intruder can impersonate them or scam their contacts. Recovery itself is dangerous ground. The United States National Institute of Standards and Technology, in its digital identity guidelines, notes that because account recovery is invoked infrequently, it “is generally less convenient than authentication” and “may involve extended waiting times”, and requires that an account recovery event always triggers a notification to the subscriber “to help detect the fraudulent use of account recovery”. Security engineers expect recovery to be slower than logging in precisely because it is where attackers aim.

When the Help Desk Became the Break-In

If anyone doubted that recovery is a battleground, 2026 supplied the proof, courtesy of the very kind of AI support that now stands between users like Ault and their accounts.

On 1 June 2026, 404 Media reported that hackers had been taking over high-profile Instagram accounts by asking Meta's AI support assistant to link the target account to a new, attacker-controlled email address. The accounts reportedly affected included the Barack Obama White House account, the account of the chief master sergeant of the US Space Force and that of the retailer Sephora. TechCrunch reported two days later that Instagram was alerting users targeted during the attacks, and quoted Meta spokesperson Andy Stone saying that “the issue that did happen has already been fixed”.

The scale emerged in a filing Meta made to the Maine Attorney General, reported by Help Net Security on 8 June. It disclosed that 20,225 Instagram accounts had been taken over through a flaw in an AI-assisted recovery system called High Touch Support, with the first unauthorised access dating to 17 April 2026 and the vulnerability identified on 31 May. Amber Hannah, an associate general counsel at Meta, explained in the notice that “due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user's Instagram account.” Accounts without two-factor authentication could then be taken over.

The episode cuts both ways. It shows why platforms are wary of easy recovery: an assistant that says yes too readily is a gift to criminals. But it also punctures the idea that automation is inherently the safer option. An AI system that handed over more than 20,000 accounts was not a cautious gatekeeper. It was a fast one. Speed and security are not the same thing.

What the Oversight Board Found Inside the Account Machine

The Oversight Board's June 2026 decision is the most detailed public examination yet of how Meta disables accounts. The case itself was not a sympathetic one. The Board upheld Meta's decision to permanently disable an Instagram account with more than 70,000 followers that had posted threats of violence against a journalist, including an image of a bullseye over her face. It did not conclude that Meta should disable fewer accounts across the board.

What it did conclude, after more than 750 public comments, was that the disablement process raised serious concerns about due process and proportionality. It found no clear framework for when “egregious” violations justify immediate permanent removal, and noted that there were no intermediate options between permanent disablement and leaving the ordinary strikes system to run its course. Where mitigating factors exist, the Board suggested, “a significant but time-bound suspension may be more proportionate.”

On appeals, the Board was pointed. Many comments described reasons people did not understand and appeals denied within moments of submission. Ault's experience, as CBC News reported it, fits that description exactly: her case was closed within minutes of her pressing appeal, with no further appeal allowed. Appeals of account disablement, the Board wrote, “are not effective if they run the same decision through the same automated process, expecting different results with the same inputs.” It accepted that advances in AI should be used to make reviews more accurate and comprehensive, but said that “prioritizing of human review in edge cases will be important”, especially for “one and done” violations, “where any error in automated decision-making has very serious consequences beyond the removal of a post or imposition of a strike.”

It also set out what users should be able to see. It called for dashboards giving “reliable information on current account status, past violations, information on available appeal options, and the status of any pending appeals”, downloadable for record-keeping. It wanted clear, prominent and timely notifications, “in a language they understand”, specifying “the rule violated, the sanction imposed and options to appeal”. It asked that users be told about “the role of automation in the review of content or behavior and the imposition of warnings or penalties”, and that people who lose access through permanent disablement can still reach the reasons and appeal mechanisms. Meta said it welcomed the decision and would respond to the recommendations after reviewing them.

Set beside Ault's account, the Board's framework reads like a checklist of what she says she did not get.

Europe Has Already Written Down What an Appeal Owes You

For users in the European Union, much of this is not aspirational. It is law, under the Digital Services Act.

Article 17 requires hosting services to give “a clear and specific statement of reasons” to anyone whose account is suspended or terminated. That statement must include the facts and circumstances relied on, and information on the use made of automated means in taking the decision. A line saying an account “doesn't follow our rules” would struggle to meet that standard.

Article 20 requires online platforms to provide, for at least six months after a decision, “an effective internal complaint-handling system” that lets people lodge complaints “electronically and free of charge”, including against decisions to suspend or terminate their account. Complaints must be handled “in a timely, non-discriminatory, diligent and non-arbitrary manner”, and where the complainant's conduct does not warrant the measure, the platform must reverse its decision “without undue delay”. Crucially, the platform must ensure that these complaint decisions “are taken under the supervision of appropriately qualified staff, and not solely on the basis of automated means.” The law does not ban AI from the process. It bans AI from being the whole of it.

Article 21 opens a door outside the platform. Users can take disputes to a certified out-of-court dispute settlement body, which should decide within 90 days, extendable to 180 for complex cases. If the body finds for the user, the platform pays the fees; if it finds for the platform, the user does not cover the platform's costs unless they acted in bad faith. The catch is that these bodies cannot impose a binding settlement.

Data protection law adds another layer. The General Data Protection Regulation gives people a right of access to their personal data and a right to portability, receiving what they provided “in a structured, commonly used and machine-readable format”, normally within one month. Article 22 protects people from decisions “based solely on automated processing” with legal or similarly significant effects and, where exceptions apply, guarantees a right “to obtain human intervention” and to contest the decision. Whether a given ban meets that threshold is a legal question, but the principle is unmistakable.

Canadians have a version of the access right too. Under the Personal Information Protection and Electronic Documents Act, organisations must answer access requests in writing within 30 days, with limited extensions, and individuals can complain to the Office of the Privacy Commissioner of Canada. For someone whose memories are trapped in a disabled account, that is a lever that exists independently of any chatbot.

The Independent Referee With an Empty Evidence File

The most revealing evidence about how these systems work in practice comes from the referee. Appeals Centre Europe, an out-of-court dispute settlement body certified by Ireland's media regulator, Coimisiún na Meán, and set up with a one-time grant from the Oversight Board Trust, began operating in late 2024 and now covers platforms including Facebook, Instagram, TikTok and YouTube. When it began handling account suspensions in 2025, its chief executive, Thomas Hughes, said: “Losing your social media account is more than an inconvenience, it affects people's lives in fundamental ways.”

In May 2026 the centre published data covering April 2025 to March 2026. As reported by RTÉ, it received more than 24,000 disputes, of which more than 12,000 were within scope. Where it reviewed decisions, it disagreed with the platforms 59 per cent of the time. On account suspensions, the figures were starker still. It had received more than 5,000 eligible disputes from users challenging suspensions, but by the end of March 2026 had made decisions after actually receiving the relevant content in fewer than 150 of them. In the rest, the platforms did not supply the material, and the centre issued “default decisions” in the user's favour. Its decisions are not binding, and the centre said it was aware of only a handful of cases where platforms had implemented its decisions on harmful content.

Independent review works only if the reviewer can see the evidence, and a legal right to escalation can fail in practice if the platform does not engage. The infrastructure for a fair appeal can exist on paper and still leave a locked-out person with nothing.

Britain and Canada Are Still Catching Up

Outside the EU, the protections are thinner. The United Kingdom's Online Safety Act 2023 requires user-to-user services to operate complaints procedures that are “easy to access, easy to use (including by children) and transparent”. Section 21 includes complaints from users who have been warned, suspended or banned, but only where that action results from content the provider considers to be illegal content. A ban for breaching a platform's own terms of service, rather than for illegal content, sits largely outside that specific duty.

Canada, where Ault lives, has no equivalent framework in force. In June 2026 the federal government introduced Bill C-34, the Safe Social Media Act, which would create a Digital Safety Commission and impose duties on platforms around seven categories of harmful content, including content that sexually victimises children, incites violence or foments hatred. The government's published summaries require “accessible reporting and complaint mechanisms” and give the commission a role in handling user complaints about harmful content. They do not, in the material the government has published, set out a specific right for users to appeal account disablement or to have such an appeal decided by a human. According to LEGISinfo, Parliament's official bill-tracking service, the bill is at second reading in the House of Commons and has not yet been referred to committee, so that gap could yet be addressed.

What an Effective Appeal Actually Looks Like

Pull together the Oversight Board's principles, the DSA, data protection law and the experience described in the Canadian Press report, and a clear picture emerges of what an effective appeal must include when AI sits at the front door. None of it is exotic.

The first element is notice with a specific reason. A person should be told, at the moment of enforcement and in plain language, which rule they are said to have broken, what the sanction is, how long it lasts and how to challenge it. If the decision was triggered by an automated system, they should be told that too. “Doesn't follow our rules” is not a reason. It is a refusal to give one.

The second is access to the evidence, as far as safety allows. A user cannot contest a decision about content they cannot identify. Where the trigger was a post, a message or a login pattern, the appeal should show what was relied on. Disclosure that would compromise a child exploitation investigation or tip off a fraud ring is a legitimate limit, but it should be an exception, not the default. The Appeals Centre's experience shows how far the default currently leans towards secrecy.

The third is a real opportunity to make one's case. The Oversight Board insists on the chance to submit written reasons, with continued access to the policy justification and any relevant exceptions. A drop-down menu and a “submit” button is not that. An AI assistant can help people understand the rules and structure an appeal. Its job should be to prepare the case for a decision-maker, not to be the decision-maker in the hardest cases.

The fourth is a guaranteed human decision for serious sanctions. Permanent disablement of a long-standing account is the platform equivalent of capital punishment for a digital identity. The DSA already requires that complaint decisions are not taken solely by automated means, and Meta's own March 2026 announcement says humans remain key in appeals of account disablement. What is missing is visibility. People should be told, in the decision itself, that a named role (not necessarily a named individual) reviewed their case, and on what basis.

The fifth is predictable timelines. An appeal that may take a day or six months is useless to someone whose income depends on their account. Platforms should publish target times, show users where they are in the queue and report on whether targets are met.

The sixth is preserved access to one's own data during the dispute. Losing access to a platform should not mean losing one's photographs, messages and contact lists for the duration of a months-long appeal. Meta's own help pages say users “may be able to request a download” of their information from the notification screen, but the Oversight Board recorded many users saying they could not. Data export during suspension should be a default, subject only to narrow, specified exceptions such as material that is itself illegal.

The seventh is escalation that leads somewhere independent. Internal appeals will always carry a structural conflict of interest. Users need a clearly signposted route beyond the platform, whether that is a DSA-style dispute settlement body, a regulator's complaints function or an ombudsperson, with the power to see the evidence and a platform that is obliged to engage.

Enforcement That Does Not Have to Be All or Nothing

The second question is harder: how do platforms keep enforcement strong without making review impossible? The answer is to stop treating account action as a single switch with two positions, on and gone.

Graduated enforcement is the obvious starting point. The Oversight Board itself observed that Meta lacked intermediate options between permanent disablement and its ordinary strikes system, and suggested time-bound suspensions where mitigating factors exist. A spectrum of measures, from restrictions on posting or messaging, to temporary suspension, to permanent removal, lets a platform act immediately on a risk while matching the long-term sanction to the confidence and seriousness of the finding.

Read-only and data-only modes are the second tool. When a platform suspects compromise, as Ault believes happened to her, it can freeze outward activity without erasing the owner's access to her own past. A frozen account cannot scam a contact list, but its owner can still see and export her photographs and messages. Where the suspected violation is the gravest kind, read-only access may be inappropriate, but such cases are a subset of disablements, not a reason to treat every case as one.

The third tool is separating the security question from the conduct question. “Is this the real owner?” and “Did this account break the rules?” are different problems, and conflating them hurts everyone. If an account appears to have been hijacked, the right response is a security recovery process with strong identity checks, such as the optional selfie video verification Meta now offers, plus notifications to the owner's known contact points of the kind NIST requires. Only once ownership is established does it make sense to ask whether the owner, rather than an intruder, did anything wrong. The High Touch Support breach shows what happens when a recovery path skips the first step.

Fourth, verification should unlock a route to a human, not replace one. If a person proves who they are through a government document or a live selfie, the platform has sharply reduced the risk of handing the account to an attacker. That is exactly the moment when a human reviewer becomes affordable and safe. The cruel irony of Ault's account is that she says she had done the paid verification step and still could not get past the chatbot. A verification-gated escalation path, honoured in practice, would answer the security objection to human support rather than ignore it.

Fifth, proportionality needs to be measured, not merely promised. The Oversight Board asked for transparency reports on account disablement that distinguish “one and done” removals from graduated enforcement and specify which rules drove them. Add overturn rates, resolution times and the share of decisions reviewed by humans, and outsiders can see whether the system works. Meta already publishes precision figures for content removal. It should publish the equivalent for accounts.

Finally, platforms should design for the worst week of a user's life, not the median one. Most support requests are trivial, and AI assistants handle them well. Account disablement is rare per user, devastating when it happens and, as the 2025 ban waves and the 2026 recovery breach showed, vulnerable to systemic error. That is exactly where human judgement and independent oversight earn their cost.

Making the Machine Answerable

It is tempting to cast AI support as the villain of Ault's experience, but that is not quite right. Nobody outside Meta knows what triggered the shutdown. What the chatbot represents is a support model built to resolve common cases cheaply and quickly, deployed in front of the rare cases where cheap and quick is the wrong answer.

Meta has said in writing that humans will play a key role in appeals of account disablement. The Oversight Board has said appeals must not re-run the same automated decision. The EU has made human supervision of complaint decisions law. An independent appeals body has shown that when platforms are asked to defend suspensions with evidence, they often do not. Every piece of a fair system exists somewhere. What is missing is the will to assemble them in front of the person who has just lost twenty years of her life and wants, above all, to be told why.

Giesler's framing is the right one. This is not a customer service problem to be optimised for response time. It is an accountability problem, and accountability means a reason you can read, evidence you can see, a person who can decide, a deadline that is kept and a referee who answers to neither side. A chatbot can help with every one of those steps. It cannot be allowed to stand in for all of them.

---

References

1. Karadeglija, Anja. “Canadian says Meta won't let her talk to a human after killing her accounts.” The Canadian Press, 29 September 2026 (updated 30 September 2026), as published by CP24. https://www.cp24.com/news/2026/09/29/canadian-says-meta-wont-let-her-talk-to-a-human-after-killing-her-accounts/
2. Jones, Alexandra Mae. “Overnight, Meta deleted all her accounts. AI customer service won't let her reach a human.” CBC News, 1 October 2026. https://www.cbc.ca/news/business/meta-accounts-deleted-ai-customer-service-9.7364543
3. Kaplan, Joel. “More Speech and Fewer Mistakes.” Meta Newsroom, 7 January 2025, updated 29 May 2025. https://about.fb.com/news/2025/01/meta-more-speech-fewer-mistakes/
4. Meta. “Integrity Reports, H1 2026.” Meta Transparency Center, March 2026. https://transparency.meta.com/reports/integrity-reports-h1-2026/
5. UPI. “Meta confirms technical error suspended thousands of Facebook Groups.” 24 June 2025. https://www.upi.com/Top_News/US/2025/06/24/meta-suspends-facebook-groups/5531750820668/
6. Perez, Sarah. “Meta users say paying for Verified support has been useless in the face of mass bans.” TechCrunch, 2 July 2025. https://techcrunch.com/2025/07/02/meta-users-say-paying-for-verified-support-has-been-useless-in-the-face-of-mass-bans/
7. Dervisevic, Hanan. “Meta falsely accuses Instagram user of breaking child exploitation rules.” ABC News (Australia), 17 July 2025. https://www.abc.net.au/news/2025-07-17/meta-wrongly-accuses-user-of-breaching-child-sex-abuse-rules/105540896
8. Moon, Joon-hyun. “Meta faces backlash in South Korea over mysterious account bans.” The Korea Herald, 13 July 2025. https://www.koreaherald.com/article/10530332
9. Meta. “Making it Easier to Access Account Support on Facebook and Instagram.” Meta Newsroom, 4 December 2025. https://about.fb.com/news/2025/12/making-it-easier-to-access-account-support-on-facebook-and-instagram/
10. Meta. “Boosting Your Support and Safety on Meta's Apps With AI.” Meta Newsroom, 19 March 2026. https://about.fb.com/news/2026/03/boosting-your-support-and-safety-on-metas-apps-with-ai/
11. Oversight Board. “Account Ban for Targeting Public Figures” (case decision BUN-ZHSLVJO2) and summary. 4 June 2026. https://www.oversightboard.com/decision/bun-zhslvjo2/
12. Perez, Sarah. “Meta's Oversight Board says account bans lack due process, transparency.” TechCrunch, 4 June 2026. https://techcrunch.com/2026/06/04/metas-oversight-board-says-account-bans-lack-due-process-transparency/
13. Koebler, Jason. “Hackers Simply Asked Meta AI to Give Them Access to High-Profile Instagram Accounts. It Worked.” 404 Media, 1 June 2026. https://www.404media.co/hackers-simply-asked-meta-ai-to-give-them-access-to-high-profile-instagram-accounts-it-worked/
14. Franceschi-Bicchierai, Lorenzo. “Instagram is alerting users who were targeted by hackers during AI chatbot attacks.” TechCrunch, 3 June 2026. https://techcrunch.com/2026/06/03/instagram-is-alerting-users-who-were-targeted-by-hackers-during-ai-chatbot-attacks/
15. Markovic, Sinisa. “Hackers used Meta's AI support system to hijack over 20,000 Instagram accounts.” Help Net Security, 8 June 2026, reporting Meta's data breach notice filed with the Maine Attorney General. https://www.helpnetsecurity.com/2026/06/08/instagram-ai-support-vulnerability-account-takeovers/
16. National Institute of Standards and Technology. “Digital Identity Guidelines: Authentication and Authenticator Management.” NIST Special Publication 800-63B-4, 2025, section on account recovery. https://pages.nist.gov/800-63-4/sp800-63b.html
17. European Parliament and Council. Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act), Articles 17, 20, 21 and 23. Official Journal of the European Union, 27 October 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2065
18. European Parliament and Council. Regulation (EU) 2016/679 (General Data Protection Regulation), Articles 12, 15, 20 and 22. https://gdpr-info.eu/
19. Office of the Privacy Commissioner of Canada. “Responding to access to information requests under PIPEDA.” https://www.priv.gc.ca/en/privacy-topics/accessing-personal-information/obligations-for-organizations/02_05_d_54_ati_02/
20. O'Donovan, Brian. “Appeals body to review social media account suspensions.” RTÉ News, 6 June 2025. https://www.rte.ie/news/2025/0606/1516958-appeals-centre-europe/
21. O'Donovan, Brian. “Increase in complaints to EU social media appeals body.” RTÉ News, 28 May 2026. https://www.rte.ie/news/2026/0528/1575608-social-media-complaints/
22. UK Parliament. Online Safety Act 2023, section 21, “Duties about complaints procedures.” legislation.gov.uk. https://www.legislation.gov.uk/ukpga/2023/50/section/21
23. Government of Canada, Canadian Heritage. “Backgrounder: Government of Canada introduces legislation to combat online harms, particularly those impacting children.” 10 June 2026. https://www.canada.ca/en/canadian-heritage/news/2026/06/government-of-canada-introduces-legislation-to-combat-online-harms-particularly-those-impacting-children.html
24. Meta Help Center. “About suspended or disabled Meta accounts.” https://www.meta.com/help/artificial-intelligence/639558911929531/
25. Parliament of Canada. “Bill C-34 (45th Parliament, 1st session).” LEGISinfo, accessed 7 October 2026. https://www.parl.ca/legisinfo/en/bill/45-1/c-34

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Oracle's most honest document this autumn was a cash flow statement.

It was filed with the US Securities and Exchange Commission on 11 September 2026, in the quarterly report covering the three months to 31 August, and it lays out with a bookkeeper's calm where the money came from and where it went. Revenue rose from $14.9 billion to $19.3 billion, and net income climbed 63 per cent to $4.76 billion. Capital expenditure, almost all of it data centres, came to $28.5 billion, against $8.5 billion a year earlier. Oracle sold roughly 141 million new shares through an “at-the-market” programme for net proceeds of $19.9 billion, using up the entire $20 billion facility in a single quarter. And in a note on restructuring, the company recorded that “subsequent to August 31, 2026, our management supplemented the 2026 Restructuring Plan by approximately $700 million to reflect additional actions that we expect to take,” lifting a plan previously estimated at up to $2.1 billion to roughly $2.8 billion.

The same note describes what the plan is for. It exists, Oracle says, “to implement certain strategic measures and further improve operational efficiencies, including through the adoption and integration of artificial intelligence technologies across certain functions”.

Three days after the quarterly report was filed, the additional actions began. On Monday 14 September, according to The Deep Dive and The Week, affected staff, including employees in India, lost access to company systems and received an email early that morning, signed by Oracle Leadership, telling them their roles had been eliminated and that “today is your last working day”. Oracle has not said how many people were affected. The only public count is partial: Worker Adjustment and Retraining Notification (WARN) notices dated 14 September cover 359 employees in Washington state, at two Seattle offices or working remotely, according to FOX 13 Seattle, and 441 in California, at Redwood City, Santa Clara and Pleasanton, according to TheStreet, with separations scheduled from 13 November. TheStreet counts at least 2,578 Oracle job cuts in US WARN notices so far in 2026, but those notices cover only some US sites, so the figure is a floor, not a total. Its annual report for the year to 31 May had already recorded a fall in headcount from about 162,000 to about 141,000, a net loss of 21,000 people, or 13 per cent.

The story of the email has been told, including in these pages. This piece is about the cash flow statement, because it answers a question the email never could. When a profitable, fast-growing company pays for its AI buildout partly by cutting its wage bill, who is actually underwriting the AI transition?

Five Parties Paid For The Quarter

Start with the ledger, because the popular version of this story gets the proportions wrong in an interesting way.

The popular version says Oracle sacked workers to pay for data centres. The filings say something more complicated. In the first quarter of fiscal 2027, Oracle's operations generated $23.1 billion of cash, up 184 per cent on a year earlier. Much of that came from customers, and according to chief financial officer Hilary Maxson on the earnings call, most of the new contracts signed in the quarter used customer prepayments, “bring your own hardware” deals or similar structures that will not need extra capital from Oracle. New shareholders put in $19.9 billion. Lenders already hold $125 billion of Oracle's senior notes and other long-term borrowings, after the company raised $42.7 billion in senior notes during fiscal 2026 alone. Those are the big cheques.

Now find the workers. Oracle's income statement shows that, compared with the same quarter a year earlier, sales and marketing expense fell from $2,063 million to $1,811 million, research and development from $2,491 million to $2,401 million, and general and administrative stayed flat at $376 million. These are the personnel-heavy lines; Oracle's own annual report attributes the $320 million fall in sales and marketing spending during fiscal 2026 “primarily” to lower employee-related expenses. Taken together, the three lines fell by about $342 million year on year.

Put those numbers next to each other and the arithmetic is almost embarrassing. Oracle's interest expense rose by $505 million over the same period. Depreciation rose by $1.8 billion. The quarter's capital expenditure was more than 80 times the year-on-year saving on those three people-heavy lines.

Even across a full year, the scale holds. Oracle's 2026 proxy statement puts the total compensation of its median global employee at $94,740. As a crude illustration only, since nobody knows what the departing staff actually earned, 21,000 people paid that median would cost about $2 billion a year. Oracle spent $55.7 billion on capital expenditure in fiscal 2026 and ran negative free cash flow of $23.7 billion. Payroll cuts cannot fund a buildout of that size. At most they trim its edges.

So the honest answer to “who pays for the data centres” is: customers first, then new shareholders, then bondholders, with workers a distant fourth in dollar terms. A fifth party, the taxpayer, appears later in this story.

The reason this matters is that it turns the question round. If payroll savings are too small to pay for the buildout, why cut payroll at all? The answer is that the payroll line is not where the money comes from. It is where the risk goes.

Why The Smallest Contribution Carries The Heaviest Loss

Every party on that ledger is underwriting the transition, but they are doing it on very different terms.

A shareholder who bought into Oracle's at-the-market programme chose to buy, at a price they accepted, in a liquid market they can leave tomorrow. A bondholder lends at a yield priced for the risk, and that risk has been spelled out. On 9 July 2026, S&P Global Ratings cut Oracle's long-term rating from BBB to BBB-, one notch above speculative grade, admitting that it had “underestimated the scale of the investments required to expand the AI business” and noting that OpenAI accounted for about half of Oracle's $638 billion contracted backlog. Lenders who stay have been told the risk and are being paid to carry it. In the first quarter alone, Oracle paid $1.565 billion in dividends to preferred and common stockholders and booked $1.428 billion of interest expense.

The departing employee's contribution works differently. It is not chosen, not priced and not diversified. Reporting by Business Insider, summarised by Business Today in April 2026, set out the US severance formula: four weeks of base salary for the first year of service, one additional week for each further year, capped at 26 weeks. Unvested stock, which for many technology workers is a substantial part of their pay, is forfeited. Reported terms for India, drawn from employee accounts rather than any company statement, are structured differently and vary by location.

Twenty-six weeks is half a year. The best long-run evidence on what job loss actually costs comes from the economists Steven J. Davis and Till von Wachter, whose 2011 Brookings Papers on Economic Activity study tracked American workers through Social Security records from 1974 to 2008. Men displaced in mass layoffs when national unemployment was below 6 per cent lost, on average, the equivalent of 1.4 years of their pre-displacement earnings in present-value terms, discounting over twenty years. When unemployment exceeded 8 per cent, the loss doubled to 2.8 years. The study is not specific to technology workers and is not a prediction for any individual. But it sets a benchmark, and the maximum severance Oracle offers in the United States covers roughly a third of the typical loss in good times.

There is one more asymmetry. In the first quarter, according to the 10-Q, Oracle made $324 million of cash payments under the 2026 Restructuring Plan. Dividend payments in the same quarter were nearly five times that figure. Both are legitimate obligations. One is a return on capital that its owners chose to commit. The other is the closing payment on a contribution its providers never agreed to make.

How Severance Learns To Look Like Modernisation

To see how a person becomes a modernisation cost, follow the accounting.

Under US generally accepted accounting principles, severance sits in one of two places. Benefits paid under an ongoing, established plan fall under ASC 712 and are accrued when they are probable and can be reasonably estimated. One-off packages fall under ASC 420, the standard for exit and disposal costs, and, as PwC's accounting guidance sets out, a liability is generally recognised when the plan has been communicated to employees in enough detail for each of them to work out what they would receive. Either way, the cost lands in a line Oracle calls “restructuring and other”.

That line is then removed from the number most investors watch. Oracle's first-quarter earnings release reconciles GAAP results with non-GAAP ones, and restructuring is one of the items stripped out, alongside stock-based compensation and the amortisation of acquired intangible assets. In the quarter, GAAP operating margin was 35 per cent. Non-GAAP operating margin, with restructuring and the other adjustments removed, was 42 per cent. The non-GAAP earnings per share figure that led the headline, $1.92, carries no trace of severance.

Excluding restructuring is standard practice and Oracle discloses every adjustment in full. The point is what the convention implies. A restructuring charge is presented as something outside the business's ongoing performance, a one-off cost of transition. Yet Oracle has recorded restructuring expense every year: $404 million in fiscal 2024, $299 million in fiscal 2025, $1.779 billion in fiscal 2026. Its quarterly report says the company expects “to incur additional restructuring expenses in future periods”. The cost recurs, but it is reported as if it were exceptional.

The most revealing sentence, though, is not in the accounting notes. It is in management's discussion, where Oracle explains what happened to the money it saved. Certain of the savings from the 2026 Restructuring Plan, it says, “were offset by investments in resources and geographies that we believe better address the development, marketing, sale and delivery of our cloud-based offerings, including investments in the development and delivery of our second-generation cloud infrastructure.”

That is the reallocated payroll, described by the company that reallocated it. Savings from one set of people and places were moved to other resources and other geographies. Severance is booked as the cost of making that move, then removed from adjusted earnings, so the move looks almost free.

The Taxpayer Was Underwriting Before Anyone Was Fired

The fifth party on the ledger did not sign anything either, and its contribution started long before September.

In 2020, Daron Acemoglu, Andrea Manera and Pascual Restrepo published a paper through the Brookings Institution and the National Bureau of Economic Research asking whether the US tax code favours automation. They found that it does. Effective tax rates on labour stood above 28.5 per cent, while the effective rate on capital invested in equipment and software had fallen to about 5 per cent. Their conclusion was that the heavy taxation of labour and light taxation of capital encourage firms to automate more tasks, and use less labour, than is socially optimal. Under the tax structure they modelled as optimal, labour would be taxed at around 9 per cent and capital at around 22 per cent.

The authors traced much of that low capital rate to generous depreciation provisions passed between 2002 and 2017. The most generous of them has since been made permanent. The 100 per cent bonus depreciation introduced in 2017 had been phasing down, reaching 40 per cent for 2025, until legislation signed in July 2025, commonly known as the One Big Beautiful Bill Act, permanently restored it for qualifying property acquired after 19 January 2025. The Internal Revenue Service issued interim guidance in Notice 2026-11 in January 2026. In general, qualifying property means assets with a recovery period of 20 years or less, which covers the servers, storage and networking equipment that fill a data centre, though not the buildings. A company buying those assets can deduct the whole cost in the year it places them in service, rather than spreading it over their useful life.

Oracle's effective tax rate in the first quarter of fiscal 2027 was 15.1 per cent. Its filings do not break out how much of that reflects accelerated depreciation, and nothing here suggests Oracle is doing anything other than using the law as written. But the structure is clear. Each dollar of salary carries payroll taxes. Each dollar of qualifying data centre equipment can now be written off immediately. When a company shifts spending from people to machines, public policy is pushing it the same way.

Then there is the safety net. In the United States, displaced workers can turn to state unemployment insurance, which typically replaces only part of lost wages for a limited period. A large share of Oracle's workforce sits outside that system altogether: of roughly 141,000 employees at the end of fiscal 2026, about 92,000 worked outside the United States, where severance, notice and benefits are governed by national law and, in many places, by much thinner public provision. Whatever severance does not cover is met by the worker's savings, by family, and by whatever state they happen to live in.

Shareholders Are Paying Too, On Different Terms

None of this means shareholders are getting a free ride, and the strongest version of Oracle's case deserves a hearing.

The share price has taken a beating. Fortune reported on 26 September 2026 that Oracle had closed at $137, down 53 per cent over the previous twelve months. The new shares sold through the at-the-market programme went for an average of about $141 each, and the share count rose by about 5 per cent in a single quarter, from 2.88 billion to 3.02 billion. Anyone holding Oracle stock owns a smaller slice of a company that S&P projected in July would run a free cash flow deficit of nearly $42 billion in fiscal 2027, up from the agency's earlier forecast of about $24 billion. Seven years ago this ledger ran the other way: in fiscal 2019, Oracle spent $36.1 billion buying back its own shares. In the most recent quarter it bought back none.

Even the executives' headline pay is less certain than it looks. Oracle's 2026 proxy statement reports fiscal 2026 total compensation of $627.5 million for co-chief executive Clayton Magouyrk and $256.6 million for co-chief executive Michael Sicilia, against $94,740 for the median employee, a ratio of 6,623 to 1 for Magouyrk. Those totals are largely the accounting value of one-time stock option awards made when the two were appointed in September 2025, awards the proxy itself values at $250 million and $100 million at target. Fortune reported that by the end of the fiscal year all of the options granted to Magouyrk, Sicilia and executive chair Larry Ellison were underwater, and the proxy says the awards “had no intrinsic value” at that date. They may never pay out.

Ellison, who according to the proxy beneficially owns 38.2 per cent of Oracle's common stock, has also signalled that he is staying put. Oracle's quarterly report disclosed that he adopted a trading plan on 22 June 2026 allowing him to sell up to 50 million shares. On 12 September, the company announced that he had cancelled it, that no shares had been sold under it, and that he had no other plans to sell.

So the people at the top are exposed to the same bet, and in paper terms they have lost a great deal on it. The difference is not whether they bear risk but what kind. A stake of 1.158 billion shares, at the $2.00 per share in common dividends Oracle declared in fiscal 2026, would generate roughly $2.3 billion a year in cash, more than the entire $1.779 billion restructuring expense of fiscal 2026. A falling share price reduces wealth that remains. A severance letter ends income that was the wealth. Both are losses, but only one can be recovered by waiting.

A Record Low For Labour's Share Of Output

Oracle is one company, but its pattern fits a national trend that was under way long before anyone mentioned large language models.

On 15 September 2026, the day after Oracle's emails went out, the US Bureau of Labor Statistics published a short note. Labour share in the nonfarm business sector, the fraction of output paid to workers as wages, salaries and benefits, was 52.8 per cent in the second quarter of 2026, “the lowest level ever recorded” in a series that begins in 1947. For most of the post-war period it ranged between 60 and 66 per cent. It began a sustained decline after the turn of the century, and it stood at 56.3 per cent at the end of 2019.

The productivity figures released a few days earlier explain why the decline matters. Nonfarm business productivity rose 1.4 per cent at an annual rate in the second quarter and 2.2 per cent on a year earlier. Real hourly compensation, meanwhile, fell at an annual rate of 3.3 per cent in the quarter. Output per hour is rising; the share of it paid to the people producing it is falling.

It would be wrong to blame that decline on AI, and the BLS makes no such claim. The trend predates the technology by two decades, and economists still disagree about what drives it. But the Oracle quarter is a small, precise example of how the macro statistic is made: revenue up 30 per cent, operating income up 57 per cent, personnel-heavy expense lines down, capital spending more than tripled. At the national level, labour share is just a ratio of totals. It falls one company at a time, through decisions like these.

Microsoft, Meta, Amazon And Salesforce Run Variations

Is Oracle the leading edge of a new corporate normal? The comparable firms suggest the pattern is real but varies, and the variations show who is underwriting the change.

Microsoft employed about 223,000 people at 30 June 2026, according to its annual report, down from about 228,000 a year earlier. In April, chief financial officer Amy Hood told investors that headcount had fallen year on year and was expected to keep falling, while the company booked about $900 million of charges for a voluntary retirement programme. Microsoft's additions to property and equipment rose by almost 80 per cent, from $64.6 billion in fiscal 2025 to $115.9 billion in fiscal 2026. And it still returned $48.7 billion to shareholders over the year, $22.3 billion in buybacks and $26.4 billion in dividends. Microsoft can pay for the buildout out of its own cash flow and keep paying shareholders. Its workforce is shrinking anyway.

Meta told staff in April that it would cut roughly 10 per cent of its workforce, about 8,000 people. Its second-quarter results recorded $1.18 billion of severance expense tied to the May reduction, headcount of 75,472, down 1 per cent on a year earlier, and revenue up 28 per cent to $60.8 billion. It narrowed its 2026 capital expenditure outlook to between $130 billion and $145 billion. Free cash flow for the quarter came to just $784 million.

Amazon announced 14,000 corporate job cuts in October 2025 and around 16,000 more on 28 January 2026, together about a tenth of its corporate workforce. By its second quarter of 2026 it was planning capital spending of around $220 billion for the year, and free cash flow on its trailing measure had turned negative, at minus $7.6 billion. Andy Jassy, the chief executive, has explicitly rejected the AI reading of his cuts. “The announcement that we made a few days ago was not really financially driven, and it's not even really AI-driven, not right now at least,” he told analysts in October 2025. “It's culture.”

Salesforce gave the clearest statement of reallocation. In September 2025, chief executive Marc Benioff said on The Logan Bartlett Show that AI agents had allowed the company to cut its support headcount from about 9,000 to about 5,000. “I've rebalanced my support headcount,” he said, “so I can now put those people into sales.”

Together the cases show both the pattern and its limits. Every one of these firms is growing revenue, holding headcount flat or cutting it, and raising capital spending to levels without precedent. Every one has made workforce reductions while doing so. But the headcount changes at Microsoft and Meta are small in percentage terms, Amazon's chief executive denies the causal link, and the cash that pays for the data centres overwhelmingly comes from somewhere other than payroll. The debt markets show where. JPMorgan, in research reported by Bloomberg in August 2026, raised its forecast for technology, media and telecoms bond issuance this year to $540 billion and estimated that hyperscaler debt alone would reach $317 billion.

Oracle stands out because it has the weakest balance sheet of the group. Microsoft funds the buildout from cash flow. Meta and Amazon are pushing their free cash flow towards zero and below. Oracle has had to sell equity at depressed prices, borrow at the lowest investment-grade rating, and ask customers to prepay. When every source of funding is under strain, the one that requires nobody's consent comes under the most pressure. That makes Oracle less an exception than a stress test. It shows what the pattern looks like when the balance sheet runs out of room.

Payroll Has Become The Adjustable Line

So is this the new normal? The evidence supports a narrower claim than the headlines, and a more durable one.

It does not support the idea that AI is about to empty corporate offices. Microsoft's headcount fell about 2 per cent in a year and Meta's 1 per cent. The cash going into data centres comes overwhelmingly from operating income, bonds and new shares. Even at Oracle, the quarter's saving on personnel-heavy expense lines came to roughly 1 per cent of its capital expenditure.

What the evidence does support is a change in the order in which claims on a company's cash are met. In the buyback era, which at Oracle peaked with $36.1 billion of repurchases in fiscal 2019, shareholders came first and payroll was a fixed cost of doing business. In the AI buildout, capital expenditure comes first, contractually locked in through long-term leases and supplier commitments that Oracle's own annual report warns could leave it “locked into multi-year commitments” for surplus capacity if demand disappoints. Shareholders come next, still paid their dividends while buybacks are largely paused. Payroll has become the variable that absorbs the difference: the one line that can be cut within a quarter, by email, without renegotiating anything with anyone.

That is what the Oracle pattern reveals about who underwrites the AI transition. Customers and investors pay most of the bill, and they choose to. Taxpayers subsidise it through a code that, on the best available evidence, taxes labour far more heavily than equipment. Workers carry the part of the risk that has nowhere else to go, at a price nobody negotiated with them, and their losses are recorded in a line that the preferred measure of corporate performance leaves out.

The arrangement is not unique to AI. What AI adds is a justification. Oracle's filings tie the restructuring explicitly to “the adoption and integration of artificial intelligence technologies across certain functions,” and its annual report states that the adoption and deployment of AI “have resulted, and may continue to result, in reductions to our workforce.” Whether AI actually did the work of the people who left, or merely made it easier to explain why they were no longer affordable, is something no filing can settle. Either way, the effect on the ledger is the same.

The Ledger Line That Cannot Be Refinanced

What becomes of the workers whose severance is booked as a cost of modernisation?

The evidence offers a sobering but bounded answer. The US unemployment rate was 4.2 per cent in September 2026, according to the Bureau of Labor Statistics, which places today's American job losers in the better of the two scenarios Davis and von Wachter measured. Their benchmark for such times is a loss equivalent to well over a year's earnings, spread across the years after displacement, rather than permanent exclusion from work. Some, particularly in countries with weak protections and in roles that AI tools can increasingly reach, will find it harder. In every case, their severance and their losses will appear in Oracle's accounts only once, in the restructuring line, and then be adjusted away.

A different arrangement is imaginable, and some of it is already on the record. Acemoglu, Manera and Restrepo argued that rebalancing the tax treatment of labour and capital would raise employment. Severance formulas, notice periods and the treatment of unvested stock are policy choices, set by companies and in part by law. The disclosure rules that already make Oracle state a cost estimate for its restructuring plan, down to the $700 million supplement, could be extended to say how many people that cost represents, and where.

Until then, the ledger works like this. Shareholders who lost half the value of their stock this year can wait for a recovery. Bondholders are paid interest for the risk they knowingly took. The company can raise capital again next quarter, and almost certainly will. The people on the restructuring line have no equivalent. A career, once booked under “restructuring and other”, cannot be refinanced.

On 18 November 2026, Oracle's shareholders will cast an advisory vote on the pay of its named executive officers. The people who left on 14 September get no ballot. Their contribution to the AI transition has already been recorded, and then adjusted out of the figures.

References & Sources

1. Oracle Corporation, “Form 10-Q for the quarterly period ended 31 August 2026”, US Securities and Exchange Commission, September 2026. https://www.sec.gov/Archives/edgar/data/1341439/000119312526389274/orcl-20260831.htm
2. Oracle Corporation, “Oracle Announces Q1 Results Driven by Triple Digit Growth in Cloud Infrastructure Revenues”, Exhibit 99.1 to Form 8-K, 10 September 2026. https://www.sec.gov/Archives/edgar/data/1341439/000119312526387905/orcl-ex99_1.htm
3. Oracle Corporation, “Form 10-K for the fiscal year ended 31 May 2026”, US Securities and Exchange Commission, 2026. https://www.sec.gov/Archives/edgar/data/0001341439/000119312526277521/orcl-20260531.htm
4. Oracle Corporation, “Larry Ellison Cancels His Plan to Sell Oracle Stock”, Exhibit 99.1 to Form 8-K, 12 September 2026. https://www.sec.gov/Archives/edgar/data/1341439/000119312526389753/d20034dex991.htm
5. Oracle Corporation, “Definitive Proxy Statement (Schedule 14A) for the 2026 Annual Meeting of Stockholders”, 25 September 2026. https://www.sec.gov/Archives/edgar/data/0001341439/000119312526402816/orcl-20260925.htm
6. Oracle Corporation, “Form 10-K for the fiscal year ended 31 May 2019”, US Securities and Exchange Commission, June 2019. https://www.sec.gov/Archives/edgar/data/1341439/000156459019023119/orcl-10k_20190531.htm
7. The Motley Fool, “Oracle (ORCL) Q1 2027 Earnings Call Transcript”, 11 September 2026. https://www.fool.com/earnings/call-transcripts/2026/09/11/oracle-orcl-q1-2027-earnings-call-transcript/
8. JP Alegre, “Oracle Starts New Layoff Round Six Months After Its Last One”, The Deep Dive, 14 September 2026; FOX 13 Seattle, “Oracle to lay off 359 Washington workers in November”, 16 September 2026; and Aparajita Chatterjee, “Oracle layoffs top 2,500 workers as AI spending surges”, TheStreet, 23 September 2026. https://thedeepdive.ca/oracle-starts-new-layoff-round-six-months-after-its-last-one/ and https://www.fox13seattle.com/news/oracle-lay-off-washington-workers and https://www.thestreet.com/employment/oracle-layoffs-top-2500-workers-ai-spending-surge-2026
9. Meera Rajeev, “Oracle layoffs: What severance pay can Indian workers expect?“, The Week, 15 September 2026. https://www.theweek.in/news/biz-tech/2026/09/15/oracle-layoffs-what-severance-pay-can-indian-workers-expect.html
10. Business Today Desk, “Oracle layoffs 2026: How much severance US employees are getting?“, Business Today, 7 April 2026. https://www.businesstoday.in/technology/story/oracle-layoffs-2026-how-much-severance-us-employees-getting-524393-2026-04-07
11. Amanda Gerut, “Oracle gave Larry Ellison and his new co-CEOs nearly $1 billion in stock options. By fiscal year end, all were underwater”, Fortune, 26 September 2026. https://fortune.com/2026/09/26/oracle-larry-ellison-ceo-pay/
12. Luis Rijo, “S&P cuts Oracle to BBB- as AI buildout widens cash deficit to $42bn”, PPC Land, 18 July 2026. https://ppc.land/s-p-cuts-oracle-to-bbb-as-ai-buildout-widens-cash-deficit-to-42bn/
13. Bloomberg, “JPMorgan sees tech bond sales surpassing half a trillion dollars in 2026”, republished by The Star, 10 August 2026. https://www.thestar.com.my/business/business-news/2026/08/10/jpmorgan-sees-tech-bond-sales-surpassing-half-a-trillion-dollars-in-2026
14. Microsoft Corporation, “Form 10-K for the fiscal year ended 30 June 2026”, US Securities and Exchange Commission, July 2026. https://www.sec.gov/Archives/edgar/data/789019/000119312526323660/msft-20260630.htm
15. Alexei Alexis, “Microsoft CFO flags workforce cuts as AI spending surges”, CFO Dive, 30 April 2026. https://www.cfodive.com/news/microsoft-cfo-flags-workforce-cuts-ai-spending-surges-layoffs/819027/
16. Meta Platforms, “Meta Reports Second Quarter 2026 Results”, Exhibit 99.1, 29 July 2026. https://s21.q4cdn.com/399680738/files/doc_financials/2026/q2/Meta-06-30-2026-Exhibit-99-1-FINAL.pdf
17. Bloomberg Law, “Meta Tells Staff It Will Cut 10% of Jobs in Push for Efficiency”, April 2026. https://news.bloomberglaw.com/tech-and-telecom-law/meta-tells-staff-it-will-cut-10-of-jobs-in-push-for-efficiency
18. Paul Thurrott, “Amazon Revenues Jump 20 Percent to $200.6 Billion”, Thurrott.com, 30 July 2026; and ABC7 New York, “Amazon cuts 16,000 jobs in the latest round of layoffs”, 28 January 2026. https://www.thurrott.com/amazon/339894/amazon-revenues-jump-20-percent-to-200-6-billion and https://abc7ny.com/post/amazon-cuts-16000-jobs-latest-round-layoffs/18491495/
19. Fortune, “Amazon CEO Andy Jassy says layoffs are about culture, not AI”, 1 November 2025. https://fortune.com/2025/11/01/ceo-andy-jassy-amazon-layoffs-about-culture-not-ai
20. Outlook Business Desk, “AI Agents Replaced 4,000 Customer Support Jobs at Salesforce, Says CEO Marc Benioff”, Outlook Business, 2 September 2025. https://www.outlookbusiness.com/artificial-intelligence/ai-agents-replaced-4000-customer-support-jobs-at-salesforce-says-ceo-marc-benioff
21. US Bureau of Labor Statistics, “Labor share at its lowest level, 52.8 percent, in second quarter 2026”, The Economics Daily, 15 September 2026. https://www.bls.gov/opub/ted/2026/labor-share-at-its-lowest-level-52-8-percent-in-second-quarter-2026.htm
22. US Bureau of Labor Statistics, “Productivity and Costs, Second Quarter 2026, Revised”, 3 September 2026; and “The Employment Situation, September 2026”, 2 October 2026. https://www.bls.gov/news.release/prod2.nr0.htm and https://www.bls.gov/news.release/empsit.nr0.htm
23. Daron Acemoglu, Andrea Manera and Pascual Restrepo, “Does the US Tax Code Favor Automation?”, Brookings Papers on Economic Activity, Spring 2020, and NBER Working Paper 27052. https://www.brookings.edu/articles/does-the-u-s-tax-code-favor-automation/
24. BDO USA, “IRS Issues Interim Guidance on Bonus Depreciation Rules” (on IRS Notice 2026-11), January 2026; and PwC Viewpoint, “Restructurings and one-time termination benefits”, Pensions and Employee Benefits Guide, section 8.5. https://www.bdo.com/insights/tax/irs-issues-interim-guidance-on-bonus-depreciation-rules and https://viewpoint.pwc.com/dt/us/en/pwc/accounting_guides/pensions-and-employee-benefitspeb/peb_guide/Chapter-8-PEB/85_Restructurings_and_onetime_termination_benefits_20.html
25. Steven J. Davis and Till von Wachter, “Recessions and the Costs of Job Loss”, Brookings Papers on Economic Activity, Fall 2011. https://www.brookings.edu/bpea-articles/recessions-and-the-costs-of-job-loss/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Somewhere in a comment thread under an image of a very thin young woman, a 12-year-old girl is asking for the workout routine. Nobody will answer her truthfully, because no routine exists. The woman in the picture never trained. She never ate, never slept, never stood in front of a mirror. She was generated.

That scene is not an invention for the purposes of this article. It is a pattern that Mia Perumal, a 19-year-old student at Cardiff University who has recovered from an eating disorder, described to the BBC in a report published on 4 October 2026. Over the past year, she said, she has watched very thin AI-generated people multiply across her social media feeds, and she has seen comments from girls as young as 12 asking how to get the body on screen. “It's all very well saying 'oh, it's AI, it's not real',” she told the BBC. “But eating disorders don't really listen to rationality.”

The same report carried a warning from Phillippa Diedrichs, professor of psychology at the Centre for Appearance Research at the University of the West of England in Bristol, whose research spans body image, eating disorder prevention and advertising. Generative systems, she explained, tend to fall back on narrowly defined traditional beauty standards unless someone deliberately prompts them to do otherwise. The result is that people of colour, and people with distinctive features, quietly disappear from the pool of images that young people measure themselves against. “If we're comparing ourselves to a very narrow beauty standard,” she said, “... that inevitably means that many of us are not going to meet those beauty ideals.”

Put those two observations side by side and a set of uncomfortable questions emerges. What happens to the way young people see their own bodies when the bodies they are invited to compare themselves with are not merely idealised, but invented? Does knowing that an image is synthetic offer any protection at all? And if the default image produced by these systems is a choice rather than a law of nature, what would happen if the industry decided to change it on purpose?

The research, some decades old and some published only this year, gives surprisingly clear answers. They are not reassuring.

Fashion Has Already Hired Models Who Do Not Exist

The BBC's warning landed in an industry that has been edging towards synthetic people for some time. In the August 2025 issue of American Vogue, an advertisement for Guess featured a flawless blonde model in a striped maxi dress and a floral playsuit. She had been created by Seraphinne Vallora, a London-based AI marketing agency founded by Valentina Gonzalez and Andreea Petrescu. A small line of text at the edge of the page disclosed that the model was AI-generated. It was the first time a synthetic person had appeared in the magazine, and the backlash was immediate.

Vanessa Longley, chief executive of the UK eating disorder charity Beat, told the BBC at the time that the development was worrying, warning that exposure to unrealistic bodies can damage self-esteem and that poor body image increases the risk of developing an eating disorder.

Guess was not alone. In March 2025, H&M announced it would create AI “digital twins” of 30 real models, working with the models and their agencies, and a campaign using those twins followed in the summer. The BBC's October report listed Balmain and Levi's among the brands that have experimented with AI models. The commercial logic is obvious: a synthetic model never ages, never gets ill and never needs a flight.

What the commercial logic leaves out is the person looking at the result. As Hannah Marriott, a lecturer in marketing and strategy at Cardiff Business School, argued in a piece for The Conversation in August 2026, consumers have long been able to recognise that a photograph has been heavily edited while still believing that a human being exists behind it. Synthetic models sever that link. “Authenticity may depend less on what a model looks like,” she wrote, “and more on whether consumers perceive them as genuinely human.”

Adult consumers, it seems, can be put off by synthetic models when they know what they are looking at. The question is whether that conscious scepticism reaches the part of the mind where body image actually lives.

Why the Mind Compares Itself With Strangers

In 1954 the American social psychologist Leon Festinger published “A Theory of Social Comparison Processes” in the journal Human Relations. His central proposal has proved remarkably durable: when there is no objective yardstick available, people evaluate themselves by comparing with others. Festinger also argued that people prefer to compare themselves with others who are reasonably similar. A novice chess player does not measure herself against a grandmaster.

Seventy years of body image research have complicated that last point. Appearance is a domain with no objective yardstick at all, and people compare themselves constantly with others who are not remotely similar: models, actors, athletes, influencers. Upward comparison with people judged more attractive is consistently associated with worse mood and greater body dissatisfaction. This is the engine behind what researchers call the thin ideal and, more recently, the fit ideal: the culturally specific picture of a desirable body that people absorb, or internalise, from the images around them.

Social media made that engine run faster by collapsing the distance between celebrity and peer. An influencer appears in the same feed as school friends and replies to comments. Festinger's similarity principle has been exploited rather than overturned. Synthetic influencers take this one step further: they are designed to look like people who might plausibly be in your feed, with gym selfies, beach shots and outfit posts. The 12-year-old asking for a workout routine is not making a mistake about genre. She has correctly read the visual cues that tell her this is a peer-like body, attainable through effort, and she has responded exactly as social comparison theory predicts. The problem is that the cues are false. The body was never attained through effort, because it was never attained at all.

There is a further twist. Even the most heavily retouched photograph of a real model began with a real body, constrained by anatomy. A retoucher could narrow a waist or lengthen a leg, but there was a skeleton underneath and a person who could, at least in principle, be met. A generated body has no such constraint. The comparison target is not merely an exaggerated human. It is a statistical composite presented as an individual.

What the Machines Learnt About Beauty

If generative systems produced a genuinely random spread of body types, the risk might be diluted. They do not. A steady accumulation of audits since 2023 shows that, left to their own devices, image generators reproduce and frequently amplify the narrowest version of conventional attractiveness.

In June 2023, Bloomberg published an analysis by Leonardo Nicoletti and Dina Bass of more than 5,000 images produced by Stable Diffusion. Under the headline “Humans Are Biased. Generative AI Is Even Worse”, it reported that the model took racial and gender disparities to extremes worse than those found in the real world.

A year later, in June 2024, The Washington Post asked three of the leading tools, Midjourney, DALL-E and Stable Diffusion, to generate pictures of a “beautiful woman”. Every single image depicted a thin woman. Just 2 per cent showed visible signs of ageing, and only 9 per cent showed women with dark skin tones. Diedrichs's warning about who gets left out is not a hypothesis. It has been measured.

The pattern holds when researchers look specifically at bodies. In a study published in Psychology of Popular Media and announced by the University of Toronto in November 2025, Delaney Thibodeau, a postdoctoral researcher in the Faculty of Kinesiology and Physical Education, and colleagues including Professor Catherine Sabiston systematically coded 300 images generated by Midjourney, DALL-E and Stable Diffusion. AI-generated women were overwhelmingly thin and AI-generated men hyper-muscular. Images of athletes were far more likely to show very low body fat and highly defined muscularity. Ninety per cent of images of an unspecified “athlete” showed a male body. Racial and age diversity was minimal, and not a single image showed a visible disability.

A preprint released in April 2026 by Justin Thomas and colleagues, titled “Underweight and Overrepresented”, went further. The researchers generated images of women in various occupations using prompts that said nothing about body size, then asked 129 American women to rate the figures' body size. The proportion of images perceived as underweight was more than three times the expected rate of underweight among American women. Women in higher-prestige occupations were more likely to be rated underweight. The study has not yet completed peer review, but its direction matches the wider literature.

A Fordham University study presented at a 2025 computational linguistics conference found the mirror image: DALL-E 3 tended to depict words such as “greedy” or “immoral” as heavier people. And in 2023 the Center for Countering Digital Hate found that image generators produced pictures glorifying unrealistic bodies in response to 32 per cent of prompts drawn from eating disorder forums.

None of this is mysterious. Generative systems are trained on vast collections of internet images saturated with advertising, fashion and stock photography, the very images body image researchers have spent decades criticising. They learn the historical bias of that visual culture and output its statistical centre with perfect lighting. Diedrichs's point about defaults is precisely this. The narrow ideal is not something the system has to be told to produce. It is what comes out when nobody tells it anything.

A Decade of Labels That Did Not Work

The intuitive response, and the one Mia Perumal says people keep offering her, is disclosure: tell young people the bodies are fake. Body image researchers have been testing that intuition for well over a decade with retouched photography, and the results are consistently discouraging.

In 2013, Marika Tiggemann and colleagues at Flinders University in Adelaide published a pair of experiments in the journal Body Image. Female undergraduates viewed fashion advertisements with no label, a generic warning that the image had been digitally altered, or a more specific warning. The labels produced no benefit. Worse, for women who habitually compared their appearance with others, the specific warning labels were associated with increased body dissatisfaction.

In 2018, Jasmine Fardouly and Elise Holland tested the social media version of the idea. They showed 164 young American women idealised Instagram images with or without disclaimer comments. Viewing the idealised images harmed body image regardless of the disclaimer. The only measurable effect of the disclaimers was that women formed a less positive impression of the person who posted them.

By 2020 there was enough evidence for a systematic review. Sarah McComb and Jennifer Mills of York University in Toronto examined 15 experimental studies on media disclaimers, publishing their findings in Body Image. Eleven found disclaimers ineffective at reducing body dissatisfaction after exposure to thin-ideal images. Three found medium-sized benefits. One found that disclaimers made things worse. Particularly troubling, specific and warning-style disclaimers appeared harmful for women who were already highly dissatisfied with their bodies, and in one case women with restrictive eating habits who saw a warning label went on to eat fewer calories than those who saw the same image without one.

A warning that draws attention to a body can increase the attention paid to it. For someone already vulnerable, a label saying “this waist has been altered” may simply confirm that the waist is the thing worth looking at.

Telling People It Is AI Changes Trust, Not Feelings

Perhaps AI is different. A label saying a body has been slimmed still implies a real body underneath; a label saying the whole person is synthetic seems more decisive. This year researchers began testing that directly, and the early answer is no.

In February 2026, the International Journal of Eating Disorders published a study by Brooke Bennett and colleagues at Clemson University in South Carolina. They recruited 296 women who were regular social media users and randomly assigned them to one of two groups. Both viewed ten AI-generated Instagram images of female content creators; only one was told they were AI-generated.

Both groups reported significantly lower body satisfaction after viewing the images. Knowing that the images were AI-generated made no significant difference to that decline. What disclosure did change was trust: the informed group showed a steeper drop in their trust in technology. The authors concluded that viewing AI-generated images of influencers negatively affects body image regardless of whether the role of AI is known, and that labels “may help keep the user informed” but “are not enough”.

A second experiment, published this year in Computers in Human Behavior: Artificial Humans by Jessica Alleva, Tuğba Türkcan, Linda Lin, Chrisa Sloutas and Jasmine Fardouly, reached a similar conclusion. The researchers showed 291 women aged 18 to 31 one of three sets of AI-generated images: bodies conforming to the fit ideal, bodies emphasising functionality and diversity, or neutral objects. Half of each group was told the images were made with generative AI. Women who saw fit-ideal imagery reported lower body appreciation than those in the control condition. Disclosure produced a small increase in one measure, appreciation of what the body can do, but made no difference to body appreciation or appearance satisfaction. And the effects of the imagery, positive and negative, were strongest among women who already tended to compare their appearance or had internalised appearance ideals.

Dove's own large-scale survey work, conducted with Edelman for its 2024 “Real State of Beauty” report across more than 33,000 respondents in 20 countries, found that one in three women felt pressure to alter their appearance because of what they saw online, even when they knew it was fake or AI-generated.

Put together, these findings make Mia Perumal's observation look less like intuition and more like a summary of the evidence.

The Gap Between Knowing and Feeling

Why should conscious knowledge offer so little protection? The research does not settle the mechanism, but the literature suggests several plausible explanations.

The first is that appearance comparison is fast and largely automatic. The comparison happens first, and any correction comes later, if at all. A label is read after the image has already been seen and evaluated.

The second is internalisation. Once a person has absorbed a particular ideal as the standard of what bodies should look like, every image that matches the ideal reinforces it, whether or not that specific image is real. A synthetic body does not need to be believed in to confirm what the viewer already thinks a good body looks like.

The third concerns vulnerability, and it is the most important for the girls Mia Perumal describes. Several of the studies cited here, from Tiggemann's in 2013 to Alleva's in 2026, found the harm concentrated among people who already compared themselves heavily or had already internalised appearance ideals. Those are precisely the people most at risk of developing eating disorders, and adolescence is a period when appearance comparison and body concern often intensify.

The scale of that risk in the UK is not small. NHS England's 2023 follow-up to its Mental Health of Children and Young People survey found that 12.5 per cent of 17 to 19 year olds in England had an eating disorder, up from 0.8 per cent in 2017. Among young women in that age group the figure was 20.8 per cent. Methods differed between survey waves, so the comparison should be treated with caution, but the direction is hard to ignore.

Eating disorders are not simply mistaken beliefs that the right information will correct. They involve compulsions, fears and distorted perceptions that resist argument. Telling someone in the grip of one that the image is not real is a little like telling someone with a phobia that the spider is harmless. The statement is true. It is also beside the point.

Laws Written for the Airbrush

Governments have tried to regulate idealised imagery before, and most interventions have been built on the disclosure model the research now calls into question.

Israel was an early mover. Its so-called “Photoshop law”, passed by the Knesset in March 2012 and in force from 1 January 2013, required fashion and commercial models to have a body mass index of at least 18.5, and required advertisers to disclose when images had been digitally altered to make models look thinner.

France followed. Under a 2016 health law and a decree that took effect on 1 October 2017, commercial photographs in which a model's body shape or silhouette had been altered by image-editing software must carry the words “photographie retouchée”. Breaches carry fines of up to 37,500 euros, or up to 30 per cent of the cost of the advertisement.

Norway went further still. Since 1 July 2022, its Marketing Control Act has required a standardised label on paid advertising, including sponsored influencer posts, in which body shape, size or skin has been altered by retouching or other manipulation, including filters. Its stated aim is to make consumers, particularly children and young people, aware that the people in advertisements deviate from reality.

In the UK, the Conservative MP Luke Evans introduced a Digitally Altered Body Images Bill in September 2020, and again as a Ten Minute Rule Bill in January 2022, proposing a logo for advertisements in which a body's proportions had been digitally altered. Neither version progressed. Instead, the Advertising Standards Authority has acted case by case. In February 2021, following the #filterdrop campaign started by the make-up artist Sasha Pallari, the ASA ruled that two tanning brands' influencer advertisements were misleading because filters exaggerated what their products could do.

All of these interventions share an assumption: that there is a real body somewhere, and that the problem is the gap between that body and its altered image. Synthetic models dissolve that assumption. There is no original body to deviate from. A Norwegian-style label stating that a body has been retouched makes little sense when the whole body has been generated, and the research suggests that changing the wording to “AI-generated” would not make the label work any better.

The newer generation of rules aimed specifically at AI largely inherits the same logic. The EU's AI Act, whose transparency obligations under Article 50 began applying on 2 August 2026, requires providers to mark synthetic images in a machine-readable format and requires deployers to disclose deepfakes, which the Act defines broadly to include content that resembles real or plausible persons. The Digital Omnibus on AI, Regulation (EU) 2026/1744, which entered into force on 27 July 2026, deferred most high-risk obligations but left these transparency rules in place. Its one concession gives providers of generative systems already on the market before 2 August 2026 until 2 December 2026 to comply with the machine-readable marking duty in Article 50(2). Machine-readable marking is aimed at detection tools rather than human eyes, and disclosure is precisely what the research finds wanting.

The UK regulator has been commendably honest about this limitation in a related context. In guidance published on 29 May 2025, the ASA stated that “disclosure alone is very unlikely to mitigate the harm caused by a fundamentally misleading message”. It warned that using an AI-generated image to show the effect of a cosmetic product that does not reflect real-world results is likely to be materially misleading, whatever the label says. Yet the same guidance noted that making clear an influencer is AI-generated “could well help to negate an otherwise misleading impression”. That may hold for whether a consumer has been misled about a product. It is a different question from whether a child's body image has been harmed.

The Online Safety Act 2023 does not fill the gap either. Since 25 July 2025, platforms have been under a legal duty to protect children from “primary priority content”, which includes material that encourages, promotes or gives instructions for an eating disorder. But a polished image of an impossibly thin synthetic fitness influencer does not, on its face, encourage an eating disorder. It simply sits in the feed, looking aspirational, accumulating comments from 12-year-olds.

Defaults Are Decisions

If disclosure is weak, the logic of the evidence points somewhere else: not to warning people about the images, but to changing the images themselves. This is where the third question raised by the BBC report becomes concrete. What would it mean for appearance standards if the industry default were changed by design rather than left to reflect whatever the systems have learnt?

Nothing about the current default is neutral. A generator that produces only thin women when asked for a beautiful one has not discovered a fact about beauty. It has reproduced decades of commercial decisions by advertisers, editors and casting directors, and leaving that output unchanged preserves those decisions at enormous scale.

Changing defaults is not a radical idea in imaging. In September 2017, Getty Images, the world's largest stock photography supplier, told contributors that from 1 October it would no longer accept “any creative content depicting models whose body shapes have been retouched to make them look thinner or larger”. That was a design decision at the level of supply, not a label at the level of the viewer.

A major brand has also tried to intervene at the level of the prompt. In April 2024, marking 20 years of its Campaign for Real Beauty, Dove pledged never to use AI to create or distort images of women in its advertising, and published what it called the Real Beauty Prompt Guidelines: guidance on the language needed to get generative tools such as Midjourney and DALL-E to produce more representative images. Dove's chief marketing officer, Alessandro Manfredi, said the company sought “a future in which women get to decide and declare what real beauty looks like, not algorithms”. Diedrichs, who has collaborated with the Dove Self-Esteem Project, observed at the launch that it was becoming “increasingly difficult to distinguish between what is real beauty and what is manufactured by AI”.

But prompt guidelines place the burden on the user, exactly the situation Diedrichs described to the BBC: diversity appears only when someone deliberately asks for it.

The Evidence That Different Images Help

Would changing the default actually help? Here the research is more encouraging than it is about labels.

In 2011, Diedrichs and Christina Lee published a study in Psychology and Health with the memorable title “Waif goodbye!”. They showed 171 women and 120 men advertisements featuring thin models, average-size models or no models. Average-size models were rated as just as effective in the advertisements as thin models. And for women who had internalised cultural beauty ideals to an average or high degree, seeing average-size models was associated with a significantly more positive body image state than seeing thin models or none.

In 2019, Rachel Cohen, Jasmine Fardouly, Toby Newton-John and Amy Slater tested body-positive Instagram content on 195 young women. Brief exposure to body-positive posts improved positive mood, body satisfaction and body appreciation compared with thin-ideal or neutral posts. There was a caveat: both thin-ideal and body-positive posts increased self-objectification, the tendency to view one's own body from an observer's perspective, relative to content that did not feature bodies at all.

And the 2026 study by Alleva and colleagues tested exactly the variable in question using generated images. AI imagery emphasising body functionality and diversity was associated with better outcomes for body appreciation and appearance satisfaction than fit-ideal imagery, and the benefits, like the harms, were strongest for the most vulnerable women. Generative tools can produce images that help as well as images that hurt. Which they produce by default is a design decision.

What a Designed Default Could and Could Not Do

Imagine that image generator companies decided that when a user asks for a person without specifying a body, the output should reflect the real range of human sizes, ages, skin tones, features and abilities. Developers already tune their systems heavily to block violence, nudity and copyrighted characters. Adjusting the distribution of body types is the same kind of intervention.

The effect on appearance standards could be significant precisely because of the scale involved. Appearance ideals are learnt from the images a culture circulates. If a growing share of those images is generated, whoever controls the default holds unusual influence over what the next generation learns a normal body looks like.

There are real limits, and honest critics have named them. A generator that produces diverse bodies on request can also be used to simulate diversity without hiring anyone. Hannah Marriott makes this point in her analysis of AI fashion models: synthetic people can mimic a range of appearances, but they cannot share the lived experience of navigating the world in a larger body, or facing discrimination because of one's appearance. A brand could generate a plus-size model, collecting the reputational credit while eliminating the actual person.

Nor would changing generator defaults solve the problem Mia Perumal describes. The thin bodies in her feed are posted by accounts that choose what to publish. A diverse default does nothing to stop a user who deliberately prompts for an extreme body. That is where recommendation algorithms and the enforcement of existing safety duties have to do the work labels cannot.

Who Gets to Decide What a Normal Body Looks Like

There is currently no regulation in the UK, the EU or the United States requiring that the default human body produced by a generative system bear any relationship to real human bodies. The decisions are being taken inside a small number of companies, under commercial pressure to produce images that users find appealing.

One argument holds that any mandated default would itself encode someone's values. But a default always encodes someone's values. The current one encodes the values of the advertising archive. Choosing not to change it is not neutrality. It is a vote for the status quo, cast on behalf of every user who never thinks to ask.

A more modest proposal would treat body representation in generative systems the way some regulators already treat other forms of measurable bias: require developers to audit the distribution of body types their systems produce by default, publish the results, and explain any significant departure from population reality. That would not force any particular aesthetic on anyone. It would simply make the default visible, and therefore contestable. The Washington Post ran such an audit using dozens of prompts.

Advertisers could go further on their own. Rather than a small disclaimer in the corner of a Vogue page, a brand could decline to generate bodies no human being could have, much as Getty did for retouching in 2017 and Israel attempted with its minimum BMI rule for real models. If a body would be unhealthy or impossible in a real person, it should not be used to sell clothes to teenagers.

The Workout Routine for a Body That Was Never Born

Go back to the 12-year-old in the comments. She is not confused about what she wants. She wants to look like the person in the picture, and she has been offered a clear visual promise that this is achievable through diet and exercise. Even if someone told her the body was generated, the evidence suggests it would make little difference to how she felt about her own.

Decades of body image research now extend into the age of generative AI with their conclusions intact. Idealised images harm body image, particularly for the young and vulnerable. Labels do little to prevent that, and telling viewers a body is synthetic reduces their trust in technology but not the damage to how they see themselves. Meanwhile the systems producing these images default, by measurement, to bodies thinner, younger, lighter-skinned and less varied than the real population.

The same research offers a way forward that does not depend on a child reading the small print. Images of varied bodies, real or generated, appear to help rather than harm, and in experiments they were rated just as effective in advertising. The default is a setting, chosen by people, and it can be chosen differently.

Mia Perumal is right that eating disorders do not listen to rationality. That is exactly why the responsibility cannot rest with the people looking at the images. It has to rest with the people deciding what the images look like before anyone sees them.

---

References and Sources

1. BBC News. “AI models could harm young people's body image, warn experts.” 4 October 2026. Includes interviews with Mia Perumal and Professor Phillippa Diedrichs. Syndicated text consulted via NewsBeep: https://www.newsbeep.com/us/870333/
2. Coverage of fashion brands' use of AI models. Guess advertisement in Vogue, August 2025: ABC News / Good Morning America, “Controversy stirs over AI generated 'models' in new Guess ads appearing in Vogue”, July 2025: https://abc7news.com/post/controversy-stirs-ai-generated-models-new-guess-ads-appearing-vogue/17390923/ ; Malay Mail (citing BBC and Beat's Vanessa Longley), 28 July 2025: https://malaymail.com/news/life/2025/07/28/lazy-and-cheap-fashion-mag-vogue-under-fire-for-debuting-ai-generated-model-in-print-campaign-video/185491 . H&M digital twins: FashionUnited, “H&M to create 'twins' of models with AI”, March 2025: https://fashionunited.uk/news/fashion/h-m-to-create-twins-of-models-with-ai-the-possibilities-are-almost-endless/2025032880819 ; “H&M turns to AI 'digital twins' in new campaign”, July 2025: https://fashionunited.uk/news/fashion/h-m-turns-to-ai-digital-twins-in-new-campaign-as-fashion-grapples-with-blurred-realities/2025070482637
3. Marriott, Hannah. “AI-generated fashion models: innovation, illusion or a step too far?” The Conversation, republished by Cyprus Mail, 29 August 2026. https://cyprus-mail.com/2026/08/29/ai-generated-fashion-models-innovation-illusion-or-a-step-too-far
4. Festinger, Leon. “A Theory of Social Comparison Processes.” Human Relations 7, no. 2 (1954): 117 to 140. https://doi.org/10.1177/001872675400700202
5. Nicoletti, Leonardo, and Dina Bass. “Humans Are Biased. Generative AI Is Even Worse.” Bloomberg, June 2023. https://www.bloomberg.com/graphics/2023-generative-ai-bias
6. The Washington Post. “What AI thinks a beautiful woman looks like: Mostly white and thin.” 6 June 2024. https://www.washingtonpost.com/technology/interactive/2024/ai-bias-beautiful-women-ugly-images/
7. University of Toronto. “U of T study asks AI to generate male and female body images, with predictable results.” 25 November 2025. https://www.utoronto.ca/news/u-t-study-asks-ai-generate-male-and-female-body-images-predictable-results ; Faculty of Kinesiology and Physical Education, “New KPE study investigates the ideal body, according to AI”: https://kpe.utoronto.ca/node/5120
8. Thomas, Justin, Fahad Al-Beyahi, Najud Al-Dugheithir and Mohammad Amin Kuhail. “Underweight and Overrepresented: Body Size Bias in AI-Generated Depictions of Women.” Research Square preprint, 6 April 2026. https://doi.org/10.21203/rs.3.rs-9266968/v1
9. Fordham University. “Fatphobia Is Fueled by AI-Created Images, Study Finds.” 2025: https://now.fordham.edu/science-and-technology/fatphobia-is-fueled-by-ai-created-images-study-finds/ ; “Decoding Fatphobia: Examining Anti-Fat and Pro-Thin Bias in AI-Generated Images”, Findings of the Association for Computational Linguistics: NAACL 2025: https://aclanthology.org/2025.findings-naacl.266/
10. Center for Countering Digital Hate. “AI and Eating Disorders: How generative AI is enabling users to generate harmful eating disorder content.” August 2023. https://counterhate.com/wp-content/uploads/2023/08/230705-AI-and-Eating-Disorders-REPORT.pdf
11. Tiggemann, Marika, Amy Slater, Belinda Bury, Kimberley Hawkins and Bonny Firth. “Disclaimer labels on fashion magazine advertisements: Effects on social comparison and body dissatisfaction.” Body Image 10, no. 1 (2013): 45 to 53. https://doi.org/10.1016/j.bodyim.2012.08.001
12. Fardouly, Jasmine, and Elise Holland. “Social media is not real life: The effect of attaching disclaimer-type labels to idealized social media images on women's body image and mood.” New Media and Society 20, no. 11 (2018): 4311 to 4328. https://doi.org/10.1177/1461444818771083
13. McComb, Sarah E., and Jennifer S. Mills. “A systematic review on the effects of media disclaimers on young women's body image and mood.” Body Image 32 (2020): 34 to 52. https://doi.org/10.1016/j.bodyim.2019.10.010 ; York University, “Disclaimers on retouched photos don't solve problem of negative body image”, 28 November 2019: https://news.yorku.ca/2019/11/28/disclaimers-on-retouched-photos-dont-solve-problem-of-negative-body-image/
14. Bennett, Brooke L., Alexis Lamere, Anastasia Kopan, Ellena Wood, Radha Hari and Evan Wright. “Filtering Trust: Disclosing the Role of Artificial Intelligence Decreases Trust in Technology, but Does Not Prevent Harm to Body Image After Viewing AI-Generated Content.” International Journal of Eating Disorders 59, no. 6 (2026): 1304 to 1312. Published online 18 February 2026. https://doi.org/10.1002/eat.70059 ; PubMed record: https://pubmed.ncbi.nlm.nih.gov/41705397/
15. Alleva, Jessica M., Tuğba Türkcan, Linda Lin, Chrisa L. Sloutas and Jasmine Fardouly. “The effects of exposure to imagery created with generative artificial intelligence (gen-AI) on young women's body image: Do image type and disclosure matter?” Computers in Human Behavior: Artificial Humans 9 (August 2026): 100339. https://doi.org/10.1016/j.chbah.2026.100339 ; Istanbul Gelisim University announcement, 17 July 2026: https://sbf.gelisim.edu.tr/en/akademik-department-nutrition-and-dietetics-english-news-new-publication-effects-of-generative-artificial-intelligence-generated-images-on-body-image-evaluated
16. Dove pledge and Real State of Beauty report, April 2024. Retail Dive, “Dove pledges no generative AI models in Real Beauty campaign”, 10 April 2024: https://www.retaildive.com/news/dove-pledges-no-generative-ai-models-real-beauty-campaign/712716/ ; FashionUnited, “Dove commits to never using AI to represent real beauty”, 11 April 2024: https://fashionunited.uk/news/business/dove-commits-to-never-using-ai-to-represent-real-beauty/2024041175082 ; Cosmetics Design Europe, “Dove pledges to 'keep beauty real' in the age of AI”, 10 April 2024: https://www.cosmeticsdesign.com/Article/2024/04/10/Dove-pledges-to-keep-beauty-real-in-the-age-of-AI
17. NHS England. “Mental Health of Children and Young People in England, 2023: wave 4 follow up to the 2017 survey.” November 2023. https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/2023-wave-4-follow-up ; University of Exeter summary: https://news.exeter.ac.uk/faculty-of-health-and-life-sciences/one-in-five-children-and-young-people-had-a-probable-mental-disorder-in-2023/
18. National laws on retouched images. Israel's 2012 “Photoshop law”: The Times of Israel, “Thin no longer in”: https://www.timesofisrael.com/underweight-models-go-out-with-2012/ ; Christian Science Monitor, “Israel tells underweight models to gain weight or get off the runway”, 4 January 2013: https://www.csmonitor.com/World/Middle-East/2013/0104/Israel-tells-underweight-models-to-gain-weight-or-get-off-the-runway . France, Décret n° 2017-738 du 4 mai 2017 relatif aux photographies à usage commercial de mannequins dont l'apparence corporelle a été modifiée: Osborne Clarke, “1 October 2017: advertisers must include label identifying retouched photographs”: https://marketinglaw.osborneclarke.com/advertising-regulation/1-october-2017-advertisers-must-include-label-identifying-retouched-photographs/ . Norway: Norwegian Consumer Authority (Forbrukertilsynet), “The Norwegian Consumer Authority's Guideline on Labelling Retouched Advertising”: https://www.forbrukertilsynet.no/english/the-norwegian-consumer-authoritys-guideline-on-labelling-retouched-advertising
19. UK Parliament. “Digitally Altered Body Images Bill”, Session 2019 to 21: https://bills.parliament.uk/bills/2778 ; Session 2021 to 22: https://bills.parliament.uk/bills/3093
20. Advertising Standards Authority. “Disclosure of AI in advertising: striking the balance between creativity and responsibility.” 29 May 2025. https://www.asa.org.uk/news/disclosure-of-ai-in-advertising-striking-the-balance-between-creativity-and-responsibility.html ; February 2021 filter rulings: Cosmetics Business, “Beauty brands and influencers banned from using 'misleading' filters, rules ASA”: https://cosmeticsbusiness.com/beauty-brands-and-influencers-banned-from-using-misleading-filters-rules-asa--174232
21. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 50 and Article 3(60). EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689/oj ; Jones Day, “European Commission Publishes Draft Code of Practice on AI Labelling and Transparency”, January 2026: https://www.jonesday.com/en/insights/2026/01/european-commission-publishes-draft-code-of-practice-on-ai-labelling-and-transparency ; Goodwin, “Not Delayed, Not Deferred: EU AI Act Transparency Obligations Are Now in Force”, August 2026: https://www.goodwinlaw.com/en/insights/publications/2026/08/alerts-technology-dpc-eu-ai-act-transparency-obligations-now-in-force
22. UK Government. “Online Safety Act: explainer.” https://www.gov.uk/government/publications/online-safety-act-explainer/online-safety-act-explainer ; Herbert Smith Freehills Kramer, “What is harmful content and what are the key duties under the OSA to protect children online?”, January 2025: https://www.hsfkramer.com/insights/2025-01/what-is-harmful-content-and-what-are-the-key-duties-under-the-osa-to-protect-children-online
23. Getty Images retouching policy, effective 1 October 2017. PetaPixel, “Getty Images Bans Photos Containing Photoshopped Weight”, 26 September 2017: https://petapixel.com/2017/09/26/getty-images-bans-photos-containing-photoshopped-weight/ ; Engadget, 27 September 2017: https://www.engadget.com/2017/09/27/getty-bans-photoshop-images-models-thinner/
24. Diedrichs, Phillippa C., and Christina Lee. “Waif goodbye! Average-size female models promote positive body image and appeal to consumers.” Psychology and Health 26, no. 10 (2011): 1273 to 1291. https://doi.org/10.1080/08870446.2010.515308 ; UWE repository: https://uwe-repository.worktribe.com/output/958730/waif-goodbye-average-size-female-models-promote-positive-body-image-and-appeal-to-consumers
25. Cohen, Rachel, Jasmine Fardouly, Toby Newton-John and Amy Slater. “#BoPo on Instagram: An experimental investigation of the effects of viewing body positive content on young women's mood and body image.” New Media and Society 21, no. 7 (2019): 1546 to 1564. https://doi.org/10.1177/1461444819826530

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...