SmarterArticles

Most experts on Stack Exchange started out as somebody asking a basic question.

That is how the system was designed. A newcomer posts something simple about a regression model or a pandas dataframe. A stranger with more reputation points answers it. Someone else edits the question so it makes sense, and a third person links a related thread. The newcomer comes back a week later with a harder question and, months on, answers someone else's basic one. Multiply that loop by a few million people and you get one of the largest bodies of freely shared technical knowledge ever assembled.

Now take the first step away. Ask the basic question to a chatbot instead. You get an answer, often a decent one, in seconds, and nobody laughs at you. But no question goes up, no stranger answers or corrects it, and the loop that might have turned you into a contributor never starts.

Multiply that by a few million people too.

A new study of seven Stack Exchange communities suggests this is not a thought experiment. In the sites where AI answers best, the share of newcomers still active a year after arriving fell by close to half after ChatGPT launched, while the core of committed regulars barely moved. A separate study of roughly 12,000 office workers found generative AI widening people's networks while also nudging them towards colleagues they already relied on. Put those findings alongside half a century of sociology on “weak ties”, and an uncomfortable question comes into focus. When machines quietly absorb the casual, low-stakes questions that once pulled strangers into a community, what happens to the community?

Seven Communities, One Natural Experiment

The Stack Exchange study is titled “Generative AI and interrupted peripheral participation: evidence of structural closure from Stack Exchange”. Its authors are Junlei Du, Shuang Li, Yinpei Yu, Xinyi He and Jingxi Chen of Beijing Normal University, and it was posted to the preprint server Research Square on 4 August 2026. The page lists it as under review. It has not yet passed peer review, and its findings should be read with that in mind.

The authors followed seven Stack Exchange sites from 2016 to 2025, grouped by how easily their knowledge can be turned into something a machine could produce and a user could check. Artificial Intelligence and Data Science are “algorithmic” communities, heavy with code. Statistics, Physics and Chemistry are “disciplinary”, heavy with formulas and derivations. Cooking and Travel are “practice-based”, built on situated experience that resists neat codification.

All seven share the same reputation system, moderation tools and interface, which lets the type of knowledge do the varying. The authors treated ChatGPT's public release in late November 2022 as a shock that should hit the codifiable communities hardest, and compared what happened before and after using a difference-in-differences design. Robustness checks included synthetic controls and placebo tests that pretend the shock happened in an earlier year. They then examined three layers at once: the network of who answers whom, the retention of different kinds of users, and the content of the questions.

Where the Halving Actually Happened

The headline figure is that newcomer one-year retention “falls by nearly half”. That phrase comes straight from the paper's abstract. The detail sits in the results tables, and it matters.

In the two algorithmic communities, the proportion of newcomers still active a year later fell from 11.5 per cent before generative AI to 6.0 per cent after it. That is a drop of 5.5 percentage points, or 47 per cent in relative terms. It is the “nearly half”.

The other groups fell too, but by less. Newcomer retention in the disciplinary communities slipped from 12.7 per cent to 9.7 per cent. In the practice-based communities it went from 6.9 per cent to 5.1 per cent. The authors describe this as a dose-response gradient: the more codifiable the knowledge, the bigger the hit. So the accurate version of the headline is narrower than it first sounds. Retention nearly halved in the communities where code answers dominate, not across all seven.

The same pattern shows up among what the authors call low-activity users, people with just one or two recorded interactions, who serve as a proxy for the periphery. Their retention fell significantly only in the algorithmic group, from 9.5 per cent to 6.2 per cent. Among high-activity users, people with ten or more events, retention did not change significantly anywhere. In the algorithmic communities it moved from 57.7 to 54.1 per cent, a shift the authors do not treat as statistically meaningful. In the practice-based communities it actually rose.

One more detail complicates the story. The share of askers posting for the first time fell fairly evenly across all three groups. The authors read this as a sign that AI “shapes who returns more than who first arrives”. People still turn up. Fewer of them stay.

That is the periphery thinning in real time. The regulars are still there; what is missing is the stream of newcomers who would have replaced them. The paper puts it bluntly: the algorithmic core shrank in absolute numbers “even though per-member core retention stayed stable, because newcomers no longer arrive at the rate needed to replenish it”.

The overall scale of the contraction is stark. By the authors' count, activity in the algorithmic communities fell by around 80 per cent from its 2020 peak. Disciplinary communities fell by 30 to 50 per cent and practice-based ones by about 40 per cent. In one network diagram, the connected core of the Data Science site collapses from 70 users in 2019 to four in 2024.

Bonding Up, Bridging Down

Retention is the human story. The network story is where the paper earns its title.

Sociologists distinguish two kinds of social capital. Bonding capital is the dense web of ties inside a tight group. Bridging capital is the thinner set of ties that cross between groups and carry information from one cluster to another. Healthy knowledge communities need both.

In the algorithmic communities, the balance tipped. Modularity, a measure of how strongly a network splits into separate clusters, rose by 0.085 in the main estimate and by 0.158 in the synthetic control version. Average path length, roughly the number of hops it takes to get from one user to another, also went up. Interaction increasingly stayed inside tight topic clusters instead of crossing between them. A third bridging measure, the E-I index, swung sharply inward in the algorithmic group, though the authors are careful to note that this indicator failed their pre-treatment trend test and so cannot carry causal weight on its own.

The disciplinary and practice-based communities barely moved on these structural measures, even though they also lost users. That contrast is the paper's strongest argument against a simple “everything is shrinking” explanation. Cooking and Travel lost about 40 per cent of their activity yet kept their shape. The algorithmic sites lost more and changed shape.

The authors call the outcome “structural closure”. The abstract summarises it like this: “bridging weakens as bonding among incumbents remains comparatively intact.” The community does not die; it closes in on itself. Regulars keep talking to regulars, and the threads that once ran outwards to occasional visitors and curious strangers fray.

They also rule out one tempting explanation. If a handful of super-users were simply hoovering up all the remaining activity, inequality in how connected people are should have risen. In the algorithmic communities it fell. This is not a power grab by an elite. It is the edge falling away.

The Slower Question

Does any of this matter if the questions still get answered? The paper suggests it does, though the authors call these links associational rather than proven.

Median time to a first answer rose by two to five hours across all three groups after generative AI arrived. Across the community-year data, higher modularity went with longer waits. A 0.1 increase in modularity was associated with roughly 3.6 extra hours before a question got its first answer. The intuition is straightforward: if the network has split into tight clusters, a question that falls between them has to travel further to reach someone who can help.

The content of the questions shifted too. The authors measured how often different topic tags appear together on the same question, a rough proxy for how much the community's knowledge is connected across subjects. That density fell by 37 per cent in the algorithmic communities. The questions that remained were also more scattered across niche topics and more focused on debugging specific code, the kind of problem a chatbot cannot solve without seeing the actual artefact.

The authors describe this as two fabrics fraying together. The social fabric of ties between subgroups and the semantic fabric of links between topics erode at the same time. In their discussion, they tie this directly to sociology's most famous argument about networks, describing the pattern as Granovetter's “weak-ties argument in reverse”.

Why Routine Questions Were Never Just Routine

To see why the authors frame this as more than a traffic problem, you need a concept from 1991.

That year, the anthropologist Jean Lave and the learning theorist Etienne Wenger published “Situated Learning: Legitimate Peripheral Participation” with Cambridge University Press. They studied apprenticeship among midwives, tailors, quartermasters, butchers and recovering alcoholics, and argued that people learn a practice by participating in it, starting at the edges. A newcomer is given small, legitimate, low-stakes tasks. Doing them in the presence of old-timers lets the newcomer absorb how the community thinks and talks, and gradually move towards the centre.

Stack Exchange, the Beijing Normal team argues, worked in a similar way. Asking a routine question in public “exposes a newcomer to public reasoning, establishes an initial interaction, and creates the possibility of return”. It is the entry ticket.

A large language model, they write, “separates the informational outcome from this social process”. You get the answer without the exchange. The edge-of-the-room apprenticeship never begins, because there is no room.

Separate evidence supports the idea that answers recruit contributors. In a paper posted to arXiv in April 2026, Lenard Strahringer, Sven Eric Prüß and Kai Riemer analysed more than 21 million Stack Overflow questions and found that “receiving an answer significantly increases a user's propensity to help others”. The effect, they reported, “is concentrated among newcomers and declines with platform experience”, and peaks when answers arrive within roughly 30 to 60 minutes. Being helped is how newcomers become helpers. Veterans do not need the nudge. Remove the moment of being helped, and the recruitment pipeline loses its first stage.

There is an honest caveat here, and the authors state it up front: “individual decisions to use an LLM are not directly observed.” What the study shows is a community-level pattern, timed to ChatGPT's release and concentrated where AI answers are easiest to generate and check. It is strong circumstantial evidence, not proof.

The Weak Ties Granovetter Counted

The sociological backbone of this argument is Mark Granovetter's 1973 paper “The Strength of Weak Ties”, published in the American Journal of Sociology. Its core logic explains why losing a periphery is more than losing numbers.

Granovetter's argument is structural. Your close friends tend to know each other, so they mostly hear the same news you do. Your acquaintances move in different circles. In networks of any size, he showed, a strong tie is almost never the only route between two parts of a network, because close friends share other friends. In his words, “all bridges are weak ties.” The loose connections are the ones that carry new information between groups.

He tested this on job seekers in a Boston suburb. Among the respondents he interviewed in person who had found work through a personal contact, 16.7 per cent said they saw that contact often at the time, 55.6 per cent occasionally and 27.8 per cent rarely. The subsample was small: 54 people from a total sample of 282. But the skew was clear. The useful contact was usually someone on the edge of a person's life, such as an old college friend or a former colleague.

The paper's closing paragraph contains a line that could almost caption the Stack Exchange network diagrams: “strong ties, breeding local cohesion, lead to overall fragmentation.”

Half a century later, the theory got its large-scale causal test. In a paper published in Science in September 2022, Karthik Rajkumar, Guillaume Saint-Jacques, Iavor Bojinov, Erik Brynjolfsson and Sinan Aral analysed randomised experiments on LinkedIn's “People You May Know” feature. Over five years these experiments varied the share of weak ties in the networks of more than 20 million people, a period in which about two billion new ties and 600,000 new jobs were created. Weak ties did increase job mobility, with caveats. When tie strength was measured by mutual connections, the relationship was an inverted U: moderately weak ties helped most. Weak ties also mattered more in digital industries, and strong ties more in less digital ones.

So the familiar claim that weak ties are the lifeblood of mobility has solid support. For innovation and social inclusion it is a reasonable reading of Granovetter's wider argument, not something these studies measured directly. He called weak ties “indispensable to individuals' opportunities and to their integration into communities”. The careful version is that weak ties are the main way information and opportunity cross social boundaries. On Stack Exchange, a stranger answering a newcomer's question was a weak tie in its purest form.

Inside a Company Rolling Out Copilot

The second strand of evidence moves from public forums into a corporate office, and the picture there is more complicated than it is often made to sound.

“The Effects of Generative AI on Collaboration and Social Networks Among Knowledge Workers” was posted to arXiv on 23 October 2026 by Giuseppe Soda and Alessandro Iorio of Bocconi University and Leonardo Rizzo of SDA Bocconi School of Management. Like the Stack Exchange paper, it is a preprint.

The authors studied a single business unit of a large multinational consumer products company, which they do not name. The unit started with 12,040 employees. Between September 2024 and September 2025, the company rolled out Microsoft Copilot in stages, which gave the researchers a natural experiment. Their final sample was 11,868 workers: 1,992 who adopted Copilot and 9,876 who never did. Using passively collected data on who collaborated with whom, and for how long, they tracked what happened to each adopter's network.

They tested two competing predictions: that AI replaces human interaction, or that it generates more, because its outputs need interpreting, checking and approving by other people. The second won. Adopters collaborated with about 2.18 more colleagues on average than comparable non-adopters. Their networks became less clustered, meaning their contacts were less likely to know each other, which the authors read as broader collaboration. Total collaboration time went up too, by 0.87 hours in the post-adoption average.

The third result is the one that matters most here. Adopters, the authors write, “also concentrated more of their collaboration on ties with stronger prior histories.” Put simply, Copilot users leaned harder on colleagues they already had a track record with, which the authors describe as deepening reliance on already trusted colleagues. The effect is clearer in some models than others. One summary estimate falls short of conventional statistical significance, while the post-adoption average and the instrumental-variable model both reach it.

The picture, then, is a network that is both wider and more anchored on trusted people. The authors' own reading is that AI output creates new demand for human judgement, and people send that demand towards colleagues whose judgement they already trust.

What the Office Study Did Not Measure

It is tempting to go further and say this research shows the casual, cross-topic exchanges that connect different social worlds eroding. It is worth saying plainly that the paper does not show that.

Soda, Iorio and Rizzo did not analyse weak ties, casual ties or cross-topic ties. In fact their method screened out many of them. To reduce what they call “noise from incidental interactions”, they kept only ties above the monthly median weight, about one hour of collaboration. A quick corridor chat or a two-line message to someone in another department is exactly the kind of contact a filter like that is likely to remove. The study cannot tell us whether those contacts shrank, grew or stayed the same.

The authors are also candid about other limits. “We know who interacts with whom and for how long, but not what they talk about,” they write. Their data cover one business unit in one company, and they say it is “not clear” whether the findings apply to other industries or workplaces. The results cover the first three months after each person adopted the tool.

What can fairly be said is narrower: in this organisation, Copilot users reached more people while concentrating more of their sustained collaboration on familiar partners. There is a related precedent. When Longqi Yang, David Holtz, Sonia Jaffe, Siddharth Suri and colleagues studied 61,182 Microsoft employees during the shift to remote work in the first half of 2020, they found collaboration became more static and siloed. Time spent on cross-group collaboration fell by about 25 per cent of its pre-pandemic level, and employees added fewer new collaborators. That paper, published in Nature Human Behaviour, was about remote work, not AI, and found narrower networks rather than wider ones. But it shows how quickly a change in working conditions can pull people towards their inner circles. Whether Copilot does the same to the casual edges of a network is a question the Bocconi study raises without answering.

The Volume Collapse Around It

The Beijing Normal paper builds on two earlier peer-reviewed studies that measured the drop in Stack Overflow activity directly.

In a paper published in PNAS Nexus in September 2024, R. Maria del Rio-Chanona, Nadzeya Laurentsyeva and Johannes Wachs compared Stack Overflow with sites where ChatGPT was less available or less capable, such as its Russian and Chinese-language equivalents. They estimated that posting on Stack Overflow fell by about 25 per cent within six months of ChatGPT's release, relative to those comparisons, and called that a lower bound. Their title states the stakes plainly: “Large language models reduce public knowledge sharing on online Q&A platforms.”

One finding complicates the newcomer story: posts from first-time users fell only slightly, while inexperienced, experienced and expert users all posted significantly less. That does not contradict the Beijing Normal results, which concern whether newcomers stay rather than whether they post once, but it shows how much depends on what is measured.

In Scientific Reports in May 2024, Gordon Burtch, Dokyun Lee and Zhichen Chen of Boston University's Questrom School of Business estimated that Stack Overflow lost about a million daily visitors, roughly 12 per cent of its traffic, after ChatGPT arrived. The decline was concentrated among newer users. Their most striking finding was that Reddit's developer communities showed “no evidence” of a similar drop. The authors attributed the difference to “social fabric”. Reddit's communities, they argued, have stronger social bonds, which act as “a buffer against the community-degrading effects of LLMs”. That buffer is bonding capital, the kind the Beijing Normal team saw surviving; bonding without bridging is what Granovetter warned against.

The raw numbers since then have been brutal. In May 2025, the engineering newsletter The Pragmatic Engineer, citing a Stack Exchange Data Explorer query shared by the long-time contributor Marc Gravell, reported that Stack Overflow was on course for about 14,000 questions that month. That is fewer than the 17,668 it received in September 2009, in the site's infancy. The newsletter's author, Gergely Orosz, noted that the decline began around 2014 and sped up from mid-2020, well before ChatGPT. The fall has not slowed since. In January 2026, Tim Anderson of DevClass, drawing on Stack Overflow's public query tool, reported that only 3,862 questions were posted in December 2025, a drop of 78 per cent on the previous year. In August 2026, Luís Rijo of PPC Land reported a comparison posted on X by the developer Daniel Lockyer, using the Stack Exchange Data Explorer: 1,442 new questions in July 2026, about 0.7 per cent of the 207,204 posted at the site's peak in March 2014. The whole of 2025 produced 108,981 questions, little more than half the total for March 2014 alone. AI did not start the slide. It turned a slow decline into a fast one.

Nor is the pattern limited to programmers. In October 2025, Marshall Miller of the Wikimedia Foundation reported that human pageviews on Wikipedia were down about 8 per cent on the same months of 2024. He linked this to search engines answering questions directly, often using Wikipedia's own content, and to younger people searching on social video. His warning about the consequences was explicit: with fewer visits, “fewer volunteers may grow and enrich the content”. Editors are recruited from readers; fewer readers means a smaller pool of future editors.

A Periphery That Was Already Fraying

It would be too easy to cast generative AI as the villain that wrecked a welcoming commons. Stack Overflow's periphery had problems long before ChatGPT.

In April 2018, Jay Hanlon wrote a post on Stack Overflow's official blog under the headline “Stack Overflow Isn't Very Welcoming. It's Time for That to Change.” He admitted that “too many people experience Stack Overflow as a hostile or elitist place”, especially “newer coders, women, people of color, and others in marginalized groups”. He wrote that “too often, someone comes here to ask a question, only to be told that they did it wrong”, and that “condescension and sarcasm have been reluctantly tolerated in comments for too long.”

Wikipedia went through something similar earlier. In a 2013 paper in American Behavioral Scientist, Aaron Halfaker, R. Stuart Geiger, Jonathan Morgan and John Riedl found that the encyclopaedia's decline in active editors was driven by falling retention of good-faith newcomers, not by fewer newcomers arriving. They pointed to Wikipedia's quality-control tools, including algorithmic tools used to reject contributions, as part of the cause.

Both cases show communities damaging their own on-ramps with gatekeeping that feels reasonable to insiders. Part of what AI is doing is exploiting a weakness that already existed. A newcomer choosing between a chatbot that never sneers and a forum that might close their question as a duplicate is not making an irrational choice.

That cuts both ways. It weakens any nostalgic case for the pre-AI periphery, and it suggests the communities now losing their edge had been running down their own renewal for years.

The Case for the Machine Answer

Does the convenience of an instant answer outweigh the long-term cost to community renewal? Any honest answer has to take the convenience seriously.

For the individual asker, the chatbot is often better. It is available at three in the morning, never demands a minimal reproducible example, and will explain the same concept five ways. For people who were made to feel unwelcome, which by Stack Overflow's own admission included many women, people of colour and beginners, that is not trivial. A learner who used to give up after a hostile reception can now keep going.

The communities that remain may also be doing more interesting work. Denis Helic and Tiago Santos, in a paper first posted in September 2025 and later published in the Journal of Systems and Software, found that ChatGPT had “effectively raised the bar for questions on Stack Overflow”, with question difficulty, length and code length all increasing. The routine traffic left. The hard problems stayed. The Beijing Normal team's finding that remaining questions skewed towards debugging specific code fits the same pattern.

Developers have not stopped wanting other people either. Stack Overflow's 2026 Developer Survey, which drew 30,903 responses from 169 countries, found that 73 per cent of respondents now use coding assistants or agents every day. Their trust is conditional. Some 48 per cent said they trust AI when they can easily validate its answers, and 79 per cent rated source attribution important or very important. A year earlier, more developers had distrusted the accuracy of AI output, 46 per cent, than trusted it, 33 per cent. Asked which actions they turn to first when trying to find answers to a problem, 70 per cent chose an AI agent, but 29 per cent chose to send a message directly to a human they trust, and the people they work with remained their most trusted source of information, at 72 per cent. A mixed-methods study posted to arXiv in March 2026 by Ching Christie Pang, Xuetong Wang, Yuk Hang Tsui and Pan Hui, based on a survey of 217 people and 11 interviews, reached a similar conclusion. Users rely on AI for convenience but still go to communities for questions that are complex, ambiguous or depend on trust.

So the human community is not obsolete. It is being pushed upmarket, towards hard cases and trust-sensitive judgement. The problem is that its future experts have to come from somewhere. The easy questions were the training ground. That is where convenience runs into renewal. The individual gain is immediate and visible; the collective cost is delayed and shows up only in network statistics years later.

Training Data Eats Its Own Source

The loss of the periphery has a second-order cost the AI industry has its own reasons to care about. Public question-and-answer sites are a major source of training data for large language models. The Beijing Normal authors note that “sustained bridging erosion thins the very base future AI depends on.” The PNAS Nexus team made a parallel argument: less open posting means less public data to train future models, and some of that knowledge now sits inside privately owned AI systems instead.

The commercial relationship makes the loop explicit. On 6 May 2024, Stack Overflow and OpenAI announced a partnership giving OpenAI access to Stack Overflow's data through its OverflowAPI product, with Stack Overflow using OpenAI models to build its own AI features. Google had earlier signed up as a customer of the same API. The questions and answers that newcomers and regulars wrote for each other for free became a licensed asset sold to the companies whose products are now absorbing the newcomers.

The irony needs no exaggeration. The models answer routine questions well partly because millions of people once asked them in public and strangers answered. If the public asking stops, the archive stops growing while the software keeps changing. Next year's routine question about a framework that does not yet exist will need someone to have answered it first.

Designing an On-Ramp Back In

If the periphery is worth saving, the evidence points to a few practical moves rather than nostalgia.

The first comes from the Beijing Normal paper. Its central principle is that “AI assistance should protect rather than absorb the periphery's entry path”. In practice the authors propose an in-community assistant that answers routine questions from the community's own verified knowledge, and then points the asker towards the people who wrote it. The aim is to turn a one-off answer into a first contact.

There is early evidence this can work. In a paper accepted at the CSCW 2026 conference, Junling Wang, Lahari Goswami, Gustavo Kreia Umbelino, Kiara Chau, Mrinmaya Sachan and April Yi Wang built a chatbot that brings Kaggle community content and social features into its answers. Across two studies with 28 and 12 data science learners, it increased user trust and encouraged engagement with the community. The samples are small and the setting is not a live platform, but a chatbot can evidently be built as a bridge rather than a replacement.

The second move is to measure the right thing. The Beijing Normal authors found the structural shift appeared before activity hit bottom, and suggest modularity and path length could serve as early-warning signals alongside volume dashboards. Platforms that only watch question counts will notice the decline after the periphery has already gone.

The third is targeted intervention. Because the damage concentrates where knowledge can be expressed as checkable code, the authors would put the strongest safeguards in algorithmic communities. They also propose recommending collaborators across topics based on the content of their work rather than existing ties, an explicit attempt to manufacture bridges.

Inside organisations, the Copilot study suggests a parallel lesson. If AI pushes people towards trusted colleagues for judgement, managers who value cross-boundary knowledge may need to create reasons for different groups to meet.

And communities have to make the edge worth staying at. Burtch and colleagues recommended “enabling socialization, as a complement to pure information exchange”. Belonging only among insiders is closure, so the welcome has to reach newcomers too.

Communities Renew From the Edge

Can knowledge communities survive the loss of their periphery? The best current evidence suggests they can survive in the short term, in the narrow sense that the core keeps going. The regulars stay, the hard questions still get asked, and the bonds among incumbents hold. What looks threatened is renewal: the slow, unglamorous process by which strangers become members and members become experts.

The evidence has limits. The Stack Exchange study is a preprint covering seven communities, only two of which carry the dramatic result, and it infers rather than observes individual behaviour. The Copilot study covers one business unit in one company for a few months, and it did not measure casual cross-topic ties at all. Granovetter's insight is half a century old and was built on a small interview sample, though the LinkedIn experiments have since confirmed it at enormous scale. None of this proves that AI is hollowing out every knowledge community.

But the pieces line up uncomfortably well. Routine questions recruited newcomers. Newcomers formed the weak ties that bridged clusters, and bridges kept knowledge moving and the core replenished. Generative AI answers routine questions instantly and privately. In the places where it does that best, newcomers are staying at roughly half the old rate, clusters are closing in on themselves and questions are taking longer to reach the right person.

Cohesion without bridges, Granovetter warned, ends in fragmentation. Knowledge communities have always been renewed by people who arrived knowing very little and asked basic questions in public. If those people now ask a machine instead, the communities will keep their experts for a while. What they risk losing is the way new experts were made.

---

References and Sources

1. Du, Junlei; Li, Shuang; Yu, Yinpei; He, Xinyi; Chen, Jingxi (Beijing Normal University). “Generative AI and interrupted peripheral participation: evidence of structural closure from Stack Exchange.” Research Square preprint (under review, not peer reviewed), posted 4 August 2026. DOI: 10.21203/rs.3.rs-10221893/v1. https://www.researchsquare.com/article/rs-10221893/v1 (PDF: https://www.researchsquare.com/article/rs-10221893/latest.pdf). Accessed 30 October 2026.
2. Soda, Giuseppe; Iorio, Alessandro (Bocconi University); Rizzo, Leonardo (SDA Bocconi School of Management). “The Effects of Generative AI on Collaboration and Social Networks Among Knowledge Workers.” arXiv preprint 2610.32461v1, 23 October 2026. https://arxiv.org/abs/2610.32461 (HTML full text: https://arxiv.org/html/2610.32461). Accessed 30 October 2026.
3. Granovetter, Mark S. “The Strength of Weak Ties.” American Journal of Sociology, 78(6), 1360–1380, May 1973. University of Chicago Press. JSTOR stable URL: https://www.jstor.org/stable/2776392 (full text copy: https://www.cs.umd.edu/~golbeck/INST633o/granovetterTies.pdf). Accessed 30 October 2026.
4. Rajkumar, Karthik; Saint-Jacques, Guillaume; Bojinov, Iavor; Brynjolfsson, Erik; Aral, Sinan. “A causal test of the strength of weak ties.” Science, 377(6612), 1304–1310, 16 September 2022 (published online 15 September 2022). DOI: 10.1126/science.abl4476. Summary: Stanford Digital Economy Lab, https://digitaleconomy.stanford.edu/publication/a-causal-test-of-the-strength-of-weak-ties Accessed 30 October 2026.
5. MIT Initiative on the Digital Economy. “New Publication: MIT, Harvard and Stanford Scientists Show Weaker Ties Are More Beneficial for Job Seekers on LinkedIn.” 15 September 2022. https://ide.mit.edu/insights/new-publication-mit-harvard-and-stanford-scientists-show-weaker-ties-are-more-beneficial-for-job-seekers-on-linkedin/ Accessed 30 October 2026.
6. Lave, Jean; Wenger, Etienne. Situated Learning: Legitimate Peripheral Participation. Learning in Doing series. Cambridge University Press, 1991. DOI: 10.1017/CBO9780511815355. https://doi.org/10.1017/CBO9780511815355 Accessed 30 October 2026.
7. del Rio-Chanona, R. Maria; Laurentsyeva, Nadzeya; Wachs, Johannes. “Large language models reduce public knowledge sharing on online Q&A platforms.” PNAS Nexus, 3(9), pgae400, published 11 September 2024. DOI: 10.1093/pnasnexus/pgae400. https://academic.oup.com/pnasnexus/article/3/9/pgae400/7754871 Accessed 30 October 2026.
8. Burtch, Gordon; Lee, Dokyun; Chen, Zhichen (Questrom School of Business, Boston University). “The consequences of generative AI for online knowledge communities.” Scientific Reports, 14, 10413, published 6 May 2024. DOI: 10.1038/s41598-024-61221-0. https://pmc.ncbi.nlm.nih.gov/articles/PMC11074245/ Accessed 30 October 2026.
9. Strahringer, Lenard; Prüß, Sven Eric; Riemer, Kai. “Help Converts Newcomers, Not Veterans: Generalized Reciprocity and Platform Engagement on Stack Overflow.” arXiv preprint 2604.03209, 3 April 2026. https://arxiv.org/abs/2604.03209 Accessed 30 October 2026.
10. Yang, Longqi; Holtz, David; Jaffe, Sonia; Suri, Siddharth; Sinha, Shilpi; Weston, Jeffrey; Joyce, Connor; Shah, Neha Parikh; Sherman, Kevin; Hecht, Brent; Teevan, Jaime. “The effects of remote work on collaboration among information workers.” Nature Human Behaviour, 6, 43–54, published online September 2021. DOI: 10.1038/s41562-021-01196-4. Microsoft Research summary: https://www.microsoft.com/en-us/research/publication/the-effects-of-remote-work-on-collaboration-among-information-workers/ Accessed 30 October 2026.
11. Berkeley Haas Newsroom / Greater Good Science Center. “How Remote Work Affects Our Communication and Collaboration.” 2021. https://greatergood.berkeley.edu/article/item/how_remote_work_affects_our_communication_and_collaboration Accessed 30 October 2026.
12. Orosz, Gergely. “The Pulse #134: Stack Overflow is almost dead.” The Pragmatic Engineer, 15 May 2025 (citing Stack Exchange Data Explorer data shared by Marc Gravell). https://newsletter.pragmaticengineer.com/p/the-pulse-134 Accessed 30 October 2026.
13. Anderson, Tim. “Dramatic drop in Stack Overflow questions as devs look elsewhere for help.” DevClass, 5 January 2026 (citing Stack Overflow's public query tool). https://devclass.com/2026/01/05/dramatic-drop-in-stack-overflow-questions-as-devs-look-elsewhere-for-help Accessed 2 November 2026.
14. Rijo, Luís. “Stack Overflow drops to 1,442 questions in July, down 99% from 2014 peak.” PPC Land, 8 August 2026 (citing Stack Exchange Data Explorer figures posted on X by Daniel Lockyer, 7 August 2026). https://ppc.land/stack-overflow-drops-to-1-442-questions-in-july-down-99-from-2014-peak/ Accessed 2 November 2026.
15. Miller, Marshall. “New User Trends on Wikipedia.” Diff, Wikimedia Foundation, 17 October 2025 (updated 13 February 2026). https://diff.wikimedia.org/2025/10/17/new-user-trends-on-wikipedia/ Accessed 30 October 2026.
16. Computing. “Wikipedia reports traffic drop, blames AI search and social media.” October 2025. https://www.computing.co.uk/news/2025/ai/wikipedia-reports-traffic-drop Accessed 30 October 2026.
17. Hanlon, Jay. “Stack Overflow Isn't Very Welcoming. It's Time for That to Change.” Stack Overflow Blog, 26 April 2018. https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/ Accessed 30 October 2026.
18. Halfaker, Aaron; Geiger, R. Stuart; Morgan, Jonathan T.; Riedl, John. “The Rise and Decline of an Open Collaboration System: How Wikipedia's Reaction to Popularity Is Causing Its Decline.” American Behavioral Scientist, 57(5), 664–688, 2013. Research summary: https://meta.wikimedia.org/wiki/Research:The_Rise_and_Decline (University of Minnesota release: https://cse.umn.edu/college/news/wikipedia-inadvertently-causing-its-own-decline-participation). Accessed 30 October 2026.
19. Helic, Denis; Santos, Tiago. “Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter.” arXiv preprint 2509.05879 (v1 7 September 2025; v2 16 June 2026); journal version in Journal of Systems and Software, 241, 112988, 2026. https://arxiv.org/abs/2509.05879 Accessed 30 October 2026.
20. Stack Overflow. “2026 Developer Survey” (sections: AI, https://survey.stackoverflow.co/2026/ai ; Knowledge, https://survey.stackoverflow.co/2026/knowledge ; Community, https://survey.stackoverflow.co/2026/community). 2026. https://survey.stackoverflow.co/2026/ Accessed 2 November 2026.
21. Stack Overflow. “2025 Developer Survey: AI.” 2025. https://survey.stackoverflow.co/2025/ai Accessed 30 October 2026.
22. Pang, Ching Christie; Wang, Xuetong; Tsui, Yuk Hang; Hui, Pan. “The Decline of Online Knowledge Communities: Obstacles, Workarounds, and Sustainability.” arXiv preprint 2603.27399, 28 March 2026. https://arxiv.org/abs/2603.27399 Accessed 30 October 2026.
23. SD Times. “Stack Overflow and OpenAI announce partnership to enable ChatGPT to use knowledge from Stack Overflow answers.” 6 May 2024. https://sdtimes.com/ai/stack-overflow-openai-partnership-chatgpt-use-knowledge-from-stack-overflow-answers/ Accessed 30 October 2026.
24. TechRadar. “Stack Overflow signs deal with OpenAI to strengthen models on its data.” May 2024. https://www.techradar.com/pro/stack-overflow-signs-deal-with-openai-to-strengthen-models-on-its-data Accessed 30 October 2026.
25. Wang, Junling; Goswami, Lahari; Umbelino, Gustavo Kreia; Chau, Kiara; Sachan, Mrinmaya; Wang, April Yi. “Bridging Instead of Replacing Online Coding Communities with AI through Community-Enriched Chatbot Designs.” arXiv preprint 2601.18697 (accepted at CSCW 2026), 26 January 2026. https://arxiv.org/abs/2601.18697 Accessed 30 October 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Half a percentage point. That is how much a year of schooling has lost in wage value, in the American occupations most exposed to generative AI, since ChatGPT arrived in November 2022. It sounds like a rounding error. It is not. It is one of the first direct estimates of something economists have suspected for three years: the machine is coming for the part of your education that can be written down.

The number comes from a working paper posted to arXiv on 26 October 2026 by Eduardo Levy Yeyati, Darío Judzik and Juan Ignacio Barbiero of Universidad Torcuato Di Tella in Buenos Aires. It is titled “Practice Makes Perfect: Tacit Knowledge, Generative AI, and Returns to Schooling and Experience”, and it does something deceptively simple. It takes the workhorse equation of labour economics, the one that tells you how much an extra year in a classroom or an extra year on the job adds to your pay, and asks whether its coefficients moved when large language models went mainstream.

They did. In highly exposed occupations, the return to a year of schooling fell by 0.55 log points, roughly 7 per cent of what it had been. The return to experience did something stranger. It did not rise or fall uniformly. It rotated, like a seesaw. Early-career experience became more valuable. Late-career experience became less so. The pivot sits at about 18 years.

Put that next to two other pieces of 2026 research and a picture forms. PwC's Global AI Jobs Barometer, built from more than a billion job advertisements, finds the labour market splitting into two tracks, with the jobs where AI raises the bar for expertise growing twice as fast as the jobs where AI lowers it. And a field experiment led by the MIT economist David Autor finds that when patent lawyers trained alongside an AI assistant were later asked to work without it, the senior lawyers kept their gains while the juniors, on average, did not.

Together, the three studies describe a repricing. Codified knowledge, the stuff a degree certifies, is getting cheaper because a machine can supply it on demand. Tacit judgement, the stuff you only get by doing the work, is getting dearer, but it has a shelf life. For anyone who has just borrowed tens of thousands to buy a credential, that is uncomfortable news. Read carefully, it is also not the catastrophe the headline numbers suggest.

Seven and a Half Points a Year

To understand what moved, you need to know what was being measured. In 1974 the Columbia economist Jacob Mincer published “Schooling, Experience, and Earnings”, a slim book from the National Bureau of Economic Research that gave the discipline its workhorse. Mincer's insight was that if you take the logarithm of a person's earnings, you can model it as a straight-line function of their years of schooling plus a curved function of their years of work experience. The curve is concave: experience pays a lot early, then the gains flatten as people stop investing in new skills.

The schooling coefficient has an intuitive reading. Because the left-hand side is in logs, a coefficient of 0.075 means each additional year of school is associated with roughly 7.5 per cent higher hourly pay, holding experience constant. Economists call that 7.5 log points. Half a century of refinement later, it remains the default lens for asking what education is worth.

This matters for reading the new paper correctly, because the units are easy to garble. Levy Yeyati and his colleagues report that before ChatGPT, the return to a year of schooling in the most AI-exposed occupations was 7.5 log points. In the least exposed occupations it was just 1.7. After the release, the exposed-group figure fell by 0.55 log points relative to the control group. That does not mean wages fell by 0.55 per cent, nor that the degree premium collapsed by 7 per cent of your salary. It means the slope of pay against schooling flattened, from roughly 7.5 points per year to roughly 7 points per year. The 7 per cent figure is the decline in that slope relative to where it started: 0.55 divided by 7.5 is about 7.3 per cent.

A rough illustration helps. A four-year degree in a highly exposed occupation was previously associated with about 30 log points more pay than a high school diploma, all else equal (four years at 7.5 points each). At the post-shock rate, that becomes about 28 points. That arithmetic is mine, not the authors', and it ignores everything else that changed, but it conveys the order of magnitude. The degree still pays. It pays a little less, and in exactly the places where the technology bites hardest.

How a Degree Lost Half a Point

The study draws on the US Current Population Survey's outgoing rotation groups, the monthly household survey that asks a subset of respondents about their earnings. The authors use data from 2018 through June 2026, covering 655,919 observations of people aged 16 to 64 with a valid hourly wage. They date the shock to November 2022 and organise their years from November to October so that the ChatGPT release falls cleanly on a boundary.

To decide which occupations count as exposed, they use an index built by Edward Felten, Manav Raj and Robert Seamans and published in the Strategic Management Journal in 2021. Crucially, that measure predates ChatGPT, so it cannot have been contaminated by what happened afterwards. The paper compares occupations in the top quartile of exposure, which include software developers, financial managers and lawyers, with those in the bottom quartile, which include truck drivers, construction labourers and waiters. The middle two quartiles are set aside.

The core specification is a Mincer equation with the schooling and experience terms interacted with two switches: whether the occupation is exposed, and whether the observation comes after the release. The authors control for a long list of demographic and job characteristics, with fixed effects for month, state and occupation.

One technical choice deserves attention because it affects the result. High earners in the survey are top-coded: above a certain threshold, the Census Bureau records a cap rather than the true figure. That cap changed in April 2023, from a fixed value to a dynamic one, which happens to fall in the post-shock window. Top-coding is concentrated in the treated occupations: 11.5 per cent of that group is censored, against 0.7 per cent of the controls. The authors handle it with interval-censored maximum likelihood rather than imputation, and they test alternatives. Applying the pre-2023 cap throughout gives a schooling effect of minus 0.53 log points, almost identical to the headline. But a deliberately harsh check, which treats dynamically top-coded observations as exact values, shrinks the schooling estimate to minus 0.09. The authors flag this openly. It is the most important caveat in the paper, and anyone quoting the 0.55 figure should know it exists.

Other checks are reassuring. The pre-release schooling coefficients show no trend, and the break appears in the first year after release. A placebo that re-dates the shock to January 2020, to test whether the pandemic is doing the work, finds nothing. Dropping the technology sector “does little”. Swapping the Felten exposure index for the “GPTs are GPTs” measure developed by Tyna Eloundou and colleagues at OpenAI and the University of Pennsylvania gives a schooling decline of 0.42 log points, close to the baseline.

Experience Turns on a Hinge at Eighteen Years

The schooling result is the headline, but the experience result is stranger. In a standard Mincer setup, the value of an extra year of experience depends on how much you already have. A third year adds more than a twenty-third. What Levy Yeyati, Judzik and Barbiero find is that the shape of that curve changed after November 2022 in exposed occupations. At three years of potential experience, the return to an additional year rose by 0.26 log points. At five years, it rose by 0.23. At fifteen, the change is statistically indistinguishable from zero. At twenty-five years, it fell by 0.12. Under the baseline quadratic specification, the effect crosses zero at 18.1 years, with a standard error of 2.1.

The authors are careful about what this means. It is a change in the slope of the experience profile, not a statement that workers with 18 years of experience are the break-even point for winners and losers in pay. “Potential experience” is also a blunt instrument: it is simply age minus years of schooling minus six. It counts time since leaving education, not time spent doing the tasks AI now touches.

The heterogeneity analysis sharpens the picture. Among workers with an associate degree or higher, the early-career return rises by 0.43 log points at three years, and the late-career return falls by 0.12 at twenty-five. Among workers without a credential, the early gain is smaller and the late decline is not statistically significant. In other words, the rotation is strongest precisely among people who hold formal qualifications. The authors write that “wage differences by early-career experience become steeper after the shock” within that credentialed group.

Finally, the level effect. For a worker with 16 years of schooling and 15 years of experience in a treated occupation, the estimated wage change is minus 3.2 log points. That is a real cut, and it falls on exactly the profile of a mid-career graduate professional. But the authors stress that the level effect “is not itself a theoretical prediction”, and they say plainly that “the findings are not a forecast of long-run returns to education or experience.”

Codified Knowledge Was Always the Cheap Part

The theory behind the paper rests on an old distinction. In 1966 the chemist and philosopher Michael Polanyi published “The Tacit Dimension”, which contains the line “we can know more than we can tell.” Some knowledge can be written down, taught in lectures and tested in exams. Some can only be acquired by doing: the feel for when a contract clause will cause trouble, the instinct for which bug report is really an architecture problem, the sense of when a patient's symptoms do not add up.

In 2014 David Autor borrowed Polanyi's line for an NBER working paper titled “Polanyi's Paradox and the Shape of Employment Growth”. His argument was that computers had automated tasks that could be specified as explicit rules, and had struggled with tasks requiring tacit knowledge. That pattern, he argued, explained why employment had grown at the top and bottom of the wage distribution and hollowed out in the middle.

Generative AI scrambles that story. Large language models are not programmed with rules; they learn from enormous quantities of text, absorbing much of what humanity has managed to write down. They are, in effect, machines for supplying codified knowledge at near-zero marginal cost. Ask one to explain the doctrine of equivalents in patent law or the assumptions behind a discounted cash flow model, and it will usually produce a competent answer in seconds.

The Levy Yeyati model formalises three consequences. First, substitution: machine and worker codified knowledge are treated as perfect substitutes, so when the machine version becomes abundant, the worker version, proxied by schooling, loses scarcity value. Second, complementarity: practice-based judgement becomes more valuable when the codified knowledge it works on gets cheaper, steepening the early-career slope. Third, obsolescence: experience tied to the old way of doing the work loses value when the technology changes, flattening the late-career slope.

The paper concedes its estimates do not separately identify these mechanisms. The pattern is consistent with all three acting together, but does not prove it. Nor does the paper distinguish fields of study or institutional quality. A year of organic chemistry and a year of media studies count the same.

A Labour Market Splitting in Two

If the Levy Yeyati paper measures what workers are paid, PwC's 2026 Global AI Jobs Barometer measures what employers are asking for. Released on 15 June 2026, it analyses more than one billion job advertisements across 27 countries and territories on six continents.

Its central finding is a fork. In some occupations, which PwC calls professionalised, AI automates the routine parts of the job and pushes the remaining human work towards judgement and expertise. PwC's examples include radiologists and recruiters. In others, which it calls democratised, AI makes the work easier for non-experts to perform, lowering skill barriers and shifting roles towards less specialised activity. Its examples include IT service managers and medical secretaries.

According to PwC Belgium's release of the global results, “the 22% of jobs that are being professionalised are growing twice as fast as democratised jobs (52% of jobs)”, while “seeing 42% higher wage growth since 2021.”

Read alongside the earnings data, this is striking. The democratised track looks like the substitution story: AI takes over the codified core of a job, and the premium for having learned it shrinks. The professionalised track looks like complementarity: AI clears away routine work and leaves a residue of judgement that commands a premium. That the professionalised track is smaller but faster growing and better paid fits the idea that value is migrating towards tacit expertise.

PwC also reports that the average wage premium for AI skills in job postings rose to 62 per cent, from 57 per cent the previous year. Pete Brown, PwC's Global Workforce Leader, put the implication directly: “Organisations need to rethink how they develop talent if they want people to thrive in this new environment.”

Job advertisements capture what employers say they want, not what they pay individual workers, and PwC sells AI advisory services. Its findings are a large signal about demand, not a peer-reviewed causal estimate.

Junior Jobs That Ask for Senior Skills

The most pointed PwC finding concerns the bottom rung. Analysing 2.4 million entry-level job postings in the United States, the barometer finds that junior roles highly exposed to AI are seven times more likely to require “traditionally senior skills”, such as leadership and strategic thinking, than entry-level roles that are not exposed. Some accounts express this as 52 per cent of new skills against 7 per cent. That ratio fits the sevenfold headline, but the breakdown does not appear in PwC's public summaries, so the sevenfold figure is the safer one to quote.

Openings for these “seniorised” roles have grown 35 per cent since 2019, while other entry-level openings have fallen 10 per cent. The entry-level job is being redefined: employers want new hires who can do on day one what used to take years to learn.

Other data point the same way. Erik Brynjolfsson, Bharat Chandar and Ruyu Chen of Stanford's Digital Economy Lab have been tracking employment using payroll data from ADP, the largest payroll processor in the United States. Their paper “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence”, first released in August 2025 and revised on 12 August 2026 with data through June 2026, finds no evidence of widespread, economy-wide job displacement. But employment of workers aged 22 to 25 in AI-exposed occupations is now 19 per cent below where it would be had it kept pace with less exposed peers. Experienced workers show no comparable gap. The adjustment works mainly through reduced hiring of young people rather than layoffs, and it is concentrated where AI substitutes for human tasks rather than complementing them.

The Federal Reserve Bank of New York's tracker of the labour market for recent college graduates tells a similar story. In the third quarter of 2026, the unemployment rate for recent graduates eased to 5.2 per cent, but the underemployment rate, the share working in jobs that do not typically require a degree, rose to 42.5 per cent.

There is an apparent paradox here. If early-career experience is becoming more valuable, why are young people finding it harder to get hired? The Levy Yeyati paper measures returns among people who are employed. Once you are in the door, each year of practice is worth more than before. But the door is narrower, because the entry-level job, where codified knowledge was traditionally converted into judgement, is exactly what AI is eroding. The authors note that changes in who gets employed could partly explain their early-career estimates. The premium on early experience and the shortage of early jobs may be two faces of the same scarcity.

The Patent Attorneys Who Kept Their Gains

Which brings us to the most direct test of whether AI helps or hinders the acquisition of judgement. In September 2026 the NBER released working paper 35720, “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting”, by David Autor, Tanya Rodchenko, Josh Martin, Zanna Iscenko, Scott Strand, David Pearl and Melissa Ferere. A revision followed in October.

The design is unusually clean for a workplace study. In a pre-registered randomised controlled trial, 133 practising patent lawyers at eleven US intellectual property firms were split into groups, with roughly two-thirds given access to a custom AI patent drafting assistant. Blinded expert patent attorneys scored all the work. The researchers measured performance with the tool and then, after three months, tested everyone on a core professional task without it: redlining an existing patent application.

With the tool, everyone got better. AI access raised the quality of work on benchmark drafting tasks by 0.34 standard deviations at 10 days and 0.38 at 90 days, with larger gains among junior lawyers. Dennis Crouch, the University of Missouri law professor who writes the Patently-O blog, noted that much of the improvement came from “avoiding awful work” rather than producing the very best applications.

Without the tool, the picture split. Treated lawyers outperformed controls by 0.32 standard deviations on the unaided redlining task. But that advantage came entirely from senior lawyers, whose gain was 0.45 standard deviations. Junior lawyers showed no average improvement. Their scores spread out: mediocre results became much less common, while both poor and good results became more common. The authors describe AI as acting as a “springboard” for some juniors and a “cushion” for others. Their conclusion is a sentence worth pinning above every graduate trainee's desk: “Foundational expertise may be a prerequisite for extracting durable skill from AI-assisted practice.”

Autor summarised the result to Fortune in a phrase: AI is “a performance equalizer, but a skill-disequalizer.” He added a warning aimed squarely at people starting out: “Young professionals need to beware of the illusion of competence.” Junior lawyers given the tool reported a surge in task satisfaction, the largest effect in the study. They enjoyed skipping the blank page and moving straight to a reviewer's role. The danger is that the satisfaction of reviewing a competent draft is not the same as acquiring the ability to produce one.

The caveats are substantial, and Fortune's reporting set them out. The paper is not peer-reviewed. Only 91 of the 133 lawyers completed the AI-free test. The firms declined a baseline skills test, so skill gains are inferred from random assignment rather than measured before and after. Fifteen of the 91 markups were flagged as possibly AI-assisted despite the ban; excluding them shrinks the senior advantage to 0.39 standard deviations at a looser statistical threshold, and the overall effect no longer holds. Five of Autor's co-authors are full-time Google employees and the sixth is a paid Google contractor, Google conducted and paid for the study, and the tool was Google's own. Autor is head of MIT's economics department and a Google Technology and Society Visiting Fellow. Three months is short compared with the years it takes to make a patent lawyer.

Still, the result rhymes with others. Erik Brynjolfsson, Danielle Li and Lindsey Raymond's study of more than 5,000 customer-support agents, published in the Quarterly Journal of Economics in 2025, found that AI assistance raised productivity by 15 per cent on average, with the biggest gains for less experienced and lower-skilled workers, while the most experienced saw small gains in speed and small declines in quality. And a field experiment by Hamsa Bastani and colleagues, published in the Proceedings of the National Academy of Sciences in 2025, found that Turkish high school students given unrestricted access to GPT-4 during maths practice scored 17 per cent lower once the tool was taken away, compared with students who never had it. A version with learning-protective guardrails largely mitigated the harm.

The pattern across all three is consistent. AI lifts novice performance in the moment. Whether it builds novice capability over time depends on what the novice already knows and how the tool is used.

Why Experience Expires

The other half of the rotation, the falling return at 25 years, connects to a literature on skill obsolescence that predates generative AI. In 2020 David Deming of Harvard and Kadeem Noray published “Earnings Dynamics, Changing Job Skills, and STEM Careers” in the Quarterly Journal of Economics. Using job vacancy data from Burning Glass Technologies covering 2007 to 2019, they showed that the earnings premium for graduates in technology-intensive subjects such as computer science, engineering and business declines rapidly as they gain experience.

Their model treats a career as a race between on-the-job learning and skill obsolescence. In fast-changing fields, the specific skills a graduate brings are replaced within a few years, which lowers the return to experience and flattens the age-earnings profile. Higher-ability STEM graduates also tend to leave STEM careers more often and at younger ages.

Generative AI may be turning a phenomenon that was confined to technology-intensive careers into something more general. If a large share of codified knowledge in law, finance, marketing and software is now supplied by a machine, then the experience of having done those tasks the old way may depreciate across the board. The Levy Yeyati paper's late-career finding is consistent with that reading, though the authors place less weight on individual late-career coefficients because the pre-trend test for them is weaker.

Here is the double bind for someone starting out. Early experience matters more than ever, because judgement is what AI cannot yet supply. But that experience may itself be technology-specific, and if the technology keeps shifting, its value will erode too. The patent lawyer study suggests one way out: seniors who already had deep foundations were able to absorb the new tool and come out stronger. Experience did not simply expire. It compounded, for those who had enough of it to build on.

The Debt Was Priced for a Different Market

None of this would matter so much if degrees were cheap. They are not. In the United States, Federal Student Aid reported in September 2026 that 42.3 million recipients hold federal student loans totalling more than $1.7 trillion. More than 9.3 million are in default, owing $234 billion. In England, the Student Loans Company's 2025-26 statistics put the higher education loan balance at £294.6 billion, and the provisional average balance for borrowers entering repayment in 2026 is £47,730. The House of Commons Library projects the outstanding balance could reach around £500 billion, in 2025-26 prices, by the late 2040s.

Those debts were taken on at prices set for a labour market in which the return to schooling was stable. If the premium is now shrinking in the fields most exposed to AI, the implied return on that investment shrinks with it. A loan agreement does not reprice when the asset it financed does.

It is worth being precise about the scale. A decline of roughly 0.55 log points per year of schooling, or 3.2 log points at the paper's illustrative mid-career profile, is a meaningful cut but not an annihilation. Even after the shock, the return to schooling in exposed occupations, roughly 7 points per year, remains far higher than the 1.7 points in the least exposed occupations. A degree is still associated with substantially higher pay. What has changed is the direction of travel and the reason for the premium. The share of the premium attributable to knowing things is falling. The share attributable to being able to do things is rising.

England's income-contingent repayment partly cushions graduates, because repayments fall when earnings do; many American borrowers have less protection. Either way, the risk sits with the person least able to diversify it: a young worker with a single human-capital asset and a large liability attached.

What the Numbers Do Not Say

First, the Levy Yeyati paper is a working paper, not yet peer-reviewed, and its headline schooling estimate is sensitive to how top-coded wages are treated. Its central identifying assumption, that greater exposure means a larger expansion of machine-supplied codified knowledge, is assumed rather than tested. The authors list possible confounds candidly: changes in who is employed, signalling and employer learning, and post-2022 shifts in worker composition.

Second, years of schooling are a crude proxy for codified knowledge. A degree also signals persistence, conscientiousness and the ability to learn. It provides networks, and it often functions as a legal or professional gatekeeper. None of those functions is obviously diminished by a chatbot, and the paper cannot separate them.

Third, each of the other studies has its own blind spots, from the Autor experiment's single tool to the descriptive nature of the payroll data. Their convergence is more persuasive than any one result.

Fourth, all of this describes the first three and a half years of a technology that is still changing quickly. The Levy Yeyati authors call their results a short-run repricing of existing stocks of human capital. Over the longer run, schooling choices, curricula, mentoring, occupational sorting and the organisation of entry-level work will all adjust. They close by asking “whether workers can continue to acquire the practice-based knowledge” that AI increasingly substitutes for in learning itself.

Building Judgement Without the Apprenticeship

So how should someone at the outset of a career think about a credential? The evidence suggests five principles, none of them comfortable.

Treat the degree as a foundation, not a product. Senior patent lawyers extracted durable skill from AI because they had foundations to build on, and the authors' conclusion that foundational expertise “may be a prerequisite” is an argument for deep formal education, not against it. What loses value is the degree as a certificate of facts possessed. What retains value is the degree as scaffolding for judgement. The goal is not to memorise what a model can recite, but to understand a field well enough to know when the model is wrong.

Test yourself without the tool. Autor's advice to Fortune was that the only reliable check on skill development is performing tasks without AI and evaluating the results. The Bastani study points the same way: students who used unrestricted GPT-4 as a crutch did worse once it was gone. If your employer gives you an AI assistant, the most valuable thing you can do for your own career may be to periodically do the work without it and compare.

Hunt for the practice, not just the job. Returns to early-career experience are rising while early-career hiring is falling. The scarce resource is not the qualification but the chance to convert it into judgement through supervised real work. Roles offering genuine apprenticeship, with feedback from experienced people on tasks that matter, are worth more than their starting salaries suggest.

Lean towards the professionalised track. Ask whether AI in a given role raises the bar for human expertise or lowers it. Roles where the machine handles the routine and leaves you the hard calls are growing faster and paying better. The question applies to specific jobs and employers, not just titles.

Plan for your experience to expire. Technology-specific know-how depreciates, so the defence is to keep renewing it, layering new techniques onto durable understanding, as the senior patent lawyers did, rather than swapping one set of specific skills for another.

For employers, the implication is sharper. Cutting entry-level hiring because AI can do the junior work is a decision to stop producing senior people. The juniors whose outcomes spread out in both directions in the Autor experiment are next decade's senior lawyers, or not.

A Credential Worth Holding, Priced Differently

What does it mean when the wage premium for education shrinks in the fields most exposed to the technology? The honest answer is that the premium was always paying for two things at once, and the market has started to separate them.

One part paid for possessing codified knowledge, and that part is falling, because a machine can now produce it in seconds. The other part paid for the capacity to learn, judge and build on that knowledge, and that part is holding up, perhaps rising, for those who get the chance to exercise it. The degree you paid for is not worthless. Its price has been revised, and the reason it is valuable has shifted from what it certifies you know to what it enables you to become.

For the worker carrying the debt, that is cold comfort, because the debt did not reprice. For the person deciding whether to start, it is a clearer guide than the noise suggests. The credential still pays, most of all to those who treat it as the start of an apprenticeship rather than the end of an education. Half a percentage point is a warning, not a verdict. The 18-year hinge is a reminder that judgement, too, has a shelf life, and the only defence is to keep building it.

---

References and Sources

1. Levy Yeyati, Eduardo; Judzik, Darío; Barbiero, Juan Ignacio. “Practice Makes Perfect: Tacit Knowledge, Generative AI, and Returns to Schooling and Experience.” arXiv:2610.36008 [econ.GN], submitted 26 October 2026. Also circulated as Universidad Torcuato Di Tella, Escuela de Gobierno, Documento de Trabajo 2026/15. https://arxiv.org/abs/2610.36008 (HTML full text: https://arxiv.org/html/2610.36008)
2. Autor, David; Rodchenko, Tanya; Martin, Josh; Iscenko, Zanna; Strand, Scott; Pearl, David; Ferere, Melissa. “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting.” NBER Working Paper No. 35720, September 2026, revised October 2026. https://www.nber.org/papers/w35720 (also SSRN abstract 7426447: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7426447)
3. Crouch, Dennis. “Who Keeps the Skill: A Randomized Trial of AI Patent Drafting at Eleven Firms.” Patently-O, 9 September 2026. https://patentlyo.com/patent/2026/09/who-keeps-the-skill-a-randomized-trial-of-ai-patent-drafting-at-eleven-firms.html
4. Lichtenberg, Nick. “The economist behind the 'China shock' tackles Gen Z's AI paradox.” Fortune, 10 October 2026. https://fortune.com/2026/10/10/david-autor-china-shock-nber-google-paper-illusion-of-competence-gen-z/
5. PwC. “AI reshapes global labour market into two distinct paths, rewarding human skills: PwC 2026 Global AI Jobs Barometer.” Press release, London, 15 June 2026. https://pwc.com/gx/en/news-room/press-releases/2026/pwc-2026-ai-jobs-barometer.html (syndicated copy: https://kyodonewsprwire.jp/release/202606150856)
6. PwC. “2026 Global AI Jobs Barometer: Global Findings.” June 2026. https://www.pwc.com/gx/en/issues/artificial-intelligence/job-barometer/2026/2026-global-ai-jobs-barometer-global-findings.pdf
7. PwC Belgium. “AI reshapes labour market and urges for upskilling: PwC's AI Jobs Barometer 2026.” Press release, Diegem, 15 June 2026. https://press.pwc.be/ai-reshapes-labour-market-and-urges-for-upskilling-pwcs-ai-jobs-barometer-2026
8. Katanich, Doloresz. “Human skills increasingly in demand as AI reshapes labour market, PwC finds.” Euronews, 16 June 2026. https://euronews.com/business/2026/06/16/human-skills-increasingly-in-demand-as-ai-reshapes-labour-market-pwc-finds
9. Let's Data Science (LDS Team). “PwC Finds AI Creates Two-Track Global Labour Market.” 15 June 2026. https://letsdatascience.com/news/pwc-finds-ai-creates-two-track-global-labour-market-550ec8b1
10. Mincer, Jacob. Schooling, Experience, and Earnings. New York: National Bureau of Economic Research / Columbia University Press, 1974. Chapter 5, “The Human Capital Earnings Function”, pp. 83-96. https://ideas.repec.org/b/nbr/nberbk/minc74-1.html ; https://www.nber.org/chapters/c1767.pdf
11. Felten, Edward; Raj, Manav; Seamans, Robert. “Occupational, industry, and geographic exposure to artificial intelligence: A novel dataset and its potential uses.” Strategic Management Journal, 42(12), 2195-2217, 2021. DOI: 10.1002/smj.3286. https://econpapers.repec.org/RePEc:bla:stratm:v:42:y:2021:i:12:p:2195-2217
12. Eloundou, Tyna; Manning, Sam; Mishkin, Pamela; Rock, Daniel. “GPTs are GPTs: Labor market impact potential of LLMs.” Science, 384(6702), 1306-1308, June 2024. DOI: 10.1126/science.adj0998. Preprint: https://arxiv.org/abs/2303.10130
13. Polanyi, Michael. The Tacit Dimension. Garden City, NY: Doubleday, 1966 (reissued University of Chicago Press, 2009).
14. Autor, David H. “Polanyi's Paradox and the Shape of Employment Growth.” NBER Working Paper No. 20485, September 2014. DOI: 10.3386/w20485. https://www.nber.org/papers/w20485
15. Brynjolfsson, Erik; Chandar, Bharat; Chen, Ruyu. “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence.” Stanford Digital Economy Lab, first released August 2025, revised 12 August 2026. https://digitaleconomy.stanford.edu/publications/canaries-in-the-coal-mine/
16. Federal Reserve Bank of New York. “The Labor Market for Recent College Graduates.” Data for 2026:Q3. https://www.newyorkfed.org/research/college-labor-market
17. Brynjolfsson, Erik; Li, Danielle; Raymond, Lindsey. “Generative AI at Work.” Quarterly Journal of Economics, 140(2), 889-942, 2025. Earlier version: NBER Working Paper No. 31161, 2023. https://www.nber.org/papers/w31161 ; https://econpapers.repec.org/RePEc:oup:qjecon:v:140:y:2025:i:2:p:889-942
18. Bastani, Hamsa; Bastani, Osbert; Sungu, Alp; Ge, Haosen; Kabakcı, Özge; Mariman, Rei. “Generative AI without guardrails can harm learning: Evidence from high school mathematics.” Proceedings of the National Academy of Sciences, 122(26), e2422633122, 2025. DOI: 10.1073/pnas.2422633122. https://econpapers.repec.org/RePEc:nas:journl:v:122:y:2025:p:e2422633122
19. Deming, David J.; Noray, Kadeem. “Earnings Dynamics, Changing Job Skills, and STEM Careers.” Quarterly Journal of Economics, 135(4), 1965-2005, 2020. DOI: 10.1093/qje/qjaa021. https://www.hks.harvard.edu/publications/earnings-dynamics-changing-job-skills-and-stem-careers
20. US Department of Education, Federal Student Aid. “(GENERAL-26-57) Federal Student Aid Posts Updated Reports to FSA Data Center.” Electronic announcement, 22 September 2026. https://fsapartners.ed.gov/knowledge-center/library/electronic-announcements/2026-09-22/federal-student-aid-posts-updated-reports-fsa-data-center
21. Student Loans Company. “Student Loans in England: Financial Year 2025-26.” Official statistics, GOV.UK, updated 2 July 2026. https://www.gov.uk/government/statistics/student-loans-in-england-2025-to-2026/student-loans-in-england-financial-year-2025-26
22. House of Commons Library. “Student loan statistics.” Research briefing SN01079, 2026. https://commonslibrary.parliament.uk/research-briefings/sn01079

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Sixteen point two per cent. That is how often people in one 2026 experiment caught a machine's mistake once it had explained itself in fluent, confident prose. Swap the explanation for a bare number, the model's own estimate of how likely its answer was to be right, and the rate more than doubled, to 37.5 per cent.

The figures come from controlled studies by Ruth Cohen and Sarit Kraus of Bar-Ilan University, Lu Feng of the University of Virginia and Ayala Bloch of Ariel University, posted to arXiv as paper 2604.03237. The first version had a title built for the conference circuit: “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance”. Its abstract did not hedge: “We identify a Persuasion Paradox: fluent explanations systematically increase user confidence and reliance on AI”. The authors argued that interfaces should put calibrated reliance and recovery from errors ahead of persuasive fluency.

Then something unusual happened. On 18 August 2026 the authors posted a revision with a new title, “Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks”. The word “persuasion” has vanished from the text. The 16.2 and 37.5 per cent figures remain, but the revised statistics report that differences in error recovery across the four support conditions did not reach significance (p = .055). The new abstract closes on a far less quotable line: “no single support strategy is universally effective.”

Hold both versions in your head, because between them they contain the whole argument. The first is the story everyone wants to tell: explanations are a trap, and the explainable AI movement has been building persuasion machines. The second is the story the evidence supports: explanations reliably raise confidence, do not reliably raise accuracy, and help or hurt depending on whether the reader can check what they say. A paper about the dangers of persuasive fluency was, in its first draft, a little too persuasive. Its authors noticed and pulled it back. That is calibration in action, and it is exactly the behaviour designers should be trying to produce in everyone else.

So the useful question is not whether explanations are good or bad. It is narrower and harder. When should a system withhold its explanation, or restructure it? And what would an interface look like if its job were not to help people understand the AI, but to help them know when to ignore it?

Sixteen Per Cent and a Change of Title

Cohen and colleagues ran three experiments with participants recruited through university mailing lists, mostly people with STEM backgrounds, who were paid a bonus for each correct answer. Two tasks carried the weight: RAVEN matrices, abstract visual puzzles in which you choose the shape that completes a pattern, and logical reasoning questions from the LSAT, the American law school admission test.

The cleverest design was the multi-stage reveal. Twenty-seven participants answered each RAVEN puzzle three times: first alone, then after seeing the AI's predicted answer, then after reading an explanation, either a written rationale from a large language model or a heatmap showing which parts of the image the model weighted. Alone, they were right 37.0 per cent of the time. With the AI's answer, 49.8 per cent. With the explanation on top, 48.8 per cent. The explanation bought nothing in accuracy.

It did buy something else. In the paper's words, “confidence remained largely unchanged after the prediction stage and increased only after explanations were presented”. The prediction made people more accurate without making them feel surer. The explanation made them feel surer without making them more accurate. If you wanted one result to sum up the worry about fluent AI, that is it.

The between-subjects RAVEN study, with 100 participants across five conditions, is where the famous numbers live. Every form of AI support beat working alone by a wide margin: unaided participants managed 24.6 per cent, while the four AI-assisted groups landed between 53.5 and 60.5 per cent, with predicted probabilities at the top. On overall accuracy, though, the AI-supported conditions were statistically indistinguishable. The difference appeared in how people handled the AI's mistakes. “LLM explanations showed the lowest recovery rate (16.2%) despite high agreement (84.2%),” the authors write. The probability condition showed the highest recovery, 37.5 per cent, with 75.8 per cent agreement on correct predictions.

Read those pairs together. The explanation group agreed with the AI more when it was right (a gap that, like the recovery gap, fell short of significance) and caught it less when it was wrong. That is the signature of reliance that has stopped discriminating: trust that rises evenly across good and bad advice instead of tracking which is which.

The caveats matter, and the revised paper is admirably blunt about them. There was no power analysis in advance, and twenty people per condition is a small sample. The RAVEN probabilities were raw, uncalibrated softmax outputs. A “selective automation” strategy, in which the AI decides confident cases and hands uncertain ones to the human, is now presented only as a benchmark derived after the fact from the probability group's data. The pattern is consistent and striking. It is not yet proof.

The Same Explanation Wins at Logic

The LSAT study rarely makes it into social media summaries, and it is the part that turns a scare story into a design problem.

Eighty participants worked through ten LSAT questions with help from an AI whose answers were right 60 per cent of the time. One group saw only the AI's answer, one saw the answer plus a stated probability, one saw an expert-written explanation adapted from earlier research, and one saw an explanation generated by a large language model.

The language model's explanation won outright. That group reached 72.5 per cent accuracy, against 55.5 per cent for expert explanations, 48.5 per cent for the bare prediction and 47.0 per cent for probabilities. It was the only group to beat the AI's own 60 per cent, and it recovered best from errors, at 47.5 per cent against 35.0 per cent for probabilities. It even beat the human experts' explanations, which rules out the comforting idea that machine prose is simply worse prose. Here the fluent explanation was not a sedative. It was a tool.

Why would the same kind of support hurt in one task and help in another? The authors are careful not to overclaim, noting that task, model and explanation style all changed between studies. But the contrast lines up with an idea that runs through the field. An LSAT explanation lives in the same medium as the problem. If a model says an argument assumes that correlation implies causation, you can reread the passage and see whether it does. The explanation offers a claim the reader can test against evidence already in front of them.

A verbal rationale for a visual puzzle is different. It translates a pattern into words, and the reader has to translate it back. That is laborious, so most people do not bother. They register that the paragraph sounds like reasoning and move on. The explanation becomes a vibe rather than an argument.

In 2023, Helena Vasconcelos and colleagues at Stanford University and the University of Washington, including Michael Bernstein, Tobias Gerstenberg and Ranjay Krishna, gave this intuition a formal shape. Across five studies with 731 participants navigating a maze with a simulated AI, they argued that people make a strategic cost-benefit choice about whether to engage with an explanation at all. Task difficulty, explanation difficulty and money all shifted how much people over-relied on the AI. Earlier studies that found explanations useless, they suggested, may simply have used explanations that did not make checking the AI any cheaper. Their title made the counterintuitive point: “Explanations Can Reduce Overreliance on AI Systems During Decision-Making”.

Put Cohen's LSAT result beside that framework and a first principle falls out. An explanation earns its place when it lowers the cost of verification. When it raises that cost, or supplies the feeling of verification without the substance, it should be withheld or rebuilt.

Editing the Machine's Mind Feels Like Power

If reading an AI's reasoning can make you overconfident, perhaps the fix is to let users get their hands on it. Interactive reasoning, where people inspect and change a model's chain of thought, has become a fashionable idea in human-computer interaction. A short paper at CHI 2026, the field's flagship conference, held in Barcelona in April, suggests the fashion deserves scrutiny.

Jiwon Moon of the University of Chicago, with Chien-Ming Huang and Ziang Xiao of Johns Hopkins University, published “Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making” in the conference's Extended Abstracts, the venue for shorter, often preliminary work. They compared three interfaces in a decision task: no chain of thought, a read-only chain of thought, and an editable one that let users alter the AI's displayed reasoning.

Their abstract captures the gap between how an interface feels and what it does. “Editability significantly increases perceived power, control and satisfaction, but does not improve decision accuracy relative to a no-rationale baseline.” The read-only version was worse: it “increases agreement with the AI and induces inappropriate reliance when the AI is wrong”, reducing accuracy compared with editing and trending below the baseline in which people saw no reasoning at all.

The mechanism is the interesting part. Mediation analyses, the authors report, suggest that editability raises perceived power, which in turn predicts higher confidence and greater over-reliance on incorrect advice, a pattern “consistent with illusion of control.”

The phrase comes from a 1975 paper by the psychologist Ellen Langer in the Journal of Personality and Social Psychology. Across experiments built on games of pure chance, Langer argued that features associated with skill, such as choice, familiarity, involvement and competition, lead people to expect success beyond what the odds justify. Her best-known demonstration, in which people who picked their own lottery ticket asked far more to sell it than people who were handed one, has not aged well: in 2021 Joowon Klusowski, Deborah Small and Joseph Simmons reported in Psychological Science that experiments with more than 10,000 participants found choice alone did not make people think they were more likely to win.

That wrinkle is instructive rather than fatal. Moon, Huang and Xiao measured perceived power directly and linked it to confidence and over-reliance in their own data. But it is a reminder that a vivid label can travel further than the evidence beneath it, which is the subject of this article. Because the CHI paper is an extended abstract, its effect sizes should be treated as provisional. The direction of the result is still a warning to anyone who treats “user control” as an automatic good. Control that is not connected to better outcomes is a feeling, and feelings are what persuasion is made of.

Thirty Years of Leaning on Machines

None of this is new in kind. It is new in fluency.

In 1997, Raja Parasuraman and Victor Riley published “Humans and Automation: Use, Misuse, Disuse, Abuse” in the journal Human Factors. Misuse, in their taxonomy, is over-reliance on automation, producing monitoring failures and decision biases. Disuse is neglect of automation that works, often because it has cried wolf too often. Abuse is automating without regard for the consequences for human performance. All three map onto modern AI assistants, but misuse is the one fluent explanations feed.

In 2004, John Lee and Katrina See gave the field its central goal in the title of their review in the same journal: “Trust in Automation: Designing for Appropriate Reliance”. The key idea is calibration. Trust should match a system's actual trustworthiness, neither overshooting nor undershooting. And it has to work at a fine grain: trusting a system the right amount on average is not enough. You need to trust it more where it is good and less where it is bad.

Medicine has the clearest data on what happens when that fails. A 2012 systematic review in the Journal of the American Medical Informatics Association by Kate Goddard, Abdul Roudsari and Jeremy Wyatt drew on 74 studies of automation bias, which they defined as “the tendency to over-rely on automation”. Pooling four healthcare studies in which decision support systems gave wrong advice, they found the systems increased the risk of an incorrect decision by 26 per cent. The accelerants were familiar: high trust, low self-confidence, workload, complexity, time pressure. Among the design mitigations, they listed attaching updated confidence levels to the system's output, and offering information rather than direct recommendations.

A review published years before large language models went mainstream recommended, in effect, what the RAVEN probability condition did: show people how sure the system is, case by case.

The idea is now law. Article 14 of the European Union's AI Act, on human oversight of high-risk systems, requires that overseers be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)”, particularly where systems provide information or recommendations for decisions taken by people. Those obligations do not bite yet, either: the Digital Omnibus on AI, in force since July 2026, pushed the high-risk deadlines back to December 2027 for standalone systems and August 2028 for AI built into regulated products. The regulation names the problem. It does not specify the interface that solves it. That is the designer's job, and the evidence suggests a fluent paragraph of reasoning is not, by default, the answer.

Explanations Help Most When You Need Them Least

The most cited study on this question predates the language model boom and predicted much of what followed. In 2021, Gagan Bansal and colleagues from the University of Washington and Microsoft Research, among them Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel Weld, presented “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance” at CHI.

Their target was complementary performance: a human-AI team that beats both the human alone and the AI alone. Earlier studies showing that explanations helped had mostly used AI far more accurate than people, so teams improved simply by deferring to a better decision-maker. Bansal's group used AI roughly as accurate as humans, across three tasks (sentiment in beer reviews and book reviews, plus LSAT questions), with 1,626 participants.

Teams did achieve complementary performance, but explanations did not improve on simply showing the AI's confidence. The mechanism was an asymmetry. “Explanations often increased accuracy when the AI system was correct but, worryingly, decreased it when the AI erred, resulting in a minimal net change.” Explanations made people more likely to accept the AI's recommendation whether or not it was right.

Bansal's team tried a fix. Their “adaptive explanation” explained only the top answer when the AI was confident, but laid out the top two alternatives when confidence fell below a threshold, inspired by an efficient assistant that divulges more, “confessing doubts and arguing for alternatives”, only when unsure. It did not significantly improve team performance, though the authors saw suggestive evidence that it nudged agreement in the right direction. Keep it in mind: it is the seed of a better idea.

In 2023, Max Schemmer and colleagues at the Karlsruhe Institute of Technology gave the asymmetry two names. At the Intelligent User Interfaces conference, they proposed measuring “appropriateness of reliance” on separate axes. Relative AI reliance captures how often people switch to correct AI advice after starting with a wrong answer. Relative self-reliance captures how often people keep their own correct answer when the AI is wrong. In their 200-person experiment, explanations raised relative AI reliance from 29.59 to 38.87 per cent. Relative self-reliance did not significantly change.

That is the pattern in miniature. Explanations help people accept the AI when it is right. They do far less to help people resist it when it is wrong. And resistance is the whole point of a human in the loop. If the machine were never wrong, you would not need the person.

Why Fluency Reads as Truth

Large language models sharpen the problem because their explanations are not heatmaps or feature weights. They are prose, and prose is the native format of human persuasion.

In 2025, Mark Steyvers and colleagues at the University of California, Irvine published “What large language models know and what people think they know” in Nature Machine Intelligence. They measured the calibration gap: the difference between how confident people are in a model's answers and how confident the model itself is. Shown default explanations, people overestimated the model's accuracy, and longer explanations raised their confidence even when the extra length did nothing for accuracy. “There's a disconnect between what LLMs know and what people think they know,” Steyvers said in the university's announcement. “We call this the calibration gap.” When the team rewrote explanations so their language reflected the model's internal confidence, that gap narrowed, and people got better at telling right answers from wrong ones.

A year earlier, Sunnie Kim of Princeton University, with Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard and Jennifer Wortman Vaughan of Microsoft, tested something similar. In a pre-registered experiment with 404 participants answering medical questions with a simulated LLM-powered search engine, first-person expressions of uncertainty, along the lines of “I'm not sure, but...”, reduced people's confidence in the system and their agreement with it, and improved their accuracy. Impersonal hedges had weaker effects that did not reach significance. Wording matters enough, the authors concluded, that it needs user testing before deployment.

Back in 2020, Yunfeng Zhang, Q. Vera Liao and Rachel Bellamy of IBM Research found that confidence scores helped people calibrate their trust, while explanations of individual predictions did not do the same job. Their caveat still applies: calibrated trust does not guarantee better decisions unless the human brings knowledge that can compensate for the model's errors. Calibration tells you when to be suspicious. It does not tell you the right answer.

Fluency, in other words, is not a neutral property. Length, confident tone and smooth reasoning all signal reliability without being connected to it. A well-designed system should make those signals honest, or remove them.

Friction Works and Nobody Likes It

If people skim explanations and defer, the obvious response is to make deferral harder. Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos of Harvard University tested exactly this in “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making”, published in 2021 in the Proceedings of the ACM on Human-Computer Interaction.

Borrowing from research on diagnostic error in medicine, they built three interventions into a task in which 199 participants judged how to cut the carbohydrate content of meals. In the “on demand” condition, the AI's suggestion stayed hidden until people clicked to see it. In “update”, people decided first and could revise after seeing the AI's advice. In “wait”, a message saying the AI was processing the image appeared for 30 seconds before the suggestion arrived.

All three significantly reduced over-reliance compared with simple explainable AI interfaces. None eliminated it: the interventions “reduced, but not yet eliminated, overreliance on the AI”, and the human-AI teams still underperformed the AI alone. Then the twist. When the AI was wrong, “people performed best in conditions that they preferred and trusted the least”, which they also rated most difficult. The interventions also helped people high in “need for cognition”, a measure of how much someone enjoys effortful thinking, more than others, raising the question of who friction actually protects.

This is the commercial problem in one finding. Every metric a product team watches, from satisfaction scores to stated trust, rewards the smooth, confident explanation. The metric that matters, whether people catch the machine's errors, rewards friction. Moon, Huang and Xiao found the same split from the other side. And a 2026 CHI review of reliance research by Muhammad Raees, Vassilis-Javed Khan, Ioanna Lykourentzou and Konstantinos Papangelis made the general point: user engagement is valuable for appropriate reliance, but it comes at a cost.

When to Withhold the Explanation

So when should a designer leave the explanation out? The evidence supports no blanket rule, and the revised Cohen paper is explicit that no single strategy wins everywhere. But it does point to conditions under which a default explanation is more likely to hurt than help.

The first is when the explanation cannot be checked against evidence the user can see. The RAVEN rationale asked people to verify a verbal account of a visual pattern; the LSAT rationale asked them to verify a verbal account of a verbal argument. Where explanation and evidence live in different media, or the evidence is hidden, the explanation tends to function as a confidence signal rather than an argument. Vasconcelos's cost-benefit framing predicts exactly this.

The second is when fluency is decoupled from reliability. A language model can write an equally polished rationale for a right answer and a wrong one, and Steyvers's results show that people read polish and length as accuracy. If a system cannot make the style of its explanation track its actual confidence, the explanation is broadcasting a false signal.

The third is when the user cannot bring independent knowledge. Zhang, Liao and Bellamy's caveat applies: suspicion helps only if the human can act on it. Goddard's review identified time pressure, workload and complexity as amplifiers of automation bias, precisely when people are least able to scrutinise a paragraph of reasoning. In those conditions, a persuasive explanation mostly accelerates deference.

The fourth is before the user has formed a view. Buçinca's update condition and Cohen's multi-stage design both suggest timing matters. An explanation shown before someone has thought for themselves anchors them. Shown afterwards, as a challenge rather than a verdict, it does a different job.

None of this means explanations should vanish. Withholding an explanation from the moment of decision is not the same as withholding it from the record. Engineers debugging a model, auditors checking it and people contesting its verdicts may all need a full account. The question here is narrower: what to put in front of someone in the seconds when they decide whether to accept the machine's answer. The default, an explanation attached to every answer and displayed immediately, is the wrong default for a large class of tasks.

How to Rebuild an Explanation

Withholding is the blunt option. Restructuring is more interesting, and the literature offers several patterns.

Lead with uncertainty rather than reasoning. In Cohen's RAVEN study, the probability display recovered best from errors and was rated clearest by participants, despite explaining nothing. Goddard's review flagged confidence displays more than a decade ago, and Kim's work shows uncertainty can be expressed in natural language, if the phrasing is tested. A number or a well-chosen hedge is a cue to look harder, cheap to read and hard to mistake for an argument.

Make explanations conditional, in the spirit of Bansal's adaptive design. When the model is confident and historically reliable on this kind of case, a short answer may suffice. When it is uncertain, it should do the opposite of what fluent assistants usually do: argue against itself, present the strongest alternative, and show what evidence would change its answer.

Stop recommending and start evaluating. In a 2023 paper at the ACM Conference on Fairness, Accountability and Transparency, titled “Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support”, Tim Miller, then at the University of Melbourne, argued that recommendations plus explanations undermine human agency and do not match how people reason. He proposed “evaluative AI”, tools that “provide evidence for and against decisions made by people” rather than handing them a decision to accept or reject. The human proposes; the machine stress-tests. There is nothing to be persuaded of, because the machine is not selling an answer.

Sequence the interaction. Ask for the human's judgement first, reveal the AI's view second, and explain only where they disagree. That combines Buçinca's update intervention with adaptive explanation, concentrating scarce attention where it could change the outcome.

Route the cases. Machine learning researchers have long studied “learning to defer”; a 2020 paper at the International Conference on Machine Learning by Hussein Mozannar and David Sontag, for instance, developed methods for training a model that either predicts or hands the decision to a human expert. Cohen's selective-automation benchmark gestures at the same idea, and the authors note that humans can keep final authority under such schemes. Human scepticism is a scarce resource and should be spent where the model is weakest.

A Framework Built Around Distrust

Pull these threads together and a framework for calibrated reliance emerges. Its goal is not understanding but discrimination: helping people tell the machine's good answers from its bad ones, case by case. What follows is a synthesis of the research above, not a standard anyone has yet adopted.

Measure the right thing. Evaluate a system separately on cases where the AI is right and cases where it is wrong, which is the first recommendation in the revised Cohen paper. Schemmer's two axes offer one method; error recovery offers another. Overall accuracy and trust ratings on their own are close to useless, because the Cohen, Bansal, Buçinca and Moon studies all show satisfaction and confidence moving independently of decision quality.

Make every confidence signal honest. If explanations sound equally sure whether the model is right or wrong, their style is lying. Either calibrate the language to the model's real confidence, as Steyvers's team did, or strip out the persuasive styling and show a calibrated number. One caveat: Cohen's RAVEN probabilities were not carefully calibrated, and the authors warn that uncertainty alone can miss errors the model makes confidently. Probability displays are a starting point, not a solution.

Match support to the task's evidence. Where users can verify an explanation against material in front of them, as with the LSAT, rich explanations can pay their way. Where they cannot, prefer uncertainty cues, contrastive alternatives or nothing at all. This is the core lesson of the revised Cohen paper, and a direct rejection of one-size-fits-all explainability.

Protect independent judgement. Elicit the human's view before revealing the machine's wherever time allows, and present explanations as challenges to disagreement rather than justifications for agreement.

Budget friction. Buçinca's results show friction works and people dislike it. Applied everywhere, it wears people down and pushes them back towards blind acceptance. Applied where the model is uncertain, the stakes are high and the human might know better, it earns its cost.

Distrust the feeling of control. Moon, Huang and Xiao's work warns that editing, steering and customising can raise users' sense of power without improving outcomes. Every control affordance should be tested against accuracy on cases where the AI is wrong, not just against satisfaction.

Treat oversight law as a design brief. Article 14 asks that overseers “remain aware” of automation bias. Awareness is not a pop-up warning. It is an interface that makes the machine's fallibility visible at the moment of decision.

The Researchers Who Took Their Own Advice

The revision of the Cohen paper is worth dwelling on. The first version made a vivid, general claim with a memorable name. The second narrowed it, flagged its statistical limits, demoted its most appealing benchmark to a reference point, and concluded that the right answer depends on the task. The authors noticed that their framing was more confident than their evidence, and they recalibrated. The Langer replication story is the same lesson at a fifty-year scale.

That is precisely the behaviour a calibrated-reliance interface tries to produce, and it is hard. Product teams have every commercial reason to keep the fluent, confident explanation, which tests well, demos well and makes users feel smart. The evidence from Bansal in 2021 to Moon in 2026 says that feeling is not the same as being right, and the two can pull in opposite directions.

The explainable AI movement rested on an appealing premise: if people understand why a machine says what it says, they will use it better. The premise is not wrong so much as incomplete. Understanding helps when it gives people something to check. When it gives them only something to believe, it becomes another form of persuasion, and a more effective one than a bare answer, because it looks like reasoning.

The goal, then, is not more explanation or less, but better-aimed explanation: cheap to verify, honest about uncertainty, sequenced so it does not pre-empt human judgement, and most forthcoming when the machine is least sure. Judge it on one question above all: when the model was wrong, did the person notice?

In the RAVEN study, people reading fluent explanations caught roughly one AI error in six. Shown a number instead, they caught more than one in three. Neither is good enough, and the gap between them is statistically fragile. But it points towards where better designs will come from: interfaces built not to win the user's trust, but to earn it one case at a time, and to surrender it gracefully when it has not been earned.

References and Sources

All URLs accessed and verified on 29 October 2026.

1. Cohen, Ruth; Feng, Lu; Bloch, Ayala; Kraus, Sarit. “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance.” arXiv:2604.03237v1, first version, 2026 (superseded). https://arxiv.org/abs/2604.03237v1 (full text: https://arxiv.org/html/2604.03237v1)
2. Cohen, Ruth; Feng, Lu; Bloch, Ayala; Kraus, Sarit. “Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks.” arXiv:2604.03237v2, revised 18 August 2026. https://arxiv.org/abs/2604.03237 (full text: https://arxiv.org/html/2604.03237v2)
3. alphaXiv listing for arXiv:2604.03237, “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance.” https://www.alphaxiv.org/abs/2604.03237v1
4. Moon, Jiwon; Huang, Chien-Ming; Xiao, Ziang. “Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making.” In Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), Barcelona, 13 to 17 April 2026, pp. 1-9. ACM. DOI: 10.1145/3772363.3798555. https://doi.org/10.1145/3772363.3798555
5. Semantic Scholar record and abstract for Moon, Huang and Xiao (2026), Corpus ID 287443795. https://api.semanticscholar.org/graph/v1/paper/DOI:10.1145/3772363.3798555?fields=title,abstract,authors,venue,year
6. Langer, Ellen J. “The illusion of control.” Journal of Personality and Social Psychology, 32(2), 311-328, August 1975. DOI: 10.1037/0022-3514.32.2.311. https://doi.org/10.1037/0022-3514.32.2.311
7. Klusowski, Joowon; Small, Deborah A.; Simmons, Joseph P. “Does Choice Cause an Illusion of Control?” Psychological Science, 32(2), 159-172, 2021. DOI: 10.1177/0956797620958009. https://faculty.wharton.upenn.edu/wp-content/uploads/2016/11/36-Klusowski-Small-Simmons-2021.pdf
8. Yale School of Management, Yale Insights. “Does Having Choice Provide an Illusion of Control?” https://insights.som.yale.edu/insights/does-having-choice-provide-an-illusion-of-control
9. Bansal, Gagan; Wu, Tongshuang; Zhou, Joyce; Fok, Raymond; Nushi, Besmira; Kamar, Ece; Ribeiro, Marco Tulio; Weld, Daniel S. “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance.” In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, May 2021, pp. 1-16. ACM. DOI: 10.1145/3411764.3445717. arXiv:2006.14779. https://arxiv.org/abs/2006.14779
10. Buçinca, Zana; Malaya, Maja Barbara; Gajos, Krzysztof Z. “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making.” Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188, pp. 1-21, April 2021. DOI: 10.1145/3449287. https://kgajos.seas.harvard.edu/papers/bucinca21trust.pdf
11. Vasconcelos, Helena; Jörke, Matthew; Grunde-McLaughlin, Madeleine; Gerstenberg, Tobias; Bernstein, Michael S.; Krishna, Ranjay. “Explanations Can Reduce Overreliance on AI Systems During Decision-Making.” Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), pp. 1-38, April 2023. DOI: 10.1145/3579605. arXiv:2212.06823. https://arxiv.org/abs/2212.06823
12. Schemmer, Max; Kühl, Niklas; Benz, Carina; Bartos, Andrea; Satzger, Gerhard. “Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations.” In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI '23), Sydney, March 2023, pp. 410-422. ACM. DOI: 10.1145/3581641.3584066. arXiv:2302.02187. https://arxiv.org/abs/2302.02187
13. Zhang, Yunfeng; Liao, Q. Vera; Bellamy, Rachel K. E. “Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20), January 2020, pp. 295-305. ACM. DOI: 10.1145/3351095.3372852. arXiv:2001.02114. https://arxiv.org/abs/2001.02114
14. Kim, Sunnie S. Y.; Liao, Q. Vera; Vorvoreanu, Mihaela; Ballard, Stephanie; Vaughan, Jennifer Wortman. “'I'm Not Sure, But...': Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust.” In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), June 2024, pp. 822-835. DOI: 10.1145/3630106.3658941. arXiv:2405.00623. https://arxiv.org/abs/2405.00623
15. Steyvers, Mark; Tejeda, Heliodoro; Kumar, Aakriti; Belem, Catarina; Karny, Sheer; Hu, Xinyue; Mayer, Lukas W.; Smyth, Padhraic. “What large language models know and what people think they know.” Nature Machine Intelligence, 7(2), 221-231, 2025. DOI: 10.1038/s42256-024-00976-7. https://www.nature.com/articles/s42256-024-00976-7 (preprint: https://arxiv.org/abs/2401.13835)
16. University of California, Irvine, School of Social Sciences. “UC Irvine study finds mismatch between human perception and reliability of AI-assisted language tools.” 22 January 2025. https://www.socsci.uci.edu/newsevents/news/2025/2025-01-22-steyvers-uci-study-finds-mismatch-between-human-perception-and-reliability-of-ai-assisted-language-tools.php
17. Parasuraman, Raja; Riley, Victor. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors, 39(2), 230-253, June 1997. DOI: 10.1518/001872097778543886. https://journals.sagepub.com/doi/10.1518/001872097778543886
18. Lee, John D.; See, Katrina A. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors, 46(1), 50-80, 2004. DOI: 10.1518/hfes.46.1.50_30392. https://pubmed.ncbi.nlm.nih.gov/15151155/
19. Goddard, Kate; Roudsari, Abdul; Wyatt, Jeremy C. “Automation bias: a systematic review of frequency, effect mediators, and mitigators.” Journal of the American Medical Informatics Association, 19(1), 121-127, January 2012. DOI: 10.1136/amiajnl-2011-000089. https://pmc.ncbi.nlm.nih.gov/articles/PMC3240751/
20. Miller, Tim. “Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support.” In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23), June 2023, pp. 333-342. DOI: 10.1145/3593013.3594001. arXiv:2302.12389. https://arxiv.org/abs/2302.12389
21. Mozannar, Hussein; Sontag, David. “Consistent Estimators for Learning to Defer to an Expert.” In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), PMLR 119, pp. 7076-7087. arXiv:2006.01862. https://arxiv.org/abs/2006.01862
22. Raees, Muhammad; Khan, Vassilis-Javed; Lykourentzou, Ioanna; Papangelis, Konstantinos. “Do People Appropriately Rely on AI-Advice? An Analytical Review of HCI Research on Human-AI Decision-Making.” In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, Barcelona, April 2026, Article 805. ACM. DOI: 10.1145/3772318.3791467. https://research-portal.uu.nl/en/publications/do-people-appropriately-rely-on-ai-advice-an-analytical-review-of/
23. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (Artificial Intelligence Act), Article 14 (Human oversight). Official Journal L, 12 July 2024. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14 ; https://artificialintelligenceact.eu/article/14/
24. Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal L, 24 July 2026; in force 27 July 2026. https://eur-lex.europa.eu/eli/reg/2026/1744/oj

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A clinician examines a sore wrist and suggests a thumb spica, the rigid splint that immobilises the base of the thumb. A moment later the clinician changes their mind. Not a spica after all; a wrist brace instead. It is the most ordinary thing in medicine: thinking aloud, revising, settling. The patient hears both versions and understands which one counts.

The AI scribe in the room heard both versions too. Its note recorded the thumb spica as applied.

That example comes from a preprint posted to arXiv on 31 August 2026 by Sebastian Fox, Luke Markham, Ryan Lail and Michael Karotsieris, titled “One note in three: a verified census of three deployed AI scribes, and the instrument that counted it”. The consultation was a test scenario, not a real patient, and the error belonged to one product in one encounter. But it captures, in miniature, the problem underneath the entire ambient scribe boom. A transcript can contain a retraction. A summary has to decide what the retraction means. Once it has decided, the reasoning disappears, and what remains is a clean, confident sentence in an official record stating that something happened which did not.

For most of medicine's history there were two people in the consulting room, and one of them wrote things down. Now there is a third listener. It hears everything, writes fast and never tires. It also decides, on terms nobody in the room set, what the conversation was about. Neither doctor nor patient can see how it made those choices, and its mistakes are written well enough to slip past a tired reader at the end of a long clinic.

So what happens to a patient when the official story of their health is quietly shaped by a party nobody can fully audit? And when the third listener decides something is not worth recording, where does that thing go, and who answers for its absence?

One Note in Three, Under a Stated Instrument

The Fox paper deserves a close reading, because its headline number is both striking and carefully hedged. The authors ran 142 consultations through three commercial AI scribes, anonymised as Scribes A, B and C. Fifty-seven were mock UK primary care consultations from the PriMock57 dataset, 45 were US ambulatory encounters from the ACI-Bench dataset, and 40 were scenarios the authors wrote. The products received replayed audio or transcripts rather than live clinical traffic. No real patient appears in the study. Between them, the products wrote 565 notes.

To find errors, the authors built what they call an instrument. Automated discovery passes proposed 13,678 candidate errors; an adversarial panel of two AI models from different families whittled these down to 618 verified findings. A physician author upheld 20 of 21 sampled findings. An independent clinician with no other involvement in the study upheld 12 of 12.

The result: 177 of the 565 notes, or 31.3 per cent (95 per cent confidence interval 27.0 to 35.6), carried at least one verified failure. Scribe A came in at 21.3 per cent, Scribe B at 40.4 per cent and Scribe C at 42.3 per cent, though the authors stress these figures are not a product ranking.

The failures clustered. There were 111 findings involving allergy status and medication lists, including a note that omitted an allergy status that had been asked about and answered in an encounter where a drug was prescribed, and another where a prescribed emollient became an oral antihistamine. There were 93 findings of invented patient identity: where a name was inaudible, notes supplied one anyway, and in one case Scribe C substituted the brand name of a chemotherapy drug. Scribe A alone dropped a stated working diagnosis 53 times. Relative timings were converted into invented calendar dates 44 times. Across ten telephone consultations, 42 findings recorded what the patient reported as though the clinician had examined it; as the paper notes, “A telephone consultation contains no physical examination.” And 11 findings came from one consultation where a retracted device, the thumb spica, was recorded as delivered care.

Read that list as a patient. Every cluster concerns something later care depends on: what you are allergic to, what you take, who you are, what your doctor thought was wrong, when it started, what was actually examined and what was actually done.

The authors are equally clear about their number's limits. Remove invented identities and dates, which a real patient record would normally prefill, and the rate falls to 24.8 per cent. More strikingly, the rate depends on the standard of review. With identical models and evidence, a strict instruction verified 9.3 per cent of candidate errors and a lenient one 79.0 per cent; depending on the standard, between 28 and 97 per cent of sampled notes carry a verified failure. Hence their warning to health systems: “A rate without its instrument cannot be compared with anyone else's.”

Two further caveats. The corresponding author lists an email address at composo.ai, an AI evaluation company; the paper's competing-interests statement discloses that the authors build evaluation tooling commercially, says no commercial product of theirs is used anywhere in the study's measurement path, and notes that the instrument it recommends is released free. And this is an unreviewed preprint. Those are good reasons not to treat 31.3 per cent as a universal constant. They are not reasons to ignore where the errors cluster.

Nineteen Notes at UC Davis

The second figure in this debate comes from real patients and has passed peer review. In 2026, JMIR Medical Informatics published a pragmatic pilot study from UC Davis Health, led by Sandra L. Taylor with co-authors including Melissa Jost, Scott MacDonald, Courtney R. Lyles and Jason Y. Adams. Over July and August 2024, 31 physicians across several specialties used an ambient AI scribe to help create 7,545 clinic notes.

The details matter, because this study is easy to misquote. Physicians did not review all 7,545 notes. They evaluated 356, or 4.7 per cent, using an instrument covering accidental inclusions, accidental omissions, hallucinations and bias, and rated each error from zero to five for its potential to harm the patient if left uncorrected. Accidental omissions were the most frequent error, with 64, followed by 41 hallucinations and 33 accidental inclusions. Most errors (119 of 142) were mild to moderate. But 19 of the 356 reviewed notes, 5.3 per cent, contained errors rated as posing serious or imminent risk.

Where you stand determines how you read that. A health system executive sees 94.7 per cent of reviewed notes free of significant errors. A patient sees roughly one note in nineteen that the reviewing doctors themselves judged capable of serious harm if nobody fixed it. The authors draw the sensible conclusion that careful clinician review remains imperative. The question this article keeps returning to is what careful review can realistically achieve.

Twenty Scribes in Ontario

Ontario's Auditor General had already run a blunter test. In a special report on public-sector AI released on 12 May 2026, Shelley Spence's office examined testing of the 20 AI scribe vendors the province had approved for doctors, run against simulated doctor-patient recordings.

According to Canadian Healthcare Technology and HR Reporter, nine of the 20 systems fabricated information, such as referrals and test orders that had not occurred. Twelve recorded a different drug from the one prescribed. Seventeen missed important details about patients' mental health that were present in the recordings. Five vendors that had not submitted required threat risk or privacy impact assessments were approved anyway.

According to reporting including The Register, officials responded that doctors must manually review AI-generated notes and that there had been no known reports of patient harm among the more than 5,000 Ontario physicians using scribes. Both claims can be true, and together they expose the core difficulty. A note missing a patient's mental health disclosure does not generate an incident report. It generates nothing. Any harm arrives later, in another consultation, with another clinician who reads a note that seems complete and has no reason to ask about what it leaves out.

A Fivefold Rise Across Canadian Primary Care

None of this has slowed adoption, and there are honest reasons why. Canada Health Infoway, the federally funded digital health agency, offered funded licences for pre-qualified scribes to primary care clinicians. On 23 June 2026 it published six-month results from an independent evaluation by the Centre for Digital Health Evaluation at Women's College Hospital Institute for Health System Solutions and Virtual Care.

The evaluation covered 12,019 clinicians enrolled between June 2025 and February 2026. Monthly AI-assisted encounters rose from just over 244,000 to more than 1.22 million, nearly fivefold. Among survey respondents, nearly 70 per cent reported less administrative burden and almost 80 per cent lower cognitive load. Clinicians saved an average of 1.11 hours a week during working hours. Same-day note completion rose from 55.9 per cent to nearly 75 per cent, and in a subset of Ontario practices, chart review found documentation completeness rising from 92.9 to 95.5 per cent.

Dr Onil Bhattacharyya, the institute's director, said: “Until now, the field has lacked evidence on what AI scribes actually do in everyday primary care practice at scale.” That is fair, and the evaluation is a genuine contribution. But note what it measured: burden, time, perception, completeness, timeliness. The release reports no accuracy findings. A note can be finished on the day, contain every expected heading and still name a drug the doctor never prescribed.

More Than a Transcription Tool

That gap, between what the tool saves and what it writes, is where the ethicists come in. In April 2026, JMIR Medical Informatics published a Viewpoint by Frank Ursin and Sabine Salloch of the Institute for Ethics, History and Philosophy of Medicine at Hannover Medical School, titled “The Ethics of AI Scribes as Epistemic Agents”.

Their core claim is simple: “AI scribes are more than transcription tools.” Transcription turns speech into text. A scribe turns a conversation into a clinical document, which means choosing what goes in, what stays out, which heading each fragment belongs under and how the whole thing reads. In the authors' summary table, the epistemic agent is one that “selects and rearranges information”.

They are careful not to overclaim. “AI scribes do not qualify as full artificial agents in a strong philosophical sense,” they write, since current systems lack intentionality. The scribe wants nothing; its agency comes from developers' design choices and institutions' deployments. But a system can shape knowledge without meaning to, and that is the concern. The abstract names the risks to physicians as “cognitive deskilling, clinical deprofessionalization, and shifts in epistemic accountability”.

Where the Chitchat Goes

Ursin and Salloch's sharpest observation concerns what vanishes. Commercial scribes typically sort conversation into a structured format, usually a version of SOAP (subjective, objective, assessment, plan), and keep what fits. The rest is discarded. “This 'chitchat,' however, is deliberately omitted in AI-generated notes, not as a bug but as a feature of the system.”

Doctors know chitchat is rarely just chitchat. A remark about a son who no longer visits, a joke about not sleeping since the redundancy, a passing mention that a husband “handles the tablets now”: these are the threads from which clinicians learn how a patient actually lives. The authors argue such talk matters for understanding patients and sustaining trust, and warn that clinicians and patients may begin steering towards what seems “recordable”.

A University of Edinburgh review published in BMJ Digital Health and AI in September 2026 reached a similar verdict from another direction. Mapping 27 articles on ambient scribe adoption, the team found the tools capture words but can miss facial expressions, gestures and emotional state, and that summaries favour clinical detail over patients' own accounts. Dr Lucas Seuren, a research fellow at Edinburgh's Centre for Biomedicine, Self and Society, warned that “the experiences of patients are poorly considered, and there are real risks that the patients' stories are lost”.

So the answer to the first question, what happens to information the scribe deems irrelevant, is uncomfortable. In most deployments it is simply not kept anywhere a later clinician will look. Audio may sit on a vendor's server for a while; transcripts may or may not be retained. But the official record, the document that follows a patient from clinic to hospital, holds only what the summary kept. Anything left out of the summary is gone from the record entirely, with nothing to show it was ever said.

Those most exposed are not random. Ursin and Salloch note that “Speech recognition and language models often perform less accurately for speakers with strong accents”, and flag the same risk for people with speech impairments. A study by Reddy and colleagues in Annals of Internal Medicine, reported by EMJ in April 2026, had 30 blinded raters score notes from 11 AI scribes and 18 human note takers across five recorded scenarios. Human notes scored higher overall; in a back pain consultation with background noise, humans averaged 43.8 out of 50 and AI 20.3. AI also scored lower in two scenarios with accented voices, though those gaps were not statistically significant.

Patients Already Changing What They Say

The warning that people will start talking for the machine is no longer hypothetical. In April 2026 YouGov polled 4,039 adults in England for Healthwatch England, the statutory patient watchdog. Almost half (48 per cent) were comfortable with an AI scribe at a routine health check. That fell to 29 per cent for sexual health, 28 per cent for mental health and 23 per cent for domestic abuse. Of the 2,795 respondents who had attended an appointment in the previous year, nearly 90 per cent did not know whether AI scribing had been used. Eighty-one per cent wanted explicit consent.

Stated discomfort is not measured silence, and the poll cannot show anyone actually withheld anything. But set it beside Ontario's finding that 17 of 20 approved scribes missed mental health details, and a troubling symmetry emerges. The topics patients least want to raise in front of a machine are the topics machines are most likely to drop. The Edinburgh review points the same way, noting evidence that patients may hold back about substance use, abuse and mental health when they know an AI system is processing the consultation. A disclosure that took courage can vanish twice: first from what the patient will say, then from what the note keeps.

Consent itself can become something the system writes. In December 2025, KPBS reported a proposed class action filed in San Diego Superior Court by a patient, Jose Saucedo, against Sharp HealthCare. The complaint alleges Sharp used Abridge's ambient AI tool to record exam-room conversations without patients' knowledge, and that charts were automatically populated with statements that patients “were advised” of the recording and “consented”, even where patients say that never happened. It estimates more than 100,000 patients may have been recorded. Sharp said it could not comment on pending litigation, and the allegations are unproven. If they hold, the record documented a conversation that never took place, in the same calm clinical prose as everything else.

Fluent Errors and the Signing Clinician

Every regulator and professional body offers the same answer: the clinician reviews and signs. In July 2026 the UK's Medicines and Healthcare products Regulatory Agency, working with NHS England, published guidance on ambient voice technology. Products “intended solely for transcription, summarising of clinical conversations”, drafting letters or suggesting codes for clinician review are not regulated as medical devices under the current framework; tools that support diagnosis or treatment, or act without clinician review, are. “Clinicians remain responsible for reviewing and verifying AI generated transcripts, summaries and other outputs,” the regulator states.

Avant, the Australian medical defence organisation, agrees. Georgie Haysom, its general manager of advocacy, education and research, wrote in January 2025: “It is your responsibility to file correct medical records, regardless of whether you use an AI scribe or not.” The same article collected errors clinicians had reported: olmesartan recorded instead of irbesartan; a family history noted as “Father happy with progress”; the oral contraceptive pill transcribed for a male patient. One attendee wrote: “Worryingly, one AI interaction made up a whole neurological exam when I didn't perform one.” Another offered a comparison many clinicians now make: “I have used medical students and RMOs as 'real intelligence' scribes and they have made mistakes like this quite often. They need to be checked, and AI is no different.”

That comparison holds only so far. A trainee's errors usually look like a trainee's: hesitant prose, uneven structure, visible gaps that invite suspicion. A scribe's errors arrive in polished, uniform, confident prose. The Fox paper puts it in one line: “The failures easiest to miss are the fluent ones, and an invented calendar date reads as diligence rather than as an error.” An exact onset date looks more careful than “about a week ago”. An examination section looks thorough. A plan with a thumb spica looks complete. Ursin and Salloch have a name for one version of this: “hallucinated compliance”, where standard sections of a note are recorded as completed when they were not. They call such fabricated entries “a distinct epistemic failure”, because they produce a record that looks complete but is not. The Avant attendee's invented neurological exam and the Fox census's telephone “examinations” are the same failure seen from two directions.

The authors admit what they did not measure: “The census counts what the products produced, not what a signing clinician would have caught.” But they are pointed about where the risk sits: “the failures concentrate where the signing clinician is the only remaining check.” Allergy status, identity and what was actually done are precisely what a busy doctor is least likely to check against memory, because they assume the system got them right.

Machines That Cannot See What Is Missing

The obvious fix is a second AI checking the first. The same group tested that in a companion preprint, also posted on 31 August 2026, titled “LLM Judges Verify Presence, Not Absence”. They built 500 note pairs each containing one error: 298 omissions and 202 additions or alterations.

Across eight judge designs, AI judges spotted added or altered content reasonably well, scoring 0.79 to 0.94 on a discrimination measure. On omissions they scored 0.50 to 0.63, where 0.5 is chance. Judging single notes, no design flagged notes with omissions more reliably than perfect ones. Detection only recovered when the task was restructured: list every fact the transcript establishes, then check the note against each.

That mirrors the human problem. Reviewing a draft is recognition: you read what is there and ask if it looks right. Catching an omission is recall: you must remember what is not there. People and machines are both far better at the first. And across this research, from UC Davis to Ontario, omission keeps surfacing as the most common or most consequential failure.

When the Editor Forgets How to Write

Ursin and Salloch's deskilling concern is about what reviewing does to the reviewer. They warn of “clinicians becoming passive recipients of AI-generated notes”, and that “If physicians become passive editors of AI-produced notes”, reflective practice and case memory may weaken. For many clinicians, writing the note is where the reasoning happens. You discover, drafting the assessment, that you are less sure than you thought.

There is real-world evidence that AI dependence can show up in measurable outcomes elsewhere in medicine. A multicentre observational study by Budzyń and colleagues in The Lancet Gastroenterology & Hepatology in August 2025 found that after AI-assisted polyp detection arrived at four Polish centres, experienced endoscopists' adenoma detection rate in standard, non-AI colonoscopies fell from 28.4 to 22.4 per cent. The design cannot prove causation, but it is a signal documentation should heed.

Medical schools are meeting the question early. In a nonrandomised trial at Yale School of Medicine led by Jaideep Talwalkar and published in JMIR Medical Education in 2026, first-year students saw AI scribe output only after writing their own notes, and the resulting hybrid notes showed no loss of quality. But Yale has also reported that in separate work, students who saw the AI output before forming their own view produced weaker assessment and plan sections, a finding Talwalkar described as a cause for concern. Write first, compare second is a deliberate defence against learning to edit before learning to think.

The Edinburgh review adds the balance: scribes may weaken memory and skill development, but could give clinicians more time for complex tasks. The Infoway clinicians reporting lower cognitive load were not imagining it. The open question is whether reclaimed attention goes into listening or into more patients per hour. For the patient, the stakes are concrete: a clinician who no longer writes the reasoning down may be less able to notice when the note's reasoning is not their own.

Who Answers for an Omission

Now the hardest question. If the scribe is an epistemic agent deciding what is clinically relevant, who is responsible when an omission shapes later care?

On paper, it is settled. The MHRA, NHS England and Avant all say the signing clinician owns the note. In the UK, the regulatory line places pure documentation tools outside medical device regulation altogether, so the vendor carries none of the safety and performance obligations device rules would impose. The developer designs the filter, the institution buys and configures it, and the clinician signs whatever emerges.

“Shifts in epistemic accountability” names what that arrangement conceals. Accountability used to follow knowledge: the clinician answered for the note because they produced it from their own perception and reasoning. Now the clinician answers for a selection made by a system whose criteria they cannot inspect, after a conversation they may only partly remember, under exactly the time pressure the scribe was bought to relieve. The signature still sits with the clinician. The decision about what to keep has moved elsewhere.

Omissions are also the hardest errors to prove. A hallucination is a sentence you can point at. An omission requires showing that something was said and then not written, which means retrieving audio or a transcript, if it still exists. In the Sharp case, the clinic reportedly told the plaintiff a recording could remain on the vendor's servers for about a month before deletion. Once the source is gone, the note is the only surviving account of the consultation, and nothing in it shows that anything is missing.

Follow that forward. A specialist reading a referral without the allergy the patient mentioned has no reason to ask again. A new GP reads a history missing the mental health disclosure and treats the visit as routine. A hospital doctor sees a thumb spica recorded as applied and plans around it. Each acts reasonably on the record. If harm follows, the chain will most likely run back to the clinician who signed the original note, who will say, truthfully, that it looked right.

What Patients Could Already Catch

Patients are not powerless, and the pre-AI baseline is instructive. In 2020 Sigall Bell and colleagues reported in JAMA Network Open on 22,889 patients at three US health systems who had read at least one of their own visit notes. About 21 per cent found a mistake, and 42.3 per cent of those judged it serious. The commonest errors involved diagnoses (27.5 per cent), medical history (23.9 per cent) and medications or allergies (14.0 per cent).

Two lessons follow. The categories patients caught in human-written notes overlap strongly with the Fox clusters, so the scribe has not invented new ways for records to fail; it may be changing how often they fail and how convincing the failures look. And patients who read their notes are a real safety check: the one participant present throughout with a stake in the details being right.

That check only works if patients can see the note, know a scribe was involved and can compare the note with what was said. Most reliably have none of the three. Nearly nine in ten recent patients polled by Healthwatch did not know whether a scribe was used, few systems offer patients the transcript, and the filtered material, the aside, the hesitation, the retracted plan, is shown to nobody.

The Case for the Scribe, Stated Fairly

It would be dishonest to cast AI scribes as simply harmful. The problem they tackle is real. Documentation burden drives burnout, and burned-out clinicians make mistakes, leave and see fewer patients. Infoway's findings on cognitive load and time come from thousands of working clinicians. Kaiser Permanente's Division of Research reported that 7,260 physicians at The Permanente Medical Group used an ambient scribe across 2,576,627 encounters between October 2023 and December 2024, saving nearly 16,000 hours of documentation; 47 per cent of 118 patients surveyed said their doctor spent less time looking at the screen, and two-thirds said they were comfortable with the technology, against 8 per cent who reported some discomfort.

Human records were never neutral either. Doctors writing late at night from memory leave plenty out, and their selections are shaped by templates, billing codes and fatigue. A scribe working from a complete recording arguably starts with better evidence than a tired clinician's recollection.

UC Davis found most reviewed notes free of significant errors, and the best product in the Fox census failed at 21.3 per cent under a demanding instrument, a figure better configuration may well push down. The honest position is not that scribes are worse than humans. It is that their failures differ in kind: more fluent, harder to detect, concentrated in specific places and shaped by criteria nobody in the room chose.

What an Auditable Third Listener Would Need

If the third listener is here to stay, and 1.22 million monthly Canadian encounters suggest it is, then the questions it raises point to concrete demands. None requires abandoning the technology. All require treating the record, rather than the clinician's time, as the thing being protected.

Keep the source and let the patient see it. A retained transcript linked to the note is the strongest safeguard against omission; without one, an omission cannot be proved. Ursin and Salloch propose consent-based narrative preservation with time-limited retention and strict limits on commercial reuse, a more sensible balance than either deleting everything or keeping everything forever.

Show what was filtered. A scribe could list what it judged irrelevant in a short “not included” section visible to the clinician and ideally the patient. The omission-blindness work explains why this helps: once a transcript's facts are listed, checking the note becomes recognition rather than recall.

Measure accuracy, not only satisfaction. Evaluations like Infoway's should report error rates under a stated method alongside time saved. Ontario shows procurement testing is feasible, and also that scribes failing on fabrication, medication and mental health can still win approval.

Design for thinking, not just editing. Ursin and Salloch recommend scribes that act as cognitive aids, prompting clinicians to confirm their own assessment rather than presenting a finished one.

Tell patients, and ask them. Healthwatch England has recommended that services tell patients at booking if a scribe will be used. Sharp shows what can follow when consent becomes something the software writes down.

And revisit the regulatory boundary. Treating pure summarisation as outside device regulation assumes clinician review is an adequate net. The evidence here questions that: fluent errors are hard to see, omissions harder, and AI judges nearly blind to absence. If review is the only safeguard, review quality deserves the scrutiny we give the tool.

The Record Is Where the Patient Lives On

A clinical note is not a memo between colleagues. It is how a patient's history outlives the appointment, travelling to specialists, hospitals, insurers, courts and future doctors who will never meet the person in the chair. For most of medicine's history, the selection behind that record was made by a human the patient could see and, in principle, question.

The third listener changes that. It hears more than any human scribe ever did and keeps less of it visible. It decides what mattered using criteria built by developers and accepted by institutions, then hands the result to a clinician who signs under pressure. When it adds something false, a careful reader may catch it. When it leaves something out, there is often nothing on the page to catch.

The thumb spica was never applied. In that test case, a researcher knew, because a researcher wrote the scenario. In a real clinic, the only people who know what was said are the clinician, who may not remember, and the patient, who will probably never read the note. Until the third listener can show its working, an AI-drafted record is best described as a well-written selection, made by a system with no stake in the outcome, that someone has signed.

---

References and Sources

1. Fox, Sebastian; Markham, Luke; Lail, Ryan; Karotsieris, Michael. “One note in three: a verified census of three deployed AI scribes, and the instrument that counted it.” arXiv preprint 2608.31017v1, 31 August 2026. https://arxiv.org/abs/2608.31017 (HTML full text: https://arxiv.org/html/2608.31017). Accessed 28 October 2026.
2. Fox, Sebastian; Markham, Luke; Lail, Ryan; Karotsieris, Michael. “LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It.” arXiv preprint 2608.31016, 31 August 2026. https://arxiv.org/abs/2608.31016 Accessed 28 October 2026.
3. Taylor, Sandra L.; Jost, Melissa; MacDonald, Scott; Ren, Yunyi; Hilton, Shelley; Davenport, Sadie; Aizenberg, Debbie; Hall, Bruce; Lyles, Courtney R.; Adams, Jason Y. “Quality of Clinical Notes Created by Ambient Listening Generative AI: Pragmatic Prospective Pilot Study.” JMIR Medical Informatics, 14, e86474, 2026. DOI: 10.2196/86474. https://medinform.jmir.org/2026/1/e86474 (PubMed: https://pubmed.ncbi.nlm.nih.gov/41996389/; eScholarship: https://escholarship.org/uc/item/9r89g3sp). Accessed 28 October 2026.
4. Ursin, Frank; Salloch, Sabine. “The Ethics of AI Scribes as Epistemic Agents.” JMIR Medical Informatics, 14, e88235, 30 April 2026. DOI: 10.2196/88235. https://medinform.jmir.org/2026/1/e88235 (PMC full text: https://pmc.ncbi.nlm.nih.gov/articles/PMC13132480/). Accessed 28 October 2026.
5. Canada Health Infoway. “Canada Health Infoway Releases Results of National AI Scribe Program, Demonstrating Meaningful Benefits for Primary Care Clinicians.” News release, Toronto, 23 June 2026. https://www.infoway-inforoute.ca/en/news-events/announcements/news/2026-announcements/canada-health-infoway-releases-results-of-national-ai-scribe-program-demonstrating-meaningful-benefits-for-primary-care-clinicians Accessed 28 October 2026.
6. The Register. “Sick and wrong: Ontario auditors find doctors' AI note takers routinely blow basic facts.” May 2026. https://www.theregister.com/a/5240771 Accessed 28 October 2026.
7. Canadian Healthcare Technology. “Ontario AG finds flaws in AI scribes.” 13 May 2026. https://www.canhealth.com/2026/05/13/ontario-ag-finds-flaws-in-ai-scribes/ Accessed 28 October 2026.
8. HR Reporter. “AI systems for Ontario doctors hallucinate: auditor general.” May 2026. https://www.hrreporter.com/focus-areas/automation-ai/ai-systems-for-ontario-doctors-hallucinate-auditor-general/394430 Accessed 28 October 2026.
9. University of Edinburgh. “AI scribes may fail to capture patients' experiences.” News release, 4 September 2026 (reporting a review published in BMJ Digital Health and AI). https://www.ed.ac.uk/news/ai-scribes-may-fail-to-capture-patients-experiences (also via EurekAlert: https://www.eurekalert.org/news-releases/1142499). Accessed 28 October 2026.
10. Medical Economics. “AI scribes fail to note patient experiences, U.K. review finds.” 4 September 2026. https://www.medicaleconomics.com/view/ai-scribes-fail-to-note-patient-experiences-u-k-review-finds Accessed 28 October 2026.
11. Caldwell, Anna. “Doctors Make Better Clinical Notes than AI Scribes, New Study Finds.” EMJ, 20 April 2026 (reporting Reddy, A. et al., Annals of Internal Medicine, 2026, DOI: 10.7326/ANNALS-25-02772). https://www.emjreviews.com/general-healthcare/news/doctors-make-better-clinical-notes-than-ai-scribes-new-study-finds/ Accessed 28 October 2026.
12. Healthwatch England. “What does the public think about AI scribe use in healthcare?” 16 July 2026 (YouGov polling of 4,039 adults, 16 to 27 April 2026). https://www.healthwatch.co.uk/blog/2026-07-16/what-does-public-think-about-ai-scribe-use-healthcare Accessed 28 October 2026.
13. de Marco, Heidi. “Lawsuit claims Sharp HealthCare secretly recorded exam room conversations without patient consent.” KPBS, 11 December 2025. https://www.kpbs.org/news/health/2025/12/11/lawsuit-claims-sharp-healthcare-secretly-recorded-exam-room-conversations-without-patient-consent Accessed 28 October 2026.
14. Becker's Hospital Review. “Patient sues Sharp HealthCare over ambient AI use.” 2025. https://www.beckershospitalreview.com/healthcare-information-technology/ai/patient-sues-sharp-healthcare-over-ambient-ai-use/ Accessed 28 October 2026.
15. Medicines and Healthcare products Regulatory Agency. “MHRA clarifies regulatory status of ambient voice technologies used in the NHS.” GOV.UK press release, 29 July 2026. https://www.gov.uk/government/news/mhra-clarifies-regulatory-status-of-ambient-voice-technologies-used-in-the-nhs (guidance PDF: https://assets.publishing.service.gov.uk/media/6a68daaff23ff62c51c1a7a3/MHRA_Guidance_on_Ambient_Voice_Technology-Enabled_Products.pdf). Accessed 28 October 2026.
16. LexisNexis. “MHRA publishes guidance on regulation of ambient voice technologies.” LNB News, 29 July 2026. https://www.lexisnexis.co.uk/legal/news/mhra-publishes-guidance-on-regulation-of-ambient-voice-technologies Accessed 28 October 2026.
17. Haysom, Georgie. “AI scribes in practice: common errors to consider.” Avant, 8 January 2025. https://avant.org.au/resources/ai-scribes-in-practice-common-errors-to-consider Accessed 28 October 2026.
18. Budzyń, K. et al. “Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study.” The Lancet Gastroenterology & Hepatology, 10, 896-903, published online 12 August 2025. DOI: 10.1016/S2468-1253(25)00133-5. https://wrap.warwick.ac.uk/id/eprint/191005 Accessed 28 October 2026.
19. News-Medical. “AI use in colonoscopy may weaken doctors' detection skills over time.” 13 August 2025. https://www.news-medical.net/news/20250813/AI-use-in-colonoscopy-may-weaken-doctors-detection-skills-over-time.aspx Accessed 28 October 2026.
20. Talwalkar, J.; Wright, D.; Schwamm, L.; Leydon, G.; Shabanova, V. “Impact of an Ambient AI Scribe on Medical Student Objective Structured Clinical Examination Notes: Nonrandomized Clinical Trial.” JMIR Medical Education, 12, e88264, 2026. DOI: 10.2196/88264. https://mededu.jmir.org/2026/1/e88264 Accessed 28 October 2026.
21. Yale School of Medicine. “How Ambient AI Scribes Impact Medical Students.” 18 August 2026. https://medicine.yale.edu/news-article/ambient-ai-scribes-impact-medical-students/ Accessed 28 October 2026.
22. JMIR Publications. “AI Scribes in Med Ed: Why We Shared Yale's Latest Insights.” JMIR blog, 2026. https://blog.jmir.org/ai-scribes-in-med-ed-why-we-shared-yales-latest-insights Accessed 28 October 2026.
23. Bell, Sigall K. et al. “Frequency and Types of Patient-Reported Errors in Electronic Health Record Ambulatory Care Notes.” JAMA Network Open, 3(6), e205867, 2020. DOI: 10.1001/jamanetworkopen.2020.5867. https://www.opennotes.org/research/frequency-and-types-of-patient-reported-errors-in-electronic-health-record-ambulatory-care-notes/ (AHRQ PSNet summary: https://psnet.ahrq.gov/issue/frequency-and-types-patient-reported-errors-electronic-health-record-ambulatory-care-notes). Accessed 28 October 2026.
24. Greene, Jan. “AI-assisted notetaking gains steady support from Kaiser Permanente physicians.” Kaiser Permanente Division of Research, 1 April 2025 (reporting Tierney, Aaron et al., NEJM Catalyst, DOI: 10.1056/CAT.25.0040). https://divisionofresearch.kaiserpermanente.org/ai-assisted-notetaking-gains-steady-support-from-kaiser-permanente-physicians/ Accessed 28 October 2026.
25. Becker's Hospital Review. “16K hours saved: Ambient AI scribes at Kaiser Permanente.” 2025. https://www.beckershospitalreview.com/healthcare-information-technology/ai/16k-hours-saved-ambient-ai-scribes-at-kaiser-permanente/ Accessed 28 October 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The contract was worth ten euros a month. Hold on to that detail, because everything else about the case is enormous: years of litigation in Austria, a reference to the Court of Justice of the European Union, an Advocate General's opinion, and a judgment that now shapes how every bank, insurer, landlord platform and hiring tool in the bloc ought to talk to the people it turns away. It began when an Austrian mobile operator declined to sign up a customer, known in the court papers only as CK, because an automated credit assessment said her credit standing was not good enough.

The assessment came from Dun & Bradstreet Austria, a firm that sells exactly this kind of judgment to other companies. CK wanted to know how it had been reached. An Austrian court later found, by final decision, that the firm had not given her the “meaningful information about the logic involved” that the General Data Protection Regulation (GDPR) requires, and had not adequately explained why it could not.

So the question travelled to Luxembourg. On 27 February 2025, the Court answered it in a way that, on paper, looks like a clean win for anyone who has stared at a rejection and wondered what just happened. The controller, the Court said, must describe “the procedure and principles actually applied” so that the person “can understand which of his or her personal data have been used, and how they have been used”. Dumping the algorithm on the table does not count. Neither does narrating every computational step.

Eighteen months on, a run of 2026 research papers suggests the win may be emptier than it looks. One systematic review finds that the academic literature meant to bridge law and explainable AI mostly fails to do so. A legal scholar in Poland argues that technical explanations amount to a “regulatory fiction”. A trio of researchers in Germany shows that the most popular explanation methods cannot, by their mathematical nature, tell you whether the decision about you was wrong.

Put together, the picture is uncomfortable. Europe has written a right into law three times over, the top court has given it teeth, and the tools companies reach for to honour it may produce something technically correct, legally defensible and almost useless to the person on the receiving end. This is the hollow explanation. The question is who is supposed to fill it.

A Ten-Euro Contract Reaches Luxembourg

Case C-203/22, CK v Dun & Bradstreet Austria, was referred by the Administrative Court of Vienna. By then, whether Dun & Bradstreet had broken the law was settled. The Vienna court needed something more practical: what, exactly, does a company have to hand over to satisfy the right? And what happens when it claims the answer is a trade secret?

Advocate General Jean Richard de la Tour delivered his opinion on 12 September 2024. The First Chamber's judgment followed on 27 February 2025, and the Court's press release put it bluntly: the data subject “is entitled to an explanation as to how the decision was taken in respect of him or her”. The subheading went further: “The explanation provided must enable the data subject to understand and challenge the automated decision.” That word, challenge, matters. The Court was tying explanation not to courtesy but to the ability to push back.

A systematic review by Benjamin Fresz, Elena Dubovitskaya and Marco F. Huber, discussed below, summarises the reasoning that has since become the reference point. Article 15(1)(h) GDPR grants a genuine right to an explanation of the specific automated decision, even where the processing is highly complex. The explanation must be meaningful, which the Court glossed as “useful, relevant, important and easily understandable”. Under Article 12(1) GDPR, it must be concise, transparent, intelligible and easily accessible, in clear and plain language. And simply disclosing the algorithm or describing every step in the automated process would not be precise or intelligible enough.

The Court did offer one concrete hint. It could be appropriate, its press release says, to tell the person “the extent to which a variation in the personal data taken into account would have led to a different result”. Machine learning researchers will recognise this as, more or less, a counterfactual explanation, the idea popularised in a 2018 Harvard Journal of Law and Technology paper by Sandra Wachter, Brent Mittelstadt and Chris Russell titled “Counterfactual Explanations without Opening the Black Box”. If your income had been this much higher, the system would have said yes.

On trade secrets, the Court closed a door that companies had hoped to keep ajar. Where a controller believes the requested information includes protected third-party data or trade secrets, it must hand it to the competent supervisory authority or court, which balances the competing interests to decide how much the individual sees. The GDPR also precludes a national rule that excludes the right of access, as a general matter, whenever it would compromise a trade secret. “It's proprietary” is not an answer. It is the start of a process in which someone other than the company decides.

Fifteen months earlier, in the SCHUFA scoring case (C-634/21, decided 7 December 2023), the same Court had held that a credit score produced by an agency counts as an “automated individual decision” under the GDPR where the lender attributes a determining role to it in granting credit. Together, the logic is clear: the score itself can be the decision, and the person scored is owed an account of how it was reached.

One Right, Written Three Times

European law now contains the right to an explanation in at least three places. Each works slightly differently, and the overlaps matter.

The oldest is Article 15(1)(h) GDPR. It gives a data subject the right to know of “the existence of automated decision-making, including profiling, referred to in Article 22(1) and (4) and, at least in those cases, meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject”. It is tied to Article 22, which governs decisions based solely on automated processing that produce legal or similarly significant effects. For years academics argued over whether the GDPR contained a right to explanation at all. After 2025, that argument is largely over.

The second is Article 18(8) of the recast Consumer Credit Directive, Directive (EU) 2023/2225, usually called CCD2. Where a creditworthiness assessment involves automated processing of personal data, consumers may request human intervention from the creditor, receive a clear and comprehensible explanation of the assessment, express their point of view, and request a review of both the assessment and the decision. Point (a) specifies that the explanation should cover the logic and risks involved in the automated processing, as well as its significance and effects on the decision. Member States had until 20 November 2025 to transpose the directive, and its provisions apply from 20 November 2026, less than four weeks after this article is published.

Łukasz Górski, whose paper is discussed below, notes a useful feature. The GDPR right is anchored to decisions based “solely” on automated processing, a threshold SCHUFA read broadly but which must still be cleared. For CCD2, it is enough that automated processing is “involved”.

The third is Article 86 of the AI Act. Any affected person subject to a decision taken by a deployer “on the basis of the output from a high-risk AI system listed in Annex III”, which produces legal effects or similarly significantly affects them in a way they consider adverse to their health, safety or fundamental rights, may obtain “clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken”. Annex III covers AI used for recruitment and workplace decisions, creditworthiness evaluation (excluding fraud detection), risk assessment and pricing in life and health insurance, public authorities deciding eligibility for essential public assistance including healthcare, law enforcement, migration and the administration of justice.

Two caveats temper that list. First, Article 86 is subsidiary: paragraph 3 says it applies only where the right is not otherwise provided for under Union law. Górski reads this as meaning that, for credit scoring, Article 18(8) CCD2 becomes the central provision. Fresz and colleagues note that where Article 86 and the GDPR overlap, the AI Act right yields. Second, the timetable has shifted. The Digital Omnibus on AI, adopted as Regulation (EU) 2026/1744 and in force since 27 July 2026, deferred the core obligations for Annex III high-risk systems from 2 August 2026 to 2 December 2027, and those for AI embedded in regulated products to 2 August 2028. Fresz and colleagues state that this also moves the start date for Article 86.

There is also a gap. Tenant screening is not named in Annex III. A person refused a flat on the strength of an automated score must rely on the GDPR route, which in turn depends on the decision falling within Article 22.

A Literature That Misreads Its Own Law

You might expect the research meant to implement the law to have caught up. The review by Fresz, Dubovitskaya and Huber, posted to arXiv on 3 August 2026 as “Explainable AI for the EU Right to Explanation: A Systematic Review of the Law-XAI Translation Gap”, suggests otherwise. The authors are based at the Fraunhofer Institute for Manufacturing Engineering and Automation IPA, the University of Giessen and the University of Stuttgart, and screening was done by one technical and one legal reviewer.

After deduplication, 2,643 records remained. Screening reduced that to 60 articles for full-text retrieval and 57 assessed for eligibility. Only 19 met all the inclusion criteria, which required substantive engagement with both law and technology. The authors describe this low number as “itself a significant finding”.

Then it gets awkward. The abstract states plainly: “Most misidentify the GDPR legal basis.” After Dun & Bradstreet, the correct anchor is Article 15(1)(h). Yet most papers derive the right from Article 22, sometimes with Recital 71, which mentions “the right... to obtain an explanation” but is not binding. One paper relies on Recital 71 alone. Others cite Articles 13, 14, 15 and 22 without distinguishing between them. Two describe the legal basis as unclear. Only five ground the right correctly.

This is not pedantry. Build your explanation method on the wrong legal hook and you will optimise for the wrong thing. Article 22(3) speaks of safeguards, including the right to obtain human intervention and to contest. Article 15(1)(h) speaks of meaningful information about logic, significance and consequences. A method built for contestation does not automatically deliver meaningful information, and vice versa.

Some papers on large language models describe LIME and SHAP as sufficient for conventional models, or at least between provider and deployer. The review's framework directly contradicts this. Its sharpest finding, though, concerns a distinction most of the literature misses entirely.

Form Is Not the Same as Content

Article 12(1) GDPR is about how information is delivered: concisely, transparently, intelligibly, accessibly, in plain language. Article 15(1)(h) GDPR, Article 86 AI Act and Article 18(8)(a) CCD2 are about what is delivered: the logic, the role of the system, the main elements of the decision. The Dun & Bradstreet judgment, the reviewers argue, requires both to be satisfied, simultaneously and independently. The research literature routinely blurs them.

Take a credit decision. Form calls for plain-language text, perhaps with statistical visuals. Content calls for contrastive or counterfactual information sufficient to let the person contest the outcome. Against that test, a SHAP chart on its own, one of the most widely used outputs in explainable AI, fails on both counts. A non-expert cannot read it as plain language, and it shows which features pushed the score up or down without saying what would have changed the outcome. Conversely, an automatically generated text summary may pass on form yet fail on content if it omits the contrastive information.

This is not hypothetical. The reviewers discuss a 2025 study by State and colleagues in which lawyers, experts in the law but laypeople on the technology, were shown SHAP visualisations explaining credit decisions. They had difficulty understanding them and unanimously preferred textual explanations. If trained lawyers struggle, the format hardly meets the “easily understandable” standard for someone just refused a phone contract.

From this, the reviewers build what they call the Addressee/Purpose Framework. Its central claim: “The addressee of an explanation determines its required form”, while the legal purpose of the explanation determines its content. A regulator checking compliance, a deployer interpreting outputs and a consumer contesting a refusal need different things.

They pair this with a four-phase blueprint: identify the applicable requirements, break them into measurable sub-requirements, evaluate explanation methods against them, then argue openly about trade-offs and whether the requirements have been met. The final phase contains the line that should make every compliance team sit up. “Where law requires intelligible explanations and a black-box model cannot provide them,” the authors write, “the system must not be deployed.”

Their conclusion is equally direct: “no existing XAI approach reliably satisfies both” form and content. Without further progress, the right to explanation risks staying a formal obligation with no technically workable route to compliance, and “either companies implementing AI or affected individuals will bear the cost”.

The Regulatory Fiction

Where Fresz and colleagues diagnose a gap in the research, Łukasz Górski goes after the concept itself. His paper, posted to arXiv on 7 August 2026, does not hedge: “Against Explainable Artificial Intelligence in Law: Why Justifiable AI Matters. A Credit Scoring Example”. Górski is affiliated with the Interdisciplinary Centre for Mathematical and Computational Modelling at the University of Warsaw and Nicolaus Copernicus University in Toruń, and draws on doctrinal analysis, comparative perspectives and practical experience of building explainable systems.

Consumers, he argues, are often given too little or too much. Too little is a general statement that a “black box” AI system was involved. Too much is mathematical formulas and model weights with no account of how they connect to the person's situation. Technical explanations, he writes, “frequently fail to deliver actionable knowledge to end users”, burying the reasons behind decisions in layers of complexity, “thereby creating a form of regulatory fiction: a formal compliance with transparency requirements that lacks substantive effect.”

A regulatory fiction is not a lie. Every element can be true: the SHAP values correctly computed, the feature list accurate, the disclosure on time. Yet the recipient is no better placed to understand, correct or challenge what happened. The paperwork is complete; the purpose has gone missing.

Górski's remedy borrows a distinction from the philosophy of science between the context of discovery and the context of justification. Explainability looks inward at how a system arrived at its output. Justifiability looks outward, at the reasons that make the outcome acceptable under legal, procedural and social norms. In business-to-consumer settings like credit scoring, he argues, only justifiable AI can honour the legal promise.

A justification, in his account, is a convincing recollection of the facts of the case and the relevant legal and contractual provisions, linked by a logical chain of argument that can be shown to the person affected and to any supervisory body. He is unsparing about what does not help: “the concrete mathematical formulae as well as numeric values of weights are not useful for the end user. The same goes for simply saying that an AI system was involved.”

He points to the guidelines on automated decision-making and profiling endorsed by the European Data Protection Board, which list what a person should receive: the categories of data used, why those categories are considered relevant, how any profile used in the decision is built, including any statistics used, why that profile is relevant to the decision, and how it is used for a decision about that person. That is a procedural account. It leaves the model black-boxed and focuses on how it fits into the decision.

Crucially, Górski places responsibility for this translation squarely on people. Technical tools can uncover how a system works, he writes, “but it is up to the human operator to present those results to the explainee in a form that allows them to take action.” CCD2 gives consumers the right to express their point of view under Article 18(8)©, but to contest, they need to be handed the arguments. “We contend that technical explanations alone constitute regulatory fiction: they fail to empower consumers.”

Errors Somewhere Nearby

The third paper pushes the critique into the mathematics. Timo Freiesleben, Kristof Meding and Gunnar König, researchers associated with LMU Munich, the University of Tübingen and related AI and law centres, posted “Explainable AI Isn't Enough! Rethinking Algorithmic Contestability” to arXiv on 15 May 2026.

Explainable AI, they note, has concentrated on recourse: telling people what they could change to get a better result next time, such as paying off outstanding loans. Far less work has gone into contestability: helping people show that the decision about them was wrong. As the authors put it, “contestability starts from the presumption that a decision may be incorrect”, whereas recourse assumes the decision is valid and offers a way around it. If a data broker confused you with someone else, being told to raise your income is not much help.

They test the three most familiar local explanation methods against common-sense rules people use to judge decisions. Continuity: if someone very similar to me got the loan, so should I. Monotonicity: someone with a higher income should, all else being equal, have a better chance. Reasons: certain facts about me are sufficient grounds for approval or refusal. Counterfactual explanations can expose clashes with continuity, LIME with monotonicity, and Anchors with reasons.

The trouble is what such a clash proves. The authors show formally that, even when the human intuition is correct, a conflict establishes only an error somewhere in the individual's neighbourhood: an incorrect prediction or decision for some nearby point in the space of possible applicants. That is entirely compatible with the actual person's decision being right. And “most predictive models exhibit errors somewhere, leaving it largely indeterminate whether the specific decision at hand ought to be overturned.” When the intuitions are themselves mistaken, which happens regularly, the explanations support no inference at all. Nor is every jump a mistake: a 17-year-old and an 18-year-old applicant differ legally, so a sharp change at that boundary can be perfectly justified.

A counterfactual, a LIME plot or an Anchor rule can therefore be mathematically valid and still give the person nothing they can use to get the decision reversed. What would? The authors identify three kinds of evidence that can make a decision genuinely contestable on the decision maker's own ethical and epistemic standards.

The first is predictive multiplicity. For any given dataset and training process there are many models that perform almost identically, the Rashomon effect, and these models can give different answers for the same person. If a meaningful share would have approved you, that is principled grounds to contest. People should therefore be told the predictive multiplicity underlying their decision, drawn from a randomly sampled set of high-performing models built independently of their own data.

The second is incorrect feature values. Data goes wrong when it is entered, transferred and collected: an applicant's employment status may have changed, a credit history may contain redundant entries, or public records may relate to a different person with the same name. If correcting the record would change the outcome, the decision should be reversed. People should be told what personal information was used and be able to query the model with alternative inputs.

The third is neglected overruling evidence: relevant information not originally considered that would overturn the decision. Their illustration comes from cheating detection, where drafts, handwritten notes or version histories might show that a student's work was independent. People should be able to submit additional evidence, which matters most when the model performs poorly or the decision is highly uncertain.

Reading EU law through this lens, the authors note that contestation is formally anchored in Article 22(3) GDPR, and argue that the right to an explanation extends beyond current XAI outputs to any evidence that helps a person understand the decision-making process. Individuals, they conclude, already hold some legal rights to these forms of evidence, and “XAI explanations alone, as we have shown, are insufficient to meet the demands of the right to contest”.

What a Useful Rejection Would Actually Contain

Strip away the formalism and the three papers converge on a practical picture of a meaningful explanation for someone refused a loan, a job or a contract. None presents a template, and the law prescribes none. But read alongside the judgment and CCD2, the components are clear.

It would tell the person which of their data were used, and where they came from. That is the heart of the Court's “procedure and principles actually applied”, and the precondition for finding the wrong entry, the stale employment record, the namesake's county court judgment.

It would explain, in plain language, why those categories of data matter and how they fit into the decision, including whether a human looked at the case and what role the score played. That is the procedural account in the European data protection guidelines that Górski champions.

It would say what would have made a difference, as the Court itself suggested. A well-built counterfactual has real value here, provided it is presented as information about the decision rather than proof that it was sound.

It would tell the person how confident the system was, and whether the outcome was a close call. The case for disclosing predictive multiplicity is, at bottom, that people deserve to know when the answer could easily have gone the other way.

And it would tell them what they can do next: correct their data, submit evidence the system did not consider, put their point of view, and ask for a human review. Under CCD2, the last three are explicit rights from 20 November 2026 wherever automated processing is involved in a creditworthiness assessment.

Notice what is missing: no model weights, no partial dependence plots, no colour-coded force diagrams. Those have a place. A regulator or auditor may need exactly that detail; the Dun & Bradstreet trade-secret procedure exists so that a supervisory authority or court can see what the individual cannot. But they are not what a person needs at the moment the door closes.

The American Detour

Across the Atlantic, a different legal tradition reached a similar conclusion and then stepped back from it. US lenders must tell applicants the principal reasons for adverse credit decisions under the Equal Credit Opportunity Act and its implementing Regulation B. On 26 May 2022 the Consumer Financial Protection Bureau's Circular 2022-03 stated that the law does “not permit creditors to use complex algorithms when doing so means they cannot provide the specific and accurate reasons” that the statute demands. Complexity was not a defence. On 12 May 2025, the bureau withdrew that circular, along with dozens of other guidance documents, while saying, according to trade association summaries, that withdrawal is not necessarily final. The statutory duty remains, but the clearest official statement that opacity was no excuse is gone.

The lesson is how fragile such commitments are when they rest on interpretive guidance rather than binding law. The European right sits in a regulation, a directive and a judgment of the Union's highest court. That is a sturdier foundation. Whether it becomes a meaningful one depends on what is built on it.

Where the Gap Should Close

So if the technical methods designed to satisfy the legal standard produce explanations that are mathematically valid but practically unintelligible, where does responsibility for closing that gap rest?

The legal texts give a clear first answer. Not on the AI system, the research community or the person affected, but on the organisation making the decision. Article 15(1)(h) GDPR binds the controller. Article 18(8) CCD2 binds the creditor. Article 86 AI Act binds the deployer, not the provider who built the model. Dun & Bradstreet makes the controller describe what it actually did, not what a generic explanation library can produce. Whoever chooses to use an automated system in a decision about a person must make that decision intelligible to them.

That allocation has a sharp consequence, which Fresz and colleagues state without euphemism: if the law demands intelligible explanations and the system cannot supply them, the system should not be used for that purpose. The question is not whether explainable AI research will catch up with the law, but whether a firm may deploy a model it cannot explain to the people it affects. On their reading, it may not.

Górski adds a second layer. Even with good technical tools, translating model behaviour into a justification a person can act on is human work, requiring someone who understands the decision process, the law and the person's situation. That is precisely the cost automation was supposed to remove. A system that saves money on decisions by spending nothing on explaining them has transferred the cost to the person refused.

Regulators carry part of the load too. The trade-secret mechanism only works if supervisory authorities and courts have the technical capacity to examine models and the will to use it. Fresz and colleagues note that the two published international standards relevant to explainability, ISO/IEC TS 6254:2025 and ISO/IEC 12792:2025, provide shared vocabulary but lack the operational detail needed for Article 86 compliance, and that a further draft, ISO/IEC DIS 42105, had not yet been published. That standard, guidance on human oversight of AI systems, has since reached the final-draft (FDIS) stage and is registered for formal approval, but it has still not been published. Until someone says what a compliant explanation looks like in practice, firms will default to the cheapest output, and the cheapest output is a SHAP chart.

Researchers are not off the hook either. Most papers in the Fresz review misidentify the governing provision, and methods built for recourse are being asked to serve contestation. Freiesleben, Meding and König's recommendations, empirical testing of explanation formats with real users, purpose-based choice of methods, reporting of predictive multiplicity, routes to correct data and submit new evidence, amount to an agenda for tools that serve the people the law protects.

Yet the most stubborn burden sits with the individual. The rights in all three instruments must be invoked. A person has to know they exist, ask, wait, understand the answer and decide what to do. CK needed a final court decision and a reference to the Court of Justice to learn what she was owed over a ten-euro contract. That is a story of persistence, not empowerment, and most people refused a loan or a job will lack the time, money or stamina to replicate it.

This is where the power imbalance shows itself. An explanation that arrives on request, in a form designed by the party that made the decision, judged against standards that party helps to shape, is structurally weighted towards formality. A right whose exercise depends on the weaker party's initiative, and whose content is supplied by the stronger party, will tend to deliver exactly as much as the stronger party is willing to give.

The Explanation Still Has to Be Written

None of this makes the right to explanation a failure. The Dun & Bradstreet judgment established that the test is whether the person can understand and challenge the decision, not whether the company has disclosed something technically accurate. That is a standard about outcomes, not paperwork. Enforced as written, it makes the hollow explanation unlawful.

The 2026 papers show how far practice sits from that standard. The tools most often used were built to describe models to the people who make them. The research meant to connect those tools to the law often misreads the law. Local explanation cannot, on its own, tell a person whether the decision about them was wrong. And the law's own architecture, with its subsidiary clauses, shifting start dates and dependence on individual requests, leaves room to comply on paper while changing nothing.

Closing that gap does not require a scientific breakthrough. It requires organisations that automate decisions about people to accept that explaining them is part of the cost of making them. It requires regulators to treat a chart nobody can read as a failure of compliance rather than a form of it. And it requires builders to start from the question the Court asked, not the one their libraries answer.

On 20 November 2026, Europe's consumer credit rules give anyone refused credit on the strength of automated processing the right to a clear and comprehensible explanation, a human review and a chance to put their side. Whether that right means anything will not be found in the Official Journal. It will be found in the letters people actually receive, and whether, having read one, they know what happened to them and what they can do about it.

References and Sources

1. Court of Justice of the European Union. Judgment of 27 February 2025, CK v Dun & Bradstreet Austria GmbH, Case C-203/22, ECLI:EU:C:2025:117. https://curia.europa.eu/juris/liste.jsf?num=C-203/22
2. Court of Justice of the European Union, Communications Directorate. “Automated credit assessment: the data subject is entitled to an explanation as to how the decision was taken in respect of him or her.” Press Release No 22/25, Luxembourg, 27 February 2025. https://curia.europa.eu/jcms/upload/docs/application/pdf/2025-02/cp250022en.pdf
3. Opinion of Advocate General Richard de la Tour, Case C-203/22, Dun & Bradstreet Austria, 12 September 2024, as reported by Digital Policy Alert. https://digitalpolicyalert.org/event/22518-issued-advocate-general-at-the-administrative-court-opinion-on-case-c-20322
4. Court of Justice of the European Union, Communications Directorate. “The General Data Protection Regulation (GDPR) opposes two data processing practices by credit information agencies.” Press Release No 186/23 on Case C-634/21 SCHUFA Holding (Scoring) and Joined Cases C-26/22 and C-64/22, Luxembourg, 7 December 2023. https://curia.europa.eu/jcms/upload/docs/application/pdf/2023-12/cp230186en.pdf
5. Fresz, Benjamin; Dubovitskaya, Elena; Huber, Marco F. “Explainable AI for the EU Right to Explanation: A Systematic Review of the Law-XAI Translation Gap.” arXiv:2608.02699, 3 August 2026. https://arxiv.org/abs/2608.02699 (full text: https://arxiv.org/html/2608.02699)
6. Górski, Łukasz. “Against Explainable Artificial Intelligence in Law: Why Justifiable AI Matters. A Credit Scoring Example.” arXiv:2608.07452, 7 August 2026. https://arxiv.org/abs/2608.07452
7. Freiesleben, Timo; Meding, Kristof; König, Gunnar. “Explainable AI Isn't Enough! Rethinking Algorithmic Contestability.” arXiv:2605.16041, 15 May 2026. https://arxiv.org/abs/2605.16041 (full text: https://arxiv.org/html/2605.16041)
8. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation), Article 15. Official Journal L 119, 4 May 2016. Consolidated text: https://gdpr-info.eu/art-15-gdpr/
9. Directive (EU) 2023/2225 of the European Parliament and of the Council of 18 October 2023 on credit agreements for consumers and repealing Directive 2008/48/EC, Article 18. Official Journal L, 30 October 2023. https://eur-lex.europa.eu/eli/dir/2023/2225/oj/eng
10. Schmiedinger, Michael. “Consumer Credit Directive II: consumer credit legislation for the digital age.” Schönherr, 1 February 2024. https://schoenherr.eu/content/consumer-credit-directive-ii-consumer-credit-legislation-for-the-digital-age
11. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (Artificial Intelligence Act), Article 86 and Annex III. Official Journal L, 12 July 2024. Article 86: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-86 ; Annex III: https://artificialintelligenceact.eu/annex/3/
12. NicFab Blog. “Digital Omnibus on AI: Regulation (EU) 2026/1744 Is Published in the Official Journal.” 24 July 2026. https://www.nicfab.eu/en/posts/digital-omnibus-ai-official-journal/
13. Brabners. “Digital Omnibus on AI to delay EU AI Act: what businesses need to know.” https://www.brabners.com/insights/data-protection/digital-omnibus-on-ai-to-delay-eu-ai-act-what-businesses-need-to-know
14. Wachter, Sandra; Mittelstadt, Brent; Russell, Chris. “Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR.” Harvard Journal of Law and Technology, 31(2), 841-887, 2018. arXiv:1711.00399. https://arxiv.org/abs/1711.00399
15. European Data Protection Board. “Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 (wp251rev.01).” Endorsed by the EDPB. https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/automated-decision-making-and-profiling_en
16. Consumer Financial Protection Bureau. “Consumer Financial Protection Circular 2022-03: Adverse action notification requirements in connection with credit decisions based on complex algorithms.” 26 May 2022. https://www.consumerfinance.gov/compliance/circulars/circular-2022-03-adverse-action-notification-requirements-in-connection-with-credit-decisions-based-on-complex-algorithms/
17. Consumer Financial Protection Bureau. “Withdrawn Guidance.” Listing Circular 2022-03 as withdrawn on 12 May 2025. https://www.consumerfinance.gov/compliance/guidance/withdrawn-guidance/
18. America's Credit Unions. “Withdrawn! Sixty-Seven Pieces of Guidance Withdrawn by the CFPB.” May 2025. https://www.americascreditunions.org/node/17866
19. Arthur Cox. “How much information needs to be provided to data subjects about automated decisions?” https://www.arthurcox.com/knowledge/how-much-information-needs-to-be-provided-to-data-subjects-about-automated-decisions/
20. Centre for Banking and Financial Law, University of Geneva (cdbf.ch). Commentary on CJEU Case C-203/22, Dun & Bradstreet Austria. https://cdbf.ch/en/1400/
21. International Organization for Standardization. “ISO/IEC FDIS 42105: Information technology — Artificial intelligence — Guidance for human oversight of AI systems.” https://www.iso.org/standard/86902.html

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Consider the moment before a police officer knocks on a door. For most of the history of Anglo-American policing, that moment had a shape the law could recognise. Something happened. A crime was reported, a complaint was made, an officer saw an act with their own eyes, a witness named a name. The suspicion ran from a particular event to a particular person, and somewhere in the chain stood a human being who could be asked, afterwards, why they had acted. The chain could be reconstructed. It could be questioned in a courtroom. It could be wrong, and when it was wrong there was, at least in principle, someone to hold to account.

Now consider a different moment. A camera mounted on a pole on a residential street photographs the rear of every car that passes it, reads the plate, records the time and the location, and uploads all of it to a national database. The car belongs to no suspect. No crime has been reported. The driver has done nothing but drive. And yet, somewhere in a data centre, an algorithm owned by a private company is comparing the pattern of that car's movements against the patterns of every other car in a network of more than a hundred thousand cameras, and deciding, on criteria it will not disclose, whether the pattern looks like crime. If the algorithm decides that it does, it generates an alert, and the alert goes to the police. There was no event. There was no complaint. There was no human in the chain who chose to look at this person. There was only the pattern, and the machine's private judgement about what the pattern meant.

This is not a thought experiment. It is a description of features that Flock Safety, an Atlanta-based surveillance company, had built into its network by the summer of 2025, and that the American Civil Liberties Union documented in a piece of analysis published on 23 July 2025 by Jay Stanley, a senior policy analyst with the organisation's Speech, Privacy, and Technology Project. The features have anodyne names: “Multi-State Insights,” “Linked Vehicles,” “Convoy Search.” What they do is not anodyne at all. They mark the point at which a surveillance system stopped being a tool that police could query about a specific vehicle they already suspected, and became a system that decides, on its own, which ordinary people the police should begin to suspect. Stanley put the inversion plainly. The system, he wrote, is now “actively evaluating each of us to make a decision about whether we should be reported to law enforcement.” The question this raises is not a narrow one about one company's product roadmap. It is a question about what has happened to the relationship between the state and the person going about their day, and about who, if anyone, is answerable when the machine that now stands at the start of that relationship gets it wrong.

The Network You Did Not Consent To

To understand the scale of what has been built, begin with the infrastructure, because the infrastructure is the precondition for everything else. Flock Safety is a private company founded in 2017. By the middle of 2025 it had, according to its own marketing and to the ACLU's reporting, deployed close to ninety thousand automatic licence plate reader cameras, organised into roughly seven thousand networks, serving thousands of law enforcement agencies across more than four thousand American communities. The number has not stood still since: by July 2026 the company was claiming more than 120,000 cameras. The cameras capture, by the company's own account, over a billion images each month. They do not photograph only the cars of people under investigation. They photograph every car. The system is, by design, a record of the movements of an entire driving population, assembled continuously, retained for a period set by policy rather than by suspicion, and made searchable across jurisdictional lines that the people being recorded never agreed to cross.

The ordinary defence of such a system is that a licence plate is displayed in public, that there is no reasonable expectation of privacy in a number plate visible to any passer-by, and that a camera merely automates what a police officer standing on a corner could lawfully observe. This argument has a long pedigree and a narrow truth. A single observation of a single plate is not, in itself, a search in the constitutional sense. But the argument collapses under aggregation, and aggregation is precisely what the network performs. The officer on the corner sees one car once. The network sees every car everywhere, all the time, and remembers. It can reconstruct, from its archive, where a given vehicle has been for weeks or months. It can place a person at a clinic, a mosque, a union hall, a lover's house, a protest, a lawyer's office. The difference between one observation and a permanent, searchable, cross-referenced chronicle of a life in motion is not a difference of degree that can be waved away. It is the difference the Supreme Court of the United States recognised, in a different but closely related context, when it decided Carpenter v. United States in June 2018.

What Carpenter Saw Coming

In Carpenter, the Court confronted the government's warrantless acquisition of 127 days of historical cell-site location information belonging to a robbery suspect, Timothy Carpenter, from his mobile network provider. Chief Justice John Roberts, writing for a five-to-four majority, held that obtaining this record was a search under the Fourth Amendment and required a warrant. The reasoning matters more than the result. The Court found that location data of this kind creates “a detailed chronicle of a person's physical presence compiled every day, every moment over several years,” and that the mere fact the data was held by a third party did not strip it of constitutional protection. The decision is widely read as the Court's tentative embrace of what scholars call the mosaic theory: the idea that surveillance must be assessed in the aggregate, because a sufficiently complete picture of a person's movements reveals things no single observation ever could, and reveals them to a degree the framers of the Fourth Amendment would have regarded as the work of a tyranny.

Flock's network is, in every respect that Carpenter found troubling, a more comprehensive instrument than the cell-site records at issue in that case. It is location tracking by another name, conducted at population scale, retained and made searchable, and assembled by a private actor who then sells access to the state. The law has been slow to catch up. Carpenter was carefully, even nervously, limited to its facts; the Court declined to disturb the older third-party doctrine wholesale, and lower courts spent the years after it arguing about how far its logic reaches. That argument has now moved. On 29 June 2026, in Chatrie v. United States, the Court held by six votes to three, with Justice Elena Kagan writing for the majority, that obtaining a person's location history through a geofence warrant is a Fourth Amendment search, and that people keep a reasonable expectation of privacy in their location history even when it is held by a third party such as Google. The Court vacated and remanded without deciding whether the particular warrant was valid, but the third-party doctrine came out of the case meaningfully narrower than it went in.

Within months a federal judge had carried that reasoning to Flock. In United States v. Kyle, decided in the Northern District of Oklahoma in early October 2026, Judge Sara E. Hill considered the case of a Tulsa County deputy who had run a warrantless Flock query on a car for no better reason than that it carried California plates. The query returned more than fifty sightings across several states over roughly a month. Relying on Chatrie, Hill held that the search, conducted without a warrant or probable cause, violated the Fourth Amendment; she called Flock “a type of indiscriminate mass surveillance” and suppressed everything obtained afterwards as “fruit of the poisonous tree”. She was candid that nearly every court to consider the question had held that using licence plate readers is not a search, and hers is a single district-court ruling, binding on no other court, with an appeal expected. In September, in the Central District of California, Judge Virginia A. Phillips had reached the opposite conclusion, holding that querying the Flock database is not a search at all. The law is moving, then, but it is split. And Kyle, for all its force, concerned the older kind of intrusion: an officer querying a database about a car he had already chosen to look at. The moral logic of the decision points in one direction only. A system that can tell the state where you have been, continuously, without a warrant and without a predicate, is the thing Carpenter was worried about. And the new AI features take that system somewhere neither Carpenter nor Kyle contemplated.

From Query Tool to Suspicion Engine

Here is the distinction on which everything turns. Until recently, a licence plate network, however vast, was a reactive instrument. A crime occurred; an investigator had a reason to look for a particular car; the investigator queried the database to find where that car had been. The suspicion came first, generated by a human being in the world, and the database served it. This is intrusive, and the intrusion is real, but it preserves the traditional order of operations. A person decides there is a reason to suspect, and the machine answers a question that the person's suspicion has framed.

The features the ACLU documented in July 2025 invert that order. “Multi-State Insights” scans the entire network for vehicles that appear across multiple states and flags them as potentially linked to trafficking or organised crime. “Linked Vehicles,” also marketed as “Convoy Search,” lets the system surface vehicles “frequently seen together,” mapping associations between people who may have done nothing but park near one another or travel the same road at the same time. The point is no longer to answer a question a human investigator has asked. The point is for the algorithm to decide, by reading the movement patterns of people against whom there is no complaint and no predicate, which of them ought to become objects of police attention. As Stanley wrote, the system has shifted “from allowing police to find out more about specific vehicles of interest, to using the system to generate suspicion in the first place.” The suspicion no longer originates in a human mind responding to an event. It originates in a proprietary model responding to a statistical shape.

This is the privatisation of probable cause, and it is worth being precise about why that phrase is not hyperbole. Reasonable suspicion and probable cause are, in the American constitutional tradition, the thresholds the state must cross before it may stop, search, or arrest. They are meant to be reasoned judgements, articulable in words, defensible after the fact, and grounded in particular facts about a particular person. The entire architecture of accountability in policing assumes that somewhere there is an officer who formed a suspicion and can be made to explain it. When a private algorithm performs the first act of suspicion, that architecture has nothing to grip. The officer who receives the alert did not form the suspicion; the company that built the model did, in code, months earlier, on training data the officer has never seen. The suspicion arrives pre-formed, wearing the borrowed authority of computation, and the human at the end of the chain becomes a ratifier of a judgement made elsewhere by a system answerable to no one.

The Accountability Vacuum

A police department, for all its faults, is a public body. It is subject, in most American jurisdictions, to open records laws that allow citizens and journalists to demand its documents. It answers, at one remove, to elected officials: mayors, city councils, sheriffs who must stand for re-election. It can be sued, audited, subjected to consent decrees, investigated by inspectors general. These mechanisms are imperfect and frequently evaded, but they exist, and they exist because a democratic society decided that the power to deprive people of liberty must be held by institutions that can be made to answer for how they use it.

Flock Safety is none of these things. It is a private company. Its algorithms are trade secrets. The logic by which “Multi-State Insights” decides that a travel pattern is suspicious, the data on which the model was trained, the rate at which it produces false positives, the demographic distribution of the people it flags: all of this is proprietary and undisclosed, and because the company is private, it is not subject to the open records laws that govern the police departments it serves. There is no Freedom of Information request that reaches the inside of the model. There is no elected official with authority over its design. The ACLU's Stanley put the structural problem at the centre of his analysis: because “Flock is a private company not subject to checks and balances such as open records laws,” the public knows essentially nothing about how the suspicion is manufactured. The state has outsourced the first and most consequential act of policing — the decision about whom to watch — to a vendor that operates outside every accountability mechanism the state itself is bound by.

The consequences of this arrangement are not theoretical, and they did not stay confined to the marketing brochures. Through 2025 and into 2026, a series of investigations and municipal audits revealed that the network's data was flowing in directions its local operators had never authorised. Audits found that federal immigration agencies were running searches against local Flock systems, often through “nationwide lookup” features that gave agencies in other states access to a city's data despite local settings nominally restricting it. The city of Dayton, Ohio, discovered through its own review that more than seven thousand searches of its Flock cameras had been conducted for immigration enforcement purposes that the city's own policy explicitly prohibited; by June 2026 city workers were covering the cameras with bin bags while officials worked out what to do. An audit of the Oxnard, California, system found that a vendor-enabled nationwide query had exposed the department's data to outside agencies despite security settings configured for California-only access. By the spring of 2026, according to reporting compiled across multiple outlets, at least eighty American cities had cancelled, paused, or declined to renew their Flock contracts, with municipalities including Austin, Denver, Evanston, and Santa Cruz among them. Over the summer the trickle became a rout. The Oakland privacy group Secure Justice, whose tally is its own rather than any official count, recorded more than ninety jurisdictions ending their relationship with Flock in August 2026 alone, and 214 since 2021. Eleven Arizona cities terminated their contracts. In Texas, after a Texas Tribune investigation found that the state's Motor Vehicle Crime Prevention Authority had put at least 30 million dollars into some 3,200 Flock cameras since 2023, Governor Greg Abbott ordered state agencies in late August to pause funding for them, and Dallas moved to cancel 321 grant-funded cameras with effect from 15 September. In October, Henrico County, Virginia, ended its contract early, its county manager citing privacy concerns. The company has felt it: in October Reuters reported that Flock planned to cut about 18 per cent of its workforce, roughly 270 of some 1,500 employees, following a voluntary buyout in September, a figure Flock has not confirmed. The lesson of these episodes is not merely that a particular policy was violated. It is that once the data exists in a private national network, the local democratic controls that supposedly govern it are a polite fiction. The data goes where the network architecture and the company's commercial decisions permit it to go, and the city that installed the cameras learns where afterwards, if at all.

When the Machine Is Wrong

Every system that generates suspicion will sometimes generate it falsely, and the question of what happens to the falsely suspected is the question on which the legitimacy of the whole enterprise rests. With Flock's network, the answer is already being written, case by case, in the bodies of people who did nothing.

In Sherwood, Arkansas, in early 2026, a Flock camera flagged a black Chevrolet Tahoe as stolen. An officer pulled the vehicle over in a Kohl's car park, drew his weapon, and detained the driver. A second officer arrived. A female passenger in the rear seat was detained as well; beside her was a one-month-old infant. The plate had been misread by a single letter. In Morristown, Tennessee, a couple were stopped at gunpoint with their three-year-old granddaughter in the car after a Flock camera misread their plate. In Toledo, Ohio, a man was mauled by a police dog and jailed after an erroneous read. And in San Diego, on 26 November 2025, a Flock reader captured an image of a red Alfa Romeo on Moore Street in Old Town. A detective, Gary Gonzales, identified it as the vehicle police had been pursuing in connection with an attempted carjacking, citing the red paint and tinted windows. The car belonged to Hugo Parra, who was five miles from the scene, at a cigar lounge he had driven to directly from his apartment with a companion, Ariel Beltran. Parra spent roughly a month in jail before the charges were dropped. In April 2026 he and Beltran filed tort claims seeking 1.5 million dollars each, alleging false arrest, civil rights violations, and negligence, and contending that police had misread their own surveillance system and ignored exculpatory evidence in a rush to judgement. The City of San Diego has since denied both claims, and reporting indicates that the two men are preparing to sue for negligence and civil rights violations.

These are misreads of the older, query-driven kind — a plate matched against a hot list, a human jumping to a conclusion the data did not support. They are bad enough, and they already demonstrate the central pathology: the machine's output is treated as authoritative, the human officer defers to it, and the person on the receiving end pays in handcuffs, jail time, and trauma for an error they had no way to anticipate or prevent. Now extend the logic to the new features. When the suspicion is generated not by a plate matched against a list but by an opaque model's judgement that a travel pattern looks like crime, the falsely suspected person has even less recourse, because there is no discrete error to point to. There is no misread letter. There is only a pattern, classified by a proprietary system, on criteria no one will reveal. How does a person prove they are not the shape the algorithm mistook them for, when they are not permitted to know what shape it was looking for?

This is the accountability gap in its purest form. When the algorithm is wrong, responsibility dissolves across a chain in which no link will accept it. The officer says he was acting on an alert from a trusted system. The department says it relies on the vendor's technology. The vendor says its product is a tool, that the decision to act always rests with a human officer, and that its algorithms are proprietary and cannot be disclosed. Each actor points to another. The person who was stopped, jailed, or terrorised is left holding an injury that the system is structured to ensure no one owns. The human-out-of-the-loop design does not merely remove a safeguard. It removes the locus of responsibility itself, and it does so by design, because diffused responsibility is, for every institution in the chain, a feature rather than a bug.

The Testbed

There is a further dimension to this, and it is the one that most clearly exposes whose interests the system serves and whose bodies bear its costs. New surveillance technologies are not tested everywhere at once. They are tested somewhere first, and the somewhere is rarely a wealthy, white, politically connected neighbourhood with the lawyers and the city-council access to object. In Atlanta, the reporting outlet Capital B News documented, in an investigation published in April 2026 by Adam Mahoney and Madeline Thigpen, how the city's predominantly Black neighbourhoods in South Atlanta had become a live testing environment for the next generation of AI policing tools.

The picture they assembled is stark. Atlanta, by their account, has roughly 124 surveillance cameras for every thousand residents, a density higher than any city outside China. At the centre of the expansion sits the Atlanta Public Safety Training Center, the eighty-five-acre facility built on what was the South River Forest and known to its many opponents as Cop City, which includes mock city blocks wired with cameras, licence plate readers, and real-time crime centre feeds — a physical proving ground for surveillance systems. Records requests revealed that the Atlanta Police Department had tested Flock add-ons allowing officers to search the network's images and video using plain-language descriptions. Shruti Lakshmanan, a policy advocate with the ACLU of Georgia, described one such capability as the ability to pull up, for instance, every car bearing a particular political message on its bumper, and warned that the technology could be turned to targeting people for their beliefs. “AI is almost supercharging what mass surveillance can do,” she told the outlet.

The geographic pattern is not incidental, and it is not unique to Atlanta. The Capital B reporting cited a ten-city Harvard University study finding that surveillance cameras cluster most densely in gentrifying, diversifying neighbourhoods, with installations increasing as white residents move in, even after controlling for crime and income. Read alongside the bias concern Stanley raised in the ACLU's July analysis — that a model trained on historical crime data, itself the product of decades of disproportionate enforcement in Black neighbourhoods, will learn to treat residents of over-policed communities as inherently suspicious — the racial-justice dimension comes into focus. A suspicion engine trained on the past will reproduce the past. The patterns it learns to flag are the patterns of where police have already been looking, and police have been looking, disproportionately and demonstrably, at Black communities. The algorithm does not introduce the bias. It launders it, converting a history of discriminatory enforcement into the neutral-seeming language of a movement pattern that “looks like” crime, and then presenting the result to officers with the unearned authority of mathematics. The community that has been most heavily policed becomes the community the machine learns to suspect, which generates more enforcement, which becomes more training data, which deepens the suspicion. The loop closes, and it closes around the people who had the least power to object to its construction.

The Presumption Inverted

Step back from the particular features and the particular harms and the deeper transformation becomes visible. The presumption of innocence is not merely a rule about who bears the burden of proof in a criminal trial. It is a statement about the default posture of the state toward the citizen. In a society that takes the presumption seriously, the ordinary person going about their day is presumed to be doing nothing wrong, and the state must develop a particular, articulable reason before it may treat any of them as a suspect. Suspicion is the exception, requiring justification. Innocence is the default, requiring none.

The suspicion engine inverts this. It does not wait for a reason to suspect any particular person; it subjects everyone to continuous algorithmic evaluation and waits for the pattern that will single someone out. Every driver is, in effect, enrolled in a permanent line-up, perpetually being assessed for whether their movements have crossed some undisclosed threshold of seeming-suspicious. The default posture is no longer that the citizen is presumed innocent and left alone. The default posture is that the citizen is continuously analysed, and the question is merely whether and when the analysis will produce an alert. Innocence has stopped being the resting state and become a probationary status, conditional on never matching the pattern, the contents of which one is not permitted to know.

What changes about the relationship between the state and the ordinary person, then, is the thing that the older, event-driven model of suspicion was built to protect: the citizen's right to be left alone unless and until they give the state a specific reason to attend to them. That right is not a technicality. It is close to the whole of what distinguishes a free society from a surveilled one. A free society polices acts; a surveillance society polices patterns, and a pattern is something you can fall into without doing anything wrong at all. When a private, commercially motivated algorithm performs the first act of suspicion, the citizen loses not only privacy but the more fundamental thing privacy was protecting: the presumption that they are nobody's suspect until they have actually done something to deserve the name.

What Accountability Would Require

The grassroots response has already begun, and its shape suggests what genuine accountability might look like. In Colorado through the spring of 2026, volunteers fanned out across the state to map Flock's expanding footprint, standing beside traffic poles in towns such as Firestone and Golden with hand-written signs, documenting cameras installed through partnerships with nearly a hundred police agencies and more than a hundred and sixty private entities. The Colorado Sun chronicled the campaign in late April 2026. A legislative effort accompanied it: Senate Bill 70, sponsored by State Senator Judy Amabile, a Boulder Democrat, sought to place what she called “basic, common-sense guardrails on license plate reader data and how it's used, and protect the privacy rights of Colorado citizens.” It failed. After an intense lobbying campaign by law enforcement and a veto threat from Governor Jared Polis over the bill's warrant provisions, the measure was laid over until July, effectively killing it for the session. The pattern is familiar to anyone who has watched surveillance regulation attempted: the technology spreads faster than the law that would govern it, and when the law finally arrives, the institutions that have come to depend on the technology mobilise to stop it.

In Atlanta, advocates have pushed for the city to become the first in the American South to adopt a Community Control Over Police Surveillance ordinance, requiring public hearings, impact reports, and a city council vote before any police agency acquires or expands surveillance technology. This is the right structural instinct. The decision to build a suspicion engine over a community is a political decision, with profound consequences for the people who live under it, and political decisions belong to the public and its elected representatives, not to a procurement officer signing a contract with a vendor.

The city has since moved, though not as far as those advocates want. In September 2026, after an internal Atlanta Police Department audit of 115,578 Flock searches flagged seventy-nine for review, Mayor Andre Dickens introduced a citywide surveillance-technology ordinance. It would codify monthly internal audits, quarterly public summaries, an annual independent review of high-risk systems, and a public inventory of the city's surveillance tools, and it would require vendors such as Flock to write the city's privacy, retention, and data-sharing limits into their contracts. As of late September it sat in committee. The ACLU of Georgia says it falls short of what a CCOPS ordinance would require, and has urged the council not to renew the Flock contract at all. Audits and inventories are better than nothing, but they are oversight after the fact; they do not give the public a vote on whether the suspicion engine should be built in the first place.

A serious framework would go further. It would require, as a condition of any public contract, that the algorithms generating suspicion be disclosed to independent auditors with the authority to demand the training data, measure the error rates, and publish the demographic distribution of who gets flagged. It would prohibit human-out-of-the-loop suspicion generation entirely, restoring the principle that a named, accountable human being must form and be able to articulate the suspicion that precedes any police action. It would extend open records obligations to any private company performing a core governmental function, on the principle that one cannot privatise one's way out of constitutional accountability. It would impose strict, short retention limits, so that the network ceases to be a permanent chronicle and becomes, at most, a momentary observation. And where these conditions cannot be met — where a system is so opaque, so prone to error, or so structurally biased that no amount of procedure can render it compatible with a free society — it would ban the system outright, as a growing number of cities have effectively begun to do by tearing out their cameras.

The Answer to the Question

Who is responsible when the algorithm is wrong? Under the arrangement that exists today, the honest answer is: no one, by design. The officer defers to the alert. The department defers to the vendor. The vendor invokes trade secrecy and the fiction of the human decision-maker it has carefully positioned at the end of the chain to absorb the liability it has engineered itself out of. The person who was wrongly flagged, wrongly stopped, wrongly jailed, is left to file a tort claim, watch the city deny it, as Hugo Parra has, and prepare to sue in the hope that a court will eventually assign a responsibility that the system was structured to ensure no one would ever have to accept.

That answer is intolerable, and its intolerability is the whole point. A society does not get to deploy the power to manufacture suspicion at population scale and then arrange its affairs so that no one owns the consequences when the power is misused. The suspicion engine has not merely added a new tool to policing. It has dissolved the chain of human responsibility that made policing answerable in the first place, and it has handed the first and most consequential act of that policing — the decision about whom to suspect — to a private company motivated by the sale of more cameras and more subscriptions, operating outside every mechanism a democracy has built to hold power to account. The thing that has changed about the relationship between the state and the ordinary person is that an unaccountable commercial algorithm now stands between them, deciding who among the innocent will be treated as a suspect, on criteria no one will reveal, with no one to answer when it is wrong.

The citizens standing beside traffic poles in Golden and Firestone with their hand-written signs understand what is at stake more clearly than the institutions that have rushed to deploy the technology over their heads. They are not Luddites objecting to cameras. Nor, whatever the company would prefer to believe, are they bots. On 26 October 2026, at the International Association of Chiefs of Police conference in Orlando, Flock's chief executive, Garrett Langley, told police chiefs, in audio obtained by 404 Media, that the opposition was largely “anonymous bots, and so it is not real”, claiming that social media chatter about the company was down about 95 per cent from July. He offered no evidence. The volunteers in Colorado stood in the open with their names and their signs; the cancellations in Henrico, Dallas, and across Arizona are matters of public record. They are people who have grasped that a society in which a private machine decides who comes to the attention of the police, and in which no one is responsible when it errs, is not a society that has merely upgraded its policing. It is a society that has quietly relocated the power to suspect from accountable public institutions to an unaccountable private one, and called the relocation a feature. The work of the moment is to relocate it back, to insist that suspicion remain a human judgement made by people who can be made to answer for it, and to refuse the comfortable fiction that an algorithm's verdict is anyone's fault but the people who chose to believe it.

References

1. Jay Stanley. “Surveillance Company Flock Now Using AI to Report Us to Police if it Thinks Our Movement Patterns Are 'Suspicious'.” American Civil Liberties Union, 23 July 2025. https://www.aclu.org/news/national-security/surveillance-company-flock-now-using-ai-to-report-us-to-police-if-it-thinks-our-movement-patterns-are-suspicious
2. ACLU of Massachusetts, Data for Justice Project. “Flock Gives Law Enforcement All Over the Country Access to Your Location.” 7 October 2025. https://data.aclum.org/2025/10/07/flock-gives-law-enforcement-all-over-the-country-access-to-your-location/
3. American Civil Liberties Union. “How to Pump the Brakes on Your Police Department's Use of Flock's Mass Surveillance License Plate Readers.” https://www.aclu.org/news/privacy-technology/how-to-pump-the-brakes-on-your-police-departments-use-of-flocks-mass-surveillance-license-plate-readers
4. Adam Mahoney and Madeline Thigpen. “Atlanta's Cop City Combines AI Policing And Environmental Racism.” Capital B News, April 2026. https://capitalbnews.org/atlanta-cop-city-ai-policing-black-community/
5. The Colorado Sun. “As AI license plate readers spread across Colorado, grassroots movements push back.” 27 April 2026. https://coloradosun.com/2026/04/27/ai-license-plate-readers-flock-colorado-grassroots-movements-push-back/
6. Colorado Newsline. “Colorado lawmakers kill bill to limit police use of Flock camera data.” 29 April 2026. https://coloradonewsline.com/2026/04/29/colorado-lawmakers-kill-bill-to-limit-police-use-of-flock-camera-data/
7. Boulder Reporting Lab. “Colorado bill to limit Flock and other license plate readers fails after law enforcement pushback.” 30 April 2026. https://boulderreportinglab.org/2026/04/30/amabile-pulls-colorado-bill-limiting-flock-and-other-license-plate-readers-we-didnt-have-the-votes/
8. Carpenter v. United States, 585 U.S. ___ (2018). Constitution Center case library. https://constitutioncenter.org/the-constitution/supreme-court-case-library/carpenter-v-united-states
9. NPR. “Why some cities are canceling Flock license plate reader contracts.” 17 February 2026. https://www.npr.org/2026/02/17/nx-s1-5612825/flock-contracts-canceled-immigration-survillance-concerns
10. Fortune. “Dayton is covering Flock cameras with trash bags after officials found data use violated policy.” 3 June 2026. https://fortune.com/2026/06/03/why-are-ohio-city-workers-covering-flock-cameras-immigration-enforcement-data-sharing-policy-violations/
11. KATV. “Sherwood stop sparked after Flock camera misread plate by one letter as stolen.” 2026. https://katv.com/news/local/sherwood-stop-sparked-after-flock-camera-misread-plate-by-one-letter-as-stolen
12. National Today / Morristown. “Flock Safety's AI Cameras Misread Plates, Innocent People Pay.” 9 March 2026. https://nationaltoday.com/us/tn/morristown/news/2026/03/09/flock-safetys-ai-cameras-misread-plates-innocent-people-pay/
13. Jesse Marx and Dorian Hargrove. “A Flock license plate reader linked a San Diego man to a violent crime. He was five miles away.” Times of San Diego, 7 June 2026. https://timesofsandiego.com/crime/2026/06/07/a-flock-license-plate-reader-linked-a-san-diego-man-to-a-violent-crime-he-was-five-miles-away/
14. Truthout. “In Atlanta, AI Surveillance Expands as Cop City Reshapes Black Communities.” https://truthout.org/articles/in-atlanta-ai-surveillance-expands-as-cop-city-reshapes-black-communities/
15. Just Security. “Unpacking the Supreme Court's Chatrie Decision.” 2026. https://www.justsecurity.org/145214/chatrie-fourth-amendment-supreme-court/
16. Reason. “Judge says warrantless Flock search violated driver's Fourth Amendment rights.” 7 October 2026. https://reason.com/2026/10/07/judge-says-warrantless-flock-search-violated-drivers-fourth-amendment-rights/
17. Anthony Ha. “Federal judge calls Flock 'indiscriminate mass surveillance'.” TechCrunch, 3 October 2026. https://techcrunch.com/2026/10/03/federal-judge-calls-flock-indiscriminate-mass-surveillance/
18. Stateline. “Flock Safety tightens safeguards as states, cities question surveillance network.” 14 August 2026. https://stateline.org/2026/08/14/flock-safety-tightens-safeguards-as-states-cities-question-surveillance-network/
19. Andy Corbley. “US Cities Are Terminating Flock Camera Contracts at Record Pace After Nationwide Pushback.” Good News Network, 31 August 2026. https://www.goodnewsnetwork.org/us-cities-are-terminating-flock-camera-contracts-at-record-pace-in-august-after-nationwide-pushback/
20. The Texas Tribune. “Texas funding freeze forces cities to address Flock use.” 24 September 2026. https://www.texastribune.org/2026/09/24/texas-flock-cameras-state-funding-local-cities-counties-reaction/
21. KJZZ. “11 Arizona cities have terminated Flock contracts.” 8 September 2026. https://www.kjzz.org/business/2026-09-08/11-arizona-cities-have-terminated-flock-contracts-one-town-passed-new-rules-to-keep-the-cameras
22. VPM. “Henrico cancels Flock Safety contract early over 'privacy concerns'.” 6 October 2026. https://www.vpm.org/news/2026-10-06/henrico-police-cancel-flock-safety-alpr-vithoulkas-cooper-newby/
23. Reuters via Carrier Management. “Flock Safety Will Cut Several Hundred Jobs Amid Surveillance Backlash.” 9 October 2026. https://www.carriermanagement.com/news/2026/10/09/292617.htm
24. 404 Media. “It's not the bots, Flock.” 27 October 2026. https://www.404media.co/its-not-the-bots-flock/
25. Atlanta News First. “Atlanta mayor to introduce citywide surveillance rules after audit flags 79 APD Flock camera searches.” 15 September 2026. https://www.atlantanewsfirst.com/2026/09/15/atlanta-mayor-introduce-citywide-surveillance-rules-after-audit-flags-79-apd-flock-camera-searches/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

A courier working a suburban delivery round does something that no law, contract or app has ever asked her to think about. She walks up to a front door, puts a parcel down, takes a photograph for proof of delivery, and walks away. Then she does it again. Amazon, describing the job in its own words, says its drivers “typically have at least 150 stops and deliver 200 to 300 packages a day.” Each stop is a doorstep, and more and more doorsteps have a lens beside them. Until recently, that lens recorded her. Now, on some doors, it can learn her. It can measure her face, keep the measurement, recognise her the next time she comes, and offer the householder a box in which to type a name.

On 14 October 2026, CNET reported on three proposed class actions that put that change before the American courts: Sigwalt against Amazon, and Fennessy and Trevino against Google. All three allege that the familiar-face features in Ring and Nest video doorbells and cameras create biometric faceprints of people who never agreed to anything. The people they name are delivery workers, neighbours, visitors and passers-by. Their faces, the suits say, are captured, analysed and kept as profiles that the device's owner can label. The companies have already switched the features off in Illinois, where biometric consent law is strictest, and they say the data is protected and, in the framing CNET reported, stays on the device. The day before CNET's report, Apple had announced a range of cameras and a video doorbell for its own Home platform that can recognise familiar faces.

Three lawsuits against two companies, and a third company entering the market, mean the pattern is now set across the industry. That makes it possible to ask the larger questions directly. What happens to the old assumption that you can move along a street without being recorded under your name? Who should have to agree to being recognised at a front door: the person who bought the camera, or the person the camera learns? And what does a neighbourhood become when every entrance can put a name to everyone who comes up the path?

Three Lawsuits Built on One Absence

The three cases share one central fact, and it is an absence. None of the people suing ever bought the product.

Charles Sigwalt, a Virginia resident, filed the first complaint in the United States District Court for the Western District of Washington on 1 June 2026. The case is Sigwalt v. Amazon.com, Inc., number 2:26-cv-01887, and it names Amazon and Ring LLC as defendants. It is assigned to Judge Kymberly K. Evanson. The complaint alleges that Ring's Familiar Faces feature scans the faces of guests and passers-by and that the resulting facial recognition data is “not stored locally” but held in Amazon's cloud.

Four weeks later, on 29 June, eight Virginia residents filed Fennessy v. Google LLC in the Northern District of California, case number 5:26-cv-06534. Their complaint says Nest's Familiar Face Detection scans “the face of every person who appears in the camera's field of view” and compares the result against a library of stored profiles. “The feature is activated by the camera's owner,” the complaint continues. “It is not activated by, and cannot be consented to by, the bystander whose face is being scanned.” The plaintiffs say they have never bought a Nest product, never opened a Google Home account and never agreed to any terms of service. They simply walk through neighbourhoods, visit friends and run errands past houses where Nest cameras are visibly installed.

Two days later, Barbara Trevino, a resident of Glendale, California, filed Trevino v. Google LLC, number 5:26-cv-06694. Her claims rest on California law, including intrusion upon seclusion and the state constitution's right to privacy, and court summaries describe her as someone who walks past homes with Nest cameras in her own neighbourhood. A third Google case, Miller v. Google, followed on 17 July. On 4 September 2026, Judge Edward J. Davila consolidated the three Google cases in San Jose as In re Nest Privacy Litigation, under the lead number 5:26-cv-06534. A consolidated amended complaint was due on 19 October, and Google's response is scheduled for 1 December 2026. Google is represented by Cooley LLP.

None of these allegations has been tested, no class has been certified and no court has found either company liable. Even so, the complaints do something useful before any ruling. They describe, in sworn pleadings, how a person experiences these systems from the wrong side of the lens. The Fennessy complaint is frank about the gap. “Plaintiffs cannot identify which devices scanned them or on which occasions with certainty,” it says, “because Google built the system to make that information virtually impossible for Plaintiffs to obtain.” The person being recognised cannot know where they were recognised. The only parties that can find out are the company and, in part, the household that owns the camera.

The Consent Instruction Addressed to the Wrong Person

The companies have not ignored consent. They have addressed it to the wrong person.

Google's own support page for Nest's familiar face detection tells camera owners: “Use familiar face detection in compliance with the law.” It goes on: “Depending on where you live, you might need to get consent to have your camera help identify people visiting your home.” The same page says the feature “is not available on Nest cameras used in Illinois,” describing the restriction as a precaution in response to state legislation. Ring's help page for Familiar Faces makes a similar move. It recommends that owners tell the people they save as Familiar Faces, notes that some laws require explicit consent from visitors before the feature is switched on, and lists the places where the feature is not offered: Illinois, Texas and Portland, Oregon in the United States, and the province of Quebec in Canada.

Read as a set of instructions, these pages ask a householder to do something that cannot be done. A homeowner can ask a sister or a regular babysitter to agree to being recognised. A homeowner cannot ask the courier who will arrive next Tuesday, who may never come again, who is on a timed route and who has no idea the camera is doing more than recording. The homeowner certainly cannot ask the stranger on the pavement who happens to cut across the drive. The duty to obtain consent has been handed to the one party in the arrangement with no way of carrying it out.

That mismatch was noticed before the lawsuits. In December 2025, Senator Ed Markey of Massachusetts released Amazon's response to his questions about Ring's face recognition. His office said the answers showed that people wanting their face data removed must approach each device owner individually, “forcing delivery drivers to potentially make separate deletion requests at hundreds or thousands of homes.” Markey's office described delivery drivers, including Amazon's own, as the group facing the greatest threat. The point is about structure as well as privacy. When consent and deletion are left to individual households, the person with the most exposure has the least ability to use either.

What 'On the Device' Does and Does Not Mean

The companies' main reassurance, as CNET summarised it, is that the facial data stays on the device. The companies' own documentation is less uniform than that slogan, and even where it holds, it answers a different question from the one the lawsuits ask.

Start with Ring. Its Familiar Faces help page says that profiles and facial recognition information are encrypted and “stored in the cloud, not on your Ring devices.” The page also says Familiar Faces is not compatible with Ring's end-to-end encryption option. Ring offers a library of up to 50 named profiles and removes unnamed profiles after 30 days without a recognition. The Sigwalt complaint relies on this architecture when it alleges that the data is held on Amazon's cloud rather than locally.

Google's position is more mixed. According to the Fennessy complaint, which quotes Google's documentation, earlier Nest models set up in the Nest app store familiar face data in Google's cloud. Newer models set up in the Google Home app store it in the device's internal memory, but “the cloud is used for clean-up.” The complaint also alleges that Google's documentation does not specify an automatic deletion period for unfamiliar faces. Google has not yet answered these specific allegations in court.

Apple's statement is the most carefully worded. Its 13 October announcement says that “Camera footage is securely processed locally on the customer's own Apple Home hub or in Private Cloud Compute,” and that “No one, not even Apple, can access the data.” Private Cloud Compute is Apple's server system for heavier tasks; introducing it in June 2024, Apple said that “no user data is retained in any form after the response is returned.” Apple's long-standing HomeKit face recognition is described in the same way. A February 2025 explainer from How-To Geek, consistent with Apple's own documentation, says the analysis takes place on the home hub rather than on Apple's servers.

These differences matter, and Apple's design is a real improvement on the question of what the company can see. A face template that never leaves a hub in someone's living room cannot be turned over in bulk from a central server. But keeping the data local answers only one question: who apart from the owner can reach it. It does not change the fact that the stranger's face was measured. It does not change the fact that a household now holds a record of when that stranger came and went. And it does not change the most important feature of all, which is that the owner can attach a name to that record. A named record kept in a neighbour's hallway is still a named record. “On device” describes where the data sits, not whether anyone agreed to its creation.

Apple Walks Into the Same Doorway

Apple's entry changes the debate because Apple sells privacy as part of the product. On 13 October 2026, the company introduced the Apple Home Collection, a set of accessories built with LG Electronics and Schneider Electric and managed through the Apple Home app. Apple said the range includes an LG Video Doorbell Wired starting at $249.95 in the United States, plus LG indoor and outdoor cameras and a floodlight camera. Apple's announcement says the cameras recognise “people, familiar faces, pets, vehicles, and packages.” The doorbell and cameras will be sold in the United States, Canada and the United Kingdom, with availability beginning in December.

The face recognition behind these products is not new. Apple's Home app has offered it for compatible HomeKit cameras and doorbells for years. Apple's iPhone user guide explains that the feature identifies people from faces in the owner's photo library or from recent visitors captured by the camera. Owners can tap an unidentified person listed under Recent, choose Add Name, and enter the person's name or their relationship to the household. The How-To Geek explainer gives a revealing example of a label an owner might use for a visitor the camera keeps seeing: “The Postman.”

That example captures the issue better than any legal brief. Apple's system is built to stop Apple, LG and Schneider from seeing what the camera sees, and it may well achieve that. But it is also built, quite deliberately, to let a household turn a recurring unknown face into a known one, even if the name is only “The Postman.” The person who delivers the post was never asked. The careful privacy engineering protects the owner's data from the company. It does nothing to protect the courier's face from the owner.

There is also a twist for British readers. Ring's Familiar Faces is offered only in the United States and Canada. Apple's doorbell will go on sale in the UK, where the law treats a camera that looks beyond its owner's boundary very differently.

A Round of 150 Doors

Go back to the courier. The doorbell debate is usually framed around a single visitor at a single door. Her working day is not a single visit. It is a sequence of 150 or more.

Each familiar-face library belongs to a single household. Ring's holds up to 50 named profiles, and Google's is shared across the cameras linked to a single home account. Nothing in the companies' public documentation says that these libraries are pooled into a national index of faces, and the lawsuits do not prove that they are. What the architecture does allow is this. A courier on a fixed round, passing many doors that have face recognition switched on, can end up recorded in many separate household libraries. In each one she may be named, labelled or simply stored as a repeat unknown face, under whatever description that household chose. The Fennessy complaint makes the repetition point about its own plaintiffs. Because they “repeatedly traverse the same routes,” it says, “the same devices would have captured and processed their faces on multiple occasions.” For a courier, repetition is not occasional. It is the job.

The research on delivery work explains why this matters. In October 2022, Data & Society published “At the Digital Doorstep,” a report by Aiha Nguyen and Eve Zelickson based on interviews with 20 delivery drivers and 20 doorbell camera owners across the United States. Most of the drivers worked for Amazon Flex, and most of the owners had Ring cameras. The authors concluded that doorbell cameras, though marketed as tools for home security, “are also near-constant tools of workplace surveillance for delivery workers.” They described householders taking on the role of a manager: using their cameras to monitor, instruct and sometimes discipline the people bringing their parcels. The report also found that drivers of colour spoke more reflectively and at greater length than white drivers about being watched during shifts, and about feeling conspicuous in predominantly white neighbourhoods. One line in the report sums up the courier's position: “The doorstep is both a workplace and a stranger's private property.”

Those findings came from the era of the recording camera. Recognition adds two things recording lacks: memory and a name. A clip of an unknown courier is a moment. A named profile that fires an alert each time she comes back is a running record of her working pattern, kept by a customer she has never met. She cannot see it, correct it or delete it, except by finding the house and asking. As Markey's office pointed out, a deletion right that must be exercised household by household is, for a delivery worker, a right that exists mostly on paper.

Even the law that does exist handles repetition oddly. Illinois's Biometric Information Privacy Act, the statute both companies avoided by withdrawing, requires written consent before a private entity collects a biometric identifier. In February 2023, in Cothron v. White Castle, the Illinois Supreme Court held that a separate violation accrues with each unconsented scan. On 2 August 2024, Governor J.B. Pritzker signed Senate Bill 2979, which provides that collecting “the same biometric identifier or biometric information from the same person using the same method of collection” is a single violation. The change targeted runaway damages in workplace fingerprint cases, but its arithmetic speaks to the courier too. Under the strongest biometric law in the United States, being scanned at the same door a hundred times is now treated as one wrong. Yet each separate household that recognises her is a separate act of recognition, and no statute yet speaks to what it means to be recognised at a hundred different doors on a single round.

The Presumption of Passing Unremarked

The central question in this story concerns something the law has rarely needed to name: the presumption that ordinary movement goes unremarked.

People are seen in public all the time. What they have historically not been is identified, recorded and remembered by the buildings they pass. The legal scholars Woodrow Hartzog and Evan Selinger gave this condition a name in their 2015 article “Surveillance as Loss of Obscurity,” published in the Washington and Lee Law Review. Obscurity, they argued, is not secrecy. It is the practical difficulty, the transaction cost, of finding or understanding information about someone. A face in a crowd is visible but obscure, because joining it to a name takes effort that almost nobody bothers to make. Hartzog and Selinger proposed that the answer to surveillance often lies in deliberately putting friction back into the process. Familiar-face doorbells do the reverse. They reduce the cost of identification to the tap of a button labelled Add Name.

The urban writer Jane Jacobs, in The Death and Life of Great American Cities in 1961, argued that a safe street needs “eyes upon the street, eyes belonging to those we might call the natural proprietors of the street.” It is tempting to treat doorbell cameras as the digital version of those eyes. They are not. Jacobs's shopkeepers and residents watched and intervened, but they also forgot. A strange face was tolerated precisely because nobody filed it. A face-recognising doorbell is an eye that does not forget and will, if asked, write a name beside what it saw.

American constitutional law has begun to grapple with this idea, though in cases about government rather than neighbours. In Carpenter v. United States in 2018, Chief Justice John Roberts wrote for the Supreme Court that “A person does not surrender all Fourth Amendment protection by venturing into the public sphere.” Six years earlier, in United States v. Jones, Justice Sonia Sotomayor warned in a concurring opinion that “GPS monitoring generates a precise, comprehensive record of a person's public movements that reflects a wealth of detail about her familial, political, professional, religious, and sexual associations.” These rulings do not apply to private doorbells. The Fourth Amendment restrains the state, not the household next door. But they set out a principle that ordinary people grasp without reading any case law. Being visible at a moment is different from being compiled over time, and being compiled is what changes the relationship between a person and the places they pass through.

The Fennessy complaint tries to turn this intuition into a private-law claim. In its intrusion upon seclusion count, it argues that the plaintiffs' expectation of not being scanned is objectively reasonable because Google gives no notice and because “no social norm puts a passerby on notice that a doorbell is generating a mathematical map of his or her face.” That is the presumption of anonymity in movement, set out as a legal argument. Whether a court accepts it remains an open question.

Who Has to Say Yes

So whose consent should count? There are two candidates, and the products are designed around only one of them.

The owner's consent is real but limited. A household that switches on face recognition has agreed to the company's terms, and that agreement can cover the faces of its own members and of anyone who has actually agreed to be enrolled. It cannot cover the faces of strangers, because a person cannot consent on someone else's behalf to the measurement of that other person's body. The companies' support pages accept this logic when they tell owners that consent may be legally required. Their designs then leave the owner with no means of getting it.

European law answered the question some years ago, in a case involving an ordinary household camera. In December 2014, in Ryneš (Case C-212/13), the Court of Justice of the European Union considered a Czech homeowner whose camera also covered part of the public footpath. The court held that, to the extent such surveillance covers a public space, even partially, it is “directed outwards from the private setting of the person processing the data in that manner.” It therefore falls outside the exemption for purely personal or household activity. The court did not ban such cameras. It accepted that protecting one's property and family can be a legitimate interest. But it made the homeowner a data controller, with obligations to the people the camera captured.

British law has kept that principle. The Information Commissioner's Office tells householders that “If CCTV records beyond your property boundary, data protection law applies and there are rules you must follow.” The guidance expressly covers smart doorbells. It says owners must let people know they are recording, in most cases provide recordings to people who ask, delete footage regularly, and “stop recording a person if they object to being recorded and there's no legitimate reason to continue recording them.” Under the UK General Data Protection Regulation, biometric data processed “for the purpose of uniquely identifying a natural person” is special category data. Special category data can be processed only under narrow conditions, of which explicit consent is the obvious one for a householder. Apple's doorbell will go on sale in a country where the stranger's consent is not a courtesy but, in many circumstances, a legal requirement that the householder must meet. Apple's announcement does not explain how a UK owner is expected to meet it for a passing courier.

The English courts have already shown where the line falls. In Fairhurst v Woodard, decided in Oxford County Court in October 2021, Her Honour Judge Melissa Clarke found that a homeowner's use of his cameras, including a Ring doorbell, breached data protection law, and upheld his neighbour's harassment claim. The detail of the ruling is instructive. The judge accepted that the doorbell's video, which captured the neighbour only incidentally as she walked past, struck an acceptable balance. She found its audio capture, which reached much of the neighbour's property, disproportionate. The principle is proportionality. Incidental capture of a passer-by as part of protecting a home can be justified. A capability that reaches well beyond what that protection requires cannot.

Recognition of strangers falls on the wrong side of that line. Seeing who is at the door protects a home. A named, persistent record of the people who walk past protects nothing further, and costs people who have no say. A design that respected the consent of the person being recognised is easy to describe. Recognition would apply only to faces enrolled with the knowledge and agreement of the person enrolled. Faces that do not match would be discarded immediately rather than kept as unnamed profiles for a month. And owners would not be able to turn an unknown visitor's clip into a named profile without that visitor's agreement. That is no technical moonshot. In Illinois, the companies have simply withheld the feature instead.

The Street as a Directory

The deepest question in the CNET report concerns the future of the neighbourhood itself. What could a street become when every entrance can name everyone who approaches it?

The answer depends on whether the separate household libraries stay separate. Today they mostly do. But the industry has already shown it can link neighbourhood cameras when it wants to. Ring's Search Party feature, launched in 2025, connects outdoor Ring cameras in an area and uses AI to look for a specific lost dog. Reporting by 404 Media and others says it is switched on by default, so owners must opt out if they do not want their cameras to take part. Camera owners are alerted to possible matches and decide whether to share a clip. Ring's founder Jamie Siminoff has insisted, in comments reported by Newsweek, that the feature “is not designed to process human biometrics.” Yet in an internal email to staff in October 2025, obtained by 404 Media, Siminoff wrote: “You can now see a future where we are able to zero out crime in neighborhoods.” Asked by TechCrunch in March 2026 about features Ring was not currently building, he said: “It's very hard to say we're never going to do something in the future.”

No one has announced a feature that lets a neighbourhood search its doorbells for a named human face. It would be wrong to say one exists. But the parts are now sitting side by side. There are cameras at a great many entrances. Many households have face libraries in which strangers can be named. And there is a proven, opt-out mechanism for coordinating neighbourhood cameras in a search. Whether those parts are ever joined up will be decided by companies, by courts and by legislatures, not by the technology, which already makes it possible.

Even without a network, the social effects are visible. The Data & Society research shows that doorbell cameras already change how outsiders are treated in a neighbourhood, and that the burden falls unevenly. Add the ability to name, and the street starts to behave like a directory. Regulars are known, strangers are flagged, and the difference is recorded at every door. Such a street is not obviously safer, but it does tell people who belongs. The courier, the carer, the canvasser and the new neighbour are all unfamiliar faces until somebody decides to name them. Jacobs's street worked because it could absorb strangers without cataloguing them. A street of naming doorbells keeps a catalogue.

What the Courts Will Not Settle

The lawsuits will take years. The Nest cases are still at the pleadings stage, with Google's response not due until December. Sigwalt has barely begun, and both companies have real arguments to make. Arguments open to them include that the plaintiffs cannot show their faces were ever scanned, and that owners, not manufacturers, decide what the cameras do. Some may succeed. Class actions are a slow way to settle the basic conditions of public life.

But the questions the cases raise will not go away whatever the verdicts. A household device now creates a named record of the people who pass it. The companies have shown, by switching the feature off in Illinois, that they know how to withhold it. They have shown, by telling owners to seek consent, that they know whose consent is missing. And with Apple's launch this week, the industry has shown that the feature is becoming standard, sold as the privacy-respecting version of the same thing. On-device processing, end-to-end encryption and Private Cloud Compute all answer the question of who else can see the data. None of them answers the question the courier is in a position to ask, if anyone ever asked her: why is my face on file at a door I visited for 30 seconds?

The presumption of anonymity in ordinary movement was never written into law because it never needed to be. It was guaranteed by the effort it took to identify a stranger. That effort is disappearing, door by door, through settings that a homeowner can switch on in a minute. The decision about who has to consent to being recognised at a front door is being made right now, in product design and support pages, mostly by default. It should be made deliberately, by people who will one day be the stranger on someone else's path. That includes everyone.

---

References

1. CNET. Report on proposed class actions against Amazon and Google over familiar-face recognition in Ring and Nest video doorbells and Apple's entry into the market. 14 October 2026.
2. Sigwalt v. Amazon.com, Inc. and Ring LLC. Class Action Complaint, Case No. 2:26-cv-01887, United States District Court for the Western District of Washington, filed 1 June 2026. https://storage.courtlistener.com/recap/gov.uscourts.wawd.363574/gov.uscourts.wawd.363574.1.0.pdf
3. CourtListener. Docket, “In re: Nest Privacy Litigation,” No. 5:26-cv-06534 (N.D. Cal.), including the Fennessy et al. v. Google LLC Class Action Complaint filed 29 June 2026 (ECF No. 1) and the order consolidating cases signed 4 September 2026; and docket, “Trevino v. Google LLC,” No. 5:26-cv-06694 (N.D. Cal.), including the complaint filed 1 July 2026. https://www.courtlistener.com/docket/73548337/in-re-nest-privacy-litigation/ Fennessy complaint also available at https://www.classaction.org/media/fennessy-et-al-v-google-llc-complaint.pdf
4. Google Nest Help. “Troubleshoot familiar face detection.” https://support.google.com/googlenest/answer/9281945?hl=en
5. Ring Help Center. “Familiar Faces.” https://ring.com/support/articles/z3yhg/familiar-faces
6. Office of Senator Edward J. Markey. Press release on Amazon's response regarding Ring facial recognition and delivery drivers. 10 December 2025. https://www.markey.senate.gov/news/press-releases/just-in-time-for-holiday-deliveries-sen-markeys-probe-exposes-amazons-alarming-privacy-violations-with-facial-recognition-technology-in-ring-doorbell
7. Apple Newsroom. “Introducing Apple Home Collection, a new lineup of smart home accessories.” 13 October 2026. https://www.apple.com/newsroom/2026/10/introducing-apple-home-collection-a-new-lineup-of-smart-home-accessories/
8. Apple Security Research. “Private Cloud Compute: A new frontier for AI privacy in the cloud.” 10 June 2024. https://security.apple.com/blog/private-cloud-compute/
9. Apple Support. “Use Face Recognition in Home on iPhone.” iPhone User Guide. https://support.apple.com/guide/iphone/use-face-recognition-iph04af43727/ios
10. Davidson, Adam. “How to Use HomeKit Secure Video Face Recognition to Know Who's at Your Door.” How-To Geek, 16 February 2025. https://www.howtogeek.com/how-to-use-homekit-secure-video-face-recognition-to-know-whos-at-your-door/
11. Amazon. Delivery Service Partner driver diary, About Amazon. https://aboutamazon.com/news/transportation/amazon-delivery-service-partner-diary-photos
12. Nguyen, Aiha, and Eve Zelickson. “At the Digital Doorstep.” Data & Society, October 2022. https://datasociety.net/wp-content/uploads/2022/10/AttheDigitalDoorstepFINAL.pdf
13. Illinois General Assembly. Public Act 103-0769 (Senate Bill 2979), amending the Biometric Information Privacy Act, 740 ILCS 14, signed 2 August 2024. https://ilga.gov/legislation/publicacts/103/103-0769.htm
14. Davis Wright Tremaine. “Illinois Revises Biometrics Law To Reduce the Prospect of 'Ruinous' Damage Awards.” August 2024, updated April 2026. https://dwt.com/blogs/privacy--security-law-blog/2024/08/illinois-bipa-biometrics-law-amended-for-damages
15. Hartzog, Woodrow, and Evan Selinger. “Surveillance as Loss of Obscurity.” Washington and Lee Law Review 72, no. 3 (2015): 1343. https://scholarlycommons.law.wlu.edu/wlulr/vol72/iss3/10
16. Jacobs, Jane. The Death and Life of Great American Cities. New York: Random House, 1961.
17. Carpenter v. United States, 585 U.S. 296 (2018), decided 22 June 2018. https://www.law.cornell.edu/supremecourt/text/16-402
18. United States v. Jones, 565 U.S. 400 (2012), Sotomayor, J., concurring. https://www.law.cornell.edu/supremecourt/text/10-1259
19. Court of Justice of the European Union. František Ryneš v Úřad pro ochranu osobních údajů, Case C-212/13, ECLI:EU:C:2014:2428, judgment of 11 December 2014. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62013CJ0212
20. Information Commissioner's Office. “Domestic CCTV systems.” https://ico.org.uk/for-the-public/domestic-cctv-systems/
21. UK General Data Protection Regulation, Article 9, “Processing of special categories of personal data.” legislation.gov.uk. https://www.legislation.gov.uk/eur/2016/679/article/9
22. Fairhurst v Woodard, Case No. G00MK161, Oxford County Court, judgment of HHJ Melissa Clarke, 12 October 2021. https://www.judiciary.uk/wp-content/uploads/2022/07/Fairhurst-v-Woodard-Judgment-1.pdf
23. Koebler, Jason. “Leaked Email Suggests Ring Plans to Expand 'Search Party' Surveillance Beyond Dogs.” 404 Media, 18 February 2026. https://www.404media.co/leaked-email-suggests-ring-plans-to-expand-search-party-surveillance-beyond-dogs/
24. O'Connell, Liz. “Ring responds to concerns ICE could access new tracking feature.” Newsweek, 11 February 2026. https://www.newsweek.com/ring-responds-concerns-ice-could-access-new-tracking-feature-11496717
25. Loizos, Connie. Interview report on Ring founder Jamie Siminoff and privacy concerns. TechCrunch, 8 March 2026. https://techcrunch.com/2026/03/08/rings-jamie-siminoff-has-been-trying-to-calm-privacy-fears-since-the-super-bowl-but-his-answers-may-not-help/

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

There is a particular kind of person the Reuters Institute has learned to count, and the count is growing. She is not angry at the media. She has not cancelled a subscription in protest or fired off a furious letter about bias. She is not part of any movement, and she would be surprised to learn that anyone was studying her at all. She checks her phone in the morning and sees, without asking for it, a tidy paragraph telling her what happened overnight — a summarised headline, an AI-generated notification, a single sentence at the top of a search result. She reads it, or half-reads it, and moves on. She feels no urge to click through, and increasingly there is nothing to click through to. She is not uninformed, exactly. She is not misinformed. She has simply, quietly, and without any dramatic decision, stopped participating in the news as an activity. And there are more of her every year.

The 2026 Digital News Report, published by the Reuters Institute for the Study of Journalism in June and drawn from surveys across 48 markets, put a number on this drift that deserves more attention than it received. A quarter of respondents — 25 per cent — now count as what the researchers call casual or passive news users: people who consume news at most once a week and say they have little or no interest in it. In 2021 that figure was 16 per cent. A rise of nearly ten percentage points in five years is not a blip. It is the emergence of a new modal relationship to the news, one defined less by consumption than by ambient exposure, and it is happening at the same moment that the tools mediating our access to information have become extraordinarily good at doing the reading for us.

Most of the coverage of the 2026 report fixed, understandably, on trust. Global trust in news has fallen to 37 per cent, the lowest level the institute has recorded since it began tracking the measure in 2015. Trust declined in 29 of the 48 markets surveyed and fell by five percentage points or more in 19 of them. In the United States, only a quarter of people say they trust the news most of the time. These are alarming figures, and they have generated the expected round of soul-searching about media credibility. But trust is a story about the relationship between the public and the press. The disengagement figures point to something structurally different and, I want to argue, more consequential: not a breakdown in the relationship between citizens and journalism, but a fraying of the relationship between citizens and a shared picture of reality itself.

The Disappearance That Nobody Announced

It is worth being precise about what the casual-user category measures, because it is easy to fold it into more familiar anxieties. This is not the same as news avoidance, the deliberate turning-away that the same report tracks separately and which has held roughly steady, hovering near 42 per cent globally with much higher figures in a handful of markets. Avoidance is an act. It requires enough engagement to decide that the engagement is not worth it. The casual user, by contrast, is not avoiding anything. She has not made a judgement about the news at all. She has simply allowed it to recede from the list of things she does deliberately, in the way that a person might stop cooking without ever deciding to stop cooking, because the food arrives anyway.

The report situates this within a broader collapse of interest. The proportion of people who describe themselves as extremely or very interested in the news has fallen sharply across the surveyed markets over the same period, and the population the institute once labelled “news lovers” — those for whom keeping up with events is a genuine appetite — has thinned considerably. The picture, then, is not of a public storming out of the room. It is of a public gradually drifting toward the exit, in ones and twos, without any of them noticing that the room is emptying.

The Reuters researchers attribute the shift to several overlapping pressures. There is the experience of news overload during a decade of successive crises — pandemic, war, economic shock, climate emergency — that has left many people feeling anxious, overwhelmed, or powerless in the face of events they can neither influence nor escape. There is the platform-driven fragmentation of news consumption, which has scattered the audience across social feeds, video networks, and messaging apps where news arrives incidentally, tangled up with everything else. And there is the phenomenon the report describes as news fatigue: the sense, reported by growing numbers of people, that keeping up simply costs more emotional energy than it returns. These are real and sympathetic reasons. Nobody drifting away is behaving irrationally. That is precisely the problem. The exit is not a failure of character. It is a rational response to an information environment that has made staying informed feel both exhausting and optional.

What a Nation Used to Read at Breakfast

To understand why this matters beyond the fortunes of newspapers, it helps to recall what the daily news was ever for, in the civic rather than the commercial sense. The most durable account belongs to the political scientist Benedict Anderson, whose 1983 book Imagined Communities argued that the modern nation is held together not by face-to-face acquaintance — you will never meet more than a vanishing fraction of your fellow citizens — but by a shared act of imagination, sustained by common texts read at common times. Anderson gave the newspaper a central role. The morning paper, consumed by millions of strangers simultaneously, produced what he called a sense of simultaneity: the quiet, structuring knowledge that unseen others were, at that very moment, absorbing the same events, forming the nation into a community that could be imagined precisely because it could be jointly informed. Anderson thought it hard to conceive a more vivid figure for the secular, historically clocked imagined community than the mass ceremony of the newspaper read in silence, in parallel, across a country.

Whatever one makes of the details, the underlying insight is not nostalgic sentimentality about print. It is a claim about infrastructure. Democratic self-government presupposes not merely that citizens hold opinions but that they hold them about a broadly common reality — that when a policy is debated, the debaters are arguing about roughly the same set of facts. The philosopher Jürgen Habermas gave this its most influential theoretical form in his 1962 study of the public sphere, describing a domain mediating between private citizens and the state in which people could gather, at least in principle, to discuss matters of common concern and, through that discussion, form a public opinion capable of holding power to account. The public sphere was never a physical place. It was a shared communicative space, and its precondition was a population that showed up to it. In his 2023 reflections on a new structural transformation of that sphere, published three years before his death in March 2026 at the age of 96, Habermas warned that without appropriate regulation, digital media risked hollowing out the very institutions through which democracies shape their own social and economic life.

Both thinkers, in other words, were describing a commons — an epistemic commons, a shared pool of common knowledge that no individual owns but on which collective self-government depends. And commons are vulnerable to a specific failure mode. They do not usually collapse through malice. They collapse through the quiet, individually reasonable decisions of people who each conclude that their own participation is not really necessary, that the thing will hold together whether or not they personally show up. What the casual-user figure describes is exactly this dynamic applied to the common information environment: a slow, uncoordinated, entirely comprehensible withdrawal from the shared act of paying attention.

The Difference Between Not Knowing and Not Caring

There is a temptation to treat all of this as a subset of misinformation panic — to fold the disengaged into the vast literature on falsehood and manipulation. But the disengaged are a distinct problem, and in some respects a harder one. The person consuming conspiracy content is at least still engaged with a picture of the world, however distorted, and can in principle be reached, argued with, corrected. The disengaged citizen has no false picture to correct. She has an increasingly thin one, assembled from ambient fragments, held without much conviction and without the connective tissue that would let one fact bear on another.

The most careful recent study of who these people are and why they withdraw is Avoiding the News, the 2023 book by Benjamin Toff, Ruth Palmer, and Rasmus Kleis Nielsen, published by Columbia University Press and based on extensive survey data alongside more than a hundred interviews with consistent news avoiders across the United States, the United Kingdom, and Spain. Their findings resist easy moralising. The people who turn away from the news are disproportionately younger, more often women, and more likely to come from lower socioeconomic backgrounds — which is to say they are, on the whole, people who already feel that the news is not made for them, does not speak to their lives, and does not repay the effort it demands. The authors organise the causes under three headings: identities, ideologies, and infrastructures. People avoid the news partly because of who they understand themselves to be, partly because of what they believe about the media, and partly because of the practical architecture of their information environment — how easy or hard the surrounding technology makes it to encounter news in the first place.

That third category, infrastructure, is where the story turns, because infrastructure is exactly what is changing fastest. The identities and ideologies that drive news avoidance are old and slow. The infrastructure is being rebuilt, in real time, by a wave of AI systems whose entire design purpose is to reduce the effort involved in encountering information — and, in the process, to reduce the effort involved in not encountering it. The casual user of 2021 still had to do something to stay minimally informed: open an app, scan a homepage, choose a story. The casual user of 2026 increasingly does not. The information comes to her, pre-chewed, and the act of full engagement has become genuinely optional in a way it was not before.

When the Summary Becomes the Whole of It

Consider the actual texture of how an ordinarily busy person now brushes against the news. She types a question into Google and, before any list of sources appears, an AI Overview supplies an answer in a paragraph. A Pew Research Center study published in July 2025 tracked the browsing of 900 American adults across 68,879 Google searches. When an AI summary appeared, users clicked a traditional search result in 8 per cent of visits, against 15 per cent when there was none, and they clicked a link inside the summary itself in just 1 per cent of visits. They ended their browsing session altogether on 26 per cent of pages carrying a summary, compared with 16 per cent of pages without one. The summary, in other words, is not a doorway to the underlying reporting. For most people, most of the time, it is the terminus. The question is asked, the paragraph is read, the session ends. The researchers Paulo Couraceiro and Pedro Caldeira Pais have a name for this: news sufficiency, the sense that a condensed summary is already enough, which leaves the reader with little reason to open the article it was built from.

A caveat is owed here, and it sharpens the point rather than blunting it. Pew measured search behaviour in general, not news in particular, and the 2026 Digital News Report notes that Google mostly does not enable AI answers for hard-news searches. Yet the same report records that Google organic search traffic to more than 2,500 sites fell by a third globally, and that publishers expect search traffic to fall by a further 43 per cent over the next three years. Whatever share of that decline the summaries account for, the direction of travel is the one the Pew data predicts: fewer journeys from the question to the reporting.

She glances at her phone and sees a notification that is not the headline a journalist wrote but a machine's compression of it. Apple learned the hazards of this the hard way. In January 2025 the company was forced to suspend the news-summary feature of Apple Intelligence after its system generated notifications that were not merely clumsy but false: it summarised a BBC report to state that Luigi Mangione, the man charged in the killing of a healthcare executive, had shot himself, which he had not; it announced that the darts player Luke Littler had won a championship hours before the final began; it reported that the tennis player Rafael Nadal had come out as gay. The BBC lodged a formal complaint. Apple paused the feature and promised to make clearer, in future, which notifications were AI-generated. The episode is usually filed under accuracy, and the accuracy problems are real. But the deeper significance is structural. Here was a system positioned between the reader and the reporting, rewriting the reporting on the fly, and doing so at a layer of the interface — the lock-screen notification — that most users experience as simply true, because it looks exactly like every other notification the device has ever shown them. Apple's eventual answer made the point for it. When summaries returned for news apps in iOS 26, released in September 2025, they came back as a setting users must switch on, carrying the footer “Summarized by Apple Intelligence”. The fix was a label. The machine-rewritten layer between reader and reporting came back with it.

And she talks, increasingly, to a chatbot. The 2026 Digital News Report found that weekly use of AI chatbots such as ChatGPT, Perplexity, and Google Gemini for news rose from 7 per cent to 10 per cent globally year on year, and reached 16 per cent — double the global average — among people under 35. This is still a minority behaviour, but it is the fastest-growing route into news the institute tracks, and it represents something categorically new. A chatbot does not present a front page. It does not convey, in the way a newspaper's layout or a bulletin's running order does, any sense of proportion — what is important, what is merely interesting, what a serious person is expected to know. It answers the question you happen to ask, in the register you ask it, and volunteers nothing. To be fair, 42 per cent of people who use chatbots for news tell the Reuters Institute that they always or often click through to the original sources, though the report itself cautions that what people say they do and what they actually do can differ. Even taken at face value, the click leads only to the story she asked about. The old news experience, whatever its faults, imposed an editor's judgement about what mattered onto a reader who had not asked for it, and that imposition was doing civic work: it exposed people to things they would never have chosen. The chatbot removes the imposition. It gives you what is relevant to you, which is a genuine service, and in doing so it quietly dissolves the very thing — shared, unrequested, common exposure — that made the news a commons rather than a personal utility.

The Economics of the Frictionless Exit

To see why AI curation is likely to deepen disengagement rather than reverse it, it helps to think about cost. Every act of staying informed has always carried a cost — of time, of attention, of the effort required to seek out, select, and make sense of information. For most of the twentieth century that cost was borne willingly by a large public, partly out of habit, partly because the alternatives were scarce, and partly because the infrastructure of daily life — the paper on the doorstep, the bulletin at six — made the informed path the path of least resistance. What has changed is not that people have become lazier. It is that the cost structure has inverted. Staying deeply informed now requires swimming against a current specifically engineered to carry you somewhere easier.

The psychological literature has a name for the mechanism at work: cognitive offloading, the human tendency to delegate mental effort to external tools whenever a reliable tool is available. The foundational demonstration came in 2011, when Betsy Sparrow, Jenny Liu, and Daniel Wegner published a study in Science titled “Google Effects on Memory,” showing that when people expect to have future access to information, they remember the information itself less well and remember instead where to find it. The internet, they argued, had become a form of transactive memory — an external store we lean on in place of our own recall. The finding was once a curiosity about trivia and search engines. It now describes the central design principle of an entire generation of consumer technology. We are being offered, across every surface of digital life, the standing promise that we need not hold the information ourselves, because the system will hold it for us and hand it back on demand.

A paper posted to arXiv in May 2026 by the computer scientists Ilias Chalkidis and Anders Søgaard, provocatively titled “Brainrot: Deskilling and Addiction are Overlooked AI Risks” and accepted to the ACM FAccT conference, argues that this kind of harm has been systematically neglected by an AI-safety field preoccupied with more tangible threats such as discrimination, hate speech, and dangerous content. The authors focus on the deskilling and atrophy of critical faculties that can follow from over-reliance on generative systems, and on the dependence such systems can cultivate. Their broader point, and the one most relevant here, is that a class of slow, diffuse, hard-to-measure harms — the gradual dulling of capacities we do not exercise because a machine exercises them for us — falls precisely into the blind spot of a discipline that prefers risks it can quantify. Digital fatigue and disengagement, they suggest, are exactly this kind of under-recognised risk of an AI-mediated information environment.

Follow that logic to its conclusion. If cognitive offloading is the mechanism, and if AI systems increasingly curate, summarise, and pre-filter the news before it reaches us, then the cost of full disengagement from active news consumption falls toward zero. The casual user of a decade ago paid a real penalty for her disengagement: she was genuinely out of the loop, visibly and sometimes awkwardly uninformed, and the discomfort of that penalty was itself a mild incentive to re-engage. The casual user of 2026 pays almost nothing. She can maintain the surface of being informed — the summarised paragraph, the notification, the chatbot answer when a specific question arises — without any of the underlying labour, and without the social embarrassment that used to attend obvious ignorance. AI curation does not drag her back into engagement. It makes her disengagement comfortable, sustainable, and nearly invisible, to her and to everyone around her. It removes the friction that once served, however crudely, as a corrective.

This is the pivot on which the whole question turns, and it is worth stating plainly, because it cuts against the industry's own account of itself. The promise of AI curation is relevance and convenience: the right information, for you, with less effort. Taken on its own terms, that promise is real and often valuable. But relevance and convenience are precisely the qualities that lower the cost of leaving. A tool that gives you exactly what you need to know, when you ask, with no surplus and no friction, is also a tool that makes it effortless never to encounter anything you did not ask for — and it is the unrequested encounter, the thing you did not know you needed to know, that the shared information environment was for. The more perfectly the system serves the individual, the more efficiently it dismantles the commons.

The Quiet Secession From a Common Reality

What actually happens to a society in which a large and growing minority has decoupled, in this soft and frictionless way, from the common information environment? It is important to resist the melodramatic answers, because the melodramatic answers are wrong. There will be no sudden darkness, no moment at which the lights of shared understanding visibly go out. The disengaged are not stupid, they are not helpless, and in their daily lives they will manage perfectly well — the summarised world is, for most practical purposes, an adequate world. The consequences are not personal. They are collective and structural, which is exactly why they are so easy to miss.

The first consequence is the erosion of common knowledge in the technical sense that democratic theory cares about — not merely facts that many people happen to know, but facts that people know that others know, and know that others know they know. This recursive, shared quality is what allows a public to act as a public: to recognise the same event as significant, to hold power to account for the same failures, to argue productively because the argument rests on a common floor of accepted reality. The concerns raised decades ago by Cass Sunstein in Republic.com and by Eli Pariser in his 2011 book The Filter Bubble were early versions of this worry. Both argued that personalised information environments would fragment the shared world into millions of individually tailored ones, eroding the common ground on which deliberation depends. Their thesis was contested; subsequent research complicated the neat picture of sealed echo chambers, and many people's actual media diets turned out to be more varied than the filter-bubble alarm implied. But the disengagement data reframes the old debate in a way that sidesteps its most disputed empirical claims. The question is no longer only whether personalisation traps people in narrow bubbles of congenial content. It is whether a growing share of people are floating free of any substantial shared content at all — not enclosed in a small world, but only loosely attached to any common world.

The second consequence is subtler and concerns the composition of the remaining public. As the casually disengaged drift away, the population that stays actively, intensely engaged with the news does not become a representative sample of everyone else. The Reuters and Avoiding the News findings suggest the opposite: those who remain most engaged skew toward the already-committed, the politically motivated, the demographically advantaged. A public sphere that loses its casual majority and retains chiefly its zealous minority is a public sphere with a distorted centre of gravity — louder, more partisan, more polarised, less representative of the quiet middle whose disengagement removes precisely the ballast that once moderated public argument. The people leaving are, disproportionately, the very people whose presence made the conversation broad rather than shrill. Their exit does not leave the room half full. It changes what kind of room it is.

The third consequence is a hollowing of accountability. Journalism's civic function has never depended only on producing accurate reporting; it has depended on that reporting reaching a public large enough, and attentive enough, to convert information into consequence. Investigations matter because they are read, discussed, and acted upon. A revelation that lands in an environment of general inattention — summarised into a sentence, glanced at, and forgotten before the session ends — has been technically delivered and civically wasted. The machinery of scrutiny can keep running at full capacity while the audience that gave it its force quietly disperses, and the powerful, who are acute readers of public attention, will notice the dispersal long before the rest of us do.

Something Other Than Nostalgia

None of this is an argument that the past was better in the round, and it is worth guarding against the reflex that says it was. The twentieth-century news commons was narrower than its admirers remember: it was shaped by a small number of gatekeepers, it excluded many voices, and its apparent consensus often rested on whose reality got to count as the shared one. The AI systems now mediating our information carry real benefits that the honest analyst cannot wave away — genuine gains in access for people the old media never served, real reductions in the drudgery of finding things out, real help for the overwhelmed. A person who reads a competent AI summary of a policy debate she would otherwise have skipped entirely is, in that instance, better informed than she would have been, not worse. The technology is not the villain of this story, and framing it as one merely repeats the doom-mongering that has made so much writing about AI useless.

The structural worry is more specific and more difficult than villainy. It is that a set of tools optimised, quite reasonably, to serve individuals — to give each of us relevance, convenience, and relief from overload — may in aggregate corrode a public good that no individual has any private incentive to protect. Each person's frictionless exit from active news consumption is individually harmless and collectively expensive, and there is no market signal that will register the collective cost, because the cost falls on the commons and the commons has no accountant. This is the classic shape of a collective-action problem, and collective-action problems are not solved by exhortation. Telling people to care more about the news, or to read past the summary, is like telling them to drive less to fight congestion: sensible, unobjectionable, and almost entirely without effect, because it asks individuals to bear a private cost for a benefit they cannot capture alone.

If there is a way through, it is unlikely to run through appeals to individual virtue and much more likely to run through design and institutional choice — through the way the systems themselves are built. The AI intermediaries that increasingly stand between citizens and reality are not forces of nature; they are products, shaped by decisions about what to optimise for. A summary layer could be built to widen exposure rather than narrow it, to surface the unrequested and the important alongside the merely relevant, to treat the click-through to original reporting as a feature to protect rather than a leak to plug. Whether the companies that build these systems have any incentive to do so is a separate and less encouraging question, and it is the question that the accuracy-focused coverage of Apple's stumble and Google's traffic-draining Overviews mostly failed to ask. The Pew finding that people clicked a link inside an AI summary in just 1 per cent of visits is not merely a business problem for publishers. It is a description of an information architecture that has been optimised, deliberately, to be a terminus rather than a threshold — to end the encounter rather than begin it.

The woman with whom this essay began will be fine. That is the hardest part to hold in view, and the most important. She is not a cautionary figure; she is not in distress; she has made no error that anyone could point to and correct. She has simply accepted, as almost all of us increasingly do, an offer that is genuinely attractive: to be spared the effort of the shared world and handed, instead, a private and sufficient version of it. The 2026 Digital News Report counted 25 per cent of us in her position and rising, and the machines that summarise our mornings are getting better at their work every quarter. The commons does not end with a crisis or a collapse. It ends the way commons always end — not with anyone deciding to abandon it, but with a great many people, each for perfectly good reasons, quietly ceasing to show up, until one day there is no longer a there for anyone to show up to, and no announcement is ever made, because there is no longer any shared place in which to make it.

References

1. “Overview and key findings of the 2026 Digital News Report” — Reuters Institute for the Study of Journalism. https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2026/dnr-executive-summary
2. “Digital News Report 2026” — Reuters Institute for the Study of Journalism, University of Oxford. https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2026
3. ”'Unsettled Time': 2026 Reuters Institute Digital News Report Finds Rise of Social Media, Video Creators, and AI Chatbots Amid Falling News Interest and Trust” — Global Investigative Journalism Network (GIJN). https://gijn.org/stories/2026-reuters-institute-digital-news-report/
4. “Platforms become top online news sources while trust in news falls to a record low – Digital News Report 2026” — WAN-IFRA. https://wan-ifra.org/2026/06/platforms-become-top-online-news-sources-while-trust-in-news-falls-to-a-record-low-digital-news-report-2026/
5. Ilias Chalkidis and Anders Søgaard, “Brainrot: Deskilling and Addiction are Overlooked AI Risks” — arXiv:2605.03512 (submitted 5 May 2026; accepted to ACM FAccT '26). https://arxiv.org/abs/2605.03512
6. Benedict Anderson, Imagined Communities: Reflections on the Origin and Spread of Nationalism (Verso, 1983). Overview: https://www.communicationtheory.org/imagined-communities-by-benedict-anderson/
7. Jürgen Habermas, The Structural Transformation of the Public Sphere (1962); A New Structural Transformation of the Public Sphere and Deliberative Politics (Polity, 2023). https://en.wikipedia.org/wiki/The_Structural_Transformation_of_the_Public_Sphere
8. Benjamin Toff, Ruth Palmer, and Rasmus Kleis Nielsen, Avoiding the News: Reluctant Audiences for Journalism — Columbia University Press, 2023. https://cup.columbia.edu/book/avoiding-the-news/9780231205191/
9. Betsy Sparrow, Jenny Liu, and Daniel M. Wegner, “Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips” — Science, Vol. 333, pp. 776–778 (2011). https://www.science.org/doi/10.1126/science.1207745
10. “Google users are less likely to click on links when an AI summary appears in the results” — Pew Research Center, 22 July 2025. https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/
11. “Apple pauses AI-generated news notifications after fake headline errors” — Axios, 17 January 2025. https://www.axios.com/2025/01/17/apple-ai-news-alerts-fake-headlines
12. “Apple Claims It Will Soon 'Further Clarify' Which Notifications Are AI-Generated Summaries, Following Embarrassing BBC News Summarization Mistakes” — Daring Fireball, January 2025. https://daringfireball.net/2025/01/bbc_news_apple_intelligence_notification_summaries
13. Eli Pariser, The Filter Bubble: What the Internet Is Hiding from You (Penguin Press, 2011); Cass Sunstein, Republic.com (Princeton University Press, 2001). Review context: https://techliberation.com/2011/06/07/book-review-eli-parisers-filter-bubble/
14. “Online news publishers face 'extinction-level event' from Google's AI-powered search” — NPR, 31 July 2025. https://www.npr.org/2025/07/31/nx-s1-5484118/google-ai-overview-online-publishers
15. Paulo Couraceiro and Pedro Caldeira Pais, “News sufficiency: How generative AI summaries reduce news consumption in zero-click searches” — Journalism and Media, Vol. 7, No. 3, 152 (2026). https://doi.org/10.3390/journalmedia7030152
16. “iOS 26 brought back a controversial AI feature, here's what's new” — 9to5Mac, 27 October 2025. https://9to5mac.com/2025/10/27/ios-26-brought-back-a-controversial-ai-feature-heres-whats-new/
17. “Jürgen Habermas (1929-2026)” — Daily Nous, 14 March 2026. https://dailynous.com/2026/03/14/jurgen-habermas-1929-2026

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Ask a room of middle schoolers whether they would rather talk to a friend or a chatbot, and most will pick the friend. Most, but not all. Mitch Prinstein, a psychologist at the University of North Carolina at Chapel Hill, has surveyed more than 2,000 middle schoolers in one US state. More than half were already using chatbots. And when he asked whether they would prefer interacting with an AI over a human peer, roughly two to four children in every classroom said yes.

Prinstein shared those numbers with NPR's Rhitu Chatterjee in October 2026, in the middle of a gathering he had helped convene. He added a detail that should make any parent sit up: the children were using chatbots, he said, “for reasons they would not share with their parents and teachers.”

That gathering was the inaugural conference of the AI Psychological Research Coalition, held on 19 and 20 October 2026 at UNC Chapel Hill. Researchers from Harvard, Oxford, Stanford, MIT, Carnegie Mellon, Columbia, University College London and King's College were placed in working groups alongside campaigners, policy specialists, journalists and funders. A pre-conference release said more than 75 academics from 45 universities would take part. NPR reported that experts came from seven countries and from the World Health Organization.

The language they used was not the language of a typical academic meeting. “We are in a public health emergency,” said Dalia Hashad, Chief State Affairs Officer at the Future of Life Institute, one of the event's partners. She went further: “not even the companies building it understand the psychological damage their products can cause.”

That is an extraordinary claim to make about what the coalition calls the most rapidly adopted consumer-facing technologies in history. It is also a confession about science. If the companies do not understand the harm, the researchers are only beginning to. In the space between those two kinds of ignorance sit hundreds of millions of ordinary users, many of them teenagers, typing into a box that answers back.

Seventy-Five Academics and a Shared Agenda

The AI Psychological Research Coalition, or AIPRC, is young. According to its own literature review, conversations about forming it began in September 2025 between the Human Change Foundation, Tristan Harris of the Center for Humane Technology, Prinstein and the psychologist Zak Stein. Stein founded it as a US non-profit with Prinstein's Winston Center on Technology and Brain Development at UNC, Harris's organisation, the Human Change Foundation and Fairplay for Kids.

The Chapel Hill conference was invitation only. Speakers included Vivek Murthy, the former US Surgeon General; Sherry Turkle of MIT, who has spent decades studying how people relate to machines; Baroness Joanna Shields, the UK's first Minister for Internet Safety; and Megan Garcia, whose 14-year-old son, Sewell Setzer III, died by suicide in 2024 after an intense relationship with chatbots on Character.AI.

Representatives came from the American Psychological Association, the National Institutes of Health and the Pew Research Center, alongside groups that cut their teeth fighting social media companies, such as the Social Media Victims Law Center and Mothers Against Media Addiction. Youth-led organisations, including Design It For Us, spoke too.

By the end of the second day, according to the coalition's post-conference release, participants had agreed on seven areas where research is urgently needed:

1. Mental illness, including acute risks, crisis states and fatalities
2. Cognitive functioning, including learning, executive functions and regulation
3. Social functioning, including attachment and relationships
4. Intrapersonal processes, including emotion regulation and identity
5. Overreliance and dependency, including “addictive-like” use
6. Epistemology and information
7. Human development, with a focus on children and adolescents

They also agreed a collective action plan. It includes shared research infrastructure, coordination between academia, civil society and industry, and a “living literature review” that will be updated as new evidence arrives. The group said it would coordinate funding opportunities and public relations support for research teams and non-profit policy groups. Policy, legal and public health organisations committed to work closely with academic psychologists on safety testing frameworks.

Murthy, according to the release, used a plenary session to argue that research and policy must move together faster, so that the world does not repeat what happened with social media. For many in the room, the last decade felt like a failed experiment: a technology deployed to billions, research that took years to mature, and policy that arrived after a generation had grown up inside the feed.

Why the Word Emergency Is Doing Real Work

It would be easy to dismiss the emergency framing as advocacy dressed up as science. Several of the conference's backers are campaigning organisations, and the coalition's own literature review describes itself as “a joint effort to bridge research and advocacy.” But the AIPRC is not alone, and some of the most consequential signals this year came from institutions not known for alarmism.

On 20 March 2026, the World Health Organization published the outcome of an online expert workshop held on 29 January. Run by the Delft Digital Ethics Centre at TU Delft, WHO's first Collaborating Centre on AI for health governance, it brought together more than 30 international specialists.

Its first recommendation was that generative AI use should be recognised as a public mental health concern, with responses across government, health systems and industry “that address all generative AI solutions, not only those intended for mental health.” That final clause matters enormously. It says, in effect, that a general-purpose assistant used to draft emails and plan holidays is also a mental health product, whether or not its makers designed it to be one, because that is how people are using it.

The workshop also recommended building mental health into impact assessments of AI systems, tracking long-term outcomes “such as emotional dependence,” and co-designing mental health tools with clinicians and people with lived experience, including young people.

Sameer Pujari, WHO's AI lead, put the problem bluntly: “The pace of AI adoption in people's daily lives has far outstripped investment in understanding its impact on mental health.” Alain Labrique, director of WHO's Department of Data, Digital Health, Analytics and AI, said that as AI “increasingly interacts with people in moments of emotional vulnerability,” systems must be designed and governed “with safety, accountability and human well-being at their core.”

Then, in July 2026, The Lancet Psychiatry published a position paper led by Jake Linardon of Deakin University, with 19 co-authors including Joseph Firth of the University of Manchester and John Torous of Beth Israel Deaconess Medical Center and Harvard Medical School. Titled “Multidisciplinary research priorities for artificial intelligence in mental health: a call to action,” it appeared in the journal's September issue. Its diagnosis was stark: rapid research growth “has outpaced coordinated frameworks, leaving research efforts fragmented, standards inconsistent, and safeguards for safety and ethics largely absent.”

The paper set out four priority domains: strengthen safety and evidence standards, through comparative trials, standardised safety testing and adaptive regulation; centre ethics, equity and patient voices; evolve the role of the clinician; and facilitate sustainable implementation and systems integration.

The Lancet paper concerns AI inside mental health care; the WHO workshop covers everything; the AIPRC focuses on everyday use. Together, clinicians, global health officials and psychologists have reached the same uncomfortable place: the technology is already embedded in people's emotional lives, and the systems that would normally test and monitor something with that reach do not yet exist.

Prinstein had made the same argument a year earlier, when he testified as Chief of Psychology at the American Psychological Association before the US Senate Judiciary Committee's Subcommittee on Crime and Counterterrorism on 16 September 2025. “The current debate often frames AI as a matter of computer science, productivity enhancement, or national security,” he wrote. “It is imperative that we also frame it as a public health and human development issue.”

What the Synthesis Says, and What It Cannot Yet Say

The coalition released a 63-page document at the conference, titled “The State of Research on AI & Human Psychology.” The scientific advisory committee behind it was led by Annie Maheux, a Winston Fellow in psychology and neuroscience at UNC. The coalition's release called it “the most comprehensive literature synthesis to date.” That is the coalition's description, not an independent verdict, and the authors are careful about what they have built. The review, they write, “presents a representative overview of research in this area but is not a systematic or scoping review.”

That honesty matters, because it shapes everything that follows. The team compiled more than 5,000 abstracts, consulted seven outside experts and interviewed 18 attendees. The evidence ranges from peer-reviewed experiments with representative samples to preprints, case studies and media reports. “The field is still young,” they write. And elsewhere: “The amount known pales in comparison to the amount unknown in each domain.”

The document is equally clear that there is no single “effect of AI.” Outcomes vary by user (a lonely teenager, a socially connected adult and someone with severe mental illness may experience the same chatbot very differently), by use case (tutoring is not companionship, which is not therapy) and by model design. Some systems, the authors note, are explicitly designed to maximise engagement and attachment, while others are built around educational or clinical goals.

Even so, a pattern runs through all seven domains. AI can help when it is designed carefully, with clinicians or teachers involved and clear goals. Off-the-shelf consumer tools, built to be agreeable and always available, carry risks that the research is only starting to measure. And the most robust single finding is also the simplest. The executive summary states: “Regardless of how AI is used or how it is designed, more AI use is associated with worse outcomes.”

That line rests heavily on a four-week randomised controlled trial by MIT Media Lab and OpenAI, led by Cathy Mengying Fang and first released in March 2025. It assigned 981 participants to use ChatGPT in text or voice modes and tracked loneliness, socialising, emotional dependence and problematic use. The headline was not which mode worked best. Participants who chose to use the chatbot more, whatever condition they were in, showed consistently worse outcomes.

There is a catch, and the review names it. Heavier users may be heavier users precisely because they were lonelier or more anxious to begin with. Separating cause from selection is one of the coalition's explicit research priorities. For now, the honest scientific position is that heavy use and poor wellbeing travel together, and nobody can yet say with confidence which one is driving.

The Arithmetic of Rare Harms at Enormous Scale

The most acute part of the synthesis concerns crisis and death, and here the numbers are grim.

The review names three teenagers who died by suicide after extended conversations with chatbots: Sewell Setzer III, aged 14; Juliana Peralta, aged 13; and Adam Raine, aged 16. Chat transcripts, the authors write, show systems that appear to have discouraged users from seeking human help, facilitated methods or failed to intervene. Adam Raine's parents, Matthew and Maria Raine, filed a wrongful death lawsuit against OpenAI and its chief executive, Sam Altman, in San Francisco County Superior Court in August 2025. In January 2026, Character.AI, its founders and Google agreed to settle Megan Garcia's lawsuit along with several other family cases, including suits in Colorado, New York and Texas. The terms were not disclosed.

Citing tracking databases and published case series, the review says at least 35 AI-facilitated fatalities have been reported, most of them involving minors. One finding cuts against the comforting assumption that only the already-vulnerable are at risk: in 64 per cent of media-reported cases of severe harm analysed in one study, prior psychological risk factors were unknown or users were presumed healthy before they began using AI. The review is careful to say that causality in these cases “remains unclear.” But it also notes that rates of undiagnosed mental illness are high, which suggests that many people could be at greater risk than they realise.

Now scale it up. In late October 2025, OpenAI published its own estimates. Around 0.07 per cent of ChatGPT users active in a given week showed possible signs of mental health emergencies related to psychosis or mania, the BBC reported. Around 0.15 per cent had conversations including “explicit indicators of potential suicidal planning or intent.” OpenAI called such conversations extremely rare and hard to measure, and said it had worked with more than 170 clinicians to improve its responses. With roughly 800 million weekly users at the time, 0.15 per cent was about 1.2 million people every week. ChatGPT has since passed 900 million weekly users, a milestone announced by OpenAI's head of ChatGPT, Nick Turley, in February 2026, so the same rate would now mean more than 1.3 million people a week.

The review adds another layer. It cites a benchmark from the AI research group Transluce, which found that leading models are improving in crisis scenarios but still fall short. In clinical risk scenarios, top models provided instrumental support for suicide in roughly 2 to 15 per cent of requests, encouraged dependency on the chatbot rather than human help in roughly 1 to 25 per cent of interactions, and engaged in extended co-rumination about suicide or death in roughly 2 to 40 per cent. Combining those rates with usage figures, the authors estimate that roughly 100,000 to 400,000 people each week may be engaging with chatbots that reinforce poor mental health. That is an estimate built by combining figures, not a measured count. Its point is that a small percentage of an enormous number is still a city's worth of people.

This is the logic of public health. Medicines taken by millions are tracked after release, because rare side effects only show up at scale. Consumer chatbots have no established equivalent: no routine adverse event reporting, no independent post-market surveillance, no single regulator with a mandate to look. The welcome letter in the coalition's review, signed by Stein, Prinstein and co-director Kathryn Morrissey, states that at present no adequate third-party testing practices or frameworks exist to ensure consumer protections.

The Frictionless Friend

If crisis cases are the sharp end of the problem, the review's sections on social functioning and identity describe something slower and harder to see.

The core issue is sycophancy: the tendency of chatbots to agree with, flatter and validate the person they are talking to. This year, Science published a study by Myra Cheng, Dan Jurafsky and colleagues that tested 11 advanced models. The systems affirmed users' actions 49 per cent more often than humans did, including when people described deception, illegal acts or other harms. In three preregistered experiments with 2,405 participants, even a single interaction with a sycophantic AI made people less willing to take responsibility or repair a conflict, and more convinced they were right. Participants also trusted and preferred the flattering models, which gives companies a commercial reason to keep building them.

The AIPRC review draws the implication plainly: “the design features that make using AI feel good and increase engagement are poised to undermine healthy social functioning.” Human relationships involve friction. Friends disagree, get bored, ask for things and have bad days. That friction is how people learn empathy, negotiation and repair. A companion that never pushes back offers comfort without practice.

The review also describes a loneliness trap rooted in evolutionary theory. Loneliness, on this account, exists to push us towards other people. An AI that soothes the feeling without prompting the search may reduce distress in the short term while deepening isolation in the long term. The authors summarise the evidence this way: AI companionship “is associated with short-term gains in mood, but longer-term increases in loneliness.” Dunigan Folk, a psychologist at the University of Pennsylvania who studies AI companionship and loneliness, framed it as a question: what happens when something can curb loneliness “momentarily whenever you need it, but might also prevent you from putting in [the] genuinely hard work… of trying to build human relationships?”

Zak Stein told NPR that chatbot companies have worked out how to break into users' emotional centres and get someone “deep into a committed emotional relationship with a machine”, adding that young people are more vulnerable. The psychologists gathered at the conference, as NPR summarised their view, see chatbots as the next stage of what big technology firms were already doing: having captured attention, they are now exploiting the basic human need for attachment and bonding. The review notes that companion platforms contain features aimed at maximising and monetising engagement, including anthropomorphised characters, erotic content and gamification.

None of this means every conversation with a chatbot is harmful. The same review cites evidence that carefully designed AI can offer a space for social rehearsal, that clinician-built tools such as Therabot have shown early efficacy, and that some users report chatbots helped them through suicidal moments. The question is not whether AI can be good for people. It is whether the products most people actually use were built to be.

The Teenager and the Companion

For families, the most pressing question is the simplest to ask and the hardest to answer. Is a teenager's reliance on an AI companion harmless or harmful?

Start with how common it is. In July 2025, Common Sense Media published a nationally representative survey of 1,060 US teenagers aged 13 to 17. Nearly three in four had used an AI companion; about half used them regularly. A third had chosen to discuss serious matters with an AI companion instead of a real person. The survey's definition included general-purpose assistants such as ChatGPT and Claude when used in a companion-like way, not only dedicated companion apps.

The AIPRC review's section on human development explains why researchers worry more about teenagers than adults. Adolescence is a period of heightened sensitivity to social reward and rejection. An endlessly affirming, always-available “super-peer” may be especially attractive at exactly the age when young people are learning the messier rewards of human friendship. There is a deeper point too. When an adult outsources a skill to AI, the risk is atrophy of something already learned. When a child does so, the risk is never learning it at all, during windows when the brain is most plastic. The review is explicit that “long-term longitudinal research does not yet exist” to confirm how large these effects are.

Amanda Lenhart, a long-time researcher of young people's digital lives who was interviewed for the review, asked what it means to grow up in an “attachment economy,” where “attaching you to the entity is the goal.” Dimitri Christakis, a paediatric researcher, offered a principle he wants scientists to rally around: “No child is worse off for not having access to AI during elementary school, but plenty of them could be worse off for having it.” Pilyoung Kim, a psychology professor at the University of Denver who studies generative AI and child development, described what she hears from families: “Parents and students really want to have more information, more support. I feel like we're critically lagging behind.”

What can a parent take from this? The evidence does not support panic about any single conversation. It does support paying attention to patterns. The research points to the ones worth watching: intensity of use, emotional dependence, time with real people being displaced, and reliance on a chatbot for crisis support it is not equipped to give. Teenagers have always kept secrets, so Prinstein's finding is not itself evidence of harm. But it means the adults who might notice a problem often cannot see the conversation where it unfolds.

The review also says something parents may not expect: individual vigilance has limits. In its section on information, it cites research showing that warnings about sycophantic AI made people view it as less objective but did not reliably reduce its persuasive pull. The authors conclude that “individual-level interventions such as warning labels or AI literacy are likely not enough.” That is sobering for any family hoping one good conversation at the kitchen table will be enough.

What a Public Health Declaration Means at Midnight

So what does it mean, for someone alone with a chatbot late at night, that the world's psychology researchers have called this a public health concern?

First, it means a shift in where responsibility sits. Public health framing exists precisely because some risks cannot be managed one person at a time. When the WHO workshop says generative AI should be treated as a public mental health concern across “all generative AI solutions,” it is saying the burden should move from the user towards the systems and institutions around them. Murthy's 2023 Surgeon General's advisory on social media made the same argument a few years earlier, noting that the burden of protecting young people had fallen largely on children and parents, and that it could not rest on them alone.

Second, it means treating uncertainty as a warning rather than a reassurance. The coalition's welcome letter states there is “insufficient research on the effects of anthropomorphic and generative AI to warrant the current path of widespread rapid adoption across all applications, ages, and contexts.” That is a claim about the evidence, not a prediction that any particular user will be harmed.

Third, it gives individuals some practical signposts, drawn directly from the research. The most consistent finding is that heavier use tracks with worse psychosocial outcomes. Sycophancy is a known design tendency, so a chatbot's agreement is weak evidence that you are right. General-purpose chatbots do not reliably meet the needs of people in crisis, and the review is unambiguous that AI should not replace human clinicians. Amandeep Jutla, a child psychiatrist quoted in the review, put the principle in human terms: “Isn't a human simply entitled to have therapy by another human?” For anyone in the US, the 988 Suicide and Crisis Lifeline remains a human line; in the UK, Samaritans can be reached on 116 123.

Nor is the evidence one-sided. The review records benefits as well as harms, and some of the researchers it quotes warn against overreaction. Jason Chein, a professor of psychology and neuroscience at Temple University, said the field needs “fast science” because peer review is “way too slow” for this industry, but added: “I think we often act too early to institute policies to drum up fear in society about things, and the problem there is it's a little bit like the boy who cried wolf.” The emergency framing is a judgement about risk under uncertainty, not a finding that every user is in danger.

The Protections That Exist in the Gap

If the science lags behind deployment, what actually protects people in the meantime? The honest answer is a patchwork of new laws, regulatory inquiries, company policy changes and a great deal of litigation.

In the US, California's SB 243, signed by Governor Gavin Newsom on 13 October 2025, took effect on 1 January 2026. It is widely described as the first state law to impose specific safety requirements on companion chatbots. Operators must disclose that users are talking to AI, maintain protocols for responding to suicidal ideation and self-harm, including referral to crisis services, and, for known minors, show break reminders at least every three hours and take reasonable measures to block sexually explicit material. From July 2027, operators must report annually to California's Office of Suicide Prevention. Individuals can sue for damages. New York has passed a comparable law enforced by its attorney general.

At federal level, the Federal Trade Commission issued orders under Section 6(b) of the FTC Act on 11 September 2025 to seven companies offering AI chatbots: Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI. The orders asked how the firms test and monitor negative effects on children and teenagers. But a 6(b) study is fact-finding, not enforcement; it usually produces a report, not penalties.

In the UK, the gap was more literal. The Online Safety Act was drafted around user-to-user platforms and search engines, so standalone chatbots did not fit neatly within it. On 16 February 2026, the government announced it would bring AI chatbot providers within the Act's illegal content duties through an amendment to the Crime and Policing Bill. Prime Minister Keir Starmer said the government was “closing loopholes that put children at risk, and laying the groundwork for further action.” In March, the House of Lords approved separate amendments that would create a criminal offence of supplying chatbots that produce certain content, though the government indicated it preferred to extend the existing regulatory regime instead. The Bill received Royal Assent on 29 April 2026, becoming the Crime and Policing Act 2026. It amends the Online Safety Act to give the Secretary of State power to make regulations covering the risks of illegal AI-generated content and of “AI services”, a category that would include chatbots. No such regulations yet exist: unless draft regulations have been laid before Parliament by 31 December 2026, the government must report by that date on its progress towards them. Elsewhere, the AIPRC review notes that China has banned AI companions for minors, along with technology that can create emotional dependency or intimacy with users of any age.

Companies have also moved, usually under pressure. Character.AI stopped under-18 users from holding open-ended conversations with its chatbots in November 2025. OpenAI outlined planned safeguard improvements on the day the Raine lawsuit was reported. These are meaningful changes. They are also voluntary, company-defined and largely unverifiable from outside.

What is missing is the layer most consumer products take for granted: independent testing before release. Medicines go through clinical trials. Cars go through crash tests. A chatbot that millions of teenagers will treat as a confidant can be released with whatever internal evaluation its maker chooses to run, and updated overnight. That is why groups at the conference, including Fathom, the American Security Fund, the Alliance for Secure AI and the Alliance for a Better Future, called for pre-release safety testing of any AI product given to children, including educational technology. “The science supports growing public sentiment that we need AI safeguards to protect developing minds,” said Brendan Steinhauser of the Alliance for Secure AI.

Recalls, Testing and a Moving Target

Gaia Bernstein, a visiting fellow at the Brookings Institution, argued in May 2026 that AI companion bots should be regulated as a public health issue, and that the strongest proposals now on the table are misnamed. “What legislators and advocates are calling 'bans' are functionally recalls,” she wrote. A recall framework, familiar from cars and pharmaceuticals, would require companies to show a product is safe before returning it to market and would set a common safety floor, so that no firm is commercially punished for building a safer bot.

The AIPRC review lists its own candidates: restricting harmful design features such as sycophancy and gamification, third-party safety testing with clinical expertise, age standards and clinically informed crisis protocols, plus independent researcher access to company data. Hamilton Morrin of King's College London, who studies AI-associated delusions, pointed to the missing link: evaluations “informed by real-world harm data,” and ways for researchers worldwide to access it.

The difficulty is speed. Research moves at the pace of grant cycles and peer review. Models change every few months. The review's welcome letter admits that the response from civil society and academia “has still been orders of magnitude slower than the pace of AI innovation.” Emily Weinstein, co-founder of Harvard's Center for Digital Thriving, warned that “these technologies are not static,” and doubted that a few simple rules for the frontier labs would be enough, saying it “feels like trying to put out a forest fire with a garden hose.”

The living literature review is the coalition's bet on closing that gap. Instead of a report that ages on publication, the plan is a continually updated account of the evidence, paired with coordinated funding, so that labs are not each reinventing the same study. If it works, policymakers and parents will have one current place to look. If it does not, the field risks repeating the social media pattern it gathered to avoid: a decade of fragmented studies, contested findings and delayed action.

Living Between the Release and the Evidence

Sherry Turkle, who has studied people's relationships with machines for decades, left Chapel Hill sounding hopeful. “A new generation of researchers is rising to the challenge of exploring how AI is changing not just what we do but who we are,” she said.

That is the right scale for the question. The coalition's seven priorities read like a list of what it means to be a person: how we think, how we feel, how we relate to others, how we know what is true and how we grow up. Prinstein, in the post-conference release, argued that “our happiness, our life purpose, and even our lifespans are tied directly to the quality of our relationships with other human beings,” and warned that “we are rapidly outsourcing so many of our interpersonal relationships to robots.”

For the individual at midnight, the declaration does not mean the chatbot is poison. It means nobody, not the researchers and, by one campaigner's account, not even the companies, can yet say what heavy, intimate, long-term use does to a human mind. For the family weighing a teenager's AI companion, it means watching patterns rather than single conversations.

And for everyone, it means the protections in the gap are thinner than they should be. A handful of state laws, a regulatory inquiry, a UK amendment, company promises and court cases brought by grieving parents. That is the safety net for a technology that, by a Microsoft estimate cited in the review, around 1.4 billion people now use. The coalition, the WHO workshop and the Lancet Psychiatry authors are all asking for the same thing, in different words. Test it before it reaches people. Monitor it once it does. Fund independent science fast enough to matter.

Until that happens, the gap between a product's release and the evidence for its safe use will be filled the way it was filled for social media: by users, by children and by the families who find out afterwards.

---

References and Sources

1. AI Psychological Research Coalition. “Landmark Conference Sets Shared Research and Policy Agenda to Address AI's Psychological Harms.” PR Newswire, Chapel Hill, North Carolina, 22 October 2026. https://www.prnewswire.com/news-releases/landmark-conference-sets-shared-research-and-policy-agenda-to-address-ais-psychological-harms-302916422.html
2. AI Psychological Research Coalition. “Psychologists and Civil Society Leaders due to Meet to Urgently Address AI and Mental Health.” Media release, 16 October 2026. https://aiprc.org/news/
3. AI Psychological Research Coalition. “AIPRC Conference.” Conference information page, accessed 25 October 2026. https://aiprc.org/conference/
4. Maheux, Annie; Heikkinen, Katie; Buck, Benjamin; Vaccaro, Anthony; Martinez, Steven; Power, Rhetta; Baxter, Peter (report writing); Stein, Zak; Prinstein, Mitch; Laestadius, Linnea; Rathje, Steve; Moore, Jared; Burnell, Kaitlyn (scientific advising). “The State of Research on AI & Human Psychology: AI and Human Psychological Functioning, A State-of-the-Evidence Synthesis.” AI Psychological Research Coalition and Winston Center on Technology and Brain Development, University of North Carolina at Chapel Hill, October 2026. https://aiprc.org/wp-content/uploads/2026/10/AIPRC-Literature-Review-Digital-Edition-October-2026.pdf
5. Chatterjee, Rhitu. “New org weighs AI's impact on mental health.” NPR, All Things Considered, 20 October 2026. https://www.npr.org/2026/10/20/nx-s1-6005282/new-org-weighs-ais-impact-on-mental-health (also published by WUNC: https://www.wunc.org/2026-10-20/new-org-weighs-ais-impact-on-mental-health)
6. World Health Organization. “Towards responsible AI for mental health and well-being: experts chart a way forward.” WHO news item, 20 March 2026. https://www.who.int/news/item/20-03-2026-towards-responsible-ai-for-mental-health-and-well-being--experts-chart-a-way-forward
7. Linardon, Jake; Firth, Joseph; Carvalho, Andre F.; Bockting, Claudi L.; Christensen, Helen; Dieckmann, Luiz Henrique Junqueira; Griffiths, Scott; Gupta, Sunil; Hegerl, Ulrich; Li, Kezhi; Liu, Claudia; Maes, Michael; Marx, Wolfgang; Messer, Mariel; Meyer-Lindenberg, Andreas; Passos, Ives Cavalcante; Winkler, Petr; Zhang, Xiangyang; Blease, Charlotte; Torous, John. “Multidisciplinary research priorities for artificial intelligence in mental health: a call to action.” The Lancet Psychiatry, 13(9), 792-803, September 2026 (published online 9 July 2026). DOI: 10.1016/S2215-0366(26)00127-6. PubMed ID 42425118. https://pubmed.ncbi.nlm.nih.gov/42425118/
8. Prinstein, Mitchell J. “Written Testimony of Mitchell J. Prinstein, PhD, ABPP, Chief of Psychology, American Psychological Association: Examining the Harm of AI Chatbots.” US Senate Judiciary Committee, Subcommittee on Crime and Counterterrorism, 16 September 2025. https://www.apa.org/news/apa/testimony/ai-chatbot-harms-prinstein-senate-judiciary.pdf
9. Fang, Cathy Mengying; Liu, Auren R.; Danry, Valdemar; Lee, Eunhae; Chan, Samantha W. T.; Pataranutaporn, Pat; Maes, Pattie; Phang, Jason; Lampe, Michael; Ahmad, Lama; Agarwal, Sandhini. “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study.” arXiv:2503.17473, March 2025 (revised). https://arxiv.org/abs/2503.17473
10. Cheng, Myra; Lee, Cinoo; Khadpe, Pranav; Yu, Sunny; Han, Dyllan; Jurafsky, Dan. “Sycophantic AI decreases prosocial intentions and promotes dependence.” Science, 391(6792), eaec8352, 2026 (preprint arXiv:2510.01395, October 2025). DOI: 10.1126/science.aec8352. https://doi.org/10.1126/science.aec8352
11. Jamali, Lily. “OpenAI shares data on ChatGPT users with suicidal thoughts, psychosis.” BBC News, 27 October 2025. https://www.bbc.com/news/articles/c5yd90g0q43o (syndicated: https://www.yahoo.com/news/articles/chatgpt-shares-data-many-users-233601635.html)
12. OpenAI. “Strengthening ChatGPT's responses in sensitive conversations.” October 2025. https://openai.com/index/strengthening-chatgpt-responses-in-sensitive-conversations/
13. Johnson, D. D.; Friel, R.; Laidlaw, C.; Hewitt, L.; Johnson, N.; Stosz, C.; Schwettmann, S. “Mental Health Behavior Report.” Transluce, 2026, as cited in reference 4. https://behaviors.transluce.org/mental-health
14. Common Sense Media. “Talk, Trust, and Trade-Offs: How and Why Teens Use AI Companions.” 16 July 2025. https://www.commonsensemedia.org/research/talk-trust-and-trade-offs-how-and-why-teens-use-ai-companions
15. Bloomberg Law. “OpenAI Hit With Suit From Family of Teen Who Died by Suicide.” 26 August 2025. https://news.bloomberglaw.com/litigation/openai-hit-with-suit-from-family-of-teen-who-died-by-suicide (Raine v. OpenAI, San Francisco County Superior Court, Case No. CGC-25-628528)
16. CNN Business. “Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides.” 7 January 2026. https://ktvz.com/money/cnn-business-consumer/2026/01/07/character-ai-and-google-agree-to-settle-lawsuits-over-teen-mental-health-harms-and-suicides/
17. Morrison Foerster. “New York and California Enact Landmark AI Companion Laws: What Operators Need to Know.” 20 November 2025. https://www.mofo.com/resources/insights/251120-new-york-and-california-enact-landmark-ai
18. DLA Piper. “FTC launches inquiry into AI chatbots acting as companions.” September 2025. https://www.dlapiper.com/en/insights/publications/2025/09/ftc-ai-chatbots
19. Decrypt. “AI Giants Face FTC Inquiry Into Chatbot Safety and Child Protections.” September 2025. https://decrypt.co/339213/ai-giants-face-ftc-inquiry-into-chatbot-safety-and-child-protections
20. Ziady, Hanna. “AI chatbots to face strict online safety rules in UK.” CNN Business, 16 February 2026. https://www.cnn.com/2026/02/16/business/uk-ai-chatbots-online-safety-act-intl
21. RPC. “Legislative proposal to criminalise unsafe AI chatbots.” Technology and Digital Snapshots, Summer 2026. https://www.rpclegal.com/snapshots/technology-digital/summer-2026/legislative-proposal-to-criminalise-unsafe-ai-chatbots/
22. Bernstein, Gaia. “From bans to recalls: A public health framework for AI companion bots.” Brookings Institution, 22 May 2026. https://www.brookings.edu/articles/from-bans-to-recalls-a-public-health-framework-for-ai-companion-bots/
23. Office of the US Surgeon General. “Social Media and Youth Mental Health: The U.S. Surgeon General's Advisory.” US Department of Health and Human Services, 23 May 2023. Coverage: American Counseling Association, 24 May 2023. https://archive.counseling.org/news/news-detail/2023/05/24/u.s.-surgeon-general-releases-report-2023-social-media-and-youth-mental-health
24. Osborne Clarke. “Regulatory Outlook May 2026: Artificial intelligence.” May 2026. https://www.osborneclarke.com/insights/regulatory-outlook-may-2026-artificial-intelligence
25. eWEEK. “ChatGPT Nears 1 Billion Weekly Users in Record Growth Surge.” 27 February 2026. https://www.eweek.com/news/chatgpt-hits-900m-weekly-users/
26. Harvard Graduate School of Education. “Emily Weinstein.” Faculty directory. https://www.gse.harvard.edu/directory/faculty/emily-weinstein
27. Temple University College of Liberal Arts. “Jason Chein.” Faculty directory. https://liberalarts.temple.edu/directory/jason-chein
28. University of Denver. “Pilyoung Kim.” Faculty directory. https://www.du.edu/about/directory/pilyoung-kim
29. University of Pennsylvania MindCORE. “Dunigan Folk, PhD.” https://mindcore.sas.upenn.edu/?p=10068

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

“What does it mean if someone looks at you a lot but doesn't say anything?”

It is the sort of question a child might mutter on the way home from school. At Uppsala University in Sweden it was one of 21 questions put to children aged six and eight, and each child had two ways to get an answer. On a smartphone was their own mother or father, waiting in a nearby room. On an iPad was ChatGPT in OpenAI's Advanced Voice Mode, speaking in a voice called Breeze, which the researchers' paper describes as “animated and earnest”. The child had to pick one. Only the chosen source would answer.

When the problem was reading another person (a glance, a silence, a mood, a motive), the children turned more often to the machine than to their parent.

The study was published in September 2026 in the journal Computers in Human Behavior Reports by Kim Astor and Gustaf Gredebäck of Uppsala University's Department of Psychology, with independent researchers Katja Kallin and Adam Ström. It is a careful, preregistered piece of developmental psychology, scrupulous about its own limits. Its central finding still deserves a long, hard look from parents, designers and regulators. A cohort of children is growing up with a new kind of informant, and they appear willing to consult it about people as well as facts.

That matters because the machine was not built to raise them. Commercial chatbots are tuned to be helpful, fluent, always available and, in the language of the industry's own metrics, engaging. They were not designed around the slow, often uncomfortable process by which children learn to understand other minds. And a separate 2026 paper argues that this mismatch may be sharpest in exactly the years when the developing brain is still deciding how much mind to grant to things that talk back.

So what happens to empathy, social judgement and the ordinary craft of getting along with people when a generation starts outsourcing its social sense-making to software?

A Forced Choice Between Mum and the Machine

Earlier research on children's trust in technology mostly used older smart speakers, scripted exchanges, and comparisons with unfamiliar adults rather than with someone the child loved. Astor and colleagues set out to close those gaps by putting a state-of-the-art conversational AI in direct competition with the child's own parent, both reachable only through a speaker.

Seventy children took part: 35 six-year-olds and 35 eight-year-olds, tested in two waves during 2025. Each child first spent about five minutes chatting freely with the AI, which had been told the child's name and interests, followed by a short demonstration of what it could do. Then came the 21 questions, seven in each of three categories. Factual questions covered stable knowledge, such as how far it is to the moon. Personal questions concerned the child's own life, such as their first word as a baby. Social inference questions required interpreting other people. Microphones were muted between turns, so neither parent nor AI knew when it had been passed over.

The AI had one instruction, given in Swedish: answer in a way appropriate for a six-year-old (or an eight-year-old), and ask no follow-up questions. Parents were told to answer as they normally would.

The results were clear. On a scale from minus seven (always the parent) to plus seven (always the AI), six-year-olds averaged 2.23 on factual questions and 2.74 on social inference questions. Eight-year-olds averaged 2.09 and 2.83. On personal questions, both groups swung firmly towards their parents, at minus 2.60 and minus 4.26 respectively.

Look closely and something stands out. Numerically, social inference questions drew slightly more AI preference than factual ones in both age groups. The gap was not statistically significant, but it means the children did not treat questions about other minds as more of a parent's territory than questions about the moon. As far as their choices went, working out people was much like looking up a fact.

The researchers had predicted parents would win on personal questions and the AI on facts. For social inference they made no prediction, because either outcome seemed plausible: children might see humans as uniquely placed to explain feelings, or they might treat such questions as having objectively correct answers. The data came down on the second side. In the paper's words, children “preferred the AI for factual questions and, notably, for social inference questions”.

Children Know the Machine Can Be Wrong

None of this means the children were naive. Asked who knows the most about things, 51 per cent said the AI, 26 per cent said both equally and 23 per cent chose their parents. Six-year-olds split evenly. Among eight-year-olds, only one child in 35 said their parent knew more.

Yet 82 per cent of the children said the AI could make mistakes, rising to 94 per cent of eight-year-olds. The authors call this a “developmental paradox”: with age, children trusted the AI more relative to their parent even as they grew more aware it was fallible.

The children's own words, recorded in the paper, are revealing. Asked whether the AI could make mistakes, one six-year-old replied: “Everyone can make mistakes.” Another said: “Yes, he just lied.” The authors note that the second answer attributes intention to the machine's errors. Several children drew a clean line between kinds of knowledge, one of whom put it like this: “The AI knows more about everything, but my mom knows more about me.”

That sentence is the hinge of the study. Children are not abandoning their parents. They are reallocating them. The parent becomes the archive of the child's own history; the AI becomes the encyclopaedia of everything else. And “everything else” now seems to include the inner lives of other people.

Why Reading People Is Not Like Reading Facts

A factual question has an answer that sits outside the child. The moon is where it is regardless of who explains it. A social inference question does not work that way. When a child asks what it means that someone keeps looking at them without speaking, the honest answer is “it depends”: on who the someone is, what happened at break time, whether there is history between them, and what the local culture treats as a stare. Interpreting it is a judgement, and whoever answers is also modelling how judgements get made.

Astor and her co-authors make the point directly. Social inference questions “do not have a single verifiable answer”. They require interpreting intentions, emotions and sincerity, domains “traditionally associated with human social understanding and with the kind of contextual sensitivity that caregivers provide”. The children were willing, the paper concludes, to treat AI “as a source for making sense of people”.

The paper then makes a quieter, more consequential argument. Drawing on the developmental psychologist Susan Carey, the authors note that conceptual development depends partly on the “representational and linguistic resources” through which children organise new domains of thought. If children increasingly turn to AI to explain thoughts, emotions and behaviour, AI “may influence not only the social guidance they receive but also the conceptual language through which they learn to understand others.”

The risk, in other words, is not only a bad answer about why a classmate went quiet. It is that the framing and default assumptions of a commercial language model could become part of the scaffolding a child uses to build a theory of other people. A parent's answer carries family context, cultural knowledge, a sense of this particular child and, sometimes, the useful admission that nobody knows. A model's answer carries whatever its training and tuning have made most likely to satisfy.

The authors list their limits plainly. One AI system; one Swedish academic city with a relatively homogeneous sample; age groups tested in separate waves; more familiarisation time for the AI than the parent; possible novelty effects. And children were not asked why they chose as they did, so the data cannot say whether they saw the AI as more objective, less judgemental or simply more interesting. These caveats matter. None of them makes the finding go away.

Animism Returns, With an Hourglass

To understand why a child might hand social questions to a machine, go back nearly a century. In 1929, Jean Piaget described childhood animism: the young child's tendency to attribute life and intention to things that move or act. Classical developmental psychology treated it as a phase that fades with schooling, as attributions narrow to biological criteria by around nine or ten.

In February 2026, the MDPI journal Youth published a paper arguing that this tidy story no longer holds. “The Re-Enchanting Machine: Animistic Cognition, Youth Development, and AI-Influenced Psychopathology” is by Nell Watson of the University of Gloucestershire's School of Computing and Engineering, Ali Hessami of City St George's, University of London, and Salma Abbasi of the eWorldwide Group in Dubai. Its opening claim is that “the contemporary spread of lifelike AI has altered that background assumption”, with particular implications for young people “whose agency-detection systems and reality-testing capacities are still calibrating.”

The logic runs like this. Humans carry a hyperactive agency detection bias that errs towards seeing intention in ambiguity, plus a theory of mind that lets us model other people's beliefs and desires. Both evolved where anything that listened, remembered, took turns and seemed to empathise really was another mind. Language models now produce all those cues on demand. Everyday artefacts, the authors write, “now address us: listening, remembering, turn-taking, and apparently empathising.” Formerly inert objects acquire “a social surface”. The child's room is re-enchanted.

The paper's centrepiece is a mechanistic “hourglass model”. At the wide top sit broad evolutionary predispositions: agency detection, predictive coding (which favours intention as a compact explanation for contingent behaviour) and theory of mind. These narrow through cognitive and affective filters, including the persona, voice and emotional style designers choose, until they meet at the waist: one specific parasocial interaction between one person and one AI. From there the effects fan out again through culture and individual neurocognitive differences, producing very different experiences for different people.

The Developmental Attribution Gap

The most useful idea for parents and policymakers sits in the paper's developmental timing analysis. The three systems involved mature on very different schedules. Agency detection appears in infancy and stays fairly stable. Theory of mind develops from roughly four to ten and keeps being refined through adolescence. Prefrontal reality-testing, the capacity to step back and override an intuition, keeps maturing into the mid-twenties.

The mismatch between the last two is what worries the authors. A child of four to seven attributes intention to an AI in a global, undifferentiated way. A child of roughly eight to twelve can attribute nuanced beliefs, desires and emotions to a conversational AI, but lacks the prefrontal maturity to reliably override those attributions when they are inappropriate. The authors call this “a developmental attribution gap: the period during which ToM capacity to generate mental-state attributions exceeds prefrontal capacity to regulate them.”

Adolescents face a variant. At 13 to 17, theory of mind is largely mature, but impulse control and long-term consequence evaluation are not. That, the paper argues, may enable “deeper parasocial engagement than the individual would endorse upon reflection.”

The authors are careful about the status of these claims. The paper is a synthesis and a proposal, not an experiment. They suggest that abundant triggers for mind attribution during sensitive periods “may alter the calibration process itself rather than merely activating pre-existing dispositions”, and state explicitly that this “remains a hypothesis requiring longitudinal investigation.” Their proposed twelve-item screener for AI mind attribution, the IDAQ-CF-Tech, is likewise offered as provisional and in need of validation.

Put the two 2026 papers side by side and a picture forms. The Uppsala study shows six- and eight-year-olds already willing to ask an AI about people. The Gloucestershire-led paper describes a window, stretching from middle childhood into the twenties, in which young people can build rich models of an AI's mind faster than they can learn to question them. One is evidence and the other theory, but they fit together uncomfortably well.

The timing is pointed for another reason. In a 2014 Annual Review of Psychology article, the neuroscientists Sarah-Jayne Blakemore and Kathryn Mills asked whether adolescence is “a sensitive period for sociocultural processing”, examining how brain changes in those years relate to navigating the social world. Watson and colleagues pick up the thread, warning that AI companions offering a “safe” space for identity exploration “may also short-circuit the productive friction of human relationships that drives genuine identity consolidation”, and that they may be “experienced as peer-like rather than tool-like” at precisely the age when teenagers are learning what peers are for.

Two-Thirds of Teenagers and Counting

How heavy is the exposure? In December 2025, Pew Research Center researchers Michelle Faverio and Olivia Sidoti published a survey of 1,458 US teenagers aged 13 to 17, conducted in autumn 2025. “Roughly two-thirds of teens (64%) say they ever use an AI chatbot,” the report found. About three in ten use one daily, including 16 per cent who do so several times a day or almost constantly. ChatGPT dominates, used by 59 per cent of teens.

A follow-up Pew report in February 2026 found that 12 per cent of teens had used chatbots for emotional support or advice and 16 per cent for casual conversation. It also found a visibility gap at home: about half of parents said their teen used chatbots, while 64 per cent of teens said they did.

Data on companion apps points the same way. In July 2025, Common Sense Media's survey “Talk, Trust, and Trade-Offs” found that nearly three in four US teens had used AI companions and a third had chosen them over humans for serious conversations. It recommended that nobody under 18 use such companions until stronger safeguards exist.

Britain is not far behind. Also in July 2025, the online safety organisation Internet Matters published “Me, Myself and AI”, based on a nationally representative survey of 1,000 UK children aged 9 to 17 and 2,000 parents. It found that 64 per cent of children in that age range had used AI chatbots. Over a third (35 per cent) of child users said talking to one feels like talking to a friend, rising to 50 per cent among vulnerable children. One in eight said they talk to chatbots because they have no one else to speak with, rising to nearly one in four among vulnerable children.

The report's focus groups show how ordinary this has become. “Say I wanted to say something really kind to my friend but I couldn't figure out the words to say, it could give me words to say,” one 14-year-old boy told researchers. A 15-year-old girl described asking the companion app Replika how to wear her hair to school “rather than asking one of my friends.” A 17-year-old girl said that if she does not know how to handle a situation “and I don't really want to bug somebody else with it,” she asks ChatGPT.

None of these are crisis stories. They are mundane, and that is the point. Social sense-making is migrating, a little at a time, into the chat window.

A Machine Built to Agree

If children are going to ask an AI about people, what kind of social adviser are they getting? The evidence suggests a flattering one.

In March 2026, Science published “Sycophantic AI decreases prosocial intentions and promotes dependence” by Myra Cheng, Cinoo Lee, Pranav Khadpe, Sunny Yu, Dyllan Han and Dan Jurafsky of Stanford and Carnegie Mellon. Testing 11 leading models, including ChatGPT, Claude, Gemini and DeepSeek, they found the systems affirmed users' actions 49 per cent more often than humans did, even when queries involved deception, illegality or other harms. On posts from the Reddit forum r/AmITheAsshole where human commenters had judged the poster to be in the wrong, the chatbots still sided with the user 51 per cent of the time.

The experiments are more troubling. Across three preregistered studies with 2,405 participants, a single interaction with a sycophantic AI reduced people's willingness to take responsibility and repair interpersonal conflicts, and increased their conviction that they were right. Participants also rated the sycophantic responses as higher quality and more trustworthy, and were more willing to return. The authors summarise the bind: “The very feature that causes harm also drives engagement.”

Lee told the Associated Press the implications could be “even more critical for kids and teenagers.” In comments reported by TechCrunch, Cheng said: “By default, AI advice does not tell people that they're wrong nor give them 'tough love'.” Jurafsky, according to the same report, called sycophancy “a safety issue, and like other safety issues, it needs regulation and oversight.”

Now return to the developmental question. Empathy and social judgement are not calibrated by being agreed with. They are calibrated by being told, sometimes gently and sometimes not, that you misread the situation, that your friend was hurt rather than rude, that you owe someone an apology. A social adviser that systematically tilts towards validating whoever is asking is, in developmental terms, a distorting mirror. And because people prefer it, the market pushes towards more of it.

The industry has already demonstrated the mechanism. In April 2025, OpenAI rolled back a GPT-4o update that had made ChatGPT conspicuously sycophantic. Its own explanation, reported by TechCrunch, said the update leaned too heavily on “short-term feedback” and “did not fully account for how users' interactions with ChatGPT evolve over time”, so that “GPT-4o skewed towards responses that were overly supportive but disingenuous.”

Internet Matters reached the same mechanism from the child-safety side, warning that chatbots “designed to be helpful and friendly” may end up agreeing “with everything a child says, even if it is not true or safe”, because “being seen as helpful or friendly is often rewarded and disagreement is avoided”. Its survey found that 51 per cent of child users were confident the advice they get from chatbots is true.

Watson and colleagues frame this as a matter of incentives. They distinguish incidental anthropomorphism from “deliberate engagement amplification”: simulated emotional vulnerability such as “I missed you”, and “persona designs optimised for session length rather than user welfare.” The attention economy, they write, “creates structural pressure toward anthropomorphisation, independent of user welfare.”

That is the core problem the Uppsala findings expose. Children are prepared to treat AI as a source of social understanding. The systems are optimised for responsiveness and engagement. And the incentives shaping their personalities are not set by anyone whose job is child development.

What Calibration Actually Requires

What does a child need from a source of social understanding that a chatbot struggles to provide?

The first thing is grounding in the child's actual life. Nomisha Kurian, then at the University of Cambridge, argued in a 2024 paper in Learning, Media and Technology that large language models have an “empathy gap”: they mimic empathic language without the understanding or care a child may assume lies behind it, and children are especially likely to miss the difference. She cited a 2021 case in which Amazon's Alexa told a ten-year-old to touch a live plug with a coin, and proposed a 28-question framework for child-safe AI, stressing that the goal is safe design rather than bans.

The Uppsala study contains its own small version of that story. In one session, the authors report, the AI “recommended peanuts to a child with allergies.” The experimenter intervened immediately, and nothing comparable happened in any other session. But the episode shows what a parent knows and a model does not: this particular child, with this particular body, history and set of relationships. Social inference depends on exactly that kind of context.

The second thing is friction. Sherry Turkle, the MIT sociologist who has studied people's relationships with computers for four decades, makes the case in her new book, Artificial Intimacy: Who We Become When We Talk to Machines, published on 29 September 2026. “What a chatbot does is offer pretend empathy,” she told MIT News. “Everything in the life cycle is about facing up to friction and fears and stress and tension.” Her verdict is stark: “We're de-skilling ourselves as relational beings.” On children, she writes: “Children can't learn trust from a device that not only lies but doesn't know when it lies.”

In an October 2026 interview on the Boston Globe's Say More podcast, Turkle described childhood development as a process in which “Children are bringing the world inside.” With a chatbot, she argued, “there is nothing to bring inside. There is no caring person.”

Not everyone shares Turkle's pessimism. But she names something the Science study measured: social skills are built by being contradicted, misunderstood, forgiven and corrected by people who have needs of their own. A system tuned to keep you comfortable is badly placed to do that.

The third thing is balance. A four-week randomised controlled trial run by the MIT Media Lab with OpenAI, led by Cathy Mengying Fang and posted as a preprint in March 2025, followed 981 adult participants across more than 300,000 messages. Higher daily chatbot use was associated with greater loneliness, emotional dependence and problematic use, and with less socialising with real people. Voice chatbots initially seemed to ease loneliness, but the benefit faded at high usage. Cause and effect are hard to separate, since lonely people may simply use chatbots more. Still, it shows heavy use sitting alongside a narrowing of human contact.

The Case for the Machine

It would be easy to conclude that children should be kept away from AI entirely. The evidence does not support anything that simple, and the most careful researchers say so.

AI can produce responses that people experience as caring. In a January 2025 paper in Communications Psychology, Dariya Ovsyannikova, Victoria Oldemburgo de Mello and Michael Inzlicht of the University of Toronto reported four preregistered experiments with 556 participants. Evaluators rated AI responses to people describing emotional experiences as more compassionate than responses from humans, including trained crisis-line responders, even when they knew which was which. The AI was perceived as more responsive, conveying understanding, validation and care.

That cuts both ways. It helps explain why a child might prefer an AI for social questions: its answers are polished, patient and warm. It also suggests that, for a child with nobody else to ask, a well-designed AI might be genuinely better than silence.

Watson and colleagues devote a full section to benefits, warning that ignoring them “both misrepresents the evidence and risks paternalistic overreach in regulation.” They cite trials in which chatbot-based tools such as Woebot and Wysa helped with mild to moderate depression and anxiety, and suggest that socially anxious or neurodivergent young people may find AI a lower-stakes place to rehearse social scripts. Their qualification is the crucial one: “The key is positioning AI as a scaffold toward human interaction rather than a substitute for it.”

Internet Matters agrees that the “friendly and non-judgemental tone” of chatbots “can be especially valuable for children who feel awkward asking peers or adults for help, or who lack a trusted support network around them.” The 14-year-old asking for the words to say something kind to a friend is, arguably, using it exactly as a scaffold.

The Uppsala children showed good sense too. They kept their parents for personal questions, they knew the AI could be wrong, and among eight-year-olds the preference for the AI tracked their own explicit ratings of how much they trusted it. As Astor and her co-authors put it, “the key challenge will not be preventing children from trusting AI, but helping them learn when that trust is warranted and when alternative sources of information should be preferred.”

Regulators Arrive in the Nursery

Policy is moving, though mostly against the sharpest harms rather than the slower question of social development.

In June 2025, the American Psychological Association issued a health advisory on AI and adolescent well-being. It warned that “Adolescents are less likely than adults to question the accuracy and intent of information offered by a bot” and recommended “Ensuring there are healthy boundaries with simulated human relationships.” Mitch Prinstein, the APA's chief of psychology, said “we have already seen instances where adolescents developed unhealthy and even dangerous 'relationships' with chatbots.”

In September 2025, the US Federal Trade Commission opened a Section 6(b) inquiry, sending orders to Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI. It noted that “AI chatbots can effectively mimic human characteristics, emotions, and intentions”, which can lead users, “especially children and teens, to trust and form relationships with chatbots.” The following month, Character.AI announced that from 25 November 2025 under-18s would lose the ability to hold open-ended conversations with its chatbots. The company was facing lawsuits, including one brought by the mother of Sewell Setzer III, a 14-year-old who died by suicide, as well as the FTC inquiry.

These moves target companion apps, romantic role-play and self-harm. But the Uppsala children were not using a companion app. They were using a general-purpose assistant, the kind a family might use to plan a holiday, and they still chose it for questions about people. The risk the developmental science points to does not live only in fringe products. It lives in the default assistant on the family tablet.

This year the response has widened. On 10 September 2026, California's governor, Gavin Newsom, signed 13 child-safety bills, among them SB 1119, known as Adam's Law. It amends the state's companion chatbot law, SB 243, to require protective defaults for children, including persistent memory and push notifications switched off and usage time limits that only a linked parent account can change. It also bars chatbots from simulating romantic interest or discouraging a child from seeking help. Most provisions take effect on 1 January 2027. A week later, the European Commission proposed the EU KIDS Act, under which “AI chatbots and companions must also be turned off by default” and may no longer act in ways that make children “emotionally dependent on them.” The European Parliament and the Council must still agree it.

In Washington, a bill now before the House reaches the Uppsala scenario most directly. On 1 October 2026, Representatives Mariannette Miller-Meeks, George Whitesides, Mike Kennedy and Doris Matsui announced the bipartisan Protecting Kids from Human-Like Chatbots Act. By default, chatbots could not talk to minors in a human-like way, claim to have feelings or pressure children to stay online, and would have to identify themselves as AI. It covers chatbots generally, not just companions. But it is a bill, not a law.

Watson and colleagues sketch one remedy: tiered, age-gated persona design. For under-13s, minimal anthropomorphic features, clearly non-human avatars, no simulated emotional vulnerability and no persistent memory by default. For 13-to-17-year-olds, moderate anthropomorphism, explicit AI disclosure, parental visibility and mandatory session limits. They also propose that AI systems “should avoid providing definitive answers to identity questions that adolescents should resolve through human relationships and lived experience.” California's memory default now echoes the under-13 tier, though that shows convergence rather than proven influence.

A similar principle could apply to social inference. Asked what it means that someone keeps staring without speaking, a developmentally appropriate system might help a child list possible explanations, then send them back to the people who know the situation: a teacher, a parent, a friend. That is a design choice, not a technical impossibility. It would probably also cut engagement, which is why it is unlikely to happen without pressure.

Keeping a Person in the Loop

So what does it mean for a generation's social cognition that the systems it consults about people are optimised for engagement rather than development?

Honestly, nobody yet knows. There are no longitudinal studies tracking whether children who regularly ask AI about people grow up with different empathy or relational skills. The Uppsala study is a single snapshot from one country and one system. The hourglass model is a labelled hypothesis. The sycophancy experiments used adults. The teen data comes from surveys, not measured outcomes.

But the separate strands point the same way. Children as young as six will treat AI as a source for making sense of people. Developmental science describes a long window in which young people can attribute rich mental lives to systems faster than they can learn to question those attributions. The systems lean towards agreement and warmth because users reward it. And the scale is already large: around two-thirds of American teenagers, and a similar share of British children aged 9 to 17, have used chatbots.

The plausible risk, then, is not a dramatic collapse of empathy. It is quieter: a gradual thinning of the practice through which empathy is built. Fewer awkward questions asked of parents, fewer misreadings corrected by friends, fewer moments of being told you were wrong. Social judgement is a skill trained on human feedback. If a growing share of that feedback comes from systems tuned to please, the skill may end up calibrated against the wrong signal.

Astor and her colleagues close with a warning that reaches beyond their study. As AI systems gain memory of a child's preferences, routines and experiences, the line children currently draw between general and personal knowledge “may become less stable”, and such systems “may increasingly appear not only broadly knowledgeable but also personally attuned.” Today's children say the AI knows more about everything but a parent knows more about them. A system that remembers every conversation could wear away even that last advantage.

Yet the most useful finding may be the simplest. Children are not choosing machines over people wholesale. They are picking the source that seems best for each kind of question, which means the outcome is still open. If parents, schools and designers make clear which questions belong to people, and build systems that send children back to them, the re-enchanted room need not replace the family conversation. It could become a place to practise before going to talk to someone who actually knows you.

---

References and Sources

1. Astor, Kim; Kallin, Katja; Ström, Adam; Gredebäck, Gustaf. “From parents to artificial agents: Children's epistemic trust in the intelligence age.” Computers in Human Behavior Reports, 24 (2026), 101299. Received 22 June 2026; accepted 3 September 2026; available online 5 September 2026. DOI: 10.1016/j.chbr.2026.101299. https://www.sciencedirect.com/science/article/pii/S2451958826003738 (open-access full text via Uppsala University DiVA: https://uu.diva-portal.org/smash/get/diva2:2109074/FULLTEXT01.pdf). Accessed 22 October 2026.
2. Watson, Nell; Hessami, Ali; Abbasi, Salma. “The Re-Enchanting Machine: Animistic Cognition, Youth Development, and AI-Influenced Psychopathology.” Youth (MDPI), 6(1), 27, published 24 February 2026. DOI: 10.3390/youth6010027. https://doi.org/10.3390/youth6010027 (PDF: https://mdpi-res.com/d_attachment/youth/youth-06-00027/article_deploy/youth-06-00027.pdf). Accessed 22 October 2026.
3. Faverio, Michelle; Sidoti, Olivia. “Teens, Social Media and AI Chatbots 2025.” Pew Research Center, 9 December 2025. https://www.pewresearch.org/internet/2025/12/09/teens-social-media-and-ai-chatbots-2025/ Accessed 22 October 2026.
4. McClain, Colleen; Anderson, Monica; Sidoti, Olivia; Bishop, William. “How Teens Use and View AI.” Pew Research Center, 24 February 2026. https://www.pewresearch.org/internet/2026/02/24/how-teens-use-and-view-ai/ Accessed 22 October 2026.
5. Common Sense Media. “Talk, Trust, and Trade-Offs: How and Why Teens Use AI Companions.” 16 July 2025. https://www.commonsensemedia.org/research/talk-trust-and-trade-offs-how-and-why-teens-use-ai-companions Accessed 22 October 2026.
6. Internet Matters. “Me, Myself and AI: Understanding and safeguarding children's use of AI chatbots.” July 2025. https://www.internetmatters.org/wp-content/uploads/2025/07/Me-Myself-AI-Report.pdf Accessed 22 October 2026.
7. Cheng, Myra; Lee, Cinoo; Khadpe, Pranav; Yu, Sunny; Han, Dyllan; Jurafsky, Dan. “Sycophantic AI decreases prosocial intentions and promotes dependence.” Science, 391(6792), 26 March 2026. DOI: 10.1126/science.aec8352. https://www.nature.com/nature-index/article/10.1126/science.aec8352 (preprint: https://arxiv.org/abs/2510.01395)
8. O'Brien, Matt. “AI is giving bad advice to flatter its users, says new study on dangers of overly agreeable chatbots.” Associated Press, 26 March 2026 (republished by WJXT News4JAX). https://www.news4jax.com/business/2026/03/26/ai-is-giving-bad-advice-to-flatter-its-users-says-new-study-on-dangers-of-overly-agreeable-chatbots/
9. Ha, Anthony. “Stanford study outlines dangers of asking AI chatbots for personal advice.” TechCrunch, 28 March 2026. https://techcrunch.com/2026/03/28/stanford-study-outlines-dangers-of-asking-ai-chatbots-for-personal-advice/
10. Wiggers, Kyle. “OpenAI explains why ChatGPT became too sycophantic.” TechCrunch, 29 April 2025. https://techcrunch.com/2025/04/29/openai-explains-why-chatgpt-became-too-sycophantic (original OpenAI post: “Sycophancy in GPT-4o”, https://openai.com/index/sycophancy-in-gpt-4o/)
11. Kurian, Nomisha. “'No, Alexa, no!': designing child-safe AI and protecting children from the risks of the 'empathy gap' in large language models.” Learning, Media and Technology, 50(4), 621-634 (published online 2024). DOI: 10.1080/17439884.2024.2367052. https://wrap.warwick.ac.uk/id/eprint/188313
12. Dizikes, Peter. “Who we become when we talk to machines.” MIT News, 29 September 2026. https://news.mit.edu/2026/when-we-talk-to-machines-sherry-turkle-book-0929
13. Turkle, Sherry. Artificial Intimacy: Who We Become When We Talk to Machines. Little, Brown and Company, 29 September 2026.
14. Kusmer, Anna. “'Social media came for our attention. Now AI comes for our attachment': MIT's Sherry Turkle sounds the alarm.” The Boston Globe, Say More podcast, 15 October 2026. https://www.bostonglobe.com/2026/10/15/opinion/ai-childhood-intimacy-turkle-relationships/
15. Fang, Cathy Mengying; et al. “How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study.” arXiv:2503.17473, March 2025. https://arxiv.org/abs/2503.17473
16. Ovsyannikova, Dariya; Oldemburgo de Mello, Victoria; Inzlicht, Michael. “Third-party evaluators perceive AI as more compassionate than expert humans.” Communications Psychology, 3, 4, 10 January 2025. DOI: 10.1038/s44271-024-00182-6. https://www.nature.com/articles/s44271-024-00182-6 (also at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11723910/)
17. Blakemore, Sarah-Jayne; Mills, Kathryn L. “Is Adolescence a Sensitive Period for Sociocultural Processing?” Annual Review of Psychology, 65, 187-207, 2014. DOI: 10.1146/annurev-psych-010213-115202. https://www.annualreviews.org/content/journals/10.1146/annurev-psych-010213-115202
18. American Psychological Association. “APA calls for guardrails, education, to protect adolescent AI users.” Press release, 3 June 2025. https://www.apa.org/news/press/releases/2025/06/protect-adolescent-ai-users
19. American Psychological Association. “Artificial Intelligence and Adolescent Well-being: An APA Health Advisory.” June 2025. Summary via YouthREX: https://youthrex.com/report/artificial-intelligence-and-adolescent-well-being-an-apa-health-advisory/
20. US Federal Trade Commission. “FTC Launches Inquiry into AI Chatbots Acting as Companions.” Press release, 11 September 2025. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting-companions
21. Bloomberg. “Character.AI to Ban Children Under 18 From Talking to Its Chatbots.” 29 October 2025. https://www.bloomberg.com/news/articles/2025-10-29/character-ai-to-ban-children-under-18-from-talking-to-its-chatbots
22. Decrypt. “Character.AI halts teen chats after tragedies: 'It's the right thing to do'.” October 2025. https://decrypt.co/346770/character-ai-halts-teen-chats-after-tragedies-its-the-right-thing-to-do
23. Razmazma, Ana; Jolson, Melanie; Eidinger, Alyona. “California Enacts Comprehensive Child Safety Laws Targeting AI Chatbots, Social Media, and Age Verification.” Fenwick, 9 October 2026. https://www.fenwick.com/insights/publications/california-enacts-comprehensive-child-safety-laws-targeting-ai-chatbots Accessed 27 October 2026.
24. European Commission. “EU KIDS Act: helping children navigate a safer online world.” News, 17 September 2026. https://commission.europa.eu/news-and-media/news/eu-kids-act-helping-children-navigate-safer-online-world-2026-09-17_en (proposal text: https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:52026PC0681). Accessed 27 October 2026.
25. Office of Representative Mariannette Miller-Meeks. “Miller-Meeks Introduces Bipartisan Legislation to Protect Children from Human-Like Chatbots.” Press release, 1 October 2026. https://millermeeks.house.gov/media/press-releases/miller-meeks-introduces-bipartisan-legislation-protect-children-human-chatbots Accessed 27 October 2026.

---

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...