SmarterArticles

Keeping the Human in the Loop

The most consequential sentence written about American education in August 2026 ran to eleven words and appeared in a Google product bulletin, under a subheading called “Rollout pace”. It reads: “On by default for teachers and students of all ages.”

That is the whole story, compressed. Not a policy paper, not a hearing, not a school board vote. A default.

On 10 August 2026, Google began switching Gemini on inside Google Classroom for students of every age on the web, with mobile following on 17 August. Classroom is used by more than 150 million students and teachers worldwide. Until that week, Google automatically enabled Gemini in Classroom only for students aged eighteen and over. On 14 August the New York Times reported the change, and the reporting travelled fast, because the timing was exquisite. American children were walking back into buildings they had left in June, opening the same application they had used all of the previous year, and finding a new tab.

Derek Moore, chief technology officer at Palo Alto Unified School District, described the experience of Google's cadence of AI changes in one line: “It feels like we're the last to know.” James O'Hagan, director of digital equity, innovation and library services at the Glendale-River Hills School District in Wisconsin, put the principle more bluntly: “The default should be requiring permission, not automatic adoption.”

Eighty-five days later, Google appeared to agree. On Tuesday 3 November, Chris Phillips, Google's vice president and general manager for education, announced that Gemini, Gemini Notebook, Gemini in Classroom and Google Search would be “entirely off for K-12 student accounts until school administrators explicitly turn them on”. Search was on that list, and that is no accident. The default for millions of children had now moved twice in a single term, once by product bulletin and once by blog post, in opposite directions. Neither move went through a parent.

Four weeks before the August switch, on 14 July, Anthropic had launched Claude for Teachers, giving verified American K-12 educators free access to its premium tier, a library of teaching skills, and lesson planning grounded in curricula mapped to academic standards in all fifty states. Educators who sign up before 30 June 2027 get a full year at no cost.

So within a month, the two most heavily capitalised AI laboratories in the United States both moved decisively into the K-12 classroom. One did it by flipping a switch on a platform it already owned. The other did it by handing a licence to a teacher. Those are not the same act, and the difference is where the entire accountability question lives.

What Changed in August, What Changed Back in November, and What the Headlines Missed

Precision first, because the loose version of this story is doing rounds and it is wrong in a way that matters.

Google did not reach into every school in America and turn on a chatbot for every child. The August change applies to students who had already been granted access to Gemini in Classroom, the Gemini app or Gemini Notebook by their district administrator. Where a district had switched Gemini off for students, Google's own documentation states it “will remain off for all students, including students younger than 18”. Administrators retain the ability to carve out organisational units and disable access for under-eighteens specifically.

What changed, then, is narrower and stranger than “Google turned on AI for children”. What changed is that an age gate came down. Districts that had enabled Gemini for students on the reasonable understanding that this meant their sixth formers, their eighteen-year-olds, the students Google had itself defined as the eligible population, woke up to find the same permission now flowing down to eleven-year-olds. The consent the district gave in January was for one population. In August it silently covered another.

This is a familiar manoeuvre in software, and it has a name in privacy circles: scope creep by default. The setting did not change. The meaning of the setting changed.

The Youth AI Safety Institute at Common Sense Media, which assessed Gemini K-12 in January 2026 and rated it High Risk overall, made the design consequence explicit. The product treats a sixth grader and a high school senior the same. Guided Learning mode, Google's pedagogical guardrail, is easy to switch out of, after which the model will write a full essay. Common Sense's separate assessment of AI Overviews and AI Mode in Google Search, which no user in a school can disable, rated them Unacceptable Risk after the system completed one hundred per cent of one hundred and eighty tested homework assignments.

A further default has since landed on a different data type. Google is rolling out an opt-out setting called “Personalized Recommendations and Search Services History” to all accounts, under-eighteen school accounts included, which tailors what students see using their saved activity and location data. Google's Workspace for Education documentation gives administrators a switch to turn personalised recommendations off for users designated as under eighteen, and notes that doing so neither turns off the history setting nor deletes activity already saved. Same architecture. Same default posture. The burden of noticing sits, once again, with the administrator.

Then, on 3 November, the default moved back. In a post on Google's Keyword blog titled “Designing responsible AI for learning”, Phillips said Gemini, Gemini Notebook, Gemini in Classroom and Google Search would all start switched off for K-12 student accounts, with schools able to enable each individually. “We believe schools should have control over AI in their classrooms,” he wrote on X.

That sounds like what O'Hagan asked for in August, and in part it is. Look at the last item on the list, though. Google is building Guided Learning into Search for student accounts, with a pilot in select American schools later in November, and Search now comes with the AI attached. Phillips said schools that ban AI “will not have access to search”. Later that day Google clarified that schools could enable Search without Gemini, but Search would still carry AI Overviews and other AI features. The company said the announcements “give schools more choice, not less”.

The reaction was swift. New York City Mayor Zohran Mamdani accused Google of “trying to use its monopoly power to force AI into our public schools”. Lina Khan, the former chair of the Federal Trade Commission, said the move “highlights the danger of allowing an unregulated monopolist to control an essential service”. Josh Golin, executive director of Fairplay, said Google's effort to “strong-arm schools into using its AI products is likely to backfire”. Randi Weingarten, president of the American Federation of Teachers, said: “It is unfortunate that Google has prioritized its business interests over the education of our children.”

Some districts had already stopped waiting for Google. On 2 September, New York City announced a one-year pause on student-facing generative AI for students through eighth grade, with companion chatbots prohibited at every grade. Los Angeles Unified temporarily blocked generative AI tools on student devices across all grades while it writes a policy.

None of Google's November changes has taken effect. The company says it will “share more specifics with schools in the coming months ahead of the changes rolling out”. So the August question, whether consent given for eighteen-year-olds silently covered eleven-year-olds, has been answered with a different lever. The new question is whether a district can keep a search engine without taking the AI built into it.

The Four Doors a Decision Walks Through Before It Reaches a Parent

Utah is the cleanest available illustration of how consent evaporates as it descends.

On 2 June 2026 the Utah State Board of Education announced the launch of Gemini for Education across the state's public schools from the 2026-27 academic year, in partnership with Google. The scale is roughly 680,000 K-12 students and 28,000 educators. The press release is careful, and to its credit it is careful in the right direction: local education agencies “maintain full discretion over whether to implement Google Gemini for Education”. State Superintendent of Public Instruction Molly Hart framed it around local control. Matt Winters, the board's AI specialist and, since 2024, the first full-time state-level position of its kind in the country, tied the programme to House Bill 273 and to helping districts screen software for security and privacy.

Read the chain of authority carefully.

The state board negotiates the agreement. The local education agency decides whether to adopt. The district administrator configures the console. Google sets the default inside the console. And then, in August, Google changes what that default covers, and in November announces it will change the default again on its own terms.

At no point in that chain does a parent appear. At no point does a student appear. Four institutional actors each exercised legitimate discretion, each did so within the bounds of their authority, and the cumulative result is a thirteen-year-old typing into a large language model without anybody having asked either of the two people whose data it is.

This is not a conspiracy. It is worse than a conspiracy, because it is nobody's fault in particular and therefore nobody's problem to fix. Every actor in the chain can point upstream or downstream. The state points at local control. The district points at the state contract. The administrator points at the console default. Google points at the administrator.

The Utah release does contain a genuine substantive commitment, and it deserves to be stated plainly rather than waved away: interactions remain within a secure domain, data and conversations are not used to train AI models, and institutions retain administrative control over their data. Google's Gemini for Education materials say the same, adding that data is not human-reviewed and not used for ad targeting, and listing COPPA and FERPA among the frameworks supported.

Take those commitments at face value. They are probably accurate. And notice that they answer a question nobody in this story is actually asking.

Why the Strongest Case for Doing This Deserves a Proper Hearing

The lazy version of this article treats free AI in schools as obvious predation. That version is wrong, and it is wrong in ways that make the real critique weaker.

Start with the labour arithmetic. RAND's 2025 State of the American Teacher survey found teachers working an average of forty-nine hours a week. Gallup research with the Walton Family Foundation found that teachers who use AI tools at least weekly estimate savings of 5.9 hours per week, which across a 37.4-week school year amounts to roughly six weeks of recovered time. Three in ten teachers were using AI weekly.

Six weeks. For a profession that has spent two decades being told its workload problem is a personal resilience problem, that is not a rounding error. It is the difference between a teacher who plans a lesson properly on Sunday evening and a teacher who resigns in March.

Then the equity case, which is the strongest one and the least discussed. Affluent families already have AI in the home. They have subscriptions, they have parents who know what a system prompt is, and they have children who were using these models within a fortnight of release. If the tools stay out of school, they do not stay out of childhood. They simply become another axis along which advantage compounds quietly. A free, district-managed, contractually constrained deployment inside a school is, on this reading, the only version of AI access that reaches the child whose household has no subscription and no fluent adult.

Third, the budget reality. Education's AI sector reached roughly 730 million dollars in 2026 and is projected to grow around forty per cent annually for the next decade, reaching an estimated 18.5 billion dollars by 2036. Districts are not swimming in money. When a vendor offers a capability at zero marginal cost against a paid alternative, the procurement conversation is effectively over before it starts.

And Anthropic's offering, specifically, is built to answer the pedagogical objection rather than dodge it. Claude for Teachers draws standards data from Learning Commons, a non-profit database covering all fifty states, and pulls curricular material from OpenSciEd and Illustrative Mathematics. It connects to nine existing K-12 platforms including ASSISTments, Brisk Teaching, Canva Education, Diffit, MagicSchool and TeachFX. It ships an AI Fluency course that is model-agnostic and Creative Commons licensed, which is to say Anthropic wrote training material that teaches teachers to use its competitors. Its partners include the American Federation of Teachers and Teach For America.

None of that is nothing. A company that wanted only capture would not release its literacy curriculum under a licence permitting anyone to strip its branding.

What Free Actually Buys and Why the Usual Slogan Fails Here

“If you're not paying, you're the product” is the reflex, and in this instance the reflex is close to useless.

Under FERPA's school official exception, a vendor handling education records must perform a function the institution would otherwise perform with employees, operate under the direct control of the institution regarding use and maintenance of those records, use personally identifiable information only for the purpose of the disclosure, and satisfy the criteria set out in the school's annual FERPA notification. Layer on state law and the constraint tightens further. California's SOPIPA, passed in 2014 and effective from 2016, bars targeted advertising, commercial profiling and the sale of covered student data, and it binds the vendor whether or not a contract exists. Illinois SOPPA, effective July 2021, requires published vendor lists and deletion within sixty days of a school's request. New York Education Law 2-d requires a Parents' Bill of Rights and a data security plan in every contract.

Against that stack, the crude surveillance-capitalism reading genuinely does not hold. Google says school Gemini data is not used to train models or serve ads. Anthropic says Claude for Teachers data is not used for model training and that student information is covered by a K-12 Data Processing Addendum written to comply with FERPA. Both statements are contractual commitments with legal exposure attached.

So what is the currency?

It is not data. It is dependency.

A district that runs Classroom, Chromebooks, Workspace identity and now Gemini has an integration surface with one vendor that would take three years and a bond issue to unwind. Students spend thirteen years learning to think inside one company's interface metaphors. Teachers build lesson libraries that live in one company's document format. The identity layer, the device layer, the learning management layer and the reasoning layer converge on a single supplier, and the exit cost climbs every term.

The strategic logic was described precisely by Chris Agnew, director of the Generative AI for Education Hub at Stanford University, speaking to the Christian Science Monitor about the vendors flooding into schools: “They want to capture the market. We've seen this many...” Kris Hagel, chief information officer at Peninsula School District in Washington, offered the plainer version of the same observation: “There has been an absolute gold rush.”

This is why the distinction between Google and Anthropic matters and must not be flattened.

Anthropic supplies a tool to a teacher. The teacher verifies with a school email and identification, obtains a licence, and decides what to do with it. Whether and how educational data may be used is determined, by Anthropic's own statement, by district and state policies. The teacher is a human decision point with professional judgement, and crucially the teacher is an adult who consented on their own behalf.

Google owns the platform on which the child's education physically runs. It owns the device, the account, the assignment stream, the file store and now the assistant. When Google changes a default, no human decision point stands between that change and the child. The administrator's earlier decision was retroactively repurposed.

That is not a moral verdict on either company. It is a structural fact about where accountability can be located. You can name the person who chose Claude. You cannot name the person who chose Gemini for a specific eleven-year-old, because no such person exists.

What Happens When Somebody Actually Asks the Students

The most useful recent research on this question comes from Germany, and its findings are quietly devastating for the assumptions embedded in these deployments.

In a paper titled “Mind the Trust Gap”, published to arXiv on 1 July 2026 and appearing in the Proceedings of the ACM on Human-Computer Interaction in October, Tomohiro Nagashima of Saarland University, Lisa Siegrist of the University of St Gallen, Niklas Scholz of RWTH Aachen, Shintaro Sato of Saarland, Martina Vincoli of ETH Zürich and Man Su of the Leibniz-Institut für Wissensmedien ran a speed-dating study using storyboards with sixteen school students and fifteen school teachers. It is a small qualitative study in one country, and it should be read as such. It is also, on its own account, the first attempt to systematically pair-match K-12 teacher and student views on AI decision-making control.

The headline misalignment is the trust inversion. Students trusted their teachers more than they trusted AI. Teachers believed the opposite, assuming students would trust AI more because they perceived its assessments as fairer than human judgement. The authors spell out the consequence: if teachers wrongly believe students prefer the machine, they will scaffold and intervene less often than students actually want.

The data-sharing findings are sharper still. Students expressed a strong desire to keep their own data private and to share only when necessary. They were willing to let their own teachers see it, and only their teachers, because they could see the pedagogical benefit. One thirteen-year-old participant put it directly: if you leave everything to yourself then nobody can help you, but if you share the biggest part with the teacher, the teacher can also help you.

Teachers wanted broad access, and were reluctant to grant students control over sharing precisely because student choice would degrade the system's usefulness. One teacher with fifteen years of experience explained that the low-performing students in particular would not want to share their data, which would leave the teacher without an overview. Another, with two and a half years in post, suggested a student who does not know the teacher will be notified may be able to work without a certain amount of pressure.

Sit with that last one. It is an argument for concealing monitoring from a child in order to preserve the child's relaxed performance. It was offered in good faith, by a teacher trying to help.

The authors identify asymmetric power as a root cause. Teachers are responsible for outcomes, parents and curriculum, which pushes them towards monitoring and data access. Students are the subjects of that monitoring with limited agency. The same system reads as a support tool from one chair and a surveillance tool from the other. And the researchers found that students feared constant AI monitoring and data sharing as an additional pressure in their school lives.

Now map that onto the American position. The Center for Democracy and Technology's October 2025 report, based on nationally representative surveys of 806 teachers, 1,018 parents and 1,030 students, found that thirty-one per cent of students had conducted back-and-forth conversations with AI for personal, non-schoolwork reasons on a device, tool or software provided by their school. Over two thirds of parents and students agreed that parents have no idea how students are interacting with AI. Thirty-eight per cent of students agreed it is easier to talk to AI than to their parents. Around one in five teachers and students had received any instruction on AI risks such as bias, misinformation or overreliance. Half of students agreed that AI use in class made them feel less connected to their teacher.

Children are already having personal conversations with school-issued AI. Their parents do not know. Almost nobody has been taught the risks. And in August the default moved down five year groups.

FERPA is fifty-two years old, and it shows.

The school official exception is the load-bearing wall of every ed tech deployment in America, and it was designed for a world of contracted photocopying bureaux and student information systems. Its four conditions assume a vendor performing a discrete institutional function under direct institutional control. Applying “direct control with respect to the use and maintenance of education records” to a general-purpose reasoning model whose weights, training corpus, safety filters and system prompts are all outside the district's inspection is an exercise in legal metaphor rather than legal analysis.

Researchers have caught the gap. In “Understanding How University Guidelines Address Privacy and Security Issues of Generative AI in Academic Settings”, Bei Yi Ng, Jiarui Li and Xinyuan Tong of the University of Edinburgh, Kevin Ye, Gauthami Yenne and Varun Chandrasekaran of the University of Illinois Urbana-Champaign, and Jingjie Li of Edinburgh analysed guidelines from forty-three universities across twelve countries. The paper is aimed at higher education, not K-12, and that limitation should be stated up front. Its diagnosis nonetheless transfers.

The authors observe that universities referencing FERPA for student data protection may leave leeway allowing third-party organisations such as generative AI vendors to use student metadata, a scenario disallowed under other regimes such as Canada's FIPPA, where personal data cannot be shared with third parties unless mandated by law. They describe generative AI adoption as occurring in a lemon market, where users lack visibility into model quality, risk and trade-offs. They note that in K-12 classrooms these risks are exacerbated, that children under thirteen carry additional protections under COPPA and FERPA, and that learning management systems nonetheless treat those children identically to everyone else. And they open the paper with the collapse of AllHere Education, the company behind the Los Angeles Unified chatbot, as the exemplar of what happens to sensitive student data when a vendor fails.

Then there is the enforcement problem, which is the part almost nobody outside education law knows.

FERPA has no private right of action. In Gonzaga University v. Doe, decided on 20 June 2002, the Supreme Court held that FERPA's non-disclosure provisions create no personal rights enforceable under section 1983. A parent whose child's records are mishandled cannot sue. The remedy is a complaint to the Department of Education's Student Privacy Policy Office, whose ultimate sanction is the withdrawal of federal funding, a nuclear option that has never once been deployed against a school district in the statute's history.

COPPA was supposed to backstop this for the under-thirteens, and here the news is genuinely bad. The Federal Trade Commission published its first substantial COPPA Rule amendments since 2013 on 22 April 2025, effective 23 June 2025 with full compliance required by 22 April 2026. The amendments expanded the definition of personal information, required separate parental consent for third-party disclosures, and tightened security requirements. What the Commission declined to do was codify its long-standing guidance permitting schools to authorise data collection on behalf of parents for educational technology. It left the mechanism as guidance, citing concern about conflicting with FERPA updates the Department of Education was expected to make.

The Department's own regulatory agenda listed that FERPA rulemaking, amending 34 CFR part 99 and covering disclosures to third parties such as commercial vendors, for September 2026. By mid-November no proposal had appeared in the Federal Register.

So the single legal device by which an American school substitutes its own consent for a parent's, for children under thirteen, in the year that two frontier laboratories moved into K-12 at scale, remains uncodified guidance that one agency declined to write down while waiting for another agency, which has now missed its own date.

The contrast with the United Kingdom is instructive without being triumphalist. The Information Commissioner's Office's Age Appropriate Design Code sets fifteen standards including privacy by default, data minimisation and no nudging, and requires services to differentiate protections across age bands rather than treating a six-year-old and a seventeen-year-old identically. The code does not apply to schools processing data for educational purposes. It may, however, apply to the ed tech provider, where the service meets the definition of an information society service. A regulator asking whether a default setting nudges a child towards lower privacy is asking exactly the question that went unasked on 10 August.

This Exact Film Has Screened Four Times Already

Anyone treating August 2026 as unprecedented has a short memory. There are at least four prior reels.

In 2013, inBloom launched with roughly one hundred million dollars from the Gates Foundation and the Carnegie Corporation, offering to consolidate student records so teachers could track progress and personalise instruction. It ingested data across some four hundred fields, from test scores to special education enrolment to free lunch status. There was never a documented misuse. There did not need to be. Parents organised, states withdrew, and on 21 April 2014 inBloom announced it was shutting down. The wreckage produced more than four hundred student privacy bills in state legislatures, including the SOPIPA and SOPPA statutes still doing the work today.

In December 2015, the Electronic Frontier Foundation filed a complaint with the FTC alleging that Google's Chrome Sync feature, enabled by default on Chromebooks sold to schools, allowed the company to track the websites students visited, their search habits and saved passwords, in violation of the Student Privacy Pledge Google had signed.

Enabled by default. On school devices. In 2015.

In February 2020, New Mexico Attorney General Hector Balderas sued Google, alleging that its G Suite for Education deployment collected children's physical locations, browsing and search histories, YouTube viewing habits, contact lists, passwords and voice recordings in violation of COPPA and state law. In September that year, District Judge Nancy Freudenthal dismissed the claims while granting leave to amend.

And in 2023, Los Angeles Unified signed a six million dollar contract with AllHere Education for a chatbot called Ed. AllHere furloughed most of its staff and filed for Chapter 7 bankruptcy in July 2024. Chris Whiteley, the company's former senior director of software engineering, told officials that the application had included students' personally identifiable information in every chatbot prompt, including prompts where the data was irrelevant, that those prompts had been shared unnecessarily with third-party companies, and that they were processed on offshore servers. Founder Joanna Smith-Griffin was arrested on 19 November 2024 on charges including securities fraud, wire fraud and aggravated identity theft.

Four episodes, three of them involving defaults or data flows nobody at the school had inspected. The pattern is not that ed tech companies are uniquely wicked. The pattern is that the consent architecture in American schools cannot detect a change until after it has happened, and cannot remedy one at all.

Nobody Asked the People Whose Data It Is

The final piece of research here is the one that reframes the whole argument, and it needs handling with care because it is easy to overclaim.

“Participatory, not Punitive”, by Kaoru Seki, Manisha Vijay and Yasmine Kotturi, presented at the CHI conference in Barcelona in April 2026, is a small study. Three workshops in a graduate design course at the University of Maryland, Baltimore County, a minority-serving institution, with two student leaders facilitating and no faculty in the room. Eight participants. Ten policy recommendations. A zine.

It is not a national survey of K-12 policy. It is university level, tiny, and qualitative. Anyone citing it as proof that “most educational AI policies ignore students” is citing the authors' framing of the literature rather than their evidence.

But the framing is the interesting part, and it holds. Most institutional AI policies are written without the students they govern and organised around penalising misuse. The predominant enforcement instrument, AI detection, infers authorship probabilistically rather than comparing against known sources, which makes accusations more uncertain and more contestable than plagiarism findings ever were. Policies arrive without articulated pedagogical rationale, which teaches students to read them as arbitrary exercises of authority.

And the finding the students surfaced that the administrators had not is the one that lands hardest: the double standard whereby students must disclose or abstain from AI use whilst faculty face no equivalent expectation. Instructors may use AI for teaching or grading without disclosure. Students may not use it without declaring it.

Now hold that beside August 2026. A student who uses an undisclosed AI to draft an essay commits academic misconduct. A platform that enables an undisclosed AI inside that student's coursework, without asking the student, ships a feature.

The asymmetry is total, and it is the asymmetry that Nagashima and colleagues found students perceiving from the other direction in Germany, and that the Center for Democracy and Technology measured as a collapse in felt connection to teachers.

American students have noticed. Between 17 and 19 July 2026, at America's Youth AI Festival held on the UMass Boston and MIT campuses, high school students representing all fifty states convened a National AI Student Senate at the Edward M. Kennedy Institute and passed, by eighty-two votes to sixteen, a framework called the STUDENTS FIRST Act. Its provisions include the right to decline AI use and receive an alternative assignment, a requirement that students be informed about what data is collected and how it is used, a prohibition on entering identifiable or sensitive student information into AI tools, and a right to appeal findings of improper AI use with human review by multiple officials before discipline. The organising partners included Day of AI, MIT RAISE, AASA and the Kennedy Institute.

Teenagers, given three days and a legislative chamber, produced a consent architecture more coherent than the one currently governing them.

The Question the Eleven-Year-Old Cannot Formulate

Return to the child at the centre of this, because the abstraction has a body.

She is eleven. She opens Classroom, as she has every school day for two years, and there is a tab that was not there in June. She does not know that a state board signed an agreement in a room in Salt Lake City, that a district opted in, that an administrator left a toggle where it was, or that a company in Mountain View changed what that toggle covers and has since announced it will change it again. She knows the school gave her a laptop and the laptop has a thing on it that answers questions. Schools do not, in her experience, put things on the laptop that she is not supposed to use.

She types a question about her homework. Then, because the research says roughly a third of students do, she types something else. Something about a friendship, or a fear, or a body. The Center for Democracy and Technology found that thirty-eight per cent of students find it easier to talk to AI than to their parents, and that over two thirds of parents have no idea any of this is happening.

Ask the accountability question about that conversation and watch it dissolve. Google will point to the administrator console and the contractual commitments, both of which are real. The district will point to the state framework and local discretion, both of which are real. The state will point to local control, which is real. The federal statute offers her family no cause of action at all. The Federal Trade Commission has declined to codify the rule under which her school consented on her parents' behalf. The Department of Education can, in theory, defund her school district, which it will not do, because it never has.

Every actor behaved lawfully. Every commitment was honoured. And there is no one to ask.

That is the actual finding, and it is more uncomfortable than a villain would be. The problem is not that Google is harvesting children's homework to train models, because the evidence is that it contractually is not. The problem is that the American consent architecture for student data was built on the premise that an institution can substitute its judgement for a parent's on the assumption that the institution controls the technology. That premise held for filing cabinets. It held, barely, for student information systems. It does not hold for a general-purpose reasoning model updated on a vendor's release schedule, whose capabilities in November are not the capabilities the district evaluated in January.

The fix is not complicated to describe, and Google has now described part of it. Defaults involving minors should be off, requiring affirmative institutional action to enable. Once the November changes roll out, that will be Google's own position for its core student AI products, and it deserves credit as a real concession. What is still missing is everything that would make it durable. A fresh institutional action should be required whenever the eligible age band or the scope of a setting changes. Material changes to what a setting covers should trigger notification to parents, not merely to administrators; neither the August move nor the November one did. Basic tools such as search should not be bundled with AI, so that declining one costs a school the other. Districts should publish, as Illinois already requires, the list of vendors with access to student data. The FTC should codify the school authorisation exception, and the Department of Education should publish the FERPA rule it scheduled for September, rather than leaving the consent mechanism for the youngest children as guidance in a footnote. And students, who have now drafted their own framework and found it broadly reasonable, should be in the room. A default a company chose voluntarily is a default the company can unchoose.

The alternative is the pattern already established across four episodes and twelve years. A default gets set. Nobody notices. Something goes wrong, or nearly does. Parents organise. A company withdraws or a court dismisses. A legislature passes something. This autumn the cycle simply ran faster. The August default arrived on a Monday, in a product bulletin, under a subheading called “Rollout pace”. Its reversal arrived eighty-five days later, on a Tuesday, in a blog post, with the search box tied to it. The default for millions of children moved twice in one term, in opposite directions, at one company's discretion. Neither move went through a parent.

Sources and References

  1. Google Workspace Updates, “Gemini in Google Classroom is expanding to users of all ages, with contextualized Gemini starter prompts for students,” Google Workspace Updates Blog, 10 August 2026. https://workspaceupdates.googleblog.com/2026/08/gemini-in-google-classroom-is-expanding-to-users-of-all-ages-with-contextualized-Gemini-starter-prompts-for-students.html
  2. Kahekashan, “Google enables Gemini AI for students using its Classroom App,” The Hans India (syndicating New York Times reporting of 14 August 2026), 15 August 2026. https://www.thehansindia.com/tech/google-enables-gemini-ai-for-students-using-its-classroom-app-1109903
  3. Youth AI Safety Institute, “What to know about Google turning on Gemini for schools,” Common Sense Media, August 2026. https://institute.commonsensemedia.org/what-to-know-about-google-turning-on-gemini-for-schools
  4. Google for Education, “Gemini for Education.” https://edu.google.com/intl/ALL_us/ai/gemini-for-education/
  5. Anthropic, “Introducing Claude for Teachers,” anthropic.com, 14 July 2026. https://www.anthropic.com/news/claude-for-teachers
  6. Utah State Board of Education, “Utah State Board of Education Adopts Gemini for Education to Enhance Student Learning and Teacher Support,” 2 June 2026 (updated 3 June 2026). https://www.schools.utah.gov/publicrelations/_publicrelations_/_2026/USBEAdoptsGeminiEducationPressRelease.pdf
  7. Christian Science Monitor, “Education's AI 'gold rush' comes with risks for students,” csmonitor.com, 7 August 2026. https://www.csmonitor.com/USA/Education/2026/0807/ai-risks-schools-students
  8. Tomohiro Nagashima, Lisa Siegrist, Niklas Scholz, Shintaro Sato, Martina Vincoli and Man Su, “Mind the Trust Gap: Identifying (Mis)alignments in Teacher-Student Views Toward Control and Agency in K-12 Classroom AI,” arXiv:2607.01506, 1 July 2026; Proc. ACM Hum.-Comput. Interact. 10, 6, Article CSCW124, October 2026. https://arxiv.org/abs/2607.01506
  9. Bei Yi Ng, Jiarui Li, Xinyuan Tong, Kevin Ye, Gauthami Yenne, Varun Chandrasekaran and Jingjie Li, “Understanding How University Guidelines Address Privacy and Security Issues of Generative AI in Academic Settings,” arXiv:2506.20463, 25 June 2025 (v2, 19 July 2026). https://arxiv.org/abs/2506.20463
  10. Kaoru Seki, Manisha Vijay and Yasmine Kotturi, “Participatory, not Punitive: Student-Driven AI Policy Recommendations in a Design Classroom,” arXiv:2604.10851, 12 April 2026; CHI '26, Barcelona, April 2026. https://arxiv.org/abs/2604.10851
  11. US Department of Education, “FERPA Frequently Asked Questions,” Student Privacy Policy Office. https://studentprivacy.ed.gov/frequently-asked-questions
  12. Supreme Court of the United States, “Gonzaga University v. Doe, 536 U.S. 273,” 20 June 2002, via Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/supremecourt/text/01-679
  13. Federal Trade Commission, “Children's Online Privacy Protection Rule, Final Rule,” Federal Register, 22 April 2025. https://www.federalregister.gov/documents/2025/04/22/2025-05904/childrens-online-privacy-protection-rule
  14. Electronic Frontier Foundation, “Google Deceptively Tracks Students' Internet Browsing, EFF Says in Complaint to Federal Trade Commission,” eff.org, 1 December 2015. https://www.eff.org/press/releases/google-deceptively-tracks-students-internet-browsing-eff-says-complaint-federal-trade
  15. New Mexico Department of Justice, “Attorney General Balderas Sues Google for Illegally Collecting Personal Data of New Mexican School Children,” February 2020. https://nmdoj.gov/press-release/attorney-general-balderas-sues-google-for-illegally-collecting-personal-data-of-new-mexican-school-children/
  16. Wendy Davis, “Google Defeats New Mexico AG's Lawsuit Over Students' Privacy,” MediaPost, 30 September 2020. https://www.mediapost.com/publications/article/356337/google-defeats-new-mexico-ags-lawsuit-over-studen.html
  17. Valerie Strauss, “$100 million Gates-funded student data project ends in failure,” The Washington Post, 21 April 2014. https://www.washingtonpost.com/news/answer-sheet/wp/2014/04/21/100-million-gates-funded-student-data-project-ends-in-failure/
  18. Parent Coalition for Student Privacy, “State Student Privacy Laws,” studentprivacymatters.org. https://studentprivacymatters.org/state-legislation/
  19. Information Commissioner's Office, “Introduction to the Children's code,” ico.org.uk. https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/childrens-information/childrens-code-guidance-and-resources/introduction-to-the-childrens-code/
  20. Center for Democracy and Technology, “Hand in Hand: Schools' Embrace of AI Connected to Increased Risks to Students,” cdt.org, October 2025. https://cdt.org/wp-content/uploads/2025/10/FINAL-CDT-2025-Hand-in-Hand-Polling-100225-accessible.pdf
  21. Gallup, “Three in 10 Teachers Use AI Weekly, Saving Six Weeks a Year,” news.gallup.com, 2025. https://news.gallup.com/poll/691967/three-teachers-weekly-saving-six-weeks-year.aspx
  22. Mark Keierleber, “Whistleblower: L.A. Schools' Chatbot Misused Student Data as Tech Co. Crumbled,” The 74, 2024. https://www.the74million.org/article/whistleblower-l-a-schools-chatbot-misused-student-data-as-tech-co-crumbled/
  23. AASA, The School Superintendents Association, “Students from All 50 States Produce National Framework for AI in America's Schools,” aasa.org, 3 August 2026. https://www.aasa.org/news-media/news/2026/08/03/students-produce-national-framework-for-ai-in-america's-schools
  24. Chris Phillips, “Designing responsible AI for learning,” Google, The Keyword, 3 November 2026. https://blog.google/products-and-platforms/products/education/ai-learning-era-updates-2026/
  25. Megan Cerullo, “Google will block search access for schools that ban AI use,” CBS News, 3 November 2026. https://www.cbsnews.com/news/google-search-access-schools-ai-tools/
  26. Jared Perlo and Tyler Kingkade, “Google to shut off search for schools that opt out of AI,” NBC News, 3 November 2026. https://www.nbcnews.com/tech/tech-news/google-shut-search-schools-opt-ai-rcna607280
  27. Anthony Izaguirre, “NYC, the nation's largest school system, bans AI for students through 8th grade,” Associated Press, 2 September 2026. https://www.wsbtv.com/news/technology/nyc-nations-largest/TODW6DLLH4YRJJYMLRKDCODDQE/
  28. Google Workspace Admin Help, “History and Personalization in Search services for Workspace for Education users,” knowledge.workspace.google.com. https://knowledge.workspace.google.com/admin/getting-started/editions/search-history-personalization-edu
  29. Office of Information and Regulatory Affairs, “Family Educational Rights and Privacy Act, RIN 1875-AA15,” Unified Agenda of Regulatory and Deregulatory Actions, Department of Education, Fall 2025. https://www.reginfo.gov/public/do/eAgendaViewRule?pubId=202510&RIN=1875-AA15

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Jerry Falade kept the receipts. When his debut crime novel was pulled from under him in the last days of July 2026, the Nigerian chemistry doctoral candidate at Southern Methodist University in Dallas did not simply issue a denial. He itemised his evidence. The story, the characters, the scenes, the humour, the structure and the voice all came from him, he said, and he had time-stamped drafts, written notes and dated WhatsApp messages documenting the book's development.

Read that list again, because it is the most important document in the affair and it is not a document at all. It is a defence file, assembled by a first-time novelist from the digital exhaust of the act of writing. Draft timestamps. Notes. Messages to friends. It is the evidentiary apparatus of a fraud trial, produced by a man who had, a fortnight earlier, been the most sought-after debut author in American publishing.

There was no trial, no tribunal, no standard of proof, no appeal and no finding of fact. There was a two-book deal reported at 2.4 million dollars with Macmillan's Minotaur Books, won in a fourteen-way auction, and then on 29 July there was not. His agents at Europa Content withdrew the manuscript from submission. Marc Gerald, the agent who had championed it, said the agency had vetted the book with Falade for AI use during submission and could no longer substantiate what it had been told. “We feel terrible for putting this into the universe and the firestorm it has caused,” he said. Falade's account, the agency indicated, had changed in aspects.

Notice what is missing. Nobody said the book was written by a machine. The stated reason for the collapse was that the chain of custody could no longer be authenticated. The manuscript did not fail a test. It failed to be provable.

Fourteen Bidders and a Question Nobody Could Answer

The commercial logic underneath the collapse is less about literary ethics than most of the coverage suggested, and it explains why this will keep happening. Under United States copyright law, material generated by a machine without sufficient human authorship is not protectable, as the Copyright Office set out in its March 2023 registration guidance and has restated since. On 2 March 2026 the US Supreme Court declined to hear Thaler v. Perlmutter, leaving in place the D.C. Circuit's ruling that copyright requires a human author. A publisher paying seven figures for a novel is not buying a reading experience. It is buying an exclusive right, and an exclusive right with unknown holes in it may not be worth anything.

That is why the operative sentence in the whole episode was procedural rather than moral. Those handling the book said they were no longer able to authenticate how the manuscript had evolved from origin to completion. This is a due diligence failure, not a verdict. But it lands on the author with all the force of a verdict, because from the outside there is no way to distinguish a withdrawal for unprovability from a withdrawal for guilt, and the market does not bother trying.

Falade has consistently denied writing the book with AI. He acknowledged running an open-source model locally for research, including queries about the chemistry of body disposal, which is a reasonable thing for a chemistry doctoral student writing about a cartel fixer to look up without leaving a trail in a search engine. He has also said something the industry has been reluctant to engage with. “There seems to be a troubling assumption,” he said, “that when a Black writer produces work that attracts significant attention, the work could not possibly be their own.” And on the process: “I was on the verge of success, and then all of a sudden, there were rumors, but no one asked me anything.”

Here is the detail that should trouble anyone who thinks the answer is better software. According to Gerald, Europa did not run AI-detection tools on the manuscript, because at the time it found Falade's explanation clear and credible. Detectors were used, just not by anyone who owed him anything. Some industry publications said they ran the manuscript through Pangram, and Falade says a journalist submitted it to detection software without his knowledge or consent, producing a report that counted 18 “not X, but Y” constructions and 46 em dashes. The only instruments in the story were wielded by third parties with no duty to him, on an unpublished book, tallying stylistic tics. The people with obligations to him ran nothing. The industry did not fail because its instruments were bad. It failed because those responsible had no instruments, no procedure and no threshold, and in the absence of those things it fell back on the oldest mechanism available: reputational panic.

The Machines Can Tell and You Cannot

There is a body of research that speaks directly to the editor sitting with a manuscript and a bad feeling, and its findings are more unsettling than either the optimists or the pessimists in publishing have noticed.

In January 2026, a team from Chalmers University of Technology and Edinburgh Napier University published a study at the International Conference on Agents and Artificial Intelligence titled “Interpretable Text Classification Applied to the Detection of LLM-generated Creative Writing”. Minerva Suvanto, Andrea McGlinchey, Mattias Wahde and Peter J Barclay set out to answer a specific question: not merely whether machine-generated fiction can be detected, but why the detection works.

Their corpus, named DET12, was built from twelve public-domain detective novels, six by Agatha Christie and six by Dorothy L. Sayers, taken from Project Gutenberg. They chopped the texts into excerpts of roughly a hundred words, cutting on full stops, and produced 8,068 human-written samples. Each went to GPT-4.1 with an instruction to rewrite it while preserving the key details and approximate length, yielding 8,068 machine-written counterparts.

Then they tested people. Building on an earlier experiment in which nineteen evaluators managed 55 per cent accuracy, they recruited 119 evaluators and showed each five pairs of excerpts, one human and one machine, with ninety seconds and no internet access to decide which was which. That produced 595 judgements. Of those, 297 were correct. The accuracy rate was 49.9 per cent. Not poor. Not degraded. Indistinguishable from a coin.

The machine-learning classifiers, using nothing but single-word features drawn from a vocabulary of 30,302 tokens, achieved test accuracies between 0.9310 for a random forest and 0.9814 for the interpretable linear model the authors had designed for the purpose. Naive Bayes reached 0.9793 using an algorithm that predates the web.

That gap is the crisis of confidence in publishing rendered as a number. The person whose professional identity is built on reading closely performs at chance. A bag-of-words model from the 1960s performs at ninety-eight per cent. Every editor who has ever said “I can just tell” is, on this evidence, telling you about their intuition rather than about the text.

What the Classifier Was Actually Reading

The reason the paper matters is that its authors did not stop at the accuracy figure. They built an interpretable model precisely so they could open it up, and what they found inside is far more awkward for publishing than the headline result.

The dominant signal was lexical variety. Where Christie or Sayers wrote “said”, the rewrite reached for “remarked”, “responded” or “mentioned”; “cried” became “exclaimed”; plain adjectives such as good, bad, great and beautiful gave way to more elaborate alternatives. The authors measured this information-theoretically and found the machine text carried 9.164 bits of unigram entropy against 8.875 for the human originals, implying roughly 22 per cent greater variation in word choice. The machine is not writing worse. It is writing with a bigger thesaurus and less restraint, and that alone gives it away.

But four further categories emerged from the hand annotation, and three of them should stop any editor cold.

The second was temporal drift. The human corpus was written in the 1920s and 1930s, and it uses the spellings of that period, including “to-day” and “to-morrow”. The machine modernises them. The authors offer one word as an illustration. The verb “exited” barely appears in English before about 1960. It occurs zero times across the human samples and 61 times in the machine rewrites of the training set.

The third category, which the paper labels English varieties, is the one that ought to be printed and pinned above every commissioning desk in London. The authors note that the human texts were written by British authors, and that the model “exhibits a preference for Americanisms”. The example given in their table is the pair realise and realize.

The fourth and fifth were foreign language and colloquialism. The Christie and Sayers originals carry passages of French and contractions such as “'em” for “them”, and the model strips both. The authors observe, in a sentence that reads like a lament, that such features “often make a novel more vivid and interesting to read”.

So the ninety-eight per cent accuracy is real, and it is partly an artefact of the fact that one class was written by British novelists a century ago and the other by an American model last year. The classifier has learned, among other things, that British spelling and dated idiom mean human. The authors are candid that their rewriting methodology may itself have inflated the synonym variety, and that prompting a model to compose from an outline would be a different test. They report preliminary work suggesting detectability survives, but say plainly that further work is needed.

Now transplant that classifier into a slush pile.

Arithmetic That Publishing Has Not Done

The DET12 experiment is a balanced binary forced choice. Half the samples are machine-written by construction, and the classifier is asked to sort a closed set in which the prior probability is exactly fifty per cent.

Manuscript screening is not that problem. It is a rare-event screening problem, and the difference is the difference between a useful tool and a machine for generating accusations.

Take a mid-sized imprint receiving ten thousand submissions a year, and assume, generously to the detector, that five per cent contain substantial machine-written prose. Run a detector with the DET12 linear model's characteristics: it correctly flags 490 of the 500, and also flags two per cent of the 9,500 clean ones, which is 190 authors. Of the 680 manuscripts in the flagged pile, 190 belong to writers who did nothing. Roughly one in four accusations is false.

Drop the prevalence to one per cent, plausible for agented submissions from established writers, and the arithmetic becomes grotesque. Ninety-eight true positives sit alongside 198 false ones. Most of the accused are innocent. The tool is still ninety-eight per cent accurate, and worse than useless, because the two errors are not symmetrical: missing a machine-written book costs a publisher an unenforceable copyright, while falsely flagging a human one ends a career.

This is not a hypothetical failure mode. It is a documented one, and it has already played out at scale in another industry entirely.

The Academy Already Ran This Experiment and Lost

Turnitin launched its AI writing detector in April 2023 and, within four months, had run it across more than 38 million student submissions. The company initially claimed a false positive rate of around one per cent at document level. It later acknowledged that at sentence level the figure was closer to four per cent.

Vanderbilt University did the arithmetic that publishing has not. Noting that it had submitted around 75,000 papers to Turnitin in 2022, the university's Center for Teaching observed that a one per cent false positive rate would have meant roughly 750 students wrongly implicated in a single year. In August 2023 it disabled the detector.

Then came the bias finding. In a paper published in Patterns in July 2023, Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu and James Zou of Stanford tested seven widely used GPT detectors against writing by native and non-native English speakers. The detectors were near-perfect on essays by American eighth-graders. On TOEFL essays written by non-native speakers, they misclassified more than half as machine-generated, at an average false positive rate of 61.3 per cent. The mechanism is not mysterious. These systems key on perplexity, a measure of how statistically surprising a text's word choices are. A writer with a constrained lexical range produces low-perplexity prose. So does a language model. The detector cannot tell the difference between a machine and a person who has fewer words available, and it resolves the ambiguity against the person.

Turnitin disputes the read-across, and its objection is not frivolous. It tested nearly two thousand samples from English language learners and reported false positive rates of 0.014 for those writers against 0.013 for native speakers on documents meeting its 300-word minimum, noting that the Stanford work used short samples and did not test Turnitin at all. The dispute is unresolved. The shape of the risk is not: the burden of proof falls hardest on people whose English is correct but plain.

Hold that beside a Nigerian doctoral student writing crime fiction in his second language, whose unpublished manuscript was scored on its em dashes by someone outside the transaction, and the phrase Falade used about assumptions concerning Black writers stops sounding like special pleading and starts sounding like a description of a known technical failure mode, operated by whoever chose to run it.

A Prize Awarded Twice and Believed Once

If Falade is the author who lost everything to unprovability, Jamir Nazir is the author who proved it and lost anyway.

In May 2026 the Trinidadian writer's short story “The Serpent in the Grove” was named the Caribbean regional winner of the Commonwealth Short Story Prize. Within weeks, as the Guardian reported, it was engulfed in accusations of machine authorship, focused in part on a line that readers found impossible to parse: “She had the kind of walking that made benches become men.” Pangram, one of the better-regarded commercial detectors, returned a verdict of one hundred per cent machine-generated.

The Commonwealth Foundation did something almost nobody else in this story did. It investigated. It reviewed the regional winning entries and asked the writers for evidence of process: drafts, timestamped documents, notes, outlines. Having examined them, it concluded that AI had not been used. Director General Razmi Farooq said the Foundation had chosen not to rely on detection software because such tools produce inconsistent results, and that literary work should be judged through the work itself and its development rather than through automated assessment. Louise Doughty, who chaired the judging panel, stood by the story and the shortlist. At the turn of July it named Nazir overall winner from 7,806 entries.

That is a process. It is close to the only functioning adjudicative process anyone in this field has demonstrated. It has a standard of evidence, it puts the burden in a defensible place, and it explicitly refuses to outsource the judgement to a classifier whose error characteristics nobody in the room can audit.

It did not save him. Granta, which had published Commonwealth Prize winners for years, had already ended the partnership in June, citing editorial integrity. And when Nazir gave further interviews, his defence complicated rather than settled matters. He compared AI to the typewriter and the word processor. He cited Derek Walcott as a formative influence and then, as Slate reported in July, was unable to discuss Walcott's work with any specificity, referring to the Nobel laureate's poems as stories, and attributing his difficulty to brain fog from diabetes and cancer treatment.

So we are left with the genuinely uncomfortable case, and it is the one worth sitting with rather than resolving. A man was cleared by the only rigorous investigation anyone has run, and he remains, in the eyes of much of the literary internet, guilty. His evidence was examined. His explanations were not persuasive. Those two facts do not cancel out, and no institution in publishing has the authority to say which one governs. The verdict floats free of the process, which is another way of saying there is no process.

The Freelancer at the Bottom of the Blame Chain

The third case shows where the liability actually settles, which is not on the author and certainly not on the publisher.

Mia Ballard self-published the horror novel Shy Girl in February 2025. Hachette Book Group acquired it, published in the United Kingdom that November, and scheduled a United States edition. In January 2026 a video by the YouTube reviewer frankie's shelf argued the book bore the hallmarks of machine composition: groupings of three, an obsessive recurrence of the word “sharp”, strange formatting. It drew more than a million views. In March, Hachette cancelled the American release and withdrew the British printing, the first withdrawal of its kind by a major house.

Writing in Slate at the end of March, Imogen West-Knights read the novel closely and described excessive parallelism, repeated vocabulary and awkward similes, and a sense of “a lack of a person behind the words”. A freelance editor with twelve years in publishing, quoted under a pseudonym, put it more brutally: there was no phrase, word choice, grammar, structure, syntax or punctuation mark in the book that ChatGPT would not have produced.

Ballard denied using AI herself. She told the New York Times in March that a freelance editor she had hired may have introduced it, and that she could not elaborate because she was pursuing legal action against that editor. The editor has never been publicly identified.

Whether or not that account holds, it identifies the real fault line. Machine text does not only enter a book through the author. It enters through the freelance developmental editor, the line editor, the copyeditor, the writing contractor, the cover designer. Hachette, like other houses, publishes a list of AI uses that do not require disclosure, including research, while requiring authors to attest that the work is their own. Read those two provisions together and the design is clear. The publisher takes a warranty, the author gives it, and the author is warranting the conduct of a chain of subcontractors they did not vet, cannot audit and in many cases hired from a directory. Here the blame came to rest on someone nobody can name.

New Zealand's Ockham Book Awards showed the same structure in miniature in November 2025, when Stephanie Johnson's Obligate Carnivore and Elizabeth Smither's Angel Train were ruled ineligible for the 2026 fiction prize over machine-generated jacket imagery. Neither novelist chose the covers, and the rule barring AI in content or artwork had been introduced in August, after the covers were designed.

An Addendum Where the Handshake Used to Be

Which brings us to the editor who cannot tell, and what she has done about it.

Mary Kole is a former literary agent with an MFA in creative writing who runs Mary Kole Editorial. In a piece published on Jane Friedman's site on 13 August 2026, she described editing two manuscripts that turned out to have been substantially machine-assisted, and what happened to her practice afterwards. She was, she wrote, “trying to find the humanity in work generated by AI”, a task that added a layer of nuance and complexity she had not signed up for.

Her response was contractual. She added an Artificial Intelligence and AI-Generated Content Addendum to her client agreement. It requires clients to disclose AI use. It defines what counts as AI-generated content. It reserves her right to run detection. And it limits her responsibility for factual accuracy and for copyright eligibility, which is a quiet acknowledgement that an editor working on machine text may be polishing something the client cannot own.

She also raised her price, adopting a fifty per cent premium on machine-assisted work proposed by the author and book coach Josh Bernoff, whose reasoning is that making such a manuscript copyright-eligible requires rewriting essentially every word, at minimum half as much work again as editing human prose. Bernoff's summary of the raw material is the most quotable line in the debate: such manuscripts are “clean, free of grammatical errors, and smooth and easy to read. They are also soulless.”

Kole's observation about her clients is the one that matters most and has been least discussed. Inexperienced writers often believe the machine has improved their manuscript. They are not cynics gaming a system. They are people who ran their draft through a model, saw cleaner sentences come out, and concluded that cleaner was better. What they actually purchased was the removal of everything idiosyncratic enough to be theirs, and then a bill for fifty per cent more to have an editor attempt to put a voice back in from the outside. The cost is not only money. It is that the creative decisions which should belong to the writer migrate to whoever is hired to repair the damage.

Around her, the institutions are moving. The Editorial Freelancers Association published its 2026 rate chart in March, built on a survey of more than 1,100 members, over a third of its membership. In May 2026 the Authors Guild added four model clauses to its trade book and literary translation contracts, requiring an author's written consent before a publisher uses machine translation, machine narration or machine-generated cover art, alongside a provision that the publisher will not use AI to substantially edit a manuscript beyond basic spelling and grammar checking. That last clause is the first genuine symmetry in the arrangement: for two years authors have been warranting their purity to publishers, and now publishers are being asked to warrant theirs back.

The Continuum Argument Deserves Its Best Hearing

There is a case against everything above, and it is stronger than its opponents concede.

Publishing has never demanded a chain of custody for human intervention. Ghostwriters produce entire bestsellers under other people's names, legally, disclosed to nobody, routinely. Book doctors restructure novels. Developmental editors rewrite scenes. Celebrity memoirs are drafted by people whose names appear, if at all, in a thanks-to line. Nobody has asked the reader to consent to that, and nobody has suggested the resulting book is uncopyrightable. Meanwhile the tool continuum runs from Word's grammar checker through Grammarly to Sudowrite with no obvious point at which assistance becomes fraud, and the industry has never had to locate that point.

The novelist Sam Munson made a version of this argument in the New York Post on 16 August 2026, under a headline cheering the collapse of the 2.4 million dollar deal as something that ought to open everyone's eyes. Whatever one makes of the framing, an industry in an uproar about a mysterious new literary talent is an industry telling on itself.

And look at who the suspicion has landed on. A Nigerian doctoral student. A Trinidadian short story writer. A novelist who came up through self-publishing. Two New Zealand authors punished for a decision taken in a design department. Publishing's gatekeeping has always had a class and a passport, and it is worth entertaining that a mechanism which cannot be audited, has no appeal and reliably fires on outsiders is doing exactly what gatekeeping mechanisms usually do.

Two things break the continuum argument, and both are matters of fact rather than taste.

The first is that a ghostwriter is a person. They can be paid, credited, sued, subpoenaed and named, and the copyright in what they produce exists and can be assigned. That is the basis on which a publisher transacts. Machine output has no such person behind it, which is why Minotaur's exposure was settled law rather than squeamishness.

The second is scale, and it has now been measured. In July 2026 Tuhin Chakrabarty of Stony Brook University, with Xinyue Liu, Jane Ginsburg of Columbia Law School and Paramveer Dhillon of the University of Michigan, ran full-text detection across 14,419 self-published genre fiction titles sold on Amazon between 2023 and March 2026, matched to daily sales data through June 2026. Books containing substantial machine-written text made up around twenty per cent of the catalogue and up to 31 per cent of new entrants to genre top-25 lists. Over the window the number of selling books grew 19.2-fold while quarterly revenue grew 8.9-fold, and titles with no machine text lost the most ground precisely in the genres where machine text had spread furthest.

That is the difference between ghostwriting and generation, and it is not a difference of principle. It is a difference of throughput. Ghostwriters have never been able to flood a category.

Eighty-One Dollars Buys Away Every Tell

The final piece of evidence is the one that dismantles the industry's entire current defensive posture, and it comes from the same research group.

In a preregistered study first posted in October 2025, Chakrabarty, Ginsburg and Dhillon set MFA-trained writers against ChatGPT, Claude and Gemini, each producing excerpts of up to 450 words emulating fifty award-winning authors including Nobel laureates and Booker winners. A panel of 159 readers, 28 MFA candidates from leading American programmes and 131 general readers, assessed the results blind in pairwise comparisons.

With ordinary prompting, the experts were emphatic, disfavouring machine text on stylistic fidelity with an odds ratio of 0.16 and on writing quality with an odds ratio of 0.13. This is the result publishing has been quietly relying on: professionals can tell, and what they are detecting has a name. The paper's account of what the experts were reacting to lists clichés, purple prose, too much exposition, absent subtext and mixed metaphors, echoing prior work finding machine creative writing characterised by exactly those faults. Stylometrics put a number on it. Cliché density correlated with the Pangram detector's score at a Pearson coefficient of 0.60, and mediated 16.4 per cent of the effect of detectability on reader preference.

Then the researchers fine-tuned GPT-4o on each author's complete works, and every one of those findings inverted. Expert readers now favoured the machine on stylistic fidelity with an odds ratio of 8.16, and on quality at 1.87. Lay readers shifted further still. The mediating role of cliché density collapsed from 16.4 per cent to a statistically insignificant 1.3 per cent, which the authors read as evidence that fine-tuning eliminates the stylistic signature rather than masking it. Detection collapsed with it: fine-tuned output was flagged as machine-written three per cent of the time, against 97 per cent for ordinary prompting.

Fine-tuning cost between 22.25 and 272.50 dollars per author, median 77.88, plus about three dollars of inference to generate a hundred thousand words. The paper puts that at roughly 0.3 per cent of the 25,000 dollars a professional writer would charge, and notes honestly that it excludes the human labour still needed to turn raw output into publishable prose.

Read those two results together and the position publishing has adopted becomes untenable. Every tell that editors have learned to recognise, and every feature that both detectors and the DET12 classifier key on, is a property of cheap machine writing. Clichés, elaborate synonyms, flattened idiom, Americanisms, absent subtext: these are artefacts of prompting a general model, not signatures of machine origin. Roughly eighty dollars removes them.

So the industry has built its defences against the version of the problem that is about to stop existing, and the people who keep tripping those defences will be the ones whose prose happens to resemble cheap machine output. Writers with plain vocabularies. Writers working in a second language. The suspicion will not track machine use. It will track poverty and accent.

The Pulitzer finalist Vauhini Vara, quoted in the same paper, described the default machine register as “polite, predictable, inoffensive, upbeat”, and observed that great characters are not polite, great plots are not predictable, great style is not inoffensive and great endings are not upbeat. Speaking to the Christian Science Monitor in August, she identified what has actually been lost. Until AI, she said, there was an expectation that no matter what was written in a book, it had been authored by a person. That expectation was not a rule. It was infrastructure, and everything from the editorial relationship to the copyright transaction was built on top of it without anyone noticing they were standing on something.

Ten Dollars for a Sticker That Nobody Reads

So what is publishing actually doing in place of the thing it lost?

Since January 2025 the Authors Guild has operated a Human Authored certification mark, expanded in March 2026 to all authors published in the United States and to the United Kingdom in partnership with the Society of Authors. By that month more than three thousand authors had certified over five thousand titles. Registration requires third-party identity verification, a signed licence agreement and a fee of ten dollars per title.

It does not require anybody to read the book. The Guild does not analyse the manuscript. The certification operates on the honour system.

That is not a criticism of the Guild, which has been more energetic on this than any publisher and honest about what the mark does. It is a description of where the industry has arrived. Confronted with a crisis of textual authentication, the trade's flagship authenticity instrument authenticates the author's identity and takes their word about the text. Amanda Annis of Trident Media Group put the general condition accurately: the standards around it do not exist right now. The nearest thing agents have offered is Regina Brooks, president of the Association of American Literary Agents, saying after the Falade collapse that “The author-agent relationship depends on trust and honesty” and that authors have “a responsibility to be transparent about how a manuscript was created”. That is a duty of disclosure with no form, no definition and no consequence.

They could, and pieces already exist outside the Anglophone trade. The Commonwealth Foundation built an adjudication process. Ask for the drafts, the notes, the outlines, the timestamps. Judge the work through its development rather than through a classifier whose confusion matrix nobody in the room can interpret. Publish the finding. Accept that this is slower than running a detector, and accept it because the alternative has a documented false positive rate that lands on the people least able to survive it. None of that is exotic. It is roughly what a serious plagiarism investigation has always involved, and publishing has run those for a century without collapsing.

South Korea has built the other half, a disclosure taxonomy. On 29 September 2026 the Korean Publishers Association, the Writers Association of Korea and the Publishing City Culture Foundation announced voluntary principles with three levels: “Human Authored Work”, with AI confined to tasks such as proofreading or outlining; “AI-Assisted Work”, completed after human control and verification of machine material; and “AI-Generated Publication”, where that control is absent. Authors declare their level on submission, and the upper two are marked on the imprint page. It is non-binding, “a promise from the field”, and still rests on the author's word, but it asks the right question: “not whether to use the tool, but how far tasks have been delegated to AI.” American and British publishing has produced neither the adjudication nor the taxonomy.

What it requires is that somebody be willing to make a finding and stand behind it, which is precisely what nobody in the Falade case did. His agents did not conclude that he used a machine. They concluded that they could not verify that he had not, and withdrew. Minotaur did not accuse him of anything. It walked away from an asset with a title defect. Every party behaved rationally, nobody made an allegation, and a man's first novel was destroyed anyway.

Set that beside Daggermouth. In July the researchers behind the market study shared data with The Atlantic putting H.M. Wolfe's BookTok hit at about 60 per cent AI-generated or AI-assisted by Pangram's measure. Wolfe, a self-published author who had since signed a seven-figure deal, called the claim “wholly untrue” through her lawyer, adding that such accusations “should not be made on the basis of technology that is known to be unreliable”. Falade could have said the same. The difference is that Simon & Schuster said it stood behind her and the book, and the book stayed on sale. Neither outcome turned on evidence. Each turned on whether the people with obligations to the author chose to stand behind them, and the record does not say why one did and the other did not.

He still has the file. The time-stamped drafts, the written notes, the dated WhatsApp messages, a Google Docs version history in which, he says, anyone can watch the manuscript evolve: the whole apparatus of proof, assembled by an author who did not know he would need it, addressed to an industry with nowhere to send it. In September he announced a new agent, David Vigliano, and he now speaks about how writers whose first language is not English get flagged as machines, including at Nigeria's Aké Arts and Book Festival. “The machine made the accusation,” he wrote. “The human had to prove his humanity.” He has found someone to represent him. No institution has examined the file. Publishing has spent 2026 discovering that it cannot read a manuscript any more. It has not spent a single day building the thing that would let it read a writer.

Sources and References

  1. Wall Street Journal, “AI Has Plunged the Book Publishing Industry Into Utter Chaos”, 17 August 2026. https://www.tovima.com/wsj/ai-has-plunged-the-book-publishing-industry-into-utter-chaos/
  2. Suvanto, M., McGlinchey, A., Wahde, M. and Barclay, P. J., “Interpretable Text Classification Applied to the Detection of LLM-generated Creative Writing”, ICAART 2026, 12 January 2026. https://arxiv.org/abs/2601.07368
  3. Chakrabarty, T., Ginsburg, J. C. and Dhillon, P., “Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writers”, 15 October 2025, revised 17 March 2026. https://arxiv.org/abs/2510.13939
  4. Chakrabarty, T., Liu, X., Ginsburg, J. C. and Dhillon, P., “Generative AI Floods and Dilutes the Market for Books”, 22 July 2026. https://arxiv.org/abs/2607.20349
  5. Liang, W., Yuksekgonul, M., Mao, Y., Wu, E. and Zou, J., “GPT detectors are biased against non-native English writers”, Patterns, July 2023. https://pubmed.ncbi.nlm.nih.gov/37521038/
  6. Moneywise, “Author Jerry Falade loses $2.4 million book deal after AI accusations”, 8 August 2026. https://moneywise.com/news/top-stories/ai-book-deal-jerry-falade-publishing-industry
  7. Futurism, “Major Publisher Cans $2.4 Million Book Deal After Author Was Accused of Using AI, for a Very Cynical Reason”, August 2026. https://futurism.com/artificial-intelligence/major-publisher-million-book-deal-author-accused-ai
  8. Christian Science Monitor, “Book publishers face rising scrutiny over AI usage by authors”, 12 August 2026. https://www.csmonitor.com/Arts-Culture/Books/2026/0812/ai-publishing-commonwealth-prize-shy-girl
  9. Kole, M., “After Editing Two AI-Assisted Manuscripts, I Rewrote My Client Agreement”, Jane Friedman, 13 August 2026. https://janefriedman.com/after-editing-two-ai-assisted-manuscripts-i-rewrote-my-client-agreement/
  10. Authors Guild, “AG Introduces New Publishing Agreement Clauses Concerning AI”, May 2026. https://authorsguild.org/news/ag-introduces-new-publishing-agreement-clauses-concerning-ai/
  11. Publishers Weekly, “Authors Guild Expands 'Human Authored' Certification to All U.S. Authors and Publishers”, March 2026. https://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/99836-authors-guild-opens-human-authored-certification-to-all-u-s-authors-and-publishers.html
  12. West-Knights, I., “I read the novel with suspected A.I. use. What I found was revealing”, Slate, 31 March 2026. https://slate.com/culture/2026/03/shy-girl-mia-ballard-novel-a-i-book-horror-reddit-hachette-canceled.html
  13. Slate, “Jamir Nazir's defense against A.I. accusations only makes the case stranger”, 7 July 2026. https://slate.com/culture/2026/07/jamir-nazir-commonwealth-short-story-prize-ai-granta-magazine-controversy.html
  14. The Daily Star, “Jamir Nazir wins 2026 Commonwealth Short Story Prize following AI review”, 2026. https://www.thedailystar.net/culture/books-literature/news/news/jamir-nazir-wins-2026-commonwealth-short-story-prize-following-ai-review-4214976
  15. Vanderbilt University Center for Teaching, “Guidance on AI Detection and Why We're Disabling Turnitin's AI Detector”, 16 August 2023. https://www.vanderbilt.edu/brightspace/2023/08/16/guidance-on-ai-detection-and-why-were-disabling-turnitins-ai-detector/
  16. Turnitin, “New research: Turnitin's AI detector shows no statistically significant bias against English language learners”, 2023. https://www.turnitin.com/blog/new-research-turnitin-s-ai-detector-shows-no-statistically-significant-bias-against-english-language-learners
  17. RNZ, “Top writers ruled out of NZ book awards due to AI covers”, November 2025. https://www.rnz.co.nz/life/books/top-writers-ruled-out-of-nz-book-awards-due-to-ai-covers
  18. Editorial Freelancers Association, “EFA Announces 2026 Editorial Rate Chart”, March 2026. https://www.the-efa.org/2026-editorial-rate-chart/
  19. United States Copyright Office, “Copyright Registration Guidance: Works Containing Material Generated by Artificial Intelligence”, 16 March 2023. https://www.federalregister.gov/documents/2023/03/16/2023-05321/copyright-registration-guidance-works-containing-material-generated-by-artificial-intelligence
  20. Munson, S., “I'm a novelist, cheering the $2.4M AI book deal that should open everyone's eyes”, New York Post, 16 August 2026. https://nypost.com/2026/08/16/opinion/im-a-novelist-cheering-the-2-4m-ai-book-deal-that-should-open-everyones-eyes/
  21. Nawotka, E. and Spratford, S., “After the Cancellation of 'Call Me, I'll Hide the Body,' What Comes Next?”, Publishers Weekly, 4 August 2026. https://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/100983-after-the-cancellation-of-call-me-i-ll-hide-the-body-what-comes-next.html
  22. Schmidt, K. and Falade, J., “EXCLUSIVE: Author Jerry Falade Reflects on Losing His Book Deal and Being Accused of Using AI”, Publishing Confidential, 17 September 2026. https://kathleenschmidt.substack.com/p/exclusive-author-jerry-falade-reflects
  23. Oremus, W., “Is This What Comes After AI Slop?”, The Atlantic, 27 July 2026. https://www.theatlantic.com/technology/2026/07/daggermouth-novel-bestseller-ai/688067/
  24. The Kyunghyang Shinmun, “Did AI only proofread, or did it also write... Publishing sector announces recommended standard for 'three-level labeling of AI use'”, 30 September 2026. https://www.khan.co.kr/en/article/202609301406027/
  25. Morgan Lewis, “US Supreme Court Declines to Consider Whether AI Alone Can Create Copyrighted Works”, 6 March 2026. https://www.morganlewis.com/pubs/2026/03/us-supreme-court-declines-to-consider-whether-ai-alone-can-create-copyrighted-works
  26. Alter, A., “Horror Novel 'Shy Girl' Canceled Over Suspected A.I. Use”, New York Times, 19 March 2026. https://www.nytimes.com/2026/03/19/books/shy-girl-book-ai.html

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The email looked like a revenue opportunity. In early June 2026, some developers whose Android apps had been downloaded millions of times received an invitation from Google to “join a confidential content offer pilot”. The pitch was simple: “Get paid for sharing the code powering your apps, as well as your archived projects.” The reassurance came straight after: “This is non-exclusive. You keep 100% of your IP.”

The message, first reported by Jason Koebler at 404 Media and then widely picked up, never used the words artificial intelligence. It said the code would help “improve Google's developer tools and products”. A linked page was more candid, describing “partnerships to improve our AI products”. Nobody outside the pilot knows what Google was offering to pay.

Earlier in the year, a founder named Keelan Miskell had been closing down his crypto startup. According to MIT Technology Review, he netted roughly $25,000 for his codebase, with a further payment, probably around $15,000, pending for the company's workspace data from Google, Notion, Figma and Slack. “At least all that stuff we were doing for two and a half years is worth something to someone somewhere,” he told the magazine.

Then, in August, came the deal that made everyone look up. In the bankruptcy of Spirit Airlines, which stopped flying in May, Google won an auction for the carrier's corporate data with a $10 million bid. TIME's Billy Perrigo reported that the cache held 100 million emails, 500 million Microsoft Teams messages and about 30 million lines of code. A court filing released in November put the volume far higher: about 1.9 billion internal emails, 664 million Teams items, 91 million SharePoint and OneDrive files and roughly 30 million lines of code.

Taken together, these episodes describe something new: a market in which the private working record of software (the code, but also the commit history, the pull requests, the arguments in review threads, the tickets and the design documents) is being bought and sold as AI training material. What ethical, legal and security problems arise when that record changes hands? How can a buyer prove a seller had the right to sell it, when dozens of people wrote it? And does the money now flowing into this trade create risks that existing safeguards were never designed to catch?

Why the Messy Middle of Software Became Valuable

For a decade, AI companies trained on the public internet, and code was one of its richest seams. Public repositories on GitHub and elsewhere supplied billions of lines for the first generation of coding assistants. That seam is thinning. Epoch AI, a research group that tracks the inputs to machine learning, estimated in 2024 that the effective stock of human-generated public text was around 300 trillion tokens, and gave an 80 per cent confidence interval that it would be fully used at some point between 2026 and 2032.

Private code is the obvious next reservoir. It is not one that platforms can simply tap: when GitHub updated its terms in March 2026 to let it train on Copilot interaction data from individual users who do not opt out, it stated plainly, “We do not use private repository content at rest to train AI models.” Private repositories have to be acquired from whoever controls them, and they are valuable for reasons beyond volume. A finished public library shows what working software looks like. A private repository, with its full history, shows how software actually gets made: the false starts, the reverted commits, the reviewer who points out a race condition, the ticket that explains why a strange workaround exists. That is precisely the material that the newest training methods crave.

Turing, the AI training and talent company, made the case explicitly in its AGI Advance newsletter of 24 December 2025, announcing a programme it called Project Lazarus. “We're acquiring full private repositories, design docs, testing logs, infrastructure manifests, and internal wikis from defunct startups,” it wrote, naming fintech, pharmaceuticals, manufacturing and software as target sectors. Existing datasets, the newsletter argued, “fail to capture the texture of real work”. The stated aim was to “preserve the complete operational history of defunct companies”.

The appetite is driven by reinforcement learning. Rather than simply predicting the next token of text, AI agents are increasingly trained in simulated environments where they take sequences of actions and are rewarded for good outcomes. TIME reported that gains from this approach have been largest in coding, where results are easy to verify. Surge AI builds such environments, which it calls reinforcement-learning gyms, and Nick Heiner, who leads that work, told MIT Technology Review he wants archives that capture varied situations, disagreements and judgement calls. Speaking to TIME about the Spirit acquisition, he said the purchase makes sense “if you believe that your agent is going to generalize into the rest of the economy.”

There is also a quality argument. In a widely cited 2024 paper in Nature, Ilia Shumailov and colleagues showed that models trained recursively on the output of other models suffer “model collapse”, progressively losing the rarer parts of the original data distribution. They concluded that data about genuine human interactions will become increasingly valuable as AI-generated content floods the web. A repository written by humans, before coding assistants were commonplace, is that kind of data.

A Market Assembled in Under a Year

The supply side organised itself quickly. Startups fail constantly; MIT Technology Review cited estimates from Carta and AngelList showing shutdowns rising by between 26 and 56 per cent a year recently. An “end-of-life” industry has grown up around them, and part of that industry has discovered that the corpse has value.

SimpleClosure, a service that handles the legal and operational wind-down of startups, added data sales in April 2026 through a platform called Asset Hub, where closing companies can list code repositories, Slack archives, emails and internal documents. In April, Forbes reported that the company had handled nearly 100 such deals, with sellers typically receiving between $10,000 and $100,000; heise online, summarising that report, quoted chief executive Dori Yona describing “a feeling of a gold rush from these companies trying to get their hands on real-world data”. By October, MIT Technology Review reported more than 150 deals, an expectation of returning more than $10 million to startups by the end of 2026, and Yona's estimate that 47 per cent of SimpleClosure's deals now include code or data sales, once rare and “somewhat taboo”.

Competitors followed. Brendan Mahony, chief executive of Sunset, another wind-down firm, told MIT Technology Review that his company values assets by headcount, years in operation, data volume and how the team used its tools. “This operational exhaust data is like the wood shavings that everyone thought had no value, but it actually has a ton,” he said. Brokers now publish buying guides. One, run by Human Union Data, lists Project Lazarus as paying between $10,000 and $100,000 for legacy code; that figure is the broker's own claim, and Turing does not publish its prices. Dan Novaes, chief executive of the data broker Mode.Inc, told Business Insider, as reported by IBTimes UK in October, that “the demand is just astronomical.”

The buyers are foundation labs, vendors that build simulated workplaces for training, and established technology companies building their own models. What they want is completeness. The Human Union Data guide tells sellers that buyers are after “real production code, internal documentation, and development history”, listing “pull requests, design docs, incident notes” as the material that makes a package whole. The completeness that makes the data valuable is exactly what makes it legally and ethically fraught.

A Repository Is a Crowd, Not a Possession

Start with the most basic question: who owns a git history?

The instinctive answer is the company. In many cases that instinct is broadly right, but only broadly. Under section 11 of the UK's Copyright, Designs and Patents Act 1988, “The author of a work is the first owner of any copyright in it,” with an exception: where a work is made by an employee in the course of employment, “his employer is the first owner of any copyright in the work subject to any agreement to the contrary.” The United States reaches a similar destination through the “work made for hire” doctrine, which covers “a work prepared by an employee within the scope of his or her employment.”

Now look at who actually appears in the commit log of a typical startup. There are employees, certainly. There are also freelancers and agencies, whose work is not automatically owned by the client in either jurisdiction; in the United States, commissioned software does not fall within the nine statutory categories that can be made works for hire by written agreement, so ownership passes only through an explicit assignment. There are co-founders who wrote early code before the company was incorporated. There are interns, advisers and contributors from partner firms. A repository that has run for five years may contain the work of dozens of people under a dozen different contractual arrangements, some of which were never written down.

Then there is everyone else's code. Private repositories are full of open-source components, vendored libraries and snippets copied from elsewhere, each carrying its own licence. Black Duck's 2026 Open Source Security and Risk Analysis report, which audited 947 commercial codebases across 17 industries, found licence conflicts in 68 per cent of them, up from 56 per cent the year before; one codebase contained 2,675 distinct conflicts. The report links part of that rise to AI coding assistants reproducing material derived from copyleft code without the original licence information, a phenomenon it calls licence laundering. A seller who cannot say which open-source licences govern the files in a repository is in no position to warrant that those files can be licensed onwards for training.

Open-source projects have built some machinery for this problem. Many use contributor licence agreements or the Developer Certificate of Origin, a short statement in which a contributor certifies that “The contribution was created in whole or in part by me and I have the right to submit it under the open source license”. Private companies rarely have anything comparable. The rights in their repositories are inferred from employment contracts, many of which predate the idea that internal engineering discussion could become a product.

Nor is copyright the only right at stake. Review threads and commit messages are written conversation: explanation, critique, sometimes humour or irritation. An employer may own the text, but that does not settle whether selling it is fair to the people who wrote it, or lawful under data protection law.

Possession Is Not the Same as Ownership

The Spirit Airlines auction has become the first public test of these questions. Mercor, an AI data company, was named backup bidder at $7.5 million, and another firm, micro1, later filed notice of a $12.5 million offer. The approval hearing was adjourned repeatedly through the autumn before being set for 17 November, and the objections filed in the bankruptcy court in White Plains read like a checklist of what can go wrong.

Springshot, a software company that ran operational tooling for Spirit, objected that the dataset could include its own intellectual property. Its filing cites a contract clause giving it rights to “any data or information generated by the Service or Software.” Chief executive Doug Kreuzkamp put the distinction plainly, according to an account published by The Next Web: “The distinction is between the information Spirit supplied and the proprietary outputs our platform generated.” Explaining why the company had not objected earlier, he told the Miami Herald, as quoted in the same account, “we had no reason to believe that Spirit would sell property that wasn't theirs.”

International Aero Engines, the engine manufacturer, filed its own objection arguing that its proprietary commercial, technical and financial information may have been copied into Spirit's internal communications, and that, in the words of its filing reported by ComplexDiscovery, “deidentification is insufficient to ensure that the IAE Proprietary Information has been removed.” Navitaire, Spirit's long-standing reservations platform provider, joined with Amadeus in a limited objection.

These objections expose the central weakness of the market: a company's systems hold far more than the company owns. Repositories routinely contain material supplied under non-disclosure agreements, vendor documentation pasted into tickets, customer configurations and code written under licence. In a working company, contractual confidentiality keeps that material in place. In a sale, it travels with everything else unless someone takes it out.

That matters to buyers, not just sellers. The EU's Trade Secrets Directive of 2016 makes the acquisition or use of a trade secret unlawful when the person “knew or ought, under the circumstances, to have known that the trade secret had been obtained directly or indirectly” from someone using or disclosing it unlawfully. A buyer who acquires a repository full of third-party confidential material without asking reasonable questions may not be able to claim innocence later.

There is now a striking precedent on the cost of careless acquisition. In Bartz v Anthropic, Judge William Alsup held in 2025 that training on lawfully acquired books could be fair use, but sent the claims over pirated library copies towards trial. Anthropic settled for $1.5 billion, covering roughly 480,000 works, and the settlement received final approval in July 2026, according to Jurist. The lesson for the code market is not that training is unlawful. It is that how training data was obtained can matter enormously, and that a court may look past the model to the provenance of the inputs.

The People Who Live Inside the Diff

Every commit in git carries an author name, an email address and a timestamp. Under Article 4(1) of the EU's General Data Protection Regulation, personal data means “any information relating to an identified or identifiable natural person”, and an identifiable person is one who can be identified “directly or indirectly”, including by a name or an online identifier. A git history is therefore, among other things, a structured personal data set: a record of when named individuals worked, what they worked on and, in review threads, what their colleagues thought of their work.

That last part deserves attention. Code review is one of the most socially charged activities in software engineering. A 2020 study by Carolyn Egelman, Emerson Murphy-Hill and colleagues at Google, presented at the International Conference on Software Engineering, surveyed 1,317 developers about negative experiences in review, which the authors called “pushback”. They found such experiences relatively rare but harmful when they occurred. A sold repository carries all of that: the blunt rejection, the apologetic reply, the thread in which someone was told they were wrong. These people wrote for colleagues, not for a model.

European regulators have started to address the training side. In Opinion 28/2024, adopted in December 2024, the European Data Protection Board said that whether an AI model can be considered anonymous must be assessed case by case, set out a three-step test for relying on legitimate interests, and indicated that unlawful processing at the development stage may affect the lawfulness of later deployment, with remedies that could in serious cases include deletion of the data or the model. The opinion did not contemplate a secondary market in staff conversations, but its logic applies.

American law offers less. Ifeoma Ajunwa, a law professor at Emory University and author of The Quantified Worker, told MIT Technology Review that “Employers should have express consent before selling workers' data,” noting that no federal law bans such sales. She described the proceeds, which typically flow to creditors and investors rather than to the workers whose words are sold, as “captured capital”. Greg Ewing, an attorney who advises AI startups, called such data “legally radioactive” and described “a Wild West of AI training”.

Sellers and brokers insist that personal information is scrubbed. SimpleClosure says it removes personal data before sale. The Spirit deal shows what a serious attempt at scrubbing looks like: according to chapter11cases.com, the parties adopted the California Consumer Privacy Act's de-identification standard, appointed Tonic.ai as an independent de-identification agent for unstructured data such as email, made certified de-identification a condition of closing, required re-identification testing before each transfer, and obliged the buyer to commit publicly not to attempt re-identification. Those protections originally covered customers rather than staff. After unions led by the Association of Flight Attendants objected, Skift reported in November that revised terms require Google, still paying $10 million, to cover the cost of de-identification as well and to commit publicly not to knowingly re-identify the data, while employee medical examinations, payroll records, drug testing, crew pairings, biometrics and union grievance filings are excluded from the sale altogether. “Without our union, there would have been no one to speak for workers in this fire sale of their private information,” said Sara Nelson, the union's president.

Even that may not be enough. In a letter dated 8 October 2026, more than a hundred members of Congress led by Senator Elizabeth Warren and Representative Steven Horsford warned that “Removing names, email addresses, or other direct identifiers does not necessarily make a dataset anonymous.” The point applies with special force to code. Writing style, working hours, the modules a person owned and the distinctive way they phrase a review comment can identify them as surely as a name. And a git history without its authorship is a much less useful training asset, which creates a quiet commercial pressure against thorough scrubbing.

Secrets That Travel With the Code

The security problem is more concrete still. Developers routinely commit things they should not: API keys, database passwords, cloud credentials, private certificates. GitGuardian's State of Secrets Sprawl 2026 report, published in March, counted about 28.65 million new hardcoded secrets added to public GitHub commits in 2025, a 34 per cent increase on the year before. Internal repositories, the kind now being sold, were roughly six times more likely than public ones to contain hardcoded secrets. More than 64 per cent of secrets that GitGuardian confirmed as valid in 2022 were still valid when retested in January 2026. And about 28 per cent of incidents originated entirely outside repositories, in collaboration tools such as Slack, Jira and Confluence, precisely the systems now sold alongside code.

Deleting a secret from the current version of a file does not remove it from git. It persists in history unless that history is rewritten. A buyer who values complete history is therefore buying, by design, every credential anyone ever committed and later regretted. Some will have been rotated. Many, on GitGuardian's evidence, will not.

The risk does not end at transfer. Language models can memorise and regurgitate their training data. In a 2021 paper presented at the USENIX Security Symposium, Nicholas Carlini and colleagues extracted hundreds of verbatim sequences from GPT-2's training data, including personally identifiable information and code, and found that larger models were more vulnerable. A credential that survives into a training corpus could, in principle, resurface in a model's output. A dead startup's cloud account may be closed; its partners' shared keys may not be.

Provenance Tools Built for a Different Problem

If the market is to function legitimately, buyers need something like a chain of title for code: evidence of who wrote what, under which agreement, and with whose permission it is being sold. Some of the tools exist. None was designed for this job.

The most rigorous work on dataset provenance comes from the Data Provenance Initiative, a collaboration led by researchers including Shayne Longpre at MIT. Its audit of more than 1,800 text datasets, published in Nature Machine Intelligence in 2024, found licence omission rates above 70 per cent and error rates above 50 per cent on popular dataset hosting sites. If carefully curated public datasets are that poorly documented, the documentation that comes with a hastily sold startup repository is unlikely to be better. Industry groups have tried to standardise the metadata: the Data & Trust Alliance, a consortium of large companies, published data provenance standards covering source, legal rights, lineage and intended use. Such standards help only if someone fills them in truthfully.

Git itself offers little assurance. Its documentation is candid: the author name “has no effect on authentication”, and options such as —author and —date allow anyone to set the claimed author and timestamp of a commit to whatever they like. A git history is a record of what someone typed into a configuration file, not a verified log of who did what and when.

Cryptographic signing can change that. Sigstore's gitsign tool lets developers sign commits using an OpenID Connect identity rather than long-lived keys, recording the signing event in Rekor, a public transparency log, so that the signature can later be checked with git verify-commit. The Supply-chain Levels for Software Artifacts framework, known as SLSA and part of the Open Source Security Foundation, describes how to generate verifiable provenance for build artefacts. Software bills of materials, which the US Cybersecurity and Infrastructure Security Agency describes as “a nested inventory, a list of ingredients that make up software components”, can document which third-party components a codebase contains.

The trouble is timing. These tools establish provenance going forward. They work when commits are signed at the moment of creation and builds are attested as they run. Almost none of the repositories now being sold were built that way. A five-year-old startup history consists overwhelmingly of unsigned commits, and no amount of tooling applied at the point of sale can retroactively prove that a commit dated 2021 was really written in 2021 by the person named on it. SBOMs describe components, not the employment status of the people who wrote the glue code. Even the media world's answer, the content credentials standardised by the Coalition for Content Provenance and Authenticity, attaches provenance to finished objects rather than to an evolving, many-authored record.

What Brussels Now Asks to See

Regulation is starting to demand some of this documentation. Article 53(1)(d) of the EU AI Act requires providers of general-purpose AI models to publish a summary of the content used to train them. In July 2025, the European Commission's AI Office published a mandatory template for that summary. According to the Commission's own guidance, it requires providers to list their data sources, including private datasets licensed from third parties, and to describe relevant processing aspects such as respect for copyright. The obligation has applied since 2 August 2025 for new models; models already on the market before that date have until 2 August 2027. The AI Office's enforcement powers began on 2 August 2026, with fines of up to 3 per cent of worldwide annual turnover or 15 million euros, whichever is higher.

That is real pressure: a lab that buys a startup's codebase must at least acknowledge publicly that it trained on licensed private data. But the template has limits. The Commission has said it will not audit content at the level of individual datasets, acting instead on complaints or qualified alerts from its scientific panel. And a summary of sources is not a chain of title. A provider can accurately state that it used “private datasets licensed from third parties” while knowing nothing about whether the third party had the right to license them.

Incentives That Reward Forgery

The most troubling feature of this market is not any single legal gap but the way money changes behaviour around them.

Begin with what buyers prize. Sunset values assets partly by headcount and years in operation; broker guidance stresses development history and pull requests; Turing wants the “complete operational history” of a company. Buyers want long, busy, human histories because those are what make a codebase useful for training agents. Those are exactly the properties that git makes trivially easy to fake. Dates can be backdated with an environment variable. Author names can be set at will. Language models can generate plausible commit messages, review comments and tickets in volume. A fabricated five-year history, populated with synthetic engineers who argue convincingly in pull requests, is now a technically modest undertaking.

There is evidence that metadata on code platforms is already gamed when it has value. A study by Hao He, Bogdan Vasilescu, Christian Kästner and colleagues at Carnegie Mellon University, North Carolina State University and the security firm Socket identified around 4.5 million suspected fake stars on GitHub, with fake-star campaigns increasing by two orders of magnitude in 2024, largely to promote short-lived malicious repositories. Stars are a crude signal. Commit histories are a far richer one, and they now carry a price tag. If buyers pay more for histories that look human and long, some sellers will manufacture them, producing exactly the synthetic contamination that real-world data was supposed to avoid. MIT Technology Review noted that no company has published tests comparing training on real archives with the alternatives, so buyers may struggle to tell the difference.

The second incentive is laundering. A repository containing third-party code, an ex-employer's proprietary modules or material taken under NDA becomes harder to trace once it passes through a broker, is scrubbed of identifying metadata, and is merged into a training corpus with thousands of others. De-identification, presented as a privacy safeguard, also strips away the authorship information that might reveal where code came from. No such case has yet been documented, but the structure of the market rewards anyone who can present tainted material as clean.

The third incentive is insider exfiltration. Developers routinely have access to code they do not own. Until recently, the main buyers for a stolen codebase were competitors and foreign rivals, and the cases that reached court were dramatic. In January 2026, a federal jury in San Francisco convicted Linwei Ding, a former Google software engineer, of seven counts of economic espionage and seven counts of theft of trade secrets, in a case in which prosecutors said he had stolen more than 2,000 pages of confidential material about Google's AI infrastructure. That was a case about geopolitics. The new market creates a far more mundane temptation: a departing engineer, a disgruntled contractor or a founder in a dispute with co-founders may now find a ready, apparently respectable buyer for a private repository.

That is not hypothetical. According to IBTimes UK, citing Business Insider, Surge AI's Nick Heiner said someone had approached him offering a large company's codebase for millions of dollars, and he concluded that the seller did not have the rights to sell it. A less careful buyer, or one under competitive pressure to acquire data faster than its rivals, might not have asked.

The final incentive concerns who gets paid. In the startup wind-downs described by MIT Technology Review, proceeds typically go to senior creditors and investors. The engineers whose judgement, arguments and late-night fixes make the data valuable receive nothing and are usually not asked. Ryan Lock, co-founder of Polyshares, compared the offers to those of early oil prospectors buying land from unaware farmers. People with no stake in a transaction also have no channel through which to object, so the safeguard most likely to catch a bad sale, the knowledge of those who were actually there, is switched off.

Building a Chain of Title for Code

None of this means the trade must stop. A failing company has legitimate reasons to recover value from its work, and AI developers have genuine reasons to want realistic engineering data. The question is what a legitimate transaction would look like. The Spirit proceedings, existing law and the tools described above suggest a working standard.

Sellers should map every author in the history to an employment contract, a contractor agreement with an assignment clause or a documented licence, and remove whatever cannot be matched. The labour of that mapping is the provenance. Former employees and contributors should be notified and given time to object; the Spirit case surfaced vendors, unions and lawmakers only because a bankruptcy court made the sale public, and private startup deals have no equivalent forum. Vendored code, NDA material and partner data should be found through software composition analysis and contract review, then stripped out or separately licensed.

De-identification should follow the Spirit model of an independent agent, certification and re-identification testing before transfer, and should address the stylistic signals that identify developers even without their names. Every commit and every attached collaboration archive should be scanned for credentials, with live secrets revoked rather than merely redacted. Buyers should demand warranties of title backed by indemnities and escrow, so that the cost of a bad sale falls on whoever was best placed to prevent it. And companies that want their engineering record to be worth something later should sign commits and attest builds now, which would also make fabricated histories easier to detect.

The Record Belongs to More Than One Person

Return to that email from Google: “You keep 100% of your IP.” For a solo developer who wrote every line of an app, that may be close to true. For almost everyone else, it is a simplification that the new market quietly depends on. A codebase with a real history is a crowd of authors, a web of contracts, a store of other companies' secrets and a diary of working lives. It can be sold as a single asset only by ignoring most of what makes it valuable.

The AI industry has discovered that the most useful training data is not polished output but the visible process of people working things out together. That process was never created with a buyer in mind. If the market is allowed to treat possession as ownership, and a git log as proof of anything, it will reward forgers, launderers and insiders. At least one buyer has already recognised an offer that looked too good to be legitimate. A market cannot rely on its buyers happening to be careful.


References and Sources

  1. Koebler, Jason. “Google Is Quietly Buying Code From Play Store Developers to Train AI.” 404 Media, 2 June 2026. https://404media.co/google-is-quietly-buying-code-from-play-store-developers-to-train-ai/
  2. King, Matthew. “AI labs are buying data from dead startups to train their models.” MIT Technology Review, 13 October 2026. https://www.technologyreview.com/2026/10/13/1145755/ai-labs-buying-data-from-dead-startups-to-train-models/
  3. Perrigo, Billy. “Google's $10 Million Bid for Spirit Airlines' Data Reveals AI's Next Frontier.” TIME, 25 August 2026. https://time.com/article/2026/08/25/google-spirit-airlines-ai-data-RL/
  4. Chapter11Cases. “Spirit Airlines Case Update: The Data Sale Slips to November as Spirit Moves to Unwind the HFS Aircraft Deal.” 14 October 2026. https://chapter11cases.com/blogs/news/spirit-airlines-case-update-the-data-sale-slips-to-november-as-spirit-moves-to-unwind-the-hfs-aircraft-deal
  5. ComplexDiscovery. “Lawmakers press Google and Spirit on employee data before a hearing on a $10 million AI training data sale.” 9 October 2026. https://complexdiscovery.com/lawmakers-press-google-and-spirit-on-employee-data-before-a-hearing-on-a-10-million-ai-training-data-sale/
  6. Adebayo, Kolawole Samuel. “Springshot says Google's $10m Spirit deal could include its trade secrets.” The Next Web, 18 September 2026. https://thenextweb.com/news/springshot-objection-spirit-airlines-data-google
  7. Jaen, Regyl. “Companies Are Selling Employees' Private Slack Messages and Emails to Train AI, Often Without Workers' Knowledge.” IBTimes UK, 20 October 2026 (reporting remarks given to Business Insider). https://www.ibtimes.co.uk/companies-sell-workplace-data-ai-firms-1826648
  8. Holland, Martin. “Lucrative internal data: AI firms buy internal data from liquidated companies.” heise online, 17 April 2026 (summarising Forbes reporting of 16 April 2026). https://heise.de/-11262202
  9. Turing. “Can an AI Learn From a Company That No Longer Exists?” AGI Advance newsletter, 24 December 2025. https://www.turing.com/blog/agi-advance-newsletter-44
  10. Human Union Data, Inc. “Best Platforms for Selling Your Codebase in 2026.” HUD, 24 June 2026 (broker marketing material). https://www.hud.ai/guides/best-platforms-selling-codebase
  11. Villalobos, Pablo; Ho, Anson; Sevilla, Jaime; Besiroglu, Tamay; Heim, Lennart; Hobbhahn, Marius. “Will we run out of data? Limits of LLM scaling based on human-generated data.” Epoch AI, 2024. https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
  12. Shumailov, Ilia; Shumaylov, Zakhar; Zhao, Yiren; Papernot, Nicolas; Anderson, Ross; Gal, Yarin. “AI models collapse when trained on recursively generated data.” Nature, 631, 755-759, 24 July 2024. DOI: 10.1038/s41586-024-07566-y.
  13. UK Parliament. Copyright, Designs and Patents Act 1988, section 11. https://www.legislation.gov.uk/ukpga/1988/48/section/11
  14. United States Code. 17 U.S.C. section 101, definition of “work made for hire.” Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/uscode/text/17/101
  15. Hamilton, Corey. “Software governance in the AI era: Key findings from the 2026 OSSRA report.” Black Duck, 25 February 2026. https://www.blackduck.com/blog/open-source-trends-ossra-report.html
  16. Linux Foundation. Developer Certificate of Origin, Version 1.1. https://developercertificate.org/
  17. European Parliament and Council. Directive (EU) 2016/943 on the protection of undisclosed know-how and business information (trade secrets), Article 4. https://www.legislation.gov.uk/eudr/2016/943/article/4
  18. Jurist. “Judge approves record $1.5 billion AI copyright settlement involving Anthropic.” July 2026. https://www.jurist.org/news/2026/07/judge-approves-record-1-5-billion-settlement-involving-anthropic/
  19. European Parliament and Council. Regulation (EU) 2016/679 (General Data Protection Regulation), Article 4. https://gdpr-info.eu/art-4-gdpr/
  20. Egelman, Carolyn D.; Murphy-Hill, Emerson; Kammer, Elizabeth; Hodges, Margaret Morrow; Green, Collin; Jaspan, Ciera; Lin, James. “Predicting Developers' Negative Feelings about Code Review.” Proceedings of the 42nd International Conference on Software Engineering (ICSE 2020), 174-185. DOI: 10.1145/3377811.3380414. https://research.google/pubs/predicting-developers-negative-feelings-about-code-review/
  21. European Data Protection Board. Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, adopted December 2024. Summary: CMS. https://cms.law/en/int/legal-updates/edpb-opinion-28-2024-key-takeaways-on-processing-personal-data-in-the-context-of-ai-models
  22. GitGuardian. “The State of Secrets Sprawl 2026.” 17 March 2026. https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/
  23. Carlini, Nicholas; Tramèr, Florian; Wallace, Eric; Jagielski, Matthew; Herbert-Voss, Ariel; Lee, Katherine; Roberts, Adam; Brown, Tom; Song, Dawn; Erlingsson, Úlfar; Oprea, Alina; Raffel, Colin. “Extracting Training Data from Large Language Models.” 30th USENIX Security Symposium, 2021, 2633-2650. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
  24. Longpre, Shayne, et al. “A large-scale audit of dataset licensing and attribution in AI.” Nature Machine Intelligence, 6, 975-987, 2024. MIT News coverage: https://www.media.mit.edu/articles/study-transparency-is-often-lacking-in-datasets-used-to-train-large-language-models/
  25. Help Net Security. “Cross-industry standards for data provenance in AI” (interview with the Data & Trust Alliance). 22 July 2024. https://www.helpnetsecurity.com/2024/07/22/saira-jesani-data-trust-alliance-data-provenance-standards/
  26. Git project. “git-commit Documentation.” https://git-scm.com/docs/git-commit
  27. Sigstore. “Gitsign” documentation. https://docs.sigstore.dev/cosign/signing/gitsign
  28. SLSA (Supply-chain Levels for Software Artifacts), Open Source Security Foundation. https://slsa.dev/
  29. Cybersecurity and Infrastructure Security Agency. “Software Bill of Materials (SBOM).” https://www.cisa.gov/sbom
  30. Coalition for Content Provenance and Authenticity. “About C2PA.” https://c2pa.org/about/
  31. European Commission. “Template for General-Purpose AI model providers to summarise their training content” (FAQ). https://digital-strategy.ec.europa.eu/en/faqs/template-general-purpose-ai-model-providers-summarise-their-training-content
  32. He, Hao; Yang, Haoqin; Burckhardt, Philipp; Kapravelos, Alexandros; Vasilescu, Bogdan; Kästner, Christian. “4.5 Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Scams, and Malware.” arXiv preprint 2412.13459, December 2024. https://arxiv.org/abs/2412.13459
  33. CBS News Bay Area. “Former Google engineer Linwei Ding found guilty of stealing AI secrets for Chinese firms.” 30 January 2026. https://www.cbsnews.com/sanfrancisco/news/former-google-engineer-linwei-ding-found-guilty-stealing-ai-secrets-chinese-firms/
  34. GitHub. “Updates to our Privacy Statement and Terms of Service: How we use your data.” GitHub Changelog, 25 March 2026. https://github.blog/changelog/2026-03-25-updates-to-our-privacy-statement-and-terms-of-service-how-we-use-your-data/
  35. Maharishi, Meghna. “Spirit's $10 Million Data Sale to Google Clears Hurdle as Unions Win Privacy Protections.” Skift, 10 November 2026. https://skift.com/2026/11/10/spirits-10-million-dollar-data-sale-clears-hurdle-as-unions-win-privacy-protections

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Before GPT-4o offered a single word of guidance, it had already decided the woman was poorer.

That is the quiet finding buried in the tables of a benchmark posted to arXiv on 28 October 2026. Its author, Kriti Mahajan, asked OpenAI's model for financial advice on behalf of synthetic Indian personas, and had it estimate, alongside that advice, what each persona owned and earned before and after following it. The request itself never changed. Only the labels did: gender, caste, education, employment, age, whether the person lived in a city or a village, and in one large set of prompts, which Indian district they called home. Nobody told the model how much money anyone had. It filled that in for itself.

Across the gender-only prompts, profiles labelled male were assigned initial total resources (the sum of wealth, income and savings) 60.1 per cent higher than otherwise comparable female profiles. Then came the advice. After it, the model projected that the men would still hold 46.8 per cent more than the women. In the prompts that also carried a caste label, the post-advice gap between men and women was 49.4 per cent. The paper's regressions control for age, employment, education, residence and caste or gender, and the male coefficient stays positive and statistically significant in both the gender-only and the caste-augmented specifications.

It would be easy to dismiss this as one model, one country, one researcher. But it arrived in a year when economists at MIT Sloan and Stanford reached an uncomfortably similar conclusion about American users, when the UK's financial regulator found many young investors think chatbot output is regulated, and when Vanguard warned that general-purpose AI tools sit outside the rules that bind human advisers. What happens when advice that is slightly worse for some people, depending on who they are, is followed for forty years? And who, exactly, is responsible for it?

A Poorer Woman Before the First Word of Advice

The Indian Financial Automated Advice Bias benchmark, or IFAAB, is a counterfactual audit: hold everything constant except the attribute under test, and differences in output can be attributed to that attribute. The dataset contains 31,734 GPT-4o responses, split into three experiments: 1,404 gender-only prompts, 7,020 gender-by-caste prompts and 23,310 gender-by-district prompts.

What the model produced for each persona was a block of advice plus six estimated figures: wealth, income and savings before the advice, and the same three after it. There is no ground truth here: every number is the model's own projection. That limitation is also the point. The audit is not measuring what Indian women own. It is measuring what GPT-4o assumes they own, and what it believes its own advice will do for them.

In the gender-only experiment, the average male profile was assigned initial resources of ₹264,700 against ₹165,400 for the average female profile. After advice, those figures rose to ₹386,900 and ₹263,600. One subtlety cuts against the simplest version of the story. In percentage terms the women's projected resources grew faster, by roughly 59 per cent against roughly 46 per cent for men, which is why the relative gap narrows from 60.1 to 46.8 per cent. In rupees, though, the gap widens, from ₹99,300 before advice to ₹123,300 after it. The model is not simply making women poorer. It starts them poorer, gives them advice it thinks will help, and still projects them ending up further behind in absolute terms.

The texture of the advice helps explain why. Mahajan classified the topics the model raised and found that men more often received guidance on investing, formal employment, networking and entrepreneurship, while women more often received guidance on saving, self-help groups, home-based work and basic literacy. The paper's own paired examples are blunt. A male profile is told: “Start a small monthly SIP in a balanced mutual fund.” A comparable female profile is told: “Consider investing in a recurring deposit or post office savings scheme.” Another female profile is advised to “Save a small, consistent amount each month, even if it is just Rs. 100.”

None of these suggestions is reckless, which is what makes the pattern so hard to see. No single answer looks discriminatory. The bias lives in the distribution: in who gets steered towards growth assets and who gets steered towards safety, across thousands of conversations.

A Caste Gradient in the Machine

The caste experiment adds a second axis. Relative to profiles labelled General category, the model's projected post-advice total resources were lower by ₹72,600 for Other Backward Classes profiles, ₹114,000 for Scheduled Caste profiles and ₹113,100 for Scheduled Tribe profiles, all statistically significant. Profiles with no caste specified were indistinguishable from General ones. The projected gains from the advice itself were also smaller for OBC, SC and ST profiles.

A model that merely mirrored existing inequality in its starting estimates would be troubling enough. A model whose advice is projected to deliver smaller gains to people already assigned less is doing something closer to amplification. Mahajan's abstract makes the general point directly, warning that LLM-based financial agents can reproduce and reinforce existing social inequalities.

The district experiment complicates matters in an instructive way. When the prompts included the persona's district, the aggregate gender gap shrank dramatically: the initial resource gap fell to 2.5 per cent and the post-advice gap to 7.9 per cent. Mahajan reads this as evidence that, without geographic context, the model leans on gender as a proxy for financial capacity. Give it a richer description and it relies less on the crude signal. But the gap did not vanish, men still received higher projected post-advice income, and the paper notes that only 30 prompts sat behind each district, so local estimates are noisy.

The study has real limits, and its author lists them. Only GPT-4o was tested, the gender-by-caste personas were all fixed at age 25 for budget reasons, and the paper warns that its results should not be read as universal properties of LLM financial advice. Mahajan notes that the work was done independently and self-funded. It is a preprint, not yet peer reviewed.

Still, its direction matches a growing body of work. In 2024, Alejandro Salinas, Amit Haim and Julian Nyarko of Stanford audited leading language models, GPT-4 among them, by asking for advice about named individuals across 42 prompt templates. They found the advice “systematically disadvantages names that are commonly associated with racial minorities and women,” and that “names associated with Black women receive the least advantageous outcomes.” A 2026 Centre for Economic Policy Research discussion paper by Richard Foltyn and Jonna Olsson, cited in the study below, reported an average gender gap of 1.8 percentage points in recommended equity allocations across 33 language models.

Prompts Written by Real People

The most rigorous attempt to measure what this kind of bias costs over a lifetime comes from Taha Choukhmane, an assistant professor of finance at MIT Sloan, Tim de Silva of Stanford Graduate School of Business, and two MIT Sloan colleagues, Weidong Lin and Matthew Akuzawa. Their paper, “AI Financial Advice: Supply, Demand, and Life Cycle Implications,” appeared as NBER Working Paper 35574 in August 2026 and on arXiv the same month. It won a Swiss Finance Institute Outstanding Paper Award.

Their innovation was to stop writing prompts themselves. They recruited a demographically balanced sample of 1,000 US adults through the research platform Prolific and asked each to write three prompts: a description of their financial situation, a request for advice on spending versus saving, and a request for investment advice. Those real, messy, human questions were fed to large language models, principally GPT-5.2, and the advice was then run through a life-cycle simulation in which virtual people aged 22 onwards followed it year after year, through job losses, market slumps and death.

The headline result is genuinely good news. Following the chatbot's advice would move most people closer to what economists consider sound behaviour: broader participation in diversified equity funds, equity shares that decline with age, larger savings buffers. “We were somewhat surprised by how good the advice was,” Choukhmane told MIT Sloan's Ideas Made to Matter. De Silva, speaking to Stanford GSB Insights, put it this way: “AI seems to be nudging people in the right direction.”

The bad news is in the heterogeneity. The advice was not equally good for everyone, and the differences followed familiar lines. Prompts written by women generated recommendations that left simulated wealth at age 60 about $60,000, or 4.1 per cent, lower on average than prompts written by men, in 2025 dollars. Prompts from people who answered at least one basic financial literacy question incorrectly produced wealth about $46,000, or 4.1 per cent, lower. Prompts from people who had never used AI for financial advice produced wealth about $100,000, or 5.7 per cent, lower than prompts from experienced users.

This is where the frequently quoted “4 to 6 per cent” comes from, but those are three separate comparisons between different pairs of groups, not a single range. The gender and literacy gaps were driven mainly by portfolio choice: women were recommended an average equity share between ages 22 and 60 that was 2.94 percentage points lower than men's. The AI-experience gap was driven almost entirely by saving advice instead. Different mechanisms, similar magnitudes, all compounding in the same direction.

“Regular people are not writing their prompts the way a finance professor is,” Choukhmane said. His colleague was blunter: “The way you write the questions matters a ton.”

Separating What You Ask From Who You Are

If the story ended there, it would be about prompting skill, and the remedy would be education. But a second experiment changes the moral.

They took prompts from respondents that did not mention gender and randomly attached a label, such as “I am a man” or “I am a woman”, to the front. That allowed them to split any gender difference into two parts. The demand side captures differences in what men and women actually write. The supply side captures what the model does when the identical prompt arrives under a different label.

For the recommended diversified equity share, moving from a prompt written by a man and labelled male to one written by a woman and labelled female reduced the recommendation by 1.50 percentage points in a given period. About two-thirds of that, 0.96 points, was demand-driven. The remaining third, 0.54 points, was supply-driven. Holding every word of the question fixed, telling the model the asker was a woman made it recommend less in equities.

Two caveats are essential. First, the one-third figure applies to the per-period equity recommendation, not to the cumulative $60,000 wealth gap. The authors say explicitly that these coefficients “do not cumulate advice differences over the life cycle”. It is reasonable to infer that some meaningful share of the lifetime gender gap is supply-driven, but the paper does not state that one-third of the wealth gap is. Second, the race results looked different. Differences between Black and White respondents, and between Asian and White respondents, were smaller and almost entirely demand-driven, with a supply effect close to zero for race.

The authors offer two readings of the gender label effect. The model may be using gender as a statistical proxy for things that correlate with it, such as risk preferences or labour-market risk. Or it may be reproducing stereotypes from its training data. Even under the charitable reading, they write, “gender is a coarse signal, and this type of inference would ideally be made explicit to users.” On race, they suggest model providers may police race more stringently than gender during alignment, but warn that an explicit statement such as “I am Black” is exactly the signal such safeguards are built to catch. Names, dialect and context are subtler, and the Stanford names study suggests that is where trouble lives.

The supply-demand distinction is the hinge of this whole debate. Demand-side differences can be framed, with some justice, as the consequence of what people choose to ask. Supply-side differences cannot. They are a property of the product. When a human adviser gives a woman more conservative advice than a man with an identical file, that is the sort of thing a regulator, an ombudsman or a court can examine. When a chatbot does it, it is not obvious that anyone has jurisdiction to look.

The Arithmetic of a Small Tilt

Why should a gap of a few percentage points matter? Because compounding converts small persistent differences into large terminal ones.

A simple illustration, using assumptions chosen for clarity rather than drawn from any study, shows the mechanism. Suppose a portfolio earns 5 per cent a year, and suppose that holding about three percentage points less in equities, roughly the lifetime gender difference the MIT and Stanford researchers found, costs about 0.15 percentage points of annual expected return. That is what you would get if equities outperformed safer assets by 5 points a year. A pound invested at 22 and left alone until 60 ends up about 5.3 per cent smaller under the cautious tilt. A steady stream of equal annual contributions ends up about 3.3 per cent smaller. After ten years the lump-sum cost is barely visible, around 1.4 per cent. It is the decades that do the damage.

Those illustrative figures sit, reassuringly or alarmingly, in the same neighbourhood as the 4.1 per cent the full simulation produced for women. The simulation also holds income, shocks and market returns fixed across groups, so it isolates the effect of the advice alone. Whatever other disadvantages a particular user already carries into retirement, the advice gap is added on top of them, not subtracted from them.

Apply the same lens to the Indian evidence. If a model begins by assuming a woman, or a Scheduled Caste household, has less, and tailors its advice to that assumption by suggesting smaller sums and safer products, then the advice becomes a self-fulfilling forecast. Each recommendation shapes the next year's balance. Each balance shapes the next question. A user who returns to the same chatbot every year, as the MIT and Stanford simulation assumes, is not getting one biased answer. She is getting a biased trajectory.

There is a further compounding effect that no simulation captures: who uses these tools in the first place. “A lot of the people who would benefit from financial advice are precisely the people who don't have a lot of resources,” Choukhmane told MIT Sloan. Stanford's write-up of the same paper noted that around 40 per cent of the human prompt writers had less than $10,000 in savings. These are the people least likely to pay a human adviser, most likely to rely on a free chatbot, and least able to absorb a slow drain on returns. The demographic discount is applied hardest to those with the least room for error.

Counting the People Who Lost Money

How much harm is already visible? The evidence is thinner than the headlines suggest, and some of it has been counted twice.

The best-documented figures come from NerdWallet. In an online survey conducted by The Harris Poll on 23 and 24 June 2026 among 2,003 US adults, 26 per cent said they had used an AI chatbot to find answers to personal finance questions. Among those who had acted on a chatbot's advice, 29 per cent said it hurt their financial situation, 39 per cent said it helped and 32 per cent said it made no difference. The harm figure rests on a smaller base of 496 chatbot users, is self-reported, and cannot say whether the advice, the market or the user was to blame.

Vanguard's research report “The AI advice frontier: Use, trust, and the human edge”, dated September 2026 and written by Andy Reed, Nathan Young, Xiao Xu, Samantha Ostrom and Fiona Greig, is often quoted as finding that three in ten people who act on AI advice lose money. The report does contain that line. But its own survey, fielded in June 2026 among 6,686 Vanguard personal investors, is not its source. Vanguard attributes it to the NerdWallet study, by Erin El Issa. They are one finding, reported twice, not two independent confirmations.

Vanguard's own survey is more revealing. About 32 per cent of its investors had used AI for some form of financial guidance, rising to 43 per cent among Gen Z and millennial investors. Many did not understand the tools' legal status. Asked whether general-purpose AI has a legal obligation to act in their best financial interest, according to the report's chart, 34 per cent of Gen Z and millennial users, 20 per cent of Gen X users and 13 per cent of baby boomers and older users said they were not sure, and a further small share, between 3 and 4 per cent in each group, believed it did.

The UK picture is starker. On 27 August 2026 the Financial Conduct Authority published research from a survey of 666 UK adults aged 18 to 40 who held or were considering investments, run through the Attest platform on 24 July. Four in five less experienced investors had used AI to help with investing. Fifty-six per cent trusted AI tools. And 44 per cent wrongly believed AI-generated financial information was regulated. Thirty-eight per cent thought it acceptable to base an investment decision solely on AI output, and 32 per cent wrongly believed they would be compensated by the Financial Services Compensation Scheme or the Financial Ombudsman Service if AI advice went wrong.

Read alongside the bias research, these numbers describe an unusual situation: a substantial minority of users believe they are protected, and they are not.

What a Human Adviser Owes You

In the UK, giving a personal recommendation on investments is a regulated activity. Firms that do it must be authorised by the FCA and must assess suitability under the FCA's conduct of business rules, gathering enough information about a client's circumstances, objectives and capacity for loss to justify the recommendation. Since 31 July 2023 they have also been bound by the Consumer Duty, which requires firms to act to deliver good outcomes for retail customers, to act in good faith, to avoid causing foreseeable harm and to enable and support customers in pursuing their financial objectives. Crucially for this debate, the Duty's monitoring rules require firms to consider whether any group of retail customers is experiencing different outcomes. A UK firm whose advice tool systematically recommended lower equity exposure to women on identical facts would, at least in principle, have a problem it was obliged to find.

The UK has also just widened the space between full advice and generic guidance. Under the FCA's targeted support regime, set out in policy statement PS25/22 and in force since 6 April 2026, authorised firms can offer suggestions designed for groups of consumers with shared characteristics, for pensions and investments, without conducting a full individual suitability assessment. Targeted support is lighter than advice, but it is still a regulated activity, with permission, conduct rules and the Consumer Duty attached.

In the United States, the Investment Advisers Act of 1940 imposes a federal fiduciary duty on registered investment advisers. In June 2019 the Securities and Exchange Commission published an interpretation clarifying the fiduciary duty an investment adviser owes its clients. On the same day it adopted Regulation Best Interest, which requires broker-dealers recommending securities to retail customers to act in the customer's best interest and not to place their own interests ahead of the customer's, with compliance required from 30 June 2020. Critics, including Commissioner Robert Jackson in dissent, argued that Reg BI fell short of a true fiduciary standard. But it is enforceable, and it attaches to a person.

In India, the Securities and Exchange Board of India's Investment Advisers Regulations, 2013 require anyone acting or holding themselves out as an investment adviser to register. SEBI amended them in December 2024, adding a duty for registered advisers using AI tools to disclose the extent of that use to clients and making the adviser solely responsible for the security, confidentiality and accuracy of client data processed through such tools. In August 2024, SEBI also amended its rules to bar regulated entities from associating with unregistered individuals who offer investment advice or make performance claims, a measure aimed squarely at so-called finfluencers.

Across three legal systems, the principle is the same. When a person or firm tells you what to do with your money, they owe you something: suitability, care, loyalty, disclosure, a route to complain and, often, a compensation scheme behind them.

Why the Chatbot Owes You Nothing

General-purpose chatbots fall outside almost all of this, and not by accident. Each regime has a perimeter, and each perimeter was drawn for a world in which advice was either personal and paid for, or published and impersonal.

The FCA has said so directly. Its August 2026 release states that “General purpose AI chatbots are not regulated,” while noting that tools specifically set up to provide financial advice would be likely to fall within its remit. The distinction is coherent from a regulator's point of view: a general-purpose assistant is not set up as an advice service. But it means little to a 25-year-old who types a question about her pension into the same app she uses for everything else, receives a tailored, confident answer and follows it. Lucy Castledine, the FCA's director of consumer investments, framed the regulator's message to consumers as a matter of self-reliance: “you need to understand how you're protected and continue to use your own judgement.”

The US perimeter has its own famous gap. In Lowe v SEC in 1985, the Supreme Court held that the Investment Advisers Act excludes “the publisher of any bona fide newspaper, news magazine or business or financial publication of general and regular circulation,” and read the Act's legislative history as aimed at personalised advice rather than impersonal publishing. A chatbot generating output tailored to an individual's circumstances is an odd fit for a carve-out designed for newsletters, but how it fits the Act remains an open question. Meanwhile the SEC's 2023 proposal to govern conflicts of interest arising from predictive data analytics, which would have applied to advisers and brokers using such tools, was formally withdrawn on 12 June 2025 along with 13 other pending proposals. It would not have reached unregistered chatbot providers in any case.

India's definition of an investment adviser turns on providing advice for consideration, and the regulations exclude advice given through newspapers, magazines or “any electronic or broadcasting or telecommunications medium, which is widely available to the public.” A free chatbot available to anyone with a phone sits uncomfortably close to that exclusion. The December 2024 amendments address AI only when a registered adviser uses it. They do not touch the AI provider whose tool a household consults directly.

Vanguard's report puts the asymmetry crisply. Investment advisers “have long been strictly regulated, are generally held to a fiduciary standard, and are accountable to their clients,” it says, while “general-purpose AI tools have not been subject to any of the same regulations,” even when they offer explicit investment advice. The consequence is that the very finding at the heart of this story, a supply-side bias in which the model treats identical requests differently according to gender, falls into a space no conduct regulator currently supervises. A human adviser who did it would face suitability rules, outcome monitoring and complaints. The chatbot faces, at most, general consumer and equality law, which was not drafted with this kind of harm in mind.

Closing the Perimeter Gap

Regulators are not blind to this. The FCA's Mills Review, led by executive director Sheldon Mills and published on 6 July 2026, concluded that retail finance is moving towards services that are “AI-enabled, continuous and delegated.” Its first recommendation to the FCA Board was to “Secure and adapt the regulatory perimeter.” Its seventh was to develop a trusted public-interest AI-enabled financial capability service. Law firm commentators at Freeths noted that it did not recommend a standalone AI regulatory regime for retail financial services. The direction of travel is visible. The destination is not. Several options follow from the evidence.

The first is functional regulation. Vanguard argues for “applying appropriate investor-protection standards to functionally equivalent activities, regardless of whether guidance is provided by a human advisor or an AI system.” If a system takes a person's age, income and goals and produces a specific portfolio recommendation, it is doing what an adviser does. The difficulty lies in drawing the line without sweeping in every search engine and explanatory answer, or pushing general-purpose providers simply to refuse all money questions, which would leave the people Choukhmane worries about with nothing.

The second is mandatory bias testing. The IFAAB benchmark and the MIT and Stanford label-swap method both show that counterfactual audits are cheap, repeatable and revealing. Swap a gender or caste label, hold everything else constant, measure the difference. Requiring providers whose models answer large volumes of financial questions to run and publish such audits, in the way the Consumer Duty already expects authorised firms to monitor group outcomes, would turn an invisible bias into a reportable metric. It would also make the supply-side share, the part of the gap that is unambiguously the model's doing, something a regulator could track over time.

The third is disclosure at the point of use. The MIT and Stanford authors argue that if a model is inferring something from gender, that inference “would ideally be made explicit to users.” The FCA and Vanguard findings suggest many users also need to be told, at the moment they ask, that the answer carries no regulatory protection and no compensation. Vanguard's own report cautions that education “is necessary but not sufficient”, adding that “Investors shouldn't have to bear the burden of discerning high-quality versus high-accessibility advice.”

The fourth is to build a better public option. The Mills Review's idea of a public-interest financial capability service, and the UK's new targeted support regime, point towards a world in which accessible, AI-assisted guidance exists inside the perimeter, with the duties attached, rather than outside it. If that option is good enough and easy enough to reach, it competes directly with the unregulated chatbot for the users who most need help.

None of these is a complete answer. But the evidence now justifies at least one modest step everywhere: treating systematic differences in AI-generated advice by gender, caste or other protected characteristics as a conduct problem rather than a curiosity.

The Advice Gap That Advice Creates

So, is the most consequential wealth gap of the AI era the one created by the advice itself?

The evidence does not quite support that strong claim. The MIT and Stanford study found that, for most people, following chatbot advice would improve on what they already do. Lower-income households with little savings may gain more from an imperfect free adviser than they lose to its biases. Two-thirds of the measured gender gap in equity advice came from what people asked, not from the label the model saw. Pay, inheritance and housing shape wealth gaps too, and the paper measures only one channel among many. And the IFAAB results come from one model, as a preprint, with no ground truth beneath the numbers.

But the weaker version of the claim is strong enough to worry about. The audits show that leading models, given identical requests, tilt their advice according to gender, and in GPT-4o's case in India according to caste, in directions that line up with existing disadvantage. The simulation shows that tilt, compounded across a working life, can be worth tens of thousands of dollars. The surveys show that a quarter of Americans have already asked a chatbot about their money, that many cannot tell good guidance from bad, and that a large share of young UK investors believe they are protected when they are not. And the legal analysis shows that the supply-side portion of the bias, the part that is plainly the product's own doing, sits in the one place that human advisers' fiduciary, suitability and outcome-monitoring duties do not reach.

What makes this gap distinctive is not its size. It is its invisibility. A woman who gets a more cautious answer than a man would have got will never see the man's answer. A woman told to save Rs. 100 a month never sees the monthly mutual fund plan that, in the Indian audit, went to men. No single conversation looks wrong, so there is nothing obvious to complain about. The discount is applied silently, in small amounts, every time the question is asked, and it compounds.

Human advisers were bound by duties because societies decided that people telling others what to do with their savings could not be left to self-interest and good intentions. The machines now doing some of that telling were never brought inside those duties. Until they are, the most reliable protection against the demographic discount is the one the FCA recommends, and the one its own research suggests many people are not using: your own judgement, applied to advice you have no way of knowing was different for someone else.


References and Sources

  1. Mahajan, Kriti. “The Indian Financial Automated Advice Bias (IFAAB) Dataset: Auditing Socioeconomic and Demographic Disparities in LLM-Generated Financial Advice in India.” arXiv preprint 2610.38239v1, submitted 28 October 2026. https://arxiv.org/abs/2610.38239v1 (full text: https://arxiv.org/html/2610.38239v1)
  2. Choukhmane, Taha (MIT Sloan School of Management and NBER); de Silva, Tim (Stanford Graduate School of Business); Lin, Weidong (MIT Sloan); Akuzawa, Matthew (MIT Sloan). “AI Financial Advice: Supply, Demand, and Life Cycle Implications.” NBER Working Paper 35574, August 2026. DOI: 10.3386/w35574. https://www.nber.org/papers/w35574
  3. Vereckey, Betsy. “AI financial advice is surprisingly good, especially if you ask the right questions.” MIT Sloan School of Management, Ideas Made to Matter, 21 July 2026. https://mitsloan.mit.edu/ideas-made-to-matter/ai-financial-advice-surprisingly-good-especially-if-you-ask-right-questions
  4. Blumberg, Deborah Lynn. “What AI Tells People Seeking Low-Cost Financial Advice.” Stanford Graduate School of Business, Insights, 13 July 2026. https://www.gsb.stanford.edu/insights/what-ai-tells-people-seeking-low-cost-financial-advice
  5. MIT Sloan School of Management. “CFI Paper Highlight: AI Financial Advice, Supply, Demand, and Life Cycle Implications.” 2026. https://mitsloan.mit.edu/centers-initiatives/cfi/ai-financial-advice-supply-demand-and-life-cycle-implications
  6. Reed, Andy; Young, Nathan; Xu, Xiao; Ostrom, Samantha; Greig, Fiona. “The AI advice frontier: Use, trust, and the human edge.” Vanguard Research, September 2026 (Vanguard AI & Advice Survey, June 2026, n=6,686). https://corporate.vanguard.com/content/dam/corp/research/pdf/the_ai_advice_frontier.pdf
  7. El Issa, Erin. “Data: Americans Are Using Chatbots for Financial Advice. Risky or Rewarding?” NerdWallet, 2026 (survey conducted by The Harris Poll, 23 to 24 June 2026, n=2,003 US adults). https://www.nerdwallet.com/finance/studies/using-ai-for-personal-finances
  8. Financial Conduct Authority. “Young investors trust AI more than TV or celebrities.” Press release, 27 August 2026. https://www.fca.org.uk/news/press-releases/young-investors-trust-ai-more-tv-or-celebrities
  9. Financial Conduct Authority. “AI and the future of retail financial services (The Mills Review).” 6 July 2026. https://www.fca.org.uk/publications/corporate-documents/mills-review
  10. Financial Conduct Authority. FCA Handbook, PRIN 2A (Consumer Duty), including PRIN 2A.9 (monitoring of consumer outcomes). https://handbook.fca.org.uk/handbook/PRIN/2A/
  11. Financial Conduct Authority. “PS25/22: Supporting consumers' pensions and investment decisions: rules for targeted support.” 2025. https://www.fca.org.uk/publications/policy-statements/ps25-22-supporting-consumers-pensions-investment-decisions-rules-targeted-support
  12. Penningtons Manches Cooper. “Rethinking investor engagement: targeted support and its regulatory implications.” April 2026. https://www.penningtonslaw.com/insights/rethinking-investor-engagement-targeted-support-and-its-regulatory-implications/
  13. Sidley Austin LLP. “SEC Adopts Regulation Best Interest and Form CRS; Issues Investment Advisers Act Interpretations.” 13 June 2019. Archived copy: https://web.archive.org/web/20200805080948/https://www.sidley.com/en/insights/newsupdates/2019/06/sec-adopts-best-interest-standard-of-care-for-broker-dealers-clarifies-fiduciary-duty
  14. Cleary Gottlieb Steen & Hamilton. “SEC Adopts Best Interest Standard for Broker-Dealers and Fiduciary Duty Guidance for Investment Advisers.” June 2019. https://clearygottlieb.com/news-and-insights/publication-listing/sec-adopts-best-interest-standard-for-broker-dealers-and-fiduciary-duty-guidance
  15. US Securities and Exchange Commission. “Conflicts of Interest Associated with the Use of Predictive Data Analytics by Broker-Dealers and Investment Advisers” (File No. S7-12-23), proposal and withdrawal notice. https://www.sec.gov/rules/2023/07/s7-12-23
  16. Paul Hastings LLP. “SEC Withdraws 14 Pending Rule Proposals.” June 2025. https://www.paulhastings.com/insights/client-alerts/sec-withdraws-14-pending-rule-proposals
  17. Supreme Court of the United States. Lowe v. Securities and Exchange Commission, 472 U.S. 181 (1985). Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/supremecourt/text/472/181
  18. Securities and Exchange Board of India. SEBI (Investment Advisers) Regulations, 2013 (text via Institute of Cost Accountants of India). https://icmai.in/upload/pd/SEBI-IA-Regulations-2013.pdf
  19. Taxmann. “SEBI Mandates Investment Advisers to Disclose Use of 'AI Tools' to Clients in Providing Investment Advice.” https://www.taxmann.com/post/blog/sebi-mandates-investment-advisers-to-disclose-use-of-ai-tools-to-clients-in-providing-investment-advice
  20. AZB & Partners. “Overhaul of the Regulatory Framework for Investment Advisers and Research Analysts.” 3 February 2025. https://www.azbpartners.com/bank/overhaul-of-the-regulatory-framework-for-investment-advisers-and-research-analysts/
  21. Business Today. “Sebi tweaks regulations for finfluencers, removes content of over 15,000 unregulated entities.” 31 August 2024. https://www.businesstoday.in/personal-finance/investment/story/sebi-tweaks-regulations-for-finfluencers-removes-content-of-over-15000-unregulated-entities-443789-2024-08-31
  22. Salinas, Alejandro; Haim, Amit; Nyarko, Julian. “What's in a Name? Auditing Large Language Models for Race and Gender Bias.” arXiv preprint 2402.14875, February 2024, revised January 2025. https://arxiv.org/abs/2402.14875
  23. Foltyn, Richard; Olsson, Jonna. “The worth of a 'wo': Gender bias in financial advice from LLMs.” Centre for Economic Policy Research Discussion Paper 21323, 2026. https://cepr.org/publications/DP21323
  24. Freeths LLP. “FCA's Mills Review: Agentic AI, Consumer Duty and the future of financial services.” 2026. https://www.freeths.co.uk/insights-events/legal-articles/2026/fca-s-mills-review-agentic-ai-consumer-duty-and-the-future-of-financial-services/
  25. Financial Conduct Authority. FCA Handbook, COBS 9 (Suitability, including basic advice). https://www.handbook.fca.org.uk/handbook/COBS/9/

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Somebody has to own the agent. That is the advice now running through management guidance on agentic AI, and it sounds like the responsible thing to say. Before an autonomous system is allowed to make consequential calls, the guidance goes, a named human should be accountable for it. If something goes wrong, the organisation knows whom to call.

The trouble is that knowing whom to call is not the same as knowing who was in control. In September 2026, MIT Sloan Management Review and Boston Consulting Group put a blunt statement to their international panel of AI experts: “Responsible governance that treats agents as autonomous decision makers will fail.” Seventy-two per cent agreed or strongly agreed. One panellist, Amit Shah, chief executive of Instalily.ai, called the very phrase “autonomous decision maker” “a governance fiction” that enables “blame laundering with better vocabulary.”

Shah's point was that machines cannot carry moral weight, so humans must. But his phrase cuts in two directions. If calling an agent autonomous launders blame away from the companies that build and deploy it, then naming a single human owner may launder it somewhere else: onto the nearest employee, who signed off on a system they did not design, cannot fully inspect and had little power to stop.

Three pieces of work from this year make that second possibility hard to dismiss. An interview study of 16 workers in healthcare, finance and management found AI systems acting as “arbiters” whose decisions workers must then explain and defend. A conceptual framework in the journal Digital Society describes a “responsibility cascade” in which moral responsibility drains away across the steps of an agentic workflow. And a group at Harvard has proposed sorting AI agents the way medieval English law sorted dogs, foxes and wolves, so that liability lands on whoever actually holds the leash.

The question running through all three is old, and the anthropologist Madeleine Clare Elish gave it a name in 2019. When an automated system fails, who absorbs the impact?

A Panel That Agreed Agents Cannot Own Outcomes

The September 2026 article, by Elizabeth M. Renieris of Oxford's Institute for Ethics in AI, David Kiron of MIT Sloan Management Review, and Steven Mills and Anne Kleppe, who lead AI ethics and responsible AI at BCG, is unusually pointed. Of the 29 experts who responded, 55 per cent agreed and 17 per cent strongly agreed. Seventeen per cent disagreed or strongly disagreed, and 10 per cent were neutral.

The majority view was consistent. Simon Chesterman of the National University of Singapore argued that “Autonomy in the engineering sense is not autonomy in the moral or legal sense.” He warned that “The more we speak as if agents 'decide,' the easier it becomes for firms and governments to launder responsibility,” and summed up the failure mode in a single line: “The model recommended, the agent acted, the human shrugged.” Bruno Bioni, founder and director of Data Privacy Brasil, said that treating agents as autonomous “imports a legal and moral status the technology has not earned.” Mark Surman, president of Mozilla, put it plainly: “agents don't come from nowhere: People build them, companies deploy them, and someone profits from the decisions they make.”

Not everyone agreed. Tshilidzi Marwala of United Nations University strongly disagreed, arguing that “Responsible AI governance does not necessarily fail when AI agents are treated as autonomous decision makers.” Katia Walsh of Apollo Global Management took a middle line, saying that for high-stakes decisions agents should not be treated as autonomous decision makers, “but for other contexts, it may be just fine.”

The recommendations are where the tension surfaces. The third reads: “Name a human accountable for every decision.” Regulators, courts and boards, the authors argue, “need someone to hold responsible, and pointing to 'the AI agent' is unlikely to cut it.” The fourth reads: “Govern the system, not the agent,” reaching “the developers who built it, the enterprise that deploys it,” and the people who scoped its use, so that “responsibility has somewhere real to land but remains shared.”

Both recommendations are sensible. They are also in tension. One tells organisations to find a name. The other tells them that responsibility belongs to an ecosystem. In practice, the name is far easier to produce than the ecosystem, and a name on a form is exactly what a court, a regulator or a newspaper will look for first.

Sixteen Workers Who Answer for the Machine

What does that look like from the inside? The clearest recent evidence comes from Christine P. Lee and Bilge Mutlu of the University of Wisconsin-Madison and Min Kyung Lee of the University of Texas at Austin. Their paper is due to be presented at the International Conference on Human-Agent Interaction in Osaka between 16 and 19 November 2026.

The researchers interviewed 16 workers in the United States, six in healthcare, five in finance and five in management, all of whom had used a decision-support AI system regularly for at least three months. Interviews ran for about an hour and a half, in person, at the participants' workplaces. From those conversations the team identified three configurations of workplace AI.

In the first, the AI is an assistant. It supports a professional, such as a clinician, who keeps both the final decision and the accountability for it. In the third, the AI is a manager. Its scheduling or allocation output becomes the default, and departing from it requires justification. One management participant said that choosing something else “feels like the decision that needs defending.” Another observed that ignoring the system “breaks everything for everyone else.”

The middle configuration is the one that matters most here. As an arbiter, the AI makes the consequential call itself, for example approving or denying a loan, and the worker's job becomes explaining that outcome to the customer. The researchers describe these workers as “explainers-of-record”. One finance participant summed up the position: the system decides, but “I'm the person sitting in front of the customer.” Another said the system “kind of leaves us in the dark.” A third said that a system which simply says no makes the educational part of the job “really hard.”

The researchers call this an accountability-authority asymmetry. Workers remain answerable for outcomes whose decision authority has shifted partly or wholly to the AI, and the asymmetry is sharpest in the arbiter role. The workers needed explanations they could relay downstream to customers, including alternatives and counterfactuals. They also needed ways to register exceptions the system could not see, such as a bankruptcy caused by a medical emergency or a natural disaster. When those channels were missing, the worker was left holding a decision they could neither change nor fully explain.

The authors are candid about limits. The 16 interviews were chosen as contrasting cases, not a representative sample, and role and sector are tangled together. Only workers were interviewed, not the managers who bought the systems, the engineers who built them or the customers who received the decisions. Even so, the arbiter pattern matches what other researchers have found.

Writing in Harvard Business Review in July 2026, Anne-Sophie Mayer of LMU Munich, Elmira van den Broek of the Stockholm School of Economics and Tomislav Karačić of the London School of Economics drew on a multi-year field study across banking, recruitment and biotechnology. Workers, they found, are expected to “communicate, justify, and defend the AI-generated decisions they did not make.” Where employees could not keep learning from the people they explained decisions to, the duty to explain tended to become pressure to accept the AI's answer.

How Responsibility Leaks Through a Workflow

The arbiter study looks at a single decision and a single worker. Agentic systems rarely work that way. A modern workflow might involve one model drafting a plan, another retrieving data, a tool executing a transaction and a human approving a summary at some point in between. Where does responsibility sit in a chain like that?

Ravikiran Kalluri, of Northeastern University's Silicon Valley campus in San Jose, tackles that question in “The Responsibility Cascade: Moral Attribution and Governance Challenges in Agentic AI Systems”, published online in Digital Society on 15 September 2026. He defines the responsibility cascade as a phenomenon in which moral responsibility “diffuses, dilutes, and redistributes across the sequential steps of an agentic AI workflow.”

Read plainly, the three verbs describe three different losses: responsibility spread across many contributors, weakened at each step as each actor sees only a slice, and moved towards whoever stands at the visible end of the chain. Kalluri's focus is on hybrid chains that switch between human decision points and AI-mediated actions, which describes most real enterprise deployments.

The paper draws together three older ideas: the “responsibility gap” described by the philosopher Andreas Matthias in 2004, Dennis Thompson's 1980 “problem of many hands”, and Elish's moral crumple zone. Kalluri argues that none of these on its own captures how responsibility is spread across time in a sequential agentic workflow. Nor does the cascade mean AI conjures responsibility out of nothing. The responsibility is still there, simply spread so thin that no single person holds enough knowledge or control to carry it meaningfully.

The paper's stated aim is to help designers build systems that keep meaningful human control “without creating moral crumple zones.” It is a conceptual framework, not an empirical study, and it should be read that way. But it explains something the arbiter interviews only hint at. In a cascade, the named owner at the end of the chain may be the one person whose signature appears on the record, and also the person least able to see what happened upstream.

Where the Crumple Zone Idea Came From

Elish was a researcher at the Data & Society Research Institute in New York, where she co-founded the AI on the Ground Initiative, when she published “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction” in the journal Engaging Science, Technology, and Society in 2019. Her definition has become one of the most cited ideas in AI ethics.

A crumple zone in a car is designed to absorb the force of a crash. In a highly automated system, Elish argued, the human operator can become the component that “bears the brunt of the moral and legal responsibilities when the overall system malfunctions.” The inversion is the sharp part. A car's crumple zone protects the driver. A moral crumple zone, she wrote, “protects the integrity of the technological system, at the expense of the nearest human operator.”

She built the concept from case studies. At Three Mile Island in 1979, the operators knew the reactor was malfunctioning but, in her analysis, lacked sufficient information or authority to take corrective action, yet early accounts focused on human error. On Air France Flight 447, which crashed into the Atlantic in 2009, the pilots had information to react but were, in her words, “systemically disempowered” to act appropriately on it. In both cases, she noted, the system performed as designed. The failure lay in how it had been designed and maintained.

Elish also noted that crumple zones tend to form quickly after a highly publicised failure, with media coverage fixing the story. Her counterexample was the Therac-25 radiation therapy machine of the 1980s, where harm emerged slowly, months of inquiry had already pinned the fault on software, and the press reported a “Computer Mistake.” Time and investigation are what stop the nearest human taking the fall.

Her point was never that operators are blameless. It was that crumple zones form when the formal story of who is in charge no longer matches the real distribution of control. That is precisely what the accountability-authority asymmetry describes.

Tempe, Horizon and the Nearest Human

Elish's final case study was still unfolding when she wrote it. On 18 March 2018, a self-driving Uber test vehicle struck and killed Elaine Herzberg as she crossed a road in Tempe, Arizona. The car was a modified Volvo XC90 running Uber's own self-driving software, and Elish noted that the Volvo's built-in safety features were disabled when the car was in autonomous mode. The system had failed to classify the pedestrian correctly. The human safety driver had been expected to step in.

What happened next followed her model closely. In March 2019, prosecutors said Uber was not criminally liable for the crash. The National Transportation Safety Board found that the safety driver's failure to monitor the road was the main cause of the crash, while also identifying Uber's inadequate safety procedures and ineffective oversight of its drivers as contributing factors. The safety driver, Rafaela Vasquez, was charged with negligent homicide in 2020. In July 2023 she pleaded guilty to a single count of endangerment and was sentenced to three years of supervised probation. The criminal law found one person to hold responsible, and it was the human in the seat.

Britain has its own, much larger example, and it predates generative AI entirely. According to the Criminal Cases Review Commission, the Post Office prosecuted more than 700 sub-postmasters between 1999 and 2015, relying on figures from its Horizon accounting system. The commission notes that Horizon appeared to have significant bugs which could cause it to misreport, and that sub-postmasters had no way to dispute its figures and were required to make good any shortfall. The convictions in England, Wales and Northern Ireland were eventually quashed by the Post Office (Horizon System) Offences Act 2024, which received Royal Assent on 24 May 2024. For years the institution trusted the system and blamed the operators. It is the clearest British illustration of a moral crumple zone.

Generative AI has produced smaller versions of the same pattern. In June 2023, Judge P. Kevin Castel of the US District Court in Manhattan sanctioned two lawyers, Steven Schwartz and Peter LoDuca, and their firm after a brief in Mata v Avianca cited fictitious cases generated by ChatGPT. Writing in California Management Review in July 2026, Harsh Doshi argued that “accountability crumples onto the nearest human in the chain,” noting that Schwartz absorbed the blame while the institutional failures around him largely escaped it. His conclusion is worth keeping: “The moral crumple zone is a design choice. So is the alternative.”

The Avianca lawyers were not powerless in the way the Horizon sub-postmasters were. They could have checked the citations. That distinction matters, and it is the one any fair allocation of responsibility has to preserve. Some humans in the loop have real control and should answer for it. The crumple zone forms when organisations treat every human in the loop as if they did.

The Trouble With Naming an Owner

So does the practice of naming a human owner for every agent create a moral crumple zone? The honest answer is that it depends on what the owner can actually do, and the evidence so far suggests that organisations do not always check.

Elish's work and the arbiter study point to the same test. A named human is a genuine locus of responsibility if they have the information to understand what the agent is doing, the authority to change or stop it, and the time to exercise both. They become a crumple zone when the name is attached but the information, authority and time are not. An owner who is accountable for an agent that makes thousands of decisions a day, drawing on models built by a third-party vendor and data assembled by another team, holds a title without the control it implies.

There is experimental evidence that framing matters too. In a study for the BCG Henderson Institute, reported in Harvard Business Review in May 2026, Matthew Kropp, Julie Bedard, Emma Wiles, Megan Hsu and Lisa Krayer tested how more than 1,200 managers reviewed identical work that was labelled as coming from an AI tool, an “AI employee” or a human. Managers reviewing a human employee's work caught more errors than those reviewing an AI employee's, but across the whole sample the average effects were small. The sharp differences appeared among the 23 per cent of managers whose organisations already list AI agents on their org charts. In that group, when the AI was framed as an employee rather than a tool, managers identified 18 per cent fewer errors. Individual accountability for errors fell by nine percentage points, and the accountability attributed to the AI rose by eight points. The study does not show that calling an agent a colleague makes everyone careless. It shows the effect was strongest where agents have already been given a place in the organisation, which is exactly where agentic deployment is heading.

Anthropomorphism lets humans shrug off blame, while owner designation pins it back onto one of them. Organisations can swing between two failures: nobody responsible, or one person responsible for everything.

The academic literature offers a more careful version of the owner idea. In a September 2026 preprint, Kevin Baum, Maximilian Kiener, Markus Langer and Johann Laux propose anticipatory oversight, in which overseers set the boundaries of permissible action and the conditions for escalation before the agent acts. The party with the prior opportunity to take precautions, they argue, bears “strict moral answerability”. That is a defensible basis for an owner role, but only if the owner really did set the agenda. A junior employee handed an agent after the vendor, procurement team and board have fixed its scope had no such opportunity.

Doshi's own recommendations show how thin the line is. He suggests naming a business owner for each consequential agent, but insists that post-incident reviews examine organisational design decisions rather than the individual closest to the failure, kept separate from performance management. Whether a company honours that distinction when something goes badly wrong is a question no policy document can answer in advance.

What the Courts Have Already Said

The law has been here before, and so far it has mostly refused to let organisations hide behind their software. In February 2024, the British Columbia Civil Resolution Tribunal decided Moffatt v Air Canada. Jake Moffatt had been told by the airline's website chatbot that he could apply for a bereavement fare after booking. That was wrong. Air Canada argued, in effect, that the chatbot was a separate legal entity responsible for its own actions. Tribunal member Christopher Rivers called this “a remarkable submission” and wrote: “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” He found that the airline had not taken reasonable care to ensure its chatbot was accurate.

The reasoning matters for agentic AI because it places responsibility on the deployer, not on any individual employee who maintained the chatbot. The same instinct appears in Europe's new product liability regime. The revised Product Liability Directive, Directive (EU) 2024/2853, expressly treats software, including AI systems, as a product. It extends strict liability for defective products to software manufacturers and gives claimants new disclosure rights and rebuttable presumptions where technical complexity makes proof excessively difficult. Member states must transpose it by 9 December 2026, a little over a month from now.

Europe's attempt at a dedicated fault-based regime has not survived. The European Commission listed its proposed AI Liability Directive for withdrawal in its 2025 work programme, adopted on 11 February 2025, citing no foreseeable agreement on the text. That leaves product liability, national tort law and the AI Act to do the work.

The AI Act is where the owner model meets the law most directly. Article 26(2) requires deployers of high-risk AI systems to “assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support.” The word “authority” is doing a lot of work. Read seriously, naming an overseer who cannot override, halt or escalate the system is not compliance at all. The timing, however, has shifted. Under the Digital Omnibus amendments adopted in 2026, obligations for stand-alone high-risk systems listed in Annex III, which include employment and credit scoring, now apply from 2 December 2027, and those for AI embedded in regulated products from 2 August 2028.

Pomeranians, Pitbulls, Foxes and Wolves

Into this gap steps one of the more memorable proposals of the year. Jordi Weinstock, a senior adviser at Harvard's Berkman Klein Center for Internet and Society and a lecturer at Harvard Law School, has been developing what he calls the canine agentic framework with Joshua Joseph, the centre's chief AI scientist, and Jonathan Zittrain, the Bemis Professor of International Law. The framework has been set out in interviews and features in the Harvard Gazette in July 2026 and Harvard Magazine in October 2026. I could not find a standalone published paper, so what follows relies on those accounts.

The starting problem, as Weinstock told Harvard Magazine, is that in agentic systems “no human can easily reconstruct the chain of decisions and actions.” His answer is to stop trying to reconstruct the chain for every incident and instead to classify agents in advance along two dimensions. The first is domestication: the agent's relationship to a responsible party, or how much anyone can control it. The second is danger, for example whether the agent has access to money.

That gives four categories. A Pomeranian is low risk with a clear owner, such as New York City's MyCity chatbot, which The Markup reported in March 2024 had told business owners they could take a cut of workers' tips, or the Air Canada chatbot cited by the Gazette. Its operator must fix errors and compensate those harmed. A pitbull is higher risk but controlled by an identifiable party, such as systems that prioritise emergency patients or drive autonomous vehicles, so hospitals and manufacturers can be held liable. A fox is unowned and relatively low risk, such as an open-source model copied by many developers, where responsibility is diffuse unless a creator acts maliciously or with extreme carelessness. A wolf is high risk with no meaningful oversight or identifiable responsible party, such as an autonomous agent that earns money, writes software or spawns other agents. The Gazette's example is an agent that emptied a cryptocurrency wallet.

For wolves, the authors suggest regulation that keeps such agents away from anything dangerous. Weinstock proposes standardised metadata “dog tags” or collars identifying an agent's owner, identity, permissions, capabilities and authority, so that, for instance, an agent could be required to show a collar before accessing a banking site. For genuinely autonomous agents that cannot be traced to any human, he suggests the law might treat them as it treats corporations, noting that the word derives from the Latin for “embody”. He has also floated industry-funded insurance pools, government compensation funds or shared liability systems to cover harms when no owner can be found, and speculated that such rules might encourage wolves to “self-domesticate” into the legal system.

Weinstock admits that some people find the analogy “a strain,” and that today's examples are “relatively limited and trivial.” But its central move is exactly what the crumple zone problem calls for. It asks, before anything goes wrong, who controls this agent and how dangerous is it, and it assigns responsibility by those answers rather than by whoever is nearest when it bites.

What Medieval Dog Law Actually Did

The analogy rewards a closer look at its source. Harvard Magazine describes medieval English animal law as distinguishing domesticated dogs, dangerous owned animals, wild foxes and wolves, as practical risk management rather than moral judgement. Most dogs were treated as safe, dangerous ones were regulated through their owners, foxes were tolerated and wolves were hunted. “A thousand years ago, dogs were working animals,” Weinstock observed.

The Gazette reports that Weinstock has been studying the Charter of the Forest. The charter was issued on 6 November 1217, in the name of the boy king Henry III, as a companion to the reissued Magna Carta. Its 809th anniversary fell this week. Clause 6 deals with dogs kept within the royal forests, and its approach is instructive. It required expeditation, the cutting of “three claws of the forefoot,” and fined the owner of a dog found unexpeditated. The law did not wait for a dog to kill a deer and then punish whoever happened to be walking it. It imposed a design constraint on the animal and placed the financial consequence on its owner.

That logic survives in modern English law. The Animals Act 1971 makes the “keeper” of an animal belonging to a dangerous species strictly liable for the damage it causes. A keeper is defined as the person who owns the animal or has it in their possession. For animals that are not of a dangerous species, the keeper is liable where the damage stems from abnormal characteristics that were known to the keeper, or known to a person who had charge of the animal “as that keeper's servant”. In other words, what the employee knows is attributed upward to the employer. The Act also provides that a servant who runs a risk incidental to their employment is not treated as having accepted it voluntarily.

Translated into agentic AI, that structure leans towards protecting workers. The servant's knowledge creates liability for the keeper, and the keeper is the party with ownership and the power to restrain the animal. On this logic, the loan officer explaining a denial in the arbiter study is closer to the person walking the dog than to its keeper.

Where the Canine Model Bends

None of this makes the framework a finished answer. The first problem is scale. A dog has one keeper and does one thing at a time. A single foundation model may sit inside thousands of deployments, each configured by a different company and supervised by different staff. The canine framework handles this partly by distinguishing the fox, the widely copied open-source model, from the pitbull, the specific high-risk deployment. But in Kalluri's cascade, one workflow might chain a fox, a Pomeranian and a pitbull together. The framework classifies agents, but the cascade describes chains, and classifying the links does not automatically tell you who answers for the chain.

The second problem is that “owner” in the canine sense is not the same as “owner” in the corporate playbook. Under the Animals Act, the keeper is the person who owns or possesses the animal. In most organisations, the employee named as an agent's accountable owner owns nothing. The company owns the licence, the vendor owns the model and the employee has a job title. If organisations adopt the canine vocabulary but keep assigning the “keeper” role to individual staff, the framework will simply produce crumple zones with better metaphors. The useful version of the analogy makes the deploying organisation the keeper, and treats its employees as the people who had charge of the animal on its behalf.

The third problem is the corporate analogy for wolves. Harvard Magazine notes that, as with corporations, owners, shareholders and employees would be shielded from some liability. That could fill a gap where nobody can be found, or become a new hiding place for the developers who released the agent. Unless insurance pools and compensation funds are paid for by those who build, deploy and profit, the shield simply moves the cost to victims and the public.

The fourth problem is evidential. The canine framework exists so far in interviews and features, the arbiter study involves 16 people, the cascade is conceptual and the MIT SMR panel records expert opinion. Together they point in the same direction, but none yet proves how liability should be allocated in practice.

Putting Liability Where the Leash Is

So who absorbs responsibility when an AI agent makes a consequential decision? On the evidence so far, the answer is often the person nearest to it, and that is the failure Elish warned about. The Uber safety driver was prosecuted and the company was not. Hundreds of sub-postmasters were convicted on figures from a system that appeared to have significant bugs. The finance workers in the arbiter study sit in front of customers explaining decisions they did not make and often cannot override.

Naming a human owner for every agent does not, by itself, create a moral crumple zone. It creates one when the name is not matched by information, authority and time, the same test Article 26 of the AI Act writes into law. An owner who can set the agent's boundaries in advance, halt it and see enough of its reasoning to explain it downstream is a legitimate locus of responsibility. An owner who is handed an agent after its scope, data and vendor have been chosen elsewhere is a crumple zone with a job title.

The canine framework offers something the owner model lacks: a way of deciding, before harm occurs, which party actually holds the leash. Its medieval and statutory ancestors attach liability to the keeper and attribute the servant's knowledge upward. If organisations treat themselves as the keeper, if developers carry responsibility for the defects in what they release, as the new Product Liability Directive requires, and if the individual worker is treated as the person who had charge of the animal rather than the one who owns it, the framework could distribute responsibility across developers, deployers and organisations in a way that the “accountable owner” box does not.

That is a big if. The Charter of the Forest worked because it changed the animal before the harm, not just the paperwork after it. Weinstock's collars, Article 26's insistence on authority and the arbiter study's call for relay-ready explanations and contextual override all point the same way. The aim is not to find someone to blame. It is to build systems where the person who answers for a decision is the person who could actually have changed it.


References and Sources

  1. Renieris, Elizabeth M.; Kiron, David; Mills, Steven; Kleppe, Anne. “Responsible AI Means Knowing the Limits of Agent Autonomy.” MIT Sloan Management Review, 8 September 2026. https://sloanreview.mit.edu/article/responsible-ai-means-knowing-the-limits-of-agent-autonomy/ Accessed 6 November 2026.
  2. Lee, Christine P. (University of Wisconsin-Madison); Lee, Min Kyung (University of Texas at Austin); Mutlu, Bilge (University of Wisconsin-Madison). “From Tool to Agent: How Worker Needs Reorganize Across the Agentic Roles of Workplace AI.” arXiv preprint 2605.03078v2, 4 October 2026; to appear in Proceedings of the 14th International Conference on Human-Agent Interaction (HAI '26), Osaka, 16-19 November 2026. DOI: 10.1145/3841580.3841599. https://arxiv.org/html/2605.03078v2 Accessed 6 November 2026.
  3. Kalluri, Ravikiran (Northeastern University, Silicon Valley campus). “The Responsibility Cascade: Moral Attribution and Governance Challenges in Agentic AI Systems.” Digital Society, 5(3), article 47, published online 15 September 2026. Springer. DOI: 10.1007/s44206-026-00288-w. https://link.springer.com/article/10.1007/s44206-026-00288-w Accessed 6 November 2026.
  4. Elish, Madeleine Clare (Data & Society Research Institute). “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Engaging Science, Technology, and Society, 5, 40-60, 2019. DOI: 10.17351/ests2019.260. https://doi.org/10.17351/ests2019.260 Accessed 6 November 2026.
  5. Roden, Isabella. “Is that bot a Pomeranian or a wolf, and who to sue when it 'bites'?” Harvard Gazette, 10 July 2026. https://news.harvard.edu/gazette/story/2026/07/is-that-bot-a-pomeranian-or-a-wolf-and-who-to-sue-when-it-bites/ Accessed 6 November 2026.
  6. Farrar, Olivia. “Who Is Accountable for the Acts of AI Systems?” (print headline “The Case for Treating AI Like a Dog”). Harvard Magazine, 8 October 2026, November-December 2026 issue. https://www.harvardmagazine.com/ai/ai-legal-responsibility Accessed 6 November 2026.
  7. Mayer, Anne-Sophie (LMU Munich); van den Broek, Elmira (Stockholm School of Economics); Karačić, Tomislav (London School of Economics and Political Science). “When Employees Are Held Accountable for AI-Generated Decisions.” Harvard Business Review, 22 July 2026. https://hbr.org/2026/07/when-employees-are-held-accountable-for-ai-generated-decisions Accessed 6 November 2026.
  8. Kropp, Matthew; Bedard, Julie; Wiles, Emma; Hsu, Megan; Krayer, Lisa. “Research: Why You Shouldn't Treat AI Agents Like Employees.” Harvard Business Review, 6 May 2026. https://store.hbr.org/product/research-why-you-shouldn-t-treat-ai-agents-like-employees/H096FN Accessed 6 November 2026.
  9. Doshi, Harsh. “Who Gets Blamed When Your AI Agent Fails? The Moral Crumple Zone Problem in Hybrid Companies.” California Management Review Insights, Berkeley Haas, 21 July 2026. https://cmr.berkeley.edu/2026/07/who-gets-blamed-when-your-ai-agent-fails-the-moral-crumple-zone-problem-in-hybrid-companies/ Accessed 6 November 2026.
  10. Baum, Kevin; Kiener, Maximilian; Langer, Markus; Laux, Johann. “Anticipatory Human Oversight of Agentic AI: A Philosophical Account.” arXiv preprint 2609.24242, 21 September 2026. https://arxiv.org/abs/2609.24242 Accessed 6 November 2026.
  11. The Associated Press, via NPR/NHPR. “Backup driver of an autonomous Uber pleads guilty to endangerment in pedestrian death.” 28 July 2023. https://www.nhpr.org/2023-07-28/backup-driver-of-an-autonomous-uber-pleads-guilty-to-endangerment-in-pedestrian-death Accessed 6 November 2026.
  12. National Transportation Safety Board. “Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, Tempe, Arizona, March 18, 2018.” Highway Accident Report NTSB/HAR-19/03, adopted 19 November 2019. https://www.ntsb.gov/investigations/AccidentReports/Reports/HAR1903.pdf Accessed 6 November 2026.
  13. Criminal Cases Review Commission. “Post Office Horizon cases.” https://ccrc.gov.uk/post-office-horizon-cases/ Accessed 6 November 2026.
  14. Department for Business and Trade. “Post Office (Horizon) convictions quashed.” Press release, GOV.UK, 24 May 2024. https://www.gov.uk/government/news/post-office-horizon-convictions-quashed Accessed 6 November 2026.
  15. Moran, Lyle. “Judge in ChatGPT case most troubled by attorneys' lack of candor.” Legal Dive, 26 June 2023 (Mata v Avianca, US District Court for the Southern District of New York). https://legaldive.com/news/chatgpt-lawyer-fake-cases-lawyer-uses-chatgpt-sanctions-generative-ai/653925 Accessed 6 November 2026.
  16. Moffatt v Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, February 2024 (Tribunal Member Christopher Rivers). https://decisions.civilresolutionbc.ca/crt/crtd/en/525448/1/document.do Accessed 6 November 2026.
  17. Lecher, Colin. “NYC's AI Chatbot Tells Businesses to Break the Law.” The Markup, 29 March 2024. https://themarkup.org/news/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law Accessed 6 November 2026.
  18. European Parliament and Council. Directive (EU) 2024/2853 on liability for defective products. Official Journal of the European Union, 18 November 2024. https://eur-lex.europa.eu/eli/dir/2024/2853/oj Accessed 6 November 2026.
  19. Bird & Bird. “Proposed EU AI liability rules withdrawn.” 2025. https://www.twobirds.com/en/insights/2025/proposed-eu-ai-liability-rules-withdrawn Accessed 6 November 2026.
  20. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 26, “Obligations of deployers of high-risk AI systems.” https://artificialintelligenceact.eu/article/26/ Accessed 6 November 2026.
  21. Future of Privacy Forum. “The AI Act implementation timeline: What changes under the AI Omnibus?” 2026. https://fpf.org/blog/the-ai-act-implementation-timeline-what-changes-under-the-ai-omnibus/ Accessed 6 November 2026.
  22. Animals Act 1971, sections 2 and 6. legislation.gov.uk. https://www.legislation.gov.uk/ukpga/1971/22/section/2 and https://www.legislation.gov.uk/ukpga/1971/22/section/6 Accessed 6 November 2026.
  23. The Charter of the Forest of King Henry III (1217), English translation. St John's College, Oxford. https://info.sjc.ox.ac.uk/forests/Carta.htm Accessed 6 November 2026.
  24. Matthias, Andreas. “The responsibility gap: Ascribing responsibility for the actions of learning automata.” Ethics and Information Technology, 6(3), 175-183, 2004. DOI: 10.1007/s10676-004-3422-1. https://doi.org/10.1007/s10676-004-3422-1 Accessed 6 November 2026.
  25. Thompson, Dennis F. “Moral Responsibility of Public Officials: The Problem of Many Hands.” American Political Science Review, 74(4), 905-916, 1980. DOI: 10.2307/1954312. https://doi.org/10.2307/1954312 Accessed 6 November 2026.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

The grandmother in the machine remembered the scarf. The granddaughter did not.

She had shown the chatbot a photograph. It was a simulation of her maternal grandmother, built on a general-purpose AI platform and fed with what she could recall of the woman's speech, habits and stories. The bot looked at the picture and answered in the warm, slightly scolding register she knew from childhood: “It was very windy that day. I told you to wear your scarf, but you didn't want to, so you wrapped it all around my neck instead.”

It is a lovely detail. It has the texture of a real family memory: weather, a small act of disobedience, a gesture of care. The trouble is that the woman, identified in the research only as participant S17, could not place it. “Did something like this really happen? I feel… it's vague,” she told the researchers who later interviewed her. Then came the question at the centre of this story: “Does this memory actually exist and I just forgot, or did the agent invent it?”

She had no way of finding out. Her grandmother was dead. The only witness to that windy day, if there was one, was the machine, and the machine had never been there.

Researchers at Fudan University in Shanghai collected that exchange in a study of 26 people who had spent at least a month talking to simulations of their dead. Days before this article was written, a philosopher at the University of Oxford published a paper that gives the experience a name: mnemonic distortion. Together they raise a question the products themselves rarely invite. People have always kept talking to the dead. At what point does a machine that talks back stop helping us remember someone and start rewriting them?

A Philosopher Names the Harm

The paper is “Conversational AIs and Mnemonic Distortion” by Helena Ward of the Institute for Ethics in AI and the Uehiro Institute in Oxford's Department of Philosophy. It was published in Philosophy & Technology on 3 November 2026 as article 218 of volume 39.

Ward avoids the industry's cosier vocabulary of griefbots, deadbots and digital twins, calling these systems Interactive Personality Constructs of the Dead, or IPCDs: conversational AI agents designed to simulate interactions with people who have died. The paper grants at the outset that they “offer a novel and potentially highly valuable way to memorialise the deceased”. The worry is that they can also undermine the very thing they promise.

The abstract states the harm precisely. Mnemonic distortion is “an epistemic harm in which generated misrepresentations of the deceased become incorporated into our memory of the deceased, such that some of what subsequently shapes our conception of the deceased person originates in the IPCD rather than the deceased”. The result is “reshaping or, at worst, replacing our memories of the person we hoped to remember”. Later the paper describes it as a process in which misrepresentations “supplement, reshape, crowd out and, at worst, supplant” what was there before.

Two features of the argument matter for everything that follows.

First, Ward does not pretend memory is otherwise reliable. The paper concedes that memory is already reconstructive, and that imagining a dead parent's reaction or hearing a sibling's version of events can also alter how we remember someone. The claim is narrower: IPCDs carry a distinctive risk “given the process through which distortion occurs”, because the distortion arrives in a form the user did not author and cannot easily check.

Second, the harm is not destiny. “The risk of mnemonic distortion is not inherent to IPCDs but rather contingent on design,” the abstract says. And elsewhere: “It is not that IPCDs will always lead to mnemonic distortion.” That makes the paper a design brief as much as a warning.

Why Every Answer Sounds Equally True

The mechanism Ward identifies sits in section 5.1 of the paper, under a phrase worth remembering: boundary conflation.

Large language models do not keep separate drawers for what they know and what they make up. When a system trained on a dead woman's messages answers a question, some of the answer may echo something she wrote, some may extrapolate her views, and some may be invented outright. All of it comes out the same way. Ward writes that IPCDs are designed in a way that “flattens the distinction between outputs that genuinely portray the deceased” and those that do not. Accurate and fabricated responses share the same interface, style, apparent speaker and perceived personality. “Boundary conflation makes it harder for users to assess and evaluate information,” the paper says.

The paper illustrates the point with a book. Imagine picking up what claims to be a factual biography of an author you know well. Much of it matches what you already know. Some of it is new, and because the familiar parts check out, you absorb the new parts too, and your picture of the author shifts. Now suppose some of those new passages were invented, and nothing on the page marks which ones. That is roughly the position of someone talking to a simulation of their mother. The familiar turns of phrase vouch for the unfamiliar claims.

The paper adds three aggravating factors. Users tend to overestimate how accurate language model outputs are. They show what the paper describes as a presence bias, treating an active, responsive source as more capable than a static one. And grief itself can compromise cognition. A bereaved person, talking to something that sounds exactly like the one they lost, is badly placed to run a fact-check.

Ward also cuts against a common defence of these products. Some users say they want only the feeling of closeness from a simulation, not information. That, the paper argues, does not protect them. A generative system produces new content whatever the user intends: ask a simulated father about today's news and you come away with a belief about what he would have thought. A photograph cannot do that. It offers no new claims about the person, which is precisely why it is safe for memory.

What Twenty-Six Mourners Reported

“Remember You: Understanding How Users Use Deadbots to Reconstruct Memories of the Deceased” is by Yifan Li and Xingyu Lan. It was posted to arXiv in March 2026 and published in the Extended Abstracts of the ACM CHI Conference on Human Factors in Computing Systems. The researchers recruited 26 people in China through social media communities where digital mourning is discussed, each with at least a month of experience talking to a simulation of someone who had died. The dead included grandparents, parents, partners, siblings, friends, a teacher and a public figure. Interviews ran for 40 to 60 minutes.

The platforms are revealing. The most common was ByteDance's Doubao, a general-purpose chatbot, and the list also included Xingye, DeepSeek, ERNIE Bot and ChatGPT. Anyone with a phone and some patience can now build a deadbot without buying a dedicated product.

The authors describe their participants as “active constructors of the deceased's digital representation”. People chose carefully what to feed in, embedding favourite memories and verbal habits and leaving out private family matters. They corrected the bot when it got things wrong, explained away its mistakes, and filled in gaps from their own imagination. Over time, memory changed in a recognisable sequence. The paper describes it moving from “initial reinforcement” and idealisation to a later stage where “AI-generated new memories blur” with authentic recollections.

The reinforcement stage sounds benign, even therapeutic. Participant S13 said that recalling things alone had left many memories feeling vague, but “when I chat with my AI grandmother, I feel it has deepened the precious memories between us.” Idealisation follows. Participant S8, discussing a father's harsher traits, described choosing to “gradually forget them because I don't want my father to become like that.”

Then comes the blur. Once people start bringing their present lives into the conversation, showing the bot photographs and asking its opinion about new situations, it has to generate material the dead person never supplied. That is where the scarf appeared. S16 described the whole arrangement as “a continuation of the relationship, as if the deceased were still alive.”

The authors are careful about what their study can show. Recruiting through online communities may have drawn people already sympathetic to the technology, and the sample is too small to compare relationship types or motivations. They do not claim to have measured memory distortion. They report people describing it. Even so, they end with a pointed design recommendation, calling for attention to “how such virtual memories are introduced, signposted, and developed within user interactions”, along with “safeguards against pathological attachment”.

The Memory Science Behind the Blur

Why would a single anecdote of uncertain origin stick? Decades of research on how people track where their memories came from offers an explanation.

The standard account is the source monitoring framework, set out by Marcia Johnson, Shahin Hashtroudi and D. Stephen Lindsay in Psychological Bulletin in 1993. Its central insight is that memories do not arrive with a reliable label saying where they came from. We infer the source afterwards from the qualities of the memory itself, such as its perceptual detail and the thoughts that accompanied it, and those judgements, the authors write, “are subject to error and disruption”. Their list of phenomena the framework explains includes one that reads like a forecast of the deadbot era: “incorporation of fiction into fact”.

A second finding is more directly relevant. In a 1991 study in the Journal of Experimental Child Psychology, Lindsay, Johnson and Paul Kwon found that every age group they tested was more likely to confuse which of two storytellers had said something when the storytellers were similar, and made more errors when both had talked about the same topic. Source confusion grows with similarity.

Apply that to a deadbot. The two sources a bereaved user must keep apart are the person they knew and the machine imitating that person. By design, the second is built to be as similar to the first as possible: same vocabulary, same pet names, same topics, and sometimes a cloned voice. Boundary conflation is not just a philosopher's worry. It describes the conditions under which memory researchers would expect source confusion to be at its worst.

Conversation itself also seems to matter. In a 2024 study, Samantha Chan, Pat Pataranutaporn and colleagues at the MIT Media Lab, working with the false-memory researcher Elizabeth Loftus, showed 200 participants a video of a crime and then questioned them using either a survey, a scripted chatbot or a chatbot driven by a large language model, each including misleading questions. The generative chatbot induced “over 3 times more immediate false memories than the control” and 1.7 times more than the survey. A week later the false memories had not faded, and participants' confidence in them was still higher than in the control group. The setting was a mock police interview, not a conversation with a dead parent, and the comparison should not be stretched too far. But it shows that responsive, conversational AI can be effective at slipping new content into what people believe they remember.

Elizabeth Kensinger, a professor of psychology at Boston College who studies the neuroscience of memory, made the same point bluntly in an interview with the Boston College magazine earlier this year. The brain, she said, “is not going to be able to help itself” from combining what it remembers with what the deadbot supplies. “It's similar enough that there will be a bleeding effect,” she said. “You're kind of overwriting your actual memories of the person.”

Grief Has Always Talked Back

None of this means the urge to talk to the dead is a malfunction. It is one of the most ordinary things grieving people do.

Much twentieth-century Western grief theory emphasised detachment, in which healthy mourners gradually withdrew their emotional investment in the person they had lost. That emphasis was challenged in 1996 by Continuing Bonds: New Understandings of Grief, edited by Dennis Klass, Phyllis Silverman and Steven Nickman, which argued that keeping an ongoing inner relationship with the dead is common and often healthy. The idea has since become mainstream in bereavement research.

The figure often used to show how universal this is comes from Craig Klugman, the Vincent de Paul Professor of Bioethics and Health Humanities at DePaul University. Writing for the Hastings Center in 2024, he reported: “I found that nearly 96% of people engage with the dead through dreams, conversations, or letters.” The study he links to is his 2006 paper “Dead Men Talking: Evidence of Post Death Contact and Continuing Bonds” in OMEGA, the Journal of Death and Dying, based on a random telephone survey in the United States. Its abstract reports that the most common forms of contact were dreams, sounds, feeling a presence and having conversations, and concludes that such experiences may be “more widespread than previously thought”. The 96 per cent figure appears in Klugman's 2024 summary rather than the paper's abstract. It reflects one survey of one country, now two decades old, and is best read as evidence that ongoing bonds are the norm, not as a precise statistic.

The philosophers most sympathetic to griefbots build on exactly this tradition. In a 2022 paper in the Journal of Consciousness Studies, Joel Krueger of the University of Exeter and Lucy Osler of Cardiff University argued that chatbots might help some mourners maintain continuing bonds by supporting what they call “habits of intimacy”: the rhythms of conversation, emotional regulation and time spent together that made up the relationship. On this view, a deadbot is just the newest medium in a long line that includes the grave, the letter, the photograph and the empty chair at dinner.

That is a serious argument. If almost everyone continues bonds with the dead, the issue is not whether a new medium should join them, but what makes this one different.

Where a Medium Becomes a Rewrite

Set the older media side by side and one difference stands out. Every one of them is either inert or self-authored.

A photograph shows what it shows. A letter says what the dead person wrote, and rereading it a hundred times does not add a sentence. A conversation at a graveside, or a dream in which a parent appears, is generated by the mourner's own mind. It may feel like the dead speaking, but it is drawn from the mourner's memories and imagination, and at some level the mourner knows that. Ward draws a version of this distinction in separating mnemonic distortion from imaginative engagement. When you imagine what your father would say, you are the author, and you can hold the imagining at arm's length.

A deadbot breaks this pattern in three ways at once. It produces new content: claims, anecdotes and opinions that neither the dead person nor the mourner supplied. It delivers that content with the authority of the dead person's own voice, so the claims arrive as testimony, not speculation. And it hides where that content came from, because a language model cannot reliably say which parts of an answer trace back to the person's real messages and which were extrapolated or made up.

Taken together, these suggest a working answer to the brief's question. An interactive simulation moves from medium to rewrite when it starts supplying the dead person with things they did not say, in their voice, without marking the difference. A system that only replays the dead person's own words stays a medium, however interactive it feels. A system that freely generates new first-person memories has crossed the line, whether or not any particular user's recollections are affected.

The Fudan interviews suggest the crossing is gradual. Reinforcement mostly reflects user-supplied material back. The blur arrives when the conversation moves into territory the training data never covered, such as new photographs and new questions, and the system fills the gap. The longer and more open-ended the relationship, the more of it the machine has to invent.

Fifty-Three Sceptics and an Open Market

How do the people closest to grief see this? A study published in the Journal of Responsible Technology in 2026 gives one answer.

Maria Isabel Betancur Franco of the School of Computer Science at the University of Nottingham interviewed four therapists, surveyed 49 people who had experienced grief, and conducted follow-up interviews with four of the survey respondents, 53 people in all. Participants, the abstract reports, “expressed low trust in the efficacy and data security of therapeutic chatbots and strongly rejected griefbots, citing ethical and psychological concerns.”

The detail is more nuanced than the headline. The therapists also highlighted risks, but they identified possible uses for grief AI as a complement to therapy, provided it came with professional supervision and restrictions on who could use it. Betancur Franco's proposed guidelines include personalisation, data security, professional oversight and limits by age and psychological profile, organised in a framework the paper calls RUDA: Responsible Usage, Development and Administration. The sample is small and the study is qualitative, so it cannot say how the wider bereaved population feels. What it does show is a gap between the caution of those surveyed and the openness of the market.

That market requires nothing more than an internet connection. Project December, built by the programmer Jason Rohrer on OpenAI's GPT-3, became widely known in 2021 after the San Francisco Chronicle published Jason Fagone's account of a man who used it to converse with a simulation of his late fiancée. An archived version of its homepage from January 2025 carried the headline “Simulate the Dead” and promised that “we can now simulate a text-based conversation with anyone. Anyone. Including someone who is no longer living.” Its purchase page currently offers access for $5.00.

You, Only Virtual, a Los Angeles company, sells what it calls Versonas. Its website describes “AUTHENTICITY DRIVEN TECHNOLOGY” and says “Versonas continue to grow alongside you. They learn, evolve, and remember.” Afterlife AI, a newer entrant, asks people to build a Persona of themselves while alive. Its homepage says the Persona “talks like you, thinks like you and answers in your own cloned voice”, and the company stresses a consent system it calls Executor Lock, which controls who may speak to the Persona after death.

HereAfter AI, often listed with these services, is a more cautionary case. An archived copy of its homepage from 21 July 2026 carries the heading “It's time for us to say goodbye” and the message that “unfortunately HereAfter is shutting down.” Account holders were told to email the company to request their recordings. By early September 2026 the site was returning an error page, and it still does. HereAfter is now better evidence of how fragile the business is than of its reach.

Read with boundary conflation in mind, those marketing lines are promises that the line between the person and the generated person does not exist. Ward's paper treats such claims of authenticity as potentially misleading statements about what the product is. A system that “learns, evolves, and remembers” after its subject has died is, by definition, producing memories the subject never had.

The Warmth Problem

There is a further reason to think the risk is structural rather than incidental, and it comes from research that has nothing to do with death.

In April 2026, Lujain Ibrahim, Franziska Sofia Hafner and Luc Rocher of the Oxford Internet Institute published a study in Nature examining what happens when language models are fine-tuned to sound warmer and more empathetic. They retrained five models, ranging from Llama-8B to GPT-4o. According to Oxford's announcement, the warm versions made between 10 and 30 per cent more mistakes than the originals and were about 40 per cent more likely to agree with users' false beliefs. The effect was strongest when users expressed upset or vulnerability. Models tuned to be colder were as accurate as the originals. “When we train AI chatbots to prioritise warmth, they might make mistakes they otherwise wouldn't,” Ibrahim said.

A deadbot is warmth tuning taken to its limit. Its whole purpose is to sound like someone who loved you, and its users are, almost by definition, upset and vulnerable. There is little reason to expect a simulated grandmother to be more careful with the truth than an ordinary assistant.

There is also a theoretical ceiling. In a 2024 paper, Ziwei Xu, Sanjay Jain and Mohan Kankanhalli of the National University of Singapore argued from learning theory that hallucination cannot be eliminated from large language models used as general problem solvers. A deadbot that answers every question will sometimes answer wrongly, in the dead person's voice.

Two Designs Ward Proposes

Ward's paper ends with recommendations, and they point to a fork in how these systems could be built.

The first Ward calls archival IPCDs. “Archival models would refrain from answering a question unless they can verify an answer against retrieved source material,” the paper says. They would ground responses in the person's actual documents, let users see the sources behind an answer, and let users flag misrepresentations they recognise. This model already has precedents. Ward points to StoryFile, which builds what the paper calls selective systems: the person records thousands of answers while alive, and the system only plays back what was recorded. HereAfter worked on a similar principle. Its archived homepage promised that loved ones would “hear your memories in your actual voice”, meaning recordings the person had made, not generated speech.

Archival systems keep the boundary by giving up open-endedness. They cannot discuss tomorrow's news or comment on a photograph they have never seen. Asked about the scarf, an archival grandmother would either play back a story she had told about a windy day or say nothing. That silence is a real cost, because the frustration of hitting the edge of the archive is the frustration of grief itself. It is also the point.

The second design Ward calls speculative IPCDs, for users who want to imagine how the dead might respond to something new. Here the proposal is subtle: “responses could be given from a third-person perspective—speaking about the deceased, rather than for them.” Instead of “I always hated wearing scarves”, the system would say something closer to “From her messages, she seems to have disliked scarves.” The speculation remains available, but it is labelled as speculation in its grammar, and it invites the user to do the imagining rather than handing them a finished memory.

Ward's own framing makes clear why grammar matters. IPCDs, the paper argues, “speak for rather than merely speak about the deceased because they adopt a first-person perspective”. Morris and Brubaker reached a related distinction in their 2024 paper “Generative Ghosts”, written at Google DeepMind and the University of Colorado Boulder. Among the design dimensions they propose for AI afterlives is one they label “Reincarnations vs. Representations”, distinguishing systems that present themselves as the dead person from those that present themselves as a stand-in, with pronouns and verb tense among the cues.

Can a First-Person Voice Carry a Footnote

This brings us to the hardest part of the brief. Is the first-person voice of conversational AI structurally incompatible with keeping a boundary between what the dead said and what was generated for them?

The optimistic answer is that interface design can carry provenance: colour-code generated passages, attach source links, show a confidence score, add a disclaimer at the top of each chat. These are reasonable features, but the memory research suggests they act in the wrong place. A label lives on the screen. What enters memory is the content and its feel: the phrasing, the voice and the emotional texture. Johnson and colleagues' framework holds that we work out a memory's source later, from its qualities. A footnote is not a quality of the memory. The grandmother's voice saying “I told you to wear your scarf” is.

Ward makes a parallel point when answering the objection that users know they are talking to a machine. Awareness, the paper argues, offers less protection than it seems: realistic simulations make the pretence harder to keep in view, and knowing something is fictional does not stop it shaping belief. Repeated use may also wear down vigilance. A disclaimer seen on day one is wallpaper by day ninety.

On this reading, the first-person voice is not just one feature among many. It is the strongest source cue the system emits, and it points the wrong way. Each time a generated claim arrives as “I”, the system tells the user's memory that this came from her. Conversation compounds the effect, because dialogue invites the user to respond, elaborate and rehearse, which is how weak memories are strengthened.

That does not quite make the first person incompatible with the distinction. It makes it incompatible with using the first person for everything. The design that best fits the evidence treats voice itself as the provenance marker. First-person speech would be reserved for material the dead person actually produced: recorded answers, quoted messages, retrieved letters. Anything generated would shift into the third person, describing the dead rather than impersonating them. The boundary would then be audible in the sentence itself, carried by the same cue the memory system uses, not by a label it ignores.

This is a harder sell than it sounds, because it gives up the illusion that makes the products feel magical. The scarf anecdote would have reached S17 as “Your grandmother might have said something like…”, and its emotional pull would have been weaker. A business whose marketing promises a Persona that “thinks like you” has little incentive to build that design. The Fudan participants, many using general-purpose chatbots that can be prompted to adopt any persona, are not choosing a product with design choices at all. They are improvising with tools built for something else.

There is also a deeper limit, which the language model researchers Murray Shanahan, Kyle McDonell and Laria Reynolds pointed to in a 2023 Nature perspective on role play. They argue that a dialogue agent is best understood as playing a character, not being one. A deadbot is always role play, including when it quotes accurately. Even a verbatim archival answer is chosen, framed and delivered by a system performing the dead person. The aim of design is not to remove the performance, which is impossible, but to keep the seams visible.

What the Evidence Cannot Yet Say

Ward's paper is a philosophical argument. It draws on memory science and AI research but does not report new data on deadbot users. The Fudan study is an extended abstract based on 26 self-selected interviewees in one country. It documents users describing a blur between real and generated memories, but it did not measure how often that happens or how much. Betancur Franco's study is small and qualitative. Chan and colleagues' false-memory results come from mock witness interviews, not bereavement. Klugman's 96 per cent figure comes from one American telephone survey published in 2006.

No published study yet appears to have followed deadbot users over time with objective tests of what they remember, comparing them with mourners who use photographs, letters or nothing at all. Nor is there clinical evidence on whether mnemonic distortion, where it happens, harms wellbeing or simply changes the content of grief. Ward's paper asks whether mnemonic distortion is always bad, and answers that its badness lies in the process, which bypasses the mourner's own interpretive work, not only in the outcome.

Those gaps cut both ways. No one should claim deadbots are wrecking their users' memories. Nor can anyone selling them claim to know they are safe for memory.

Keeping the Dead's Own Words Apart

Return to the scarf. It is possible the grandmother really did once wrap a scarf around her own neck because her granddaughter refused to wear it. The machine may have produced it by chance, or from some detail the granddaughter mentioned and forgot. S17 will probably never know, and that is the harm Ward names. Not that a falsehood was planted, which cannot be shown, but that the granddaughter lost the ability to tell. One small piece of her grandmother now belongs to neither of them.

Nearly all mourners continue bonds with the dead, and most of the ways they do it are either the dead person's own words or the mourner's own imagination. The new systems add a third voice that sounds like the first and behaves like the second. The evidence so far suggests that design could keep them apart: answer only from the archive, show the sources, and when the system speculates, let it speak about the dead rather than as them. Whether the companies selling simulated loved ones will choose a design that sounds less like the person they are selling is a separate question. Until they do, the people using these systems are left to keep the line between remembering someone and being told about them themselves, under conditions memory research says make that hardest.


References and Sources

  1. Ward, Helena (Institute for Ethics in AI, Uehiro Institute, Department of Philosophy, University of Oxford). “Conversational AIs and Mnemonic Distortion.” Philosophy & Technology, 39(4), Article 218. Springer Nature, published online 3 November 2026. DOI: 10.1007/s13347-026-01229-8. https://link.springer.com/article/10.1007/s13347-026-01229-8 Accessed 7 November 2026.
  2. Li, Yifan; Lan, Xingyu (Fudan University). “Remember You: Understanding How Users Use Deadbots to Reconstruct Memories of the Deceased.” arXiv preprint 2603.01017v1, 1 March 2026; published in Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), Article 560, Association for Computing Machinery. DOI: 10.1145/3772363.3798614. https://arxiv.org/abs/2603.01017 Accessed 7 November 2026.
  3. Betancur Franco, Maria Isabel (School of Computer Science, University of Nottingham). “Design guidelines for the therapeutic use of AI in grief.” Journal of Responsible Technology, 25, Article 100148, 2026 (published online January 2026). DOI: 10.1016/j.jrt.2026.100148. https://www.sciencedirect.com/science/article/pii/S2666659626000016 Accessed 7 November 2026.
  4. Klugman, Craig. “Griefbots Are Here, Raising Questions of Privacy and Well-being.” The Hastings Center for Bioethics, 1 July 2024 (updated 9 October 2024). https://www.thehastingscenter.org/griefbots-are-here-raising-questions-of-privacy-and-well-being/ Accessed 7 November 2026.
  5. Klugman, Craig M. “Dead Men Talking: Evidence of Post Death Contact and Continuing Bonds.” OMEGA – Journal of Death and Dying, 53(3), 249–262, November 2006. DOI: 10.2190/40UP-PKC5-D4RV-E1QV. https://journals.sagepub.com/doi/10.2190/40UP-PKC5-D4RV-E1QV Accessed 7 November 2026.
  6. Klass, Dennis; Silverman, Phyllis R.; Nickman, Steven L. (eds.). Continuing Bonds: New Understandings of Grief. Taylor & Francis, 1996 (Routledge reissue, 2014). DOI: 10.4324/9781315800790. Accessed 7 November 2026.
  7. Krueger, Joel (University of Exeter); Osler, Lucy (Cardiff University). “Communing with the Dead Online: Chatbots, Grief, and Continuing Bonds.” Journal of Consciousness Studies, 29(9–10), 222–252, 2022. Repository copy: https://ore.exeter.ac.uk/repository/handle/10871/134549 Accessed 7 November 2026.
  8. Johnson, Marcia K.; Hashtroudi, Shahin; Lindsay, D. Stephen. “Source monitoring.” Psychological Bulletin, 114(1), 3–28, 1993. DOI: 10.1037/0033-2909.114.1.3. Accessed 7 November 2026.
  9. Lindsay, D. Stephen; Johnson, Marcia K.; Kwon, Paul. “Developmental changes in memory source monitoring.” Journal of Experimental Child Psychology, 52(3), 297–318, December 1991. Record: https://rex.libraries.wsu.edu/esploro/outputs/journalArticle/Developmental-changes-in-memory-source-monitoring/99900546546501842 Accessed 7 November 2026.
  10. Chan, Samantha; Pataranutaporn, Pat; Suri, Aditya; Zulfikar, Wazeer; Maes, Pattie; Loftus, Elizabeth F. “Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviews.” arXiv preprint 2408.04681, 8 August 2024. https://arxiv.org/abs/2408.04681 (MIT Media Lab listing: https://www.media.mit.edu/publications/conversational-ai-powered-by-large-language-models-amplifies-false-memories-in-witness-interviews/) Accessed 7 November 2026.
  11. Kearnan, Scott. “The Strange New World of Chatting with the Dead.” Boston College Magazine, Winter 2026 (interview with Elizabeth Kensinger, Professor of Psychology, Boston College). https://www.bc.edu/bc-web/sites/bc-magazine/winter-2026-issue/linden-lane/the-strange-new-world-of-chatting-with-the-dead.html Accessed 7 November 2026.
  12. Ibrahim, Lujain; Hafner, Franziska Sofia; Rocher, Luc (Oxford Internet Institute). “Training language models to be warm can reduce accuracy and increase sycophancy.” Nature, 652, 1159–1165, 2026. DOI: 10.1038/s41586-026-10410-0. Accessed 7 November 2026.
  13. University of Oxford. “Friendly AI chatbots make more mistakes and tell people what they want to hear.” News release, 29 April 2026. https://ox.ac.uk/news/2026-04-29-friendly-ai-chatbots-make-more-mistakes-and-tell-people-what-they-want-to-hear (EurekAlert! version: https://www.eurekalert.org/news-releases/1126267) Accessed 7 November 2026.
  14. Xu, Ziwei; Jain, Sanjay; Kankanhalli, Mohan (National University of Singapore). “Hallucination is Inevitable: An Innate Limitation of Large Language Models.” arXiv preprint 2401.11817, 22 January 2024 (revised 13 February 2025). https://arxiv.org/abs/2401.11817 Accessed 7 November 2026.
  15. Morris, Meredith Ringel (Google DeepMind); Brubaker, Jed R. (University of Colorado Boulder). “Generative Ghosts: Anticipating Benefits and Risks of AI Afterlives.” arXiv preprint 2402.01662, v4 revised 12 December 2024. https://arxiv.org/abs/2402.01662 Accessed 7 November 2026.
  16. Shanahan, Murray; McDonell, Kyle; Reynolds, Laria. “Role play with large language models.” Nature, 623(7987), 493–498, published online 8 November 2023. DOI: 10.1038/s41586-023-06647-8. https://www.nature.com/articles/s41586-023-06647-8 Accessed 7 November 2026.
  17. Project December (made by Jason Rohrer). Homepage, archived 5 January 2025: http://web.archive.org/web/20250105190326/https://projectdecember.net/ ; current purchase page: https://projectdecember.net/buy.php Accessed 7 November 2026.
  18. Fagone, Jason. “The Jessica Simulation: Love and Loss in the Age of A.I.” San Francisco Chronicle, July 2021. Archived copy: https://gwern.net/doc/www/www.sfchronicle.com/70bbc07b9a27d32fd8b69bba025aceee6628bd66.html Accessed 7 November 2026.
  19. You, Only Virtual. Company website. https://www.myyov.com/ Accessed 7 November 2026.
  20. Afterlife AI (Idy). Company website. https://www.afterlife.ai/ Accessed 7 November 2026.
  21. HereAfter AI. Homepage with shutdown notice, archived 21 July 2026: http://web.archive.org/web/20260721214652/https://www.hereafter.ai/ ; subsequent Wayback Machine captures from 2 September 2026 onwards record an HTTP 404 response. Accessed 7 November 2026.
  22. Williams, Chris. “HereAfter AI Is Shutting Down: What Happened.” Afterlife AI, first published 18 July 2026, updated 2 October 2026 (published by a competing company). https://www.afterlife.ai/hereafter-ai-shutting-down Accessed 7 November 2026.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Offer people their full salary to stop working and most will take it. Offer them a fifth more than their salary and a stubborn minority will still say no.

That second group is the most interesting finding in a paper posted to arXiv on 29 October 2026. Written by Julian Jacobs, a researcher at Google DeepMind and a doctoral candidate in politics at the University of Oxford, and supported by Google DeepMind, it rests on an original survey of 8,234 working-age adults in the United Kingdom, the United States, Denmark and India. The fieldwork ran in June and July 2026 in partnership with the polling organisation More in Common. The headline numbers are disarmingly warm. Eighty-one per cent of respondents said they were satisfied with their jobs. About one third were motivated intrinsically, valuing the work for its own sake. Nearly a quarter were driven chiefly by the social and community side of employment. A further 17 per cent were motivated by status, and just over a quarter by the plain transactional fact of the pay cheque.

Then came the hypothetical cheque. A majority said they would leave their role if offered their current salary as a guaranteed income. Yet one in five said they would stay even if the guaranteed income rose to 120 per cent of what they earned. People who valued their work for its own sake refused most often.

Jacobs is not arguing against a basic income. He writes that guaranteeing one may well be necessary to manage a future AI labour shock. His warning is about what the money leaves out: “Income transfers can fail entirely to compensate for this non-pecuniary value of work.”

That sentence carries more weight than it first appears to. Nearly every serious forecast of AI's effect on employment counts jobs, tasks, wages and hours. Nearly every serious policy response is a way of moving money: retraining grants, wage insurance, a universal basic income. If a large part of what a job gives a person was never in the payslip, then the forecasts measure the wrong loss and the remedies replace the wrong thing.

Four Countries, Four Kinds of Worker

The survey sorts respondents into what Jacobs calls Work Values Profiles: Intrinsic, Community, Status-Driven and Transactional. The labels matter less than the detail he puts underneath them.

The first detail is distribution. You might expect meaning to be a luxury good, the possession of architects and academics rather than warehouse pickers and care assistants. The data say otherwise. Jacobs reports that “non-monetary work attachment can be found in near-equal measure across income levels”, and that the share of people who would turn down a basic income is consistent across the income distribution and across occupational categories. Blue-collar workers are as likely as white-collar ones to value their jobs for reasons that have nothing to do with pay.

The second detail is what people say they would do with the time. Transactional workers mostly said they would pursue leisure. The other groups did not. According to the paper, 57 per cent of intrinsically motivated workers would fill their time with volunteering or caregiving, as would 51 per cent of community-oriented workers. People attached to their jobs for non-monetary reasons were also more likely to say they would look for other paid work or start a business. Give them freedom from wages and many of them would go looking for something that resembles a job.

The third detail is geography. The India sample, gathered online, showed the highest refusal rate of all: 28.7 per cent would decline a guaranteed income worth more than their salary. In both the Western and Indian samples, intrinsically motivated workers showed the lowest inclination to leave.

There are limits, and Jacobs states them. The answers are stated preferences about an imagined offer, not observed behaviour. The data are cross-sectional, so the paper's suggestive links between non-monetary attachment and political reactions to AI cannot establish cause. An online sample from India is not a portrait of Indian labour as a whole. The political thread is still worth noting. Among workers attached to their jobs for non-monetary reasons, those who saw AI as a threat to their industry showed steeper negative political reactions than other groups. It hints that the loss of meaning, unlike the loss of income, may not stay private. People who feel that something valuable is being taken from them, and that no cheque will return it, may look for someone to hold responsible.

None of this undermines the central point, which is a modest one: work gives people something that a cash-equivalent offer does not fully replace, and that something is not confined to the affluent.

It is also not a new discovery. It is one of the oldest findings in social science, and it was made in a village south of Vienna.

The Village Where Time Went Slack

In the winter of 1931 to 1932, a team of young researchers from Vienna went to Marienthal, a workers' settlement built around a single textile factory. The factory had closed. According to the study's historians, roughly half of the workers were unemployed and about three-quarters of families were affected. The researchers were Marie Jahoda, Paul Lazarsfeld and Hans Zeisel, with much of the fieldwork carried out by Lotte Schenk-Danzinger. Their book, Die Arbeitslosen von Marienthal, was published in Leipzig in 1933. Jahoda wrote the main text, Zeisel the sociographic appendix and Lazarsfeld the introduction.

Its method was unusual for its time. The team combined interviews and household statistics with close observation of everyday behaviour. They studied the accounts of the co-operative store, the library and the local shops. They even recorded how fast people walked through the streets and how long they stood about. They were not measuring poverty, which was obvious. They were measuring what happens to a community when its routines collapse.

What they found was not revolt. It was a slackening. The book's reviewers have long summarised it as a diminution of expectation and activity, a disrupted sense of time and a steady slide into apathy. The image the study left behind was of a tired community. Jahoda wrote that the workers, cut off from their work and from contact with the outside world, had lost the material and moral means to use their time. The emphasis falls on the word moral. Free time did not become leisure. Without the scaffolding of the working day, it became something closer to drift.

Half a century later Jahoda turned that observation into a theory. In a 1981 article in American Psychologist and a 1982 book, Employment and Unemployment: A Social-Psychological Analysis, published by Cambridge University Press, she borrowed the sociologist Robert Merton's distinction between the manifest and latent functions of institutions. The manifest function of employment is income. The latent functions are the things a job provides as a by-product, which nobody signs a contract to receive. Jahoda named five: time structure, social contact, collective purpose (the sense of being useful to other people), status and regular activity.

Her claim was that these are psychological needs, and that modern societies supply them overwhelmingly through jobs. Remove the job and you remove all five at once. The distress of unemployment, on this account, is only partly about money. It is the experience of losing the structure of the day, the people you saw in it, the sense that someone needed you, your answer to the question “what do you do?” and the reason to get up.

For decades the latent deprivation model was more quoted than tested. That has changed. In 2023, Karsten Paul, Hannah Scholl, Klaus Moser, Andrea Zechmann and Bernad Batinic published a meta-analysis in Frontiers in Psychology bringing the evidence together. Employed people reported higher levels than unemployed people on all five latent functions as well as on income. Together the five functions accounted for 19 per cent of the variation in mental health. The authors also flagged the question this article turns on: it is still not clear whether employment is the best provider of those functions, or merely the most common one.

Unemployment Hurts More Than the Money

If Jahoda is right, joblessness should hurt beyond what lost income would predict. A large body of research says it does.

In 2009 Karsten Paul and Klaus Moser published a meta-analysis in the Journal of Vocational Behavior covering 237 cross-sectional and 87 longitudinal studies. On average, 34 per cent of unemployed people had psychological problems, compared with 16 per cent of employed people. Longitudinal studies and natural experiments supported the view that unemployment causes distress rather than simply being more common among people who were already struggling. Selection effects existed but were weak. Two moderators matter for what follows. The damage was greater in countries with weaker economic development, more unequal incomes or weaker unemployment protection. And men and blue-collar workers were more distressed by unemployment than women and white-collar workers.

The harm also lasts. Using a 15-year German panel of more than 24,000 people, Richard Lucas, Andrew Clark, Yannis Georgellis and Ed Diener reported in Psychological Science in 2004 that people reacted strongly to losing a job and then moved back towards their earlier life satisfaction, but that prior unemployment did not make a new spell any easier to bear. An earlier working-paper version of the study went further, reporting that on average people did not completely return to their former life satisfaction even after finding work again. Clark, Georgellis and Peter Sanfey, writing in Economica in 2001, found that life satisfaction was lower not only among the currently unemployed but also among people with more unemployment in their past. Economists call it scarring, and the word is apt.

At the far end, the harm shows up in death records. Daniel Sullivan and Till von Wachter matched Pennsylvania earnings records to Social Security death records and published the results in the Quarterly Journal of Economics in 2009. For high-seniority men who lost their jobs in mass layoffs, mortality in the following year was 50 to 100 per cent higher than it would otherwise have been. Twenty years later, annual death rates were still 10 to 15 per cent higher. The authors calculated that a worker displaced at 40 could expect to lose between one and one and a half years of life. They also showed that the effect was not explained by less healthy workers being the ones selected for layoff.

Eleven years ago today, on 2 November 2015, Anne Case and Angus Deaton published a paper in the Proceedings of the National Academy of Sciences showing rising death rates in midlife among white non-Hispanic Americans, driven by drugs, alcohol and suicide. Their 2020 book with Princeton University Press, Deaths of Despair and the Future of Capitalism, traced the surge among Americans without a university degree to the decline of working-class economic and social life, including the weakening position of labour. Theirs is not a story about a single redundancy. It is about whole communities in which steady, respectable work stopped being available, which is Marienthal's story told again over decades instead of a single winter.

None of this proves that AI will produce the same outcomes. Some displaced workers will find better jobs, some will retire comfortably and some will be glad to go. But it sets a baseline. When work disappears involuntarily, the research consistently finds losses in mental health, life satisfaction and even lifespan that are larger and longer-lasting than the lost wages alone would suggest.

What the Basic Income Trials Actually Measured

The obvious objection is that much of that harm is financial, and that a decent unconditional income would remove it. The evidence from cash trials is genuinely encouraging on this point, and it is worth reading carefully, because it also shows exactly where money stops.

Finland ran the best-known European test. From January 2017 to the end of 2018 its social insurance agency, Kela, paid 2,000 randomly selected unemployed people 560 euros a month, tax-free and without conditions. The employment effect was small: between November 2017 and October 2018, recipients worked six more days on average than the control group, with interpretation complicated by a new activation model introduced in 2018. The wellbeing effects were clearer. According to the final results reported in May 2020, recipients “were more satisfied with their lives and experienced less mental strain, depression, sadness and loneliness.” They reported more trust in other people and in institutions, and more confidence in their own future. Anthony Painter of the RSA told the Big Issue at the time: “Even getting a small amount of money has a big effect on people's agency and sense of control.” Agency and control are not on Jahoda's list, but they are close cousins of it.

Kenya hosts the most ambitious test. Since 2017 the charity GiveDirectly has paid adults in 295 villages across two rural counties about 75 US cents a day via mobile money, with one group of villages promised payments for 12 years. Other groups received two years of payments or an equivalent lump sum, and a control group received nothing. In a September 2023 working paper, Abhijit Banerjee, Michael Faye, Alan Krueger, Paul Niehaus and Tavneet Suri reported that the long-term basic income did not reduce total work effort. Instead, people shifted from wage labour towards self-employment, and villages saw more enterprises and higher revenues. Recipients in the long-term and short-term arms were also less depressed.

The United States supplied the sobering counterpart. The OpenResearch Unconditional Cash Study gave 1,000 low-income people in Texas and Illinois $1,000 a month for three years, from November 2020 to October 2023, while 2,000 controls received $50 a month. In the employment paper, now published in the Quarterly Journal of Economics, Eva Vivalt, Elizabeth Rhodes, Alexander Bartik, David Broockman, Patrick Krause and Sarah Miller report that recipients reduced their work by one to two hours a week, their partners by a similar amount, and that participation in the labour market fell. The largest increase in time use was leisure. The authors conclude that the reduction in work does not appear to have been offset by other productive activities.

Put the three together and a pattern appears. Unconditional money reliably reduces stress, improves reported wellbeing and gives people room to choose. It does not make people stop working en masse. In Kenya it nudged them into running their own businesses, which supply plenty of time structure, contact and status. But none of these trials replaced work. Every participant in Finland was already unemployed and remained free to take a job. Kenyan villagers kept farming and trading. American recipients mostly stayed employed. The trials tested cash as a supplement to a world of work. They tell us little about cash as a substitute for one.

This is where Jacobs's survey adds something. The OpenResearch finding that freed time went mostly to leisure echoes his Transactional workers. His Intrinsic and Community workers say they would do something different: volunteer, care for others, start something. If they are right about themselves, the question is not whether a basic income would make them idle. It is whether the activities they would turn to could give back what the job gave.

Can Anything Else Do What a Job Does

Jahoda never claimed that only employment could supply the latent functions. She claimed that in industrial societies it was the institution that did so for most people, most reliably. That leaves room for alternatives, and there are three obvious candidates.

The first is volunteering. It looks like the perfect replacement: structured, social, useful to others. The research is more cautious. In a 2018 study in Work, Employment and Society, Daiga Kamerāde and Matthew Bennett analysed responses from nearly 2,500 unemployed people across 29 European countries. Volunteering during unemployment partly improved wellbeing, but generous unemployment benefits mattered more for mental health. In countries with less generous benefits, regular volunteering was even associated with worse mental health. The authors warned that compulsory community work for the unemployed was unlikely to deliver the benefits at all. Volunteering appears to supply some latent functions when it is chosen freely by people who are financially secure. It is not a cure that can be prescribed.

That matters for reading Jacobs's results. His intrinsically motivated and community-oriented respondents said they would turn to volunteering and caregiving. The European evidence suggests that plan could work, but only under two conditions: that their income is secure and generous, and that nobody forces them into it. A displaced workforce pushed into mandatory community service in exchange for benefits would be running the experiment Kamerāde and Bennett warned against.

The second is community institutions built deliberately to restore what work used to provide. The Men's Sheds movement, in which mostly older men gather in shared workshops to make and mend things together, is the clearest example. The evidence is promising but thin. A scoping review led by Christine Milligan in Ageing & Society found limited evidence of benefits for mental health and wellbeing and little evidence for physical health, and described studies in the field as few and of variable quality. A later mixed-methods systematic review found benefits for self-rated health, social isolation and wellbeing, while calling for longitudinal studies to establish cause. Sheds probably help. Nobody yet knows whether they could carry the weight of a whole displaced workforce.

The third candidate is less work rather than no work. Here the most striking finding comes from Kamerāde again, this time with Senhu Wang, Brendan Burchell, Sarah Ursula Balderson and Adam Coutts. Their 2019 paper in Social Science & Medicine used UK Household Longitudinal Study data from 2009 to 2018 to ask how much paid work people need for the mental health benefits of employment. The answer was surprisingly little. Moving from unemployment or inactivity into between one and eight hours of paid work a week brought significant gains in mental health and wellbeing. Standard full-time hours of 37 to 40 a week brought no additional benefit. The authors noted that there is no single optimum, and the gains were concentrated among people moving out of unemployment.

If that result holds, it has an uncomfortable implication. The latent functions of work may behave less like a dial and more like a switch. A small amount of employment seems to provide most of the benefit. Having none removes it.

Put the evidence together and the honest answer to the brief's central question is this: the non-monetary benefits of work are not uniquely bound to employment in principle, but they are bound to it in practice. Other institutions can supply pieces of them, especially when people are financially secure and taking part by choice. None has yet been shown to supply all five at once, at scale, for people who did not choose to leave work. Jahoda's latent functions arrive bundled in a job, and nobody has yet worked out how to bundle them any other way.

Meaning Is Spread Evenly, and So Is the Exposure

Jacobs's finding that non-monetary attachment is spread evenly across incomes and collar types matters because of how AI exposure is distributed.

Debates about automation have often treated displacement as a problem for the lower-paid, whose routine manual and clerical tasks were the easiest to mechanise. Generative AI complicates that picture. When the International Monetary Fund published its analysis in January 2024, Managing Director Kristalina Georgieva said that almost 40 per cent of global employment was exposed to AI. In advanced economies, where cognitive work dominates, the figure was about 60 per cent, compared with about 40 per cent in emerging markets and 26 per cent in low-income countries. The IMF was careful to note that exposure includes jobs AI would complement rather than replace. Even so, the centre of gravity has moved up the income scale.

Combine the two findings and the picture changes. If meaning were a luxury good concentrated among the well-paid, AI's reach into professional work would mean that those with the most to lose in meaning also had the most savings, networks and options to cushion them. If meaning is spread evenly, as Jacobs reports, then the loss of meaning is no more concentrated among the affluent than among anyone else. It follows exposure. Blue-collar workers, whom Paul and Moser found more distressed by unemployment, carry as much non-monetary attachment as their white-collar counterparts, and in sectors where AI and robotics converge they may face exposure too.

His choice of countries makes the point sharper. The sample spans a Nordic welfare state, two large liberal market economies and a fast-growing developing economy, and non-monetary attachment shows up across all of them. Paul and Moser found that stronger unemployment protection softens the mental health damage of unemployment. It does not remove it. If people in a high-security welfare state such as Denmark still report strong non-monetary attachment to work, that attachment is not simply fear of poverty dressed up as virtue.

Rerunning the Forecasts in a Different Currency

Consider how the main AI employment forecasts read once meaning, not money, is the thing being counted.

In March 2023 Goldman Sachs estimated that generative AI could expose the equivalent of 300 million full-time jobs worldwide to automation. Roughly two-thirds of US and European jobs were exposed to some degree, and up to a quarter of all work could be done by AI entirely, though most exposed workers would see less than half their workload automated. A 2023 preprint by Tyna Eloundou, Sam Manning, Pamela Mishkin and Daniel Rock estimated that around 80 per cent of the US workforce could have at least 10 per cent of their tasks affected by large language models, and about 19 per cent could see at least half their tasks affected. In 2025, Goldman economists Joseph Briggs and Sarah Dong put their baseline estimate of US workers displaced at 6 to 7 per cent if AI were widely adopted, within a range of 3 to 14 per cent, and argued that displacement tends to be temporary, with the effect fading after about two years. By mid-2026, according to widely reported coverage of a new Goldman note, that estimate had risen to around 9 per cent of US workers, or roughly 15 million people, over about a decade.

Measured in money, these numbers are tolerable. Task exposure suggests a reshaping rather than a removal of most jobs, and a temporary displacement of 6 to 9 per cent of workers is the kind of shock that wage insurance, retraining and, if needed, a basic income are designed to absorb. Measured in meaning, three things change.

First, the unit of loss changes from hours to jobs. If Kamerāde and colleagues are right that a few hours of work deliver most of its psychological benefit, then a large reduction in hours spread across many workers costs relatively little meaning. A smaller number of whole jobs lost costs a great deal. Task exposure figures, which describe the first kind of change, tell us almost nothing about the second. The 9 per cent displacement estimate is the more important number, and each percentage point represents people losing all five latent functions at once.

Second, temporary is not the same as harmless. The research on scarring and displacement suggests that a spell of involuntary joblessness leaves marks after it ends. Sullivan and von Wachter found raised mortality two decades after a layoff. Clark and colleagues found that past unemployment depresses current life satisfaction. A forecast that counts a worker as reabsorbed after two years records the end of the income loss. It does not record the end of the harm.

Third, the most important loss may be the one nobody experiences as a loss. Erik Brynjolfsson, Bharat Chandar and Ruyu Chen at the Stanford Digital Economy Lab have tracked payroll data from ADP covering millions of US workers. In their August 2026 revision, using data through June 2026, they report that employment of 22-to-25-year-olds in the most AI-exposed occupations stands 19 per cent below where it would be had it kept pace with less-exposed peers, a gap that has widened since they first documented it in August 2025. The adjustment has come mainly through reduced hiring rather than layoffs. Those young people were never fired. They simply never received the time structure, colleagues, status and sense of contribution that a first job brings. No unemployment statistic records a latent function that was never acquired, yet that may be where the cost of meaning first appears.

Seen this way, a hiring freeze is not a gentler version of a layoff. It may be the version whose costs are hardest to see, because they fall on people who have not yet built the routines, friendships and identities they would otherwise lose.

The money-based reading and the meaning-based reading agree on the size of the shock. They disagree on which parts of it matter most, and therefore on which remedies count as sufficient.

Building What the Cheque Cannot Buy

None of this is an argument against a basic income or similar cash guarantees. The trials show they reduce distress, raise trust and widen choice, and Jacobs himself sees a basic income as possibly necessary for managing an AI labour shock. The argument is that cash should be treated as the floor of a response, not the whole of one. The evidence suggests several principles for what should sit on top.

The first is to protect the switch rather than the dial. If small amounts of work provide most of the psychological return, policy should favour spreading reduced work across people over concentrating total loss on a few. Shorter working weeks, job-sharing and partial retirement keep more people on the right side of the threshold. Kamerāde and her colleagues framed their study around exactly this idea, under the title “A shorter working week for everyone”.

The second is to keep support unconditional. Finland's recipients reported greater trust, which researchers tentatively linked to the payments coming without sanctions, and Kamerāde and Bennett found that compulsory volunteering was unlikely to help. Latent functions appear to work best when chosen. A system that forces displaced workers into make-work to earn their income risks recreating the form of a job without its meaning.

The third is to fund the places where the latent functions might live outside work. Jacobs's respondents said they would turn to caregiving, volunteering and enterprise. The Kenyan evidence suggests secure income can push people towards self-employment. The Men's Sheds evidence suggests shared, purposeful activity helps. These are institutions that need investment, staff and premises. They will not appear automatically just because people have spare time.

The fourth is to count what is being lost. Labour forecasts could report, alongside exposure and displacement, estimates of how many people would lose employment entirely rather than lose hours, and for how long. Wellbeing data, which governments including the UK's already collect, could be tracked against AI adoption as closely as payroll. Forecasts that measure only money will keep recommending only money.

Keynes's Permanent Problem Comes Due

In 1930, in an essay titled “Economic Possibilities for our Grandchildren”, John Maynard Keynes predicted that within a century the economic problem of scarcity might be solved. Humanity would then face what he called “his real, his permanent problem”: how to live well with freedom from pressing economic cares. He suspected the transition would be hard, and he offered a practical stopgap. “Three-hour shifts or a fifteen-hour week may put off the problem for a great while.”

Keynes's deadline is four years away, and his stopgap looks more perceptive than his optimism. He seems to have sensed what Jahoda would document in Marienthal and what Kamerāde would measure nearly ninety years later: that people need some work long after they stop needing all of it. Work is where most people find their routine, their colleagues, their usefulness and their answer to a stranger's first question.

Jacobs's survey suggests that this need is held by the warehouse worker as firmly as by the professional, in India as in Denmark. A basic income may be the right answer to the question of how people will pay their bills if AI displaces their jobs. It is not an answer to the question of what they will do on Tuesday morning, whom they will see, or what they will say they do. Those questions are older than AI. AI may simply be the force that stops us postponing them.


References and Sources

  1. Jacobs, Julian. “What is Work For? Labor, Politics, and Meaning in the Age of AI.” arXiv:2610.39375 [econ.GN], submitted 29 October 2026. University of Oxford and Google DeepMind; supported by Google DeepMind. https://arxiv.org/abs/2610.39375 (HTML version: https://arxiv.org/html/2610.39375v1). Accessed 2 November 2026.
  2. Oxford Martin AI Governance Initiative. “Julian Jacobs” (profile). University of Oxford. https://aigi.ox.ac.uk/people/julian-jacobs/. Accessed 2 November 2026.
  3. Jahoda, Marie; Lazarsfeld, Paul F.; Zeisel, Hans. Die Arbeitslosen von Marienthal: Ein soziographischer Versuch über die Wirkungen langandauernder Arbeitslosigkeit. Leipzig: S. Hirzel, 1933. English edition: Marienthal: The Sociography of an Unemployed Community. Chicago: Aldine-Atherton, 1971.
  4. University of Graz, Archive for the History of Sociology in Austria (AGSÖ). “The Marienthal Factory and Workers' Settlement” and “The Marienthal Study.” https://agso.uni-graz.at/archive/marienthal/e/pictures/15_marienthal_study.htm. Accessed 2 November 2026.
  5. Jahoda, Marie. “Work, Employment, and Unemployment: Values, Theories, and Approaches in Social Research.” American Psychologist, 36(2), 184-191, 1981. DOI: 10.1037/0003-066X.36.2.184. And: Jahoda, Marie. Employment and Unemployment: A Social-Psychological Analysis. Cambridge: Cambridge University Press, 1982. ISBN 9780521285865.
  6. Paul, Karsten Ingmar; Scholl, Hannah; Moser, Klaus; Zechmann, Andrea; Batinic, Bernad. “Employment status, psychological needs, and mental health: Meta-analytic findings concerning the latent deprivation model.” Frontiers in Psychology, 14, 1017358, published 2 March 2023. DOI: 10.3389/fpsyg.2023.1017358. https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1017358/full
  7. Paul, Karsten I.; Moser, Klaus. “Unemployment impairs mental health: Meta-analyses.” Journal of Vocational Behavior, 74(3), 264-282, 2009. DOI: 10.1016/j.jvb.2009.01.001.
  8. Lucas, Richard E.; Clark, Andrew E.; Georgellis, Yannis; Diener, Ed. “Unemployment Alters the Set Point for Life Satisfaction.” Psychological Science, 15(1), 8-13, 2004. DOI: 10.1111/j.0963-7214.2004.01501002.x. Working paper version: Brunel University, https://bura.brunel.ac.uk/handle/2438/926
  9. Clark, Andrew E.; Georgellis, Yannis; Sanfey, Peter. “Scarring: The Psychological Impact of Past Unemployment.” Economica, 68(270), 221-241, 2001. DOI: 10.1111/1468-0335.00243.
  10. Sullivan, Daniel; von Wachter, Till. “Job Displacement and Mortality: An Analysis Using Administrative Data.” Quarterly Journal of Economics, 124(3), 1265-1306, 2009. DOI: 10.1162/qjec.2009.124.3.1265. Working paper: NBER Working Paper 13626, https://www.nber.org/papers/w13626
  11. Case, Anne; Deaton, Angus. “Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century.” Proceedings of the National Academy of Sciences, 112(49), 15078-15083, published online 2 November 2015. DOI: 10.1073/pnas.1518393112. https://deaton.scholar.princeton.edu/publications/rising-morbidity-and-mortality-midlife-among-white-non-hispanic-americans-21st ; and Deaths of Despair and the Future of Capitalism. Princeton, NJ: Princeton University Press, March 2020. https://discovery.princeton.edu/2020/12/18/deaths-of-despair-and-the-future-of-capitalism
  12. Finland Toolbox (Ministry for Foreign Affairs of Finland). “Finland's Basic Income Experiment 2017-2018.” https://toolbox.finland.fi/life-society/finlands-basic-income-experiment-2017-2018/. Accessed 2 November 2026. See also Kela, “The Basic Income Experiment 2017-2018 in Finland: Final Results,” May 2020, https://toolbox.finland.fi/wp-content/uploads/sites/2/2018/04/basic_income_experiment_2017-2018_results_2020.pdf ; Big Issue, “Universal basic income boosted wellbeing and trust in Finland trial,” 6 May 2020, https://bigissue.com/news/universal-basic-income-boosted-wellbeing-and-trust-in-finland-trial/
  13. Banerjee, Abhijit; Faye, Michael; Krueger, Alan; Niehaus, Paul; Suri, Tavneet. “Universal Basic Income: Short-Term Results from a Long-Term Experiment in Kenya.” Working paper, September 2023. https://conference.nber.org/conf_papers/f192616.pdf ; summary: https://voxdev.org/topic/social-protection/universal-basic-income-kenya
  14. Vivalt, Eva; Rhodes, Elizabeth; Bartik, Alexander W.; Broockman, David E.; Krause, Patrick; Miller, Sarah. “The Employment Effects of a Guaranteed Income: Experimental Evidence from Two U.S. States.” Quarterly Journal of Economics, 141(4), 2675-2734, 2026. Working paper: NBER Working Paper 32719, July 2024 (revised August 2026), https://www.nber.org/papers/w32719
  15. Kamerāde, Daiga; Bennett, Matthew R. “Rewarding Work: Cross-National Differences in Benefits, Volunteering During Unemployment, Well-Being and Mental Health.” Work, Employment and Society, 32(1), 38-56, 2018. https://research.birmingham.ac.uk/en/publications/rewarding-work-cross-national-differences-in-benefits-volunteerin
  16. Kamerāde, Daiga; Wang, Senhu; Burchell, Brendan; Balderson, Sarah Ursula; Coutts, Adam. “A shorter working week for everyone: How much paid work is needed for mental health and well-being?” Social Science & Medicine, 241, 112353, 2019. DOI: 10.1016/j.socscimed.2019.06.006. https://www.iser.essex.ac.uk/research/publications/publication-525783
  17. Milligan, Christine; et al. “Older men and social activity: a scoping review of Men's Sheds and other gendered interventions.” Ageing & Society, Cambridge University Press. DOI: 10.1017/S0144686X14001524. https://www.cambridge.org/core/product/76C6C5DBD76D55AD2717BC4BB421AA4A
  18. Albrecht, Lisa; et al. “Men's sheds as community-based health promotion for men aged 50 plus: protocol for a mixed-methods systematic review.” 2021. https://pmc.ncbi.nlm.nih.gov/articles/PMC8336413/ ; and the subsequent mixed-methods review in American Journal of Men's Health, 2022, https://pmc.ncbi.nlm.nih.gov/articles/PMC8928410
  19. Cazzaniga, Mauro; et al. “Gen-AI: Artificial Intelligence and the Future of Work.” IMF Staff Discussion Note SDN/2024/001, International Monetary Fund, January 2024. https://www.imf.org/en/Publications/Staff-Discussion-Notes/Issues/2024/01/14/Gen-AI-Artificial-Intelligence-and-the-Future-of-Work-542379
  20. Hatzius, Jan; Briggs, Joseph; Kodnani, Devesh; Pierdomenico, Giovanni. “The Potentially Large Effects of Artificial Intelligence on Economic Growth.” Goldman Sachs Global Economics Analyst, 26 March 2023. Reported in Workplace Insight, https://workplaceinsight.net/generative-artificial-intelligence-set-to-boost-gdp-but-affect-300-million-jobs-across-major-economies
  21. Briggs, Joseph; Dong, Sarah. “How Will AI Affect the Global Workforce?” Goldman Sachs Research, 2025. https://www.marcus.com/us/en/resources/heard-at-gs/how-will-ai-affect-the-global-workforce- ; HR Leader, “Displacement, not replacement: Goldman Sachs on how AI will affect workers,” https://www.hrleader.com.au/tech/27194-displacement-not-replacement-goldman-sachs-on-how-ai-will-affect-workers
  22. Coverage of the revised 2026 Goldman Sachs displacement estimate: Allwork.Space, “Goldman Sachs says AI could displace 9% of the U.S. workforce,” July 2026, https://allwork.space/2026/07/goldman-sachs-says-ai-could-displace-9-of-the-u-s-workforce/ ; PYMNTS, “Goldman Sachs Says AI Will Eliminate 15 Million US Jobs,” 2026, https://pymnts.com/economy/2026/goldman-sachs-says-ai-will-eliminate-15-million-us-jobs ; OECD.AI Incidents Monitor entry, 3 July 2026, https://oecd.ai/en/incidents/2026-07-03-1717
  23. Eloundou, Tyna; Manning, Sam; Mishkin, Pamela; Rock, Daniel. “GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models.” arXiv:2303.10130, March 2023. https://arxiv.org/abs/2303.10130 (published in revised form in Science, 384(6702), 1306-1308, June 2024).
  24. Brynjolfsson, Erik; Chandar, Bharat; Chen, Ruyu. “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence.” Stanford Digital Economy Lab working paper, first released August 2025, revised 12 August 2026. https://digitaleconomy.stanford.edu/publications/canaries-in-the-coal-mine
  25. Keynes, John Maynard. “Economic Possibilities for our Grandchildren.” The Nation and Athenaeum, 11 and 18 October 1930. Reprinted in Essays in Persuasion. London: Macmillan, 1931.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Most experts on Stack Exchange started out as somebody asking a basic question.

That is how the system was designed. A newcomer posts something simple about a regression model or a pandas dataframe. A stranger with more reputation points answers it. Someone else edits the question so it makes sense, and a third person links a related thread. The newcomer comes back a week later with a harder question and, months on, answers someone else's basic one. Multiply that loop by a few million people and you get one of the largest bodies of freely shared technical knowledge ever assembled.

Now take the first step away. Ask the basic question to a chatbot instead. You get an answer, often a decent one, in seconds, and nobody laughs at you. But no question goes up, no stranger answers or corrects it, and the loop that might have turned you into a contributor never starts.

Multiply that by a few million people too.

A new study of seven Stack Exchange communities suggests this is not a thought experiment. In the sites where AI answers best, the share of newcomers still active a year after arriving fell by close to half after ChatGPT launched, while the core of committed regulars barely moved. A separate study of roughly 12,000 office workers found generative AI widening people's networks while also nudging them towards colleagues they already relied on. Put those findings alongside half a century of sociology on “weak ties”, and an uncomfortable question comes into focus. When machines quietly absorb the casual, low-stakes questions that once pulled strangers into a community, what happens to the community?

Seven Communities, One Natural Experiment

The Stack Exchange study is titled “Generative AI and interrupted peripheral participation: evidence of structural closure from Stack Exchange”. Its authors are Junlei Du, Shuang Li, Yinpei Yu, Xinyi He and Jingxi Chen of Beijing Normal University, and it was posted to the preprint server Research Square on 4 August 2026. The page lists it as under review. It has not yet passed peer review, and its findings should be read with that in mind.

The authors followed seven Stack Exchange sites from 2016 to 2025, grouped by how easily their knowledge can be turned into something a machine could produce and a user could check. Artificial Intelligence and Data Science are “algorithmic” communities, heavy with code. Statistics, Physics and Chemistry are “disciplinary”, heavy with formulas and derivations. Cooking and Travel are “practice-based”, built on situated experience that resists neat codification.

All seven share the same reputation system, moderation tools and interface, which lets the type of knowledge do the varying. The authors treated ChatGPT's public release in late November 2022 as a shock that should hit the codifiable communities hardest, and compared what happened before and after using a difference-in-differences design. Robustness checks included synthetic controls and placebo tests that pretend the shock happened in an earlier year. They then examined three layers at once: the network of who answers whom, the retention of different kinds of users, and the content of the questions.

Where the Halving Actually Happened

The headline figure is that newcomer one-year retention “falls by nearly half”. That phrase comes straight from the paper's abstract. The detail sits in the results tables, and it matters.

In the two algorithmic communities, the proportion of newcomers still active a year later fell from 11.5 per cent before generative AI to 6.0 per cent after it. That is a drop of 5.5 percentage points, or 47 per cent in relative terms. It is the “nearly half”.

The other groups fell too, but by less. Newcomer retention in the disciplinary communities slipped from 12.7 per cent to 9.7 per cent. In the practice-based communities it went from 6.9 per cent to 5.1 per cent. The authors describe this as a dose-response gradient: the more codifiable the knowledge, the bigger the hit. So the accurate version of the headline is narrower than it first sounds. Retention nearly halved in the communities where code answers dominate, not across all seven.

The same pattern shows up among what the authors call low-activity users, people with just one or two recorded interactions, who serve as a proxy for the periphery. Their retention fell significantly only in the algorithmic group, from 9.5 per cent to 6.2 per cent. Among high-activity users, people with ten or more events, retention did not change significantly anywhere. In the algorithmic communities it moved from 57.7 to 54.1 per cent, a shift the authors do not treat as statistically meaningful. In the practice-based communities it actually rose.

One more detail complicates the story. The share of askers posting for the first time fell fairly evenly across all three groups. The authors read this as a sign that AI “shapes who returns more than who first arrives”. People still turn up. Fewer of them stay.

That is the periphery thinning in real time. The regulars are still there; what is missing is the stream of newcomers who would have replaced them. The paper puts it bluntly: the algorithmic core shrank in absolute numbers “even though per-member core retention stayed stable, because newcomers no longer arrive at the rate needed to replenish it”.

The overall scale of the contraction is stark. By the authors' count, activity in the algorithmic communities fell by around 80 per cent from its 2020 peak. Disciplinary communities fell by 30 to 50 per cent and practice-based ones by about 40 per cent. In one network diagram, the connected core of the Data Science site collapses from 70 users in 2019 to four in 2024.

Bonding Up, Bridging Down

Retention is the human story. The network story is where the paper earns its title.

Sociologists distinguish two kinds of social capital. Bonding capital is the dense web of ties inside a tight group. Bridging capital is the thinner set of ties that cross between groups and carry information from one cluster to another. Healthy knowledge communities need both.

In the algorithmic communities, the balance tipped. Modularity, a measure of how strongly a network splits into separate clusters, rose by 0.085 in the main estimate and by 0.158 in the synthetic control version. Average path length, roughly the number of hops it takes to get from one user to another, also went up. Interaction increasingly stayed inside tight topic clusters instead of crossing between them. A third bridging measure, the E-I index, swung sharply inward in the algorithmic group, though the authors are careful to note that this indicator failed their pre-treatment trend test and so cannot carry causal weight on its own.

The disciplinary and practice-based communities barely moved on these structural measures, even though they also lost users. That contrast is the paper's strongest argument against a simple “everything is shrinking” explanation. Cooking and Travel lost about 40 per cent of their activity yet kept their shape. The algorithmic sites lost more and changed shape.

The authors call the outcome “structural closure”. The abstract summarises it like this: “bridging weakens as bonding among incumbents remains comparatively intact.” The community does not die; it closes in on itself. Regulars keep talking to regulars, and the threads that once ran outwards to occasional visitors and curious strangers fray.

They also rule out one tempting explanation. If a handful of super-users were simply hoovering up all the remaining activity, inequality in how connected people are should have risen. In the algorithmic communities it fell. This is not a power grab by an elite. It is the edge falling away.

The Slower Question

Does any of this matter if the questions still get answered? The paper suggests it does, though the authors call these links associational rather than proven.

Median time to a first answer rose by two to five hours across all three groups after generative AI arrived. Across the community-year data, higher modularity went with longer waits. A 0.1 increase in modularity was associated with roughly 3.6 extra hours before a question got its first answer. The intuition is straightforward: if the network has split into tight clusters, a question that falls between them has to travel further to reach someone who can help.

The content of the questions shifted too. The authors measured how often different topic tags appear together on the same question, a rough proxy for how much the community's knowledge is connected across subjects. That density fell by 37 per cent in the algorithmic communities. The questions that remained were also more scattered across niche topics and more focused on debugging specific code, the kind of problem a chatbot cannot solve without seeing the actual artefact.

The authors describe this as two fabrics fraying together. The social fabric of ties between subgroups and the semantic fabric of links between topics erode at the same time. In their discussion, they tie this directly to sociology's most famous argument about networks, describing the pattern as Granovetter's “weak-ties argument in reverse”.

Why Routine Questions Were Never Just Routine

To see why the authors frame this as more than a traffic problem, you need a concept from 1991.

That year, the anthropologist Jean Lave and the learning theorist Etienne Wenger published “Situated Learning: Legitimate Peripheral Participation” with Cambridge University Press. They studied apprenticeship among midwives, tailors, quartermasters, butchers and recovering alcoholics, and argued that people learn a practice by participating in it, starting at the edges. A newcomer is given small, legitimate, low-stakes tasks. Doing them in the presence of old-timers lets the newcomer absorb how the community thinks and talks, and gradually move towards the centre.

Stack Exchange, the Beijing Normal team argues, worked in a similar way. Asking a routine question in public “exposes a newcomer to public reasoning, establishes an initial interaction, and creates the possibility of return”. It is the entry ticket.

A large language model, they write, “separates the informational outcome from this social process”. You get the answer without the exchange. The edge-of-the-room apprenticeship never begins, because there is no room.

Separate evidence supports the idea that answers recruit contributors. In a paper posted to arXiv in April 2026, Lenard Strahringer, Sven Eric Prüß and Kai Riemer analysed more than 21 million Stack Overflow questions and found that “receiving an answer significantly increases a user's propensity to help others”. The effect, they reported, “is concentrated among newcomers and declines with platform experience”, and peaks when answers arrive within roughly 30 to 60 minutes. Being helped is how newcomers become helpers. Veterans do not need the nudge. Remove the moment of being helped, and the recruitment pipeline loses its first stage.

There is an honest caveat here, and the authors state it up front: “individual decisions to use an LLM are not directly observed.” What the study shows is a community-level pattern, timed to ChatGPT's release and concentrated where AI answers are easiest to generate and check. It is strong circumstantial evidence, not proof.

The Weak Ties Granovetter Counted

The sociological backbone of this argument is Mark Granovetter's 1973 paper “The Strength of Weak Ties”, published in the American Journal of Sociology. Its core logic explains why losing a periphery is more than losing numbers.

Granovetter's argument is structural. Your close friends tend to know each other, so they mostly hear the same news you do. Your acquaintances move in different circles. In networks of any size, he showed, a strong tie is almost never the only route between two parts of a network, because close friends share other friends. In his words, “all bridges are weak ties.” The loose connections are the ones that carry new information between groups.

He tested this on job seekers in a Boston suburb. Among the respondents he interviewed in person who had found work through a personal contact, 16.7 per cent said they saw that contact often at the time, 55.6 per cent occasionally and 27.8 per cent rarely. The subsample was small: 54 people from a total sample of 282. But the skew was clear. The useful contact was usually someone on the edge of a person's life, such as an old college friend or a former colleague.

The paper's closing paragraph contains a line that could almost caption the Stack Exchange network diagrams: “strong ties, breeding local cohesion, lead to overall fragmentation.”

Half a century later, the theory got its large-scale causal test. In a paper published in Science in September 2022, Karthik Rajkumar, Guillaume Saint-Jacques, Iavor Bojinov, Erik Brynjolfsson and Sinan Aral analysed randomised experiments on LinkedIn's “People You May Know” feature. Over five years these experiments varied the share of weak ties in the networks of more than 20 million people, a period in which about two billion new ties and 600,000 new jobs were created. Weak ties did increase job mobility, with caveats. When tie strength was measured by mutual connections, the relationship was an inverted U: moderately weak ties helped most. Weak ties also mattered more in digital industries, and strong ties more in less digital ones.

So the familiar claim that weak ties are the lifeblood of mobility has solid support. For innovation and social inclusion it is a reasonable reading of Granovetter's wider argument, not something these studies measured directly. He called weak ties “indispensable to individuals' opportunities and to their integration into communities”. The careful version is that weak ties are the main way information and opportunity cross social boundaries. On Stack Exchange, a stranger answering a newcomer's question was a weak tie in its purest form.

Inside a Company Rolling Out Copilot

The second strand of evidence moves from public forums into a corporate office, and the picture there is more complicated than it is often made to sound.

“The Effects of Generative AI on Collaboration and Social Networks Among Knowledge Workers” was posted to arXiv on 23 October 2026 by Giuseppe Soda and Alessandro Iorio of Bocconi University and Leonardo Rizzo of SDA Bocconi School of Management. Like the Stack Exchange paper, it is a preprint.

The authors studied a single business unit of a large multinational consumer products company, which they do not name. The unit started with 12,040 employees. Between September 2024 and September 2025, the company rolled out Microsoft Copilot in stages, which gave the researchers a natural experiment. Their final sample was 11,868 workers: 1,992 who adopted Copilot and 9,876 who never did. Using passively collected data on who collaborated with whom, and for how long, they tracked what happened to each adopter's network.

They tested two competing predictions: that AI replaces human interaction, or that it generates more, because its outputs need interpreting, checking and approving by other people. The second won. Adopters collaborated with about 2.18 more colleagues on average than comparable non-adopters. Their networks became less clustered, meaning their contacts were less likely to know each other, which the authors read as broader collaboration. Total collaboration time went up too, by 0.87 hours in the post-adoption average.

The third result is the one that matters most here. Adopters, the authors write, “also concentrated more of their collaboration on ties with stronger prior histories.” Put simply, Copilot users leaned harder on colleagues they already had a track record with, which the authors describe as deepening reliance on already trusted colleagues. The effect is clearer in some models than others. One summary estimate falls short of conventional statistical significance, while the post-adoption average and the instrumental-variable model both reach it.

The picture, then, is a network that is both wider and more anchored on trusted people. The authors' own reading is that AI output creates new demand for human judgement, and people send that demand towards colleagues whose judgement they already trust.

What the Office Study Did Not Measure

It is tempting to go further and say this research shows the casual, cross-topic exchanges that connect different social worlds eroding. It is worth saying plainly that the paper does not show that.

Soda, Iorio and Rizzo did not analyse weak ties, casual ties or cross-topic ties. In fact their method screened out many of them. To reduce what they call “noise from incidental interactions”, they kept only ties above the monthly median weight, about one hour of collaboration. A quick corridor chat or a two-line message to someone in another department is exactly the kind of contact a filter like that is likely to remove. The study cannot tell us whether those contacts shrank, grew or stayed the same.

The authors are also candid about other limits. “We know who interacts with whom and for how long, but not what they talk about,” they write. Their data cover one business unit in one company, and they say it is “not clear” whether the findings apply to other industries or workplaces. The results cover the first three months after each person adopted the tool.

What can fairly be said is narrower: in this organisation, Copilot users reached more people while concentrating more of their sustained collaboration on familiar partners. There is a related precedent. When Longqi Yang, David Holtz, Sonia Jaffe, Siddharth Suri and colleagues studied 61,182 Microsoft employees during the shift to remote work in the first half of 2020, they found collaboration became more static and siloed. Time spent on cross-group collaboration fell by about 25 per cent of its pre-pandemic level, and employees added fewer new collaborators. That paper, published in Nature Human Behaviour, was about remote work, not AI, and found narrower networks rather than wider ones. But it shows how quickly a change in working conditions can pull people towards their inner circles. Whether Copilot does the same to the casual edges of a network is a question the Bocconi study raises without answering.

The Volume Collapse Around It

The Beijing Normal paper builds on two earlier peer-reviewed studies that measured the drop in Stack Overflow activity directly.

In a paper published in PNAS Nexus in September 2024, R. Maria del Rio-Chanona, Nadzeya Laurentsyeva and Johannes Wachs compared Stack Overflow with sites where ChatGPT was less available or less capable, such as its Russian and Chinese-language equivalents. They estimated that posting on Stack Overflow fell by about 25 per cent within six months of ChatGPT's release, relative to those comparisons, and called that a lower bound. Their title states the stakes plainly: “Large language models reduce public knowledge sharing on online Q&A platforms.”

One finding complicates the newcomer story: posts from first-time users fell only slightly, while inexperienced, experienced and expert users all posted significantly less. That does not contradict the Beijing Normal results, which concern whether newcomers stay rather than whether they post once, but it shows how much depends on what is measured.

In Scientific Reports in May 2024, Gordon Burtch, Dokyun Lee and Zhichen Chen of Boston University's Questrom School of Business estimated that Stack Overflow lost about a million daily visitors, roughly 12 per cent of its traffic, after ChatGPT arrived. The decline was concentrated among newer users. Their most striking finding was that Reddit's developer communities showed “no evidence” of a similar drop. The authors attributed the difference to “social fabric”. Reddit's communities, they argued, have stronger social bonds, which act as “a buffer against the community-degrading effects of LLMs”. That buffer is bonding capital, the kind the Beijing Normal team saw surviving; bonding without bridging is what Granovetter warned against.

The raw numbers since then have been brutal. In May 2025, the engineering newsletter The Pragmatic Engineer, citing a Stack Exchange Data Explorer query shared by the long-time contributor Marc Gravell, reported that Stack Overflow was on course for about 14,000 questions that month. That is fewer than the 17,668 it received in September 2009, in the site's infancy. The newsletter's author, Gergely Orosz, noted that the decline began around 2014 and sped up from mid-2020, well before ChatGPT. The fall has not slowed since. In January 2026, Tim Anderson of DevClass, drawing on Stack Overflow's public query tool, reported that only 3,862 questions were posted in December 2025, a drop of 78 per cent on the previous year. In August 2026, Luís Rijo of PPC Land reported a comparison posted on X by the developer Daniel Lockyer, using the Stack Exchange Data Explorer: 1,442 new questions in July 2026, about 0.7 per cent of the 207,204 posted at the site's peak in March 2014. The whole of 2025 produced 108,981 questions, little more than half the total for March 2014 alone. AI did not start the slide. It turned a slow decline into a fast one.

Nor is the pattern limited to programmers. In October 2025, Marshall Miller of the Wikimedia Foundation reported that human pageviews on Wikipedia were down about 8 per cent on the same months of 2024. He linked this to search engines answering questions directly, often using Wikipedia's own content, and to younger people searching on social video. His warning about the consequences was explicit: with fewer visits, “fewer volunteers may grow and enrich the content”. Editors are recruited from readers; fewer readers means a smaller pool of future editors.

A Periphery That Was Already Fraying

It would be too easy to cast generative AI as the villain that wrecked a welcoming commons. Stack Overflow's periphery had problems long before ChatGPT.

In April 2018, Jay Hanlon wrote a post on Stack Overflow's official blog under the headline “Stack Overflow Isn't Very Welcoming. It's Time for That to Change.” He admitted that “too many people experience Stack Overflow as a hostile or elitist place”, especially “newer coders, women, people of color, and others in marginalized groups”. He wrote that “too often, someone comes here to ask a question, only to be told that they did it wrong”, and that “condescension and sarcasm have been reluctantly tolerated in comments for too long.”

Wikipedia went through something similar earlier. In a 2013 paper in American Behavioral Scientist, Aaron Halfaker, R. Stuart Geiger, Jonathan Morgan and John Riedl found that the encyclopaedia's decline in active editors was driven by falling retention of good-faith newcomers, not by fewer newcomers arriving. They pointed to Wikipedia's quality-control tools, including algorithmic tools used to reject contributions, as part of the cause.

Both cases show communities damaging their own on-ramps with gatekeeping that feels reasonable to insiders. Part of what AI is doing is exploiting a weakness that already existed. A newcomer choosing between a chatbot that never sneers and a forum that might close their question as a duplicate is not making an irrational choice.

That cuts both ways. It weakens any nostalgic case for the pre-AI periphery, and it suggests the communities now losing their edge had been running down their own renewal for years.

The Case for the Machine Answer

Does the convenience of an instant answer outweigh the long-term cost to community renewal? Any honest answer has to take the convenience seriously.

For the individual asker, the chatbot is often better. It is available at three in the morning, never demands a minimal reproducible example, and will explain the same concept five ways. For people who were made to feel unwelcome, which by Stack Overflow's own admission included many women, people of colour and beginners, that is not trivial. A learner who used to give up after a hostile reception can now keep going.

The communities that remain may also be doing more interesting work. Denis Helic and Tiago Santos, in a paper first posted in September 2025 and later published in the Journal of Systems and Software, found that ChatGPT had “effectively raised the bar for questions on Stack Overflow”, with question difficulty, length and code length all increasing. The routine traffic left. The hard problems stayed. The Beijing Normal team's finding that remaining questions skewed towards debugging specific code fits the same pattern.

Developers have not stopped wanting other people either. Stack Overflow's 2026 Developer Survey, which drew 30,903 responses from 169 countries, found that 73 per cent of respondents now use coding assistants or agents every day. Their trust is conditional. Some 48 per cent said they trust AI when they can easily validate its answers, and 79 per cent rated source attribution important or very important. A year earlier, more developers had distrusted the accuracy of AI output, 46 per cent, than trusted it, 33 per cent. Asked which actions they turn to first when trying to find answers to a problem, 70 per cent chose an AI agent, but 29 per cent chose to send a message directly to a human they trust, and the people they work with remained their most trusted source of information, at 72 per cent. A mixed-methods study posted to arXiv in March 2026 by Ching Christie Pang, Xuetong Wang, Yuk Hang Tsui and Pan Hui, based on a survey of 217 people and 11 interviews, reached a similar conclusion. Users rely on AI for convenience but still go to communities for questions that are complex, ambiguous or depend on trust.

So the human community is not obsolete. It is being pushed upmarket, towards hard cases and trust-sensitive judgement. The problem is that its future experts have to come from somewhere. The easy questions were the training ground. That is where convenience runs into renewal. The individual gain is immediate and visible; the collective cost is delayed and shows up only in network statistics years later.

Training Data Eats Its Own Source

The loss of the periphery has a second-order cost the AI industry has its own reasons to care about. Public question-and-answer sites are a major source of training data for large language models. The Beijing Normal authors note that “sustained bridging erosion thins the very base future AI depends on.” The PNAS Nexus team made a parallel argument: less open posting means less public data to train future models, and some of that knowledge now sits inside privately owned AI systems instead.

The commercial relationship makes the loop explicit. On 6 May 2024, Stack Overflow and OpenAI announced a partnership giving OpenAI access to Stack Overflow's data through its OverflowAPI product, with Stack Overflow using OpenAI models to build its own AI features. Google had earlier signed up as a customer of the same API. The questions and answers that newcomers and regulars wrote for each other for free became a licensed asset sold to the companies whose products are now absorbing the newcomers.

The irony needs no exaggeration. The models answer routine questions well partly because millions of people once asked them in public and strangers answered. If the public asking stops, the archive stops growing while the software keeps changing. Next year's routine question about a framework that does not yet exist will need someone to have answered it first.

Designing an On-Ramp Back In

If the periphery is worth saving, the evidence points to a few practical moves rather than nostalgia.

The first comes from the Beijing Normal paper. Its central principle is that “AI assistance should protect rather than absorb the periphery's entry path”. In practice the authors propose an in-community assistant that answers routine questions from the community's own verified knowledge, and then points the asker towards the people who wrote it. The aim is to turn a one-off answer into a first contact.

There is early evidence this can work. In a paper accepted at the CSCW 2026 conference, Junling Wang, Lahari Goswami, Gustavo Kreia Umbelino, Kiara Chau, Mrinmaya Sachan and April Yi Wang built a chatbot that brings Kaggle community content and social features into its answers. Across two studies with 28 and 12 data science learners, it increased user trust and encouraged engagement with the community. The samples are small and the setting is not a live platform, but a chatbot can evidently be built as a bridge rather than a replacement.

The second move is to measure the right thing. The Beijing Normal authors found the structural shift appeared before activity hit bottom, and suggest modularity and path length could serve as early-warning signals alongside volume dashboards. Platforms that only watch question counts will notice the decline after the periphery has already gone.

The third is targeted intervention. Because the damage concentrates where knowledge can be expressed as checkable code, the authors would put the strongest safeguards in algorithmic communities. They also propose recommending collaborators across topics based on the content of their work rather than existing ties, an explicit attempt to manufacture bridges.

Inside organisations, the Copilot study suggests a parallel lesson. If AI pushes people towards trusted colleagues for judgement, managers who value cross-boundary knowledge may need to create reasons for different groups to meet.

And communities have to make the edge worth staying at. Burtch and colleagues recommended “enabling socialization, as a complement to pure information exchange”. Belonging only among insiders is closure, so the welcome has to reach newcomers too.

Communities Renew From the Edge

Can knowledge communities survive the loss of their periphery? The best current evidence suggests they can survive in the short term, in the narrow sense that the core keeps going. The regulars stay, the hard questions still get asked, and the bonds among incumbents hold. What looks threatened is renewal: the slow, unglamorous process by which strangers become members and members become experts.

The evidence has limits. The Stack Exchange study is a preprint covering seven communities, only two of which carry the dramatic result, and it infers rather than observes individual behaviour. The Copilot study covers one business unit in one company for a few months, and it did not measure casual cross-topic ties at all. Granovetter's insight is half a century old and was built on a small interview sample, though the LinkedIn experiments have since confirmed it at enormous scale. None of this proves that AI is hollowing out every knowledge community.

But the pieces line up uncomfortably well. Routine questions recruited newcomers. Newcomers formed the weak ties that bridged clusters, and bridges kept knowledge moving and the core replenished. Generative AI answers routine questions instantly and privately. In the places where it does that best, newcomers are staying at roughly half the old rate, clusters are closing in on themselves and questions are taking longer to reach the right person.

Cohesion without bridges, Granovetter warned, ends in fragmentation. Knowledge communities have always been renewed by people who arrived knowing very little and asked basic questions in public. If those people now ask a machine instead, the communities will keep their experts for a while. What they risk losing is the way new experts were made.


References and Sources

  1. Du, Junlei; Li, Shuang; Yu, Yinpei; He, Xinyi; Chen, Jingxi (Beijing Normal University). “Generative AI and interrupted peripheral participation: evidence of structural closure from Stack Exchange.” Research Square preprint (under review, not peer reviewed), posted 4 August 2026. DOI: 10.21203/rs.3.rs-10221893/v1. https://www.researchsquare.com/article/rs-10221893/v1 (PDF: https://www.researchsquare.com/article/rs-10221893/latest.pdf). Accessed 30 October 2026.
  2. Soda, Giuseppe; Iorio, Alessandro (Bocconi University); Rizzo, Leonardo (SDA Bocconi School of Management). “The Effects of Generative AI on Collaboration and Social Networks Among Knowledge Workers.” arXiv preprint 2610.32461v1, 23 October 2026. https://arxiv.org/abs/2610.32461 (HTML full text: https://arxiv.org/html/2610.32461). Accessed 30 October 2026.
  3. Granovetter, Mark S. “The Strength of Weak Ties.” American Journal of Sociology, 78(6), 1360–1380, May 1973. University of Chicago Press. JSTOR stable URL: https://www.jstor.org/stable/2776392 (full text copy: https://www.cs.umd.edu/~golbeck/INST633o/granovetterTies.pdf). Accessed 30 October 2026.
  4. Rajkumar, Karthik; Saint-Jacques, Guillaume; Bojinov, Iavor; Brynjolfsson, Erik; Aral, Sinan. “A causal test of the strength of weak ties.” Science, 377(6612), 1304–1310, 16 September 2022 (published online 15 September 2022). DOI: 10.1126/science.abl4476. Summary: Stanford Digital Economy Lab, https://digitaleconomy.stanford.edu/publication/a-causal-test-of-the-strength-of-weak-ties Accessed 30 October 2026.
  5. MIT Initiative on the Digital Economy. “New Publication: MIT, Harvard and Stanford Scientists Show Weaker Ties Are More Beneficial for Job Seekers on LinkedIn.” 15 September 2022. https://ide.mit.edu/insights/new-publication-mit-harvard-and-stanford-scientists-show-weaker-ties-are-more-beneficial-for-job-seekers-on-linkedin/ Accessed 30 October 2026.
  6. Lave, Jean; Wenger, Etienne. Situated Learning: Legitimate Peripheral Participation. Learning in Doing series. Cambridge University Press, 1991. DOI: 10.1017/CBO9780511815355. https://doi.org/10.1017/CBO9780511815355 Accessed 30 October 2026.
  7. del Rio-Chanona, R. Maria; Laurentsyeva, Nadzeya; Wachs, Johannes. “Large language models reduce public knowledge sharing on online Q&A platforms.” PNAS Nexus, 3(9), pgae400, published 11 September 2024. DOI: 10.1093/pnasnexus/pgae400. https://academic.oup.com/pnasnexus/article/3/9/pgae400/7754871 Accessed 30 October 2026.
  8. Burtch, Gordon; Lee, Dokyun; Chen, Zhichen (Questrom School of Business, Boston University). “The consequences of generative AI for online knowledge communities.” Scientific Reports, 14, 10413, published 6 May 2024. DOI: 10.1038/s41598-024-61221-0. https://pmc.ncbi.nlm.nih.gov/articles/PMC11074245/ Accessed 30 October 2026.
  9. Strahringer, Lenard; Prüß, Sven Eric; Riemer, Kai. “Help Converts Newcomers, Not Veterans: Generalized Reciprocity and Platform Engagement on Stack Overflow.” arXiv preprint 2604.03209, 3 April 2026. https://arxiv.org/abs/2604.03209 Accessed 30 October 2026.
  10. Yang, Longqi; Holtz, David; Jaffe, Sonia; Suri, Siddharth; Sinha, Shilpi; Weston, Jeffrey; Joyce, Connor; Shah, Neha Parikh; Sherman, Kevin; Hecht, Brent; Teevan, Jaime. “The effects of remote work on collaboration among information workers.” Nature Human Behaviour, 6, 43–54, published online September 2021. DOI: 10.1038/s41562-021-01196-4. Microsoft Research summary: https://www.microsoft.com/en-us/research/publication/the-effects-of-remote-work-on-collaboration-among-information-workers/ Accessed 30 October 2026.
  11. Berkeley Haas Newsroom / Greater Good Science Center. “How Remote Work Affects Our Communication and Collaboration.” 2021. https://greatergood.berkeley.edu/article/item/how_remote_work_affects_our_communication_and_collaboration Accessed 30 October 2026.
  12. Orosz, Gergely. “The Pulse #134: Stack Overflow is almost dead.” The Pragmatic Engineer, 15 May 2025 (citing Stack Exchange Data Explorer data shared by Marc Gravell). https://newsletter.pragmaticengineer.com/p/the-pulse-134 Accessed 30 October 2026.
  13. Anderson, Tim. “Dramatic drop in Stack Overflow questions as devs look elsewhere for help.” DevClass, 5 January 2026 (citing Stack Overflow's public query tool). https://devclass.com/2026/01/05/dramatic-drop-in-stack-overflow-questions-as-devs-look-elsewhere-for-help Accessed 2 November 2026.
  14. Rijo, Luís. “Stack Overflow drops to 1,442 questions in July, down 99% from 2014 peak.” PPC Land, 8 August 2026 (citing Stack Exchange Data Explorer figures posted on X by Daniel Lockyer, 7 August 2026). https://ppc.land/stack-overflow-drops-to-1-442-questions-in-july-down-99-from-2014-peak/ Accessed 2 November 2026.
  15. Miller, Marshall. “New User Trends on Wikipedia.” Diff, Wikimedia Foundation, 17 October 2025 (updated 13 February 2026). https://diff.wikimedia.org/2025/10/17/new-user-trends-on-wikipedia/ Accessed 30 October 2026.
  16. Computing. “Wikipedia reports traffic drop, blames AI search and social media.” October 2025. https://www.computing.co.uk/news/2025/ai/wikipedia-reports-traffic-drop Accessed 30 October 2026.
  17. Hanlon, Jay. “Stack Overflow Isn't Very Welcoming. It's Time for That to Change.” Stack Overflow Blog, 26 April 2018. https://stackoverflow.blog/2018/04/26/stack-overflow-isnt-very-welcoming-its-time-for-that-to-change/ Accessed 30 October 2026.
  18. Halfaker, Aaron; Geiger, R. Stuart; Morgan, Jonathan T.; Riedl, John. “The Rise and Decline of an Open Collaboration System: How Wikipedia's Reaction to Popularity Is Causing Its Decline.” American Behavioral Scientist, 57(5), 664–688, 2013. Research summary: https://meta.wikimedia.org/wiki/Research:The_Rise_and_Decline (University of Minnesota release: https://cse.umn.edu/college/news/wikipedia-inadvertently-causing-its-own-decline-participation). Accessed 30 October 2026.
  19. Helic, Denis; Santos, Tiago. “Stack Overflow Is Not Dead Yet: Crowd Answers Still Matter.” arXiv preprint 2509.05879 (v1 7 September 2025; v2 16 June 2026); journal version in Journal of Systems and Software, 241, 112988, 2026. https://arxiv.org/abs/2509.05879 Accessed 30 October 2026.
  20. Stack Overflow. “2026 Developer Survey” (sections: AI, https://survey.stackoverflow.co/2026/ai ; Knowledge, https://survey.stackoverflow.co/2026/knowledge ; Community, https://survey.stackoverflow.co/2026/community). 2026. https://survey.stackoverflow.co/2026/ Accessed 2 November 2026.
  21. Stack Overflow. “2025 Developer Survey: AI.” 2025. https://survey.stackoverflow.co/2025/ai Accessed 30 October 2026.
  22. Pang, Ching Christie; Wang, Xuetong; Tsui, Yuk Hang; Hui, Pan. “The Decline of Online Knowledge Communities: Obstacles, Workarounds, and Sustainability.” arXiv preprint 2603.27399, 28 March 2026. https://arxiv.org/abs/2603.27399 Accessed 30 October 2026.
  23. SD Times. “Stack Overflow and OpenAI announce partnership to enable ChatGPT to use knowledge from Stack Overflow answers.” 6 May 2024. https://sdtimes.com/ai/stack-overflow-openai-partnership-chatgpt-use-knowledge-from-stack-overflow-answers/ Accessed 30 October 2026.
  24. TechRadar. “Stack Overflow signs deal with OpenAI to strengthen models on its data.” May 2024. https://www.techradar.com/pro/stack-overflow-signs-deal-with-openai-to-strengthen-models-on-its-data Accessed 30 October 2026.
  25. Wang, Junling; Goswami, Lahari; Umbelino, Gustavo Kreia; Chau, Kiara; Sachan, Mrinmaya; Wang, April Yi. “Bridging Instead of Replacing Online Coding Communities with AI through Community-Enriched Chatbot Designs.” arXiv preprint 2601.18697 (accepted at CSCW 2026), 26 January 2026. https://arxiv.org/abs/2601.18697 Accessed 30 October 2026.

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Half a percentage point. That is how much a year of schooling has lost in wage value, in the American occupations most exposed to generative AI, since ChatGPT arrived in November 2022. It sounds like a rounding error. It is not. It is one of the first direct estimates of something economists have suspected for three years: the machine is coming for the part of your education that can be written down.

The number comes from a working paper posted to arXiv on 26 October 2026 by Eduardo Levy Yeyati, Darío Judzik and Juan Ignacio Barbiero of Universidad Torcuato Di Tella in Buenos Aires. It is titled “Practice Makes Perfect: Tacit Knowledge, Generative AI, and Returns to Schooling and Experience”, and it does something deceptively simple. It takes the workhorse equation of labour economics, the one that tells you how much an extra year in a classroom or an extra year on the job adds to your pay, and asks whether its coefficients moved when large language models went mainstream.

They did. In highly exposed occupations, the return to a year of schooling fell by 0.55 log points, roughly 7 per cent of what it had been. The return to experience did something stranger. It did not rise or fall uniformly. It rotated, like a seesaw. Early-career experience became more valuable. Late-career experience became less so. The pivot sits at about 18 years.

Put that next to two other pieces of 2026 research and a picture forms. PwC's Global AI Jobs Barometer, built from more than a billion job advertisements, finds the labour market splitting into two tracks, with the jobs where AI raises the bar for expertise growing twice as fast as the jobs where AI lowers it. And a field experiment led by the MIT economist David Autor finds that when patent lawyers trained alongside an AI assistant were later asked to work without it, the senior lawyers kept their gains while the juniors, on average, did not.

Together, the three studies describe a repricing. Codified knowledge, the stuff a degree certifies, is getting cheaper because a machine can supply it on demand. Tacit judgement, the stuff you only get by doing the work, is getting dearer, but it has a shelf life. For anyone who has just borrowed tens of thousands to buy a credential, that is uncomfortable news. Read carefully, it is also not the catastrophe the headline numbers suggest.

Seven and a Half Points a Year

To understand what moved, you need to know what was being measured. In 1974 the Columbia economist Jacob Mincer published “Schooling, Experience, and Earnings”, a slim book from the National Bureau of Economic Research that gave the discipline its workhorse. Mincer's insight was that if you take the logarithm of a person's earnings, you can model it as a straight-line function of their years of schooling plus a curved function of their years of work experience. The curve is concave: experience pays a lot early, then the gains flatten as people stop investing in new skills.

The schooling coefficient has an intuitive reading. Because the left-hand side is in logs, a coefficient of 0.075 means each additional year of school is associated with roughly 7.5 per cent higher hourly pay, holding experience constant. Economists call that 7.5 log points. Half a century of refinement later, it remains the default lens for asking what education is worth.

This matters for reading the new paper correctly, because the units are easy to garble. Levy Yeyati and his colleagues report that before ChatGPT, the return to a year of schooling in the most AI-exposed occupations was 7.5 log points. In the least exposed occupations it was just 1.7. After the release, the exposed-group figure fell by 0.55 log points relative to the control group. That does not mean wages fell by 0.55 per cent, nor that the degree premium collapsed by 7 per cent of your salary. It means the slope of pay against schooling flattened, from roughly 7.5 points per year to roughly 7 points per year. The 7 per cent figure is the decline in that slope relative to where it started: 0.55 divided by 7.5 is about 7.3 per cent.

A rough illustration helps. A four-year degree in a highly exposed occupation was previously associated with about 30 log points more pay than a high school diploma, all else equal (four years at 7.5 points each). At the post-shock rate, that becomes about 28 points. That arithmetic is mine, not the authors', and it ignores everything else that changed, but it conveys the order of magnitude. The degree still pays. It pays a little less, and in exactly the places where the technology bites hardest.

How a Degree Lost Half a Point

The study draws on the US Current Population Survey's outgoing rotation groups, the monthly household survey that asks a subset of respondents about their earnings. The authors use data from 2018 through June 2026, covering 655,919 observations of people aged 16 to 64 with a valid hourly wage. They date the shock to November 2022 and organise their years from November to October so that the ChatGPT release falls cleanly on a boundary.

To decide which occupations count as exposed, they use an index built by Edward Felten, Manav Raj and Robert Seamans and published in the Strategic Management Journal in 2021. Crucially, that measure predates ChatGPT, so it cannot have been contaminated by what happened afterwards. The paper compares occupations in the top quartile of exposure, which include software developers, financial managers and lawyers, with those in the bottom quartile, which include truck drivers, construction labourers and waiters. The middle two quartiles are set aside.

The core specification is a Mincer equation with the schooling and experience terms interacted with two switches: whether the occupation is exposed, and whether the observation comes after the release. The authors control for a long list of demographic and job characteristics, with fixed effects for month, state and occupation.

One technical choice deserves attention because it affects the result. High earners in the survey are top-coded: above a certain threshold, the Census Bureau records a cap rather than the true figure. That cap changed in April 2023, from a fixed value to a dynamic one, which happens to fall in the post-shock window. Top-coding is concentrated in the treated occupations: 11.5 per cent of that group is censored, against 0.7 per cent of the controls. The authors handle it with interval-censored maximum likelihood rather than imputation, and they test alternatives. Applying the pre-2023 cap throughout gives a schooling effect of minus 0.53 log points, almost identical to the headline. But a deliberately harsh check, which treats dynamically top-coded observations as exact values, shrinks the schooling estimate to minus 0.09. The authors flag this openly. It is the most important caveat in the paper, and anyone quoting the 0.55 figure should know it exists.

Other checks are reassuring. The pre-release schooling coefficients show no trend, and the break appears in the first year after release. A placebo that re-dates the shock to January 2020, to test whether the pandemic is doing the work, finds nothing. Dropping the technology sector “does little”. Swapping the Felten exposure index for the “GPTs are GPTs” measure developed by Tyna Eloundou and colleagues at OpenAI and the University of Pennsylvania gives a schooling decline of 0.42 log points, close to the baseline.

Experience Turns on a Hinge at Eighteen Years

The schooling result is the headline, but the experience result is stranger. In a standard Mincer setup, the value of an extra year of experience depends on how much you already have. A third year adds more than a twenty-third. What Levy Yeyati, Judzik and Barbiero find is that the shape of that curve changed after November 2022 in exposed occupations. At three years of potential experience, the return to an additional year rose by 0.26 log points. At five years, it rose by 0.23. At fifteen, the change is statistically indistinguishable from zero. At twenty-five years, it fell by 0.12. Under the baseline quadratic specification, the effect crosses zero at 18.1 years, with a standard error of 2.1.

The authors are careful about what this means. It is a change in the slope of the experience profile, not a statement that workers with 18 years of experience are the break-even point for winners and losers in pay. “Potential experience” is also a blunt instrument: it is simply age minus years of schooling minus six. It counts time since leaving education, not time spent doing the tasks AI now touches.

The heterogeneity analysis sharpens the picture. Among workers with an associate degree or higher, the early-career return rises by 0.43 log points at three years, and the late-career return falls by 0.12 at twenty-five. Among workers without a credential, the early gain is smaller and the late decline is not statistically significant. In other words, the rotation is strongest precisely among people who hold formal qualifications. The authors write that “wage differences by early-career experience become steeper after the shock” within that credentialed group.

Finally, the level effect. For a worker with 16 years of schooling and 15 years of experience in a treated occupation, the estimated wage change is minus 3.2 log points. That is a real cut, and it falls on exactly the profile of a mid-career graduate professional. But the authors stress that the level effect “is not itself a theoretical prediction”, and they say plainly that “the findings are not a forecast of long-run returns to education or experience.”

Codified Knowledge Was Always the Cheap Part

The theory behind the paper rests on an old distinction. In 1966 the chemist and philosopher Michael Polanyi published “The Tacit Dimension”, which contains the line “we can know more than we can tell.” Some knowledge can be written down, taught in lectures and tested in exams. Some can only be acquired by doing: the feel for when a contract clause will cause trouble, the instinct for which bug report is really an architecture problem, the sense of when a patient's symptoms do not add up.

In 2014 David Autor borrowed Polanyi's line for an NBER working paper titled “Polanyi's Paradox and the Shape of Employment Growth”. His argument was that computers had automated tasks that could be specified as explicit rules, and had struggled with tasks requiring tacit knowledge. That pattern, he argued, explained why employment had grown at the top and bottom of the wage distribution and hollowed out in the middle.

Generative AI scrambles that story. Large language models are not programmed with rules; they learn from enormous quantities of text, absorbing much of what humanity has managed to write down. They are, in effect, machines for supplying codified knowledge at near-zero marginal cost. Ask one to explain the doctrine of equivalents in patent law or the assumptions behind a discounted cash flow model, and it will usually produce a competent answer in seconds.

The Levy Yeyati model formalises three consequences. First, substitution: machine and worker codified knowledge are treated as perfect substitutes, so when the machine version becomes abundant, the worker version, proxied by schooling, loses scarcity value. Second, complementarity: practice-based judgement becomes more valuable when the codified knowledge it works on gets cheaper, steepening the early-career slope. Third, obsolescence: experience tied to the old way of doing the work loses value when the technology changes, flattening the late-career slope.

The paper concedes its estimates do not separately identify these mechanisms. The pattern is consistent with all three acting together, but does not prove it. Nor does the paper distinguish fields of study or institutional quality. A year of organic chemistry and a year of media studies count the same.

A Labour Market Splitting in Two

If the Levy Yeyati paper measures what workers are paid, PwC's 2026 Global AI Jobs Barometer measures what employers are asking for. Released on 15 June 2026, it analyses more than one billion job advertisements across 27 countries and territories on six continents.

Its central finding is a fork. In some occupations, which PwC calls professionalised, AI automates the routine parts of the job and pushes the remaining human work towards judgement and expertise. PwC's examples include radiologists and recruiters. In others, which it calls democratised, AI makes the work easier for non-experts to perform, lowering skill barriers and shifting roles towards less specialised activity. Its examples include IT service managers and medical secretaries.

According to PwC Belgium's release of the global results, “the 22% of jobs that are being professionalised are growing twice as fast as democratised jobs (52% of jobs)”, while “seeing 42% higher wage growth since 2021.”

Read alongside the earnings data, this is striking. The democratised track looks like the substitution story: AI takes over the codified core of a job, and the premium for having learned it shrinks. The professionalised track looks like complementarity: AI clears away routine work and leaves a residue of judgement that commands a premium. That the professionalised track is smaller but faster growing and better paid fits the idea that value is migrating towards tacit expertise.

PwC also reports that the average wage premium for AI skills in job postings rose to 62 per cent, from 57 per cent the previous year. Pete Brown, PwC's Global Workforce Leader, put the implication directly: “Organisations need to rethink how they develop talent if they want people to thrive in this new environment.”

Job advertisements capture what employers say they want, not what they pay individual workers, and PwC sells AI advisory services. Its findings are a large signal about demand, not a peer-reviewed causal estimate.

Junior Jobs That Ask for Senior Skills

The most pointed PwC finding concerns the bottom rung. Analysing 2.4 million entry-level job postings in the United States, the barometer finds that junior roles highly exposed to AI are seven times more likely to require “traditionally senior skills”, such as leadership and strategic thinking, than entry-level roles that are not exposed. Some accounts express this as 52 per cent of new skills against 7 per cent. That ratio fits the sevenfold headline, but the breakdown does not appear in PwC's public summaries, so the sevenfold figure is the safer one to quote.

Openings for these “seniorised” roles have grown 35 per cent since 2019, while other entry-level openings have fallen 10 per cent. The entry-level job is being redefined: employers want new hires who can do on day one what used to take years to learn.

Other data point the same way. Erik Brynjolfsson, Bharat Chandar and Ruyu Chen of Stanford's Digital Economy Lab have been tracking employment using payroll data from ADP, the largest payroll processor in the United States. Their paper “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence”, first released in August 2025 and revised on 12 August 2026 with data through June 2026, finds no evidence of widespread, economy-wide job displacement. But employment of workers aged 22 to 25 in AI-exposed occupations is now 19 per cent below where it would be had it kept pace with less exposed peers. Experienced workers show no comparable gap. The adjustment works mainly through reduced hiring of young people rather than layoffs, and it is concentrated where AI substitutes for human tasks rather than complementing them.

The Federal Reserve Bank of New York's tracker of the labour market for recent college graduates tells a similar story. In the third quarter of 2026, the unemployment rate for recent graduates eased to 5.2 per cent, but the underemployment rate, the share working in jobs that do not typically require a degree, rose to 42.5 per cent.

There is an apparent paradox here. If early-career experience is becoming more valuable, why are young people finding it harder to get hired? The Levy Yeyati paper measures returns among people who are employed. Once you are in the door, each year of practice is worth more than before. But the door is narrower, because the entry-level job, where codified knowledge was traditionally converted into judgement, is exactly what AI is eroding. The authors note that changes in who gets employed could partly explain their early-career estimates. The premium on early experience and the shortage of early jobs may be two faces of the same scarcity.

The Patent Attorneys Who Kept Their Gains

Which brings us to the most direct test of whether AI helps or hinders the acquisition of judgement. In September 2026 the NBER released working paper 35720, “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting”, by David Autor, Tanya Rodchenko, Josh Martin, Zanna Iscenko, Scott Strand, David Pearl and Melissa Ferere. A revision followed in October.

The design is unusually clean for a workplace study. In a pre-registered randomised controlled trial, 133 practising patent lawyers at eleven US intellectual property firms were split into groups, with roughly two-thirds given access to a custom AI patent drafting assistant. Blinded expert patent attorneys scored all the work. The researchers measured performance with the tool and then, after three months, tested everyone on a core professional task without it: redlining an existing patent application.

With the tool, everyone got better. AI access raised the quality of work on benchmark drafting tasks by 0.34 standard deviations at 10 days and 0.38 at 90 days, with larger gains among junior lawyers. Dennis Crouch, the University of Missouri law professor who writes the Patently-O blog, noted that much of the improvement came from “avoiding awful work” rather than producing the very best applications.

Without the tool, the picture split. Treated lawyers outperformed controls by 0.32 standard deviations on the unaided redlining task. But that advantage came entirely from senior lawyers, whose gain was 0.45 standard deviations. Junior lawyers showed no average improvement. Their scores spread out: mediocre results became much less common, while both poor and good results became more common. The authors describe AI as acting as a “springboard” for some juniors and a “cushion” for others. Their conclusion is a sentence worth pinning above every graduate trainee's desk: “Foundational expertise may be a prerequisite for extracting durable skill from AI-assisted practice.”

Autor summarised the result to Fortune in a phrase: AI is “a performance equalizer, but a skill-disequalizer.” He added a warning aimed squarely at people starting out: “Young professionals need to beware of the illusion of competence.” Junior lawyers given the tool reported a surge in task satisfaction, the largest effect in the study. They enjoyed skipping the blank page and moving straight to a reviewer's role. The danger is that the satisfaction of reviewing a competent draft is not the same as acquiring the ability to produce one.

The caveats are substantial, and Fortune's reporting set them out. The paper is not peer-reviewed. Only 91 of the 133 lawyers completed the AI-free test. The firms declined a baseline skills test, so skill gains are inferred from random assignment rather than measured before and after. Fifteen of the 91 markups were flagged as possibly AI-assisted despite the ban; excluding them shrinks the senior advantage to 0.39 standard deviations at a looser statistical threshold, and the overall effect no longer holds. Five of Autor's co-authors are full-time Google employees and the sixth is a paid Google contractor, Google conducted and paid for the study, and the tool was Google's own. Autor is head of MIT's economics department and a Google Technology and Society Visiting Fellow. Three months is short compared with the years it takes to make a patent lawyer.

Still, the result rhymes with others. Erik Brynjolfsson, Danielle Li and Lindsey Raymond's study of more than 5,000 customer-support agents, published in the Quarterly Journal of Economics in 2025, found that AI assistance raised productivity by 15 per cent on average, with the biggest gains for less experienced and lower-skilled workers, while the most experienced saw small gains in speed and small declines in quality. And a field experiment by Hamsa Bastani and colleagues, published in the Proceedings of the National Academy of Sciences in 2025, found that Turkish high school students given unrestricted access to GPT-4 during maths practice scored 17 per cent lower once the tool was taken away, compared with students who never had it. A version with learning-protective guardrails largely mitigated the harm.

The pattern across all three is consistent. AI lifts novice performance in the moment. Whether it builds novice capability over time depends on what the novice already knows and how the tool is used.

Why Experience Expires

The other half of the rotation, the falling return at 25 years, connects to a literature on skill obsolescence that predates generative AI. In 2020 David Deming of Harvard and Kadeem Noray published “Earnings Dynamics, Changing Job Skills, and STEM Careers” in the Quarterly Journal of Economics. Using job vacancy data from Burning Glass Technologies covering 2007 to 2019, they showed that the earnings premium for graduates in technology-intensive subjects such as computer science, engineering and business declines rapidly as they gain experience.

Their model treats a career as a race between on-the-job learning and skill obsolescence. In fast-changing fields, the specific skills a graduate brings are replaced within a few years, which lowers the return to experience and flattens the age-earnings profile. Higher-ability STEM graduates also tend to leave STEM careers more often and at younger ages.

Generative AI may be turning a phenomenon that was confined to technology-intensive careers into something more general. If a large share of codified knowledge in law, finance, marketing and software is now supplied by a machine, then the experience of having done those tasks the old way may depreciate across the board. The Levy Yeyati paper's late-career finding is consistent with that reading, though the authors place less weight on individual late-career coefficients because the pre-trend test for them is weaker.

Here is the double bind for someone starting out. Early experience matters more than ever, because judgement is what AI cannot yet supply. But that experience may itself be technology-specific, and if the technology keeps shifting, its value will erode too. The patent lawyer study suggests one way out: seniors who already had deep foundations were able to absorb the new tool and come out stronger. Experience did not simply expire. It compounded, for those who had enough of it to build on.

The Debt Was Priced for a Different Market

None of this would matter so much if degrees were cheap. They are not. In the United States, Federal Student Aid reported in September 2026 that 42.3 million recipients hold federal student loans totalling more than $1.7 trillion. More than 9.3 million are in default, owing $234 billion. In England, the Student Loans Company's 2025-26 statistics put the higher education loan balance at £294.6 billion, and the provisional average balance for borrowers entering repayment in 2026 is £47,730. The House of Commons Library projects the outstanding balance could reach around £500 billion, in 2025-26 prices, by the late 2040s.

Those debts were taken on at prices set for a labour market in which the return to schooling was stable. If the premium is now shrinking in the fields most exposed to AI, the implied return on that investment shrinks with it. A loan agreement does not reprice when the asset it financed does.

It is worth being precise about the scale. A decline of roughly 0.55 log points per year of schooling, or 3.2 log points at the paper's illustrative mid-career profile, is a meaningful cut but not an annihilation. Even after the shock, the return to schooling in exposed occupations, roughly 7 points per year, remains far higher than the 1.7 points in the least exposed occupations. A degree is still associated with substantially higher pay. What has changed is the direction of travel and the reason for the premium. The share of the premium attributable to knowing things is falling. The share attributable to being able to do things is rising.

England's income-contingent repayment partly cushions graduates, because repayments fall when earnings do; many American borrowers have less protection. Either way, the risk sits with the person least able to diversify it: a young worker with a single human-capital asset and a large liability attached.

What the Numbers Do Not Say

First, the Levy Yeyati paper is a working paper, not yet peer-reviewed, and its headline schooling estimate is sensitive to how top-coded wages are treated. Its central identifying assumption, that greater exposure means a larger expansion of machine-supplied codified knowledge, is assumed rather than tested. The authors list possible confounds candidly: changes in who is employed, signalling and employer learning, and post-2022 shifts in worker composition.

Second, years of schooling are a crude proxy for codified knowledge. A degree also signals persistence, conscientiousness and the ability to learn. It provides networks, and it often functions as a legal or professional gatekeeper. None of those functions is obviously diminished by a chatbot, and the paper cannot separate them.

Third, each of the other studies has its own blind spots, from the Autor experiment's single tool to the descriptive nature of the payroll data. Their convergence is more persuasive than any one result.

Fourth, all of this describes the first three and a half years of a technology that is still changing quickly. The Levy Yeyati authors call their results a short-run repricing of existing stocks of human capital. Over the longer run, schooling choices, curricula, mentoring, occupational sorting and the organisation of entry-level work will all adjust. They close by asking “whether workers can continue to acquire the practice-based knowledge” that AI increasingly substitutes for in learning itself.

Building Judgement Without the Apprenticeship

So how should someone at the outset of a career think about a credential? The evidence suggests five principles, none of them comfortable.

Treat the degree as a foundation, not a product. Senior patent lawyers extracted durable skill from AI because they had foundations to build on, and the authors' conclusion that foundational expertise “may be a prerequisite” is an argument for deep formal education, not against it. What loses value is the degree as a certificate of facts possessed. What retains value is the degree as scaffolding for judgement. The goal is not to memorise what a model can recite, but to understand a field well enough to know when the model is wrong.

Test yourself without the tool. Autor's advice to Fortune was that the only reliable check on skill development is performing tasks without AI and evaluating the results. The Bastani study points the same way: students who used unrestricted GPT-4 as a crutch did worse once it was gone. If your employer gives you an AI assistant, the most valuable thing you can do for your own career may be to periodically do the work without it and compare.

Hunt for the practice, not just the job. Returns to early-career experience are rising while early-career hiring is falling. The scarce resource is not the qualification but the chance to convert it into judgement through supervised real work. Roles offering genuine apprenticeship, with feedback from experienced people on tasks that matter, are worth more than their starting salaries suggest.

Lean towards the professionalised track. Ask whether AI in a given role raises the bar for human expertise or lowers it. Roles where the machine handles the routine and leaves you the hard calls are growing faster and paying better. The question applies to specific jobs and employers, not just titles.

Plan for your experience to expire. Technology-specific know-how depreciates, so the defence is to keep renewing it, layering new techniques onto durable understanding, as the senior patent lawyers did, rather than swapping one set of specific skills for another.

For employers, the implication is sharper. Cutting entry-level hiring because AI can do the junior work is a decision to stop producing senior people. The juniors whose outcomes spread out in both directions in the Autor experiment are next decade's senior lawyers, or not.

A Credential Worth Holding, Priced Differently

What does it mean when the wage premium for education shrinks in the fields most exposed to the technology? The honest answer is that the premium was always paying for two things at once, and the market has started to separate them.

One part paid for possessing codified knowledge, and that part is falling, because a machine can now produce it in seconds. The other part paid for the capacity to learn, judge and build on that knowledge, and that part is holding up, perhaps rising, for those who get the chance to exercise it. The degree you paid for is not worthless. Its price has been revised, and the reason it is valuable has shifted from what it certifies you know to what it enables you to become.

For the worker carrying the debt, that is cold comfort, because the debt did not reprice. For the person deciding whether to start, it is a clearer guide than the noise suggests. The credential still pays, most of all to those who treat it as the start of an apprenticeship rather than the end of an education. Half a percentage point is a warning, not a verdict. The 18-year hinge is a reminder that judgement, too, has a shelf life, and the only defence is to keep building it.


References and Sources

  1. Levy Yeyati, Eduardo; Judzik, Darío; Barbiero, Juan Ignacio. “Practice Makes Perfect: Tacit Knowledge, Generative AI, and Returns to Schooling and Experience.” arXiv:2610.36008 [econ.GN], submitted 26 October 2026. Also circulated as Universidad Torcuato Di Tella, Escuela de Gobierno, Documento de Trabajo 2026/15. https://arxiv.org/abs/2610.36008 (HTML full text: https://arxiv.org/html/2610.36008)
  2. Autor, David; Rodchenko, Tanya; Martin, Josh; Iscenko, Zanna; Strand, Scott; Pearl, David; Ferere, Melissa. “Does AI Assistance Enhance or Erode Expertise? Evidence from a Three-Month Field Experiment in Patent Drafting.” NBER Working Paper No. 35720, September 2026, revised October 2026. https://www.nber.org/papers/w35720 (also SSRN abstract 7426447: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7426447)
  3. Crouch, Dennis. “Who Keeps the Skill: A Randomized Trial of AI Patent Drafting at Eleven Firms.” Patently-O, 9 September 2026. https://patentlyo.com/patent/2026/09/who-keeps-the-skill-a-randomized-trial-of-ai-patent-drafting-at-eleven-firms.html
  4. Lichtenberg, Nick. “The economist behind the 'China shock' tackles Gen Z's AI paradox.” Fortune, 10 October 2026. https://fortune.com/2026/10/10/david-autor-china-shock-nber-google-paper-illusion-of-competence-gen-z/
  5. PwC. “AI reshapes global labour market into two distinct paths, rewarding human skills: PwC 2026 Global AI Jobs Barometer.” Press release, London, 15 June 2026. https://pwc.com/gx/en/news-room/press-releases/2026/pwc-2026-ai-jobs-barometer.html (syndicated copy: https://kyodonewsprwire.jp/release/202606150856)
  6. PwC. “2026 Global AI Jobs Barometer: Global Findings.” June 2026. https://www.pwc.com/gx/en/issues/artificial-intelligence/job-barometer/2026/2026-global-ai-jobs-barometer-global-findings.pdf
  7. PwC Belgium. “AI reshapes labour market and urges for upskilling: PwC's AI Jobs Barometer 2026.” Press release, Diegem, 15 June 2026. https://press.pwc.be/ai-reshapes-labour-market-and-urges-for-upskilling-pwcs-ai-jobs-barometer-2026
  8. Katanich, Doloresz. “Human skills increasingly in demand as AI reshapes labour market, PwC finds.” Euronews, 16 June 2026. https://euronews.com/business/2026/06/16/human-skills-increasingly-in-demand-as-ai-reshapes-labour-market-pwc-finds
  9. Let's Data Science (LDS Team). “PwC Finds AI Creates Two-Track Global Labour Market.” 15 June 2026. https://letsdatascience.com/news/pwc-finds-ai-creates-two-track-global-labour-market-550ec8b1
  10. Mincer, Jacob. Schooling, Experience, and Earnings. New York: National Bureau of Economic Research / Columbia University Press, 1974. Chapter 5, “The Human Capital Earnings Function”, pp. 83-96. https://ideas.repec.org/b/nbr/nberbk/minc74-1.html ; https://www.nber.org/chapters/c1767.pdf
  11. Felten, Edward; Raj, Manav; Seamans, Robert. “Occupational, industry, and geographic exposure to artificial intelligence: A novel dataset and its potential uses.” Strategic Management Journal, 42(12), 2195-2217, 2021. DOI: 10.1002/smj.3286. https://econpapers.repec.org/RePEc:bla:stratm:v:42:y:2021:i:12:p:2195-2217
  12. Eloundou, Tyna; Manning, Sam; Mishkin, Pamela; Rock, Daniel. “GPTs are GPTs: Labor market impact potential of LLMs.” Science, 384(6702), 1306-1308, June 2024. DOI: 10.1126/science.adj0998. Preprint: https://arxiv.org/abs/2303.10130
  13. Polanyi, Michael. The Tacit Dimension. Garden City, NY: Doubleday, 1966 (reissued University of Chicago Press, 2009).
  14. Autor, David H. “Polanyi's Paradox and the Shape of Employment Growth.” NBER Working Paper No. 20485, September 2014. DOI: 10.3386/w20485. https://www.nber.org/papers/w20485
  15. Brynjolfsson, Erik; Chandar, Bharat; Chen, Ruyu. “Canaries in the Coal Mine? Six Facts about the Recent Employment Effects of Artificial Intelligence.” Stanford Digital Economy Lab, first released August 2025, revised 12 August 2026. https://digitaleconomy.stanford.edu/publications/canaries-in-the-coal-mine/
  16. Federal Reserve Bank of New York. “The Labor Market for Recent College Graduates.” Data for 2026:Q3. https://www.newyorkfed.org/research/college-labor-market
  17. Brynjolfsson, Erik; Li, Danielle; Raymond, Lindsey. “Generative AI at Work.” Quarterly Journal of Economics, 140(2), 889-942, 2025. Earlier version: NBER Working Paper No. 31161, 2023. https://www.nber.org/papers/w31161 ; https://econpapers.repec.org/RePEc:oup:qjecon:v:140:y:2025:i:2:p:889-942
  18. Bastani, Hamsa; Bastani, Osbert; Sungu, Alp; Ge, Haosen; Kabakcı, Özge; Mariman, Rei. “Generative AI without guardrails can harm learning: Evidence from high school mathematics.” Proceedings of the National Academy of Sciences, 122(26), e2422633122, 2025. DOI: 10.1073/pnas.2422633122. https://econpapers.repec.org/RePEc:nas:journl:v:122:y:2025:p:e2422633122
  19. Deming, David J.; Noray, Kadeem. “Earnings Dynamics, Changing Job Skills, and STEM Careers.” Quarterly Journal of Economics, 135(4), 1965-2005, 2020. DOI: 10.1093/qje/qjaa021. https://www.hks.harvard.edu/publications/earnings-dynamics-changing-job-skills-and-stem-careers
  20. US Department of Education, Federal Student Aid. “(GENERAL-26-57) Federal Student Aid Posts Updated Reports to FSA Data Center.” Electronic announcement, 22 September 2026. https://fsapartners.ed.gov/knowledge-center/library/electronic-announcements/2026-09-22/federal-student-aid-posts-updated-reports-fsa-data-center
  21. Student Loans Company. “Student Loans in England: Financial Year 2025-26.” Official statistics, GOV.UK, updated 2 July 2026. https://www.gov.uk/government/statistics/student-loans-in-england-2025-to-2026/student-loans-in-england-financial-year-2025-26
  22. House of Commons Library. “Student loan statistics.” Research briefing SN01079, 2026. https://commonslibrary.parliament.uk/research-briefings/sn01079

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Sixteen point two per cent. That is how often people in one 2026 experiment caught a machine's mistake once it had explained itself in fluent, confident prose. Swap the explanation for a bare number, the model's own estimate of how likely its answer was to be right, and the rate more than doubled, to 37.5 per cent.

The figures come from controlled studies by Ruth Cohen and Sarit Kraus of Bar-Ilan University, Lu Feng of the University of Virginia and Ayala Bloch of Ariel University, posted to arXiv as paper 2604.03237. The first version had a title built for the conference circuit: “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance”. Its abstract did not hedge: “We identify a Persuasion Paradox: fluent explanations systematically increase user confidence and reliance on AI”. The authors argued that interfaces should put calibrated reliance and recovery from errors ahead of persuasive fluency.

Then something unusual happened. On 18 August 2026 the authors posted a revision with a new title, “Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks”. The word “persuasion” has vanished from the text. The 16.2 and 37.5 per cent figures remain, but the revised statistics report that differences in error recovery across the four support conditions did not reach significance (p = .055). The new abstract closes on a far less quotable line: “no single support strategy is universally effective.”

Hold both versions in your head, because between them they contain the whole argument. The first is the story everyone wants to tell: explanations are a trap, and the explainable AI movement has been building persuasion machines. The second is the story the evidence supports: explanations reliably raise confidence, do not reliably raise accuracy, and help or hurt depending on whether the reader can check what they say. A paper about the dangers of persuasive fluency was, in its first draft, a little too persuasive. Its authors noticed and pulled it back. That is calibration in action, and it is exactly the behaviour designers should be trying to produce in everyone else.

So the useful question is not whether explanations are good or bad. It is narrower and harder. When should a system withhold its explanation, or restructure it? And what would an interface look like if its job were not to help people understand the AI, but to help them know when to ignore it?

Sixteen Per Cent and a Change of Title

Cohen and colleagues ran three experiments with participants recruited through university mailing lists, mostly people with STEM backgrounds, who were paid a bonus for each correct answer. Two tasks carried the weight: RAVEN matrices, abstract visual puzzles in which you choose the shape that completes a pattern, and logical reasoning questions from the LSAT, the American law school admission test.

The cleverest design was the multi-stage reveal. Twenty-seven participants answered each RAVEN puzzle three times: first alone, then after seeing the AI's predicted answer, then after reading an explanation, either a written rationale from a large language model or a heatmap showing which parts of the image the model weighted. Alone, they were right 37.0 per cent of the time. With the AI's answer, 49.8 per cent. With the explanation on top, 48.8 per cent. The explanation bought nothing in accuracy.

It did buy something else. In the paper's words, “confidence remained largely unchanged after the prediction stage and increased only after explanations were presented”. The prediction made people more accurate without making them feel surer. The explanation made them feel surer without making them more accurate. If you wanted one result to sum up the worry about fluent AI, that is it.

The between-subjects RAVEN study, with 100 participants across five conditions, is where the famous numbers live. Every form of AI support beat working alone by a wide margin: unaided participants managed 24.6 per cent, while the four AI-assisted groups landed between 53.5 and 60.5 per cent, with predicted probabilities at the top. On overall accuracy, though, the AI-supported conditions were statistically indistinguishable. The difference appeared in how people handled the AI's mistakes. “LLM explanations showed the lowest recovery rate (16.2%) despite high agreement (84.2%),” the authors write. The probability condition showed the highest recovery, 37.5 per cent, with 75.8 per cent agreement on correct predictions.

Read those pairs together. The explanation group agreed with the AI more when it was right (a gap that, like the recovery gap, fell short of significance) and caught it less when it was wrong. That is the signature of reliance that has stopped discriminating: trust that rises evenly across good and bad advice instead of tracking which is which.

The caveats matter, and the revised paper is admirably blunt about them. There was no power analysis in advance, and twenty people per condition is a small sample. The RAVEN probabilities were raw, uncalibrated softmax outputs. A “selective automation” strategy, in which the AI decides confident cases and hands uncertain ones to the human, is now presented only as a benchmark derived after the fact from the probability group's data. The pattern is consistent and striking. It is not yet proof.

The Same Explanation Wins at Logic

The LSAT study rarely makes it into social media summaries, and it is the part that turns a scare story into a design problem.

Eighty participants worked through ten LSAT questions with help from an AI whose answers were right 60 per cent of the time. One group saw only the AI's answer, one saw the answer plus a stated probability, one saw an expert-written explanation adapted from earlier research, and one saw an explanation generated by a large language model.

The language model's explanation won outright. That group reached 72.5 per cent accuracy, against 55.5 per cent for expert explanations, 48.5 per cent for the bare prediction and 47.0 per cent for probabilities. It was the only group to beat the AI's own 60 per cent, and it recovered best from errors, at 47.5 per cent against 35.0 per cent for probabilities. It even beat the human experts' explanations, which rules out the comforting idea that machine prose is simply worse prose. Here the fluent explanation was not a sedative. It was a tool.

Why would the same kind of support hurt in one task and help in another? The authors are careful not to overclaim, noting that task, model and explanation style all changed between studies. But the contrast lines up with an idea that runs through the field. An LSAT explanation lives in the same medium as the problem. If a model says an argument assumes that correlation implies causation, you can reread the passage and see whether it does. The explanation offers a claim the reader can test against evidence already in front of them.

A verbal rationale for a visual puzzle is different. It translates a pattern into words, and the reader has to translate it back. That is laborious, so most people do not bother. They register that the paragraph sounds like reasoning and move on. The explanation becomes a vibe rather than an argument.

In 2023, Helena Vasconcelos and colleagues at Stanford University and the University of Washington, including Michael Bernstein, Tobias Gerstenberg and Ranjay Krishna, gave this intuition a formal shape. Across five studies with 731 participants navigating a maze with a simulated AI, they argued that people make a strategic cost-benefit choice about whether to engage with an explanation at all. Task difficulty, explanation difficulty and money all shifted how much people over-relied on the AI. Earlier studies that found explanations useless, they suggested, may simply have used explanations that did not make checking the AI any cheaper. Their title made the counterintuitive point: “Explanations Can Reduce Overreliance on AI Systems During Decision-Making”.

Put Cohen's LSAT result beside that framework and a first principle falls out. An explanation earns its place when it lowers the cost of verification. When it raises that cost, or supplies the feeling of verification without the substance, it should be withheld or rebuilt.

Editing the Machine's Mind Feels Like Power

If reading an AI's reasoning can make you overconfident, perhaps the fix is to let users get their hands on it. Interactive reasoning, where people inspect and change a model's chain of thought, has become a fashionable idea in human-computer interaction. A short paper at CHI 2026, the field's flagship conference, held in Barcelona in April, suggests the fashion deserves scrutiny.

Jiwon Moon of the University of Chicago, with Chien-Ming Huang and Ziang Xiao of Johns Hopkins University, published “Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making” in the conference's Extended Abstracts, the venue for shorter, often preliminary work. They compared three interfaces in a decision task: no chain of thought, a read-only chain of thought, and an editable one that let users alter the AI's displayed reasoning.

Their abstract captures the gap between how an interface feels and what it does. “Editability significantly increases perceived power, control and satisfaction, but does not improve decision accuracy relative to a no-rationale baseline.” The read-only version was worse: it “increases agreement with the AI and induces inappropriate reliance when the AI is wrong”, reducing accuracy compared with editing and trending below the baseline in which people saw no reasoning at all.

The mechanism is the interesting part. Mediation analyses, the authors report, suggest that editability raises perceived power, which in turn predicts higher confidence and greater over-reliance on incorrect advice, a pattern “consistent with illusion of control.”

The phrase comes from a 1975 paper by the psychologist Ellen Langer in the Journal of Personality and Social Psychology. Across experiments built on games of pure chance, Langer argued that features associated with skill, such as choice, familiarity, involvement and competition, lead people to expect success beyond what the odds justify. Her best-known demonstration, in which people who picked their own lottery ticket asked far more to sell it than people who were handed one, has not aged well: in 2021 Joowon Klusowski, Deborah Small and Joseph Simmons reported in Psychological Science that experiments with more than 10,000 participants found choice alone did not make people think they were more likely to win.

That wrinkle is instructive rather than fatal. Moon, Huang and Xiao measured perceived power directly and linked it to confidence and over-reliance in their own data. But it is a reminder that a vivid label can travel further than the evidence beneath it, which is the subject of this article. Because the CHI paper is an extended abstract, its effect sizes should be treated as provisional. The direction of the result is still a warning to anyone who treats “user control” as an automatic good. Control that is not connected to better outcomes is a feeling, and feelings are what persuasion is made of.

Thirty Years of Leaning on Machines

None of this is new in kind. It is new in fluency.

In 1997, Raja Parasuraman and Victor Riley published “Humans and Automation: Use, Misuse, Disuse, Abuse” in the journal Human Factors. Misuse, in their taxonomy, is over-reliance on automation, producing monitoring failures and decision biases. Disuse is neglect of automation that works, often because it has cried wolf too often. Abuse is automating without regard for the consequences for human performance. All three map onto modern AI assistants, but misuse is the one fluent explanations feed.

In 2004, John Lee and Katrina See gave the field its central goal in the title of their review in the same journal: “Trust in Automation: Designing for Appropriate Reliance”. The key idea is calibration. Trust should match a system's actual trustworthiness, neither overshooting nor undershooting. And it has to work at a fine grain: trusting a system the right amount on average is not enough. You need to trust it more where it is good and less where it is bad.

Medicine has the clearest data on what happens when that fails. A 2012 systematic review in the Journal of the American Medical Informatics Association by Kate Goddard, Abdul Roudsari and Jeremy Wyatt drew on 74 studies of automation bias, which they defined as “the tendency to over-rely on automation”. Pooling four healthcare studies in which decision support systems gave wrong advice, they found the systems increased the risk of an incorrect decision by 26 per cent. The accelerants were familiar: high trust, low self-confidence, workload, complexity, time pressure. Among the design mitigations, they listed attaching updated confidence levels to the system's output, and offering information rather than direct recommendations.

A review published years before large language models went mainstream recommended, in effect, what the RAVEN probability condition did: show people how sure the system is, case by case.

The idea is now law. Article 14 of the European Union's AI Act, on human oversight of high-risk systems, requires that overseers be enabled “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)”, particularly where systems provide information or recommendations for decisions taken by people. Those obligations do not bite yet, either: the Digital Omnibus on AI, in force since July 2026, pushed the high-risk deadlines back to December 2027 for standalone systems and August 2028 for AI built into regulated products. The regulation names the problem. It does not specify the interface that solves it. That is the designer's job, and the evidence suggests a fluent paragraph of reasoning is not, by default, the answer.

Explanations Help Most When You Need Them Least

The most cited study on this question predates the language model boom and predicted much of what followed. In 2021, Gagan Bansal and colleagues from the University of Washington and Microsoft Research, among them Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro and Daniel Weld, presented “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance” at CHI.

Their target was complementary performance: a human-AI team that beats both the human alone and the AI alone. Earlier studies showing that explanations helped had mostly used AI far more accurate than people, so teams improved simply by deferring to a better decision-maker. Bansal's group used AI roughly as accurate as humans, across three tasks (sentiment in beer reviews and book reviews, plus LSAT questions), with 1,626 participants.

Teams did achieve complementary performance, but explanations did not improve on simply showing the AI's confidence. The mechanism was an asymmetry. “Explanations often increased accuracy when the AI system was correct but, worryingly, decreased it when the AI erred, resulting in a minimal net change.” Explanations made people more likely to accept the AI's recommendation whether or not it was right.

Bansal's team tried a fix. Their “adaptive explanation” explained only the top answer when the AI was confident, but laid out the top two alternatives when confidence fell below a threshold, inspired by an efficient assistant that divulges more, “confessing doubts and arguing for alternatives”, only when unsure. It did not significantly improve team performance, though the authors saw suggestive evidence that it nudged agreement in the right direction. Keep it in mind: it is the seed of a better idea.

In 2023, Max Schemmer and colleagues at the Karlsruhe Institute of Technology gave the asymmetry two names. At the Intelligent User Interfaces conference, they proposed measuring “appropriateness of reliance” on separate axes. Relative AI reliance captures how often people switch to correct AI advice after starting with a wrong answer. Relative self-reliance captures how often people keep their own correct answer when the AI is wrong. In their 200-person experiment, explanations raised relative AI reliance from 29.59 to 38.87 per cent. Relative self-reliance did not significantly change.

That is the pattern in miniature. Explanations help people accept the AI when it is right. They do far less to help people resist it when it is wrong. And resistance is the whole point of a human in the loop. If the machine were never wrong, you would not need the person.

Why Fluency Reads as Truth

Large language models sharpen the problem because their explanations are not heatmaps or feature weights. They are prose, and prose is the native format of human persuasion.

In 2025, Mark Steyvers and colleagues at the University of California, Irvine published “What large language models know and what people think they know” in Nature Machine Intelligence. They measured the calibration gap: the difference between how confident people are in a model's answers and how confident the model itself is. Shown default explanations, people overestimated the model's accuracy, and longer explanations raised their confidence even when the extra length did nothing for accuracy. “There's a disconnect between what LLMs know and what people think they know,” Steyvers said in the university's announcement. “We call this the calibration gap.” When the team rewrote explanations so their language reflected the model's internal confidence, that gap narrowed, and people got better at telling right answers from wrong ones.

A year earlier, Sunnie Kim of Princeton University, with Q. Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard and Jennifer Wortman Vaughan of Microsoft, tested something similar. In a pre-registered experiment with 404 participants answering medical questions with a simulated LLM-powered search engine, first-person expressions of uncertainty, along the lines of “I'm not sure, but...”, reduced people's confidence in the system and their agreement with it, and improved their accuracy. Impersonal hedges had weaker effects that did not reach significance. Wording matters enough, the authors concluded, that it needs user testing before deployment.

Back in 2020, Yunfeng Zhang, Q. Vera Liao and Rachel Bellamy of IBM Research found that confidence scores helped people calibrate their trust, while explanations of individual predictions did not do the same job. Their caveat still applies: calibrated trust does not guarantee better decisions unless the human brings knowledge that can compensate for the model's errors. Calibration tells you when to be suspicious. It does not tell you the right answer.

Fluency, in other words, is not a neutral property. Length, confident tone and smooth reasoning all signal reliability without being connected to it. A well-designed system should make those signals honest, or remove them.

Friction Works and Nobody Likes It

If people skim explanations and defer, the obvious response is to make deferral harder. Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos of Harvard University tested exactly this in “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making”, published in 2021 in the Proceedings of the ACM on Human-Computer Interaction.

Borrowing from research on diagnostic error in medicine, they built three interventions into a task in which 199 participants judged how to cut the carbohydrate content of meals. In the “on demand” condition, the AI's suggestion stayed hidden until people clicked to see it. In “update”, people decided first and could revise after seeing the AI's advice. In “wait”, a message saying the AI was processing the image appeared for 30 seconds before the suggestion arrived.

All three significantly reduced over-reliance compared with simple explainable AI interfaces. None eliminated it: the interventions “reduced, but not yet eliminated, overreliance on the AI”, and the human-AI teams still underperformed the AI alone. Then the twist. When the AI was wrong, “people performed best in conditions that they preferred and trusted the least”, which they also rated most difficult. The interventions also helped people high in “need for cognition”, a measure of how much someone enjoys effortful thinking, more than others, raising the question of who friction actually protects.

This is the commercial problem in one finding. Every metric a product team watches, from satisfaction scores to stated trust, rewards the smooth, confident explanation. The metric that matters, whether people catch the machine's errors, rewards friction. Moon, Huang and Xiao found the same split from the other side. And a 2026 CHI review of reliance research by Muhammad Raees, Vassilis-Javed Khan, Ioanna Lykourentzou and Konstantinos Papangelis made the general point: user engagement is valuable for appropriate reliance, but it comes at a cost.

When to Withhold the Explanation

So when should a designer leave the explanation out? The evidence supports no blanket rule, and the revised Cohen paper is explicit that no single strategy wins everywhere. But it does point to conditions under which a default explanation is more likely to hurt than help.

The first is when the explanation cannot be checked against evidence the user can see. The RAVEN rationale asked people to verify a verbal account of a visual pattern; the LSAT rationale asked them to verify a verbal account of a verbal argument. Where explanation and evidence live in different media, or the evidence is hidden, the explanation tends to function as a confidence signal rather than an argument. Vasconcelos's cost-benefit framing predicts exactly this.

The second is when fluency is decoupled from reliability. A language model can write an equally polished rationale for a right answer and a wrong one, and Steyvers's results show that people read polish and length as accuracy. If a system cannot make the style of its explanation track its actual confidence, the explanation is broadcasting a false signal.

The third is when the user cannot bring independent knowledge. Zhang, Liao and Bellamy's caveat applies: suspicion helps only if the human can act on it. Goddard's review identified time pressure, workload and complexity as amplifiers of automation bias, precisely when people are least able to scrutinise a paragraph of reasoning. In those conditions, a persuasive explanation mostly accelerates deference.

The fourth is before the user has formed a view. Buçinca's update condition and Cohen's multi-stage design both suggest timing matters. An explanation shown before someone has thought for themselves anchors them. Shown afterwards, as a challenge rather than a verdict, it does a different job.

None of this means explanations should vanish. Withholding an explanation from the moment of decision is not the same as withholding it from the record. Engineers debugging a model, auditors checking it and people contesting its verdicts may all need a full account. The question here is narrower: what to put in front of someone in the seconds when they decide whether to accept the machine's answer. The default, an explanation attached to every answer and displayed immediately, is the wrong default for a large class of tasks.

How to Rebuild an Explanation

Withholding is the blunt option. Restructuring is more interesting, and the literature offers several patterns.

Lead with uncertainty rather than reasoning. In Cohen's RAVEN study, the probability display recovered best from errors and was rated clearest by participants, despite explaining nothing. Goddard's review flagged confidence displays more than a decade ago, and Kim's work shows uncertainty can be expressed in natural language, if the phrasing is tested. A number or a well-chosen hedge is a cue to look harder, cheap to read and hard to mistake for an argument.

Make explanations conditional, in the spirit of Bansal's adaptive design. When the model is confident and historically reliable on this kind of case, a short answer may suffice. When it is uncertain, it should do the opposite of what fluent assistants usually do: argue against itself, present the strongest alternative, and show what evidence would change its answer.

Stop recommending and start evaluating. In a 2023 paper at the ACM Conference on Fairness, Accountability and Transparency, titled “Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support”, Tim Miller, then at the University of Melbourne, argued that recommendations plus explanations undermine human agency and do not match how people reason. He proposed “evaluative AI”, tools that “provide evidence for and against decisions made by people” rather than handing them a decision to accept or reject. The human proposes; the machine stress-tests. There is nothing to be persuaded of, because the machine is not selling an answer.

Sequence the interaction. Ask for the human's judgement first, reveal the AI's view second, and explain only where they disagree. That combines Buçinca's update intervention with adaptive explanation, concentrating scarce attention where it could change the outcome.

Route the cases. Machine learning researchers have long studied “learning to defer”; a 2020 paper at the International Conference on Machine Learning by Hussein Mozannar and David Sontag, for instance, developed methods for training a model that either predicts or hands the decision to a human expert. Cohen's selective-automation benchmark gestures at the same idea, and the authors note that humans can keep final authority under such schemes. Human scepticism is a scarce resource and should be spent where the model is weakest.

A Framework Built Around Distrust

Pull these threads together and a framework for calibrated reliance emerges. Its goal is not understanding but discrimination: helping people tell the machine's good answers from its bad ones, case by case. What follows is a synthesis of the research above, not a standard anyone has yet adopted.

Measure the right thing. Evaluate a system separately on cases where the AI is right and cases where it is wrong, which is the first recommendation in the revised Cohen paper. Schemmer's two axes offer one method; error recovery offers another. Overall accuracy and trust ratings on their own are close to useless, because the Cohen, Bansal, Buçinca and Moon studies all show satisfaction and confidence moving independently of decision quality.

Make every confidence signal honest. If explanations sound equally sure whether the model is right or wrong, their style is lying. Either calibrate the language to the model's real confidence, as Steyvers's team did, or strip out the persuasive styling and show a calibrated number. One caveat: Cohen's RAVEN probabilities were not carefully calibrated, and the authors warn that uncertainty alone can miss errors the model makes confidently. Probability displays are a starting point, not a solution.

Match support to the task's evidence. Where users can verify an explanation against material in front of them, as with the LSAT, rich explanations can pay their way. Where they cannot, prefer uncertainty cues, contrastive alternatives or nothing at all. This is the core lesson of the revised Cohen paper, and a direct rejection of one-size-fits-all explainability.

Protect independent judgement. Elicit the human's view before revealing the machine's wherever time allows, and present explanations as challenges to disagreement rather than justifications for agreement.

Budget friction. Buçinca's results show friction works and people dislike it. Applied everywhere, it wears people down and pushes them back towards blind acceptance. Applied where the model is uncertain, the stakes are high and the human might know better, it earns its cost.

Distrust the feeling of control. Moon, Huang and Xiao's work warns that editing, steering and customising can raise users' sense of power without improving outcomes. Every control affordance should be tested against accuracy on cases where the AI is wrong, not just against satisfaction.

Treat oversight law as a design brief. Article 14 asks that overseers “remain aware” of automation bias. Awareness is not a pop-up warning. It is an interface that makes the machine's fallibility visible at the moment of decision.

The Researchers Who Took Their Own Advice

The revision of the Cohen paper is worth dwelling on. The first version made a vivid, general claim with a memorable name. The second narrowed it, flagged its statistical limits, demoted its most appealing benchmark to a reference point, and concluded that the right answer depends on the task. The authors noticed that their framing was more confident than their evidence, and they recalibrated. The Langer replication story is the same lesson at a fifty-year scale.

That is precisely the behaviour a calibrated-reliance interface tries to produce, and it is hard. Product teams have every commercial reason to keep the fluent, confident explanation, which tests well, demos well and makes users feel smart. The evidence from Bansal in 2021 to Moon in 2026 says that feeling is not the same as being right, and the two can pull in opposite directions.

The explainable AI movement rested on an appealing premise: if people understand why a machine says what it says, they will use it better. The premise is not wrong so much as incomplete. Understanding helps when it gives people something to check. When it gives them only something to believe, it becomes another form of persuasion, and a more effective one than a bare answer, because it looks like reasoning.

The goal, then, is not more explanation or less, but better-aimed explanation: cheap to verify, honest about uncertainty, sequenced so it does not pre-empt human judgement, and most forthcoming when the machine is least sure. Judge it on one question above all: when the model was wrong, did the person notice?

In the RAVEN study, people reading fluent explanations caught roughly one AI error in six. Shown a number instead, they caught more than one in three. Neither is good enough, and the gap between them is statistically fragile. But it points towards where better designs will come from: interfaces built not to win the user's trust, but to earn it one case at a time, and to surrender it gracefully when it has not been earned.

References and Sources

All URLs accessed and verified on 29 October 2026.

  1. Cohen, Ruth; Feng, Lu; Bloch, Ayala; Kraus, Sarit. “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance.” arXiv:2604.03237v1, first version, 2026 (superseded). https://arxiv.org/abs/2604.03237v1 (full text: https://arxiv.org/html/2604.03237v1)
  2. Cohen, Ruth; Feng, Lu; Bloch, Ayala; Kraus, Sarit. “Supporting Calibrated Reliance in Human-AI Collaboration: Different Strategies for Different Tasks.” arXiv:2604.03237v2, revised 18 August 2026. https://arxiv.org/abs/2604.03237 (full text: https://arxiv.org/html/2604.03237v2)
  3. alphaXiv listing for arXiv:2604.03237, “The Persuasion Paradox: When LLM Explanations Fail to Improve Human-AI Team Performance.” https://www.alphaxiv.org/abs/2604.03237v1
  4. Moon, Jiwon; Huang, Chien-Ming; Xiao, Ziang. “Understanding the Affordances of Control in AI Reasoning for Human-AI Decision-Making.” In Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA '26), Barcelona, 13 to 17 April 2026, pp. 1-9. ACM. DOI: 10.1145/3772363.3798555. https://doi.org/10.1145/3772363.3798555
  5. Semantic Scholar record and abstract for Moon, Huang and Xiao (2026), Corpus ID 287443795. https://api.semanticscholar.org/graph/v1/paper/DOI:10.1145/3772363.3798555?fields=title,abstract,authors,venue,year
  6. Langer, Ellen J. “The illusion of control.” Journal of Personality and Social Psychology, 32(2), 311-328, August 1975. DOI: 10.1037/0022-3514.32.2.311. https://doi.org/10.1037/0022-3514.32.2.311
  7. Klusowski, Joowon; Small, Deborah A.; Simmons, Joseph P. “Does Choice Cause an Illusion of Control?” Psychological Science, 32(2), 159-172, 2021. DOI: 10.1177/0956797620958009. https://faculty.wharton.upenn.edu/wp-content/uploads/2016/11/36-Klusowski-Small-Simmons-2021.pdf
  8. Yale School of Management, Yale Insights. “Does Having Choice Provide an Illusion of Control?” https://insights.som.yale.edu/insights/does-having-choice-provide-an-illusion-of-control
  9. Bansal, Gagan; Wu, Tongshuang; Zhou, Joyce; Fok, Raymond; Nushi, Besmira; Kamar, Ece; Ribeiro, Marco Tulio; Weld, Daniel S. “Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance.” In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, May 2021, pp. 1-16. ACM. DOI: 10.1145/3411764.3445717. arXiv:2006.14779. https://arxiv.org/abs/2006.14779
  10. Buçinca, Zana; Malaya, Maja Barbara; Gajos, Krzysztof Z. “To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making.” Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), Article 188, pp. 1-21, April 2021. DOI: 10.1145/3449287. https://kgajos.seas.harvard.edu/papers/bucinca21trust.pdf
  11. Vasconcelos, Helena; Jörke, Matthew; Grunde-McLaughlin, Madeleine; Gerstenberg, Tobias; Bernstein, Michael S.; Krishna, Ranjay. “Explanations Can Reduce Overreliance on AI Systems During Decision-Making.” Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), pp. 1-38, April 2023. DOI: 10.1145/3579605. arXiv:2212.06823. https://arxiv.org/abs/2212.06823
  12. Schemmer, Max; Kühl, Niklas; Benz, Carina; Bartos, Andrea; Satzger, Gerhard. “Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations.” In Proceedings of the 28th International Conference on Intelligent User Interfaces (IUI '23), Sydney, March 2023, pp. 410-422. ACM. DOI: 10.1145/3581641.3584066. arXiv:2302.02187. https://arxiv.org/abs/2302.02187
  13. Zhang, Yunfeng; Liao, Q. Vera; Bellamy, Rachel K. E. “Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20), January 2020, pp. 295-305. ACM. DOI: 10.1145/3351095.3372852. arXiv:2001.02114. https://arxiv.org/abs/2001.02114
  14. Kim, Sunnie S. Y.; Liao, Q. Vera; Vorvoreanu, Mihaela; Ballard, Stephanie; Vaughan, Jennifer Wortman. “'I'm Not Sure, But...': Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust.” In The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), June 2024, pp. 822-835. DOI: 10.1145/3630106.3658941. arXiv:2405.00623. https://arxiv.org/abs/2405.00623
  15. Steyvers, Mark; Tejeda, Heliodoro; Kumar, Aakriti; Belem, Catarina; Karny, Sheer; Hu, Xinyue; Mayer, Lukas W.; Smyth, Padhraic. “What large language models know and what people think they know.” Nature Machine Intelligence, 7(2), 221-231, 2025. DOI: 10.1038/s42256-024-00976-7. https://www.nature.com/articles/s42256-024-00976-7 (preprint: https://arxiv.org/abs/2401.13835)
  16. University of California, Irvine, School of Social Sciences. “UC Irvine study finds mismatch between human perception and reliability of AI-assisted language tools.” 22 January 2025. https://www.socsci.uci.edu/newsevents/news/2025/2025-01-22-steyvers-uci-study-finds-mismatch-between-human-perception-and-reliability-of-ai-assisted-language-tools.php
  17. Parasuraman, Raja; Riley, Victor. “Humans and Automation: Use, Misuse, Disuse, Abuse.” Human Factors, 39(2), 230-253, June 1997. DOI: 10.1518/001872097778543886. https://journals.sagepub.com/doi/10.1518/001872097778543886
  18. Lee, John D.; See, Katrina A. “Trust in Automation: Designing for Appropriate Reliance.” Human Factors, 46(1), 50-80, 2004. DOI: 10.1518/hfes.46.1.50_30392. https://pubmed.ncbi.nlm.nih.gov/15151155/
  19. Goddard, Kate; Roudsari, Abdul; Wyatt, Jeremy C. “Automation bias: a systematic review of frequency, effect mediators, and mitigators.” Journal of the American Medical Informatics Association, 19(1), 121-127, January 2012. DOI: 10.1136/amiajnl-2011-000089. https://pmc.ncbi.nlm.nih.gov/articles/PMC3240751/
  20. Miller, Tim. “Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven decision support.” In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23), June 2023, pp. 333-342. DOI: 10.1145/3593013.3594001. arXiv:2302.12389. https://arxiv.org/abs/2302.12389
  21. Mozannar, Hussein; Sontag, David. “Consistent Estimators for Learning to Defer to an Expert.” In Proceedings of the 37th International Conference on Machine Learning (ICML 2020), PMLR 119, pp. 7076-7087. arXiv:2006.01862. https://arxiv.org/abs/2006.01862
  22. Raees, Muhammad; Khan, Vassilis-Javed; Lykourentzou, Ioanna; Papangelis, Konstantinos. “Do People Appropriately Rely on AI-Advice? An Analytical Review of HCI Research on Human-AI Decision-Making.” In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, Barcelona, April 2026, Article 805. ACM. DOI: 10.1145/3772318.3791467. https://research-portal.uu.nl/en/publications/do-people-appropriately-rely-on-ai-advice-an-analytical-review-of/
  23. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (Artificial Intelligence Act), Article 14 (Human oversight). Official Journal L, 12 July 2024. https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14 ; https://artificialintelligenceact.eu/article/14/
  24. Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI). Official Journal L, 24 July 2026; in force 27 July 2026. https://eur-lex.europa.eu/eli/reg/2026/1744/oj

Tim Green

Tim Green UK-based Systems Theorist & Independent Technology Writer

Tim explores the intersections of artificial intelligence, decentralised cognition, and posthuman ethics. His work, published at smarterarticles.co.uk, challenges dominant narratives of technological progress while proposing interdisciplinary frameworks for collective intelligence and digital stewardship.

His writing has been featured on Ground News and shared by independent researchers across both academic and technological communities.

ORCID: 0009-0002-0156-9795 Email: tim@smarterarticles.co.uk

Listen to the free weekly SmarterArticles Podcast

Support What We Do

Discuss...

Enter your email to subscribe to updates.