In this article
- Why the Turing Test Refuses to Retire
- Inside the Study: 72,000 Judgments, 1,916 Judges, 37 Models
- Barely Above Chance: What the Turing Test Scores Actually Mean
- When Interrogation Works: The Live-Chat Experiments
- Why Follow-Up Questions Trip Up the Turing Test’s Best Imitations
- The Fine Print: Limits Every Reader Should Keep in Mind
- What Builders Should Do With the Narrowing Turing Test Gap
- Conclusion: Ask One More Question
- Frequently Asked Questions
- References
Why the Turing Test Refuses to Retire
Seventy-six years after Alan Turing proposed the imitation game, the Turing test has been declared dead more often than almost any other idea in computing. Large language models made the question unavoidable again: when a chat replies fluently, empathically and on topic, has the machine finally won? A new Turing test study suggests a more interesting answer. AI often passes for human, yes – but the ancient move of asking follow-up questions still trips it up.
The imitation game, briefly
Turing’s 1950 proposal was deliberately practical. Instead of arguing about whether machines can truly think, he described a party game: a judge exchanges written messages with a person and a machine hidden behind screens, and tries to tell which is which. If the machine fools the judge often enough, Turing argued, the question of machine thinking has lost most of its meaning.
Why the test matters more than ever in 2026
That game stopped being hypothetical the moment chatbots entered daily life. The Turing test now runs constantly, informally and without anyone’s consent: in customer support chats, news comment sections, dating apps, classrooms and family group messages. Every one of those encounters is an uncontrolled experiment in whether people can tell a person from a product.
The study described in this article ran that experiment properly, at a scale nobody had attempted before, with more than 72,000 judgments from almost two thousand people. Its conclusion is uncomfortable middle ground: the machines are good enough that judges barely beat a coin flip, and the human tool of interrogation is still sharp enough to matter.
How the Turing Test Migrated Into Everyday Software
Alan Turing’s thought experiment began in a 1950 paper on machines and philosophy and ended up as an uninvited guest in product meetings. Every verification flow, CAPTCHA replacement and bot-detection dashboard quietly asks the imitation-game question, and usually answers it badly. The new study puts a number on that hidden industry’s problem: untrained humans perform near chance.
That number should reframe arguments about chatbot disclosure, AI labeling and platform policy. When people cannot reliably tell, consent-based design collapses – users agree to talk to a system they cannot identify as a system. The Turing test stops being trivia and becomes a compliance requirement hiding in plain sight.
Inside the Study: 72,000 Judgments, 1,916 Judges, 37 Models
The research was led by Mengmi Zhang of Nanyang Technological University in Singapore and published in Nature Human Behaviour. Its central design question was deceptively simple: how good are humans, really, at telling AI-written responses apart from human ones – and under which conditions does that ability improve?
Six categories, one question
The team assembled sample responses from 636 people and 37 different AI models, then recruited a separate pool of 1,916 human judges. The responses fell into six categories spanning language and vision: writing captions for photos, holding conversations, producing word associations, and identifying colours and objects in images. Judges read each response and voted on whether a human or a machine produced it.
Deliberately older models
Initial data collection took place in 2023 and 2024, with additional control experiments in 2025. Some tests included GPT-4o, Claude 3.7 and Llama 3.3 – capable systems, but not the newest ones. The authors are explicit that the findings describe an earlier stage of AI development, not a live measurement of today’s frontier chatbots.
The study at a glance
| Parameter | Value |
|---|---|
| Sample providers | 636 people + 37 AI models |
| Human judges | 1,916 |
| Total judgments | More than 72,000 |
| Task categories | Six, spanning language and vision |
| Collection window | 2023-2024, control experiments 2025 |
| Notable models | GPT-4o, Claude 3.7, Llama 3.3 |
What Counted as a Win
There was no single pass threshold in this Turing test study – the authors deliberately avoided crowning a victor. Instead every condition produced an accuracy band, and the interesting story lives across the bands. A method that performs at chance on transcripts but climbs during live interrogation is not measuring the same skill twice; it is measuring two different things while both go by the name of AI detection.
Barely Above Chance: What the Turing Test Scores Actually Mean
Judges performing like coin flips
In the main language tests, judge accuracy landed between 53% and 57% – only modestly above the 50% expected from random guessing. Judges correctly identified captions as AI-generated just 46% of the time, and caught AI-written conversational responses only 47% of the time. Humans were mislabeled often too, so the errors ran in both directions.
Those numbers deserve to sit in for a moment. When untrained observers read a chat reply, they are effectively guessing. The Turing test, conducted exactly as Turing framed it, is now a near-coin-flip for some of the most common language tasks – and that was measured on models that are already a generation old.
Where the machine fooled judges most
The beach photo lesson
The judges’ scores also separated two ideas people constantly merge: being a convincing imitator is not the same as being accurate. In one caption example, GPT-4 correctly described people on a beach but missed a visual trick that made a man appear to hold his own detached head. “Being right or wrong is not the same as imitation,” the authors write – and that distinction is the philosophical heart of the whole study.
Why Barely Above Chance Is the Honest Headline
Five percentage points of edge sounds like a verdict when sample sizes are huge, and 72,000 judgments make this one statistically real. But statistical and practical significance part company here: a detection method that works roughly 55% of the time cannot anchor policy, moderation queues or fraud screening. A near-coin-flip Turing test is a warning label for anyone outsourcing the judgment to crowds, not a green light.
It also reframes the opposite error. Humans did not merely fail to catch the machines; they convicted each other at similar rates, mislabeling genuine people as bots often enough to poison any process that trusts such flags. Both directions of mistake matter when the flagged party is a person seeking help from customer service.
When Interrogation Works: The Live-Chat Experiments
Seven conversations, seven detections
In a smaller experiment, the researchers let judges question a person and a chatbot directly. The judges chose their own topics and asked their own follow-up questions before deciding which was which. In seven conversations where GPT-3.5-Turbo was instructed to pose as a person, the judges identified the machine every single time – seven out of seven.
When safeguards give the game away
That perfect score arrived with asterisks. Response style and response timing could offer clues, and some probing questions triggered safety responses that effectively announced the chatbot as AI. In other words, part of that perfect run may reflect guardrails confessing rather than judges seeing through an otherwise convincing imitation.
The harder Llama trial
To reduce those accidental disclosures, the team ran a larger follow-up with a different model, Llama-3-70B-Instruct, with safeguards disabled. Across 188 trials involving 28 participants, judges made mistakes – but they became better at identifying machine responses as conversations grew longer. The longer the thread, the worse the chatbot’s hiding place.
| Aspect | Transcript Turing Test | Interrogation Test |
|---|---|---|
| Material | Prepared answers, some obvious AI failures removed | Real-time chat with live follow-ups |
| Judge’s power | Read and vote only | Choose topics, ask questions, press harder |
| Best result | 53-57% accuracy, barely above chance | 7 of 7 catches against GPT-3.5-Turbo |
| Main weakness | Flatters the chatbot; curated samples | Style, timing and safety tells leak the answer |
Why the Seven-of-Seven Result Still Matters
Skeptics are right to asterisk the perfect score – timing, style and safety confessions did much of the work. But the direction of the effect is the finding: interrogation created separations that transcripts never produced, and the larger safeguard-free Llama 3.3 trial preserved that direction while losing the perfection. In Turing test research, a consistent direction across two independent designs is rarer, and more useful, than any single flawless run.
Why Follow-Up Questions Trip Up the Turing Test's Best Imitations
Consistency is a different skill than fluency
A single polished answer is cheap. Keeping that answer’s story straight under three rounds of pressing questions is something else entirely. Follow-up questions test memory, self-consistency and the ability to recognise what both parties now know – the background machinery that human conversations run on and that the Turing test was never designed to probe directly.
Grounding, repair and common ground
Humans constantly repair: they notice when a conversation has drifted, restate, clarify and apologise. Chatbots can simulate repair, but each turn is generated against a finite context, and maintaining a believable persona across unpredictable questions is a genuinely harder problem than producing one convincing reply. Interrogation converts a one-shot generation task into a consistency marathon – and that is where the seams show.
Why transcripts flatter chatbots
The transcript half of the study handed AI its best possible highlight reel: prepared material with repetitive loops and premature endings removed. Live questioning removes that editing room. The study cannot isolate the effect of questioning alone – the live and transcript experiments used different models under different instructions – but the pattern matches everyday experience: anyone who has argued with a chatbot knows the second follow-up is where the mask slips.
The Tells Nobody Coded
Several of the strongest cues in the live experiments were designed in by no one. Response timing betrays a machine that produces a paragraph instantly and an equally instant rebuttal. Typographical consistency – flawless spelling under pressure, zero mid-thought edits – reads as uncanny to people even when they cannot name why. And safety refusals remain a confession delivered in a customer-service voice.
Humans are leaky, and that is their tell. People contradict themselves, drift off topic, restate a point for the third time with rising irritation, and let mood color the prose. A chatbot that never gets bored during a Turing test interrogation is suspicious in precisely the way the study’s judges kept sensing without being able to articulate.
The Fine Print: Limits Every Reader Should Keep in Mind
A snapshot, not a forecast
The study describes an earlier stage of AI development. Models such as GPT-4o, Claude 3.7 and Llama 3.3 anchor the data; today’s newest systems are almost certainly harder to catch, not easier. Nothing in the paper establishes how current frontier models would score if judges ran the same Turing test protocol against them tomorrow.
Small samples and confounded conditions
Seven conversations is not a verdict; 188 trials across 28 participants is better but still modest. The transcript and live experiments differed in models and instructions, so the effect of questioning alone cannot be cleanly isolated. And recognizing one chatbot in one test does not establish that people can reliably distinguish AI from humans across different systems, languages and situations.
Prepared material is a kind filter
Researchers discarded conversations that became repetitive or ended prematurely, which means the transcript results measure imitation under controlled conditions, not the messy reality of detecting every kind of chatbot output in the wild. The results describe performance under controlled conditions rather than detection of every kind of chatbot output.
What a Stronger Turing Test Would Need
The study’s own limitations sketch the specification for its successor: the same models tested under both transcript and live conditions, pre-registered follow-up scripts, longer conversations sampled from real workloads, and judges calibrated against known-AI control groups. Until such a run exists, the Turing test’s 2026 verdict should be quoted with its caveats attached – the numbers are real, and their portability to frontier models is genuinely unknown.
What Builders Should Do With the Narrowing Turing Test Gap
Design for disclosure, not detection
If untrained judges cannot beat a coin flip, products that quietly let AI impersonate humans are betting on a detection ability users do not have. Honest disclosure – clear labels, persistent indicators, handoff moments – is engineering ethics, not marketing. The same logic underpins the regulatory push for watermarking; our report on OpenAI’s plan to watermark ChatGPT text in the EU shows the industry reaching for exactly that conclusion.
Conversation is the detector
The practical lesson for anyone who runs chatbots is that the transcript is the flattering format and the live thread is the honest one. Teams shipping conversational features should test under interrogation-style evaluation – multi-turn, topic-switching, contradiction-seeking – rather than single-prompt demos. We explored the mirror-image problem, AIs judging AIs, in our piece on reverse Turing tests where frontier models failed to detect each other; neither species of detector is reliable enough to stand alone.
Trust, institutions and the human in the loop
Judiciary experiments like the UK proposal to let an AI judge decide low-value claims will lean on public confidence that humans can tell machine judgments from fair ones – a confidence this study says nothing good about. Meanwhile the industry keeps selling automated decisions: our breakdown of Human in the Loop 4.0 shows why the human review layer is exactly the machinery – memory, consistency, the right to ask why – that the follow-up question tests.
A Checklist for Conversational AI Testing
For teams shipping assistants, the study converts neatly into a QA backlog. Run multi-turn adversarial scripts that contradict earlier statements and measure how often the assistant agrees with itself. Profile the latency of answers that change their minds. Test refusal behaviour for character breaks under pressure. Track topic-switch recovery and persona drift across long sessions. Treat every interrogation-style bug you skip in QA as a question you are hoping no real user will ever ask.
The transcript demo always wins; the live interrogation always tells. That asymmetry is the most actionable sentence in the entire Turing test literature, and it applies whether you are building the assistant, procuring one, or writing the policy that governs it.
Conclusion: Ask One More Question
The 2026 MIT study closes with a strange symmetry: the machines got better at sounding human, and the humans got suspicious of everything, including themselves. Somewhere in that mess the Turing test matures from a parlor trick into an engineering constraint. The imitation game is solvable at the level of style and unsolved at the level of accountability – being answerable, over time, for what you said a minute ago.
For builders, the lesson is practical rather than philosophical: ship the memory, the consistency and the willingness to be asked why, and follow-up questions stop being a vulnerability. For everyone else, the lesson is civic: trust your unease, and spend one extra question per conversation. It is the cheapest Turing test ever devised and, per this study, the only one still scoring above chance.
Working on AI that people can trust?
Get a free 30-minute consultation on conversational AI strategy, detection or safety – or read how we build and audit AI systems.
Book a Free ConsultationFrequently Asked Questions
Did AI finally pass the Turing test?
No – and the study’s authors would resist the framing. Judges performed barely above chance on several transcript tasks, which means AI passed in some individual comparisons, not that the Turing test as a whole was passed. Models from 2023-2024 anchor the data, so the result is a snapshot rather than a finish line.
Can follow-up questions always catch a chatbot?
Not always, and not by magic. The seven perfect detections against GPT-3.5-Turbo partly reflected timing, style and safety-refusal tells. With safeguards off, judges made mistakes – they simply improved as conversations grew longer. Follow-up questions raise the odds; they are not a reliable detector on their own.
Should I assume every chat reply might be AI?
Practically, the study’s numbers imply that reading alone cannot tell you. If the relationship matters – contracts, healthcare, finances, safety – verify identity through channels and callbacks rather than conversational impressions, as our guide to spotting AI-powered scams recommends.
Is the Turing test still a useful goal?
As a finish line, no – researchers abandoned it as a benchmark decades ago. As a diagnostic, this study suggests it remains healthier than ever: by changing one rule of one Turing test and watching behaviour change, the authors located precisely which conversational abilities machines have acquired and which they have not. That makes the test less a trophy and more a measuring instrument.
The follow-up question is the practical residue of that instrument, and the single most usable finding for anyone who simply wants to know who is typing. Ask one more question than the conversation requires, and listen to what the answer costs.
What should schools and employers do about AI writing?
Oral defence is the oldest detection method humans have, and the study explains why it still works: it is interactive by design. Asking a student or candidate to explain, extend and defend a submitted text recreates the interrogation condition that separated humans from models. Detection software, by contrast, operates in the transcript condition where both sides perform at chance.
