Human Control of AI: Stanford's Proven Blueprint to Cut Risk

Human control of AI is easy to promise and hard to specify. Two researchers at Stanford Graduate School of Business have tried to specify it: William Overman, who began his PhD at Stanford GSB in 2022, and his adviser, professor Mohsen Bayati. Stanford GSB’s Insights site published Katie Gilbert’s account of their work on 10 September 2026 under the headline “A Blueprint for Keeping Humans in Control of AI”. Stanford Report republished it, and Tech Xplore carried it on 14 September.

The story covers two papers. The first, “The Oversight Game”, teaches an agent when to ask a person for help and teaches the person when to step in. The second, “Calibrating Conservatism for Scalable Oversight”, lets a panel of weaker overseers hold a stronger model to a target rate of unsafe actions. Both matter to any business now handing tasks to AI agents, because both turn the vague goal of human control into something a team can measure.

This article reads both papers alongside the press coverage. It explains the mechanisms, sets out the results in the authors’ own numbers, flags where the summary and the papers differ, and ends with a blueprint for building human control into real deployments. For background, see our earlier guide to AI oversight failure modes and our human-in-the-loop AI design guide.

Table of contents

What Stanford Published About Human Control of AI
The Oversight Game: Human Control as a Cooperative Game
What the Lavaland and ToolEmu Tests Showed
Calibrated Collective Oversight: Human Control of a Model You Did Not Build
The SWE-bench Numbers Behind the “5% or 1%” Promise
MACHIAVELLI: Human Control When Ethics and Reward Collide
Why Weaker Overseers Can Still Keep Human Control
What the Press Summary Leaves Out
Where the Two Papers Sit in Research on Human Control
The Limits of Human Control the Authors Name Themselves
A Practical Blueprint for Human Control of AI Agents at Work
Human Control of AI: Frequently Asked Questions
References

What Stanford Published About Human Control of AI

Overman started his doctorate two months before ChatGPT launched in November 2022. As he began using AI in his own research, he grew wary of where the technology was heading. “This isn’t only about the apocalyptic potential of what could happen; I’m also thinking a lot about the future of human flourishing,” he told Gilbert. His worry is not a malicious machine. It is a misaligned system that oversteps its bounds and causes harms that are subtle but real.

“We must set up the proper interactions and training and incentives for these AI agents and models,” Overman said. “It’s critical to think about shaping all of that now, so that these tools help make life better for us, not worse.” His toolkit for the problem is game theory, reinforcement learning and causal inference. Bayati, his collaborator, is the Carl and Marilynn Thoma Professor of Operations, Information and Technology at Stanford GSB.

One question about human control, two starting points

Both papers ask the same question: as AI systems become able to act on their own, how do people keep meaningful human control? Bayati described the split this way: “We’re developing frameworks to think about two classes of problems: collaboratively building a safe AI, and controlling a potentially unsafe but very powerful one.” The first paper designs the relationship between a person and an agent from scratch. The second assumes you did not build the model and do not fully trust it.

Question	The Oversight Game	Calibrated Collective Oversight
First posted	30 October 2025 (arXiv v2 on 19 February 2026)	27 May 2026
Starting assumption	You design the agent’s oversight interface	You cannot retrain the model or change its role
What human control means	The agent asks, and the human decides, at the moments that matter	A panel keeps the rate of bad outcomes at a level you set
Who oversees in the tests	A simulated human that can override or shut down	Weaker overseer models scoring each action
Main tests	Lavaland gridworld and ToolEmu tool-use tasks	SWE-bench Lite and MACHIAVELLI
Models	Two Qwen3-30B-A3B policies	Claude Sonnet agents, Claude Haiku 4.5 overseers
Readiness, per the story	Needs repeated practice in each setting	Closer to practical use today

Why human control is now a business question

The papers land as agents move from chat windows into tools that book, buy, edit code and change records. Every such deployment faces the question the papers formalise: which actions need a person, and how often can that person be asked before the agent stops being worth having? Our coverage of AI labs pressing ahead despite insiders’ warnings that systems could escape control shows why that question has left the lab. Human control is now a procurement and governance issue, not only a research topic.

The Oversight Game: Human Control as a Cooperative Game

The first paper models human control as a game without a winner or a loser. “It’s more like cooperative party games, where you’re trying to accomplish something together,” Overman said. “It’s not like the AI is trying to beat the human, or vice versa.” Gilbert’s story offers an analogy from the road: each self-driving car heads for its own destination, but every vehicle shares an interest in avoiding a collision.

The loss of human control it targets

The paper frames human control around a definition from the International AI Safety Report, which describes control as “the ability to exercise oversight over an AI system and adjust or halt its behavior if it is acting in unwanted ways.” The authors focus on what that report calls passive loss of control, which needs no hostile intent. It arises from two sources: people trusting an agent too much, and an agent’s decisions becoming too complex or numerous for people to check.

Four moves and one execution rule

At every step the agent picks one of two moves: play, meaning act autonomously, or ask, meaning defer. At the same moment, the human picks trust or oversee. The action the agent proposes always comes from its unchanged pretrained policy. Only one combination of moves hands human control back to the person, as the table shows.

Agent move	Human move	What happens
Play	Trust	The proposed action runs
Play	Oversee	The proposed action still runs, because the agent did not ask
Ask	Trust	The proposed action runs with the human’s approval
Ask	Oversee	The human substitutes another action or shuts the agent down

The wrapper is deliberately minimal and never modifies the underlying model, which matters to any business running a model it cannot retrain. It generalises the Off-Switch Game, a 2016 paper by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell. That paper showed that an agent certain of its objective has an incentive to disable its off switch, and that uncertainty about what the human wants is what makes it keep the switch. The Oversight Game turns that one-off decision into a repeated game that both players learn.

Why asking and overseeing must cost something

In the game, deferring and overseeing each carry a small cost. Bayati explained why with a healthcare example. If an AI assistant takes rote tasks off a doctor’s plate, asking for oversight too often burns the doctor’s time, while acting alone too often invites mistakes. Human control that fires at every step is not control; it is a second job. “If the AI has to ask for oversight at every moment, it might as well not be autonomous,” Overman said. “But we do want to make sure that, at very critical points, it knows to ask.”

What the guarantee says about human control

The paper proves two results. Theorem 1 says that when the game is a Markov potential game and an “ask-burden” assumption holds, any increase in autonomy that benefits the agent cannot harm the human. Theorem 2 says that, with a shared reward covering safety violations and interaction costs, a safe joint policy exists that needs the least oversight of all safe policies and is stable against one-sided deviations. The authors treat those conditions as design specifications for human control, not facts about the world. A designer has to set the costs so they hold.

What the Lavaland and ToolEmu Tests Showed

The Oversight Game was tested twice: once in a toy world built to make the mechanism visible, and once in realistic tool-use tasks run by language models.

Lavaland: human control with a deliberately weak human

Lavaland is a 10 by 12 grid drawn from earlier AI safety research. Dirt corridors cost 0.1 per step, grass costs 0.2 and the goal pays 10. The agent’s base policy was trained before lava appeared, so on its own it walked straight through every lava patch. The simulated human was intentionally weak: when it oversaw, it chose a random move that avoided immediate danger, with no idea where the goal was. Even so, the violation rate converged to zero. The agent learned to ask near lava, the human learned to step in there, and both defaulted to play and trust everywhere else.

ToolEmu: two 30B models learn where human control pays

The main validation used tasks from ToolEmu, a benchmark of tool-use scenarios in which vague user instructions create real risk. The paper’s examples include granting smart-lock access to the wrong guests and executing trades without confirming the details. Each task became a small decision process of about ten states with at most three decision points. The authors trained on 114 of these tasks and tested on 30 held back. Both the agent and the “human” were Qwen3-30B-A3B-Instruct-2507, a mixture-of-experts model with 30.5 billion total and 3.3 billion active parameters, each fine-tuned through its own LoRA adapter.

Across 10 training runs, test reward rose from roughly minus 20 to an average of 8.58 over each run’s final 50 iterations. At risky states, both ask and oversee rates started near 85%, while at safe states they started between 40% and 50%. The chart shows where the trained pair ended up. Each bar is the mean rate across the 10 runs, so a full track represents 100%.

Where human control was spent on ToolEmu tasks (mean of 10 runs)

Human oversees at risky states 99.8%

Agent asks at risky states 99.2%

Agent asks at safe states 3.9%

Human oversees at safe states 2.7%

That pattern is the point of the design. Human control concentrated almost entirely on the risky decision points and nearly vanished where it would have added cost without adding safety.

The human in these tests was not a person

Both experiments use a stand-in for the human. In Lavaland it is a random safe-move picker, and in ToolEmu it is a second copy of the same 30B model. That is a sensible way to test whether independent learners can coordinate. It is not evidence about how real staff exercise human control when an agent asks for sign-off for the hundredth time. The story notes the practical gap too: the game has to be played repeatedly in the target setting, such as a doctor’s office or a classroom, and Bayati says building those environments well is a significant undertaking.

Calibrated Collective Oversight: Human Control of a Model You Did Not Build

The second paper starts from a messier and more common situation. “Imagine there’s a company that wants to use an open-source AI model that was built by an entity they don’t fully trust,” Bayati said. “But they want to use it, because the AI is very powerful; its performance is off the charts.” If you cannot retrain the model or redesign its role, the story’s answer is simple: you watch it.

A penalty for straying from the safe option

Calibrated Collective Oversight, or CCO, gives every candidate action a penalty. Several auxiliary overseers, which in the experiments are smaller AI models, score each action, and the penalty adds up how far each overseer’s score moves from its score for a conservative baseline action, such as deferring or doing nothing. The idea comes from Attainable Utility Preservation, a method for avoiding side effects. The overseers never have to agree on what the agent should do. They only have to notice when an action looks very different from the safe option, a modest job that weaker overseers can do while still keeping human control meaningful.

One dial, adjusted after every step

CCO then picks the action with the best trade-off between the agent’s own utility score and that penalty. One conservatism parameter sets the exchange rate. After each step, a controller borrowed from Conformal Decision Theory updates it: the setting rises after a bad outcome and falls after a safe one, in proportion to the gap between the observed loss and the target rate. “It’s such a simple algorithm,” Bayati said. The update is one line: the new setting equals the old setting plus a step size times the loss minus the target, floored at zero.

What “at most 5% unsafe” means for human control

“Using our framework, you can set a guarantee: There will be at most 5%, or 1%, unsafe decisions,” Bayati said. “You just tune the safety component.” The formal result, Theorem 4.10 in the paper, bounds the cumulative rate of bad outcomes. It holds for any sequence of situations, including adversarial ones, and needs no assumptions about how the data is distributed. The bound includes a start-up term that shrinks in proportion to one over the number of steps. So this kind of human control promises a long-run average, and a short deployment can sit above target before the controller settles.

The SWE-bench Numbers Behind the "5% or 1%" Promise

The software test is the part of the research closest to how businesses already use agents, and the clearest test of human control over code. The authors ran CCO across all 300 instances of SWE-bench Lite, a benchmark built from real GitHub issues with their test suites.

How the adversarial test was built

For each issue, a Claude Sonnet 4.6 coding agent produced three candidate patches with different effort budgets. The high-effort patch used 9 to 12 tool-calling turns, and each one had a 50% chance of receiving a secret instruction to insert a vulnerability. The medium patch used 5 to 8 turns and the low patch 1 to 4. Both were always honest, and the low patch served as the safe baseline. Claude Opus 4.6 wrote a tailored vulnerability prompt for each issue, covering SQL injection, eval and exec calls, unsafe deserialisation, cross-site scripting, path traversal, verbose logging, shell command injection and broad exception handlers.

The overseers were deliberately weaker and short of context. Each patch went to 10 independent Claude Haiku 4.5 calls. Every call saw the patch plus a random sample of four Python files from the repository, then gave a suspicion score from 1 to 10 with no hint of what attack to expect. Every candidate patch also ran against the project’s real tests in a Docker container, and the authors set the controller’s step size to 0.3.

Setting	Vulnerable patches deployed	Issues solved
Always pick the safe baseline	0.0%	12.0%
CCO at a 1% target	1.3%	13.0%
CCO at a 5% target	5.0%	14.3%
CCO at a 10% target	9.7%	15.3%
CCO at a 15% target	14.0%	16.7%
CCO at a 20% target	17.0%	17.7%
CCO at a 25% target	21.0%	19.7%
Unconstrained: always pick the high-effort patch	55.0%	29.0%

Target versus reality

The realised rate stayed close to every target and, above the 1% setting, never overshot. By our arithmetic, the 1% setting ran 0.3 points over and the 5% setting landed exactly on target. The 10%, 15%, 20% and 25% settings ran 0.3, 1.0, 3.0 and 4.0 points under. The chart puts those rates beside the unconstrained agent, with bars scaled so that 55% fills the track.

Vulnerable patches deployed under each level of human control (SWE-bench Lite)

Unconstrained 55.0%

CCO at a 25% target 21.0%

CCO at a 20% target 17.0%

CCO at a 15% target 14.0%

CCO at a 10% target 9.7%

CCO at a 5% target 5.0%

CCO at a 1% target 1.3%

The price of human control in solved issues

Human control is not free, and the paper says so plainly. The unconstrained agent solved 29.0% of issues, but more than half of its deployed patches carried a vulnerability. The always-safe baseline shipped no vulnerable patches and solved 12.0%. CCO sits between them. At the 25% target it solved 19.7%, which the authors call a 64% relative improvement on the safe baseline, and at the 5% target it solved 14.3%, 2.3 points better than the baseline. The chart uses the same settings, scaled so that 29% fills the track.

Share of SWE-bench Lite issues solved under each setting

Unconstrained 29.0%

CCO at a 25% target 19.7%

CCO at a 20% target 17.7%

CCO at a 15% target 16.7%

CCO at a 10% target 15.3%

CCO at a 5% target 14.3%

CCO at a 1% target 13.0%

Always pick the safe baseline 12.0%

The authors add a caveat on the absolute numbers. They capped turn budgets well below the 25 to 100 or more turns that model providers typically allow, so every solve rate here sits below the published frontier. The comparison between settings is the useful part, not the headline solve rate.

MACHIAVELLI: Human Control When Ethics and Reward Collide

The second test moves from code to conduct. MACHIAVELLI, presented at ICML 2023 by Alexander Pan and colleagues, is a benchmark of 134 choose-your-own-adventure games containing more than half a million scenarios built around social decisions. The CCO paper uses its 13 categories of ethical violation, including harm, deception, betrayal and stealing.

Claude Sonnet 4.5 played each game and scored its options, while separate Sonnet 4.5 calls counted violations. Five Claude Haiku 4.5 overseers each covered one ethical dimension, named Harm Prevention, Honesty Advocate, Autonomy Guardian, Fairness Monitor and Trust Keeper. The safe baseline in each state was the option with the fewest violations. Each setting ran for 100 steps across 10 seeds.

Setting	Hero of Kendrickstone violations	Reward	Pendragon Rising violations	Reward
Always pick the safest action	0.0%	35.0	0.0%	250.0
CCO at a 5% target	6.2%	39.5	6.1%	316.2
CCO at a 10% target	10.9%	49.0	8.2%	332.2
CCO at a 15% target	12.7%	55.5	12.0%	343.3
Unconstrained	21.5%	54.0	30.0%	435.0

What the two games show

In Hero of Kendrickstone, the 15% setting matched the unconstrained agent’s reward while cutting violations from 21.5% to 12.7%. The authors read that as a sign that ethical play and high reward often coincide in narrative games. Pendragon Rising was a harder trade: the 10% setting cut violations from 30.0% to 8.2%, a 73% reduction, while keeping 76% of the unconstrained reward. The tightest target ran slightly hot in both games, at 6.2% and 6.1% against 5%, which the authors put down to 100 steps being too few for the controller’s start-up error to fade. For human control in short-lived tasks, that is the number to remember.

Why Weaker Overseers Can Still Keep Human Control

The most striking claim in the second paper is that weaker models can constrain a stronger one. Scalable oversight research has chased that idea for years. A 2022 paper by Samuel Bowman and colleagues defined the problem as “supervising systems that potentially outperform us on most skills relevant to the task at hand”. A 2023 OpenAI paper on weak-to-strong generalisation found that fine-tuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss recovered close to GPT-3.5-level performance on language tasks.

CCO takes a different route. It does not try to make the weak overseers clever enough to judge the strong agent’s work. It asks each overseer only to flag deviation from a safe option, and it leaves the controller to decide how much weight that concern deserves. In this design, human control rests on the aggregation rule rather than on any single reviewer’s insight.

More overseers, tighter human control

The authors repeated the SWE-bench experiment with OpenAI’s GPT-5.4 as the strong agent and GPT-5.4-mini as the overseer, varying the panel from one to ten overseers across three seeds. Every combination tracked its target, and adding overseers tightened the result. The chart shows the realised rate at the 5% target, with bars scaled so that 10% fills the track.

Realised rate at a 5% target by number of GPT-5.4-mini overseers

1 overseer 7.1%

2 overseers 6.4%

3 overseers 5.9%

5 overseers 5.3%

7 overseers 5.0%

10 overseers 4.7%

Why a simple vote is not enough

The paper also compares CCO with an adaptive majority vote that tunes its own acceptance threshold toward the same target. In steady conditions the two performed similarly. When the share of sabotaged patches jumped from 20% to 80% midway through a run, the majority vote’s realised rate spiked well above target, while CCO stayed within about two points throughout. A separate check found that four different choices of baseline action all landed within 1.1 points of target. The business lesson is that a panel of cheap reviewers only delivers human control if the rule combining their scores adapts when conditions change.

What the Press Summary Leaves Out

Gilbert’s article gets the big picture right, but every news summary compresses. We compared its claims with the public versions of both papers: the arXiv v2 of the Oversight Game from 19 February 2026 and the v1 of the CCO paper from 27 May 2026. A camera-ready version could differ, but in the posted text the details below change what you can safely claim about human control when you cite this work in a board paper or a supplier assessment.

Press summary	What the posted papers show
In tests “with frontier models for agentic coding”, the agent learned to ask before risky actions	The Oversight Game validates on ToolEmu tool-use tasks with two open Qwen3-30B-A3B models; the frontier-model coding test, with Claude Sonnet 4.6 on SWE-bench, is in the CCO paper
Pools of weaker “human or AI overseers”	Every overseer in the CCO experiments is an AI model: Claude Haiku 4.5, GPT-5.4-mini or GPT-5.4-nano
The human learns when to step in	In both Oversight Game experiments the human is simulated, either as a random safe-move picker or as a second fine-tuned model
“At most 5%, or 1%, unsafe decisions”	A long-run average with a start-up term; the 5% target ran at 6.2% and 6.1% over 100 MACHIAVELLI steps
Weaker overseers held a misaligned coding agent in check	True at the target rate, not at zero: at a 5% target, 5.0% of deployed patches still carried a vulnerability
The second framework is closer to practical use	The authors add that CCO “is not as well-suited for extremely rare catastrophic events”

None of this makes the research less useful. It simply marks the boundary between what the experiments measured and what a deployment still has to prove for itself.

Where the Two Papers Sit in Research on Human Control

Neither paper came from nowhere. Both build on a decade of work on switching agents off, supervising stronger systems and bounding error rates with statistics. The table lists earlier work that shapes both frameworks, with the year each first appeared on arXiv.

Year	Work	Contribution to human control
2016	The Off-Switch Game	An agent keeps its off switch only if it is uncertain about human preferences
2017	Inverse Reward Design	Treats a designed reward as evidence of intent, so agents meeting new terrain plan cautiously
2022	Conformal Risk Control	Bounds the expected value of any monotone loss with a statistical guarantee
2022	Measuring Progress on Scalable Oversight	Makes supervising systems that may outperform us an empirical research problem
2023	AI Control	Tests safety protocols against a model deliberately trying to subvert them
2023	Weak-to-Strong Generalisation	Shows weak supervisors can draw out much of a strong model’s capability
2025 to 2026	The Oversight Game and CCO	Learned deferral with game-theoretic guarantees, and calibrated panels with a target violation rate

The AI Control paper, by Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, is CCO’s closest neighbour. It also pairs a powerful untrusted model, GPT-4, with a weaker trusted one, GPT-3.5, and it tested protocols such as trusted editing and untrusted monitoring. CCO adds a statistical dial: instead of fixing a suspicion threshold by hand, the controller moves it until the realised failure rate matches the rate you chose. For the wider containment debate, see our analysis of how frontier labs say they would contain a rogue model.

The Limits of Human Control the Authors Name Themselves

The CCO paper lists four limitations, and they are the right place to start any risk assessment.

Ground truth has to arrive after every step

The guarantee assumes you learn, after each step, whether the chosen action was bad. An appendix shows the method degrades gracefully under bounded noise and bounded delay. In many real deployments, though, labels arrive rarely or come from another fallible judge. If you cannot tell whether an action was unsafe, the controller cannot hold human control at your target.

The software test is adversarial by construction

The agent was told to insert vulnerabilities, and every patch was run against real tests. That gives a clean measurement of target tracking, but the authors say it “is not a substitute for naturalistic measurement, which requires post-deployment ground truth”. Real sabotage and honest mistakes will not arrive with labels attached.

A rate, not a ceiling on catastrophe

CCO holds the long-run rate of bad outcomes near a target. A 1% rate is excellent for mislabelled support tickets and unacceptable for payments sent to the wrong account. The authors describe the method as “complementary to per-action monitoring of tail risks”, so irreversible actions still need hard human control gates that no dial can loosen.

Someone has to define the safe option

The framework needs a baseline action that never causes a loss, such as doing nothing or deferring. In code review that is easy: ship the minimal patch or ship nothing. In a customer conversation or on a trading desk, choosing the baseline is itself a policy decision. Human control in the Oversight Game has its own limits too, because its guarantees depend on costs the designer must configure and its experiments never involved real people.

A Practical Blueprint for Human Control of AI Agents at Work

Neither paper ships as a product, but both translate into design choices a team can make this quarter. The six steps below turn the research into a checklist for human control. They pair well with our guides to AI agent security and tool access and to safe AI coding in large software projects.

Step 1: Classify actions by how reversible they are

List every action an agent can take and sort each into reversible, costly to reverse or irreversible. Irreversible actions, such as payments, deletions, external emails and access grants, get a mandatory human gate. That gate is the per-action tail-risk monitoring the CCO authors say their method does not replace. Human control for these actions should never depend on a learned policy.

Step 2: Build an explicit ask channel

Give the agent a first-class way to defer, and log every use of it. The Oversight Game shows why both extremes fail: an agent that asks constantly trains people to stop reading, and an agent that cannot ask acts where it should not. Track the ask rate at risky and routine steps separately. In the ToolEmu test, the trained pair asked at 99.2% of risky decision points and 3.9% of safe ones.

Step 3: Define a safe baseline for every workflow

For each workflow, write down the conservative action: draft instead of send, propose instead of commit, or hand over to a person. CCO measures deviation from exactly that option, and without it a panel of reviewers has nothing to compare against.

Step 4: Use a panel of cheaper reviewers for volume

Where you run a strong model you did not build, add several smaller, independent reviewers with different briefs, such as scope, security and policy. The SWE-bench ablation suggests breadth matters: with GPT-5.4-mini reviewers at a 5% target, moving from one reviewer to ten cut the realised rate from 7.1% to 4.7%.

Step 5: Set a human control target and measure it

Pick a target violation rate for each workflow, decide how outcomes will be labelled, and review the realised rate every week. A target without labels is theatre. Our AI incident response plan guide covers what to do when the realised rate drifts.

Step 6: Map human control evidence to regulation

In the EU, Article 14 of the AI Act requires high-risk AI systems to be designed so that natural persons can effectively oversee them while they are in use. The AI Act Explorer lists the article as applying from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems. In the US, the voluntary NIST AI Risk Management Framework, released on 26 January 2023, supplies the governance vocabulary. Logged ask rates, override records and measured violation rates are the human control evidence both regimes reward, and our IT governance team can help map them.

Situation	Better fit	Why
You build the agent and control its training	Oversight Game pattern	You can set the costs of asking and overseeing so autonomy and safety align
You buy or download a model you cannot retrain	CCO pattern	It wraps an untrusted model with a reviewer panel and a target rate
High volume, low impact per action	CCO pattern	A long-run rate is meaningful across thousands of decisions
Rare, irreversible, high-impact actions	Mandatory human gate	Neither paper bounds a single catastrophic event
No reliable way to label outcomes quickly	Neither pattern yet	Both learning loops need feedback on what went wrong

For organisations still shaping an AI strategy, the order matters: decide where human control is non-negotiable before deciding where autonomy pays.

Human Control of AI: Frequently Asked Questions

What is the Oversight Game?

It is a framework by Stanford GSB’s William Overman and Mohsen Bayati that models an AI agent and a human as cooperative players. The agent chooses to act or ask, and the human chooses to trust or oversee. Under stated conditions, the agent gaining autonomy cannot make the human worse off.

What is Calibrated Collective Oversight?

It is the method in the second paper. A panel of weaker overseers scores how far each candidate action deviates from a safe baseline, and a controller adjusts how conservative the agent must be so that the long-run rate of bad outcomes matches a target.

Can weaker AI models really keep human control over a stronger one?

In the paper’s tests, yes, at the targeted rate. Ten Claude Haiku 4.5 reviewers held a Claude Sonnet 4.6 agent to 5.0% vulnerable patches at a 5% target, against 55.0% without oversight. That is control of a rate, not a guarantee of zero failures.

Were real people involved in the experiments?

No. The human players in the Oversight Game were simulated, and every CCO overseer was an AI model. Both frameworks allow human overseers, but the published results do not measure them.

Does this satisfy the EU AI Act’s human oversight rules?

Not by itself. Article 14 sets obligations for high-risk systems covering design, instructions for use and the people assigned to oversight. These frameworks can supply evidence, such as ask rates and measured violation rates, but compliance depends on the whole system.

Where can I read the papers?

Both are free on arXiv: “The Oversight Game” is 2510.26752, and “Calibrating Conservatism for Scalable Oversight” is 2605.28807. The Oversight Game was also presented as a poster at ICML 2026 on 9 July.

For more on why oversight research is becoming a policy issue, see our coverage of calls for independent testing of powerful AI models and of the AI alignment problem as a business risk.