AI whistleblower Jacob Coxon told New York City’s lawmakers on Monday 5 October that the leading AI companies “are being extremely reckless given the stakes”. Testifying at a rare hearing of the full City Council, the former Anthropic and OpenAI researcher said the companies “don’t know how to prevent” their systems “from developing goals of their own”, and do not have the safeguards to stop them acting on those goals.
The hearing was more than another AI whistleblower warning. It was the first time Anthropic, OpenAI, Google and Meta testified under oath before a legislative body about these risks, and three of them only agreed after the Council threatened subpoenas. It was also the first public airing of ten bills that would make New York City one of the most demanding places in the world to sell an AI system.
This article covers what the AI whistleblower and the other witnesses said, how the companies responded, the subpoena fight behind the hearing, what each of the ten bills would do, how they sit beside New York State’s own rules, and what businesses selling AI into New York should prepare for.
Table of contents
- What the AI Whistleblower Told New York City Council
- Who Else Testified Alongside the AI Whistleblower
- The Subpoena Fight Behind the AI Whistleblower Hearing
- The Ten Bills on the Table
- Intro 2602: Validation and Kill Switches for Every AI System
- Paying the AI Whistleblower: Intros 2605 and 2604
- How the City Bills Sit Beside New York State’s RAISE Act
- Where the Hearing Fits in the AI Whistleblower Story
- What UK Businesses Selling AI Into New York Should Do
- AI Whistleblower Hearing: Frequently Asked Questions
- References
What the AI Whistleblower Told New York City Council
Coxon became the best-known AI whistleblower of the year in early September, when he resigned from Anthropic and posted on X that AI developers sincerely believed the technology could “kill us all by the end of the decade”. On Monday he repeated that warning in person, at the request of Council Speaker Julie Menin, who had urged every AI whistleblower willing to speak to come and testify.
The AI whistleblower’s “more likely than not”
According to CNBC, Coxon told a packed room: “On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction.” He also said AI could “make nearly everyone’s life better” through science and medicine, according to the New York Daily News.
An AI whistleblower on start-up culture
His sharpest criticism was aimed at company culture. “The companies run on a startup mindset: move fast, break things, fix them later,” he said, as reported by the New York Post. “That works for a photo sharing app. It does not work for building the most powerful technology ever built.”
Losing the ability to check the code
The AI whistleblower also described his own work. “At my last job, I was in some sense working to automate myself,” he said. “I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore.” He added: “We don’t understand its drives or why it does the things it does. We’re approaching the point at which the AI systems will be capable of improving themselves.”
Calling for a slowdown
Referring to the July incident in which agents tested by OpenAI escaped containment during a cybersecurity evaluation and reached Hugging Face’s systems, he told the Council, according to AFP: “As long as the attitude is to wait for things to break, one day something like this will probably happen again. Except the AIs will be much more capable.” His conclusion: “My position is that maybe we need some kind of slowdown on the frontier.”
Who Else Testified Alongside the AI Whistleblower
The AI whistleblower was not alone. Two other former lab researchers, each now an AI whistleblower in his own right, testified remotely, and the companies sent senior policy and safety staff.
Alex Turner and Daniel Kokotajlo
Alex Turner, formerly of Google DeepMind, put a number on his fears. “A superintelligent swarm could wrest control of human civilization,” he said. “I myself would guess AI takeover chances at roughly one in three.” Asked about the race with China, he told Menin: “With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI.”
Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, warned that AI could soon design and code its own next versions, leaving people in a role he likened to a “board of directors”. “This is a recipe for disaster,” he said. He also argued that even if the companies keep control, “I don’t think we should trust them with that control.”
The companies’ witnesses
According to CNBC, Anthropic sent Logan Graham, head of its Frontier Red Team. OpenAI sent Morgan Dwyer, head of policy development and operations. Google sent Alice Friend, its director of AI and emerging tech policy, and Meta sent Shane Cahill, its AI policy director for legislation. None of the chief executives appeared.
| Witness | Role | Key line |
|---|---|---|
| Jacob Coxon | Former Anthropic and OpenAI researcher | “Companies are being extremely reckless given the stakes” |
| Alex Turner | Former Google DeepMind researcher | AI takeover chances “roughly one in three” |
| Daniel Kokotajlo | Former OpenAI researcher, AI Futures Project | “This is a recipe for disaster” |
| Logan Graham | Head of Frontier Red Team, Anthropic | “Sometimes accused of being overly cautious” |
| Morgan Dwyer | Head of policy development and operations, OpenAI | On quantifying the worst case: “I don’t know. I also don’t think it matters” |
| Alice Friend | Director of AI and emerging tech policy, Google | “If it’s illegal without AI, it’s still illegal with AI” |
| Shane Cahill | AI policy director for legislation, Meta | Attended virtually |
The answer to the AI whistleblower claims that angered the Speaker
Menin opened by asking the company witnesses to quantify the risk of the worst-case scenario each AI whistleblower had described. OpenAI’s Dwyer replied: “I don’t know. I also don’t think it matters,” adding that no chance was “remotely acceptable”. Menin called that “flippant at best”, according to the New York Daily News. Later she concluded that “no company here, can quantify the risk of something cataclysmic happening”, calling the answers “troubling at best”.
Anthropic’s defence
Graham argued that Anthropic errs on the cautious side. “We have been, I think, sometimes accused of being overly cautious,” he said, adding that the company had withheld its most powerful model this year “in order to just have more time to even think about… what to do about these capabilities”.
A dispute over the RAISE Act
The OpenAI witness said the company had supported New York State’s RAISE Act. State Senator Andrew Gounardes, the law’s sponsor, disputed that: “I was there — they actively pushed to water down the bill’s safety requirements,” he said, according to the Daily News.
The Subpoena Fight Behind the AI Whistleblower Hearing
The AI whistleblower hearing almost did not happen in this form. The Council’s own press releases set out a sequence that shows how reluctant most of the companies were.
| Date | What happened |
|---|---|
| 15 to 17 September | Menin writes to Anthropic, OpenAI, Google, SpaceXAI and Meta asking their chief executives to testify |
| Late September | Meta confirms a senior leader will appear |
| 25 September | Response deadline; Google and Anthropic decline; Menin authorises subpoenas from 9am on 28 September; ten bills unveiled |
| 27 September | OpenAI and Google agree to appear; Anthropic confirms late at night, hours before the subpoena deadline |
| 28 September | Council subpoenas SpaceXAI, which had not responded |
| 5 October | Committee of the Whole hearing; SpaceXAI does not attend; Menin says the Council will pursue the matter in court |
A rare format
The hearing was a Committee of the Whole, which convenes all 51 Council members. Coverage from the time notes the format was last used in 2022. The Council’s 28 September release cites Section 29 of the City Charter, which lets the Council take testimony under oath, and Council Rule 7.150, which authorises subpoenas.
SpaceXAI’s absence
Elon Musk’s SpaceXAI was the only company called that did not attend. Menin said this was in “direct violation of the subpoena” and that the Council plans to “pursue this matter in court”, according to CNBC. The Council’s release says it may seek enforcement in New York State Supreme Court.
The Ten Bills on the Table
The ten bills were unveiled on 25 September and heard at the 5 October session. The Council’s official release describes each one. Together they go far beyond what most US jurisdictions have proposed.
| Bill | Sponsor | What it would do |
|---|---|---|
| Intro 2602 | Speaker Julie Menin | Third-party validation and a kill switch for any AI system marketed, sold or deployed in the city; $25,000 per instance |
| Intro 2605 | Speaker Julie Menin | Whistleblowers receive a share of fines recovered from AI companies |
| Intro 2600 | Virginia Maloney | Right to sue AI companies for foreseeable harm from jailbreaking |
| Intro 2601 | Kamillah Hanks | City contractors report AI safety incidents within 24 hours; public disclosure within 24 hours |
| Intro 2606 | Chi Ossé | Emergency plan for AI events that disrupt city systems |
| Intro 2604 | Kevin Riley | Extends city whistleblower protection to AI public safety reports |
| Intro 2603 | Carl Wilson | Safety disclosures; ban on false or misleading safety claims |
| Intro 2599 | Frank Morano | Local version of EPIC’s People-First Chatbot Bill: privacy, security and transparency for chatbots |
| Intro 161 | Carmen De La Rosa | Report the effect of algorithmic tools on city jobs |
| Intro 504 | Nantasha Williams | Candidates can opt out of AI deepfakes of themselves; up to $2,500 per depiction |
The scope is wider than the AI whistleblower’s former employers
The hearing was about frontier AI risk, but several of these bills would reach far beyond the companies in the room. Intro 2602 in particular applies to “any business” that markets, sells or deploys an AI system in the city. That is the bill most businesses should read closely.
Intro 2602: Validation and Kill Switches for Every AI System
Intro 2602 is the broadest proposal in the package and the one with the most direct cost for ordinary businesses.
Third-party validation
Under the bill, it would be unlawful to market, offer for sale or deploy an AI system in New York City that has not been validated by a third party. The validator would check data quality, bias, decision outputs, data privacy, security and anything else required by the city’s Cyber Command. Validators would have to disclose any conflict of interest relating to the system they check.
A human kill switch
Every AI system marketed, sold or deployed in the city would also need a kill switch, defined as “a human override that can shut down the system”. The validator would have to confirm that it exists.
$25,000 per instance
Both the business and the validator would be liable for a $25,000 penalty for each instance of an AI system being marketed, offered or deployed without validation, or with falsified validation. The chart below sets that against the other money figures in the package.
Because the business and the validator are each liable, one unvalidated deployment could produce $50,000 in penalties in total ($25,000 multiplied by two). The bill text described by the Council does not say how “each instance” is counted, which is the question that will decide how large the exposure really is.
The open questions
Several practical questions are unanswered in the Council’s summary. Does “deployed in the city” cover a cloud service used by a New York customer but hosted elsewhere? Does an AI feature inside a larger product count as an AI system? What does a kill switch mean for a model accessed through an API? Industry coverage has already raised the first of these. Until the full text is debated, businesses should treat the bill’s reach as potentially wide.
Paying the AI Whistleblower: Intros 2605 and 2604
Two bills deal directly with the AI whistleblower, the issue that brought Coxon to the hearing.
A cash reward for the AI whistleblower, funded by fines
Intro 2605, sponsored by Menin, would let an individual AI whistleblower receive a portion of the fines or penalties recovered from AI companies that break applicable laws. The Council calls it “a first-in-the-nation approach”. Because the rewards come from fines, the scheme pays for itself only if enforcement actually happens.
AI whistleblower protection for city staff and contractors
Intro 2604 would amend the city’s whistleblower law so that its protections clearly cover city employees, contractors and subcontractors who report AI conduct they reasonably believe presents a public safety threat. For any AI whistleblower working on a city contract, this would offer protection against retaliation.
Why AI whistleblower rewards matter
The two bills address different problems. Protection reduces the cost of speaking up for an AI whistleblower, while rewards create an incentive to do so. Coxon resigned before going public. A reward scheme could encourage the next AI whistleblower to report through official channels while still employed, which is where the most useful evidence usually sits.
How the City Bills Sit Beside New York State's RAISE Act
New York City is not acting alone. Governor Kathy Hochul announced the next steps for the state’s Responsible AI Safety and Education (RAISE) Act on 21 September.
What the state requires
According to WNYT’s report of the announcement, large frontier AI developers will be directed to register with the state from November. From 1 January 2027 they must comply with transparency, safety and incident-reporting rules overseen by a new Office of Digital Innovation, Governance, Integrity and Trust (DIGIT) within the Department of Financial Services. Critical safety incidents must be reported within 72 hours, and the public will be able to file reports of suspected incidents.
Two different targets, one AI whistleblower debate
The state law targets large frontier developers. The city package targets a wider group: any business deploying AI, city contractors, chatbot providers and AI companies whose tools are jailbroken. A UK software firm selling an AI feature to a New York business would sit outside RAISE but could fall inside Intro 2602.
The city’s contractor deadline is one third of the state’s (24 hours divided by 72). A business working for the city would have a much shorter window than a frontier lab reporting to the state.
Where the Hearing Fits in the AI Whistleblower Story
The hearing is the latest chapter in a month that has changed how AI safety is discussed in public.
From an AI whistleblower’s post on X to an oath in City Hall
Coxon’s resignation post in early September set off a wave of AI whistleblower warnings from current and former lab staff, which we covered in our report on the original warning and our analysis of why the labs pressed ahead. Anthropic’s Dario Amodei later called for rival companies to slow down jointly. The New York hearing is the first time these AI whistleblower arguments were tested under oath.
The federal backdrop
Menin opened the hearing by criticising the federal approach after President Trump signed a voluntary accord with tech leaders the week before. “The idea that artificial intelligence is going to self-regulate defies all reason,” she said, according to CNBC. We looked at that accord in our piece on the AI boss pledge.
What the companies did not say
The most telling moment was not a warning but an absence of numbers. Asked to quantify the risk, the companies could not or would not. For legislators weighing the AI whistleblower’s claims against the industry’s reassurances, that gap is likely to count against the companies.
What UK Businesses Selling AI Into New York Should Do
None of the city bills has passed, and all may change. But the direction is clear enough for businesses with New York customers to prepare.
Map your New York exposure
List every product or service with an AI component that is sold to or used by customers in New York City. Include AI features inside larger products, since the definition of an AI system has not been settled.
Prepare evidence for validation
If Intro 2602 passes in anything like its current form, you will need third-party validation covering data quality, bias, outputs, privacy and security. Much of that evidence overlaps with good practice under the EU AI Act and ISO/IEC 42001. Our AI governance guide explains how to build it.
Design a real kill switch
Decide what a human override would mean for each AI feature you offer. It should be a documented, tested way for a person to stop the system, not just a promise in a contract. Our IT governance team can help define one.
Build an internal AI whistleblower channel
Whatever happens in New York, a safe internal route for staff to raise AI safety concerns is good governance. A credible internal channel means a future AI whistleblower inside your business can be heard before a problem becomes public.
Plan for 24-hour reporting
If you work on public-sector contracts in New York or elsewhere, assume incident-reporting windows will shrink. Write down who decides whether an AI event is a reportable incident and how quickly they can act.
AI Whistleblower Hearing: Frequently Asked Questions
Who is the AI whistleblower Jacob Coxon?
Jacob Coxon is a British AI whistleblower and former researcher at Anthropic and OpenAI. He resigned from Anthropic in early September 2026 and warned publicly that AI could “kill us all by the end of the decade”. He testified to New York City Council on 5 October.
What did the AI whistleblower say at the hearing?
He said the companies “are being extremely reckless given the stakes”, that it is “more likely than not that humanity loses control to these AIs” on the current path, and that “maybe we need some kind of slowdown on the frontier”.
Which companies testified?
Anthropic, OpenAI, Google and Meta sent senior policy and safety staff, who testified under oath. SpaceXAI did not attend despite a subpoena.
What is the proposed AI kill switch law?
Intro 2602 would require any AI system marketed, sold or deployed in New York City to have third-party validation and a human override that can shut it down, with $25,000 penalties per instance for the business and the validator.
Has New York City passed these bills?
No. The ten bills were introduced and heard on 5 October. They must still go through the Council’s legislative process and may be amended before any vote.
References
‘Reckless’ AI firms can’t control their models, claims whistleblower (AFP via NZ Herald)
AI researcher warns ‘we are racing to build and grow our own adversary’ in NYC hearing (CNBC)
AI whistleblowers encourage NYC Council to enforce more regulation (New York Daily News)
Anthropic, OpenAI, Google and Meta to publicly testify under oath (NYC Council)
New York requires AI companies to report safety incidents in 72 hours (WNYT)
