An AI scraping opt-out is finally on the table that works across more than one company. OpenAI, Google, Microsoft and Anthropic said on Friday 30 October 2026 that they will honour one shared line in a website’s robots.txt file, so a publisher can refuse generative AI training in a single step instead of writing a separate rule for every company’s bot.
The word “some” in the headline is doing real work. This AI scraping opt-out covers training only, it starts at different times at each company, and it rests on an individual Internet-Draft that the IETF itself says it has not endorsed. Most importantly, the specification says the signal is not a directive for crawling behaviour. A bot can still arrive at your site. What changes is the stated preference about what happens to your pages afterwards.
Table of contents
- What the AI Scraping Opt-Out Actually Promises
- How the AI Scraping Opt-Out Works in Robots.txt
- Why This AI Scraping Opt-Out Does Not Block Crawlers
- Who Starts When: The AI Scraping Opt-Out Timetable
- Google and Microsoft Read the Same AI Scraping Opt-Out Differently
- What the AI Scraping Opt-Out Leaves Unsolved
- What Publishers Should Do About the AI Scraping Opt-Out Now
- Frequently Asked Questions About the AI Scraping Opt-Out
- References
What the AI Scraping Opt-Out Actually Promises
Four companies, one line, one Friday
Three companies published posts on the same day, and the fourth made its commitment on a mailing list. Google’s John Mueller wrote on the Search Central Blog, Microsoft’s Krishna Madhavan wrote on the Bing Blog, and OpenAI published an unsigned company post. Kevin Kelley, a standards lead at Anthropic, submitted the draft and told the IETF’s AI Preferences working group that “Anthropic has already started implementing support for this draft.”
The pitch is the same in each case. OpenAI says operators previously had to “identify individual crawlers and maintain separate rules for each and every provider.” Google says site owners will stop “managing separate instructions for every AI developer.” That is the whole of the AI scraping opt-out in plain terms: fewer rules to maintain, with the same answer going to every company that agrees to read it.
Why the announcement landed now
AFP framed the news against a surge of bots and a growing pile of litigation. The New York Times sued OpenAI and Microsoft in December 2023, and we have since covered the latest allegations in that case and USA Today’s own suit against OpenAI. OpenAI and Microsoft invoke fair use in those cases.
Regulation is pushing too. Google says the AI scraping opt-out is consistent with the EU’s Code of Practice for general-purpose AI model providers, which it says “recognizes robots.txt as a primary method for web publishers to express machine-readable preferences.” It also says the proposal reflects feedback from publishers, civil society groups and regulators.
Who has not signed up
Meta communications head Andy Stone welcomed the idea on X, calling a common signal “an important step.” The Verge noted that he did not say whether Meta would honour it, which matters because Meta’s own terms for crawling business websites have been contested this month, and Meta delayed those terms after publisher pushback.
Amazon and Apple are not named in the commitments either. Smaller labs, open-source projects and data brokers have promised nothing. So this AI scraping opt-out reaches only the companies that volunteer for it, however large those four companies are.
How the AI Scraping Opt-Out Works in Robots.txt
A policy token, not a crawler name
Robots.txt normally addresses a named bot on its User-Agent line. Kelley’s draft, titled “Machine-Readable Training Preference Signal”, writes a made-up name there instead, which it calls a policy token. The draft’s own example looks like this, with each directive on its own line in the real file: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/.
According to the draft, that group “expresses a preference against AI training for all paths except those under /press-kit/.” Paths use ordinary robots.txt matching from RFC 9309, so the AI scraping opt-out can apply to a whole site or a single folder. The double underscore exists because some parsers match loosely, and the draft says “the __ prefix makes accidental matches unlikely.”
Three rules that catch people out
First, a wildcard does not count. The draft says groups for User-Agent: * “MUST NOT be interpreted as matching the policy token.” Kelley told the working group that a file not using this mechanism, “as is true of every one published today, expresses no preference at all.” If you block all bots today, you have not yet used this AI scraping opt-out.
Second, your existing vendor rules do not transfer. A Google-Extended block keeps its current meaning at Google, but the draft says consumers “MUST NOT treat it as expressing a preference in the context of this document.” Third, “Private agreements MAY override preferences expressed via this mechanism,” so a licensing deal can sit above a blanket refusal.
What “AI training” means here
The definition is borrowed from the working group’s vocabulary draft: “Using an asset to modify the learned parameters of a generative AI model.” Models used only for classification, ranking or scoring fall outside it. That narrows the AI scraping opt-out further than the word “scraping” suggests, because a ranking model trained on your pages is not covered.
OpenAI’s wording may narrow it again. Its post refers to “foundational generative AI training” and promises implementation details later. Whether “foundational” means something tighter than the shared definition is an open question until that update appears.
The people shaping this were not only AI vendors. The acknowledgements list 46 contributors, with affiliations “for identification only”. They include staff from the Financial Times, the BBC, News Corp and the European Publishers Council, plus the Electronic Frontier Foundation and Creative Commons.
Bars scale to the largest count of 4. Anthropic’s Kelley is the author and a further colleague is thanked separately, and being thanked does not imply endorsement.
Why This AI Scraping Opt-Out Does Not Block Crawlers
The sentence most coverage skips
Headlines say this blocks scraping, and The Verge reported that the four companies will restrict their crawlers from accessing a site when they meet the token. The draft says otherwise: “This policy token is not a directive for crawling behavior.” It describes content-usage preferences and traffic management as “independent and orthogonal.” On the primary text, this AI scraping opt-out is about use, not access.
Microsoft is equally blunt: “These settings govern use of content, not crawl access.” AFP’s report of the Microsoft post makes the same point, saying the setting does not prevent bots from crawling. If your problem is server load or bandwidth, nothing here helps, and you still need a rule for each crawler’s own name.
A small gap between draft and author
Kelley told the working group that “the draft does ask crawlers not to fetch what they can’t use.” The text is softer. A consumer that crawls only for covered purposes “MAY refrain from retrieving resources,” and under IETF conventions MAY marks an option rather than a request. PPC Land spotted this difference, and it is worth keeping in mind before you treat the AI scraping opt-out as a way to reduce traffic.
There is a good structural reason for the split. Googlebot and Bingbot gather pages that serve search and AI products at once, so a crawl block also costs you search visibility. A statement about use can sit alongside normal crawling. That is the trade this AI scraping opt-out makes: search stays intact, but the bots keep visiting.
| Control | Stops the fetch? | States a training preference? | Who honours it |
|---|---|---|---|
| Shared token in robots.txt | No | Yes | Participating providers, once live |
| Rule naming a crawler | Asks it not to | Not under this draft | That crawler, voluntarily |
| Google-Extended | No, it is a control and not a crawler | At Google only | |
| Cloudflare default bot blocking | Blocks at the network edge | Not by itself | Enforced on new customer sites since July 2025 |
| Really Simple Licensing | No | Yes, with licence terms | Those who adopt the standard |
Who Starts When: The AI Scraping Opt-Out Timetable
Four start dates, and only one is fixed
AFP’s summary is accurate but compresses a lot. Microsoft gives an exact date, 1 February 2027. Google’s blog says “in 2027”, while a note added to its crawler documentation says “early 2027”. OpenAI says “in the coming months”. Anthropic has started building, with Kelley cautioning that “It’ll take us some time to add support across our services.”
So a publisher adding the AI scraping opt-out today is writing a line that, on these statements, nobody has confirmed as live. That is harmless, because any compliant parser that does not implement the token simply ignores it, which is one of the draft’s stated goals. Adding it early costs nothing, but you should not assume it is protecting anything yet.
| Company | Stated timing | Existing controls | Where it said so |
|---|---|---|---|
| Anthropic | “already started implementing”, no live date | Kept alongside | IETF mailing list |
| OpenAI | “in the coming months” | Kept alongside | Company post |
| Microsoft | 1 February 2027, as it crawls | NoArchive and NoCache retired | Bing Blog |
| “in 2027”, docs say “early 2027” | Google-Extended kept alongside | Search Central Blog |
The dates that matter if you use NoArchive or NoCache
Microsoft is the only company with a transition plan, and it has teeth. Temporary grounding preferences open in Bing Webmaster Tools on 15 December 2026, for sites already using those tags. After 1 March 2027, “NoArchive and NoCache will no longer be read as a preference for training or grounding.” Any temporary preference still in place expires on 30 September 2027.
The arithmetic below shows the order of events: the transition window opens before the AI scraping opt-out starts at Microsoft, and this version of the draft lapses before the temporary preferences do, unless it is renewed or replaced.
Bars scale to the longest gap of 335 days. Days are counted from the published dates, and expiry applies to this version of the draft.
Standing: support arrived before the standard
The IETF’s datatracker states that this document “is not endorsed by the IETF and has no formal standing in the IETF standards process.” It is an individual submission, and Kelley calls it “input to the WG’s attachment work.” Four large consumers have therefore committed to a text that could still change. Google and Microsoft both acknowledge that the working group’s discussions continue, so this AI scraping opt-out may look different by the time a standard exists.
Google and Microsoft Read the Same AI Scraping Opt-Out Differently
Training is not grounding
Training changes what a future model knows. Grounding pulls a live page into an answer being written now, usually with a link. For most publishers grounding is the part that can send a reader back, so how each company treats it may matter more than the training rule. This is where the AI scraping opt-out stops being one policy and becomes two.
Google ties the two together
Google says that when a site uses the token and “Google-Extended is not explicitly specified”, it “plans to interpret that as if the publisher had disallowed Google-Extended.” Google’s documentation says Google-Extended covers Gemini training and also grounding in Gemini Apps and Vertex AI. Read together, the AI scraping opt-out at Google also appears to withdraw you from Gemini grounding.
Google’s post approaches the same point from the other side: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” You can override the shared token with an explicit Google-Extended allow. But then you have also allowed training at Google, because it is one control. PPC Land concludes there is no single Google setting that lets a site be cited without being trained on, and we read it the same way.
Microsoft separates them
Microsoft says the opposite design is possible: “you can set AI Training to Disallow and AI Grounding to Allow.” It adds a page-level grounding control with three values, and the new one, DisallowExtended, keeps you in Microsoft’s own products while excluding grounding it sells to others, such as Web IQ. The company also warns that restricting grounding “will significantly reduce your content’s visibility in AI-generated experiences.”
Note what none of this touches. AI Overviews and AI Mode in Google Search are outside Google-Extended, and PPC Land reports they sit behind a separate Search Console setting that followed the UK regulator’s June requirement. We looked at that wider shift in how Google’s AI search is upending publishing. The AI scraping opt-out does not reach it.
| Question | Microsoft | |
|---|---|---|
| Is the shared token honoured for training? | Honoured from 1 February 2027 | Planned for early 2027 |
| Does it also affect grounding? | No, grounding is separate | Yes, treated as a Google-Extended disallow |
| Page-level training control? | Yes, meta tag or header | Not described in the post |
| Train no, cite yes? | Supported | No single setting |
| Effect on classic search results? | None stated | None, Google-Extended is not a ranking signal |
What the AI Scraping Opt-Out Leaves Unsolved
No enforcement, and a poor record for polite signals
Nothing in the draft penalises a company that ignores the token, and OpenAI itself calls this a “voluntary effort.” TollBit’s State of the Bots report for the first half of 2026, as described by Search Engine Journal and PPC Land, found about 15% of identified AI page fetchers reached URLs that European sites had disallowed. These are real-time fetchers rather than training crawlers, and TollBit counts any request to a disallowed URL as a bypass. Still, it shows how an unenforced instruction behaves.
So treat the AI scraping opt-out as a commitment you can hold named companies to, not a technical barrier. It gives a publisher something to point at. It does not give you any way to verify that a page you refused was left out of a training set, because nothing in these announcements describes independent audit.
Everything already collected
The draft ties a preference to “the robots.txt file in effect when the asset was retrieved.” Microsoft says the settings cover “future AI generative model training”, and AFP reports the setting applies only to future training. Nobody here offers to remove your archive from an existing dataset or an existing model, so the AI scraping opt-out looks forward only. Years of scraped work stay where they are, and the courts are still arguing over that.
Default on, and no money
The burden stays with the publisher. Microsoft says both of its preferences default to Allow, and AFP reports scraping is allowed by default until a publisher changes the setting. Microsoft adds that the preferences “do not establish licensing or payment arrangements” and that “Neither choice guarantees inclusion, traffic, or payment.” The AI scraping opt-out gives you a way to say no, not a way to get paid.
Payment routes exist elsewhere. Really Simple Licensing declares machine-readable licence terms, Cloudflare has blocked AI bots by default on new customer sites since July 2025, and we profiled Cloudflare chief executive Matthew Prince’s campaign last month. Our earlier piece on how websites are responding to AI crawlers shows why many publishers want more than a refusal.
Lawsuits are untouched
Nothing about an AI scraping opt-out settles whether training on copyrighted work needed permission in the first place. OpenAI published a post titled “Fair use benefits everyone” on the same day. A publisher who sets the token today gives up no legal claim that we can see in these documents, but we are not lawyers, and anyone with a live dispute should ask theirs. The Wikipedia copyright filing we covered shows how unsettled the open web’s own position remains.
What Publishers Should Do About the AI Scraping Opt-Out Now
Decide what you actually want first
Separate three questions before touching the file. Do you want to refuse model training? Do you want to appear, with links, in AI answers? Do you want less bot traffic? The AI scraping opt-out answers only the first. A news publisher relying on citations may make a different call from a software vendor protecting paid documentation, and neither call is wrong.
For a business site, the case is often less clear-cut than it is for a newsroom. Marketing pages and product descriptions may benefit from being known to AI models, while gated guides and customer material may not. Path-level rules suit that split, and it belongs in the same conversation as your wider SEO services and AI visibility plan.
A practical order of work
- Audit your robots.txt. List which vendor tokens you already use, and whether a tool such as Cloudflare writes them for you. PPC Land reports HasData found a Cloudflare change removed AI opt-outs from 118 sites’ files.
- Keep vendor rules. Do not delete GPTBot, ClaudeBot or Google-Extended lines. OpenAI, Google and Anthropic all say their existing controls continue, and named rules are what govern fetching.
- Add the shared group deliberately. Use the AI scraping opt-out at the path level if only some content is sensitive, and allow folders such as a press kit.
- Decide on Gemini. Without an explicit Google-Extended allow, expect to lose Gemini grounding as well as training once Google goes live.
- Check Microsoft’s tags. If you use NoArchive or NoCache, diary 15 December 2026 and 1 March 2027.
- Log what visits. Server logs are the only evidence you will have of who actually respects the signal.
One trap for NoCache users
Microsoft says that “If we detect new AI Training or AI Grounding preferences on a site, we will disregard any NoArchive and NoCache preferences that may remain on that site.” Read literally, a site relying on NoCache to stay out of Copilot answers could lose that restriction by adding only the shared training line. PPC Land flagged this, and Microsoft’s post does not discuss the case directly. Publish the page-level grounding preference at the same time as your AI scraping opt-out.
Who should treat this as urgent
Few sites need to rush. Nothing changes at Microsoft before February, and the wording could still shift. The sites with a real deadline are those on NoArchive or NoCache that want to stay out of grounding. For everyone else, this is a good moment to write down your policy, so that the AI scraping opt-out becomes a one-line change when providers go live. If you are shaping that wider position, our AI models and tools hub tracks the vendors involved.
Frequently Asked Questions About the AI Scraping Opt-Out
What is the AI scraping opt-out in one sentence?
It is a robots.txt group naming the policy token __usage_ai-training, which tells participating AI companies you prefer that your content is not used to train generative models.
Does the AI scraping opt-out stop bots from visiting my site?
No. The draft says the token “is not a directive for crawling behavior.” To limit fetching, you still need rules for each crawler’s own name, or blocking at the network level.
Is the AI scraping opt-out live today?
We found no company stating that it is live. Anthropic says it has started implementing support, OpenAI says the coming months, Google says 2027 and Microsoft says 1 February 2027.
Will using it hurt my search rankings?
Microsoft says eligibility for traditional Bing results does not change, and Google says Google-Extended does not affect Google Search inclusion or ranking. Visibility in AI answers is a separate matter, and both companies warn it can fall.
Does the AI scraping opt-out remove content already scraped?
Nothing in these announcements says so. The draft ties preferences to the file in effect when a page was retrieved, and Microsoft refers to future training.
Does my existing Google-Extended or GPTBot rule count?
Those rules keep working for those companies. But the draft says a proprietary token does not express a preference under the shared AI scraping opt-out, so it will not carry over to anyone else.
Do Meta, Amazon and Apple have to follow it?
No. Meta has voiced support without saying it will honour the signal, and Amazon and Apple are not named in the commitments. OpenAI and Anthropic have both asked other providers to join.
References
Google Search Central Blog: Supporting a shared AI training control for websites
Microsoft Bing Blog: Your content, your choice
OpenAI: Making it easier for websites to express AI training preferences
IETF Datatracker: Machine-Readable Training Preference Signal
IETF ai-control mailing list: Kevin Kelley’s announcement
IETF: A Vocabulary For Expressing AI Usage Preferences
Google: Common crawlers documentation
The Verge: The biggest names in AI are backing a new way to opt out of scraping
AFP via Malay Mail: After scraping the Internet, AI giants pledge no-scraping tool for publishers
PPC Land: One robots.txt token lets sites refuse AI training
PPC Land: 15% of AI page fetchers in Europe reached disallowed URLs
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.
