Agent blocking has become the next hurdle for the personal AI assistants that launched this autumn. Meta’s Muse, Instinct, OpenAI’s Dots and others promise to book flights, reserve tables and buy groceries for you. In practice, TechCrunch’s Sarah Perez reported on Tuesday 6 October 2026, many of them are running into websites that refuse to let them finish the job.
Some of that agent blocking is deliberate. Amazon shut Muse out of its store in September, and Yelp says it bars non-human traffic unless an agent pays for its data. Some is accidental: Walmart is a Muse partner, yet shoppers report its human-verification button throwing their agents out. Users rarely know which kind they have hit.
This article maps who is blocking and why, explains the mechanics on the website side, including a Cloudflare default change on 15 September, and looks at the Personal Agent Protocol that Meta and partners announced the same day. It ends with a practical framework for site owners deciding whether to block, verify or welcome agents, and advice for people using them.
Table of contents
- Why Agent Blocking Became the Next Hurdle for AI Agents
- The Agent Blocking Map: Who Blocks and Who Welcomes
- How Agent Blocking Actually Happens on a Website
- Cloudflare’s 15 September Change and Accidental Agent Blocking
- Meta’s Answer to Agent Blocking: The Personal Agent Protocol
- Other Standards That Could End Blind Agent Blocking
- The Business Cost of Agent Blocking
- A Framework for Site Owners: Block, Verify or Welcome
- What Agent Blocking Means for People Using AI Agents
- Agent Blocking: Frequently Asked Questions
- References
Why Agent Blocking Became the Next Hurdle for AI Agents
Until this year most web automation was either a search crawler or a scraper. The new personal agents are different: each one acts for a single, identifiable customer who usually wants to buy something. That makes agent blocking a commercial decision, not just a security setting.
Personal agents now do the clicking
TechCrunch notes that these assistants get things done “all without having to be technical enough to set up OpenClaw or some other agent on your own computer.” Muse, Instinct and Dots open web pages, fill forms and complete checkouts on a user’s behalf, often in a browser the user never sees.
Amazon went first
On 21 September, Bloomberg reported that Amazon had begun blocking Muse from browsing or buying on its retail site. We covered the dispute, and the legal background from Amazon’s earlier case against Perplexity, in our report on Amazon barring Meta’s Muse. Since then, complaints about agent blocking have spread to many more brands.
Users cannot tell deliberate from accidental
The person behind the agent sees only a failed task. TechCrunch notes they cannot tell whether the block was intentional, which leaves them frustrated with both the retailer and the agent. That confusion about agent blocking damages trust on both sides, even when nobody meant to turn the customer away.
The Agent Blocking Map: Who Blocks and Who Welcomes
TechCrunch gathered statements on agent blocking from several companies named in user complaints. The table summarises what each said. It reflects their public positions on or around 6 October, not a technical test of each site.
| Company | What users reported | What the company said | Type of block |
|---|---|---|---|
| Amazon | Muse cannot browse or buy | Began blocking Muse (Bloomberg, 21 Sept) | Deliberate |
| Walmart | Muse fails at checkout | Not intentional; Walmart is a Muse partner | Accidental |
| Yelp | Bookings and quotes fail | No non-human traffic unless the agent pays for data | Deliberate, paid access |
| eBay | Blocks and some account suspensions | Restricts unauthorised agents, scraping and training | Conditional |
| Delta | Flight bookings rejected | Protects against unauthorised automation; evaluating agents | Policy, no partnership |
| United | Flight bookings rejected | Pointed to terms banning robots without permission | Terms of use |
| Instinct removed while cleaning Gmail | Not stated in the report | Unclear | |
| Adidas, Zillow, Pizza Hut | Agents blocked | No response or declined to comment | Unknown |
Deliberate blocks: Amazon and Yelp
Yelp told TechCrunch it does not permit non-human traffic unless the agent has paid for access through its data licensing programme. TechCrunch concludes that an agent trying to get a quote, join a waitlist or book a table will likely fail without a formal partnership. That is agent blocking as a business model: access is a product, and agents are expected to buy it.
Accidental blocks: Walmart’s human check
Walmart’s case of agent blocking is the most telling. A spokesperson said the failures were not intentional, and the company told TechCrunch it wants to be where its customers are. The problem appears to be a button visitors must press to prove they are human. If the agent interrupts that check, it fails, and the agent is booted out.
Policy by terms: the airlines
Delta’s Heena Chavda said the airline has no partnership enabling third-party agents to book on its platforms but is “continuing to evaluate” how external agents could help. United pointed to terms that bar “any robot, spider, other automatic device” without written permission. For travel, agent blocking is currently the default.
Conditional access: eBay
eBay told TechCrunch it does not ban all third-party shopping agents. Its policies restrict unauthorised agents and actions such as automated scraping and model training. Its user agreement, updated with effect from 20 February 2026, names “buy-for-me agents, LLM-driven bots, or any end-to-end flow that attempts to place orders without human review” among the restricted tools.
Counting the positions
The chart counts the eight entries in the table above by type. It is a tally of public positions, not a measure of how often agents fail.
Each bar is the count divided by the eight table rows: 2 divided by 8 is 25% and 1 divided by 8 is 12.5%. The Adidas, Zillow and Pizza Hut row counts once.
How Agent Blocking Actually Happens on a Website
Few sites have written a rule that says “no AI agents“. Most agent blocking happens through cybersecurity tools built for older problems: spam, credential stuffing, ticket scalping and scraping. Knowing which layer is responsible for agent blocking is the first step to fixing it.
Human-verification buttons and challenges
Press-and-hold buttons, puzzles and invisible challenges score whether a visitor behaves like a person. An agent driving a browser can trip them in ways a human never would, for example by moving too quickly or pausing mid-check. TechCrunch calls this “an indication that the current technology we use to verify humans” may not fit the agent-first web.
Bot-management scores and rate limits
Content delivery networks and web application firewalls assign each request a bot score from signals such as browser fingerprint, network origin and behaviour. Cloud-hosted agents often come from data-centre addresses that score badly. A rule meant to stop scrapers then becomes agent blocking that catches a paying customer’s assistant.
Terms of use and account action
Some agent blocking happens after the fact. A site’s terms forbid automation, and the account that used an agent is flagged or suspended. TechCrunch reports that a couple of people said eBay suspended their accounts over agent use, which puts the risk on the customer rather than the agent maker.
Sign-in and session checks
Agents that work inside a logged-in account, such as an email inbox, face extra checks for unusual activity. One user complained that Google kicked out Instinct while it was cleaning their Gmail inbox. These checks protect against account takeover, which is exactly what an unfamiliar agent can look like.
Cloudflare's 15 September Change and Accidental Agent Blocking
TechCrunch reports that some people suspect content delivery networks are disrupting Muse traffic, pointing to a Cloudflare change that took effect on 15 September. Cloudflare’s own blog post sets out what changed, and it explains how a site could end up blocking agents without anyone choosing to.
Search, Training and Agent controls
Cloudflare now classifies bots by behaviour into three controls: Search, for building a search index; Training, for training or fine-tuning models; and Agent, for “user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents”. Each can be set to allow, block, or block only on pages that serve ads.
What migrated automatically
Sites that had used the older “Block AI” setting were migrated to the new controls. Under the migration table in Cloudflare’s post, a legacy “Block” became Allow for Search, Disallow AI Training for Training, and “Block on pages with ads” for Agent. In other words, an old anti-scraping choice can now turn away personal agents on every page with an advert.
Ads are the dividing line
Cloudflare’s explanation is economic. “Ad revenue depends on a human actually seeing the page,” it says, adding that “agents fetch the page with nobody there to see the ads.” New domains that earn from ads are offered a preset that blocks agents on ad-carrying pages, while sites without ads are offered Allow.
The numbers Cloudflare did publish
Cloudflare told TechCrunch it had no specific data on blocking of personal agents and pointed to its public Radar hub. Its 15 September post does give two related figures: under 1% of Cloudflare sites block search bots, while 17% enable some way of blocking AI training.
Bars are drawn on a 0 to 100% scale, so the training bar is 17% wide and the search bar is shown at its 1% ceiling. Cloudflare published no equivalent figure for agents.
Cloudflare’s wider stake
Cloudflare has its own interests here. It runs a marketplace where AI bots pay for the data they access, and in August it began testing Kitesurf, a lightweight browser built for AI agents. We looked at its chief executive’s agenda in whether Matthew Prince can save the web from AI.
Meta's Answer to Agent Blocking: The Personal Agent Protocol
On 6 October, Meta published “A new way for businesses and personal agents to work together”. It describes the Personal Agent Protocol, “an open standard” that Meta (Muse and Meta Business Agents) and Sierra are developing with partners at Genesys, NiCE, Decagon, Rocket, Shopify, Stripe and Walmart. TechCrunch’s list omits Shopify; Meta’s includes it.
The concerns Meta says it heard
Meta acknowledges that some businesses have responded to personal agents by blocking them, and asks directly: “What about load spikes, abuse, liability?” Its answer is not that the concerns are wrong, but that agent blocking throws away the customer. “Turning away a personal agent means turning away the customer behind it,” Meta writes.
Muse’s own behaviour rules
Meta describes two tests Muse applies before acting. The “one honest person” test asks whether “a reasonable, honest person doing this by hand” would do it this way. The “if every agent did this” test asks whether the system would cope if every Muse user made the same request. For consequential actions, Meta says, Muse “is designed to continue only with explicit user approval.”
Three stages of cooperation
The protocol is meant to evolve in stages. First, businesses get predictable ways to guide agents to preferred paths through their site. Next, connectors give agents direct access instead of driving a browser. Finally, business agents and personal agents talk to each other directly. Meta also keeps a growing list of connector partners inside the Muse app.
What is still missing
Meta’s post sets out principles rather than a wire format. It does not, as published, specify how an agent proves its identity to a site. Without that, a website cannot tell a well-behaved Muse request from a scraper pretending to be one, and blanket agent blocking will remain the safer default for many security teams.
Other Standards That Could End Blind Agent Blocking
Identity is the missing piece in agent blocking, and several efforts already target it. None is universal yet, but together they show how agent blocking could become agent verification.
Web Bot Auth and signed requests
Web Bot Auth proposals let a bot or agent sign its HTTP requests using HTTP Message Signatures, the IETF’s RFC 9421. A site can check the signature against the operator’s published keys and know who sent the request. That turns “is this a bot?” into “which bot is this, and do I trust it?”
Payment networks’ agent protocols
Visa and Cloudflare announced the Trusted Agent Protocol in October 2025. It builds on Web Bot Auth and adds a tag saying whether an agent is browsing or paying, plus a nonce against replay attacks. Mastercard has a parallel Agent Pay programme. These let merchants trust an agent at checkout without trusting every bot.
Paid access and licensing
The third route is commercial. Cloudflare’s pay-per-crawl marketplace and Yelp’s data licensing turn access into a paid product. For sites whose value is the data itself, this may be the right answer, but it raises costs for agent makers and, eventually, for their users.
| Approach | Who is behind it | What it solves | What it leaves open |
|---|---|---|---|
| Personal Agent Protocol | Meta, Sierra and partners | How agents and businesses cooperate | Agent identity, as published |
| Web Bot Auth | IETF proposals, Cloudflare | Cryptographic proof of who sent a request | Adoption by agent makers |
| Trusted Agent Protocol | Visa, Cloudflare | Trusting an agent at checkout | Non-payment journeys |
| Agent setting in bot management | Cloudflare and other CDNs | A site-level allow or block choice | No standard agent directive yet |
| Paid data access | Yelp, Cloudflare marketplace | Compensation for data | Cost for agents and users |
The Business Cost of Agent Blocking
For a retailer, a hotel or a restaurant group, agent blocking is no longer a pure security question. It is a sales question with a security side.
Turning away the customer behind the agent
An agent that fails on your site usually tries the next one. If a competitor’s checkout works and yours does not, the agent will learn the difference faster than any shopper would. Walmart’s response, partnering and fixing accidental blocks, reflects that risk.
The risks that justify caution
The reasons for agent blocking are real. Agents can generate load spikes, scrape prices and inventory, and act on accounts in ways that look like fraud. Liability is unclear when an agent buys the wrong thing. Our coverage of the five ways rogue agents are messing with the internet shows these are not hypothetical.
The advertising problem
For publishers and content sites, the maths is different. An agent that reads a page and leaves sees no adverts. Cloudflare’s presets build that into the default, and many ad-funded sites will choose agent blocking on purpose, at least until agents arrive with a way to pay.
A Framework for Site Owners: Block, Verify or Welcome
The worst outcome is accidental agent blocking that nobody chose. The steps below help UK and European businesses make a deliberate decision. Our cybersecurity team helps clients tune bot rules without locking out real customers.
Map your agent traffic first
Check your CDN or firewall logs for requests labelled as agents, browser-automation signatures and data-centre traffic on checkout and booking pages. If you use Cloudflare, review the Agent setting that the 15 September migration may have applied. Many site owners will find a choice was made for them.
Decide per journey, not per site
Browsing a catalogue, buying, and acting inside an account carry different risks. Treat them separately. A blanket block on everything is simple, but it gives up the low-risk journeys where agents bring buyers.
Replace fragile human checks on key flows
If your checkout relies on a press-and-hold button or puzzle, consider risk-based checks that look at payment and account signals instead. For known agents, verify identity through signed requests where available, rather than asking an agent to pretend to be a person.
Publish your policy
Set out your agent blocking policy in your terms, your robots.txt and a short page for agent makers. Clear rules turn surprise failures into reportable bugs, the outcome TechCrunch credits Meta’s partnership approach with achieving.
Offer a connector where it pays
For high-volume journeys, an API or connector is cheaper to run and easier to control than a browser-driving agent. It also gives you rate limits, audit logs and a contract with the agent maker.
| Journey | Main risk | Suggested stance |
|---|---|---|
| Browse product or menu pages | Scraping, load | Welcome with rate limits |
| Check availability or prices | Competitor harvesting | Verify identity, or offer a connector |
| Checkout or booking | Fraud, wrong orders | Verify, and require user approval |
| Act inside a user account | Account takeover | Allow only verified agents with consent |
| Bulk data or archives | Loss of data value | Block, or license access |
What Agent Blocking Means for People Using AI Agents
For users, the practical advice on agent blocking is about expectations and risk. Agents are useful, but the web has not agreed to let them in everywhere yet.
Why your agent failed
If a task fails at a particular site, assume agent blocking before assuming the agent is broken. Try the site’s own app or a partnered connector. In Muse, check the connector list, which TechCrunch says continues to grow.
Your account carries the risk
If a site’s terms ban automation, the penalty falls on your account, not on the agent maker. The reported eBay suspensions show this can happen. Read the terms of any site where a lost account would hurt, and keep agents away from it until the site says yes.
Prefer approved routes
Partnered connectors and sites that publish an agent policy are more reliable and less risky. Our round-up of AI agents you can text notes which assistants lean on partnerships rather than raw browsing.
Agent Blocking: Frequently Asked Questions
Why are websites blocking AI agents?
For several reasons: to stop scraping and fraud, to protect advertising revenue, to control access to valuable data, or by accident, when anti-bot tools built for spam catch a legitimate agent.
Is Walmart blocking Meta’s Muse?
Walmart told TechCrunch the failures were not intentional and that it is a Muse partner. The problem appears to be a human-verification button that agents can fail.
Did Cloudflare start blocking AI agents?
Not across the board. From 15 September, sites that had used its legacy “Block AI” setting were migrated to block agents on pages that carry ads, and new ad-funded domains are offered the same preset.
What is the Personal Agent Protocol?
It is an open standard announced by Meta on 6 October 2026, developed with Sierra, Genesys, NiCE, Decagon, Rocket, Shopify, Stripe and Walmart, to define how personal agents interact with businesses.
Can I get banned for using an AI agent?
Yes, if a site’s terms forbid automation. Users reported eBay account suspensions linked to agent use, so check the terms of any site that matters to you.
References
The next hurdle for AI agents: getting websites to let them in (TechCrunch)
A new way for businesses and personal agents to work together (Meta for Business)
Have it both ways: stay discoverable in search while disallowing AI training (Cloudflare)
Amazon blocks Meta’s Muse AI agent from its retail site (Bloomberg)
Big Tech’s AI agents are fighting over your shopping cart (NBC News)
Yelp data licensing (Yelp for Business)
eBay explicitly bans AI buy-for-me agents in user agreement update (Value Added Resource)
Securing agentic commerce: helping AI agents transact with Visa and Mastercard (Cloudflare)
More AI coverage: explore Progressive Robot's AI Models, Tools & Releases hub — hands-on reviews, setup guides and benchmarks in one place.
