RMIT University

ai crawlers eating website traffic websites blocking ai a solid castle battlement wall

AI Is Eating Website Traffic, Websites Are Blocking AI: Reliable Information Is Getting Harder to Find

On 31 August 2026 Dana McKay and Damiano Spina of RMIT University argued in The Conversation that the web’s thirty-year deal between crawlers and websites has collapsed: AI crawlers take content to train models and write answers while sending almost no traffic back, reputable publishers respond by blocking them, and AI answers increasingly draw on the low-quality and AI-generated sites that do not block. This piece sets out their argument, the Pew, Wikipedia and Cloudflare data behind it, the three peer-reviewed studies that show the quality trap is real, Cloudflare’s 15 September default block, pay-per-use and licensing, and what readers and UK businesses should do now.

Read more
CHAT