From 15 September 2026, Cloudflare flips a default that affects anyone whose blog or content sits behind its network. Mixed-use AI crawlers — bots that blend search indexing, AI model training and AI-agent retrieval under a single user agent — will be blocked by default from any page that carries advertising. If you have never touched your crawler settings, this changes what gets through without you doing anything.
Who this hits, and who it does not
The new default applies to new Cloudflare customers, newly added sites, and all existing free-plan customers. Paid customers with an existing configuration keep whatever they already have set — this is a default for people who have not made a deliberate choice, not a forced change for everyone. If your site runs on Cloudflare's free tier, which describes a large share of indie SaaS blogs, you are in the group the new default applies to.
The distinction that matters is what counts as "mixed-use." A crawler that only does search indexing is unaffected and still welcome — Cloudflare is not blocking discovery. A fetch triggered directly by a user asking an AI agent to look at a specific page is treated separately and also generally allowed. What gets caught is the bot that combines all three purposes under one identity, because that combination makes it impossible for a site owner to consent to one use while declining another.
Why this is not just another robots.txt update
Robots.txt has always been advisory — a request that well-behaved crawlers choose to honour. This is enforced at Cloudflare's edge, ahead of the request reaching your server, for every site sitting behind Cloudflare's default configuration. That is a meaningfully stronger control than anything a text file has ever provided, and it is why the date is worth putting on your calendar rather than filing away as background noise.
It also arrives alongside a business model shift. Cloudflare is evolving its Pay Per Crawl experiment into Pay Per Use, where publishers are compensated when their content is actually used inside an AI-generated answer, rather than simply whenever a bot fetches the page. Ceramic.ai and You.com are the launch partners for the new marketplace, with other AI companies expected to negotiate their own arrangements.
The decision this forces, whether you notice it or not
If you have spent any effort trying to get cited by ChatGPT, Perplexity or Google's AI answers, this default works directly against that goal unless you override it. A mixed-use crawler blocked by default on an ad-supported page is a crawler that cannot read your content to cite it, full stop — the same block does not distinguish between "training on my content without permission" and "reading my content to answer a user's question about my product," because from the crawler's identity alone, Cloudflare cannot tell them apart.
That means the decision is genuinely yours to make deliberately, not one to leave to a default that was set for you. If AI citation is part of your growth strategy, you likely want to allow these crawlers and opt into whatever Pay Per Use arrangement becomes available once your traffic justifies it. If your priority is protecting original content from being used to train someone else's model without compensation, the new default is doing exactly what you want without any action on your part.
What to actually do before 15 September
Check your Cloudflare crawler and bot management settings now, particularly if you are on the free plan — that is the group whose default is changing. Decide deliberately whether you want AI crawlers reading your content, based on whether citation or content protection matters more to your specific business. Confirm whether your pages count as ad-supported under this policy, since that is the condition that triggers the default block. And if you are a heavier publisher, watch for the Pay Per Use marketplace expanding beyond its initial two partners — it is the mechanism that could eventually pay you for exactly the traffic this default is otherwise built to block.
The bottom line
A default that used to let ambiguous crawlers through now blocks them, starting 15 September, for free-tier and new Cloudflare sites. Whichever side of the citation-versus-protection trade-off your business sits on, make that choice yourself this week rather than letting an infrastructure default make it for you.