Topic
robots.txt directives, managed blocklists and content signals used to govern AI crawler and AI search access to websites.
Cloudflare's September 15 split of AI bot controls into Search, Agent and Training can block Googlebot when a site blocks Training. Site owners who depend on organic search should check their settings, a dev.to analysis says.
Reality
Cloudflare's Disallow AI Training control lets Googlebot and Applebot keep indexing a site while a directive Cloudflare writes into robots.txt tells them not to train on it. Bingbot is expected to honor it in early 2027.
Reality
Sony Music and Warner have sued Anthropic over pirated music and lyrics, joining Universal. For publishers the live question is which crawlers still get the full text, and what evidence would show a training promise held.
Reality
Bot Preference Sync writes Cloudflare's AI bot settings into your robots.txt from the Free tier up. The file becomes a rendering of dashboard state, and Cloudflare's bot database decides what it says.
Reality
The managed robots.txt file lists eight crawlers. Three of them are the training half of a pair whose search half is left allowed, and Perplexity is not in the file at all.
Reality