Decide your policy for AI crawlers on Cloudflare
Why it matters
This is a summary of which AI crawlers your robots.txt allows or blocks, not a fault. Whether to allow them is your choice, because asking training crawlers to stay away is a request that well-behaved ones honour, while blocking AI search crawlers can reduce how often you are cited in AI answers.
The fix
- Cloudflare can publish a managed robots.txt that tells known AI crawlers to stay away (Cloudflare's docs call it managed robots.txt, found under the bots settings in Security). Turn it on if you want that default policy; it only expresses a preference.
- To enforce a decision, use Cloudflare's AI Crawl Control to allow or block each crawler. Edit your own robots.txt as shown in the generic tab for finer choices.
Related
- Reported by the AI crawler access rules check
Scan your domain to see whether this applies to you.