robots.txt and sitemap
What it can report
- · info 404 (info: no restrictions is a legitimate state) How to fix it
- ! low 5xx (low; crawlers must assume a full disallow) How to fix it
- ! low an html page or garbage with 200, rules before any User-agent, a misspelt directive, a redirect chain that never ends (low) How to fix it
- ! low over 500 KiB, which Google truncates (low) How to fix it
- ! low Googlebot and Bingbot are both disallowed from "/" (low: right for staging, wrong for a public site) How to fix it
- ! low no Sitemap line and no /sitemap.xml (low) How to fix it
- · info /sitemap.xml exists but robots.txt does not name it (info) How to fix it
- ! low a sitemap fetch failed or answered non-2xx (low) How to fix it
- ! low not parseable as a sitemap (low) How to fix it
- ! low valid but lists nothing (low) How to fix it
- ! low more than 50,000 URLs or child sitemaps in one file (low) How to fix it
- ! low URLs on another registrable domain (low) How to fix it
How it counts
The score is computed by fixed rules from the checks that ran, and this check has a weight of 1 within Hygiene. A check that errors or is skipped is shown as "could not check" and left out of the score, never counted as a pass. Fixes and the AI summary are written after scoring and never change it.
Related checks
Scan your own domain and see this check run.