Fix the server error on robots.txt
Why it matters
While robots.txt answers with a 5xx error, Google stops crawling the site for about 12 hours, then relies on its last good copy for up to 30 days. The cause is usually a failing backend, a script that builds the file, or a firewall.
The fix
- Request https://example.com/robots.txt and note the status. Read the server or application error log around the time of the request.
- Fix the cause. Common ones are a crashed or overloaded backend, a plugin or script that generates the file and fails, wrong file permissions, and a firewall or rate limiter that answers 5xx.
- Serve robots.txt as a plain static file if you can; it is the most reliable. Return 200 with the rules, or 404 if you want no rules at all.
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the syntax of robots.txt
- Shrink the robots.txt file
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.