Fix the server error on robots.txt on nginx

Why it matters

While robots.txt answers with a 5xx error, Google stops crawling the site for about 12 hours, then relies on its last good copy for up to 30 days. The cause is usually a failing backend, a script that builds the file, or a firewall.

The fix

1. Check the nginx error log (often /var/log/nginx/error.log) for the failing request, and check whether the upstream app answers.
2. Create /var/www/html/robots.txt first (the generic fix shows what to put in it). Then add an exact-match location that serves it regardless of the app, run nginx -t and reload nginx. Without the file this location answers 404 instead of the app's answer.

location = /robots.txt {
    root /var/www/html;
    default_type text/plain;
}

Read the documentation ↗(opens in a new tab)

Related

• Reported by the robots.txt and sitemap check
• Add a robots.txt file
• Fix the syntax of robots.txt
• Shrink the robots.txt file
• Allow search engines in robots.txt
• Publish an XML sitemap
• List the sitemap in robots.txt
• Make the sitemap reachable
• Fix the sitemap format
• Put URLs into the sitemap
• Split the sitemap into several files
• Remove URLs from other domains in the sitemap