Fix the server error on robots.txt on WordPress

Why it matters

While robots.txt answers with a 5xx error, Google stops crawling the site for about 12 hours, then relies on its last good copy for up to 30 days. The cause is usually a failing backend, a script that builds the file, or a firewall.

The fix

1. Turn on logging by setting WP_DEBUG and WP_DEBUG_LOG to true and WP_DEBUG_DISPLAY to false in wp-config.php, request /robots.txt, then read wp-content/debug.log. When you are done, set them back to false and delete debug.log, which can hold sensitive details and may be readable from the web.
2. Deactivate plugins that add or change robots.txt (SEO, cache and security plugins) one at a time to find the one that fails, or upload a static robots.txt to the web root, which bypasses WordPress.

define( 'WP_DEBUG', true );
define( 'WP_DEBUG_LOG', true );
define( 'WP_DEBUG_DISPLAY', false );
// Set these back to false and delete wp-content/debug.log when you are done

Read the documentation ↗(opens in a new tab)

Related

• Reported by the robots.txt and sitemap check
• Add a robots.txt file
• Fix the syntax of robots.txt
• Shrink the robots.txt file
• Allow search engines in robots.txt
• Publish an XML sitemap
• List the sitemap in robots.txt
• Make the sitemap reachable
• Fix the sitemap format
• Put URLs into the sitemap
• Split the sitemap into several files
• Remove URLs from other domains in the sitemap