Add a robots.txt file on WordPress
Why it matters
Without a robots.txt, crawlers assume they may fetch everything, which is a legitimate choice. A small file still lets you name your sitemap and gives you a place to set rules later.
The fix
- WordPress serves a virtual robots.txt when no physical file exists, and it already points to the core sitemap. If /robots.txt returns 404, save Settings, Permalinks once to refresh the rewrite rules.
- If it still fails, look for a security or cache plugin or a server rule that blocks the path. To change the rules, use your SEO plugin's robots.txt editor or upload a physical robots.txt to the web root.
Related
- Reported by the robots.txt and sitemap check
- Fix the server error on robots.txt
- Fix the syntax of robots.txt
- Shrink the robots.txt file
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.