Add a robots.txt file

Why it matters

Without a robots.txt, crawlers assume they may fetch everything, which is a legitimate choice. A small file still lets you name your sitemap and gives you a place to set rules later.

The fix

1. Create a UTF-8 text file named robots.txt in the root of the site, so it is served at https://example.com/robots.txt.
2. Start with an allow-all rule (and a Sitemap line if you have a sitemap), and add Disallow rules only for paths you really want crawlers to skip (such as internal search results). Robots.txt is not a way to hide private pages.
3. Open the address in a browser to confirm it returns the text with a 200 status.

User-agent: *
Disallow:

# Keep the next line only if the sitemap exists at this address
Sitemap: https://example.com/sitemap.xml

The examples use example.com. Replace it with your own domain.

Read the documentation ↗(opens in a new tab)

Related

• Reported by the robots.txt and sitemap check
• Fix the server error on robots.txt
• Fix the syntax of robots.txt
• Shrink the robots.txt file
• Allow search engines in robots.txt
• Publish an XML sitemap
• List the sitemap in robots.txt
• Make the sitemap reachable
• Fix the sitemap format
• Put URLs into the sitemap
• Split the sitemap into several files
• Remove URLs from other domains in the sitemap