Fix the syntax of robots.txt

Why it matters

A robots.txt that is really an html page, has rules before any User-agent line, or misspells a directive is partly or wholly ignored by crawlers. Rules you meant to apply then do nothing.

The fix

1. Open https://example.com/robots.txt. If it shows an html page (a 404 page or your single-page app), the real file is missing and a catch-all rule is answering; add a static robots.txt that is served before that rule.
2. Start every group with a User-agent line, followed by its rules. Spell directives exactly as Allow, Disallow and Sitemap, each on its own line, with comments starting with a
3. Save the file as UTF-8, then test it again, for example with the robots.txt report in Google Search Console.

User-agent: *
Disallow: /search/
Allow: /

# Keep the next line only if the sitemap exists at this address
Sitemap: https://example.com/sitemap.xml

The examples use example.com. Replace it with your own domain.

Read the documentation ↗(opens in a new tab)

Related

• Reported by the robots.txt and sitemap check
• Add a robots.txt file
• Fix the server error on robots.txt
• Shrink the robots.txt file
• Allow search engines in robots.txt
• Publish an XML sitemap
• List the sitemap in robots.txt
• Make the sitemap reachable
• Fix the sitemap format
• Put URLs into the sitemap
• Split the sitemap into several files
• Remove URLs from other domains in the sitemap