Fix the syntax of robots.txt on Netlify
Why it matters
A robots.txt that is really an html page, has rules before any User-agent line, or misspells a directive is partly or wholly ignored by crawlers. Rules you meant to apply then do nothing.
The fix
- Put robots.txt in the published folder so it is served as a static file. An existing file is served before a catch-all rewrite such as /* /index.html 200, unless that rule has force enabled; remove force from it.
- Redeploy and check https://example.com/robots.txt.
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Shrink the robots.txt file
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.