Shrink the robots.txt file
Why it matters
Google reads only the first 500 KiB of robots.txt and ignores the rest, so rules at the end may never apply. Very long files are also hard to maintain.
The fix
- Remove duplicate and obsolete rules, and replace long lists of single URLs with a pattern using * or $, for example Disallow: /*?sessionid=.
- For individual pages that should stay out of search results, use a noindex meta tag or header instead of one Disallow line per page, and put your most important rules at the top of the file.
- Reload https://example.com/robots.txt and check it is well under 500 KiB.
The examples use example.com. Replace it with your own domain.
Related
- Reported by the robots.txt and sitemap check
- Add a robots.txt file
- Fix the server error on robots.txt
- Fix the syntax of robots.txt
- Allow search engines in robots.txt
- Publish an XML sitemap
- List the sitemap in robots.txt
- Make the sitemap reachable
- Fix the sitemap format
- Put URLs into the sitemap
- Split the sitemap into several files
- Remove URLs from other domains in the sitemap
Scan your domain to see whether this applies to you.