robots.txt & sitemap check
robots.txt
File content
Sitemaps
| Sitemap | Status | Type | URL | lastmod | Size | Notes |
|---|
How to read it
- robots.txt: unknown directives are ignored; an empty User-agent line voids the group; a Sitemap line is the easiest way to let engines find the map. Google ignores Crawl-delay. 'Disallow: /' closes the whole site.
- Limits: 50,000 URLs and 50 MB uncompressed per sitemap; split with an index when you exceed them.
- lastmod: write the real change date only; Google ignores maps that say 'today' every day.
- HTML returned: the sitemap address returns a page (often a 404 page that answers 200); the content type should be XML.
- For an index, at most 20 child sitemaps are fetched; the URL total is computed from those children.