SEO Intelligence
Robots.txt & Sitemap Checker
Catch the contradiction most sites never notice: pages submitted for indexing in your sitemap that robots.txt actually blocks.
What it checks
- ✓Parses every user-agent group, allow/disallow rule, and declared sitemap in robots.txt
- ✓Recursively expands nested sitemap indexes with cycle protection, to any depth
- ✓Cross-checks every sitemap URL against robots.txt and lists exactly which ones are blocked from crawling
- ✓Also powers URL discovery used elsewhere in the platform, like URL Matcher