Free tool
robots.txt Checker
Reads https://your-host/robots.txt and shows the rules a polite crawler would honor. Does not impersonate Googlebot.
What this check covers
- File present vs 404
- User-agent blocks and blanket Disallow: /
- Sitemap: lines
Read it honestly
- robots.txt is not a security boundary. It is a crawl hint.
Questions
Does a Disallow hide the page from Google?
It asks Googlebot not to crawl. URLs can still appear if they are linked. noindex is a different signal — run the full audit for meta robots.
Run the complete SiteRune audit
This tool is one chapter. The full brief walks interiors, scores every chapter, and attaches ship files.