AI search
Robots.txt for GPTBot, ClaudeBot, and PerplexityBot
· Configuration notes, not a newsroom.
How a Disallow on AI crawlers silently zeros your citation share — and how SiteRune reads robots.txt without guessing.
The new 404 is a robots rule
If GPTBot, ClaudeBot, PerplexityBot, or Google-Extended are Disallow: /, you will not show up as a citation no matter how good the prose is. SiteRune parses robots.txt and scores the AI-bot policy as its own finding. We do not crawl as those bots. We read the file.
A lot of WordPress security plugins ship a 'block AI' preset. That is a product decision, not a default anyone meant. Check the file, not the plugin marketing page.
A policy that still lets you hide staging
You can allow the named AI crawlers on public marketing and docs while still Disallowing /wp-admin/, /cart/, /checkout/, and preview hosts. SiteRune's playbook writes that split. It will not recommend opening the entire origin if your robots already protect admin.
Run it on a live URL
The brief will name your CMS and attach the files. Three guest scans, no card.