AI crawler robots.txt checker
Your robots.txt decides which AI crawlers may read your site. We test GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and CCBot against your homepage. Free. No signup.
We read your homepage, robots.txt, sitemap.xml, and llms.txt. Nothing else. Results are cached for 6 hours.
See the State of AI readiness data page. Need a page-by-page audit? The deep report crawls 25 pages and lists fixes by priority.
Questions
Should I block AI crawlers?
It depends on your goals. Blocking search crawlers like OAI-SearchBot can keep you out of AI answers. Blocking training crawlers like GPTBot or CCBot does not affect that. Decide per bot.
My robots.txt has no AI rules. Am I blocked?
No. If no rule matches a bot, the bot falls back to the * rules. If those allow your homepage, the bot is allowed.
What does the check test?
Six things: llms.txt, AI crawler access in robots.txt, JSON-LD schema, sitemap.xml, a meta description, and text readable without JavaScript. The score runs from 0 to 100.
Which AI crawlers do you check?
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and CCBot. We read your robots.txt and test each one against your homepage.
Is it free?
Yes. No signup. Results are cached for 6 hours.
What do you read on my site?
Your homepage, robots.txt, sitemap.xml, and llms.txt. Nothing else.
Other checks: AI readiness checker · llms.txt checker · llms.txt validator · State of AI readiness · llms.txt generator
Also from us: Trades With AI (AI tools for trades businesses) · SaaS Price Watch (SaaS pricing, tracked weekly) · WrapMockup (Free vehicle wrap mockups for trades)
Get an email when AI crawlers, robots.txt rules, or llms.txt practice change. At most twice a month.
We send a confirmation link first. See the privacy policy.