AI crawler user agents
Each page shows who runs the agent, what it is for, the robots.txt rules to block or allow it, and a link to the operator documentation.
- GPTBot (OpenAI, training)
- OAI-SearchBot (OpenAI, search)
- ChatGPT-User (OpenAI, user fetch)
- ClaudeBot (Anthropic, training)
- Claude-SearchBot (Anthropic, search)
- Claude-User (Anthropic, user fetch)
- PerplexityBot (Perplexity, search)
- Perplexity-User (Perplexity, user fetch)
- Google-Extended (Google, AI training control)
- Googlebot (Google, search)
- Bingbot (Microsoft, search)
- Applebot-Extended (Apple, AI training control)
- Amazonbot (Amazon, search and AI)
- Meta-ExternalAgent (Meta, training)
- CCBot (Common Crawl, open web dataset)
- DuckAssistBot (DuckDuckGo, AI answers)
- MistralAI-User (Mistral, user fetch)
Get an email when AI crawlers, robots.txt rules, or llms.txt practice change. At most twice a month.
We send a confirmation link first. See the privacy policy.