Home / Guides

AI crawlers in robots.txt: GPTBot, ClaudeBot, PerplexityBot rules

AI companies send several bots to your site, and each one does a different job. One collects pages that may train a model. Another builds a search index so the assistant can cite you. A third fetches a page when a user asks about it. Your robots.txt can treat each one on its own.

This guide lists the main AI user agents, what each vendor says about them, and rules you can paste. Check your live file with the AI crawler robots.txt checker.

The user agents that matter

OpenAI

From >OpenAI's crawler docs:

• GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals that your content should not be used for training.
• OAI-SearchBot surfaces websites in ChatGPT's search features. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can appear as navigational links.
• ChatGPT-User visits a page when a ChatGPT user or a Custom GPT asks for it. OpenAI says it is not used for automatic crawling, that robots.txt rules "may not apply" because a user started the action, and that it is not used to decide what appears in Search.

OpenAI says each setting works on its own, and that search results can take about 24 hours to reflect a robots.txt change.

Anthropic

From >Anthropic's help article:

• ClaudeBot collects web content that could contribute to model training. Blocking it signals that future content should be excluded from training datasets.
• Claude-User fetches pages when a Claude user asks a question. Blocking it may reduce your visibility in user-directed web search.
• Claude-SearchBot indexes content to improve Claude's search results. Blocking it may reduce your visibility and accuracy in search answers.

Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay rule. It asks site owners to add rules for every subdomain they want to opt out, and warns that blocking its IP addresses may not work because that stops its bots from reading your robots.txt.

Perplexity

From >Perplexity's bot docs:

• PerplexityBot surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.
• Perplexity-User fetches pages when a user asks a question. Perplexity says that because a user requested the fetch, it "generally ignores robots.txt rules."

Perplexity says changes can take up to 24 hours to take effect.

Google

Google-Extended is a robots.txt token, not a separate crawler. Google says sites can use it to control whether their content helps train future Gemini models, and that using it "doesn't affect a site's inclusion in Search," nor is it a ranking signal (>Google crawling docs). Googlebot rules control Google Search.

Common Crawl

CCBot builds Common Crawl's open web archive, which many AI projects use as data. Common Crawl publishes the rule User-agent: CCBot / Disallow: / and warns that some crawlers pose as CCBot (>CCBot page).

Three policies you can paste

1. Allow everything

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Best if you want the most AI visibility and do not object to training use.

2. Allow AI search, block training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This keeps you eligible for ChatGPT search, Claude search, and Perplexity results, while telling the training crawlers named above to stay out. It does not affect Googlebot.

3. Block all listed AI bots

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Grouping several User-agent lines over one rule is valid robots.txt syntax. Per the vendor docs above, this removes you from ChatGPT search answers and may cut your Claude search visibility. User-triggered fetchers from OpenAI and Perplexity may visit, since both vendors say robots.txt may not apply to them.

What blocking costs you

A business that wants new customers from AI answers should weigh policy 3 with care. If OAI-SearchBot cannot read your pages, OpenAI says your site will not be shown in ChatGPT search answers. Blocking Claude-SearchBot and Claude-User may cut your visibility in Claude. Training bots are a separate choice: blocking GPTBot does not block OAI-SearchBot.

What big SaaS sites do

On October 17, 2026 we checked robots.txt on 49 popular SaaS websites for eight AI user agents. 45 allowed all eight. Four blocked at least one:

• Figma blocked GPTBot, ClaudeBot, Google-Extended, and CCBot.
• Canva blocked GPTBot, ClaudeBot, and CCBot.
• Loom blocked GPTBot.
• Calendly blocked CCBot.

None of the 49 blocked OAI-SearchBot, Claude-SearchBot, or PerplexityBot. The full data is in the State of AI readiness report.

Mistakes we see

• A rule under the wrong group. Rules apply to the User-agent lines right above them. A Disallow placed under User-agent: * does not override a bot that has its own group.
• Blocking by IP. Anthropic warns this may stop its bots from reading your robots.txt at all.
• Forgetting subdomains. Each subdomain, such as docs.example.com, needs its own robots.txt.
• A firewall that blocks the bot before robots.txt loads. Perplexity's docs explain how to allow its bots in Cloudflare and AWS WAF rules.

Check your site

Run the AI readiness check on your domain. It reports which of the eight AI user agents your robots.txt allows, along with llms.txt, JSON-LD, sitemap, and text readable without JavaScript. For the llms.txt side, read llms.txt example and format.

FAQ

Does blocking GPTBot remove me from ChatGPT search? No. OpenAI says GPTBot and OAI-SearchBot are separate settings. Block GPTBot for training and allow OAI-SearchBot for search.

Do AI crawlers obey robots.txt? OpenAI, Anthropic, Perplexity, Google, and Common Crawl all document robots.txt controls for their crawlers. OpenAI and Perplexity say their user-triggered fetchers may not follow robots.txt.

Does Google-Extended affect my Google rankings? Google says it does not affect inclusion in Search and is not a ranking signal.

How long until a change takes effect? OpenAI says about 24 hours for search. Perplexity says up to 24 hours.

Checked 2026-10-17. Prices and policies change. Confirm on the source before you buy.

Sources

• https://platform.openai.com/docs/bots
• https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• https://docs.perplexity.ai/guides/bots
• https://developers.google.com/crawling/docs/about-crawling
• https://developers.google.com/crawling
• https://commoncrawl.org/ccbot
• https://checkaiready.com/state-of-ai-readiness.json

More guides

• llms.txt example and format: how to write one that passes (2026)