CCBot in robots.txt
CCBot is run by Common Crawl. Purpose: open web dataset. Source: Common Crawl documentation.
What blocking CCBot changes
Common Crawl publishes an open archive of the web that many AI projects use. Blocking it keeps new pages out of that archive.
Block CCBot
User-agent: CCBot Disallow: /
Block CCBot from one folder
User-agent: CCBot Disallow: /private/
Allow CCBot
If no group names CCBot, it follows your "User-agent: *" rules. To allow it while other bots are blocked:
User-agent: CCBot Allow: /
How many sites block CCBot?
Of the 49 SaaS sites in our State of AI readiness check on 2026-10-17, 3 block CCBot from their homepage: figma.com, canva.com, calendly.com.
More: the guide AI crawlers in robots.txt and the AI crawler checker.
Questions
What is CCBot?
CCBot is the user agent token for an agent run by Common Crawl. Its purpose: open web dataset. See the Common Crawl documentation linked on this page.
How do I block CCBot?
Add a group to robots.txt with "User-agent: CCBot" and "Disallow: /". The rule applies to agents that follow robots.txt.
How do I check if my site blocks CCBot?
Run the free check on this site. It tests your robots.txt against 8 AI agents on your homepage. The $19 deep report tests 18 agents per path.
Get an email when AI crawlers, robots.txt rules, or llms.txt practice change. Up to two emails a month.
We send a confirmation link first. See the privacy policy.