Claude-User in robots.txt
Claude-User is run by Anthropic. Purpose: user fetch. Source: Anthropic documentation.
What blocking Claude-User changes
This agent fetches a page when a person asks the assistant about it. Blocking it stops those live reads.
Block Claude-User
User-agent: Claude-User Disallow: /
Block Claude-User from one folder
User-agent: Claude-User Disallow: /private/
Allow Claude-User
If no group names Claude-User, it follows your "User-agent: *" rules. To allow it while other bots are blocked:
User-agent: Claude-User Allow: /
More: the guide AI crawlers in robots.txt and the AI crawler checker.
Questions
What is Claude-User?
Claude-User is the user agent token for an agent run by Anthropic. Its purpose: user fetch. See the Anthropic documentation linked on this page.
How do I block Claude-User?
Add a group to robots.txt with "User-agent: Claude-User" and "Disallow: /". The rule applies to agents that follow robots.txt.
How do I check if my site blocks Claude-User?
Run the free check on this site. It tests your robots.txt against 8 AI agents on your homepage. The $19 deep report tests 18 agents per path.
Get an email when AI crawlers, robots.txt rules, or llms.txt practice change. At most twice a month.
We send a confirmation link first. See the privacy policy.