Google-Extended in robots.txt
Google-Extended is run by Google. Purpose: AI training control. Source: Google documentation.
What blocking Google-Extended changes
This is a control token, not a separate crawler. Blocking it tells the operator not to use your pages for its AI models. Regular search crawling is not affected.
Block Google-Extended
User-agent: Google-Extended Disallow: /
Block Google-Extended from one folder
User-agent: Google-Extended Disallow: /private/
Allow Google-Extended
If no group names Google-Extended, it follows your "User-agent: *" rules. To allow it while other bots are blocked:
User-agent: Google-Extended Allow: /
How many sites block Google-Extended?
Of the 49 SaaS sites in our State of AI readiness check on 2026-10-17, 1 block Google-Extended from their homepage: figma.com.
More: the guide AI crawlers in robots.txt and the AI crawler checker.
Questions
What is Google-Extended?
Google-Extended is the user agent token for an agent run by Google. Its purpose: AI training control. See the Google documentation linked on this page.
How do I block Google-Extended?
Add a group to robots.txt with "User-agent: Google-Extended" and "Disallow: /". The rule applies to agents that follow robots.txt.
How do I check if my site blocks Google-Extended?
Run the free check on this site. It tests your robots.txt against 8 AI agents on your homepage. The $19 deep report tests 18 agents per path.