Guides

Each guide quotes its sources and the date we checked them.

• AI crawlers in robots.txt: GPTBot, ClaudeBot, PerplexityBot rules (2026)
  Which AI bots read robots.txt, what blocking each one costs you, and copy-paste rules to allow search but block training. Sourced from vendor docs.
• llms.txt example and format: how to write one that passes (2026)
  The llms.txt format, a copy-ready example, and the 5 mistakes we see most. Based on the llmstxt.org spec and checks of 49 SaaS sites.