AI readiness of screamingfrog.co.uk: 85/100

85/100

We checked screamingfrog.co.uk on 2026-10-18 with our free checker: six scored tests plus evidence for each finding. The score reflects the site on that date. The site may have changed after that date.

Test	Result	Points
llms.txt	No	0 of 15
JSON-LD schema on the homepage	Yes	20 of 20
Meta description	Yes	10 of 10
Text readable without JavaScript	Yes	20 of 20
Sitemap	Yes	10 of 10
AI crawlers allowed in robots.txt	8 of 8	25 of 25

Compared with SEO tools

screamingfrog.co.uk ranks 12 of 20 seo tools sites in this scan (ties share the order by domain name). The median score in this group is 85. 6 of 20 have an llms.txt file.

Findings with evidence

FixNo llms.txt

GET /llms.txt returned 404.

Fix: Create /llms.txt: a # title, a > one-line summary, then ## sections that list key pages as [name](url). Draft one free at checkaiready.com/llms-txt-generator.

ImproveServer responded in 1238 ms

Time to response headers from our checker, after 1 redirect. Final URL: https://www.screamingfrog.co.uk/

Fix: Slow responses can make crawlers give up. Add page caching or a CDN in front of the homepage.

18 checks passed or noted

PassGPTBot (OpenAI, training): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassOAI-SearchBot (OpenAI, ChatGPT search): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassChatGPT-User (OpenAI, live fetch for users): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassClaudeBot (Anthropic, training): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassClaude-SearchBot (Anthropic, Claude search): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassPerplexityBot (Perplexity, search index): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassGoogle-Extended (Google, Gemini training control): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassCCBot (Common Crawl, open dataset): allowed on /

User-agent: *
Disallow: /products/
Disallow: /?s=*
Disallow: /*?post_type*
Disallow: /search/
Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php
(1 more lines)
Group "*" has no rule for /.

PassJSON-LD: 1 block, types WebPage, ReadAction, BreadcrumbList, ListItem, WebSite, SearchAction, EntryPoint, PropertyValueSpecification, Organization, ImageObject

Every typed block has its required properties.

PassTitle, 27 characters

<title>SEO Agency - Screaming Frog</title>

PassMeta description, 148 characters

content="We run remarkably successful search engine optimisation (SEO) and paid search (PPC) campaigns using a unique blend of data, technology & creativity."

PassOpen Graph tags present

og:title: Screaming Frog | SEO, Search Engine Marketing Agency
og:description: We run remarkably successful search engine optimisation (SEO) and paid search (PPC) campa…
og:image: https://www.screamingfrog.co.uk/wp-content/uploads/2021/09/screaming-frog-social.png

PassCanonical link set

<link rel="canonical" href="https://www.screamingfrog.co.uk/">

PassLanguage declared: en

<html lang="en">

Pass1 H1 heading

<h1>We are a UK based SEO agency and the creators of the famous website crawler and log file analyser t…</h1>

Pass9,609 characters of text without JavaScript

Text starts: "SEO Agency - Screaming Frog About Team Our Story Clients Case Studies Careers Agency Services Search Engine Marketing Search Engine Optimisation AI Search Opti…"

PassSitemap found, 617 URLs

robots.txt has no Sitemap: line.
GET https://screamingfrog.co.uk/sitemap.xml returned 200.

Fix: Add a "Sitemap:" line to robots.txt so every crawler finds it.

PassHTML size 196 KB, 58 script tags

2 render-blocking scripts in <head>: https://html5shiv.googlecode.com/svn/trunk/html5.js

Fix: Add defer or async to scripts in <head> that are not needed for first paint.

Is this your site? Run a fresh free check. Get the $19 deep report (up to 25 pages, 18 AI agents per path), or add an AI-ready badge after your check.

See how all 20 SEO tools sites compare: common problems, real evidence and the full table.

Scoring: llms.txt 15, JSON-LD 20, meta description 10, readable text 20, sitemap 10, AI crawler access 25. Want this page changed or removed? Email [email protected].

AI crawler and llms.txt updates

Get an email when AI crawlers, robots.txt rules, or llms.txt practice change. Up to two emails a month.

We send a confirmation link first. See the privacy policy.