Four sites, four different webs.
What the reports found on a paywalled publisher, a site that blocks everyone, a documentation site that serves one page to all, and a baseline with nothing to fix. Each links to a live re-run.
nytimes.com
35 · Invisible to machinesThe clearest case of a deliberate split. robots.txt has 63 agent groups and disallows every AI token individually: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider and CCBot all get "Disallow: /". Googlebot and Bingbot are allowed by policy but received an HTTP 403 challenge from our network, because the Times verifies search crawlers by IP range. The human fetch got the full page: 1,121 words, 57 script tags, canonical set. Zero of fourteen AI agents get the page. For a paywalled publisher with licensing deals, that is the intended outcome, and the report says so.
reddit.com
20 · Invisible to machinesReddit blocks with a single wildcard: "User-agent: *" then "Disallow: /", so every crawler without its own group is refused, Googlebot included. On top of the policy, the live fetches came back as 403s and 429s, and the one response that did arrive was a JavaScript shell with a single visible word. This is the lowest score in the set and the most interesting, because Reddit sells its content to AI companies under contract while refusing it to everyone else. The robots file is the contract boundary.
docs.astro.build
95 · Mostly one webA documentation site that gets it right. Every agent, human or machine, receives the same 768 words of server-rendered HTML with a 200. robots.txt is two lines that allow everything. The only deduction is a missing llms.txt. One detail worth knowing: the site root at docs.astro.build/ is an 80-byte meta-refresh stub that forwards browsers to /en/getting-started/. Browsers follow it. Most crawlers do not. Two Webs flags that as a gap on the root URL and recommends a real 301.
runsonlinux.com
100 · Mostly one webA static Astro site on Cloudflare with no bot management. All fifteen live fetches returned the same 560 words with HTTP 200, robots.txt allows every agent, and llms.txt is present. Nothing to fix. It is included as the baseline for what a full score looks like.