The page you published is not the page machines receive.
Two Webs fetches any URL fifteen ways, as a Chrome browser, as Googlebot and Bingbot, and as twelve AI crawlers and assistants, then lines the responses up side by side. A 403 for GPTBot. A Cloudflare challenge for Googlebot. Nine words of HTML where you see nine hundred. Whatever machines get, you get to see it.
One URL, two responses
| Check | Chrome | GPTBot |
|---|---|---|
| Status | 200 | 403 |
| Title | How we rebuilt checkout | Just a moment… |
| Words | 1,860 | 3 |
| robots.txt | allowed | Disallow: / |
| Challenge | none | cf-mitigated: challenge |
| Canonical | /blog/checkout | none |
Illustrative. A real report covers 18 agents and links every finding to the rule or header that caused it.
What it checks
robots.txt, read the way crawlers read it
RFC 9309 matching for 18 crawler tokens: most specific group wins, longest rule wins, Allow beats Disallow on a tie. Plus llms.txt and llms-full.txt.
Live fetches with real user agents
Every crawler that makes HTTP requests is imitated with its published user agent string. Status, X-Robots-Tag, meta robots, canonical and visible word count are compared against what the browser got.
Findings you can act on
Blocked, challenged, degraded, cloaked, app shell, noindex, meta-refresh stub. Each one names the agent and the rule or header behind it.
Why the two drift apart
Blocking decisions pile up in different places. A robots.txt group someone added in 2023, a CDN firewall rule from the last bot scare, a framework that renders nothing until JavaScript runs. Nobody adds them up from the crawler's side. So a site ends up open to every training crawler and closed to the one search bot that would have cited it, or a React app that looks finished in Chrome sends 40 bytes to everyone else. One request shows the difference.
Questions
Does Two Webs run JavaScript when it fetches a page?
No. It reads the raw HTML each server sends, which is what GPTBot, ClaudeBot, PerplexityBot and Googlebot's first pass read too. If your page is an empty div until a bundle runs, the report shows it that way.
Which crawlers are checked?
Eighteen tokens: a Chrome browser, Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider and CCBot. Fifteen of them make live requests. The other three are robots-only tokens.
Why does a site show as blocked for every agent, including the human one?
Some sites verify crawlers by reverse DNS. Our fetches come from Cloudflare's network rather than each crawler's published IP ranges, so those sites challenge everything we send. The report says so when it happens.
What does the score mean?
It starts at 100 and drops for AI agents that are blocked or degraded, for empty app shells, for noindex, for a blocked Googlebot and, lightly, for a missing llms.txt. A publisher that blocks training crawlers on purpose can score in the 30s and be getting exactly what it wants.
Is there an API?
Yes. GET /api/compare/?url= returns the same report as JSON. Results are cached for ten minutes and limited to ten comparisons a minute per IP.