How a comparison is made.
Eighteen points of view on one URL. Here is what happens, step by step, when you press Compare.
Method
1. Policy: robots.txt and llms.txt
We fetch /robots.txt and evaluate the page path against every crawler token below using RFC 9309 rules: the most specific matching group wins, the longest matching rule wins, and Allow beats Disallow on a tie. We also check whether /llms.txt and /llms-full.txt exist and are real text files rather than soft-404 HTML pages.
2. Reality: a live fetch as each agent
For every crawler that actually makes HTTP requests, we fetch the page with that crawler's published user agent and compare the response to the human one: status code, bot-management challenges, X-Robots-Tag, meta robots, canonical, and the amount of visible text in the raw HTML. Two agents that get different text for the same URL are flagged, because that is what cloaking looks like from the outside.
We do not execute JavaScript, and neither do most AI crawlers. If the raw HTML is an empty shell, an empty shell is what they read.
3. Score
The score starts at 100 and drops for AI agents that are blocked or degraded, for empty app shells, for noindex, for a blocked Googlebot and, lightly, for a missing llms.txt. The score measures visibility only. A publisher that blocks every training crawler on purpose will score in the 30s and be getting exactly what it wants. The gap list is where intent gets sorted from accident.
Agents checked
| Agent | Owner | Type | robots token | Live fetch |
|---|---|---|---|---|
| Human (Chrome) | You | Human | * | yes |
| GooglebotClassic search index. Also feeds AI Overviews. | Search engine | Googlebot | yes | |
| Google-ExtendedRobots-only token controlling Gemini training use. Never fetches on its own. | AI training | Google-Extended | robots only | |
| GPTBotTraining crawler for OpenAI models. | OpenAI | AI training | GPTBot | yes |
| OAI-SearchBotIndexes for ChatGPT search results and citations. | OpenAI | AI search index | OAI-SearchBot | yes |
| ChatGPT-UserLive fetch when a user asks ChatGPT about a page. | OpenAI | AI assistant (live fetch) | ChatGPT-User | yes |
| ClaudeBotTraining crawler for Claude models. | Anthropic | AI training | ClaudeBot | yes |
| Claude-SearchBotIndexes for Claude web search. | Anthropic | AI search index | Claude-SearchBot | yes |
| Claude-UserLive fetch when a user asks Claude about a page. | Anthropic | AI assistant (live fetch) | Claude-User | yes |
| PerplexityBot | Perplexity | AI search index | PerplexityBot | yes |
| Perplexity-User | Perplexity | AI assistant (live fetch) | Perplexity-User | yes |
| BingbotBing index. Also feeds Copilot and ChatGPT search. | Microsoft | Search engine | Bingbot | yes |
| Applebot-ExtendedRobots-only token for Apple Intelligence training. | Apple | AI training | Applebot-Extended | robots only |
| Meta-ExternalAgent | Meta | AI training | Meta-ExternalAgent | yes |
| AmazonbotAlexa and Amazon AI. | Amazon | AI training | Amazonbot | yes |
| BytespiderKnown for ignoring robots.txt. | ByteDance | AI training | Bytespider | yes |
| CCBotOpen dataset used to train most LLMs. | Common Crawl | AI training | CCBot | yes |
Privacy and who runs this
A comparison fetches the URL you enter, plus that site's robots.txt, llms.txt and llms-full.txt. The report is cached at Cloudflare's edge for ten minutes and then discarded. URLs are not logged, stored or shared beyond that cache. The only measurement is Cloudflare Web Analytics, which is cookieless and records page views without personal data. The theme choice lives in your browser's local storage only.
Two Webs is built and run by Patrick Jennings, a digital strategist in Adelaide, Australia. The source is public on GitHub.
Limits
- Requests come from Cloudflare's network, not from the crawler's real IP ranges. Sites that verify crawlers by reverse DNS will treat all of our fetches as impostors, which usually means a challenge or 403 for every agent including the human one. When that happens the report says so.
- Bodies are capped at 1.5 MB and fetches at 12 seconds.
- Results are cached for ten minutes per URL.