Publishers / britannica.com
Invisible to machines
2 of 14 AI agents get the full page at https://britannica.com/. Checked 2026-09-25 09:33 UTC.
britannica.com scores 1 out of 100 for machine visibility, number 87 of the 100 content publishers checked on 2026-09-25. Even a plain request carrying a Chrome user agent was refused (HTTP 403), so britannica.com serves its pages only to clients that pass a full browser check. Every crawler below got the same treatment, which is why the score is so low. All 12 refusals happen at the HTTP level with no matching robots.txt rule, so a crawler cannot tell the block is deliberate. Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot received a bot-management challenge instead of the page. Googlebot itself is blocked: Bot challenge served (HTTP 403, cf-mitigated=challenge). The raw HTML is an app shell with 3 words; content appears only after JavaScript runs. There is no llms.txt.
youfull pagedegradedblockedunknown
Human
- Status
- 403 → https://www.britannica.com/
- Title
- Just a moment...
- H1
- none
- Words
- 3
- Scripts
- 1 tags · 3 KB · app shell
- Meta robots
- noindex,nofollow
- Canonical
- none
- Server
- cloudflare
Machine
- robots.txt
- 21 groups · 1 sitemaps
- llms.txt
- missing
- llms-full.txt
- missing
- X-Robots-Tag
- none
- Blocked
- GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
- Degraded
- none
- Challenged
- Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
Gaps
- highHumans cannot load this page eitherHTTP 403. Every other check is measured against a broken baseline.
- high12 of 14 AI agents are blockedAssistants asked about this page will guess, or cite someone else. Deliberate for a paywalled publisher. Otherwise this is the biggest gap on the list.
- mediumBot-management challenge served to crawlersGooglebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot received an interstitial instead of the page. A robots.txt rule is an honest block. A challenge hides the page without saying so.
- lowNo llms.txtOptional, but a short curated index at /llms.txt is the cheapest way to tell assistants what matters on this site.
- highGooglebot is blockedBot challenge served (HTTP 403, cf-mitigated=challenge)
- infoGoogle-Extended not listedYou block other training crawlers but Gemini training via Google-Extended is still allowed. Add it to the same group if the policy is meant to be even-handed.
Every agent
| Agent | Type | robots.txt | Fetch | Words | Result |
|---|---|---|---|---|---|
| Human (Chrome)You | Human | allowed | 403 challenge1387 ms | 3 | blocked HTTP 403 |
| GooglebotGoogle · Classic search index. Also feeds AI Overviews. | Search engine | allowed | 403 challenge1369 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| Google-ExtendedGoogle · Robots-only token controlling Gemini training use. Never fetches on its own. | AI training | allowed | n/a | visible Robots-only token; allowed by robots.txt | |
| GPTBotOpenAI · Training crawler for OpenAI models. | AI training | allowed | 403 challenge1368 ms | 0 | blocked Bot challenge served (HTTP 403) |
| OAI-SearchBotOpenAI · Indexes for ChatGPT search results and citations. | AI search index | allowed | 403 challenge1386 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| ChatGPT-UserOpenAI · Live fetch when a user asks ChatGPT about a page. | AI assistant (live fetch) | allowed | 403 challenge1386 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| ClaudeBotAnthropic · Training crawler for Claude models. | AI training | allowed | 403 challenge1386 ms | 0 | blocked Bot challenge served (HTTP 403) |
| Claude-SearchBotAnthropic · Indexes for Claude web search. | AI search index | allowed | 403 challenge1385 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| Claude-UserAnthropic · Live fetch when a user asks Claude about a page. | AI assistant (live fetch) | allowed | 403 challenge1385 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| PerplexityBotPerplexity | AI search index | allowed | 403 challenge1384 ms | 0 | blocked Bot challenge served (HTTP 403) |
| Perplexity-UserPerplexity | AI assistant (live fetch) | allowed | 403 challenge1385 ms | 0 | blocked Bot challenge served (HTTP 403) |
| BingbotMicrosoft · Bing index. Also feeds Copilot and ChatGPT search. | Search engine | allowed | 403 challenge1384 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| Applebot-ExtendedApple · Robots-only token for Apple Intelligence training. | AI training | allowed | n/a | visible Robots-only token; allowed by robots.txt | |
| Meta-ExternalAgentMeta | AI training | allowed | 403 challenge1384 ms | 0 | blocked Bot challenge served (HTTP 403) |
| AmazonbotAmazon · Alexa and Amazon AI. | AI training | allowed | 403 challenge1392 ms | 3 | blocked Bot challenge served (HTTP 403, cf-mitigated=challenge) |
| BytespiderByteDance · Known for ignoring robots.txt. | AI training | allowed | 403 challenge1389 ms | 0 | blocked Bot challenge served (HTTP 403) |
| CCBotCommon Crawl · Open dataset used to train most LLMs. | AI training | allowed | 403 challenge1383 ms | 0 | blocked Bot challenge served (HTTP 403) |
Snapshot
Measured 2026-09-25 09:33 UTC from a residential connection. Sites change their rules often, and sites that verify crawler IPs answer differently to different networks. Run a live comparison to see what britannica.com does right now, from Cloudflare's network.