Publishers / usatoday.com
50
Two different webs
0 of 14 AI agents get the full page at https://usatoday.com/. Checked 2026-09-25 09:33 UTC.
usatoday.com scores 50 out of 100 for machine visibility, number 51 of the 100 content publishers checked on 2026-09-25. None of the 14 AI agents receive the page. All 14 refusals are written in robots.txt, which is the transparent way to do it. There is no llms.txt.
youfull pagedegradedblockedunknown
Human
- Status
- 200 → https://www.usatoday.com/
- Title
- USA TODAY - Breaking News and Latest News Today
- H1
- none
- Words
- 620
- Scripts
- 9 tags · 147 KB
- Meta robots
- none
- Canonical
- none
- Server
- hidden
Machine
- robots.txt
- 287 groups · 7 sitemaps
- llms.txt
- missing
- llms-full.txt
- missing
- X-Robots-Tag
- noarchive,nocache
- Blocked
- Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
- Degraded
- none
- Challenged
- none
Gaps
- high14 of 14 AI agents are blockedAssistants asked about this page will guess, or cite someone else. Deliberate for a paywalled publisher. Otherwise this is the biggest gap on the list.
- lowNo llms.txtOptional, but a short curated index at /llms.txt is the cheapest way to tell assistants what matters on this site.
Every agent
| Agent | Type | robots.txt | Fetch | Words | Result |
|---|---|---|---|---|---|
| Human (Chrome)You | Human | allowed | 200357 ms | 620 | visible HTTP 200, 620 words |
| GooglebotGoogle · Classic search index. Also feeds AI Overviews. | Search engine | allowed | 200349 ms | 620 | visible HTTP 200, 620 words |
| Google-ExtendedGoogle · Robots-only token controlling Gemini training use. Never fetches on its own. | AI training | disallowedDisallow: / | n/a | blocked robots.txt Disallow: / (group: Google-Extended) | |
| GPTBotOpenAI · Training crawler for OpenAI models. | AI training | disallowedDisallow: / | 402933 ms | 29 | blocked robots.txt Disallow: / (group: GPTBot) |
| OAI-SearchBotOpenAI · Indexes for ChatGPT search results and citations. | AI search index | disallowedDisallow: / | 402930 ms | 29 | blocked robots.txt Disallow: / (group: OAI-SearchBot) |
| ChatGPT-UserOpenAI · Live fetch when a user asks ChatGPT about a page. | AI assistant (live fetch) | disallowedDisallow: / | 402941 ms | 29 | blocked robots.txt Disallow: / (group: ChatGPT-User) |
| ClaudeBotAnthropic · Training crawler for Claude models. | AI training | disallowedDisallow: / | 402933 ms | 29 | blocked robots.txt Disallow: / (group: ClaudeBot) |
| Claude-SearchBotAnthropic · Indexes for Claude web search. | AI search index | disallowedDisallow: / | 402941 ms | 29 | blocked robots.txt Disallow: / (group: Claude-SearchBot) |
| Claude-UserAnthropic · Live fetch when a user asks Claude about a page. | AI assistant (live fetch) | disallowedDisallow: / | 402947 ms | 29 | blocked robots.txt Disallow: / (group: Claude-User) |
| PerplexityBotPerplexity | AI search index | disallowedDisallow: / | 402721 ms | 29 | blocked robots.txt Disallow: / (group: PerplexityBot) |
| Perplexity-UserPerplexity | AI assistant (live fetch) | disallowedDisallow: / | 402955 ms | 29 | blocked robots.txt Disallow: / (group: Perplexity-User) |
| BingbotMicrosoft · Bing index. Also feeds Copilot and ChatGPT search. | Search engine | allowed | 200364 ms | 620 | visible HTTP 200, 620 words |
| Applebot-ExtendedApple · Robots-only token for Apple Intelligence training. | AI training | disallowedDisallow: / | n/a | blocked robots.txt Disallow: / (group: Applebot-Extended) | |
| Meta-ExternalAgentMeta | AI training | disallowedDisallow: / | 402943 ms | 29 | blocked robots.txt Disallow: / (group: Meta-ExternalAgent) |
| AmazonbotAmazon · Alexa and Amazon AI. | AI training | disallowedDisallow: / | 402946 ms | 29 | blocked robots.txt Disallow: / (group: Amazonbot) |
| BytespiderByteDance · Known for ignoring robots.txt. | AI training | disallowedDisallow: / | 402942 ms | 29 | blocked robots.txt Disallow: / (group: Bytespider) |
| CCBotCommon Crawl · Open dataset used to train most LLMs. | AI training | disallowedDisallow: / | 402313 ms | 29 | blocked robots.txt Disallow: / (group: CCBot) |
Snapshot
Measured 2026-09-25 09:33 UTC from a residential connection. Sites change their rules often, and sites that verify crawler IPs answer differently to different networks. Run a live comparison to see what usatoday.com does right now, from Cloudflare's network.