Top 100 / t.co
45
Invisible to machines
0 of 14 AI agents get the full page at https://t.co/. Checked 2026-09-25 09:19 UTC.
t.co scores 45 out of 100 for machine visibility, number 90 of the 100 most visited websites checked on 2026-09-25. None of the 14 AI agents receive the page. All 14 refusals are written in robots.txt, which is the transparent way to do it. There is no llms.txt.
youfull pagedegradedblockedunknown
Human
- Status
- 200
- Title
- t.co / X
- H1
- none
- Words
- 53
- Scripts
- 1 tags · 0 KB
- Meta robots
- none
- Canonical
- none
- Server
- cloudflare envoy
Machine
- robots.txt
- 3 groups · 0 sitemaps
- llms.txt
- missing
- llms-full.txt
- missing
- X-Robots-Tag
- none
- Blocked
- Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot
- Degraded
- none
- Challenged
- none
Gaps
- high14 of 14 AI agents are blockedAssistants asked about this page will guess, or cite someone else. Deliberate for a paywalled publisher. Otherwise this is the biggest gap on the list.
- lowVery little text in the raw HTML53 words. Thin for both humans and machines.
- lowNo llms.txtOptional, but a short curated index at /llms.txt is the cheapest way to tell assistants what matters on this site.
Every agent
| Agent | Type | robots.txt | Fetch | Words | Result |
|---|---|---|---|---|---|
| Human (Chrome)You | Human | disallowedDisallow: / | 200476 ms | 53 | visible HTTP 200, 53 words |
| GooglebotGoogle · Classic search index. Also feeds AI Overviews. | Search engine | allowedAllow: (empty) | 200479 ms | 53 | visible HTTP 200, 53 words |
| Google-ExtendedGoogle · Robots-only token controlling Gemini training use. Never fetches on its own. | AI training | disallowedDisallow: / | n/a | blocked robots.txt Disallow: / (group: *) | |
| GPTBotOpenAI · Training crawler for OpenAI models. | AI training | disallowedDisallow: / | 200509 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| OAI-SearchBotOpenAI · Indexes for ChatGPT search results and citations. | AI search index | disallowedDisallow: / | 200509 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| ChatGPT-UserOpenAI · Live fetch when a user asks ChatGPT about a page. | AI assistant (live fetch) | disallowedDisallow: / | 200508 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| ClaudeBotAnthropic · Training crawler for Claude models. | AI training | disallowedDisallow: / | 200480 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| Claude-SearchBotAnthropic · Indexes for Claude web search. | AI search index | disallowedDisallow: / | 200509 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| Claude-UserAnthropic · Live fetch when a user asks Claude about a page. | AI assistant (live fetch) | disallowedDisallow: / | 200524 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| PerplexityBotPerplexity | AI search index | disallowedDisallow: / | 200523 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| Perplexity-UserPerplexity | AI assistant (live fetch) | disallowedDisallow: / | 200519 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| BingbotMicrosoft · Bing index. Also feeds Copilot and ChatGPT search. | Search engine | allowedAllow: (empty) | 200520 ms | 53 | visible HTTP 200, 53 words |
| Applebot-ExtendedApple · Robots-only token for Apple Intelligence training. | AI training | disallowedDisallow: / | n/a | blocked robots.txt Disallow: / (group: *) | |
| Meta-ExternalAgentMeta | AI training | disallowedDisallow: / | 200520 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| AmazonbotAmazon · Alexa and Amazon AI. | AI training | disallowedDisallow: / | 200537 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| BytespiderByteDance · Known for ignoring robots.txt. | AI training | disallowedDisallow: / | 200507 ms | 53 | blocked robots.txt Disallow: / (group: *) |
| CCBotCommon Crawl · Open dataset used to train most LLMs. | AI training | disallowedDisallow: / | 200525 ms | 53 | blocked robots.txt Disallow: / (group: *) |
Snapshot
Measured 2026-09-25 09:19 UTC from a residential connection. Sites change their rules often, and sites that verify crawler IPs answer differently to different networks. Run a live comparison to see what t.co does right now, from Cloudflare's network.