Mostly one web

11 of 14 AI agents get the full page at https://fortune.com/. Checked 2026-09-25 09:33 UTC.

fortune.com scores 85 out of 100 for machine visibility, number 15 of the 100 content publishers checked on 2026-09-25. 11 of 14 AI agents receive the full page: 5 of 8 training crawlers, 3 of 3 AI search indexes and 3 of 3 assistant fetchers. 2 are refused in robots.txt and 1 is refused at the HTTP level. The pattern is "cite me, don't train on me": 3 of 8 training crawlers are refused while 6 of 6 search and assistant fetchers get through. There is no llms.txt.

youfull pagedegradedblockedunknown

Human

Status: 200
Title: Fortune - Fortune 500 Daily & Breaking Business News
H1: none
Words: 2,361
Scripts: 71 tags · 337 KB
Meta robots: index
Canonical: https://fortune.com/
Server: nginx

Machine

robots.txt: 7 groups · 10 sitemaps
llms.txt: missing
llms-full.txt: missing
X-Robots-Tag: none
Blocked: Google-Extended, Bytespider, CCBot
Degraded: none
Challenged: none

Gaps

medium
3 of 14 AI agents are blockedGoogle-Extended: robots.txt Disallow: / (group: Google-Extended) · Bytespider: HTTP 403 for this user agent · CCBot: robots.txt Disallow: / (group: CCBot)
info
Training blocked, citation allowedTraining crawlers are refused while search and assistant fetches get through. This is the common "cite me, don't train on me" posture.
low
No llms.txtOptional, but a short curated index at /llms.txt is the cheapest way to tell assistants what matters on this site.

Every agent

Agent	Type	robots.txt	Fetch	Words	Result
Human (Chrome)You	Human	allowed	200201 ms	2,361	visible HTTP 200, 2361 words
GooglebotGoogle · Classic search index. Also feeds AI Overviews.	Search engine	allowed	200229 ms	2,361	visible HTTP 200, 2361 words
Google-ExtendedGoogle · Robots-only token controlling Gemini training use. Never fetches on its own.	AI training	disallowedDisallow: /	n/a		blocked robots.txt Disallow: / (group: Google-Extended)
GPTBotOpenAI · Training crawler for OpenAI models.	AI training	allowed	200201 ms	2,361	visible HTTP 200, 2361 words
OAI-SearchBotOpenAI · Indexes for ChatGPT search results and citations.	AI search index	allowed	200200 ms	2,361	visible HTTP 200, 2361 words
ChatGPT-UserOpenAI · Live fetch when a user asks ChatGPT about a page.	AI assistant (live fetch)	allowed	200142 ms	2,361	visible HTTP 200, 2361 words
ClaudeBotAnthropic · Training crawler for Claude models.	AI training	allowed	200202 ms	2,361	visible HTTP 200, 2361 words
Claude-SearchBotAnthropic · Indexes for Claude web search.	AI search index	allowed	200173 ms	2,361	visible HTTP 200, 2361 words
Claude-UserAnthropic · Live fetch when a user asks Claude about a page.	AI assistant (live fetch)	allowed	200173 ms	2,361	visible HTTP 200, 2361 words
PerplexityBotPerplexity	AI search index	allowed	200162 ms	2,361	visible HTTP 200, 2361 words
Perplexity-UserPerplexity	AI assistant (live fetch)	allowed	200155 ms	2,361	visible HTTP 200, 2361 words
BingbotMicrosoft · Bing index. Also feeds Copilot and ChatGPT search.	Search engine	allowed	200221 ms	2,361	visible HTTP 200, 2361 words
Applebot-ExtendedApple · Robots-only token for Apple Intelligence training.	AI training	allowed	n/a		visible Robots-only token; allowed by robots.txt
Meta-ExternalAgentMeta	AI training	allowed	200154 ms	2,361	visible HTTP 200, 2361 words
AmazonbotAmazon · Alexa and Amazon AI.	AI training	allowed	200234 ms	2,361	visible HTTP 200, 2361 words
BytespiderByteDance · Known for ignoring robots.txt.	AI training	allowed	403113 ms	82	blocked HTTP 403 for this user agent
CCBotCommon Crawl · Open dataset used to train most LLMs.	AI training	disallowedDisallow: /	200207 ms	2,361	blocked robots.txt Disallow: / (group: CCBot)

Snapshot

Measured 2026-09-25 09:33 UTC from a residential connection. Sites change their rules often, and sites that verify crawler IPs answer differently to different networks. Run a live comparison to see what fortune.com does right now, from Cloudflare's network.

← tomshardware.com (85)All 100 publishers bleacherreport.com (85) →