How 100 publishers treat AI crawlers, measured.

1. The median publisher is half closed

The median visibility score is 51 out of 100 and the average is 49. For comparison, the 100 most visited sites average 76. Publishers are the most guarded corner of the web we have measured, and by a wide margin.

26 of the 100 refuse every one of the fourteen AI tokens. Only 5 let all of them through: foxnews.com, scmp.com, engadget.com, salon.com, thedailybeast.com. Everyone else is somewhere in between, and the "in between" has a shape.

2. "Cite me, don't train on me" is the dominant posture

41 publishers block at least one training crawler while letting at least one search index or assistant fetcher through. That is the largest single group in the data. The split is sharpest inside the two biggest vendors: 15 sites refuse GPTBot yet admit OAI-SearchBot, and 15 refuse ClaudeBot yet admit Claude-SearchBot or Claude-User. Publishers have learned the vendors' token names and are using them to say two different things to the same company.

Examples: theguardian.com, cbsnews.com, abcnews.go.com, latimes.com, nypost.com, thetimes.com, independent.co.uk, mirror.co.uk, thesun.co.uk, aljazeera.com.

3. Which crawlers get shut out

Bytespider, ByteDance's crawler, is refused by 87 of 100, the most of any agent. CCBot, the Common Crawl bot whose archive trained most early LLMs, is next at 86. ClaudeBot is blocked by 80, GPTBot by 73, PerplexityBot by 73, Meta's crawler by 73. The assistant fetchers fare better: ChatGPT-User at 66, Claude-User at 62, Perplexity-User at 63.

The robots-only tokens are the ones most often forgotten. Google-Extended, which governs Gemini training, is blocked by 56, and Applebot-Extended by 53. 22 sites block GPTBot, ClaudeBot and CCBot but leave Google-Extended open, which almost certainly means nobody decided that; the group was written before the token existed.

4. Most blocks are a request, not a wall

A robots.txt rule asks a crawler to stay out. A 403 makes it. Of the publishers that block anything, 40 block only in robots.txt, 12 block only at the HTTP level, and 42 do both. On 41 sites, at least one disallowed training crawler still received the full page with a 200 when it asked, because nothing behind robots.txt enforces it. Bytespider, which has a reputation for ignoring robots.txt, would have found the page waiting on 19 of the sites that told it to leave.

Whether that matters depends on the crawler. The major vendors publish their tokens and honour the file. The ones that publishers worry about most do not.

5. 21 publishers are closed to anything that is not a real browser

nytimes.com, apnews.com, bloomberg.com, wsj.com, ft.com, economist.com, npr.org, telegraph.co.uk, japantimes.co.jp, theatlantic.com and 11 more refused even a plain request carrying a Chrome user agent from our connection, with a 403 or a challenge page. Their robots.txt files are often detailed, but the file never gets a chance to apply: a bot-management layer in front of the site decides on TLS fingerprint and behaviour before any user agent string is read. These sites score close to zero here, which is the honest result of the test, and a reminder that the test measures what a non-browser client receives.

37 of the 100 served a challenge page to at least one agent. Where the server named itself, Cloudflare and nginx front the most of them.

6. Googlebot is collateral

34 publishers blocked Googlebot in our fetch, and only 2 of those did it in robots.txt. The rest are HTTP refusals from bot management that cannot tell our request from an impostor, because Google verifies its crawler by IP range and we were not in it. From Google's real addresses these sites are almost certainly open. From anywhere else, including the AI vendors' data centres unless they are allow-listed, they are not. That is the second web in one sentence: the same URL, a different answer depending on who is asking and from where.

7. Almost nobody publishes an llms.txt

5 of 100: foxnews.com, timesofindia.indiatimes.com, hindustantimes.com, ndtv.com, si.com. Among the most visited sites the figure is 20 of 100. The file is a proposal, not a standard, and no crawler has been observed requesting it. Publishers are, reasonably, waiting.

13 publishers serve an app shell whose HTML holds almost no text until JavaScript runs, and 4 have no robots.txt at all, which means every crawler is welcome by default whether or not that was the intent.

Method and limits

The list was curated by hand: 100 publishers across news, magazines, broadcasting, reference, health, food, sport, technology and entertainment, weighted toward reach. Each home page was fetched with the published user agent of every crawler that makes requests, from a residential connection in Australia rather than from any vendor's network. robots.txt was evaluated with RFC 9309 precedence. Scores follow the same rules as a live comparison; the Method page has them. Sites that verify crawler IP ranges will answer real crawlers differently from how they answered us, and we say so on every page where it happened. The snapshot is dated 2026-09-25; every site page links to a live re-run.