# Two Webs

> Two Webs shows what humans get versus what AI crawlers and assistants get for any URL. It fetches the page live as a Chrome browser, Googlebot, Bingbot and twelve AI user agents, evaluates robots.txt (RFC 9309) and llms.txt, then scores visibility and lists gaps: blocked, challenged, degraded, cloaked, empty JavaScript shell, noindex, meta-refresh stub.

Use it when a page that looks fine in a browser is missing from AI answers or citations, or to check whether a site's crawler policy matches its intent.

## Pages

- [Home](https://twowebs.dev/): enter a URL to compare.
- [Examples](https://twowebs.dev/examples/): four real reports explained (nytimes.com 35, reddit.com 20, docs.astro.build 95, runsonlinux.com 100).
- [Method](https://twowebs.dev/about/): how robots.txt is evaluated, which 18 agents are checked, how the score is computed, and the tool's limits.

## API

- `GET https://twowebs.dev/api/compare/?url=<url>` returns the same report as JSON: per-agent robots verdict, live fetch status, word count, outcome and reason, plus a score and gap list. Results are cached ten minutes. Rate limit: ten comparisons per minute per IP.

## Notes

- Fetches originate from Cloudflare's network, not the crawlers' real IP ranges. Sites that verify crawlers by reverse DNS will challenge every agent, including the human one.
- The tool does not execute JavaScript. Neither do most AI crawlers.
- Source: https://github.com/AllyScan/twowebs