Crawler and bot policy

The principle

Blocking everything automated is the easy answer and the wrong one. A large and growing share of discovery happens through machine clients: search crawlers, AI-search crawlers, social preview unfurlers, monitoring, and accessibility tooling. Blocking them costs reach and buys almost no security, because the clients you actually want to stop lie about who they are.

So the policy has two independent halves. robots.txt states our preference to well-behaved crawlers, which is a courtesy protocol, not enforcement. The edge makes the real decision from behaviour and reputation, which is enforcement and does not care what the User-Agent header claims.

Allowed

Search engine crawlers: Googlebot, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider.

AI and AI-search crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Applebot-Extended, Amazonbot, Bytespider, Google-Extended, cohere-ai, Meta-ExternalAgent, DuckAssistBot, MistralAI-User.

Archival and research: CCBot (Common Crawl), ia_archiver and the Internet Archive, and academic research crawlers that identify themselves.

Social preview: facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Discordbot, TelegramBot, WhatsApp, Pinterest, Redditbot.

Operational: uptime and TLS monitoring, and accessibility evaluation tools.

Challenged

Unknown automated clients, clients whose request rate is well above human, clients whose TLS or header fingerprint does not match the browser they claim to be, and datacenter addresses behaving unlike a person. These get a managed challenge rather than a block, so a legitimate but unrecognised client can still get through.

Blocked

Vulnerability scanners and exploit probes, credential stuffing, enumeration of users, IDs or endpoints, aggressive scraping, request floods, malformed and oversized requests, and requests for paths that exist only in an attack toolkit — .env, .git, wp-login.php, phpMyAdmin, backup archives.

Blocked probes are answered with a plain 404, without revealing whether the path exists or what the stack is.

Why not User-Agent alone

A User-Agent header is a string the client chooses. Anyone can send Googlebot. Verifying a crawler means reverse DNS to the crawler operator domain and forward DNS back to the same address, or checking the published address ranges. The edge does that; robots.txt cannot.

The corollary matters too: because the header is not trusted for blocking, it is also not used for serving. The same HTML goes to every client.

Rate limits

Public API reads are limited per address per minute; public writes are limited harder and additionally protected by Turnstile. Limits return 429 with a Retry-After header rather than a silent drop, so a well-behaved client can back off correctly.

Sources

Where this page states a standard, an obligation or a research result, this is what it is drawn from. All external, all checkable.

1. RFC 9309 — Robots Exclusion Protocol — IETF — robots.txt, finally standardised
2. Verifying Googlebot and other Google crawlers — Google — reverse DNS verification, the reason we do not trust User-Agent
3. Google crawlers and user-triggered fetchers — Google
4. GPTBot — OpenAI
5. Anthropic crawlers — Anthropic
6. PerplexityBot — Perplexity
7. Applebot — Apple
8. Sitemaps XML format — sitemaps.org
9. Verified bots — Cloudflare — how verification actually works at the edge

Other documents