The wildcard rule would already admit all of them. They are named anyway, because some operators obey only a group that names them, and because a named group is a statement rather than an accident. The two lists below are read from the same module that builds the served robots.txt, so this page cannot claim an agent the file does not.
Bulk crawlers: indexing and training corpora
Fetching pages nobody is waiting on. These get the whole public reading surface: the marketing pages, the engine and template catalogue, the help centre, tutorials, blog, glossary, community and creator pages, and the machine-readable endpoints below.
- GPTBot
- OAI-SearchBot
- ClaudeBot
- Claude-Web
- Claude-SearchBot
- anthropic-ai
- PerplexityBot
- CCBot
- Google-Extended
- Applebot-Extended
- Bytespider
- Meta-ExternalAgent
- Amazonbot
- cohere-ai
- GrokBot
- xAI-Grok
- xAI-SearchBot
Two pages are held back from this group and only from this group: /editor and /studio. That is not a policy about training data. They are client-only workspaces that server-render almost no text, a measured 34 characters in /editor’s body at load, so a bulk fetch of them indexes an empty shell and spends crawl budget for nothing.
User-initiated assistants: someone asked for this page
An AI acting for a person who wants this page, right now. The crawl-budget argument above does not apply at all when there is no budget being spent, so this group additionally gets the two workspaces. Every engine, control and export in this product runs in the browser inside them, which makes an AI browser driving the real page the only way an AI can actually use this site rather than just describe it.
- ChatGPT-User
- Claude-User
- Perplexity-User
- DuckAssistBot
- MistralAI-User
- Meta-ExternalFetcher
- Grok-DeepSearch