Issue
robots.txt in the starter is still the unmodified Drupal scaffold, with no rules distinguishing AI crawlers by vendor or purpose. Each vendor runs independently-controlled bots for training, search-indexing, and live-retrieval — a blanket "AI bot" rule can't express "allow retrieval, block training," so every new client build currently inherits this gap and has to solve it from scratch.
To Do
- Ship an explicit, commented
robots.txt baseline naming each AI bot family separately (OpenAI's GPTBot/OAI-SearchBot/ChatGPT-User, Anthropic's ClaudeBot/Claude-SearchBot/Claude-User, Perplexity's PerplexityBot/Perplexity-User, Meta-ExternalAgent), defaulting to allow retrieval/index bots and allow training bots for original content
- Add a
Disallow: Google-Extended rule (free opt-out from Gemini/Vertex training, no effect on Search or AI Overviews)
- Document the path-scoped pattern for restricting training bots to a specific content type (e.g. licensed/rights-restricted collections), for projects that need it
- Add a CDN/WAF rule that verifies AI-bot traffic against each vendor's published IP range/reverse-DNS before honoring its user-agent string
- Write tests
Issue
robots.txtin the starter is still the unmodified Drupal scaffold, with no rules distinguishing AI crawlers by vendor or purpose. Each vendor runs independently-controlled bots for training, search-indexing, and live-retrieval — a blanket "AI bot" rule can't express "allow retrieval, block training," so every new client build currently inherits this gap and has to solve it from scratch.To Do
robots.txtbaseline naming each AI bot family separately (OpenAI'sGPTBot/OAI-SearchBot/ChatGPT-User, Anthropic'sClaudeBot/Claude-SearchBot/Claude-User, Perplexity'sPerplexityBot/Perplexity-User,Meta-ExternalAgent), defaulting to allow retrieval/index bots and allow training bots for original contentDisallow: Google-Extendedrule (free opt-out from Gemini/Vertex training, no effect on Search or AI Overviews)