We read your robots.txt and check 15 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more). If it blocks a crawler AI answers rely on, we hand you back a corrected file you can paste.
AI engines respect robots.txt. If OAI-SearchBot or PerplexityBot is blocked, ChatGPT search and Perplexity cannot fetch your pages to cite them, even if everything else is in order.
A copied "block AI bots" list can name the answer crawlers too, so a site that meant to refuse training data also stops AI answers fetching its pages. A "Disallow: /" left over from a staging site does the same.
We read your live robots.txt and check it against 15 AI crawlers. If an answer crawler is blocked, the corrected file keeps your existing rules, lets it in, and leaves your choices about training crawlers alone.
The list covers training crawlers (used to build models) and answer crawlers (used when someone asks an assistant a question). Blocking an answer crawler stops that assistant fetching your pages for its answers. Turning away a training crawler is a choice, and we do not treat it as a problem.
GPTBotTraining dataOAI-SearchBotChatGPT browse and searchChatGPT-UserPer-user fetches from ChatGPTGoogle-ExtendedGemini training and Gemini app answersClaudeBotClaude trainingClaude-SearchBotClaude search answersClaude-UserPer-user fetches from ClaudePerplexityBotIndex for Perplexity answersPerplexity-UserPer-user fetches from PerplexityApplebot-ExtendedApple Intelligence trainingcohere-aiCohere, no crawler documentedBytespiderTikTok and DoubaoCCBotTraining data for many LLMsAmazonbotAmazon AI trainingMeta-ExternalAgentLlama training