GEO & AI Search

AI Crawler

An AI crawler is a bot operated by an AI company to collect or retrieve web content for training or for answering live queries, such as OpenAI's GPTBot, Google-Extended, and Anthropic's ClaudeBot.

Site owners can allow or block these in robots.txt, which controls whether their content can feed and be cited by AI systems. Blocking them protects content but can also remove a brand from AI answers. Deciding which to allow is now part of technical SEO strategy.