Crawler tokens used by other AI vendors

Beyond OpenAI and Anthropic, the vendors' own pages name distinct tokens per purpose. Google: Google-Extended is a robots.txt product token with no user agent of its own (crawling uses existing Google user agents) that governs whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI; GoogleOther, GoogleOther-Image and GoogleOther-Video are generic crawlers whose rules affect no specific product. Apple: Applebot powers search in Spotlight, Siri and Safari, and Applebot-Extended does not crawl but, when disallowed, keeps content out of Apple's foundation-model training while pages can still appear in search results. Meta: Meta-ExternalAgent crawls for uses such as training foundation models, Meta-WebIndexer improves Meta AI search and lets it cite and link content, Meta-ExternalAds crawls for advertising and other business products, Meta-ExternalFetcher fetches links at a user's request and may bypass robots.txt, and FacebookExternalHit fetches shared links to build previews (title, description, thumbnail) and may bypass robots.txt when performing security or integrity checks. Microsoft: bingbot is Bing's standard crawler, AdIdxBot crawls ad landing pages and BingPreview/MicrosoftPreview generate page snapshots. Amazon: Amazonbot's content may be used to train Amazon AI models, while Amzn-SearchBot (search) and Amzn-User (live user queries such as Alexa) do not crawl for model training. Perplexity: PerplexityBot surfaces sites in Perplexity search and is not used for model training; Perplexity-User generally ignores robots.txt because a user asked for the page. Common Crawl: CCBot builds an open repository of web crawl data and publishes its IP ranges.