Crawler tokens
Every token below is a User-agent value a site can name in robots.txt and see in its logs. They
are not interchangeable: a vendor’s training crawler, its search index crawler and its user-initiated
fetcher are separate programs with separate consequences. Vercatus fetches sampled pages as several of
them and reports each one’s treatment in the machine-access matrix.
Each section opens with the one sentence that answers “what is this bot”. Sources are the vendors’
own documentation; the knowledge base entries behind this page are access.ai-crawler-token-catalogue
and geo.search-vs-training-bots.
GPTBot
Section titled “GPTBot”GPTBot is OpenAI’s training crawler; it collects public pages to improve OpenAI’s foundation models,
and blocking it withholds the site from training without affecting ChatGPT search. Vendor: OpenAI.
Honours robots.txt. Published IP ranges at openai.com/gptbot.json. robots.txt line to block:
User-agent: GPTBot / Disallow: /. Allowing both GPTBot and OAI-SearchBot lets OpenAI reuse one
crawl for both purposes rather than fetching twice.
OAI-SearchBot
Section titled “OAI-SearchBot”OAI-SearchBot is OpenAI’s search crawler; it builds the index that ChatGPT search draws its results
and citations from, so blocking it removes the site from ChatGPT search while leaving training
unaffected. Vendor: OpenAI. Honours robots.txt; a change takes about 24 hours to take effect.
Published IP ranges at openai.com/searchbot.json. To be cited by ChatGPT, this is the token to allow.
ChatGPT-User
Section titled “ChatGPT-User”ChatGPT-User is OpenAI’s user-initiated fetcher; it retrieves a specific page when a person asks
ChatGPT about it, on that person’s behalf, and is not a crawler. Vendor: OpenAI. It does not build an
index and is not governed by robots.txt the way crawlers are, because the request is a user’s, so a
page blocked to GPTBot and OAI-SearchBot can still be read when someone pastes its address.
OAI-AdsBot
Section titled “OAI-AdsBot”OAI-AdsBot is OpenAI’s advertising crawler; it visits only pages submitted as ads in ChatGPT, to check
landing-page safety and relevance, and the data is not used to train models. Vendor: OpenAI.
Published IP ranges at openai.com/adsbot.json. Sites that do not advertise in ChatGPT never see it.
ClaudeBot
Section titled “ClaudeBot”ClaudeBot is Anthropic’s crawler for collecting public pages that may be used to train and improve
Claude; blocking it withholds the site from that collection. Vendor: Anthropic. Honours robots.txt.
It is distinct from Claude’s search and user fetchers below, so a site can block training and still
be cited.
anthropic-ai
Section titled “anthropic-ai”anthropic-ai is Anthropic’s earlier token, still named in many robots.txt files; ClaudeBot is the
crawler in use, and a rule for anthropic-ai alone does not cover it. Vendor: Anthropic. Honours
robots.txt.
Claude-SearchBot
Section titled “Claude-SearchBot”Claude-SearchBot is Anthropic’s search crawler; it indexes pages so Claude’s web search can find and
cite them, and blocking it removes the site from those results without any training effect. Vendor:
Anthropic. Honours robots.txt.
Claude-User
Section titled “Claude-User”Claude-User is Anthropic’s user-initiated fetcher; it retrieves a page when a person asks Claude to read that page, on the person’s behalf. Vendor: Anthropic. Like other user-initiated fetchers it acts on a specific request rather than crawling.
PerplexityBot
Section titled “PerplexityBot”PerplexityBot is Perplexity’s index crawler; it collects pages for the index Perplexity answers cite,
and blocking it removes the site from that index but not from pages a user asks Perplexity to open.
Vendor: Perplexity. Honours robots.txt. Perplexity favours recently updated, clearly dated pages for
time-sensitive questions, so a dated page competes better than an undated one.
Perplexity-User
Section titled “Perplexity-User”Perplexity-User is Perplexity’s user-initiated fetcher; it retrieves a specific page when a person asks about it and is not used to build the index. Vendor: Perplexity.
Googlebot
Section titled “Googlebot”Googlebot is Google’s crawler for Google Search; it fetches pages, renders them with a headless Chrome
and indexes what it finds, and it is the identity that decides classic search visibility. Vendor:
Google. Honours robots.txt. Genuine Googlebot is verified by reverse DNS to googlebot.com or
google.com and forward DNS back to the same address, or by Google’s published IP ranges; a request
with the Googlebot user agent from any other network is a spoof, and a site is right to refuse it.
Googlebot reads only the first 2 MB of a page.
Google-Extended
Section titled “Google-Extended”Google-Extended is a robots.txt control, not a separate crawler; it tells Google whether a site’s
content may be used to train Gemini models and for grounding, and blocking it does not change the
site’s presence in Google Search, AI Overviews or AI Mode. Vendor: Google. It has no user agent of
its own: Google’s ordinary crawlers fetch the page and consult the Google-Extended rule afterwards.
Inclusion in AI Overviews and AI Mode is governed separately, by the Search generative AI control in
Search Console.
Bingbot
Section titled “Bingbot”Bingbot is Microsoft’s crawler for Bing Search, and the index it builds also serves Microsoft
Copilot; blocking it removes the site from both. Vendor: Microsoft. Honours robots.txt. Bing gives
noarchive and nocache Copilot-specific meaning: noarchive keeps a page out of Copilot answers,
nocache limits Copilot to the page’s URL, title and snippet.
CCBot is Common Crawl’s crawler; it builds the open Common Crawl corpus, which many model builders
train on, so blocking it withholds the site from that corpus rather than from any one vendor.
Vendor: Common Crawl, a non-profit. Honours robots.txt.
BingPreview
Section titled “BingPreview”BingPreview is Microsoft’s page-preview fetcher; it renders pages to build the snapshots Bing shows
beside results, and is not the index crawler. Vendor: Microsoft. Honours robots.txt.
Applebot
Section titled “Applebot”Applebot is Apple’s crawler; it builds the index behind Siri and Spotlight suggestions, and the same
crawl feeds Apple’s models unless Applebot-Extended is disallowed. Vendor: Apple. Honours
robots.txt. Blocking Applebot itself removes the site from Siri and Spotlight; to keep those and
refuse training, block the control token below instead.
Applebot-Extended
Section titled “Applebot-Extended”Applebot-Extended is Apple’s control token; it never fetches a page, and disallowing it keeps pages Applebot crawled out of Apple’s foundation model training without removing them from Siri and Spotlight. Vendor: Apple. The same pattern as Google-Extended.
Meta-ExternalAgent
Section titled “Meta-ExternalAgent”Meta-ExternalAgent is Meta’s training crawler; it collects pages to train Meta’s AI models and to
improve its products, so blocking it withholds the site from Meta’s training without touching Meta
AI’s search. Vendor: Meta. Honours robots.txt. Older robots.txt files spell it meta-externalagent;
the match is case-insensitive.
Meta-ExternalFetcher
Section titled “Meta-ExternalFetcher”Meta-ExternalFetcher is Meta’s user-initiated fetcher; it retrieves a page a person shared or asked Meta AI about, on that person’s behalf, and is not a crawler. Vendor: Meta.
Meta-WebIndexer
Section titled “Meta-WebIndexer”Meta-WebIndexer is Meta’s search index crawler; it builds the index Meta AI’s search answers draw on,
so blocking it removes the site from those answers. Vendor: Meta. Honours robots.txt. This is the
Meta token to allow to be cited by Meta AI.
GoogleOther
Section titled “GoogleOther”GoogleOther is Google’s crawler for research and development crawls by teams other than Search; it
does not feed the search index, so blocking it changes nothing in Google Search or AI Overviews.
Vendor: Google. Honours robots.txt.
GoogleOther-Image
Section titled “GoogleOther-Image”GoogleOther-Image is the image-fetching form of GoogleOther, for research crawls that need image
URLs. Vendor: Google. Honours robots.txt.
GoogleOther-Video
Section titled “GoogleOther-Video”GoogleOther-Video is the video-fetching form of GoogleOther. Vendor: Google. Honours robots.txt.
Amazonbot
Section titled “Amazonbot”Amazonbot is Amazon’s crawler; it collects pages to improve Alexa’s answers and Amazon’s other
services, so blocking it removes the site from what Alexa can answer from the web. Vendor: Amazon.
Honours robots.txt.
Bytespider
Section titled “Bytespider”Bytespider is ByteDance’s crawler, associated with training its models and feeding its products.
Vendor: ByteDance, which publishes no page about it; site owners widely report it ignoring
robots.txt, so the only reliable control is a WAF or bot-management rule on the token or its
published address ranges.
cohere-ai
Section titled “cohere-ai”cohere-ai is the token Cohere’s crawling has been seen under. Vendor: Cohere, which publishes no
crawler page a site owner can rely on; treat robots.txt compliance as unstated.
Diffbot
Section titled “Diffbot”Diffbot is Diffbot’s crawler; it extracts pages into a commercial knowledge graph and datasets that
its customers, including model builders, buy, so blocking it withholds the site from that resale
rather than from any one engine. Vendor: Diffbot. Compliance with robots.txt is unstated.
omgili
Section titled “omgili”omgili is Webz.io’s crawler (the name is older than the company’s); it collects pages into datasets
sold to others, including for training. Also seen as omgilibot. Vendor: Webz.io. Compliance with
robots.txt is unstated.
YouBot
Section titled “YouBot”YouBot is You.com’s crawler; it builds the index You.com’s AI search answers draw on, so blocking it
removes the site from those answers. Vendor: You.com. Compliance with robots.txt is unstated.
DuckAssistBot
Section titled “DuckAssistBot”DuckAssistBot is DuckDuckGo’s fetcher for DuckAssist answers; it retrieves pages to summarise beside
the search results. Vendor: DuckDuckGo. Honours robots.txt.
MistralAI-User
Section titled “MistralAI-User”MistralAI-User is Mistral’s user-initiated fetcher; it retrieves a page a person asked Le Chat about, on that person’s behalf, and is not a crawler. Vendor: Mistral.
How to decide
Section titled “How to decide”Allow the search and answer indexes of the engines you want to be found and cited by (Googlebot,
Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amazonbot).
Decide separately whether to allow the training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent,
Bytespider) and the control tokens (Google-Extended, Applebot-Extended), and whether to allow the
data brokers (CCBot, Diffbot, omgili); blocking those costs no citations. User-initiated fetchers act
for a person and are rarely worth blocking. State the choice per token in robots.txt, and publish
the same preference as a Content-Signal line so it reads the same to every crawler. Every full run
reads the site’s robots.txt against this whole list and reports the policy per category on the run’s
Access tab; the daily access watch requests pages as Applebot, Meta-ExternalAgent, Bytespider and
CCBot beside the five AI crawlers it already sends. Allow-list the crawler
shows how sites behind bot management do this without opening the door to spoofs.