AI Search Crawlers vs Training Bots: What to Allow in robots.txt

In our 18 September 2026 robots.txt readings of 64 well-known sites, 5 disallowed GPTBot, the training token, while leaving OAI-SearchBot, the search token, unblocked, and 5 disallowed both. Sites already treat the two purposes separately, so write each token’s rule from the purpose it serves, not from one allow-or-block switch.

Choose AI crawler rules by purpose. A search crawler helps a service discover sources; a training crawler collects material for model development; a user fetcher retrieves a page during a request. A policy that permits one purpose does not automatically permit the others, and allowing a bot does not ensure that an assistant will cite you.

In short

• Treat search discovery, training and user-requested fetching as three separate decisions.
• Write the policy per host and content area first; then choose the tokens that implement it.
• Test the policy, the delivery and the content separately, and keep the evidence.
• A permitted bot is not a citation, and a robots rule is not access control.

This guide reflects vendor documentation checked on 28 September 2026. Keep the policy owner and review date alongside your configuration because crawler names and controls can change.

Which names mean which purpose?

OpenAI describes search and training controls as independent. Its user-triggered fetcher is a separate category, for which robots rules may not apply. Anthropic documents the three Claude agents and says its bots honor robots directives. Perplexity distinguishes its search bot from user fetches; user-triggered requests generally ignore robots rules. Do not copy one vendor’s policy assumptions to another.

Google needs a separate explanation. Googlebot is a crawler, while Google-Extended is a robots control token without its own HTTP user-agent. Its documented scope includes certain Gemini training and grounding uses; it does not determine inclusion or ranking in Google Search. See Google’s current crawler documentation.

Write down the desired policy first

For each hostname and content area, record the purpose you permit, the owner who approved it and exceptions such as private pages or licensed archives. A publisher might allow public articles for search discovery while declining training. A help center might allow both. These are business choices; the bot table alone cannot make them.

A minimal illustrative OpenAI policy allowing search and declining training is:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Merge this with your existing file. More-specific groups can change which rules a bot uses, so inspect the entire file rather than appending a snippet without testing. Include needed path restrictions in each applicable group. Publish robots.txt at the root of each relevant host; a rule on the marketing host does not configure a separate documentation host.

Robots.txt expresses instructions to cooperating crawlers. It does not authenticate a requester or protect confidential information. Keep access controls on private content.

Check three layers independently

1. Policy: Does the applicable robots group permit the actual URL? Check redirects, case-sensitive paths and the final hostname.
2. Delivery: Does the server return the intended page, rather than a block or challenge? A permissive robots file cannot override a firewall decision.
3. Content: Is the useful answer present in the returned HTML? A successful status with an empty application shell is still a content problem.

Retain the rule, URL, date, status and response evidence. When inspecting traffic, verify identity using the vendor’s supported method and current address information where available. A request with a copied bot name is a probe, not proof that the vendor visited.

Separate access from citation outcomes

Use logs to establish verified visits, and saved answer observations to establish citations. A training bot request cannot stand in for a search citation; an allowed search bot does not establish source selection.

What the vendors’ own documents say, and where they differ

The details matter, because the same word, “block”, does different things at different vendors.

• OpenAI. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. A disallowed page can also appear as a bare link and title when OpenAI learns its URL from a third-party search provider or from links on other pages. To prevent that, the page needs a noindex robots meta tag, and OAI-SearchBot can only read it if robots.txt lets it crawl the page (the token catalogue, disallowed pages as links, both grade A). If a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes.
• Anthropic. All three of its bots honour robots.txt, including Crawl-delay, and do not bypass CAPTCHAs. Disallowing Claude-User stops Claude retrieving the page for a user (Claude’s fetch behaviour, grade A).
• Google. Google-Extended is a robots.txt token, not a crawler with its own user agent; it governs whether crawled content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Whether a site appears in AI Overviews and AI Mode is a different control: Search Console’s site-level Search generative AI setting (other vendors’ tokens, the Search Console control, both grade A). Newer user-triggered agents such as Google-Agent generally ignore robots.txt (Google’s non-search tokens, grade A).

Match the goal to the rule

These are options, not recommendations. Which row fits is a business decision.

What a robots rule cannot do

• Agents in a person’s browser. Agentic browsers that run in the user’s own browser present to websites as ordinary Chrome and carry no robots.txt token (agentic browsers, grade C). A robots rule cannot single them out.
• Declared identity is a claim. A request that says GPTBot may not be OpenAI. Vendors publish verification methods, and signed requests (Web Bot Auth) let a client prove who it is; the signature identifies the client, not the page (Web Bot Auth, grade A).
• Private content. Robots.txt asks cooperating crawlers to stay away. It does not authenticate anyone.

Common mistakes

• Pasting one vendor’s rules for every vendor.
• Adding a Disallow: / group for a training bot and discovering that a wildcard group above it already changed what other bots read.
• Putting robots.txt on the marketing host and assuming it covers the documentation host.
• Treating allowed GPTBot traffic as evidence of ChatGPT citations.
• Letting a CDN rule, installed later, override the robots intention without anyone noticing.

Questions people ask

Does blocking a training bot hide my site from AI search?

Not by that rule alone. OpenAI and Anthropic document search and training as separate controls. Check the delivery layer too: a firewall can refuse a search bot that robots.txt allows.

Does blocking Google-Extended affect my Google ranking?

Google documents it as a control for certain Gemini uses and not a ranking or inclusion signal in Search.

How often should I review this?

Whenever a vendor adds a bot, and on a set schedule. The token lists above changed within the last year; keep an owner and a review date next to the file.

Vercatus’s crawler-token reference and its plays, which include the access fixes that come first, help connect policy checks to readable content. For a refusal, follow the 403 diagnostic workflow. Once access is correct, assess question coverage and measure the same citation panel again.