AI Crawler 403 Errors on Cloudflare: Diagnose the Rule Before Changing It

A 403 response says that a request was refused. It does not identify the rule, prove that a real AI vendor is blocked, or tell you whether your robots.txt policy is responsible. Start with the exact request and the matching security event before changing your Cloudflare configuration.

In short

• A 403 tells you a request was refused, not which rule refused it or whether the requester was genuine.
• Reproduce the refusal, read the matching security event, and only then change a rule.
• Make the smallest change that fits your policy, scoped to a verified identity and public paths.
• Retest the content, not just the status code, and keep a rollback.

This is a diagnostic guide, not a report from an inspected customer zone. The worksheet below is illustrative; we have not supplied invented security events or a measured fix.

Diagnose a Cloudflare 403 for an AI crawler in five steps

Checked against the vendors’ own pages on 12 October 2026. Replace example.com/guides/setup/ with a public page on your site.

1. Reproduce the refusal once per crawler identity, plus a plain control. Send the same URL with each user agent and keep the status line and the headers. The strings below are the ones OpenAI, Anthropic, Perplexity and Google publish for their crawlers, and the ones a Vercatus access probe sends; check each vendor’s page for the current version number.

   URL=https://example.com/guides/setup/
   curl -sS -o /dev/null -D - "$URL"                                  # control: no crawler identity
   curl -sS -o /dev/null -D - -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" "$URL"
   curl -sS -o /dev/null -D - -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" "$URL"
   curl -sS -o /dev/null -D - -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "$URL"
   curl -sS -o /dev/null -D - -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" "$URL"
   curl -sS -o /dev/null -D - -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "$URL"

   Read three things in each response: the status code, the cf-ray header (it identifies the request in Cloudflare’s logs), and whether a cf-mitigated: challenge header is present. Cloudflare documents that header as set to challenge on a challenge page response (detecting a challenge response). A 200 whose body is a challenge page is a refusal too.

2. Separate robots.txt from an HTTP refusal. Fetch https://example.com/robots.txt and read the group that applies to the crawler’s token (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot). A disallow there is a policy that a well-behaved crawler obeys by not asking; it never returns a 403. A 403 or a challenge on a URL that robots.txt allows comes from the edge or the origin.

3. Match the event to the layer that issued it. In the Cloudflare dashboard open the zone’s Analytics page and its Events tab, filter to the time of your test, and expand the event (Cloudflare: security events). The expanded event shows the action taken and the security feature that applied it, such as managed rules or rate limiting rules; where the feature records a rule, its identifier is in the same details, and that identifier, with the feature and the action, is what to write down. Cloudflare’s page does not name every field, so record what your account shows. If there is no event for that minute, the refusal may be rate limiting, the origin or an application rule, so read the origin logs next.

4. Verify who is really asking before you allow anything. A user-agent string is self-declared, so a 403 to a copy of GPTBot sent from your laptop can be correct. Check a request from your own logs against the vendor’s published method:

   # OpenAI publishes one IP list per crawler (searchbot.json, gptbot.json, chatgpt-user.json)
   curl -s https://openai.com/searchbot.json | jq -r '.prefixes[].ipv4Prefix' | head
   curl -s https://www.perplexity.com/perplexitybot.json | jq -r '.prefixes[].ipv4Prefix' | head
   # Google: reverse DNS, then forward DNS on the name it returns (googlebot.com, google.com or googleusercontent.com)
   host 66.249.66.1                          # crawl-66-249-66-1.googlebot.com
   host crawl-66-249-66-1.googlebot.com      # must return the same address

   Google also publishes JSON lists of its address ranges (verify requests from Google crawlers), and Bing offers a verification tool (which crawlers does Bing use). An address that is not on the vendor’s list is not that vendor, whatever the header says (the evidence, grade A).

5. Change the narrowest control, then repeat step 1. Scope any exception to the verified identity and the public paths it should read, keep the previous configuration for a rollback, and rerun the curl commands from step 1. Then check that the body holds the page and not a challenge.

Symptom in step 1	Likely cause	What to do
Every identity and the control get the same 403	A rule that is not about bots: a geographic rule, an origin authorisation rule or a redirect	Read the origin logs and the event’s feature; fix that rule
Only crawler identities get 403, the control gets 200	A bot rule or a bot protection setting	Match the event, then allow the verified identity on public paths only
200 with cf-mitigated: challenge	A challenge, which a crawler that does not run JavaScript cannot pass	Exempt the verified crawler from the challenge on public paths (JavaScript challenges, grade A)
429, or a 403 that comes and goes	Rate limiting, or origin load	Read the rate rule and the origin’s response times
200 from your laptop, 403 to the real vendor	The vendor’s addresses are treated differently from your copied user agent	Test from the logs of real vendor requests, not from a copied string
200, but the answer is missing from the body	Not an access problem: a shell that needs JavaScript	See JavaScript and AI crawlers

Capture the refusal you can reproduce

Record the hostname, path, timestamp with timezone, response status, response body and request identifier when present. Note whether the client followed redirects. Compare a public article and the home page because they may use different delivery rules.

Observation	What to inspect next
Robots disallows the URL	The applicable crawler policy; this is different from an HTTP refusal
403 with a matching edge event	Event action, product and rule identifier
Challenge page instead of the article	The challenge-generating product and whether the legitimate client can satisfy it
429 or intermittent refusals	Rate-limit rule, request burst and retry behavior
No matching edge event	Origin logs, application authorization, geographic rules and redirects
200 with an application shell	Readable-content and rendering checks

A missing event is not proof that the edge played no part. Check the time window, retained logs and sampling before drawing that conclusion.

Establish who made the request

The user-agent header is self-declared. Someone can send GPTBot from an unrelated machine. Compare verified bot signals and the vendor’s published verification method, rather than treating that string as authentication. Cloudflare explains how verified bots are identified, and OpenAI publishes crawler details.

A Vercatus access probe comes from our network. It can show what your site served to that probe under a crawler-like identity. It cannot prove that a verified vendor’s own addresses receive the same treatment. Keep those evidence types separate in a support ticket.

Match the event to the responsible control

In the authorized zone, inspect security events around the timestamp. Record the action and rule identifier before deciding whether the refusal conflicts with your intended policy. Consider custom rules, managed protections, AI bot controls, rate limiting and origin restrictions separately.

Avoid advice that assumes all products have the same exception mechanism. Cloudflare’s Bot Fight Mode documentation states that its behavior cannot be bypassed using WAF custom-rule skip actions. A rule designed for a different product may therefore have no effect. Have the zone owner select the supported control for the identified product.

The phrase “Cloudflare blocks AI bots by default” is too broad to diagnose your site. Onboarding choices, account configuration, product and rule history matter. Inspect the actual zone settings.

Make the smallest policy-consistent change

If a verified search crawler should read public editorial pages, scope any exception to the identity, public paths and responsible protection that need it. Do not extend access to authenticated content or trust every request carrying a familiar bot name. Save the previous configuration, change ticket and rollback condition.

If the refusal implements your intended training policy, preserve it. Fix a mislabeled measurement rather than treating every denied AI request as an SEO defect.

Retest delivery and then observe real traffic

Repeat the original probe and inspect the returned content, not only the status. Confirm that the requested page is present and essential passages exist before JavaScript runs. Check that unrelated protections still behave as intended.

Then monitor verified vendor traffic. Record whether the relevant real requests return useful content over a reasonable observation window. Policy propagation and recrawling are separate from an immediate local retest; neither establishes an answer citation.

Why a refusal can come from several places

Cloudflare’s own documentation and the sources behind this site describe several layers that can say no to a crawler. Each needs a different fix.

• A managed challenge. Cloudflare’s interstitial challenge pages run JavaScript in the visitor’s browser, and Cloudflare lists command-line tools without JavaScript execution and automated browsers as unsupported. A crawler that is not exempted receives the challenge page instead of the content (JavaScript challenges, grade A). For Google, content from a 4xx response is not used or indexed, and a 2xx page that shows an error is reported as a soft 404.
• AI Crawl Control. Cloudflare’s AI Crawl Control shows which AI crawlers request a zone’s content, lets owners allow or block per crawler and tracks robots.txt compliance (AI Crawl Control, grade A). It is the quickest way to reconcile a refusal with what you intended.
• A default you did not set. New domains are asked at signup whether to allow AI crawlers, and since September 2026 new domains block training and agent traffic by default on pages that display ads (the default, grade A).
• Rate limiting or the origin. A 429, or an intermittent 403, can be a rate rule or an origin authorisation rule rather than a bot rule. Slow or erroring responses also lower how fast Google crawls (crawl rate, grade A).

A caution about undeclared crawlers

In August 2025 Cloudflare reported that on test domains that disallowed Perplexity in robots.txt and blocked its declared bots at the WAF, content was still retrieved by an undeclared crawler using a generic browser user agent, and in the same test ChatGPT-User fetched robots.txt and stopped (the dispute, grade C, one party’s measurement). The lesson is not about any one vendor. It is that a declared name is a claim, which is why verification comes before an exception.

A worked example, with invented values

A site owner sees 403 for a bot named “SearchBot-X” on /guides/setup/. The worksheet reads: timestamp noted, event found, action “managed challenge”, rule from a bot-protection product, requester unverified. The decision: do not add an exception for the name alone. If the vendor publishes a verification method and the owner wants the traffic, scope an allowance to the verified identity and to public guide paths; otherwise leave the challenge in place. Everything in this example is illustrative.

Common mistakes

• Opening the whole zone because one probe returned 403.
• Using a rule type that does not apply to the product that issued the refusal.
• Confusing a robots disallow with an HTTP refusal.
• Testing only the home page.
• Calling the job done when the status is 200 but the body is a challenge page.

Questions people ask

Does Cloudflare block all AI bots by default?

Not on every site. It depends on when and how the zone was set up, and on the product and rule history. Inspect your own zone.

Will fixing the 403 get me cited?

It removes a barrier. Citation depends on whether the content answers the question.

Should I allow every bot that asks nicely?

No. Allow identities you have verified and want, on the paths you want them to read.

Use the crawler traffic guide for observation sources and the 64-site access study for the limits of external probes. Once delivery is correct, the next question is whether your content answers the user’s task, followed by a repeatable citation check.