Vercatus
Sign inGet started
Get started

Who lets AI assistants read them

Machine access on 64 well-known websites across eight niches. Can the crawlers behind ChatGPT, Claude and Perplexity fetch the pages, what does robots.txt tell them, and how much is there to read without JavaScript?

Read the resultsHow it was measured
  • 64 sites
  • 8 niches
  • 2 pages each
  • 6 identities
  • public pages only

The headline · 18 September 2026

33 of 64

sites served their home page to all three named crawler probes from our network.

  • All three served 33
  • Some 4
  • None 27

Each cell is a dated response to our probe, not a verified vendor visit or an observed citation.

The study
in five figures.

Of 64 sites measured on 18 September 2026, from Vercatus's network. Measured, not endorsed.

33
served all three crawler probes their home page
27
refuse or challenge all three: a 403, 401 or bot interstitial
4
let some AI crawlers in and not others
17
publish an llms.txt
9
keep over half the deep page behind JavaScript

Who lets them in,
by niche.

Eight names in each niche, chosen for recognition, not for size: sites a general reader has used. Each niche has its own reason the question matters.

Home pages served to GPTBot, ClaudeBot and PerplexityBot

  • All three served
  • Some
  • None
64 sites, public pages, from our network, 18 September 2026. Measured, not endorsed.

Site by site,
every response.

The home page and one deep page, requested with each identity from Vercatus's network. The robots column is what robots.txt says to the AI training and search tokens, whatever the server then does.

  • serveda 200 with the page's content
  • refuseda 401, 403 or 451
  • challengea bot-management interstitial instead of the page

Swipe a table sideways for the other crawlers; the site name stays in view.

News and media

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
BBCwww.bbc.comservedservedservedservedservedservedGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extendedno8% of words
The New York Timeswww.nytimes.comchallengechallengerefused 403refused 403refused 403refused 403GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extendedno0% of words
The Guardianwww.theguardian.comservedservedservedservedrefused 403refused 403ClaudeBot, PerplexityBot, CCBot, Applebot-Extendedno0% of words
Reuterswww.reuters.comchallengechallengechallengechallengechallengechallengeApplebot-Extendedno0% of words
Bloombergwww.bloomberg.comchallengedeep servedchallengedeep servedchallengedeep servedchallengedeep servedchallengedeep servedchallengedeep servedGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extendedno0% of words
Der Spiegelwww.spiegel.deservedserved470served470servedGPTBot, OAI-SearchBot, ClaudeBot, CCBot, Applebot-Extendedno0% of words
Le Mondewww.lemonde.frservedservedserveddeep 402servedservedservedClaudeBot, Google-Extended, CCBot, Applebot-Extendedno33% of words
El Paíselpais.comserveddeep challengeservedserveddeep challengeservedservedserveddeep challengeClaudeBot, PerplexityBot, CCBotno17% of words

E-commerce and retail

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Amazonwww.amazon.comservedservedservedservedservedservedGPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBotno–
IKEAwww.ikea.comservedservedservedservedservedservedno AI token disallowedno–
Zalandowww.zalando.comrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words
Walmartwww.walmart.comchallengedeep 200challengechallengechallengechallengechallengeno AI token disallowedno0% of words
Etsywww.etsy.comchallenge429challengechallengedeep servedchallengechallengeno AI token disallowedno0% of words
eBaywww.ebay.comrefused 403deep 418no replyno replyno replyno replyno replyGPTBot, ClaudeBot, PerplexityBot, CCBot, Applebot-Extendedno99% of words
Decathlonwww.decathlon.comservedservedservedservedservedservedno AI token disallowedyes0% of words
H&Mwww.hm.comrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words

Software and SaaS

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Notionwww.notion.comservedchallengeservedservedservedservedno AI token disallowedyes25% of words
Slackslack.comservedservedservedservedservedservedno AI token disallowedyes100% of words
Atlassianwww.atlassian.comservedservedservedservedservedservedno AI token disallowedyes30% of words
HubSpotwww.hubspot.comservedservedservedservedservedservedno AI token disallowedyes58% of words
Shopifywww.shopify.comservedservedservedservedservedservedno AI token disallowedyes0% of words
Salesforcewww.salesforce.comservedrefused 403refused 403refused 403refused 403refused 403no AI token disallowedyes20% of words
Figmawww.figma.comservedservedservedservedservedservedGPTBot, ClaudeBot, Google-Extended, CCBotno4% of words
Zoomzoom.usservedservedservedservedservedservedno AI token disallowedno85% of words

Travel and hospitality

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Booking.comwww.booking.comchallengedeep servedchallengedeep servedservedchallengedeep servedchallengedeep servedchallengedeep servedno AI token disallowedno14% of words
Airbnbwww.airbnb.comservedrefused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words
Expediawww.expedia.com429deep served429deep served429deep served429deep served429deep served429deep servedno AI token disallowedyes–
Ryanairwww.ryanair.comservedservedservedservedservedservedno AI token disallowedno100% of words
Lufthansawww.lufthansa.comchallengechallengechallengechallengechallengechallengeno AI token disallowedyes16% of words
Marriottwww.marriott.comrefused 403deep servedrefused 403refused 403refused 403refused 403refused 403no AI token disallowedno12% of words
Tripadvisorwww.tripadvisor.comchallengechallengechallengeservedchallengechallengeGPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extendedno0% of words
trivagowww.trivago.comrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno9% of words

Banking and payments

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Revolutwww.revolut.comchallengechallengechallengechallengechallengechallengeno AI token disallowedno14% of words
Wisewise.comservedservedservedservedservedservedno AI token disallowedno–
PayPalwww.paypal.comservedservedservedservedservedservedno AI token disallowedyes11% of words
Stripestripe.comservedservedservedservedservedservedno AI token disallowedyes1% of words
HSBCwww.hsbc.comservedservedservedservedservedservedno AI token disallowedno18% of words
INGwww.ing.comservedservedservedservedservedservedno AI token disallowedno–
Deutsche Bankwww.db.comservedservedservedservedservedservedno AI token disallowedyes41% of words
Coinbasewww.coinbase.comchallengechallengechallengechallengechallengechallengeno AI token disallowedyes17% of words

Consumer brands and automotive

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Applewww.apple.comserveddeep refused 403servedservedservedservedservedno AI token disallowedno68% of words
Samsungwww.samsung.comservedrefused 403servedservedservedservedno AI token disallowedyes0% of words
Nikewww.nike.comservedrefused 403deep servedrefused 403deep servedrefused 403deep servedrefused 403deep servedrefused 403deep servedno AI token disallowedno0% of words
adidaswww.adidas.comrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words
BMWwww.bmw.comservedno replyno replyno replyno replyno replyno AI token disallowedno0% of words
Teslawww.tesla.comrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words
Volkswagenwww.volkswagen.comservedservedservedservedservedservedno AI token disallowedno21% of words
Dysonwww.dyson.comchallengechallengechallengechallengechallengechallengeno AI token disallowedyes18% of words

Health

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Mayo Clinicwww.mayoclinic.orgrefused 403refused 403refused 403refused 403refused 403refused 403no AI token disallowedno0% of words
WebMDwww.webmd.comserveddeep 404challengeserveddeep 404serveddeep 404serveddeep 404serveddeep 404GPTBot, ClaudeBot, CCBotno15% of words
NHSwww.nhs.ukservedservedservedservedservedservedno AI token disallowedno13% of words
Cleveland Clinicmy.clevelandclinic.orgservedservedservedservedservedservedno AI token disallowedyes0% of words
Healthlinewww.healthline.comservedservedservedservedservedservedGPTBot, ClaudeBot, CCBot, Applebot-Extendedno35% of words
World Health Organizationwww.who.intservedservedservedservedservedservedCCBotno–
Philipswww.philips.comservedservedservedservedservedservedno AI token disallowedno7% of words
Bayerwww.bayer.comrefused 403serveddeep 404refused 403refused 403refused 403refused 403no AI token disallowedno0% of words

Education and reference

SitePlain HTTP clientGooglebotGPTBotOAI-SearchBotClaudeBotPerplexityBotrobots.txt, AI tokensllms.txtNeeds JavaScript
Wikipediaen.wikipedia.orgserveddeep 404refused 403serveddeep 404serveddeep 404refused 403serveddeep 404no AI token disallowedno0% of words
Courserawww.coursera.orgservedservedservedservedservedservedno AI token disallowedyes56% of words
Khan Academywww.khanacademy.orgservedservedservedservedservedservedno AI token disallowedno–
Duolingowww.duolingo.comservedservedservedservedservedservedno AI token disallowedno100% of words
Britannicawww.britannica.comchallengechallengerefused 403challengerefused 403refused 403no AI token disallowedno32% of words
MIT OpenCourseWareocw.mit.eduservedservedservedservedservedservedno AI token disallowedno0% of words
Udemywww.udemy.comchallengechallengechallengechallengechallengechallengeno AI token disallowedno59% of words
Stack Overflowstackoverflow.comchallengechallengerefused 403deep challengechallengerefused 403deep challengerefused 403deep challengeno AI token disallowedno–

What was measured,
and what was not.

One day, two pages, one network: a snapshot, not a ranking. Read the methodology

Pages. Two per site: the home page and one deep page chosen from the site's own sitemap or navigation (an article, a product, a help page). Both are public and linked from the site.

Identities. Each page was requested 6 ways: as a plain HTTP client with no browser, and with the published user-agent strings of Googlebot, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Requests came from Vercatus's own network and carried Vercatus's signed identity, so a site that verifies crawler IP ranges will see a non-Google, non-OpenAI client presenting that string. A refusal establishes what our probe received from this origin. It does not establish what a verified vendor request receives; address verification and other security conditions can change the response.

Purposes. GPTBot and ClaudeBot are training crawlers; OAI-SearchBot and PerplexityBot serve search-related purposes. Our three-probe headline therefore combines different purposes. It is not a search-visibility score. Use the per-identity rows and the crawler-purpose guide for a policy decision.

robots.txt. Read once per site and evaluated for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended. "Disallowed" means the token is named and disallowed on the paths measured; "unlisted" means the file says nothing about it.

llms.txt. Whether /llms.txt returns a text document.

Needs JavaScript. The deep page loaded in a browser with scripts disabled and enabled, from the same client; the share of words present only after scripts ran. The difference identifies content absent from the initial response under this test. It does not establish how every assistant, browser agent or vendor renders the page.

Not measured. Whether any assistant actually cites these sites, how their content answers questions, or anything behind a login. Sites change their rules often; the date is on every figure.

Icons. The small icon beside each name is the site's own favicon, shown the way a browser tab shows it, to identify the measured site. It implies no relationship with Vercatus.

Corrections. If you run one of these sites and the observation is wrong or out of date, write to hello@vercatus.com and we will re-measure and note the change here.

Every observation,
in a file.

The observations the tables are drawn from: one row per site, 64 rows, measured 18 September 2026. Nothing is counted differently for the download.

Using it
Quote, chart or republish these figures with attribution to Vercatus and a link to this page. Please carry the measurement date with the figure: sites change their rules often, and every number here is a reading from one day.
Interpreting a row
First identify the request origin, crawler token, page and observation date. Compare its HTTP response with its robots policy separately. A disagreement is a diagnostic lead, not evidence that an assistant cited or rejected that site. Follow the refusal diagnostic workflow and research publication checklist.
Citing it
Vercatus, Who lets AI assistants read them: Machine access on 64 well-known websites across eight niches, measured 18 September 2026. https://vercatus.com/research/who-lets-ai-read-them/

Start with what AI can read on your own domain.

Crawler access is where AI visibility starts. Create your account and see what Vercatus already reads on your domain; a plan adds Answer Share on your topics, the reasons behind it, and the plays to move it.

Get started

Cancel any time · Data stored in the EU