VercatusAdditional AI crawler tokens documented in 2026
Vendors not covered by the older token lists now publish per-purpose agents. Mistral: MistralAI-User (user-requested visits in its assistant, governed by robots.txt, not used for training), MistralAI-Index (automated crawling for its search index only, not for training) and MistralAI-Training (crawls to build training datasets). Amazon: Amazonbot (improves products and services and may be used to train Amazon AI models), Amzn-SearchBot (makes content eligible for search experiences such as Alexa, not used for generative AI training, and follows the rules given to other search bots when robots.txt does not name it) and Amzn-User (live fetches for Alexa answers, may not follow all robots.txt directives, not used for training); Amazon says changes take about 24 hours. DuckDuckGo: DuckAssistBot crawls in real time for its cited AI-assisted answers, is not used to train models, and a disallow takes effect within 72 hours without affecting organic rankings.
Sources
- 1
- 2
- 3
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries