VercatusGoogle agent and non-search crawler controls
Google's crawler documentation (updated July–August 2026) lists tokens beyond Googlebot and Google-Extended that now show up in logs: Google-Agent is a user-triggered fetcher used by agents hosted on Google infrastructure to navigate the web and act on a user's request; it generally ignores robots.txt, uses IP ranges from user-triggered-agents.json, and some of its requests are signed experimentally with Web Bot Auth as https://agent.bot.goog. Google-GeminiNotebook fetches URLs users add as sources in Gemini Notebook and replaces Google-NotebookLM (supported until August 2026). Google-CloudVertexBot obeys robots.txt and crawls only at a site owner's request to build Vertex AI Agents, with no effect on Search. Google-Extended covers Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI, and does not affect inclusion in Google Search.
Sources
- 1
- 2
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Access and crawlers, or search the whole base.
All Access and crawlers entries