VercatusGet started
Grade AR-158Technical

Crawl budget signals

Google's crawl-budget guide targets sites with 1 million+ unique pages changing about weekly, 10,000+ pages changing daily, or many URLs in 'Discovered – currently not indexed' (rough estimates, not exact thresholds); others need only an up-to-date sitemap. Budget per hostname is crawl capacity (rises with stable, fast responses; falls with slow responses, 5xx and 429) times crawl demand (perceived inventory, popularity, staleness). Google advises consolidating duplicates, robots.txt (not noindex) for URLs that shouldn't be crawled, 404/410 for removed pages, eliminating soft 404s, fresh sitemaps with lastmod, avoiding long redirect chains, faster pages and 304 support.

Sources

  1. 1
    Google · 2026Optimize your crawl budgetdevelopers.google.com
  2. 2
    Google · 2026How HTTP status codes affect Google's crawlersdevelopers.google.com

AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.

Put this evidence to work

Related entries