Crawl budget signals
Google's crawl-budget guide targets sites with 1 million+ unique pages changing about weekly, 10,000+ pages changing daily, or many URLs in 'Discovered – currently not indexed' (rough estimates, not exact thresholds); others need only an up-to-date sitemap. Budget per hostname is crawl capacity (rises with stable, fast responses; falls with slow responses, 5xx and 429) times crawl demand (perceived inventory, popularity, staleness). Google advises consolidating duplicates, robots.txt (not noindex) for URLs that shouldn't be crawled, 404/410 for removed pages, eliminating soft 404s, fresh sitemaps with lastmod, avoiding long redirect chains, faster pages and 304 support.
Sources
- 1
- 2
AGrade A: official documentation or a standard. A grade describes the provenance of the sources above; it is not a probability that a change will work. The effect basis says how the size of the effect is known: measured by a study, taken from an industry report, only documented without a size, or not quantified.
Put this evidence to work
Related entries
More in Technical, or search the whole base.
All Technical entries