Sampling and templates
Discovery
Section titled “Discovery”Vercatus resolves the sitemap tree and parses each child sitemap it reads, recording an exact count and a kind (articles, tags, archives, products, news, locale …) for each. A 1.3-million-URL news site with 336 sitemaps is mapped in under a minute and costs nothing.
One run reads up to 1,000 sitemap files and 400 MB of sitemap XML. On a tree larger than that, the files read are taken in a stride across the whole index, so every part of it is represented, and the run says how many files it did not read: that is a limit of ours, not a fault in the sitemaps, and it is never reported as one.
The URL universe kept for a run is capped (50,000 by default); the cap is allocated across the sitemaps read proportionally with a floor of one, most recently modified URLs first, so no sitemap that was read is silently skipped.
A site with no sitemap, at the usual places or in robots.txt, is read from its home page and the pages on the same
site that the home page links to, up to the run’s page budget, and the run says that no sitemap was found. That
universe is only what the home page links to, so pages reached by search, filters or deeper navigation are not in
it: a sitemap is recommended, and it is what lets a run sample across the whole site.
URL patterns and templates
Section titled “URL patterns and templates”URLs are clustered into patterns such as /blog/{slug} and /{locale}/products/{id} by segment shape and
cardinality. After fetching, pages are clustered again by the structure of their HTML, which is what a
template actually is. A finding scoped to a template holds for every page that shares that structure,
including the ones not sampled.
The page budget
Section titled “The page budget”The sample is stratified by pattern and language so every stratum is represented, with the budget spread by the logarithm of stratum size so long-tail templates are not drowned by the largest one. Tag, archive and author pages are capped at 20% of the budget: they expose thin-template findings without crowding out the content.
A larger budget improves long-tail coverage of intents, claims and per-page findings; it does not much change template-level findings, which stabilise quickly. The run dialog prices any size before you commit.
Reading scope
Section titled “Reading scope”- site: holds for the whole site (or the sample as a whole).
- template: holds for every page sharing that structure.
- page: holds for that URL only.
Page-level findings are reported only when a pattern does not generalise; when a page-level issue affects most of a template or of the sample, it is reported once at the wider scope.