Pull-request citability check

A redesign that moves the price table into client-side JavaScript, an edge rule that starts refusing GPTBot, a template that drops its headings: each can cost citations, and each shows up weeks later in your Answer Share. The citability check finds them on the pull request’s preview deploy, before merge.

For the pages a change touches, it fetches production and the preview as the nine identities of a full run (plain HTTP, a browser with and without JavaScript, Googlebot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended), reads what a crawler that runs no JavaScript gets, and runs the retrieval simulation over both with the questions of the site’s latest full run. It then comments on the pull request, Vercatus citability check: passed or failed, with how many of the questions lost or gained grounding, the prices or text the JavaScript-off body of a page loses, and the AI crawlers newly refused, then the detail: the questions that lost or gained grounding, each page’s words, headings, prices and structured data without JavaScript, the share of its text that needs JavaScript, and the identities newly refused. The check fails past the thresholds you set.

No answer engine is asked and no paid model runs: retrieval is simulated on the CI runner. It needs a completed full run of the site, for its questions. The GitHub Action and the command-line client the check runs on are provided on request: ask at hello@vercatus.com.

GitHub

.github/workflows/vercatus.yml

name: Vercatus
on: deployment_status
permissions:
  contents: read
  pull-requests: write
jobs:
  citability:
    if: github.event.deployment_status.state == 'success' && github.event.deployment.environment != 'Production'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: vercatus/check-preview-action@v1
        with:
          preview-url: ${{ github.event.deployment_status.target_url }}
          site-id: ${{ vars.VERCATUS_SITE_ID }}
          api-key: ${{ secrets.VERCATUS_API_KEY }}

Use a key with the read scope limited to the site (API keys). The site id is in the site’s settings. One comment per pull request, updated on every push; the result is also in the job summary.

GitLab

Copy vercatus-check-preview.yml into the repository (as ci/vercatus-check-preview.yml), include it, and extend the job with the review app’s address:

include:
  - local: ci/vercatus-check-preview.yml
vercatus:
  extends: .vercatus-check-preview
  needs: [deploy_review]
  variables:
    VERCATUS_PREVIEW_URL: $CI_ENVIRONMENT_URL
    VERCATUS_SITE_ID: "<site id>"

With VERCATUS_GITLAB_TOKEN (a project access token with the api scope) it posts one note on the merge request and updates it; without, the result is in the job log and in the vercatus-check.md artifact.

Anywhere else

With the command-line client (provided on request):

export VERCATUS_API_KEY=ca_…
vercatus check-preview https://preview.example.dev --site <site id> --paths /,/pricing,/blog/launch

Which pages

Without --paths, the files changed since the base branch are mapped to pages by the framework’s routing: Next.js (app and pages routers), Astro, SvelteKit, Nuxt and content/ folders. A dynamic route such as app/blog/[slug]/page.tsx is checked on two pages of the latest full run that match it. A file that maps to no page (a layout, a component, a stylesheet) can change any template, so one page per template is checked, largest first. Tests, docs and CI files are ignored. For anything else, a route map:

{ "routes": [{ "files": "src/components/pricing/**", "paths": ["/pricing", "/enterprise"] }] }

Options

Option	Default
--paths	pages the changes touch	comma-separated paths, or git:<base ref>
--baseline	production	the site’s own origin, or another origin to compare with
--max-lost	0	fail when more questions than this lose grounding
--no-fail-on-blocked		do not fail when an identity is newly refused
--no-fail-on-body-loss		do not fail when the body without JavaScript loses a price, a heading, structured data or a third of its words
--no-render		skip the browser identities: no browser to install, and no JavaScript comparison
--preview-query		a query string on every preview request, such as x-vercel-protection-bypass=<secret>
--header		a header on the preview’s HTTP requests, repeatable
--route-map		the JSON above
--markdown, --json		write the comment and the full result to files

Exit status: 0 passed, 1 failed, 2 nothing could be compared (a preview behind protection answers every identity with 401; pass its bypass).

How to read it

A question is grounded on a side when the best passage of the checked pages is close enough to answer it (the same measure as a run’s retrieval simulation without the model judge). “Lost” means production’s pages ground it and the preview’s do not; “weaker” that it went from answered to partly answered. The simulation reads the text a crawler without JavaScript gets, because GPTBot, ClaudeBot and PerplexityBot run none. It predicts where citations can come from, not the citations themselves; the next run’s panel measures those.