Verdicts

Every play you ship gets a verdict within 45 days: whether your Answer Share on the play’s topics moved, against the topics you left alone over the same days. A verdict also says how strong its evidence is, with a grade, and what else changed in the window. It is the answer to the fourth question, did it work?

How a verdict is measured

Treated and left alone. A play is aimed at some topics and engines: its treated cells. A page listing cited by two engines treats those two engines; a change to your own page, or a search play, treats every engine. The comparison is the topics no play touched in the same weeks and that share no questions with the play’s, on the same engines.

The verdict compares the change in your Answer Share on the treated topics, before and after, with the change on the topics left alone over the same days. A rise everywhere, from a market trend or an engine update, moves both and cancels out; what remains is the play’s.

Never against competitors. Answer Share is shared out among the brands in an answer, so when you gain, someone loses. A competitor’s share is not a fair comparison, and it is never used as one.

From the day engines could see it. The clock starts when the change became visible to the engines, not when it was planned:

• Off the site: the first version of the page that names your brand, which the page’s history dates to the day.
• On your site: the later of the day the acceptance check passed and the day the engine’s own crawler first fetched the changed page, read from your crawler traffic. With crawler logs connected, the first read waits for that fetch, for up to two weeks. Without them, the check’s date is used, and the verdict says so.

Answers per question, honestly. The poll asks many questions and few of them twice, and the range on each verdict allows for the fact that repeat asks of one question agree with each other, and for how few questions a single topic holds. A play on one topic therefore needs more answers to reach a firm verdict than a play on several.

Reads on day 14, 30 and 45

Each play is read on day 14, day 30 and day 45 after the change became visible.

• Day 14 is an early read. It calls a win only when the effect is overwhelming, and it never calls No effect.
• Day 30 is the main read.
• Day 45 is final. The verdict is then frozen and kept as it was: what we told you stays told.

States

State	Meaning
Won	Answer Share on the play’s topics rose against the topics left alone: the whole range is above zero, and the rise is large enough to matter.
Lost ground	It fell against the topics left alone: the whole range is below zero, by enough to matter.
No effect	The range lies within the smallest change this much data could detect: the play did not move your share by more than that, either way. Drawn hatched, within noise.
Inconclusive	The window is complete, but the range crosses zero and is wider than what could be detected. It says how many more answers would decide it, and roughly how long that takes at the current pace.
Too early	The window is not complete yet, or the play’s topics do not have enough answers since the change.

Each verdict has one sentence in plain words, in this form (illustrative): “Won: +4.2 points on the play’s topics (range +1.8 to +6.5), against the topics left alone. Grade B: measured together with two other plays.”

Grades

Grade	When
A	One play on its own, nothing else changed in the window, the checks that should stay flat did, and plenty of answers.
B	Clear, with one caveat: measured together with other plays, one other change in the window, or fewer answers.
C	The direction only: the range crosses zero, but the estimate and the checks that should stay flat agree.
D	Before and after only, with nothing to compare against: every topic was treated, or a check that should stay flat moved the same way.

Only grades A and B can be Won or Lost ground, and only A and B may say a play caused a change, anywhere in the product. A read whose numbers alone would say Won but grades C or D is held back: Inconclusive at day 45, Too early before it. A grade D read says plainly that nothing was left alone to compare with, and suggests holding topics out for the next play.

The checks that should stay flat

Two comparisons should not move if the play is the cause:

• Answers without search, for a change to your own pages. A page change cannot move what an engine knows from training within weeks; if memory moves as much as search, something else is going on.
• Engines the play does not treat. If they move as much as the treated engines, the move is not the play’s.

When either moves clearly, the same way, the verdict is graded down.

What else changed

Recorded with every verdict, and graded down:

• an engine changed its model or what it cites in the window;
• another play shipped on the same topics;
• a competitor made a large move on those topics;
• the poll was interrupted, with far fewer answers than expected in a week.

Plays that ship close together on overlapping topics are measured together as a bundle. The bundle gets the verdict, and each play says it was measured together with the others. Each play is compared only with topics that were untreated in its own window, never with topics a play treated earlier.

Wins

A win is a topic and engine where your Answer Share rose beyond its range after a play, against the topics left alone. Wins are written when a Won verdict is frozen at day 45, one per treated topic (and per engine when the play treats only some), and they are the record on Results. Wins and the verdicts behind them also teach Vercatus Intelligence what works.

Making verdicts stronger: hold topics out

The strongest comparison is one you set up on purpose. On Results you can hold some of your topics out: no play ships on them while the experiment runs, so every verdict has topics that were left alone by design. See Results.

The verdict design is also summarised on the methodology page, and Limits of the method says what it cannot prove.