In short: The third group costs a quarter or a year rather than a week, because nothing about them looks wrong at the time. They are found late, if at all, and usually by somebody else.
Measuring without a control, editing the question set, one person holding the method, and claiming a win from a change that has not been re-crawled yet.
All four are cheap to prevent and expensive to discover, which is the worst ratio in this field and the reason this lesson exists at the end rather than the start.
Measuring Without A Control
The slowest failure and the most common. A number is recorded, it moves, and nobody knows how much it moves when nothing has happened.
Without a noise floor there is no scale, so every subsequent comparison is against zero, and movement inside the floor gets reported as a result. A year of that produces a chart, a story and no knowledge.
The correction costs twenty minutes a quarter: three questions, three runs each, same day, per platform. Measuring Your Own Noise Floor also carries the arithmetic trap inside it, which moved a published headline on this site by 14 points.
Improving The Question Set
This one looks like diligence, which is exactly why it survives.
A question is reworded to read more naturally, or two are swapped for better ones, and the trend line now spans two different measurements. Nobody notices, because the sheet still has the same number of rows and the chart still goes left to right.
Add new questions as new rows with their own history. Never edit an old one. The sheet is worth what it is because nothing about it changes between runs.
One Person Holding The Method
Everything works until that person is on leave, changes team, or leaves.
Sessions carry history and accounts carry settings, so a second person running the same set produces a different sheet and neither is wrong. If the method lived in somebody's head, the difference is undetectable and permanent: the trend breaks at the handover and nobody can tell that is what happened.
Six lines on a tab of the sheet fixes it. The questions, the platforms, the runs per question, the session rule, who ran it, and the date. Anybody who has tried to inherit a measurement knows this is the difference between a record and a folder.
Claiming A Win Too Early
Work shipped two weeks before a measurement, the number rose, and the rise goes in the report.
A page has to be re-crawled, re-indexed and re-considered before anything written on it can change an answer, and that runs longer than two weeks in most cases. So the rise is almost certainly something else: a model release, an index refresh, or the ordinary variance between runs.
The check that catches it is whether the other sites named moved too. Yours moves you; a platform change moves everyone. What Moves That Is Not You is the rest of that test.
Why These Are Worse Than The Obvious Ones
A blocked crawler is found in an afternoon and fixed the same day. These four are found by somebody inheriting the work, or by a stakeholder asking a question nobody can answer, and by then the record they would have needed is the thing that was never kept.
The cost is not the mistake. It is that the evidence for the previous year cannot be reconstructed, so the decision about the next year gets made the same way it was made before anybody started measuring.
If you do one thing from this module: write the six-line method on a tab of the sheet and re-measure the floor whenever you measure anything. Both take minutes and both are only ever valuable in retrospect, which is why neither gets done.
Key Takeaways
- Without a noise floor, movement inside the noise gets reported as a result, and a year of it produces a chart with no knowledge in it.
- Improving a question's wording between quarters restarts its history while looking like diligence.
- A method held in one person's head breaks silently at handover, because two people running the same set legitimately get different answers.
- Two weeks is usually not enough for a re-crawl, so a rise in that window is probably not yours. Check whether the other named sites moved too.
- All four are cheap to prevent and expensive to find, and the cost is that last year's evidence cannot be reconstructed.
Check yourself
Before you move on
Not scored, not recorded, and not part of the certificate. Both answers are settled by a sentence in this lesson, and the reasoning appears whichever option you pick.
- 01
What makes this group of mistakes more expensive than a blocked crawler?
- 02
Why does a measurement break silently when the person who ran it hands it over?