Did the fix actually work? Measuring before and after

The team ships an optimisation. Someone runs a speed test, the score is four points higher, a screenshot goes in the channel, everyone moves on. Two weeks later the same page tests four points lower and nobody posts that screenshot.

Neither number meant much. Optimisation without trustworthy measurement is just hoping with extra steps, so here is how to measure a fix in a way that makes the answer real.

Rule 1. Compare like with like

A test result depends on more than your page. It also depends on the server that ran it, the simulated network, the hardware, and whether caches were warm. Change any of those between your before and your after and you're comparing two different experiments, not two versions of your page. (This is the same reason Webperf Cloud and PageSpeed Insights disagree. Different conditions give different numbers, and both are internally consistent.)

The fix is to use the same tool, under the same conditions, for both measurements. Controlled lab testing exists for this reason. When conditions are held constant, a change in the score comes from a change in your page and nothing else.

Rule 2. One run is an anecdote

Even under controlled conditions, every measurement wobbles a little. A single before-run and a single after-run can differ by a few points on identical pages, which means a small improvement can be pure noise, cheerfully screenshotted.

Two habits protect you:

Rule 3. Change one thing at a time (when it matters)

If a release contains a new image pipeline, two removed scripts and a redesigned header, and the score improves, which change did it? Nobody knows, and it matters the next time you have to choose what to invest in.

You don't need scientific purity for routine work. But for the expensive question, should we roll this approach out everywhere?, isolate the change. Ship it to one template, measure, then decide with evidence instead of folklore.

Rule 4. Respect the lag in field data

Lab data answers today whether the fix worked mechanically. But the question your stakeholders actually care about, do real visitors feel it?, is answered by field data (CrUX, Google's real-Chrome-user measurements), and field data is slow on purpose. It's collected over a rolling 28-day window.

So a fix shipped today reaches full effect in the field numbers about a month later, diluted until then by pre-fix visits. The practical rhythm:

  1. Day 0: re-test in the lab under the same conditions. This is your proof the fix works.
  2. Weeks 2–6: watch the field trend move. This is your proof the fix matters.

Teams that don't know about the lag conclude their fix failed after a week of flat field data and sometimes undo perfectly good work. Now you know.

Rule 5. Write down what you expected before you look

This is the cheapest reality check there is. Before shipping, write one sentence. Removing the carousel script should cut TBT roughly in half on product pages. Then look.

If the prediction hits, you've confirmed you understand your site. If it misses, you've learned something more valuable than the fix. Your mental model was wrong, and the next optimisation guess would have been wrong too. Either way you win, but only if the sentence was written before the measurement.

Do all of this properly and something changes in how the work is funded. Instead of we did some performance work, you can say we cut p75 LCP from 4.1 s to 2.4 s on 600 product pages, here's the before/after. One of those gets next quarter's budget.

The measurement is the part of the work that makes the optimisation count.

With continuous testing, your before/after evidence is already collected by the time you need it. See the plans, or email hello@webperf.cloud.