Evaluating Practical Significance in Regression

Why Simple Comparisons Are Misleading

A single metric like the median is a point estimate—one number trying to represent thousands or millions of measurements. It's useful, but it hides the full story of the data's distribution and variability. A 15ms change in the median can mean very different things depending on the dispersion (or spread) of the results in each test.

Consider two scenarios, both with a 15ms median increase:

1. High Variability: Both the baseline and candidate tests show a wide spread of response times. The distributions of the two tests overlap significantly. In this case, the 15ms difference might just be random chance—a product of the inherent noise in the system.
2. Low Variability: Both tests produce very consistent results with a narrow spread of response times. The distributions are tight and barely overlap. Here, that same 15ms difference is much more likely to be a real, repeatable effect of the code change.

To make a sound judgment, we need to account for this variability. But first, let's clarify two crucial and often confused concepts: statistical significance and practical significance.

Statistical vs. Practical Significance

In performance engineering, not all measurable differences are meaningful. A change might be "statistically significant," meaning it's very unlikely to be due to random chance, but still be too small to have any real-world impact. This is the concept of "practical significance."

The following article from "Statistics By Jim" provides an excellent explanation of this distinction. As you read, focus on how sample size can create statistically significant results for tiny, unimportant effects.

As the article makes clear, your subject-area expertise is required to define what is practically significant. For your system, is a 10ms regression acceptable? 50ms? Does it depend on the endpoint? This threshold is your benchmark for practical importance, often informed by Service Level Objectives (SLOs). A change might be statistically real, but if it's well within your SLO, it may not be worth blocking a release over.

Quantifying Uncertainty with Confidence Intervals

So, how do we account for the sample variability we discussed earlier? We use confidence intervals (CIs). Instead of a single point estimate (e.g., "the median is 165ms"), a confidence interval gives us a range of plausible values for the true median (e.g., "we are 95% confident the true median is between 161ms and 169ms"). This range is calculated from your sample data and incorporates its variability.

This short video gives a great conceptual introduction to what confidence intervals are and how they're used.

The key takeaway is that the CI provides a more honest representation of our measurement. It acknowledges that our test run is just one sample, and the "true" performance characteristic lies within a range. The width of this range tells you how much uncertainty there is: a narrow CI means high precision, while a wide CI indicates high variability in your measurements.

Comparing Two Samples Using Confidence Intervals

Now we can apply this tool to our problem of comparing a baseline and a candidate build. By calculating a confidence interval for the median of each test, we can perform a quick visual statistical test.

This video from StatQuest provides a brilliantly clear visual explanation of how to do this.

As you saw, the rule is simple and powerful:

• If the 95% confidence intervals of the two samples do not overlap, the difference between them is statistically significant.
• If they do overlap, the result is inconclusive; the difference may not be statistically significant.

Let's look at what this looks like on a typical plot you might generate.

This visual check is your first step in rigorous analysis.

Putting It All Together: A Workflow for Regression Analysis

Let's consolidate these ideas into a practical workflow you can use on the job.

1. Define Practical Significance: Before you test, determine the smallest performance change that matters. For example, "any regression greater than 20ms or 5% of the median response time is practically significant." This often comes from your SLOs.

2. Run and Measure: Execute your load tests for the baseline and candidate builds, ensuring you analyze only the steady-state data.

3. Calculate Medians and CIs: For each test run, calculate the median response time and its 95% confidence interval. Most statistical packages or libraries (like Python's scipy and numpy) can do this.

   • A quick note: Because performance data is often heavily skewed (think of the long tail of p99 latencies), the median is a more robust measure of central tendency than the mean. The academic paper below discusses why this is the case.

4. Check for Statistical Significance: Plot the two medians and their CIs, just like in the interval plot above. Do they overlap?

   • No Overlap: The regression is statistically significant. Proceed to the next step.
   • Overlap: You cannot conclude there is a significant difference. The change is likely just noise. You can stop here.

5. Check for Practical Significance: If the result was statistically significant, now compare the magnitude of the change to your predefined threshold.

   • Look at the gap between the intervals. For a regression, the most pessimistic (and safest) estimate of the difference is the difference between the bottom of the candidate's CI and the top of the baseline's CI.
   • Is this difference greater than your practical significance threshold?
     • Yes: You have strong evidence of a real, meaningful performance regression. Time to raise a flag and investigate.
     • No: You have found a statistically real effect, but its magnitude is too small to be concerned about. You can likely approve the change.

This structured process provides a defensible, evidence-based foundation for your performance recommendations, which is a hallmark of a senior engineer.

For a deeper dive into the statistical reasoning, especially regarding the use of the median for performance data, the following research paper provides a formal perspective.

Conclusion

You are now equipped to move beyond simple "eyeball" comparisons of performance metrics. By embracing a bit of statistical rigor, you can make far more reliable and defensible decisions about whether a performance change is real and whether it matters.

Key Takeaways:

• Comparing single-point estimates like the median is unreliable because it ignores data variability.
• Statistical significance indicates an effect is likely real (not random chance), while practical significance asks if the effect is large enough to care about.
• Confidence intervals (CIs) provide a range of plausible values for a metric, accounting for measurement uncertainty.
• A powerful visual test for statistical significance is to check for overlap between 95% confidence intervals. No overlap implies a significant difference.
• A complete analysis involves checking for both statistical significance (using CIs) and practical significance (using your domain expertise and SLOs).

In this lesson, we've learned how to properly interpret measurement data. However, this assumes the data itself is correct. In our next lesson, we will investigate a subtle but critical measurement problem known as coordinated omission, which can cause your tools to lie to you by systematically ignoring the slowest requests, making your system appear much faster than it really is.