Choosing the Right Test for Pre-Post Data

From Continuous Monitoring to Formal Evaluation

The Bayesian approach from our previous lesson is ideal for modeling your evolving understanding during an intervention. The frequentist pre-post analysis serves a complementary purpose: it provides a formal, summative assessment. Imagine a client has completed a four-week intervention designed to improve their relationship satisfaction. You have their daily scores from before the intervention (the baseline phase, 'A') and after (the intervention phase, 'B'). A pre-post analysis allows you to answer the question: "Was the change in average satisfaction from phase A to phase B statistically significant?"

Choosing the Right Tool: A Decision Framework

Selecting the correct statistical test is a systematic process. The appropriate choice depends on the type of data you have, the question you're asking, and whether certain statistical assumptions are met.

Let's trace the path for a typical pre-post analysis in your coaching context:

1. Type of data? You are measuring constructs like satisfaction, compersion, or time spent on logistics, which are numerical.
2. How is the data organized? You are measuring the same client at two different points in time (or averaging over two phases). This is paired data.
3. This leads us to a crucial checkpoint: Do we meet the data distribution assumptions for a parametric test?

For paired data, the most common parametric test is the paired t-test. Let's examine it in detail.

The Paired t-Test: The Standard Approach

The core insight of a paired t-test is that it analyzes the differences between the paired measurements for each individual (or, in your case, for each observation pair in time). Instead of comparing the average of all 'pre' scores to the average of all 'post' scores, it calculates the difference for each pair, and then tests if the average of these differences is significantly different from zero.

Hypotheses in a Coaching Context

The statistical hypotheses translate directly into coaching questions:

• Null Hypothesis ($H_0$): The mean of the differences is zero (${\mu }_d=0$). This implies the intervention had no effect on the client's average score.
• Alternative Hypothesis ($H_a$): The mean of the differences is not zero (${\mu }_d\ne 0$). This implies the intervention had some effect. You could also use a one-tailed test (e.g., ${\mu }_d>0$) if you are only interested in whether the intervention improved the score.

The Mechanics of the Test

The following video provides an excellent step-by-step walkthrough of conducting a paired-samples t-test on a "before and after" weight loss study. The logic is identical to analyzing pre- and post-intervention data for a client.

As the video demonstrates, the key steps are:

1. Calculate the difference $d_i={\text{post}}_i-{\text{pre}}_i$ for each pair of observations.
2. Calculate the mean of these differences, $\bar{d}$.
3. Calculate the standard deviation of the differences, $s_d$.
4. Calculate the t-statistic: $t=\frac{\bar{d}-{\mu }_0}{s_d/\sqrt{n}}$, where ${\mu }_0$ is the hypothesized mean difference (usually 0) and $n$ is the number of pairs.
5. Compare the calculated t-statistic to a critical value from the t-distribution (based on your chosen alpha and degrees of freedom, $n-1$), or simply find the p-value.

Assumptions and Non-Parametric Alternatives

Referring back to the flowchart, the paired t-test is a parametric test, meaning it relies on an assumption about the data's distribution. Specifically, the paired t-test assumes that the differences are approximately normally distributed. This assumption is robust, especially with larger sample sizes ($n>30$), but for the small N-of-1 datasets common in coaching, it's worth checking (e.g., with a Shapiro-Wilk test or a Q-Q plot).

If the assumption of normality is violated (e.g., due to strong skewness or outliers), the appropriate alternative is a non-parametric test. For paired data, this is the Wilcoxon signed-rank test. It works by ranking the absolute values of the differences and comparing the sum of ranks for positive and negative differences. It's less powerful than a t-test when assumptions are met, but more reliable when they are not.

Increasing Power with ANCOVA

While the paired t-test is a valid and standard choice, your statistical background allows us to consider a more powerful and precise method: Analysis of Covariance (ANCOVA).

A paired t-test is essentially an ANOVA on the change scores. ANCOVA offers a more sophisticated approach: instead of analyzing the change score, it analyzes the post-intervention score while statistically controlling for the pre-intervention score (by including it as a covariate in the model).

This method is generally more powerful because it accounts for regression to the mean and explains more of the variance. It isolates the intervention's effect more cleanly by parsing out the variation that is simply due to clients starting at different baseline levels. For your practice, this means a greater ability to detect a true intervention effect, even if it's small.

The following paper, written for a clinical research audience, provides a rigorous comparison of these methods.

The practical implication for your work is clear: While a paired t-test is good, using ANCOVA is better. It represents a more sophisticated analysis that can enhance the credibility and accuracy of your findings, giving you a competitive advantage.

Reporting and Interpretation

A p-value alone is insufficient for conveying the full story to a client. A complete report should provide context and magnitude.

Presenting this full suite of information allows you to move beyond "it worked" or "it didn't work." You can say, "After implementing the communication exercise, your average daily satisfaction score increased from 6.5 to 7.8. This 1.3-point average increase is statistically significant (p < .05), and we are 95% confident that the true average increase lies between 0.8 and 1.8 points." This is a much richer and more professional form of feedback.

Conclusion

Today, we've added a key set of frequentist tools to your quantitative coaching toolkit. You now have a framework for formally evaluating the impact of your interventions using pre-post data.

Key Takeaways:

• Frequentist tests like the paired t-test provide a formal method for determining if an observed pre-post change is statistically significant.
• The test choice is systematic: for paired numerical data, the paired t-test is standard, but you must check the assumption of normality for the differences. If violated, the Wilcoxon signed-rank test is the appropriate non-parametric alternative.
• For greater statistical power and precision, ANCOVA is a superior method that controls for baseline scores, offering a more refined analysis.
• Effective reporting goes beyond the p-value to include descriptive statistics (means, standard deviations) and a confidence interval for the mean difference, providing a complete picture of the intervention's effect.

In our previous lessons, we've designed experiments, created data collection tools, and explored both Bayesian and frequentist methods for analysis. You now have the means to generate and interpret quantitative findings. In our next lesson, we will focus on the crucial final step: constructing compelling data visualizations. We will learn how to turn your statistical results into clear, intuitive graphs and charts that effectively communicate progress and insights to your clients.