Sign Test for Paired Satisfaction Scores

The Evidence: Before and After

Here is the data from Dr. Eve's 12 client couples. We are looking for a consistent trend, but a glance at the raw numbers shows a mixed picture. Some improvements are large, others are small, and in at least one case, satisfaction appears to have worsened.

Before we attempt to quantify the magnitude of these changes, let's first focus on the most fundamental piece of information: the direction of change. Did the score go up, down, or stay the same? This approach is powerful because it makes no assumptions about the data being perfectly interval-scaled (i.e., that a jump from 2 to 4 is "equal" to a jump from 7 to 9). It's a robust, non-parametric method.

The Tool: The Sign Test

We will reduce each pair of scores to a simple symbol: a '+' for an improvement, a '−' for a decline, and a '0' for no change. This process forms the basis of the sign test. This tool is ideal for our situation as it directly tests for a consistent directional shift in paired data.

The core logic of the sign test should feel familiar. Once we have our list of '+' and '−' signs, we find ourselves back in the world of the binomial distribution. The null hypothesis, our "chance-only" scenario, is that there is no real change, meaning a '+' and a '−' are equally likely outcomes ($p=0.5$).

The following guide from statstutor.ac.uk provides a concise overview of the formal procedure.

Applying the Test to Dr. Eve's Data

Let's apply this procedure to our evidence. First, we calculate the difference (After - Before) for each couple and assign the corresponding sign.

Now, we tally the signs:

• Number of positive signs ($r^{+}$): 9
• Number of negative signs ($r^{-}$): 1
• Number of ties (zeros): 2

As the procedure dictates, we ignore the ties. This leaves us with an effective sample size of $n^{\prime }=r^{+}+r^{-}=9+1=10$.

Our investigative question is whether the program leads to improvement. This gives us a clear set of hypotheses:

• Null Hypothesis ($H_0$): The program has no effect. The median of the differences is zero. Any change is random, so P(Improvement) = P(Decline) = 0.5.
• Alternative Hypothesis ($H_1$): The program is effective. The median of the differences is greater than zero. P(Improvement) > 0.5.

This is a one-sided test. We need to find the probability of observing a result as extreme or more extreme than our finding (9 positives out of 10) if the null hypothesis were true. We are calculating $P(X\ge 9)$ for a binomial distribution with $n=10$ and $p=0.5$.

Using the binomial formula $P(X=k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$:

This p-value of approximately 1.1% is the probability of seeing such a strong positive trend (9 or more improvements out of 10) just by chance. The image below shows a similar analysis performed in a spreadsheet, where the final p-value is the key piece of evidence.

Our manual calculation is the workhorse behind such a summary. The following image provides another example, detailing the discrete steps of counting signs and applying the binomial distribution for a small sample.

For a more rigorous review of the test's formal assumptions and further examples, the following intelligence brief from Wikipedia is an excellent resource.

Conclusion

Our analysis has yielded a crucial piece of evidence. The p-value of $\approx 0.0107$ is well below the conventional significance level of 0.05. This allows us to confidently reject the null hypothesis.

The strong directional pattern of improvement is not a random occurrence. While her claim of a 100% success rate was slightly exaggerated (we found one couple whose satisfaction decreased and two with no change), the data provides statistically significant evidence that her program does, in fact, tend to produce positive results. We have moved from identifying a suspicious claim to verifying a real underlying effect.

Key Takeaways:

• The sign test is a non-parametric tool used to test for a consistent directional change in paired data.
• It operates by converting paired differences into +, -, or 0 signs.
• Ties (0s) are discarded, and the test statistic is based on the counts of + and - signs.
• Under the null hypothesis of no effect, the number of positive signs follows a binomial distribution with $p=0.5$.
• The p-value is the probability of observing a result as, or more, extreme than the one found, assuming the null hypothesis is true.

Preview of the Next Lesson

We've confirmed a significant directional effect. But in our zeal to be robust, we discarded information about the magnitude of the improvement. Are the improvements minor or life-changing? Is there a relationship between a couple's initial satisfaction and how much they benefit from the program?

In our next session, we will re-examine the Before and After scores, this time using correlation analysis. We will investigate whether there is a linear or ranked relationship between the starting and ending scores, which could reveal deeper patterns in how Dr. Eve's "miracle cure" actually works.