Applying the Test for Excess Success

1. The Logic of "Too Good to Be True"

At its heart, science that relies on statistical inference is probabilistic. Even if an effect is real and your experimental design is sound, random sampling variability means you won't get a "successful" outcome every single time. A study's statistical power is the probability of detecting an effect that is truly there. If you have a set of studies with an average power of 60%, you would expect to find a significant result in about 6 out of 10 attempts, not 10 out of 10.

When a series of experiments, particularly underpowered ones, all report successful outcomes, it defies statistical expectations. This "excess success" can be a red flag, suggesting that the results may have been influenced by factors other than the phenomenon under study.

As the article explains, the TES formalizes this intuition. It calculates the probability (P_TES) that a set of experiments, given their estimated power, would produce at least as many successful outcomes as were actually observed. A low P_TES (typically below a criterion of 0.1) suggests the findings are suspiciously consistent and may be biased.

2. Formalizing the Consistency Test

The Test for Excess Success is also known as a "consistency test" because it checks whether the observed rate of success is statistically consistent with the estimated power of the experiments. Given your background in statistics, the formal mechanics will be straightforward.

The core of the test involves comparing the observed number of successful studies, $O$, with the expected number of successful studies, $E$. For a set of $M$ independent experiments, the expected number is simply the sum of their individual power values:

If the observed number of successes $O$ is much larger than the expected number $E$, the results are inconsistent. The P_TES value quantifies this inconsistency. In the simplest case, where all $M$ experiments are reported as successful ($O=M$), the P_TES is the joint probability of this happening, calculated as the product of all the individual power values:

3. Estimating Power: The Critical Ingredient

The entire TES calculation hinges on having a reasonable estimate of each experiment's power. This is the most challenging step, as the true effect size is unknown. There are several approaches to this, each with its own assumptions and trade-offs. For our purposes as a self-diagnostic, two methods are most relevant.

1. Post Hoc Power: You estimate the power of each experiment using its own observed effect size. This is computationally simple and directly uses the data you have. However, as statisticians have noted, it has known biases. Specifically, it tends to underestimate power when true power is high (>0.5) and overestimate it when true power is low (<0.5).
2. Pooled Fixed Effect Size: You assume all experiments in the set are measuring the same underlying effect. You then calculate a weighted average of the observed effect sizes from all experiments to get a more stable, pooled estimate. This pooled effect size is then used to calculate the power for each individual experiment. This method is generally more robust than post hoc power, provided the assumption of a fixed effect is reasonable.

For evaluating pilot data, both methods can be useful. If your pilot consists of several distinct but related mini-experiments (e.g., testing different activities), a pooled approach might be best. If the pilot involves repeating the same experimental contrast under slightly different conditions, the post hoc method is a very direct way to assess the results you've just generated.

4. Self-Diagnosis: Applying TES to the Vee Experiment Pilot

Let's ground this in the context of your proposed N-of-1 crossover experiment. Imagine you run a pilot phase consisting of four sequential blocks. In each block, you compare the "Nesting" arrangement to one of the "Separate" arrangements over a period of weeks, collecting daily satisfaction scores. Your primary hypothesis for each block is that Triadic_Satisfaction is higher during the Nesting phase.

Suppose that after running the four blocks, you analyze the data for each block separately and find a significant result ($p<.05$) in all four cases. This feels like a strong success. But is it too strong? Let's apply the TES as a self-diagnostic.

Step 1: Define Success and Count Observations

• Success criterion: A statistically significant result ($p<.05$) for the planned contrast in a given block.
• Observed successes (O): You found 4 successes out of 4 blocks. So, $O=4$ and $M=4$.

Step 2: Estimate Power for Each Block (using Post Hoc Power)
You would go back to the analysis of each of the four blocks:

• Block 1: You observed an effect size of $g=0.45$. Given your sample size for that block, the post hoc power is calculated to be 0.57.
• Block 2: You observed a larger effect, $g=0.60$. The power is 0.72.
• Block 3: A smaller effect, $g=0.41$, gives a power of 0.52.
• Block 4: A similar effect to the first, $g=0.46$, gives a power of 0.58.

Step 3: Calculate P_TES
Since all experiments were successful, we simply multiply the power estimates:

Step 4: Interpret the Result
Your calculated P_TES is 0.124. This is greater than the conventional criterion of 0.1. Therefore, this set of results does not show evidence of excess success. While fortunate, a run of four successes is statistically plausible given the power of your pilot experiments.

What if the result had been different?
Suppose your power estimates were lower: 0.45, 0.51, 0.42, and 0.48. Then:

Here, P_TES is less than 0.1. This would be a red flag. It doesn't mean your theory is wrong, but it suggests the perfect results might be an artifact. This is the point where you, as a diligent researcher, would pause and critically examine your process. Could optional stopping or some form of p-hacking have crept into your pilot analysis?

This is a crucial step because it allows you to identify and correct potentially biased research habits before you conduct your main, pre-registered experiment, thereby strengthening the credibility of your final results.

Conclusion

Today we've added a powerful self-diagnostic technique to your methodological toolkit. The Test for Excess Success provides a formal way to check if a set of findings is statistically credible or suspiciously perfect.

Key Takeaways:

• Excess Success: A set of experiments that are all "successful" can be a statistical red flag, especially if the individual experiments have low to moderate power.
• The Test for Excess Success (TES): This test quantifies this suspicion by calculating P_TES, the probability of observing a success rate at least as high as what was reported, given the estimated power of the studies. A low P_TES (e.g., < 0.1) indicates excess success.
• Application as a Self-Diagnostic: By applying the TES to pilot data, you can identify potential biases in your research practices (like optional stopping or unconscious p-hacking) before your main pre-registered study, protecting your final work from these threats.
• Power Estimation is Key: The validity of the TES depends on the power estimates used. Methods like post hoc power or pooled effect size estimation are practical choices for analyzing pilot data.

By internalizing this "skeptical scrutiny" of even your own successful results, you elevate the rigor and credibility of your research. This mindset is a cornerstone of navigating the replication crisis.

In our next lesson, we will directly tackle one of the primary culprits behind excess success: optional stopping. We will propose specific stopping rules for data collection within each period of your experiment that are robust to this bias, ensuring your data collection process is as rigorous as your analysis plan.