Type-I, Type-II Errors, and Statistical Power

1. The Four Outcomes of a Hypothesis Test

Every hypothesis test, like a verdict in a trial, has four possible outcomes. Two are correct decisions, and two are errors. Let's formalize this.

We start with our two hypotheses:

• Null Hypothesis ($H_0$): There is no effect or no difference. This is the default assumption.
• Alternative Hypothesis ($H_1$): There is an effect or a difference. This is what we are typically trying to find evidence for.

The four outcomes can be summarized in a confusion matrix:

• Type I Error ($\alpha$): The probability of rejecting the null hypothesis when it is actually true. This is also called the significance level of the test.
• Type II Error ($\beta$): The probability of failing to reject the null hypothesis when it is actually false.
• Statistical Power ($1-\beta$): The probability of correctly rejecting the null hypothesis when it is false. This is the test's ability to detect an effect when one truly exists. It is the probability of a "true positive."

To build a strong intuition for these concepts, please watch the following video. It uses a clear courtroom analogy before transitioning to the statistical definitions.

2. The Visual Trade-off Between $\alpha$ and $\beta$

For a fixed sample size, $\alpha$ and $\beta$ have an inverse relationship. Decreasing one inevitably increases the other. The best way to understand this is by visualizing the sampling distributions of our test statistic under both the null and alternative hypotheses.

Imagine we are testing the mean of a population, $\mu$, with $H_0:\mu ={\mu }_0$ vs. $H_1:\mu ={\mu }_1$. Our test statistic is the sample mean, $\bar{X}$.

1. Under $H_0$, $\bar{X}$ follows a distribution centered at ${\mu }_0$.
2. Under $H_1$, $\bar{X}$ follows a distribution centered at ${\mu }_1$.

Our decision rule is to reject $H_0$ if $\bar{X}$ exceeds some critical value, $c$.

• $\alpha$ (the blue shaded area) is the probability of getting an $\bar{X}$ in the rejection region if $H_0$ is true.
• $\beta$ (the green shaded area) is the probability of getting an $\bar{X}$ in the acceptance region if $H_1$ is true.
• Power ($1-\beta$) is the remaining area under the $H_1$ curve (the unshaded part of the green distribution).

Now, what happens if we change the critical value $c$?

• To decrease $\alpha$, we must make our test stricter by increasing $c$ (shifting the decision boundary to the right). As the diagram shows, this simultaneously increases the area of $\beta$.
• To decrease $\beta$ (and increase power), we must make our test more lenient by decreasing $c$ (shifting the decision boundary to the left). This simultaneously increases the area of $\alpha$.

This illustrates the fundamental trade-off. For a fixed sample size, you can't have your cake and eat it too. The following segment of the video you started watching provides a dynamic visualization of this trade-off.

3. Escaping the Trade-off: The Role of Sample Size and Effect Size

If we are stuck with this trade-off, how can we design a test that is both highly significant (low $\alpha$) and highly powerful (low $\beta$)? There are two main levers we can pull:

1. Increase the Sample Size ($n$): As we've seen before, the standard error of the mean is $\sigma /\sqrt{n}$. A larger sample size $n$ leads to a smaller standard error. This makes the sampling distributions for $\bar{X}$ under both $H_0$ and $H_1$ much narrower, or "skinnier." When the distributions become narrower, their overlap decreases. This allows us to find a critical value $c$ that keeps $\alpha$ low while also making $\beta$ small. This is the primary way researchers design powerful experiments.

2. Increase the Effect Size (${\mu }_1-{\mu }_0$): The "effect size" is the magnitude of the difference between the null and alternative hypotheses. If the true effect is very large (i.e., ${\mu }_1$ is far from ${\mu }_0$), their distributions will be far apart, with little overlap. This makes it very easy to distinguish between them, leading to a high-power test. While we usually can't control the true effect size, understanding its role is crucial for interpreting power.

The final part of the zedstatistics video explains these two points very clearly.

4. A Mathematical Formulation

As you appreciate the mathematical details, let's formalize the relationship we just visualized. We can derive an equation that connects $\alpha$, $\beta$, sample size $n$, variance ${\sigma }^2$, and the effect size.

The following document provides a concise and clear derivation.

Let's summarize the key steps of the derivation for a one-sided test $H_0:\theta ={\theta }_0$ vs $H_1:\theta ={\theta }_1$ where ${\theta }_1>{\theta }_0$.

1. Define $\alpha$: The probability of rejecting $H_0$ when it's true.
   $\alpha =P(\hat{\theta }>c|\theta ={\theta }_0)$
   Standardizing this gives:
   $c={\theta }_0+z_{\alpha }\frac{\sigma }{\sqrt{n}}$
   (Note: The resource uses $B_{1-\alpha }$ which is the same as $z_{\alpha }$, the critical value for a one-tailed test).

2. Define $\beta$: The probability of failing to reject $H_0$ when it's false.
   $\beta =P(\hat{\theta }\le c|\theta ={\theta }_1)$
   Standardizing this gives:
   $c={\theta }_1+z_{1-\beta }\frac{\sigma }{\sqrt{n}}={\theta }_1-z_{\beta }\frac{\sigma }{\sqrt{n}}$

3. Equate and Solve for $n$: Since $c$ is the same in both equations, we can set them equal to each other and solve for the sample size $n$:
   ${\theta }_0+z_{\alpha }\frac{\sigma }{\sqrt{n}}={\theta }_1-z_{\beta }\frac{\sigma }{\sqrt{n}}$
   Rearranging yields the famous sample size formula:
   $n=\frac{(z_{\alpha }+z_{\beta }{)}^2{\sigma }^2}{({\theta }_1-{\theta }_0{)}^2}$

This equation is the mathematical embodiment of the trade-offs:

• To achieve smaller $\alpha$ and $\beta$ (which means larger $z_{\alpha }$ and $z_{\beta }$), you need a larger sample size $n$.
• A larger effect size $({\theta }_1-{\theta }_0)$ reduces the required sample size $n$.

5. The Power Function

So far, we've focused on simple alternative hypotheses ($\theta ={\theta }_1$). But what if our alternative is composite, like $H_1:\theta >{\theta }_0$? We can evaluate our test's performance by defining a power function, $\pi (\theta )$, which gives the probability of rejecting $H_0$ for any possible value of $\theta$.

• Power Function: $\pi (\theta )=P(\text{Reject }{H}_0|\text{the true parameter is }\theta )$.

This function has two key properties:

• When $\theta$ is in the null region (e.g., $\theta ={\theta }_0$), the power function gives the Type I error rate: $\pi ({\theta }_0)=\alpha$.
• When $\theta$ is in the alternative region, the power function gives the power: $\pi (\theta )=1-\beta (\theta )$.

Ideally, we want a power function that is close to 0 for all $\theta$ under $H_0$ and close to 1 for all $\theta$ under $H_1$. The following reading explains this concept formally.

Conclusion

In this lesson, you explored the critical relationship between the errors in hypothesis testing. This is a crucial concept that separates elementary applications of statistics from the thoughtful design of experiments.

Key Takeaways:

• Hypothesis testing involves a trade-off between Type I error ($\alpha$) and Type II error ($\beta$). For a fixed sample size, they are inversely related.
• Statistical power ($1-\beta$) is the probability of detecting a real effect, making it a key measure of a test's sensitivity.
• The trade-off can be overcome by increasing the sample size ($n$), which reduces the overlap between the distributions under $H_0$ and $H_1$.
• Power is also higher for larger effect sizes (greater differences between the null and alternative hypotheses).
• The power function $\pi (\theta )$ generalizes the concept of power to composite hypotheses, showing how a test's ability to reject $H_0$ changes as a function of the true parameter value.

Preview of the Next Lesson:

In our examples, we have often assumed the population variance ${\sigma }^2$ is known. In the real world, this is rarely the case. In our next lesson, we will address this by deriving the one-sample t-test statistic, a workhorse of applied statistics used when both the mean and variance must be estimated from the data. This will involve introducing a new probability distribution: Student's t-distribution.