Fundamentals of Hypothesis Testing

Hello! Welcome to the final lesson in our module on Statistical Estimation Theory.

In our previous lessons, we focused on parameter estimation, where our goal was to find the "best" value for an unknown parameter using methods like Maximum Likelihood (MLE) and Maximum a Posteriori (MAP). Today, we shift our focus to a different but related pillar of statistical inference: hypothesis testing. Instead of asking "what is the value of this parameter?", hypothesis testing helps us answer questions like "is it plausible that this parameter is equal to a specific value?".

Your learning outcome for this lesson is to formulate the statistical hypothesis testing framework, including null/alternative hypotheses and Type I/II errors. This framework is the bedrock for making data-driven decisions and is fundamental to interpreting the results of many machine learning models.

1. Formulating Hypotheses

At its core, a statistical hypothesis is a claim or statement about a property of a population, which we can translate into a statement about a parameter $\theta$. Hypothesis testing provides a formal procedure for deciding between two competing hypotheses.

To understand the formal definitions, let's start with a foundational resource.

As the reading explains, we always have two competing hypotheses:

• The Null Hypothesis ($H_0$): This is the default assumption, the status quo, or the statement of "no effect." It often represents a skeptical position. For example, if we're testing a new drug, $H_0$ would be that the drug has no effect. In a parametric model, this corresponds to the parameter $\theta$ belonging to a specific set, ${\Theta }_0$. A simple hypothesis specifies an exact value, e.g., $H_0:\mu =50$. A composite hypothesis specifies a range, e.g., $H_0:\mu \le 50$.

• The Alternative Hypothesis ($H_1$ or $H_a$): This is the claim we wish to find evidence for. It is the competing, mutually exclusive statement to the null hypothesis. It corresponds to the parameter $\theta$ belonging to the set ${\Theta }_1$. For the simple null $H_0:\mu =50$, an alternative could be one-sided ($H_1:\mu >50$) or two-sided ($H_1:\mu \ne 50$).

The logic of hypothesis testing is analogous to a criminal trial: the defendant is presumed innocent ($H_0$) unless the prosecution can provide evidence "beyond a reasonable doubt" to prove guilt ($H_1$). We start by assuming $H_0$ is true and only reject it if our observed data is highly unlikely under that assumption.

2. Decisions and Errors

Once we've set up our hypotheses, we collect data and make one of two possible decisions:

1. Reject $H_0$ in favor of $H_1$.
2. Fail to reject $H_0$.

It's crucial to note that we never "accept" the null hypothesis. We either find enough evidence to reject it, or we don't. Failing to reject $H_0$ simply means the data was not strong enough to convince us to abandon the default position.

Since the true state of nature is unknown to us, our decision can either be correct or incorrect. This leads to four possible outcomes, which are best understood visually.

This leads us to the formal definitions of the two types of errors we can make:

• Type I Error: Rejecting the null hypothesis ($H_0$) when it is actually true. This is a "false positive." In the trial analogy, this is convicting an innocent person.
• Type II Error: Failing to reject the null hypothesis ($H_0$) when it is actually false. This is a "false negative." In the trial analogy, this is acquitting a guilty person.

The following video provides an excellent summary of these concepts.

3. Quantifying Errors: Significance, Power, $\alpha$, and $\beta$

In a statistical framework, we want to control the probabilities of making these errors. These probabilities are given special names:

• Significance Level ($\alpha$): The probability of making a Type I error, given that the null hypothesis is true.

  $$\alpha =P(\text{Reject }{H}_0|H_0\text{ is true})$$

  As the experimenter, you choose the value of $\alpha$ before conducting the test. It represents the risk you are willing to take of making a false positive claim. Common choices are $\alpha =0.05$ or $\alpha =0.01$. Setting $\alpha =0.05$ means you are willing to accept a 5% chance of incorrectly rejecting a true null hypothesis.

• Beta ($\beta$): The probability of making a Type II error, given that the alternative hypothesis is true.

  $$\beta =P(\text{Fail to reject }{H}_0|H_1\text{ is true})$$

  Unlike $\alpha$, $\beta$ is not a single value. It depends on the specific, true value of the parameter under $H_1$. For instance, it's harder to detect a small effect (true mean is 51 when $H_0:\mu =50$) than a large effect (true mean is 70), so $\beta$ would be higher in the first case.

• Power of the Test (1 - $\beta$): This is the probability of correctly rejecting a false null hypothesis.

  $$\text{Power}=1-\beta =P(\text{Reject }{H}_0|H_1\text{ is true})$$

  Power is the probability that our test will successfully detect an effect that is actually there. We generally want tests with high power.

There is an inherent trade-off between $\alpha$ and $\beta$. Making it harder to reject $H_0$ (i.e., decreasing $\alpha$) will naturally make it more likely that we'll fail to reject a false $H_0$ (i.e., increasing $\beta$).

Let's continue with the same video to see how these probabilities are defined.

4. Visualizing the Framework: Test Statistics and Critical Regions

How do we actually decide whether to reject $H_0$? We calculate a test statistic, $T$, from our data. This is a single number that summarizes the evidence. Then, we compare this statistic to a pre-defined critical region (or rejection region), $R$.

• If our test statistic $T$ falls inside the critical region $R$, we reject $H_0$.
• If $T$ falls outside $R$, we fail to reject $H_0$.

The critical region $R$ is defined such that the probability of our test statistic falling into it, assuming $H_0$ is true, is exactly $\alpha$.

The following video provides a brilliant visualization of how $\alpha$, $\beta$, and the critical region all fit together in a concrete example.

Watching those segments should make the abstract definitions much more concrete. You can see two sampling distributions:

1. The distribution of the test statistic if $H_0$ is true. The critical region is the tail(s) of this distribution with a total area of $\alpha$.
2. The distribution of the test statistic if $H_1$ is true (for a specific parameter value). The area of this distribution that overlaps with the rejection region is the power, and the area that falls outside is $\beta$.

Conclusion

Today you have built the foundational vocabulary and conceptual structure of statistical hypothesis testing. This is a cornerstone of scientific inquiry and data-driven decision-making.

Key Takeaways:

• Hypothesis testing involves a choice between a null hypothesis ($H_0$) and an alternative hypothesis ($H_1$).
• We can make two types of errors:
  • Type I Error: Rejecting a true $H_0$. The probability of this is the significance level, $\alpha$.
  • Type II Error: Failing to reject a false $H_0$. The probability of this is $\beta$.
• The power of a test ($1-\beta$) is its ability to correctly detect a true effect.
• The decision is made by comparing a test statistic to a critical region, whose size is determined by $\alpha$.

Preview of the Next Lesson:
With this lesson, you have now concluded our modules on the mathematical foundations of linear algebra, calculus, optimization, probability, and statistics. You've explored the two main branches of inference: parameter estimation and hypothesis testing.

In our next module, we will begin applying all these powerful mathematical tools to build and understand our first machine learning model: Linear Regression. You will see how concepts like gradients, matrix operations, and probabilistic assumptions come together to create a predictive model.