Hypotheses for A/B Tests and Model Comparisons

1. The Core Idea: Is It Real or Just Randomness?

Whenever we see a difference between two groups in our data—like a higher click-through rate for a new website design or better accuracy from a new model—we face a critical question: is this a real effect, or is it just due to random chance in our sample?

Hypothesis testing is the statistical procedure for answering this question. It helps us separate a meaningful "signal" from random "noise." The central idea is to start by assuming there is no effect and then see if our data provides enough evidence to convince us otherwise.

To get an intuition for this, let's watch a video that explains the problem of randomness and the purpose of hypothesis testing.

2. The Language of Hypotheses: Null and Alternative

To conduct a test, we need to translate our question into two precise, opposing statements. These are the null hypothesis and the alternative hypothesis.

• The Null Hypothesis ($H_0$): This is the default position, the "status quo." It's a statement of no effect, no difference, or no relationship. Think of it as the "skeptic's view." In our model comparison example, the null hypothesis would be: "There is no difference in performance between the old model and the new model."

• The Alternative Hypothesis ($H_a$ or $H_1$): This is the claim we want to find evidence for. It's a statement that there is an effect, a difference, or a relationship. For our example, the alternative hypothesis would be: "The new model's performance is different from (or better than) the old model's."

The entire process works like a court trial, a great analogy you'll see in the next clip. We assume the null hypothesis is true ("innocent until proven guilty") and then look for strong evidence in our data to reject that assumption in favor of the alternative.

To solidify this, let's read a short, clear definition from an article focused on AI/ML.

3. Application 1: A/B Testing

A/B testing is one of the most common applications of hypothesis testing in industry. Companies use it to test everything from website button colors to marketing email subject lines.

Let's say an e-commerce company wants to test if changing its "Buy Now" button from blue to green increases the conversion rate.

• Group A sees the blue button (the control).
• Group B sees the green button (the variant).

We want to compare the true population conversion rates, $p_{blue}$ and $p_{green}$. Our hypotheses would be:

• Null Hypothesis ($H_0$): $p_{green}=p_{blue}$. There is no difference in conversion rate between the green and blue buttons.
• Alternative Hypothesis ($H_a$): $p_{green}>p_{blue}$. The green button has a higher conversion rate.

Notice that the alternative hypothesis is directional ($>$). This is called a one-tailed test. We are specifically testing for an improvement. If we only wanted to know if there was any difference (better or worse), we would use $p_{green}\ne p_{blue}$, which is a two-tailed test.

This image helps visualize the difference.

Let's see one more crisp example from a data science interview perspective.

4. Application 2: Machine Learning Model Comparison

This framework applies directly to your goal of building and justifying ML models. Every time you try a new architecture, add a feature, or tune a hyperparameter, you should ask: "Is this change producing a statistically significant improvement, or did I just get lucky on my test set?"

Let's formalize our earlier example. You have Model_Old and you've trained Model_New. You want to know if the new model is genuinely more accurate. Let ${\text{acc}}_{\text{Old}}$ and ${\text{acc}}_{\text{New}}$ be the true (but unknown) accuracies of these models on all possible data.

• Null Hypothesis ($H_0$): ${\text{acc}}_{\text{New}}={\text{acc}}_{\text{Old}}$. The new model is not better than the old one. Any difference we see in our test set is due to random chance.
• Alternative Hypothesis ($H_a$): ${\text{acc}}_{\text{New}}>{\text{acc}}_{\text{Old}}$. The new model is genuinely more accurate than the old one.

By testing this, you can provide statistical evidence to back up your claim that your new model is an improvement, which is exactly the kind of rigorous explanation you're aiming to provide.

The following reading details how to operationalize this for various ML tasks.

Conclusion

Today you've learned the conceptual foundation of all hypothesis testing. By correctly formulating a null and alternative hypothesis, you create a clear, testable question from a real-world problem. This is the first and most important step in using data to make credible, defensible decisions.

Key Takeaways:

• Hypothesis testing is a formal procedure to determine if an observed effect in data is statistically significant or just due to random chance.
• The Null Hypothesis ($H_0$) is the default assumption of "no effect" or "no difference."
• The Alternative Hypothesis ($H_a$) is the claim you want to prove, stating that there is an effect.
• The process works by assuming $H_0$ is true and looking for evidence to reject it.
• This framework is used extensively in A/B testing (e.g., comparing website conversion rates) and ML model evaluation (e.g., comparing model accuracies).
• The choice between a one-tailed ($>$ or $<$) and two-tailed ($\ne$) test depends on whether you're testing for a change in a specific direction or any change at all.

Preview of the Next Lesson:

Now that we know how to state our hypotheses, what's next? We need to figure out how to use our sample data to decide between them. In the next lesson, we will learn how to calculate p-values from test statistics and use them to make data-driven decisions. This will complete the core logic of hypothesis testing.