Crafting A/B Test Hypotheses

1. From Problem to Hypothesis: The A/B Testing Workflow

First, let's briefly define A/B testing. It's an experiment where you compare two versions of something—like a webpage, an app screen, or an email—to see which one performs better. You show the original version (the 'control' or 'A') to one group of users, and a new version (the 'variation' or 'B') to another group. By measuring the results, you can make decisions based on data, not just intuition.

The entire A/B testing process is guided by a hypothesis. A hypothesis isn't just a random idea; it's a clear, testable statement about the relationship between a change you want to make and the outcome you expect. It's based on the research and analysis you've already done—like the cohort analysis from our last lesson. Without a solid, data-backed hypothesis, you're essentially guessing, which can be a costly mistake.

2. How to Write a Good Hypothesis

A well-formulated hypothesis turns a business problem into a scientific question. To start, let's explore a foundational article that breaks down the components of a strong hypothesis and common pitfalls to avoid.

As the article explains, a very effective way to structure your thinking is the "If... Then... Because..." framework.

Let's dissect it:

• If you make a specific change (the independent variable). This is the action you will take.
  • Example: "If we change the 'Sign Up' button color from blue to green..."
• Then you expect a specific outcome (the dependent variable). This is the metric you predict will change.
  • Example: "...then the number of new user sign-ups will increase..."
• Because you have a specific reason or rationale. This is the most important part, as it's your underlying assumption based on data, user research, or analysis.
  • Example: "...because our user survey indicated that the blue button doesn't stand out against our website's background."

The "Because" clause is your core hypothesis. The "If/Then" part is your prediction for a specific experiment designed to test that hypothesis. In a BA role, your stakeholders will want to know why you believe a change will work, and your "Because" is the answer.

3. A Formal Framework for Documenting Hypotheses

While "If/Then/Because" is great for thinking and communicating, in a formal setting, you'll often need a more detailed structure to document your testing plan. This ensures all stakeholders are aligned and that you've thought through every aspect of the experiment.

The following video introduces a five-part framework that is perfect for creating this kind of formal documentation.

This more rigorous structure—including the research, the specific change, the expected result, and even the required sample size—is exactly the kind of thoroughness expected from a BA in a data-driven company. It forces you to prove you've done your homework before committing development resources to an experiment.

4. The Statistical Hypothesis: Null vs. Alternative

When you get closer to data science teams or in technical interviews, you'll hear hypotheses discussed in statistical terms: the Null Hypothesis ($H_0$) and the Alternative Hypothesis ($H_A$).

• Null Hypothesis ($H_0$): This is the default assumption that there is no difference between the control and the variation. It states that your proposed change will have no effect on the outcome. For example: "Changing the button color has no effect on the sign-up rate."
• Alternative Hypothesis ($H_A$): This is what you are trying to prove. It states that your change will have an effect on the outcome. For example: "Changing the button color will increase the sign-up rate."

The goal of A/B testing is to collect enough evidence to confidently reject the null hypothesis in favor of the alternative.

The following short clips from a Google Data Scientist clearly explain this concept in the context of an interview.

Understanding this terminology shows a deeper level of analytical maturity and is essential when communicating with data analysts and data scientists.

5. Putting It All Together

Let's apply what you've learned. Imagine you are a Business Analyst at an AdTech company.

Conclusion

Today you've learned how to translate a business problem into a structured, testable hypothesis—a crucial skill for any Business Analyst. This process ensures that your proposed solutions are not just guesses, but are grounded in data and designed to be measured.

Key Takeaways:

• A hypothesis is a data-backed, testable statement that guides an A/B test.
• The "If... Then... Because..." framework is a simple, powerful tool for structuring your hypothesis, ensuring you've identified the change, the expected outcome, and the reason why.
• For formal documentation, a more detailed framework (including research and sample size) adds necessary rigor.
• The Null Hypothesis ($H_0$) assumes no effect, while the Alternative Hypothesis ($H_A$) assumes your change will have an effect. Your goal is to gather enough evidence to reject the null.

Preview of the Next Lesson:

We've formulated a hypothesis, but how do we measure success? Just saying "it will increase" isn't enough. In our next lesson, we will focus on how to define primary and secondary metrics for evaluating an A/B test. This will enable you to quantify the impact of your changes and understand their effects on the broader user experience.