AdTech Case Study: Problem Analysis

1. The AdTech Landscape: A Quick Refresher

Before we dive into the case, let's quickly align on the core concepts of the programmatic advertising world. Given your experience, these ideas will be familiar, but a brief review will help set the stage for our problem.

Programmatic advertising is the automated buying and selling of digital ad space. At its heart is a process called Real-Time Bidding (RTB).

To visualize this ecosystem, let's start with a short video.

The process described in the video can be summarized in the following diagram:

In this lesson, our case study will focus on the Demand-Side Platform (DSP), the platform used by advertisers to manage their ad buys and bid on inventory.

2. The Case: Optimizing a DSP's Bidding Algorithm

Here is the business problem we will analyze today:

You are a Business Analyst at "BidWise," a popular Demand-Side Platform (DSP). Your clients are advertisers who want to show their ads to the right users at the right price.

The engineering team has developed a new, experimental bidding algorithm. The current algorithm primarily bids based on historical conversion rates for similar ad placements. The new algorithm uses machine learning to also predict the likelihood a user will engage with the ad, hoping to win more effective ad placements.

Leadership wants to know if they should roll out this new algorithm to all clients. They've tasked you with designing a test and providing a recommendation.

3. A Framework for Analysis

Tackling a business case like this requires a structured approach. A vague or disorganized analysis won't impress stakeholders or interviewers. We will use a professional decision-making framework to guide our work.

Let's apply this framework to our BidWise case.

Step 1: Clarify the Context

First, we need to ensure we fully understand the problem and constraints. In an interview, you'd ask clarifying questions. Here, we'll state our assumptions:

• Problem: The core business driver is the need to improve the performance of the DSP's bidding algorithm to deliver better results for clients, which in turn increases client retention and platform revenue.
• Business Goal: Increase the value delivered to clients without making the platform prohibitively expensive.
• Constraint: The solution should not significantly hurt key client performance metrics. A client whose costs skyrocket will leave the platform, even if the DSP's metrics look good.

Step 2: Define Goals and Success Criteria

This is where we define what "better" means by setting clear metrics. This is arguably the most critical step for a BA. For this case, we need to balance the DSP's goals with its clients' goals.

• Primary Metric: This is the main metric we expect the new algorithm to improve.

  • Win Rate: The percentage of auctions won out of the total number of bids submitted. A higher win rate means our clients' ads are being shown more often.

• Guardrail Metrics: These are metrics we must monitor to ensure we're not causing unintended harm. Improving the primary metric at the expense of these could make the whole change a failure.

  • Cost Per Acquisition (CPA): This is a key client-facing metric. It's the total cost of a campaign divided by the number of conversions (e.g., sales, sign-ups). If our new algorithm wins more auctions but does so by bidding on extremely expensive inventory, the client's CPA might increase, making them unhappy.
  • Click-Through Rate (CTR): The number of clicks an ad receives divided by the number of times it's shown (impressions). The new algorithm claims to predict engagement. A higher CTR would help validate this claim.
  • Cost Per Mille (CPM): The cost for 1,000 ad impressions. We must track this to understand the cost implications for the DSP and its clients.

Step 3: Design the Experiment

Now, we'll design an A/B test to compare the new algorithm (variant) against the current one (control).

• Hypothesis:

  • Null Hypothesis ($H_0$): The new bidding algorithm has no statistically significant effect on the Win Rate compared to the current algorithm.
  • Alternative Hypothesis ($H_A$): The new bidding algorithm results in a statistically significant increase in the Win Rate.

• Experiment Setup:

  • Randomization Unit: We need to decide what to split into control and variant groups. We'll choose the campaign level. We will randomly assign entire advertising campaigns to either the control group (using the old algorithm) or the variant group (using the new one). This avoids having the same advertiser's campaigns in both groups, which could muddy the results.
  • Duration: We should run the test for at least two full weeks to capture any day-of-the-week effects on user behavior and ad prices. A four-week test would be even more robust.
  • Data Collection: For the duration of the test, we will log the Win Rate, CPA, CTR, and CPM for all campaigns in both the control and variant groups.

Step 4 & 5: Evaluate Systematically & Recommend

After running the experiment, we'll get the results. As a BA, your job is to interpret these results and recommend a course of action. This is where you apply the "Ship, Retest, or No-Ship" framework.

Conclusion

Congratulations on completing this lesson and the entire module! You've successfully navigated a complex, domain-specific case study from start to finish.

Key Takeaways:

• Structure is Key: A formal framework (Clarify -> Define Goals -> Design -> Evaluate -> Recommend) is essential for analyzing business problems and is highly valued in interviews.
• Metrics Define Success: The most critical BA task in an analytical project is defining the primary and guardrail metrics. This step aligns the technical solution with the business goals.
• AdTech is About Trade-offs: Business problems in AdTech often involve balancing competing incentives, such as increasing volume (Win Rate) while controlling costs (CPA).
• The Recommendation is the Goal: Your final output isn't just data; it's a clear, actionable recommendation that considers statistical results, business context, and potential risks.

Preview of the Next Lesson:

This lesson concludes our module on "Business Analytics and Experimentation." You have built a strong foundation in framing problems, defining metrics, and using data to drive decisions.

You are now ready for the Capstone. In the next module, we will move into full, end-to-end case studies that simulate the challenges you'll face in a BA interview and on the job. We'll begin by having you solve a requirements-focused case study, from elicitation to documentation. This will be an excellent opportunity to practice another core BA competency.