Conjugate Priors and Posterior Distributions

1. Revisiting the Core Calculation: The Beta-Binomial Model

Let's ground our work in the most common conjugate pair: the Beta-Binomial model. This model is used when we want to infer a probability, $\pi$ (e.g., a conversion rate, the chance of a coin landing heads), which can be any value between 0 and 1.

The setup is:

• Prior: Our initial belief about $\pi$ is modeled by a Beta distribution, $\pi \sim \text{Beta}(\alpha ,\beta )$.
• Likelihood: The evidence from our data comes from a Binomial process. We observe $y$ successes in $n$ trials, so the likelihood is given by the Binomial PMF, $P(y|\pi )=(\genfrac{}{}{0ex}{}{n}{y}){\pi }^y(1-\pi {)}^{n-y}$.

The key insight from our previous lesson was that the posterior is proportional to the product of the prior and the likelihood. Let's watch a short video that walks through this multiplication step-by-step.

As the video demonstrated, the mathematical core of the update is this:

When we drop the constant terms (those that don't depend on $\pi$), the multiplication looks like this:

By grouping the terms with $\pi$ and $(1-\pi )$, we get:

This final expression is the kernel of a new Beta distribution. Because we recognize this form, we know our posterior distribution is:

This is the beauty of conjugacy: a potentially complex calculus problem becomes a simple act of addition.

2. The Beta-Binomial Update Rule in Practice

Let's make this even more concrete. The resource below, Chapter 3 of the online book Bayes Rules!, provides a fantastic, in-depth walkthrough with a practical example.

The key takeaway from that reading is the general update rule for any Beta-Binomial model:

• Prior: $\pi \sim \text{Beta}(\alpha ,\beta )$
• Data: Observe $y$ successes and $n-y$ failures.
• Posterior: $\pi |y\sim \text{Beta}({\alpha }_{post},{\beta }_{post})$, where:
  • ${\alpha }_{post}=\alpha +y$ (Prior pseudo-successes + Observed successes)
  • ${\beta }_{post}=\beta +n-y$ (Prior pseudo-failures + Observed failures)

This simple addition is the computational heart of the process.

3. Example: The Milgram Obedience Study

Let's apply this rule to a real-world example discussed in the Bayes Rules! chapter: the famous Milgram experiment, which tested people's obedience to authority.

Suppose a psychologist, before running the study, has a strong prior belief that very few people would obey orders to inflict harm. They model their belief about $\pi$ (the proportion of people who would obey) with a Beta(1, 10) distribution. This prior is heavily skewed towards zero, with a mean of $\frac{1}{1+10}\approx 0.09$, reflecting their belief that obedience is unlikely.

Now, they run the experiment with n = 40 participants and observe that y = 26 of them obey the authority figure completely.

Let's compute the posterior distribution for $\pi$.

1. Identify Prior Parameters:

   • $\alpha =1$
   • $\beta =10$

2. Identify Data:

   • $y=26$ (successes, i.e., people who obeyed)
   • $n=40$ (total trials)
   • $n-y=40-26=14$ (failures)

3. Apply the Update Rule:

   • ${\alpha }_{post}=\alpha +y=1+26=27$
   • ${\beta }_{post}=\beta +(n-y)=10+14=24$

4. State the Posterior Distribution:

   • The posterior distribution for $\pi$ is Beta(27, 24).

Interpreting the Result:

The psychologist's belief has shifted dramatically. The posterior distribution is now centered around $\frac{27}{27+24}\approx 0.53$. The strong evidence from the data has overwhelmed the initial, skeptical prior.

4. Beyond the Beta-Binomial

The principle of finding a posterior by updating parameters is not limited to the Beta-Binomial case. It applies to all conjugate pairs. Let's briefly look at another example to see the same pattern at play.

Consider modeling data from a Normal distribution with a known mean $\mu$ but an unknown variance ${\sigma }^2$. The conjugate prior for the variance ${\sigma }^2$ is the Inverse Gamma distribution.

Just as with the Beta-Binomial case, the end result is a simple set of rules for updating the parameters of the Inverse Gamma prior to get the Inverse Gamma posterior. This reinforces the idea that conjugacy provides a computational shortcut across many different types of models.

Conclusion

In this lesson, we moved from theory to practice, focusing on the mechanics of computing a posterior distribution. You learned how to combine a prior belief with observed data to form an updated, posterior belief, all without performing any direct integration.

Key Takeaways:

• Computing the posterior for a conjugate pair involves multiplying the prior and likelihood functions and identifying the mathematical "kernel" of the resulting distribution.
• For the Beta-Binomial model, this process simplifies to a straightforward update rule:
  • ${\alpha }_{post}={\alpha }_{prior}+y$
  • ${\beta }_{post}={\beta }_{prior}+(n-y)$
• This process of updating parameters is a general feature of all conjugate models, not just the Beta-Binomial pair.
• The posterior distribution represents a mathematical blend of prior knowledge and observed evidence.

Preview of the Next Lesson:

We have now successfully computed the full posterior distribution, which describes all we know about our parameter of interest. But how do we summarize and communicate this knowledge? In the next lesson, we will learn how to construct and interpret Bayesian credible intervals and compare them to the frequentist confidence intervals you may already know. This will give us a powerful tool for quantifying our uncertainty.