Gaussian-Gaussian Conjugate Priors

1. The Core of Bayesian Inference: Updating Beliefs

At its heart, Bayesian inference is a formal system for updating our beliefs in light of new data. It combines what we knew before seeing the data with what the data tells us. The process uses three key ingredients, all connected by Bayes' theorem.

1. Prior Distribution, $p(\theta )$: This distribution represents our belief about a parameter $\theta$ before we observe any data. A "strong" prior might be a narrow distribution, indicating high certainty, while a "weak" or "uninformative" prior might be very broad, indicating low certainty.

2. Likelihood, $p(D|\theta )$: This is the same likelihood function we used in MLE. It tells us the probability of observing our data $D$ for a given value of the parameter $\theta$. It represents the evidence from the data.

3. Posterior Distribution, $p(\theta |D)$: This is our updated belief about the parameter $\theta$ after observing the data $D$. It is the result we want to compute.

Bayes' theorem elegantly combines these parts:
$p(\theta |D)=\frac{p(D|\theta )p(\theta )}{p(D)}$
The term in the denominator, $p(D)=\int p(D|\theta )p(\theta )d\theta$, is called the marginal likelihood or the evidence. It acts as a normalization constant, ensuring the posterior integrates to 1. In many cases, we can ignore it and work with the proportionality:
$\underset{\text{Posterior}}{\underbrace{p(\theta |D)}}\propto \underset{\text{Likelihood}}{\underbrace{p(D|\theta )}}\times \underset{\text{Prior}}{\underbrace{p(\theta )}}$
This simple formula is the engine of all Bayesian inference: Posterior belief is proportional to Likelihood of data times Prior belief.

2. Inferring a Gaussian Mean with a Gaussian Prior

Let's make this concrete. Our goal is to infer the unknown mean $\mu$ of a Gaussian distribution, assuming we know its variance ${\sigma }^2$.

The Setup:

• Data: We have $N$ independent and identically distributed (i.i.d.) observations $D=\{x_1,\dots ,x_N\}$ drawn from a Gaussian distribution: $x_i\sim \mathcal{N}(\mu ,{\sigma }^2)$.
• Likelihood: As we saw in previous lessons, the likelihood function is the product of the individual probability densities. As a function of $\mu$, its exponent is a quadratic in $\mu$.
  $p(D|\mu )={\prod }_{i=1}^N\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)\propto \exp \left(-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^N(x_i-\mu {)}^2\right)$
• Prior: We need to specify a prior belief for $\mu$. A mathematically convenient choice is a prior that has the same functional form as the likelihood. Since the likelihood's exponent is quadratic in $\mu$, we choose a Gaussian prior for $\mu$:
  $p(\mu )=\mathcal{N}(\mu |{\mu }_0,{\sigma }_0^2)\propto \exp \left(-\frac{(\mu -{\mu }_0{)}^2}{2{\sigma }_0^2}\right)$
  Here, ${\mu }_0$ is our prior guess for the mean, and ${\sigma }_0^2$ is the variance of our prior belief (how certain we are). This is a conjugate prior because, as we will see, the resulting posterior will also be a Gaussian distribution.

3. Deriving the Posterior Distribution

We now have all the ingredients. The posterior is proportional to the product of the likelihood and the prior. Since both are exponentials, their product is the exponential of the sum of their arguments.

Your interest in mathematical rigor will be well-served here. The key step in this derivation involves an algebraic technique called "completing the square" to rearrange the exponent into the form of a new Gaussian distribution.

The following reading provides a detailed, step-by-step derivation.

After working through the derivation, you will find that the posterior distribution is indeed a Gaussian:
$p(\mu |D)=\mathcal{N}(\mu |{\mu }_N,{\sigma }_N^2)$
with the following parameters for the posterior mean ${\mu }_N$ and variance ${\sigma }_N^2$:

• Posterior Variance:
  ${\sigma }_N^2={\left(\frac{N}{{\sigma }^2}+\frac{1}{{\sigma }_0^2}\right)}^{-1}$
• Posterior Mean:
  ${\mu }_N={\sigma }_N^2\left(\frac{N\bar{x}}{{\sigma }^2}+\frac{{\mu }_0}{{\sigma }_0^2}\right)$
  where $\bar{x}=\frac{1}{N}\sum x_i$ is the sample mean of the data.

4. Interpreting the Result: Precision and Weighted Averages

The derived formulas are mathematically correct, but their true beauty lies in their interpretation. The following video provides an excellent conceptual breakdown.

Let's summarize the key intuitions from the video.

4.1. The Additivity of Precision

The formula for the posterior variance is much clearer if we think in terms of precision, which is the inverse of variance ($\lambda =1/{\sigma }^2$). High precision means low variance (high certainty).

$\frac{1}{{\sigma }_N^2}=\frac{1}{{\sigma }_0^2}+\frac{N}{{\sigma }^2}$
In terms of precision, this is simply:
${\lambda }_N={\lambda }_0+N\cdot {\lambda }_{\text{data}}$
This is a beautiful result! It states that:

The posterior precision is the sum of the prior precision and the total precision contributed by the data.

Every data point we observe adds to our certainty about the parameter $\mu$, causing the posterior distribution to become narrower (higher precision, lower variance).

4.2. The Posterior Mean as a Weighted Average

The formula for the posterior mean, ${\mu }_N$, can be rewritten to reveal a compelling story. It is a precision-weighted average of the prior mean (${\mu }_0$) and the sample mean ($\bar{x}$, which is the MLE of $\mu$).

${\mu }_N=\left(\frac{{\lambda }_0}{{\lambda }_0+N{\lambda }_{\text{data}}}\right){\mu }_0+\left(\frac{N{\lambda }_{\text{data}}}{{\lambda }_0+N{\lambda }_{\text{data}}}\right)\bar{x}$
The posterior mean is a compromise, or a point on the line between the prior mean and the data's sample mean.

• If our prior is very strong (${\lambda }_0$ is large), the posterior mean will be pulled closer to ${\mu }_0$.
• If our data is very strong (large $N$), the posterior mean will be pulled closer to the sample mean $\bar{x}$.
• As $N\to \infty$, the weight on the prior goes to zero, and the posterior mean converges to the sample mean. The Bayesian estimate converges to the frequentist (MLE) estimate.
• If we have a very uninformative prior (${\lambda }_0\to 0$), the posterior mean is simply the sample mean.

This second short video reinforces this weighted-average intuition with clear algebraic steps.

5. Sequential Nature of Bayesian Inference

One of the most powerful aspects of the Bayesian framework is its natural ability to handle data sequentially. The posterior distribution after observing $N-1$ data points can serve as the prior distribution for the $N$-th data point.

This "online learning" property is not an add-on; it falls directly out of the mathematics. The product of the likelihoods for $N$ data points can be grouped in any way we choose:
$p(\mu |D_N)\propto \left({\prod }_{i=1}^{N}p(x_i|\mu )\right)p(\mu )=\underset{\text{Likelihood for Nth point}}{\underbrace{p(x_N|\mu )}}\times \underset{\text{Proportional to Posterior after N-1 points}}{\underbrace{\left(\left(\prod _{i=1}^{N-1}p(x_i|\mu )\right)p(\mu )\right)}}$
This shows that the posterior after $N$ observations is what you get when you update the posterior after $N-1$ observations with the final data point. The Bayesian framework provides a natural and elegant way to continuously learn as data arrives.

Conclusion

This lesson served as your formal entry into Bayesian inference, demonstrating its core mechanics with the canonical example of a Gaussian model.

Key Takeaways:

• Bayesian Framework: The posterior belief about a parameter is derived by multiplying a prior belief with the likelihood of observed data.
• Conjugate Priors: Choosing a prior from a distributional family that is "conjugate" to the likelihood (e.g., Gaussian prior for a Gaussian likelihood) results in a posterior from the same family, which simplifies the analysis immensely.
• Posterior for Gaussian Mean: The posterior distribution for the mean $\mu$ of a Gaussian, given a Gaussian prior, is another Gaussian distribution, $\mathcal{N}(\mu |{\mu }_N,{\sigma }_N^2)$.
• Intuitive Posterior Parameters: The posterior parameters have clear interpretations:
  • The posterior precision is the sum of the prior precision and the data's precision.
  • The posterior mean is a precision-weighted average of the prior mean and the sample mean.
• Sequential Updating: Bayesian inference is naturally sequential; the posterior from one set of data can serve as the prior for the next.

Preview of the Next Lesson:

Now that we have derived a full posterior distribution for our parameter, a natural question arises: how do we get a single point estimate from it? In the next lesson, we will explore two of the most common ways to do this. We will derive the Minimum Mean Square Error (MMSE) estimator, which corresponds to the mean of the posterior distribution, and the Maximum A Posteriori (MAP) estimator, which corresponds to its mode. You will also see the fascinating connection between MAP estimation, MLE, and regularization.