Exponential Family Form of Normal and Gamma Distributions

Recalling the Exponential Family Form

As a quick refresher, a distribution is in the exponential family if its PDF or PMF can be written as:

The components are:

• $\theta$: The natural parameter, related to the mean.
• $\varphi$: The dispersion parameter, related to the variance or scale.
• $a(\varphi )$: A function of the dispersion parameter. We'll see this is often just $\varphi$ itself.
• $b(\theta )$: The cumulant function, which helps determine the mean and variance.
• $c(y,\varphi )$: A function of the data and dispersion, acting as a normalizing constant.

Our strategy remains the same: take the PDF, get everything inside an exp() function, and algebraically rearrange it to match this target structure.

Worked Example 1: The Normal Distribution

Let's begin with the familiar bell curve. The Normal distribution is the foundation of standard linear regression, and seeing it in this new context helps bridge the gap to GLMs.

The PDF for a Normal distribution with mean $\mu$ and variance ${\sigma }^2$ is:

Our task is to manipulate this into the exponential family form. The process involves some algebraic steps like expanding squares and grouping terms. To see this done clearly, step-by-step, the following video is an excellent guide.

Summary of the Derivation and Components

As shown in the video, the algebraic manipulation leads to this form:

Now, we can match this to our target structure $\frac{y\theta -b(\theta )}{a(\varphi )}+c(y,\varphi )$ and identify each component.

• Natural Parameter ($\theta$): The term multiplying $y$ inside the main fraction is $\mu$.

  $$\theta =\mu$$

• Dispersion Parameter ($\varphi$): The variance ${\sigma }^2$ serves as the dispersion parameter.

  $$\varphi ={\sigma }^2$$

• Function $a(\varphi )$: The denominator of the main fraction is ${\sigma }^2$, which is just $\varphi$.

  $$a(\varphi )=\varphi$$

• Cumulant Function ($b(\theta )$): The term subtracted from $y\theta$ in the numerator is ${\mu }^2/2$. Since $\theta =\mu$, we have:

  $$b(\theta )=\frac{{\mu }^2}{2}=\frac{{\theta }^2}{2}$$

• Function $c(y,\varphi )$: The remaining terms make up $c(y,\varphi )$.

  $$c(y,\varphi )=-\frac{1}{2}\left(\frac{y^2}{{\sigma }^2}+\ln (2\pi {\sigma }^2)\right)=-\frac{1}{2}\left(\frac{y^2}{\varphi }+\ln (2\pi \varphi )\right)$$

The lecture notes from Purdue University's statistics department provide a concise summary of this same result, which can be useful as a quick reference.

Worked Example 2: The Gamma Distribution

The Gamma distribution is common for modeling right-skewed, strictly positive continuous data, such as insurance claims or waiting times. Its derivation is algebraically more involved than the Normal distribution's.

The Gamma PDF is often parameterized with a shape parameter $\alpha$ and a rate parameter $\beta$:

Here, $\Gamma (\alpha )$ is the gamma function.

Let's convert this into the exponential family form.

Step 1: Get everything into an exp() function.
We use the fact that $x=\exp (\ln x)$.

Step 2: Rearrange to match the target form.
This is the trickiest part. We need to create the structure $\frac{y\theta -b(\theta )}{a(\varphi )}$. The key insight is to define the dispersion parameter $\varphi$ in terms of the shape parameter $\alpha$, and the natural parameter $\theta$ in terms of the rate parameter $\beta$.

Let's define our parameters as follows:

• Natural parameter: $\theta =-\beta$
• Dispersion parameter: $\varphi =\alpha$

With these definitions, we can't directly form $y\theta$. Let's use a different, more standard set of definitions from GLM theory:

• $\theta =-\beta /\alpha$
• $\varphi =1/\alpha$

This choice may seem arbitrary, but it leads to a clean separation of terms. With $\alpha =1/\varphi$ and $\beta =-\theta /\varphi$, we can substitute these back into the log-likelihood expression. The algebra is dense, but it results in:

While you are not expected to derive this from scratch on the exam, you are expected to be able to identify the components from the final form.

Let's identify the components from this expression:

• Natural Parameter ($\theta$): Defined as $\theta =-\beta /\alpha$.
• Dispersion Parameter ($\varphi$): Defined as $\varphi =1/\alpha$.
• Function $a(\varphi )$: The denominator of the main fraction is simply $\varphi$.
• Cumulant Function ($b(\theta )$): The term subtracted in the numerator is $-\ln (-\theta )$. $$b(\theta )=-\ln (-\theta )$$
• Function $c(y,\varphi )$: All the leftover terms. $$c(y,\varphi )=(\frac{1}{\varphi }-1)\ln y-\ln \Gamma (\frac{1}{\varphi })+\frac{1}{\varphi }\ln \varphi$$

The video below attempts this derivation. While its notation and algebraic steps can be a bit difficult to follow, watching it can give you an appreciation for the process. Focus on the final identification of the components rather than the intermediate steps.

Why Does This Matter? Mean and Variance

A beautiful property of the exponential family is that you can derive the mean and variance of the distribution directly from the $b(\theta )$ and $a(\varphi )$ functions:

• Mean: $E(Y)=\mu =b^{\prime }(\theta )$ (the first derivative of $b(\theta )$)
• Variance: $Var(Y)=b^{\prime \prime }(\theta )a(\varphi )$ (the second derivative of $b(\theta )$ times $a(\varphi )$)

Let's check this for the Normal distribution:

• $b(\theta )={\theta }^2/2⟹b^{\prime }(\theta )=\theta$. Since $\theta =\mu$, $E(Y)=\mu$. Correct.
• $b^{\prime \prime }(\theta )=1$. $a(\varphi )=\varphi ={\sigma }^2$. So $Var(Y)=1\cdot {\sigma }^2={\sigma }^2$. Correct.

This powerful result is why the $b(\theta )$ function is so important in GLMs. For your exam, knowing these derivative relationships can be a quick way to check your work.

Conclusion

In this lesson, we built on our previous work by converting the continuous Normal and Gamma distributions into the exponential family form. Crucially, this introduced the concept of a non-trivial dispersion parameter $\varphi$.

Key Takeaways:

• The process of converting a PDF to the exponential family form involves manipulating it to match the structure $\exp \left(\frac{y\theta -b(\theta )}{a(\varphi )}+c(y,\varphi )\right)$.
• For the Normal($\mu ,{\sigma }^2$) distribution:
  • Natural Parameter: $\theta =\mu$
  • Dispersion Parameter: $\varphi ={\sigma }^2$
  • Cumulant Function: $b(\theta )={\theta }^2/2$
• For the Gamma($\alpha ,\beta$) distribution (shape/rate):
  • Natural Parameter: $\theta =-\beta /\alpha$
  • Dispersion Parameter: $\varphi =1/\alpha$
  • Cumulant Function: $b(\theta )=-\ln (-\theta )$
• The mean and variance of any exponential family distribution are given by $E(Y)=b^{\prime }(\theta )$ and $Var(Y)=b^{\prime \prime }(\theta )a(\varphi )$.

In our next lesson, we will tackle the last distribution for this topic: the Negative Binomial. We'll then consolidate this knowledge by learning how to choose the right distribution for a given dataset based on its characteristics, a key skill for your exam.