Exponential Family Forms of Binomial and Poisson Distributions

The Exponential Family: A Standard Form for Distributions

The power of GLMs comes from the fact that many common probability distributions—including the Normal, Binomial, Poisson, and Gamma—are members of the exponential family. This means their probability density function (PDF) or probability mass function (PMF) can be written in a specific, standardized form.

A distribution belongs to the exponential family if its PMF or PDF, $f(y;\theta )$, can be expressed as:

Let's break down these components:

• $y$: The observed data or a sufficient statistic (a function of the data that summarizes all the relevant information).
• $\theta$: The natural parameter (or canonical parameter). It's a function of the distribution's original mean parameter (like $p$ in the Binomial or $\lambda$ in the Poisson).
• $\varphi$: The dispersion parameter. For the Binomial and Poisson distributions, this is fixed at 1, so the $a(\varphi )$ term simplifies to 1. We will explore this more in the next lesson.
• $b(\theta )$: A function that depends only on the natural parameter $\theta$. It's often called the cumulant function and is crucial for finding the mean and variance of the distribution.
• $c(y,\varphi )$: A function that depends only on the data $y$ and the dispersion parameter $\varphi$. It acts as a normalizing constant.

Our goal is to take a familiar PMF, like for the Binomial distribution, and algebraically manipulate it until it matches this structure.

The Algebraic Strategy

The most direct way to convert a PMF/PDF into this form is to work with its logarithm. This turns the products in the original PMF into sums, which are easier to rearrange.

1. Write down the log-likelihood: Start with the PMF, $f(y)$, and take its natural logarithm, $\ln (f(y))$. This is the log-likelihood, $l$.
2. Rearrange and Isolate: Algebraically rearrange the log-likelihood to match the form $y\theta -b(\theta )+c(y)$. (We can ignore $\varphi$ for now).
3. Identify Components: Once the expression is in the target form, you can directly identify $\theta$, $b(\theta )$, and $c(y)$.

Let's apply this strategy.

Worked Example 1: The Binomial Distribution

Your goal is to show that the Binomial distribution is in the exponential family and identify its components.

First, let's review the step-by-step derivation in a clear, formal text. The following resource works through the log-likelihood, which is the exact method we will use.

Let's summarize the key steps from that reading.

Step 1: Write the log-likelihood

The PMF for a Binomial random variable $Y$ is $f(y;\pi )=\left(\genfrac{}{}{0ex}{}{n}{y}\right){\pi }^y(1-\pi {)}^{n-y}$, where we use $\pi$ for the probability of success to avoid confusion with p-values.

The log-likelihood, ignoring the constant term $\ln \left(\genfrac{}{}{0ex}{}{n}{y}\right)$ which doesn't involve the parameter $\pi$, is:

Step 2: Rearrange and Isolate

We want to group all terms involving the data $y$ together:

Step 3: Identify Components

Now, we match this to the structure $y\theta -b(\theta )$.

• Natural Parameter ($\theta$): The term multiplying $y$ is our natural parameter.

  $$\theta =\ln \left(\frac{\pi }{1-\pi }\right)$$

  You might recognize this as the logit function. This is precisely why the logit is the canonical link for the Binomial distribution!

• Cumulant Function ($b(\theta )$): To find $b(\theta )$, we first need to express the remaining part, $-n\ln (1-\pi )$, in terms of $\theta$. As shown in the reading, we first solve for $\pi$ in terms of $\theta$:

  $$\pi =\frac{e^{\theta }}{1+e^{\theta }}\text{and}1-\pi =\frac{1}{1+e^{\theta }}$$

  Substituting this back gives:

  $$b(\theta )=-n\ln (1-\pi )=-n\ln \left(\frac{1}{1+e^{\theta }}\right)=n\ln (1+e^{\theta })$$

The other components are $a(\varphi )=1$ and $c(y,\varphi )=\ln \left(\genfrac{}{}{0ex}{}{n}{y}\right)$.

The following image provides a great visual summary of this entire derivation.

For a different perspective, the following video walks through the same process but uses a slightly different algebraic path (exponentiating earlier). Watching it can help solidify your understanding.

Worked Example 2: The Poisson Distribution

Now, let's repeat the process for the Poisson distribution. This one is more straightforward.

Again, let's start with the formal derivation from the same set of lecture notes.

Here is a summary of that process:

Step 1: Write the log-likelihood

The PMF for a Poisson random variable $Y$ is $f(y;\lambda )=\frac{{\lambda }^y{e}^{-\lambda }}{y!}$.

The log-likelihood, ignoring the constant term $-\ln (y!)$, is:

Step 2 & 3: Rearrange and Identify

This expression is already in the form $y\theta -b(\theta )$!

• Natural Parameter ($\theta$): The term multiplying $y$ is $\ln (\lambda )$.

  $$\theta =\ln (\lambda )$$

  This is the log function, which is the canonical link for the Poisson distribution.

• Cumulant Function ($b(\theta )$): The remaining term is $\lambda$. We must express it in terms of $\theta$. Since $\theta =\ln (\lambda )$, it follows that $\lambda =e^{\theta }$.

  $$b(\theta )=\lambda =e^{\theta }$$

The other components are $a(\varphi )=1$ and $c(y,\varphi )=-\ln (y!)$.

To reinforce this, you can watch the following video walkthrough.

Conclusion

In this lesson, we demystified the mathematical structure that unifies different models within the GLM framework. You practiced the essential exam skill of converting the Binomial and Poisson PMFs into the exponential family form.

Key Takeaways:

• Distributions in the exponential family can be written in a standard form: $f(y)=\exp \left(\frac{y\theta -b(\theta )}{a(\varphi )}+c(y,\varphi )\right)$.
• A reliable method for conversion is to take the natural log of the PMF/PDF, rearrange the terms to isolate the data $y$, and identify the components $\theta$ and $b(\theta )$.
• For the Binomial distribution, the natural parameter is $\theta =\text{logit}(\pi )$, and the cumulant function is $b(\theta )=n\ln (1+e^{\theta })$.
• For the Poisson distribution, the natural parameter is $\theta =\ln (\lambda )$, and the cumulant function is $b(\theta )=e^{\theta }$.
• The form of the natural parameter $\theta$ directly gives rise to the canonical link function for that distribution.

In the next lesson, we will extend this skill to continuous distributions, specifically the Normal and Gamma distributions. This will introduce the role of the dispersion parameter, $\varphi$, which we have set aside for today.