Sufficient Statistics with the Factorization Theorem

A Strategy for Factorization

Applying the Neyman-Fisher theorem is an exercise in algebraic manipulation. The general strategy involves three main steps:

1. Write the Likelihood Function: For an independent and identically distributed (iid) sample $x_1,\dots ,x_n$, the likelihood is the joint probability mass/density function, which is the product of the individual functions: $L(\theta |x)={\prod }_{i=1}^{n}f(x_i;\theta )$.
2. Simplify and Collect Terms: Use algebraic rules (especially for exponents and products) to consolidate the expression. The goal is to gather all terms involving the parameter $\theta$.
3. Factor the Expression: Separate the simplified likelihood into the two components required by the theorem:
   • A function $h(T(x),\theta )$, which contains the parameter $\theta$ and depends on the data only through a statistic $T(x)$.
   • A function $g(x_1,\dots ,x_n)$, which can depend on the full dataset but must not depend on $\theta$.

Let's apply this strategy to a familiar example.

Applying the Theorem: The Poisson Distribution

Suppose we have a random sample $X_1,X_2,\dots ,X_n$ from a Poisson distribution with parameter $\lambda$. Our goal is to find a sufficient statistic for $\lambda$.

First, let's walk through the factorization process together by reading a clear, step-by-step example.

As the text demonstrates, the joint probability mass function (the likelihood) is:

By combining terms, we arrive at:

This expression is now perfectly factored.

Let's identify our components:

• Statistic: $T(X)={\sum }_{i=1}^n{X}_i$
• Function $h$: $h(T(x),\lambda )=e^{-n\lambda }{\lambda }^{T(x)}$. This function depends on the data only through the sum $T(x)$.
• Function $g$: $g(x_1,\dots ,x_n)=\frac{1}{{\prod }_{i=1}^n{x}_i!}$. This function does not depend on $\lambda$.

Since we have successfully factored the likelihood, the Neyman-Fisher theorem confirms that $T(X)=\sum X_i$ is a sufficient statistic for $\lambda$.

Practice with a Continuous Distribution: The Normal Mean

Now let's tackle a continuous distribution. Consider a random sample $X_1,\dots ,X_n$ from a Normal distribution with an unknown mean $\mu$ and a known variance, which we'll set to 1 for simplicity (i.e., $X_i\sim N(\mu ,1)$).

Can you apply the three-step strategy to find a sufficient statistic for $\mu$?

1. Write the Likelihood: Start with the joint PDF, $L(\mu |x)={\prod }_{i=1}^{n}f(x_i;\mu )$.
2. Simplify: Combine the exponential terms.
3. Factor: This is the trickiest part. The expression inside the exponent, $\sum (x_i-\mu {)}^2$, needs to be rearranged to isolate $\mu$ with a statistic.

The following video and text resource explain the key algebraic manipulation needed to complete the factorization. After attempting it yourself, use these to check your work or to get unstuck.

The crucial step, as shown in the video and also detailed in the Penn State resource you looked at earlier, is rewriting the sum of squares:

Expanding this and using the fact that $\sum (x_i-\bar{x})=0$, the likelihood function separates nicely.

After the algebra, the likelihood function factors into:

From this, we can identify:

• Statistic: $T(X)=\bar{X}$ (the sample mean)
• Function $h$: $h(T(x),\mu )=\exp \left[-\frac{n}{2}(T(x)-\mu {)}^2\right]$
• Function $g$: $g(x)=\frac{1}{(2\pi {)}^{n/2}}\exp \left[-\frac{1}{2}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2\right]$

Thus, the sample mean $\bar{X}$ is a sufficient statistic for the population mean $\mu$.

Are Sufficient Statistics Unique?

You may have noticed in the Poisson example that since $\sum X_i=n\bar{X}$, we could have just as easily written the $h$ function in terms of $\bar{X}$. Does this mean $\bar{X}$ is also sufficient? Yes.

A sufficient statistic is not unique. In general, if $T(X)$ is a sufficient statistic, then any one-to-one function of $T(X)$ is also a sufficient statistic.

• A function $f$ is one-to-one if for every possible output value, there is only one unique input value that produces it. For example, $f(t)=nt$ is one-to-one. If you know $\sum X_i$, you know $\bar{X}$, and vice-versa.
• A function like $f(t)=t^2$ is not one-to-one (for real numbers), because knowing $t^2=4$ doesn't tell you if $t$ was 2 or -2.

Therefore, for the Poisson and Normal examples, both the sum $\sum X_i$ and the sample mean $\bar{X}$ are sufficient statistics for their respective parameters.

Conclusion

In this lesson, you put the Neyman-Fisher Factorization Theorem to work. The key has been to see the theorem not just as a definition, but as an instruction set for manipulating the likelihood function.

Key Takeaways:

• Applying the factorization theorem follows a clear process: write the likelihood, simplify, and algebraically factor it into the required $g(x)$ and $h(T(x),\theta )$ components.
• The goal of the algebraic manipulation is to isolate the parameter $\theta$ such that it only interacts with the data through a single function, the statistic $T(x)$.
• Sufficient statistics are not unique; any one-to-one transformation of a sufficient statistic is also sufficient. For many common distributions, both the sum and the sample mean are sufficient.

You have now solidified your understanding of how to find a sufficient statistic. This concept of data reduction—summarizing a potentially massive dataset into one or a few numbers without losing information about the parameter—is central to statistical theory. In our next lesson, we will explore this idea further by introducing the Likelihood Principle, which formalizes the idea that all evidence in a sample is contained in the likelihood function, a concept closely tied to sufficiency.