Neyman-Fisher Factorization Theorem

In our previous lesson, we established the log-likelihood function as the cornerstone of Maximum Likelihood Estimation, noting its mathematical convenience and numerical stability. Today, we'll build on that foundation by exploring a fundamental question: does the entire dataset need to be carried through our analysis, or can a simpler summary suffice? This leads us to the powerful concept of sufficiency.

Our goal in this lesson is to introduce the Neyman-Fisher factorization theorem, a practical tool for identifying statistics that perfectly summarize all the information a sample contains about a parameter. Understanding this will allow us to simplify complex data without losing inferential power.

What is a Sufficient Statistic?

Imagine two statisticians, A and B, who both want to estimate a parameter $θ$ from the same dataset $x = (x_{1}, \dots, x_{n})$ .

Statistician A has access to the complete dataset, all $n$ individual data points.
Statistician B is only given a single number, which is a function of the data, such as the sum or the average. This function of the data is called a statistic.

If Statistician B can perform the estimation of $θ$ just as well as Statistician A, despite having "less" information, then the statistic they were given is considered sufficient. It contains all the relevant information about $θ$ that was present in the original data. The word "sufficient" is quite literal here—the statistic is enough.

For instance, when estimating the parameter $p$ (probability of heads) from a series of Bernoulli trials (coin flips), knowing the total number of heads is just as good as knowing the exact sequence of heads and tails. The total count is a sufficient statistic.

Properties of Estimators III 7.8.1 Sufficiency

To formalize this idea, let's turn to this chapter from Alex Tsun's "Probability & Statistics with Applications to Computing".

First, read the introductory section starting from the beginning to grasp the intuitive explanation using the two-statistician analogy. Then, focus on the formal definition under Definition 7.8.2.

The formal definition states that a statistic $T (X)$ is sufficient for $θ$ if the conditional distribution of the data $X$ , given the value of the statistic $T (X) = t$ , does not depend on $θ$ .

P (X = x_{1}, \dots, x_{n} ∣ T = t, θ) = P (X = x_{1}, \dots, x_{n} ∣ T = t)

In simple terms, once you know the value of the sufficient statistic $t$ , the parameter $θ$ no longer tells you anything more about the probability of having observed your specific dataset. All of $θ$ 's influence is channeled through the statistic $T$ .

The Neyman-Fisher Factorization Theorem

While the formal definition is theoretically sound, it can be difficult to apply directly. This is where the Neyman-Fisher factorization theorem comes in. It provides a straightforward and powerful criterion to check for sufficiency.

The Theorem: A statistic $T (X_{1}, \dots, X_{n})$ is sufficient for a parameter $θ$ if and only if the likelihood function $L (θ ∣ x_{1}, \dots, x_{n})$ can be factored into two non-negative functions:

L (θ ∣ x_{1}, \dots, x_{n}) = g (x_{1}, \dots, x_{n}) \cdot h (T (x_{1}, \dots, x_{n}), θ)

Let's break down the two parts of this product:

$h (T (x), θ)$ : This function contains the parameter $θ$ . Crucially, it depends on the data only through the statistic $T (x)$ . All interaction between the parameter and the data happens within this function, mediated by the sufficient statistic.
$g (x_{1}, \dots, x_{n})$ : This function can depend on the full dataset, but it must not depend on the parameter $θ$ . It represents the part of the likelihood that is irrelevant for estimating $θ$ . Sometimes, this function is just a constant (e.g., $g (x) = 1$ ).

The following video provides an excellent walkthrough of the theorem and a practical strategy for applying it.

Sufficient Statistics and the Factorization Theorem

In this video from "Stats with Brian", the factorization theorem is presented as the primary method for finding sufficient statistics.

First, watch the section that formally states the theorem and contrasts it with the conditional probability definition. Then, observe the strategy in action with the Poisson distribution example. The key strategy here is to algebraically rearrange the likelihood function to "drag the thetas to the left", separating them from the parts of the data that don't interact with them. Finally, watch the summary at the end.

Applying the Theorem: Examples

As you saw in the video, applying the theorem is a process of algebraic manipulation of the likelihood function. Let's work through the Bernoulli case, which you'll recall from our last lesson.

Example: Bernoulli Distribution

For $n$ iid samples from a Bernoulli( $p$ ) distribution, the likelihood function is:

L (p ∣ x_{1}, \dots, x_{n}) = i = 1 \prod n p^{x_{i}} (1 - p)^{1 - x_{i}} = p^{\sum x_{i}} (1 - p)^{n - \sum x_{i}}

This expression is already perfectly factored for us! Let's identify the components:

Let the statistic be $T (x) = \sum_{i = 1}^{n} x_{i}$ , which is the total number of successes.
Let $h (T (x), p) = p^{T (x)} (1 - p)^{n - T (x)}$ . This function depends on the data only through the sum $T (x)$ .
Let $g (x_{1}, \dots, x_{n}) = 1$ . This function does not depend on $p$ .

Since we have successfully factored the likelihood according to the theorem, we conclude that $T (X) = \sum X_{i}$ is a sufficient statistic for $p$ . This confirms our intuition: to estimate the probability of a coin landing heads, you only need to know the total number of heads in your sample, not the specific order in which they appeared.

The document we looked at earlier provides more clear, worked-out examples for other common distributions.

Properties of Estimators III 7.8.1 Sufficiency

Now, let's solidify your understanding by reading through the examples in the same Stanford document. This will show you how the factorization is applied to different likelihoods.

Read the three examples provided in the document. Follow the derivation for each: The Uniform distribution, which shows a continuous example where the sufficient statistic is the maximum value. The Poisson distribution, which reinforces the example from the video. Note how the g(x) term isolates the factorials. The Bernoulli distribution, which recaps what we just discussed. Pay close attention in each case to how the likelihood is algebraically separated into the g(x) and h(T(x), \theta) components.

Conclusion

In this lesson, we moved from using the full likelihood function to understanding how to distill the essential information from our data into a sufficient statistic. This is a crucial data reduction step in statistics that loses no information about the parameter of interest.

Our key takeaways are:

A statistic is any function of the data. A statistic is sufficient if it contains all the information available in the sample about the parameter $θ$ .
The Neyman-Fisher Factorization Theorem is the workhorse for identifying sufficient statistics.
To apply the theorem, you must show that the likelihood function $L (θ ∣ x)$ can be factored into two parts: $g (x)$ , which is independent of $θ$ , and $h (T (x), θ)$ , where the data $x$ and parameter $θ$ interact only through the statistic $T (x)$ .

So far, we've identified sufficient statistics and seen that they are "enough." In the next lesson, we will explore this idea further by asking what it means for a statistic to be a complete summary of the evidence in the data. This will lead us to formally interpret the meaning of sufficiency and introduce the related Likelihood Principle, which has profound implications for statistical inference.

Recommended resources

What is a Sufficient Statistic?

The Neyman-Fisher Factorization Theorem

Applying the Theorem: Examples

Example: Bernoulli Distribution

Conclusion