The Likelihood Principle and Its Implications

What's the Evidence? A Tale of Two Statisticians

Let's begin with a classic thought experiment, presented here in the form of a popular webcomic.

In the comic, two statisticians get a "YES" answer from a machine designed to detect if the sun has gone nova. The machine is mostly reliable, but it lies 1/36 of the time (when two dice come up six).

• The Frequentist calculates the probability of getting this result by chance, assuming the sun has not exploded. This probability is simply the chance the machine lies, which is $1/36\approx 0.027$. Since this p-value is less than the conventional 0.05 threshold, the frequentist rejects the "null hypothesis" (that the sun is fine) and concludes it has exploded.
• The Bayesian, on the other hand, is willing to bet $50 that the sun has not exploded. Why? Because their reasoning incorporates an extremely strong prior belief: the sun exploding is an incredibly rare event. They weigh the evidence from the detector against this prior knowledge.

The frequentist's conclusion depends on the properties of the procedure and considers outcomes that did not happen (i.e., the dice could have rolled something else). The Bayesian's conclusion, however, focuses only on the data observed in the context of prior knowledge. The Likelihood Principle lies at the heart of this distinction.

Stating the Likelihood Principle

The Likelihood Principle (LP) gives a formal answer to the question: "What part of the experimental result constitutes evidence about the parameter $\theta$?" The answer, it claims, is simple: everything is in the likelihood function, and nothing else matters.

More formally, the Strong Likelihood Principle (SLP) compares the evidence from two potentially different experiments.

In essence, the principle states that if two different experiments, $E_1$ and $E_2$, produce outcomes $x_1$ and $x_2$ such that their likelihood functions are proportional, then they provide the exact same evidence about $\theta$.

The constant of proportionality $c$ can depend on the data ($x_1,x_2$) but crucially, not on the parameter $\theta$. This means the shape of the likelihood function is all that matters, not its absolute magnitude or the experimental procedure that generated it.

This might seem like a bold claim. Its justification comes from a landmark result by Allan Birnbaum, who showed that the Likelihood Principle is the logical consequence of two simpler, widely accepted principles: the Weak Sufficiency Principle and the Weak Conditionality Principle. The implication is that if you accept those two "self-evident" principles, you are logically forced to accept the Likelihood Principle.

Implications for Statistical Inference

Accepting the Likelihood Principle has major consequences for how we analyze data. Many standard statistical procedures, particularly in the frequentist school, are incompatible with it.

1. Justification for the Sufficiency Principle

In our last lesson, we defined a sufficient statistic $T(X)$ as one that captures all the information about $\theta$. The Likelihood Principle provides the formal justification for this. By the Neyman-Fisher factorization, the likelihood can be written as $L(\theta |x)=h(T(x),\theta )\cdot g(x)$. If two data points $x_1$ and $x_2$ have the same value for the sufficient statistic, $s(x_1)=s(x_2)$, their likelihood functions are proportional (with the constant being the ratio $g(x_1)/g(x_2)$). According to the LP, the evidence must therefore be the same.

2. The Stopping Rule Principle

One of the most striking implications of the LP is that the reason an experiment was stopped should have no bearing on the inference drawn from the data. This is known as the Stopping Rule Principle.

Consider two scenarios for an experiment on a Bernoulli parameter $\theta$ (e.g., probability of heads):

1. Experiment 1 (Binomial): An engineer decides to test 12 microchips and count the number of failures. He observes 3 failures.
2. Experiment 2 (Negative Binomial): Another engineer decides to test microchips until she finds 9 that pass. This happens on the 12th test, meaning she also observed 3 failures.

The data are identical: 3 failures and 9 successes in 12 trials. The likelihood functions are:

• Binomial: $L_1(\theta )=\left(\genfrac{}{}{0ex}{}{12}{3}\right){\theta }^3(1-\theta {)}^9=220\cdot {\theta }^3(1-\theta {)}^9$
• Negative Binomial: $L_2(\theta )=\left(\genfrac{}{}{0ex}{}{11}{2}\right){\theta }^3(1-\theta {)}^9=55\cdot {\theta }^3(1-\theta {)}^9$

Notice that $L_1(\theta )=4\cdot L_2(\theta )$. The likelihood functions are proportional. According to the Likelihood Principle, the conclusion about $\theta$ must be identical in both cases.

However, in a traditional frequentist analysis, the conclusions might differ. Why? Because a p-value is the probability of observing data as extreme or more extreme than what was actually seen, assuming a null hypothesis is true. The set of "more extreme" outcomes depends on the stopping rule.

As you read, a frequentist testing $H_0:\theta =0.5$ would calculate different p-values because the sample spaces are different. In the Binomial case, "more extreme" means fewer than 3 successes, while in the Negative Binomial case, it means taking more than 12 trials to get 9 successes. The resulting p-values lead to different conclusions at the 5% significance level, a direct violation of the Likelihood Principle.

To a practitioner who accepts the LP, this is deeply problematic. The evidence from the 3 failures and 9 successes should not depend on the private intentions of the engineer running the test.

3. The Bayesian-Frequentist Divide

This brings us back to our opening comic. The Likelihood Principle is a natural fit for Bayesian inference but is often at odds with frequentist methods.

• Bayesian Inference: The core of Bayesian updating is Bayes' Theorem: $\pi (\theta |x)\propto L(\theta |x)\pi (\theta )$. The posterior distribution is a direct function of the likelihood. If two experiments yield proportional likelihoods, $L_1\propto L_2$, and use the same prior $\pi (\theta )$, their posterior distributions will be identical. Therefore, all subsequent inferences (posterior means, credible intervals, etc.) will also be identical. Bayesian analysis inherently respects the LP.

• Frequentist Inference: As we saw with the stopping rule example, many frequentist tools—such as p-values and confidence intervals—are defined by considering the long-run properties of a procedure over all possible datasets that could have been generated. Their calculation explicitly involves parts of the sample space where $x$ was not observed. This dependence on the experimental design and unobserved data is a violation of the LP.

The comic highlights this perfectly. The frequentist relies on a p-value ($p\approx 0.027$), which is calculated based on the properties of the detector's randomizer. The Bayesian effectively ignores the detector's design and focuses on the likelihood of the two competing hypotheses ("sun exploded" vs. "sun fine") given the "YES" signal, combined with a very strong prior belief that the sun has not exploded.

Conclusion

This lesson introduced one of the most fundamental and debated ideas in statistical theory. The Likelihood Principle asserts that all the evidence obtained from an experiment about an unknown parameter is fully contained in the likelihood function of the observed data.

Key Takeaways:

• The Likelihood Principle (LP): If two experiments yield proportional likelihood functions for the same parameter, the evidence about that parameter is identical.
• Implications: The LP provides a formal justification for the Sufficiency Principle and implies the Stopping Rule Principle (i.e., inference should not depend on why data collection ceased).
• Philosophical Divide: The LP highlights a core difference between statistical philosophies. Bayesian methods are naturally compliant with the LP, whereas many classical frequentist procedures (like p-value significance testing) violate it by design, as they depend on the sample space of unobserved outcomes.

Understanding the Likelihood Principle is crucial as it shapes our approach to modeling and interpretation. It forces us to decide whether inference should be based only on the data we saw, or also on the data we might have seen.

This concludes our first module on the foundations of likelihood. We have defined the likelihood function, learned how to use it to find sufficient statistics, and explored the principle that it contains all the evidence. In the next module, we will begin to quantify this evidence by examining the geometry of the likelihood function itself. We will start by defining the score function, the gradient of the log-likelihood, which tells us the direction in which the evidence is pointing.