Probability as Belief vs. Frequency

1. Two Ways of Thinking: An Analogy

Let's start with a simple analogy that captures the essence of the difference. Imagine you've misplaced your phone at home. You use the "find my phone" feature, and it starts beeping.

• A Frequentist approach to finding your phone would be to listen carefully to the beeps and use only that sound—the data you are currently observing—to determine where the phone is. The reasoning is focused entirely on the properties of the data from this specific "experiment."

• A Bayesian approach would also use the beeping sound, but it would combine that information with your prior knowledge. You know you often leave your phone on the kitchen counter or by your bed. You would integrate the sound of the beep (the data) with your past experience (your prior belief) to narrow down the search area more effectively.

This analogy, adapted from a discussion on Stack Exchange, introduces the core idea: Bayesian reasoning explicitly combines prior knowledge with new evidence, while frequentist reasoning focuses on the properties of a procedure given the evidence at hand.

2. A Tale of Two Probabilities: The Coin Flip

This difference in approach stems from two very different definitions of "probability." To see this in action, let's watch a short video that brilliantly illustrates the divide with a simple coin flip.

As the video explains, once the coin has landed but is still hidden, there are two valid ways to answer the question about the probability of it being heads:

1. The Frequentist Answer: The event is over. The coin is either heads or it is tails. The probability is therefore either 100% or 0%. We just don't know which one. In this view, probability is a statement about the long-run frequency of an event over many repeated trials. It doesn't make sense to talk about the probability of a single event that has already occurred. The underlying state of the world is fixed, even if it's unknown to us.

2. The Bayesian Answer: Probability is a measure of our personal belief or uncertainty about a proposition. Since we don't have the information yet, our state of knowledge is uncertain. Our belief that the coin is heads is 50%. This probability is not a property of the coin itself, but a property of our knowledge about the coin. As we gain more information (i.e., by looking at the coin), our belief will update to 100% or 0%.

This brings us to the core definition for this lesson.

Bayesian probability is a 'degree of belief.' It quantifies our uncertainty about something. This 'something' can be the outcome of a coin flip, a model parameter in a machine learning algorithm, or whether a scientific hypothesis is true.

3. When Long-Run Frequencies Fall Short

The frequentist definition of probability as a long-run frequency is powerful and works well for repeatable events like coin flips or card games. But what about questions where no repeated experiment is possible?

Consider the following example from the textbook The Art of Insight in Science and Engineering.

The question, "How sure am I that the candidate number is my phone number?" is a question about a single, fixed state of the world. The number is either correct or it is not. There is no "long-run frequency" of it being correct. The only way to answer this is to express a degree of belief, which is precisely what the Bayesian framework is designed to do.

4. What is Fixed and What is Random?

We can summarize the philosophical divide with a simple diagram and a key distinction.

This image highlights a subtle but profound difference:

• Frequentist: Treats the theory or model parameter as a fixed, unknown constant. The data is a random sample from a repeatable process. The key question is: "How likely are my observed data, assuming my theory is true?" This is why you calculate things like p-values: P(observing this data or more extreme | null hypothesis is true).

• Bayesian: Treats the data as fixed (it's what you observed). The theory or model parameter is the uncertain quantity. The key question is: "How plausible is my theory, now that I have seen this data?" You make direct probability statements about the parameter: P(theory | observed data).

This leads to a very useful, albeit crude, summary you can find in many statistics discussions:

Frequentist: Parameters are fixed; data is random.
Bayesian: Data is fixed; parameters are random (i.e., uncertain).

To hear this distinction explained clearly, you can watch a short segment from another video.

Conclusion

In this introductory lesson, we've established the fundamental philosophical divide between frequentist and Bayesian statistics. This isn't just an academic curiosity; it has profound implications for how we frame questions, conduct analyses, and interpret results.

Key Takeaways:

• Frequentist Probability = Long-Run Frequency: It's an objective property of a system, measured by repeating an experiment many times. Parameters are considered fixed constants.
• Bayesian Probability = Degree of Belief: It's a subjective measure of our uncertainty about a statement, which depends on our current knowledge. Parameters are treated as uncertain quantities about which we can have beliefs.
• The core difference lies in what is considered random and what is considered fixed. For Frequentists, parameters are fixed and data are random. For Bayesians, the observed data are fixed, and the parameters are the uncertain quantities we reason about.

You've probably noticed that the Bayesian interpretation often aligns with how we intuitively talk about probability in everyday life. One of the great appeals of the Bayesian framework is that it allows us to make direct probability statements about the things we are most interested in—our hypotheses and model parameters.

Next time: We will explore how this "degree of belief" definition allows us to use the basic rules of probability—the sum and product rules—to logically update our beliefs as we collect more evidence. This will lead us directly to the cornerstone of the entire field: Bayes' theorem.