Fundamentals of Probability Theory

Hello! Welcome to the first lesson in our module on the Foundations of Probability Theory.

In the previous modules, we built a robust understanding of linear algebra, multivariate calculus, and optimization theory. We finished by deriving the Karush-Kuhn-Tucker (KKT) conditions, which provide the necessary criteria for identifying optimal solutions in complex, constrained problems. These tools are essential for training machine learning models by minimizing loss functions.

Now, we pivot to the second great pillar of machine learning foundations: probability theory. This field provides the mathematical language to model uncertainty, which is at the heart of nearly every machine learning problem. Today, we will start from first principles to answer the question: What, precisely, is a probability? Our goal is to define probability spaces, sample spaces, events, and the axioms of probability. By the end of this lesson, you will understand the rigorous, axiomatic framework developed by Andrey Kolmogorov that underpins all of modern probability.

1. Experiments, Outcomes, and the Sample Space

At its core, probability theory is concerned with modeling an experiment—any process that has an uncertain outcome. The first step in building a probabilistic model is to define the set of all possible outcomes.

• A sample space, denoted by the Greek letter Omega ($\Omega$), is the set of all possible outcomes of an experiment.
• An individual outcome, or element of the sample space, is often denoted by $\omega$.

Sample spaces can be finite, countably infinite, or uncountably infinite.

• Finite: Rolling a standard six-sided die. $\Omega =\{1,2,3,4,5,6\}$.
• Countably Infinite: Tossing a coin until the first "Heads" appears. $\Omega =\{H,TH,TTH,TTTH,\dots \}$.
• Uncountably Infinite: Measuring the exact height of a randomly selected person in meters. $\Omega =[0,\infty )$.

This initial step seems simple, but it's a crucial part of modeling. The choice of sample space frames the entire problem.

2. Events: The Subsets of Interest

In many situations, we are interested not in a single outcome, but in a set of outcomes. For example, when rolling a die, we might care about the event "the result is even" rather than the specific outcome of rolling a 4.

An event is any subset of the sample space $\Omega$. We say an event has occurred if the outcome of the experiment is an element of that set.

• In the die roll experiment ($\Omega =\{1,2,3,4,5,6\}$), the event "the result is even" corresponds to the set $A=\{2,4,6\}$.
• The event "the result is greater than 4" corresponds to the set $B=\{5,6\}$.

Since events are sets, we can use set operations to define new events:

• $A\cup B$ (A or B): The outcome is in A, in B, or both. For our example, $A\cup B=\{2,4,5,6\}$.
• $A\cap B$ (A and B): The outcome is in both A and B. For our example, $A\cap B=\{6\}$.
• $A^c$ (not A): The outcome is not in A. For our example, $A^c=\{1,3,5\}$.

A critical question arises: should we assign probabilities to individual outcomes or to events? The following video explains why focusing on events is a more general and powerful approach, especially when dealing with continuous sample spaces.

3. The Probability Space: A Formal Structure

Having established the sample space and events, we can now assemble the complete formal structure of a probabilistic model, known as a probability space. As you appreciate mathematical rigor, you will see how this formalization provides a solid foundation for everything that follows.

A probability space is a triplet $(\Omega ,\mathcal{F},P)$, consisting of:

1. $\Omega$: The sample space (the set of all outcomes).
2. $\mathcal{F}$: The event space (a collection of events to which we can assign probabilities).
3. $P$: A probability measure (a function that assigns a probability to each event in $\mathcal{F}$).

Let's examine the two new components, $\mathcal{F}$ and $P$, in detail.

3.1 The Event Space: The $\sigma$-algebra

For finite sample spaces, the event space $\mathcal{F}$ can simply be the power set of $\Omega$ (denoted $2^{\Omega }$), which is the set of all possible subsets of $\Omega$.

However, for uncountable sample spaces (like the real number line), using the power set leads to mathematical paradoxes. It turns out that it's impossible to define a probability measure consistently for all subsets. We must restrict ourselves to a "well-behaved" collection of subsets. This collection is called a $\sigma$-algebra (or sigma-field).

A collection of subsets $\mathcal{F}$ is a $\sigma$-algebra if it satisfies three properties:

1. The sample space itself is an event: $\Omega \in \mathcal{F}$.
2. It is closed under complementation: If $A\in \mathcal{F}$, then its complement $A^c\in \mathcal{F}$.
3. It is closed under countable unions: If $A_1,A_2,\cdots \in \mathcal{F}$, then their union ${\bigcup }_{i=1}^{\infty }{A}_i$ is also in $\mathcal{F}$.

These properties ensure that if we can talk about the probability of certain events, we can also logically talk about the probabilities of their complements, and their countable unions and intersections (via De Morgan's laws).

3.2 The Probability Measure: The Axioms of Probability

The final piece is the probability measure $P$, a function that maps each event in the $\sigma$-algebra $\mathcal{F}$ to a real number. This function is not arbitrary; it must obey three fundamental rules known as the Kolmogorov Axioms.

Formally, a function $P:\mathcal{F}\to \mathbb{R}$ is a probability measure if it satisfies:

1. Axiom 1 (Non-negativity): For any event $A\in \mathcal{F}$, $P(A)\ge 0$.
2. Axiom 2 (Normalization): The probability of the entire sample space is 1, i.e., $P(\Omega )=1$.
3. Axiom 3 (Countable Additivity): For any countable sequence of pairwise disjoint events $A_1,A_2,\dots$ (meaning $A_i\cap A_j=\varnothing$ for $i\ne j$), the probability of their union is the sum of their individual probabilities:
   $P\left({\bigcup }_{i=1}^{\infty }{A}_i\right)={\sum }_{i=1}^{\infty }P(A_i)$

These three simple axioms are the bedrock from which all other properties of probability are derived.

4. Consequences of the Axioms

The power of the axiomatic approach is that we can derive all other familiar rules of probability directly from these three axioms. This is the kind of rigorous deduction you are interested in. Let's prove a couple of fundamental properties.

Let's reconstruct the proof for $P(A^c)=1-P(A)$:

1. By definition, an event $A$ and its complement $A^c$ are disjoint ($A\cap A^c=\varnothing$).
2. Their union is the entire sample space ($A\cup A^c=\Omega$).
3. From Axiom 3 (Additivity), we have $P(A\cup A^c)=P(A)+P(A^c)$.
4. Since $A\cup A^c=\Omega$, we have $P(A\cup A^c)=P(\Omega )$.
5. From Axiom 2 (Normalization), $P(\Omega )=1$.
6. Putting it all together: $P(A)+P(A^c)=1$.
7. Rearranging gives the desired result: $P(A^c)=1-P(A)$.

This small proof demonstrates the elegance of the axiomatic system: a familiar rule is not just an intuitive idea but a logical consequence of the foundational axioms.

Conclusion

In this lesson, we have laid the formal groundwork for our study of probability. We moved from the intuitive idea of an "experiment" to a rigorous mathematical structure—the probability space—that allows us to reason about uncertainty in a consistent way.

Key Takeaways:

• A probability space is a triplet $(\Omega ,\mathcal{F},P)$.
• The sample space $\Omega$ is the set of all possible outcomes of an experiment.
• An event is a subset of the sample space. The collection of measurable events, the event space $\mathcal{F}$, must form a $\sigma$-algebra.
• The probability measure $P$ is a function mapping events to numbers that must satisfy Kolmogorov's three axioms: non-negativity, normalization, and countable additivity.
• This axiomatic framework is the foundation from which all other rules and theorems of probability are derived.

Preview of the Next Lesson:
With the axioms firmly in place, our next step will be to use them to derive some of the most fundamental rules for calculating probabilities. We will cover the addition rule for non-disjoint events, the definition of conditional probability, and the multiplication rule, which are essential tools for practical problem-solving.