Probability Mass Functions (PMFs) for Discrete Variables

Hello! Welcome to the next lesson in our course on the mathematical foundations of AI.

In the previous lesson, you learned to define a random variable as a measurable function mapping outcomes from a sample space to real numbers. We distinguished between discrete random variables, which take on a countable set of values, and continuous ones, which take values from an uncountable set like an interval.

This lesson focuses on the first of these two types. We will address the learning outcome: Define and compute probability mass functions (PMFs) for discrete random variables. A PMF is the primary tool for describing the probabilistic behavior of a discrete random variable, assigning a specific probability to each possible value it can take. This concept is the discrete counterpart to the Probability Density Function (PDF) we will encounter later for continuous variables.

1. Defining the Probability Mass Function (PMF)

Let's start by formalizing what a PMF is. For a discrete random variable $X$, we are interested in the probability that it takes on each of its possible numerical values. The function that provides these probabilities is the PMF.

Definition: Let $X$ be a discrete random variable with support (range) $R_X=\{x_1,x_2,x_3,\dots \}$. The Probability Mass Function (PMF) of $X$, denoted $p_X(x)$, is a function that assigns a probability to each possible value of $X$.
$p_X(x)=P(X=x)$
It's crucial to remember the notation from our last lesson: the event $\{X=x\}$ is shorthand for the set of all outcomes $\omega$ in the sample space $\Omega$ such that the random variable function maps that outcome to the value $x$. That is, $\{\omega \in \Omega \mid X(\omega )=x\}$. The PMF, therefore, tells us the total probability of this set of outcomes.

To see this definition in action, let's watch a short video that introduces the PMF, illustrates its construction, and outlines its fundamental properties.

Properties of a PMF

As you saw in the video, any function must satisfy two essential properties to be a valid PMF. For a discrete random variable $X$ with support $R_X$, its PMF $p_X(x)$ must exhibit:

1. Non-negativity: The probability of any event must be non-negative.
   $p_X(x)\ge 0\text{for all }x$
2. Normalization: The sum of the probabilities of all possible outcomes must be 1. Since the values in the support $R_X$ correspond to disjoint events that cover the entire sample space, their probabilities must sum to 1.
   ${\sum }_{x\in R_X}{p}_X(x)=1$

For any value $x$ not in the support $R_X$, the event $X=x$ is impossible, so we define $p_X(x)=0$.

To see these properties stated formally, please review the following short text.

2. Computing a PMF

Now let's establish a systematic procedure for finding the PMF of a discrete random variable $X$ based on an experiment:

1. Define the Sample Space ($\Omega$): List all possible outcomes of the experiment and determine their probabilities.
2. Define the Random Variable ($X$): Specify the function that maps each outcome in $\Omega$ to a real number.
3. Determine the Support ($R_X$): Identify the set of all possible numerical values that $X$ can take.
4. Calculate Probabilities: For each value $x_k$ in the support $R_X$:
   a. Find the subset of outcomes in $\Omega$ that map to $x_k$.
   b. Sum the probabilities of these outcomes to find $p_X(x_k)$.
5. Verify: Check that all $p_X(x_k)$ are non-negative and that they sum to 1.

The following video segment demonstrates this exact process for a random variable defined as the sum of two dice rolls.

3. Common Discrete Distributions

While we can derive a PMF for any discrete random variable from first principles, many common experimental setups give rise to PMFs that have a standard, parameterized form. Recognizing these "named distributions" is a cornerstone of statistical modeling. Your background in computer science will make the "stories" behind these distributions quite intuitive.

The reading below introduces three of the most fundamental discrete distributions.

Let's summarize the PMFs for these three key distributions.

a) The Bernoulli Distribution: Bernoulli(p)

• Story: A single trial with two possible outcomes: "success" (value 1) or "failure" (value 0).
• Parameter: $p$, the probability of success.
• Support: $R_X=\{0,1\}$.
• PMF: $p_X(1)=p$ and $p_X(0)=1-p$. This can be written elegantly as a single formula:
  $p_X(k)=p^k(1-p{)}^{1-k}\text{for }k\in \{0,1\}$

b) The Binomial Distribution: Bin(n, p)

• Story: The number of successes in $n$ independent Bernoulli trials.
• Parameters: $n$ (number of trials) and $p$ (probability of success in each trial).
• Support: $R_X=\{0,1,\dots ,n\}$.
• PMF: The probability of getting exactly $k$ successes is:
  $p_X(k)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}\text{for }k\in \{0,1,\dots ,n\}$
  The term $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is the binomial coefficient, "n choose k", which counts the number of ways to arrange the $k$ successes among the $n$ trials. Given your background, this should be familiar from combinatorics.

c) The Geometric Distribution: Geo(p)

• Story: The number of trials until the first success is observed in a sequence of independent Bernoulli trials.
• Parameter: $p$, the probability of success on any given trial.
• Support: $R_X=\{1,2,3,\dots \}$ (a countably infinite set).
• PMF: The probability that the first success occurs on the $k$-th trial is:
  $p_X(k)=(1-p{)}^{k-1}p\text{for }k\in \{1,2,3,\dots \}$
  This corresponds to having $k-1$ failures, each with probability $1-p$, followed by one success with probability $p$.

Note: Some definitions, including the one in the resource LINK, define the Geometric distribution as the number of failures before the first success. This shifts the support to $\{0,1,2,\dots \}$ and the PMF to $(1-p{)}^{k}p$. Both are valid; it's a matter of convention. We will stick to the "number of trials until success" definition, which has support starting at 1.

Conclusion

In this lesson, we have formally defined the Probability Mass Function (PMF) as the definitive tool for characterizing discrete random variables. You have learned how to derive a PMF from the basic principles of an experiment and have been introduced to the PMFs of three foundational discrete distributions that appear frequently in machine learning and data science.

Key Takeaways:

• The PMF, $p_X(x)=P(X=x)$, gives the probability that a discrete random variable $X$ takes on a specific value $x$.
• A valid PMF must be non-negative for all inputs and its values must sum to 1 over the support of the random variable.
• You can compute a PMF by identifying all outcomes that map to a given value and summing their probabilities.
• The Bernoulli, Binomial, and Geometric distributions are fundamental building blocks with well-defined PMFs that model common probabilistic scenarios.

Preview of the Next Lesson:
With discrete random variables and their PMFs now understood, our next step is to address their continuous counterparts. In the next lesson, we will define and compute Probability Density Functions (PDFs), which describe the probability distributions of continuous random variables. We will see how probabilities for continuous variables are defined over intervals rather than single points, a crucial shift that requires the tools of calculus.