Probability Density Functions for Continuous Random Variables

Hello! Welcome back to our course on the mathematical foundations of AI.

In our last lesson, we explored Probability Mass Functions (PMFs), which perfectly describe discrete random variables by assigning a specific probability to each possible value. We saw that for a discrete variable $X$, the probability $P(X=x)$ is often positive and meaningful.

Today, we shift our focus to continuous random variables. As we've touched upon, for any continuous variable $X$, the probability of it taking on any single exact value is zero, i.e., $P(X=x)=0$. This renders the PMF useless, as it would simply be a function that is zero everywhere. To handle this, we need a new concept.

This lesson addresses the learning outcome: Define and compute probability density functions (PDFs) for continuous random variables. We will develop the idea of "probability density" and see how it allows us to characterize the distribution of a continuous random variable using the tools of calculus.

1. From Probability Mass to Probability Density

For a discrete random variable, we can think of the PMF as distributing a total probability mass of 1 among a set of distinct points. For a continuous random variable, this mass is spread smoothly over an interval, like a fine layer of dust. Instead of asking about the mass at a single point (which is zero), we ask about the density of the mass at that point.

The following video provides an excellent analogy to build this intuition, contrasting the "point masses" of a PMF with the "spread-out mass" of a PDF.

This idea of "density" can be formalized by thinking about the probability of the variable falling into a very small interval around a point $x$. Let's consider the probability that our random variable $X$ falls in the interval $(x,x+\Delta ]$ for a very small $\Delta >0$. While this probability, $P(x<X\le x+\Delta )$, approaches zero as $\Delta$ approaches zero, the ratio of the probability to the interval's length can approach a finite, non-zero value. This ratio is the probability density.

The Probability Density Function (PDF), denoted $f_X(x)$, is defined as this limit:
$f_X(x)={\lim }_{\Delta \to 0^{+}}\frac{P(x<X\le x+\Delta )}{\Delta }$
This gives us an interpretation of the PDF as probability per unit length. For a small $\Delta$, we can approximate the probability of a small interval as:
$P(x<X\le x+\Delta )\approx f_X(x)\cdot \Delta$

2. The Formal Definition of a PDF

Your background in calculus will be useful here. We can connect the PDF directly to another fundamental concept: the Cumulative Distribution Function (CDF).

The CDF, which we will explore in depth in the next lesson, is defined as $F_X(x)=P(X\le x)$. It represents the total accumulated probability up to the value $x$. Using this, we can write the probability of our small interval as:
$P(x<X\le x+\Delta )=F_X(x+\Delta )-F_X(x)$

Substituting this into our limit definition for the PDF gives:
$f_X(x)={\lim }_{\Delta \to 0}\frac{F_X(x+\Delta )-F_X(x)}{\Delta }$
You will immediately recognize this as the definition of the derivative.

Definition: For a continuous random variable $X$ with a differentiable CDF $F_X(x)$, the Probability Density Function (PDF) $f_X(x)$ is the derivative of the CDF.
$f_X(x)=\frac{d{F}_X(x)}{dx}=F_X^{\prime }(x)$
The following video walks through this derivation and introduces the key properties of the PDF that stem from this definition.

3. Properties of the Probability Density Function

As you just saw in the video, and as a direct consequence of its definition, any valid PDF $f_X(x)$ must satisfy a few fundamental properties. These are the continuous analogues of the PMF properties.

For a continuous random variable $X$ with PDF $f_X(x)$:

1. Non-negativity: The density can't be negative.
   $f_X(x)\ge 0\text{ for all }x$
   Important: Unlike a PMF, it is possible for $f_X(x)>1$. This is because the PDF represents a density, not a probability. For example, if probability is concentrated in an interval of length 0.1, the density in that region could be around 10 to ensure the total probability integrates to 1.

2. Normalization: The total area under the PDF curve must equal 1. This is because the probability of the random variable taking some value on the real line is 1.
   ${\int }_{-\infty }^{\infty }{f}_X(x)dx=1$

3. Probability Calculation: By the Fundamental Theorem of Calculus, the probability that $X$ falls within an interval $[a,b]$ is the integral of the PDF over that interval.
   $P(a\le X\le b)={\int }_a^b{f}_X(x)dx=F_X(b)-F_X(a)$
   Since $P(X=a)=P(X=b)=0$, it makes no difference whether the interval is open or closed.
   $P(a\le X\le b)=P(a<X<b)=P(a\le X<b)=P(a<X\le b)$

You can also read through a concise summary of these concepts in the text that accompanies the video you just watched.

4. A Worked Example: Computing with PDFs

Let's put these concepts into practice. A common task is to be given the functional form of a PDF with an unknown constant and then use it to find probabilities. The following example, involving the exponential distribution, is a classic.

We'll follow three steps:

1. Find the unknown constant using the normalization property.
2. Find the CDF by integrating the PDF.
3. Calculate the probability of an interval using both the PDF (integration) and the CDF (subtraction).

The final segment of the video from Probability Course provides a perfect, step-by-step walkthrough.

Conclusion

In this lesson, we transitioned from the discrete world of PMFs to the continuous world of PDFs. We established that for continuous random variables, probability is not found at points but is measured over intervals as an area under a curve.

Key Takeaways:

• A Probability Density Function (PDF), $f_X(x)$, describes the distribution of a continuous random variable. It represents probability density, not probability itself.
• The PDF is formally defined as the derivative of the CDF: $f_X(x)=F_X^{\prime }(x)$.
• A valid PDF must be non-negative ($f_X(x)\ge 0$) and normalize to one (${\int }_{-\infty }^{\infty }{f}_X(x)dx=1$).
• Probabilities for intervals are calculated by integrating the PDF: $P(a\le X\le b)={\int }_a^b{f}_X(x)dx$.

Preview of the Next Lesson:
We briefly introduced the Cumulative Distribution Function (CDF) today as a means to define the PDF. In the next lesson, we will give the CDF its full attention. We will formally define it for both discrete and continuous variables, explore its universal properties, and see why it is, in many ways, an even more fundamental description of a random variable's distribution than either the PMF or the PDF.