Probability and Random Variables in Deep Learning

Welcome to our new module on probability and statistics for generative models. In our previous module, we delved deep into the mechanics of optimization, culminating in an understanding of why the stochasticity in SGD is a crucial feature, not a bug, for navigating the complex, high-dimensional loss landscapes of neural networks. We saw how random noise helps escape the ubiquitous saddle points that would otherwise stall training.

This lesson marks our formal entry into the world of probability, providing the language to precisely describe and reason about the "stochastic" and "random" phenomena we've so far only treated intuitively. Our goal is to refresh the fundamentals of probability spaces and random variables, framing them through deep learning examples like stochastic mini-batches and dropout. For someone with your mathematical background, this will be a brisk but foundational review, aimed at solidifying the conceptual bridge between pure probability theory and its application in modern machine learning.

1. The Language of Chance: Probability Spaces

At its core, probability theory provides a formal framework for reasoning about uncertainty. This framework is built upon a triplet of concepts known as a probability space, which consists of a sample space, an event space, and a probability measure.

Let's watch a short video from the Intelligent Systems Lab that introduces these three ingredients.

Probability spaces and random variables

This video provides a conceptual overview of the three components of a probability space and explains their roles in building a formal model of an experiment.

Please watch the introduction and the segments covering the three core concepts: The three ingredients: sample space (\Omega), event space (\mathcal{A}), and probability measure (P). The sample space: the set of all possible outcomes. The event space: a collection of "interesting" subsets of outcomes that we can assign probabilities to. Pay attention to the brief mention of a \sigma-algebra; you don't need to dive into the measure-theoretic details, just appreciate that it's a mathematically consistent way to handle collections of events. The probability measure: the function that assigns a value in [0, 1] to each event.

Now, let's formalize these ideas and connect them directly to a deep learning context. The Stanford CS229 review notes provide a concise summary of these definitions.

Review of Probability Theory - CS229

This section provides the formal definitions for the elements of probability we just saw in the video.

Read Section 1, "Elements of probability," focusing on the definitions of Sample space \Omega, Set of events \mathcal{F}, and Probability measure P.

Let's map these concepts to the familiar process of stochastic mini-batch sampling:

Sample Space ( $Ω$ ): This is the set of all possible outcomes of our experiment. In SGD, the "experiment" is selecting one mini-batch. If your dataset has $N$ samples and your batch size is $B$ , then the sample space $Ω$ is the set of all possible subsets of data indices of size $B$ . The size of this space, $(B N)$ , is astronomically large for any non-trivial dataset, which is precisely why we can't just compute the loss over the whole thing at every step. Each element $ω \in Ω$ is a specific set of $B$ indices, i.e., one concrete mini-batch.
Event Space ( $F$ ): This is a set of subsets of $Ω$ . An "event" is a collection of outcomes we might care about. For example, an event $A \in F$ could be "The mini-batch contains the specific image with index 1337." This event $A$ is the set of all mini-batches of size $B$ that happen to include the index 1337.
Probability Measure ( $P$ ): This function assigns a probability to every event. When you use a standard DataLoader in PyTorch with shuffle=True, you are implicitly defining a probability measure. Assuming uniform sampling without replacement, the probability of picking any single, specific mini-batch $ω$ is $P ({ω}) = 1/ (B N)$ . The probability of the event $A$ (the batch containing image 1337) can then be calculated from this.

This formal structure, ( $Ω, F, P$ ), allows us to move from a vague notion of "picking something at random" to a precise mathematical model of the data sampling process that underpins SGD.

2. From Outcomes to Numbers: Random Variables

While the sample space $Ω$ can be abstract (like sets of images), we typically want to work with numbers. This is where random variables come in. As the video you just saw cheekily notes, the name "random variable" is terrible—it's not really a variable, and it's not inherently random. It's a function that maps outcomes from the sample space to a set of numerical values.

Probability spaces and random variables

This segment clarifies the true nature of a random variable as a function.

Watch the final part of the video explaining what a random variable is, from its definition as a function to how it connects back to the underlying probability space.

Let's get the formal definition and distinguish between the two main types.

Review of Probability Theory - CS229

These sections formally define random variables and the functions used to describe their behavior.

Please read the following sections: Start of Section 2: Read the definition of a random variable and the distinction between discrete and continuous types. Section 2.2: Read about Probability Mass Functions (PMFs) for discrete random variables. Section 2.3: Read about Probability Density Functions (PDFs) for continuous random variables. Note the crucial point that for a continuous variable, f_X(x) \neq P(X=x).

Let's ground this with our DL examples.

Continuous Random Variable: Network Initialization
When you initialize the weights of a neural network, you are instantiating millions of random variables. For instance, using Kaiming He initialization, each weight $W_{ij}$ is a draw from a Gaussian distribution with mean 0 and a specific variance. Here:

The outcome $ω$ is an abstract "roll of the dice" in the universe of randomness.
The random variable $W_{ij}$ is a function that maps this $ω$ to a real number.
The distribution of these values is described by a Probability Density Function (PDF), in this case, the familiar bell curve of the Gaussian distribution.

Discrete Random Variable: Dropout
Dropout is a perfect example of leveraging discrete random variables for regularization. During training, for each neuron in a layer, you decide whether to "keep" or "drop" it.

The outcome $ω$ is again an abstract random seed.
We define a Bernoulli random variable $M_{i}$ for the $i$ -th neuron's mask. A Bernoulli variable is the simplest discrete RV, taking two values (0 or 1).
$M_{i}$ is a function: $M_{i} (ω)$ equals 1 (keep) with probability $1 - p$ and 0 (drop) with probability $p$ , where $p$ is the dropout rate.
The behavior of $M_{i}$ is described by a Probability Mass Function (PMF): $P (M_{i} = 1) = 1 - p$ and $P (M_{i} = 0) = p$ .

The following video offers a clear, practical explanation of how dropout works.

What is Dropout Regularization | How is it different?

This video introduces dropout regularization and how it randomly deactivates neurons during training.

Watch the first segment, which explains the basic mechanism of dropout and how the dropout rate p corresponds to the probability of a neuron being inactive. This is a direct application of a Bernoulli random variable for every neuron.

3. Tying It Together: The Power of Expectation

The real power of this formalism comes when we start calculating properties of these random variables, chief among them the expected value. The expectation is the long-run average value of a random variable if you were to repeat the experiment many times.

Review of Probability Theory - CS229

This section defines expectation for both discrete and continuous random variables.

Read the first paragraph of Section 2.4, on Expectation.

Let's see how this concept resolves a key implementation detail in both SGD and dropout.

Expectation in SGD
Let $L_{ba t c h} (ω)$ be the loss on a randomly sampled mini-batch $ω$ . The stochastic gradient is $\nabla L_{ba t c h} (ω)$ . This is a random vector. The reason SGD works is that the expected value of the stochastic gradient is equal to the true gradient over the full dataset, assuming uniform sampling.

E [\nabla L_{ba t c h} (ω)] = \nabla L_{f u l l}

The stochastic gradient is thus an unbiased estimator of the true gradient. The "noise" we discussed in the last module is simply the variance around this expectation.

Expectation in Dropout
Dropout presents a mismatch: we drop neurons during training but use the full network for inference (testing). How do we reconcile this? The answer lies in ensuring the expected output of any given neuron is the same in both modes.

The Mısra Turp video explains this beautifully.

What is Dropout Regularization | How is it different?

This video explains the practical problem of using dropout and how it's solved by scaling activations, a direct application of managing expectations.

Please watch the rest of the video, focusing on: The mismatch between training and test time. The reason for scaling: to make the expected input to the next layer consistent. If the keep probability is 1-p = 0.75, a neuron on average receives input from 75% of the preceding neurons during training. At test time, it receives input from 100%. To compensate, we must scale the test-time output down. The concept of "inverted dropout," which is the standard implementation. Here, we scale the activations up during training by dividing by the keep probability. This makes the math work out so that no scaling is needed at test time.

Let's write out the inverted dropout logic formally. Let $a$ be the output of a neuron. During training, with keep probability $k = 1 - p$ , the output is scaled to $a^{'} = a / k$ . The random mask is $M \sim Bernoulli (k)$ . The final output is $M \cdot a^{'}$ . The expected value of this output is:

E [M \cdot a^{'}] = E [M] \cdot a^{'} = (1 \cdot k + 0 \cdot (1 - k)) \cdot \frac{a}{k} = k \cdot \frac{a}{k} = a

The expected output during training is now simply $a$ , which is exactly the output at test time (when $M = 1$ and no scaling is applied). This elegant trick, now standard in all DL frameworks, is a direct application of matching the expectations of random variables.

Conclusion

In this lesson, we've taken the first step in formalizing our understanding of probability for deep learning. We've seen that the abstract concepts of probability theory have direct, tangible analogues in the daily practice of training neural networks.

Key Takeaways:

A probability space $(Ω, F, P)$ provides the formal language to model random experiments like mini-batch sampling.
A random variable is not a variable, but a function that maps outcomes to numbers, allowing us to quantify the results of stochastic processes like weight initialization or dropout.
Discrete RVs are described by PMFs (e.g., the Bernoulli mask in dropout), while continuous RVs are described by PDFs (e.g., initial network weights).
The concept of expectation is a powerful tool for bridging the gap between stochastic training-time behavior and deterministic test-time behavior, as demonstrated by the scaling trick in dropout.

We have now refreshed the absolute fundamentals. In our next lesson, we will build on this by exploring expectation, variance, and covariance in greater depth, focusing on their role in Monte Carlo estimators, a cornerstone technique for approximating the intractable quantities that arise in generative modeling.

Recommended resources

1. The Language of Chance: Probability Spaces

2. From Outcomes to Numbers: Random Variables

3. Tying It Together: The Power of Expectation

Conclusion