Implementing Probability Distributions with SciPy

Introduction

In the previous module, we assembled a powerful toolkit for data manipulation and visualization. We capped it off by loading, cleaning, and exploring the Titanic dataset, preparing it for a modeling task. This process of Exploratory Data Analysis (EDA) is crucial, but it's only half the story. The other half is understanding the theoretical principles that allow us to build models in the first place.

This module dives into those principles, starting with the fundamental building blocks of statistical modeling: probability distributions. A probability distribution is a mathematical function that describes the likelihood of different outcomes for a random variable. Think of it as a blueprint for a random process. If we can identify the correct blueprint for the data we're observing, we can make powerful inferences and predictions.

In this lesson, we will review three of the most important distributions in data science: the Binomial, Poisson, and Normal (Gaussian) distributions. For each, we'll cover:

• The phenomena they typically model.
• Their mathematical definitions (PMF/PDF) and key parameters.
• How to implement and work with them in Python using the scipy.stats library.

This will lay the groundwork for the more advanced statistical inference topics we'll cover in the rest of the course.

1. Discrete Distributions: Modeling Count-Based Events

We'll begin with discrete distributions, which are used to model outcomes that can be counted (e.g., 0, 1, 2, ...).

The Binomial Distribution

Imagine an experiment with a fixed number of independent trials, where each trial has only two possible outcomes: "success" or "failure." The Binomial distribution models the number of successes in these trials.

A classic example is flipping a coin. Let's explore this with a video.

As the video explained, the shape of the binomial distribution is determined by n and p. Now, let's formalize this with the Probability Mass Function (PMF), which gives the probability of observing exactly $k$ successes.

The PMF for a Binomial distribution is:
$P(k;n,p)=\left(\genfrac{}{}{0ex}{}{n}{k}\right)p^k(1-p{)}^{n-k}$
where:

• $k$ is the number of successes.
• $n$ is the number of trials.
• $p$ is the probability of success in one trial.
• $\left(\genfrac{}{}{0ex}{}{n}{k}\right)=\frac{n!}{k!(n-k)!}$ is the binomial coefficient, representing the number of ways to choose $k$ successes from $n$ trials.

The next segment of the video provides a clear, step-by-step derivation of this formula, which will appeal to your interest in mathematical foundations.

The Poisson Distribution

Next, consider a different scenario. Instead of a fixed number of trials, we are interested in the number of events that occur within a fixed interval of time or space, when we only know the average rate of occurrence. For example:

• The number of emails you receive per hour.
• The number of defects per square meter of a material.
• The number of cars passing a point on a highway in one minute.

This is modeled by the Poisson distribution.

The PMF for the Poisson distribution is:
$P(k;\lambda )=\frac{e^{-\lambda }{\lambda }^k}{k!}$
where:

• $k$ is the number of events observed.
• $\lambda$ is the average number of events per interval.
• $e$ is Euler's number (approximately 2.71828).

A Deeper Connection: From Binomial to Poisson

The Binomial and Poisson distributions are closely related. The Poisson distribution can be derived as a limiting case of the Binomial distribution when the number of trials $n$ is very large and the probability of success $p$ is very small. In this limit, the expected number of successes, $np$, remains constant and becomes the Poisson parameter $\lambda$.

This connection is not just a mathematical curiosity; it's a powerful concept. It means that if you have a process with many opportunities for an event to occur, but each opportunity has a tiny chance of success, the total number of events will follow a Poisson distribution.

The following resources provide two perspectives on this derivation—a visual, conceptual one and a rigorous, step-by-step mathematical proof.

2. Continuous Distributions: Modeling Measured Quantities

Now let's shift to continuous distributions, which model outcomes on an unbroken scale, like height, temperature, or time. For these, we use a Probability Density Function (PDF). The area under the PDF curve over a certain range gives the probability that the outcome will fall within that range. The probability of any single specific value is technically zero.

The Normal (Gaussian) Distribution

The Normal distribution is arguably the most important distribution in all of statistics. It arises naturally in countless situations.

The PDF for the Normal distribution is given by:
$P(x;\mu ,\sigma )=\frac{1}{\sigma \sqrt{2\pi }}{e}^{-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2}$

A key feature of the Normal distribution is the Empirical Rule (or 68-95-99.7 rule), which provides a quick way to understand the spread of data.

3. Implementation with SciPy

Theory is essential, but a modern data scientist needs to be able to apply it. The scipy.stats module in Python is the standard tool for working with probability distributions. It provides a consistent and powerful API for a vast library of distributions.

The following resource from a "Machine Learning for Physics" course provides a fantastic overview. Given your background, you'll find the context familiar.

The scipy.stats objects for each distribution have several key methods:

• .pmf(k, ...) or .pdf(x, ...): Evaluates the Probability Mass/Density Function at point k or x.
• .cdf(x, ...): Evaluates the Cumulative Distribution Function up to point x (i.e., the probability of getting a value less than or equal to x).
• .rvs(size, ...): Generates size random variates (samples) from the distribution.

Here is a practical example putting it all together to plot the three distributions we've discussed.

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import binom, poisson, norm




# Set up the plot
fig, axs = plt.subplots(1, 3, figsize=(18, 5))




# 1. Binomial Distribution
n, p = 10, 0.3
x_binom = np.arange(0, n + 1)
pmf_binom = binom.pmf(x_binom, n, p)
axs[0].bar(x_binom, pmf_binom, label=f'n={n}, p={p}')
axs[0].set_title('Binomial PMF')
axs[0].set_xlabel('Number of Successes (k)')
axs[0].set_ylabel('Probability')
axs[0].legend()




# 2. Poisson Distribution
mu = 3
x_poisson = np.arange(0, 10)
pmf_poisson = poisson.pmf(x_poisson, mu)
axs[1].bar(x_poisson, pmf_poisson, label=f'λ={mu}')
axs[1].set_title('Poisson PMF')
axs[1].set_xlabel('Number of Events (k)')
axs[1].legend()




# 3. Normal Distribution
mu, sigma = 0, 1
x_norm = np.linspace(-4, 4, 100)
pdf_norm = norm.pdf(x_norm, mu, sigma)
axs[2].plot(x_norm, pdf_norm, label=f'μ={mu}, σ={sigma}')
axs[2].set_title('Normal PDF')
axs[2].set_xlabel('Value (x)')
axs[2].legend()

plt.tight_layout()
plt.show()

Feel free to run this code and experiment by changing the parameters (n, p, λ, μ, σ) to see how the distributions change shape.

Conclusion

In this lesson, we took our first step into the probabilistic foundations of data science. We moved from simply handling data to understanding the theoretical models that can describe it.

Key Takeaways:

• Probability Distributions are mathematical functions that model the likelihood of outcomes for a random variable.
• Discrete vs. Continuous: Discrete distributions (like Binomial and Poisson) use a Probability Mass Function (PMF) for countable outcomes. Continuous distributions (like Normal) use a Probability Density Function (PDF) for measured quantities.
• Key Distributions and Their Use Cases:
  • Binomial: Number of successes in $n$ independent trials, with probability $p$.
  • Poisson: Number of events in a fixed interval, given an average rate $\lambda$. It's a limiting case of the Binomial distribution.
  • Normal: A ubiquitous bell-shaped curve defined by its mean $\mu$ and standard deviation $\sigma$, often modeling natural phenomena and measurement errors.
• SciPy for Implementation: The scipy.stats library provides a unified API (.pmf, .pdf, .cdf, .rvs) for working with a vast collection of distributions in Python.

Preview of the Next Lesson:
We've reviewed some standard probability distributions. But what happens if our data is a function of a random variable? For example, if a random variable $X$ follows a Normal distribution, what is the distribution of $Y=X^2$? In the next lesson, we will explore the transformation of random variables and derive the powerful Jacobian method to answer such questions.