Modeling Binary and Count Data

1. The Building Block: The Bernoulli Distribution

The simplest probabilistic scenario involves a single event with only two possible outcomes, often labeled as "success" (1) and "failure" (0). Think of flipping a coin once, a user clicking on an ad or not, or a single manufactured part being defective or not. This is modeled by the Bernoulli distribution.

The Bernoulli distribution is defined by a single parameter, $p$, which is the probability of success.

• $P(X=1)=p$
• $P(X=0)=1-p$

The following short video provides a clear definition of the Bernoulli distribution and, importantly, connects it to the concepts from our last lesson by stating its mean and variance.

As the video explained, the expected value and variance are straightforward to calculate:

• Expected Value: $E[X]=p$
• Variance: $Var(X)=p(1-p)$

The article below provides a quick, practical example of a Bernoulli distribution in the context of a medical trial and shows how you might visualize it.

2. Counting Successes: The Binomial Distribution

What happens if we repeat a Bernoulli trial multiple times? For example, what is the probability of getting 7 heads in 10 coin flips? Or the probability that 15 out of 100 users will click on an ad?

This is where the Binomial distribution comes in. It models the number of successes, $k$, in a fixed number of independent Bernoulli trials, $n$.

The Binomial distribution has two parameters:

• $n$: The total number of trials.
• $p$: The probability of success on any single trial.

The probability of observing exactly $k$ successes in $n$ trials is given by the probability mass function (PMF):

$P(k|n,p)=(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}$

Where $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ ("n choose k") is the binomial coefficient, which counts the number of ways to choose $k$ successes from $n$ trials.

The next segment of the video you just watched explains this formula and provides a powerful insight into the mean and variance of the Binomial distribution.

This connection is crucial: since a Binomial experiment is the sum of $n$ independent Bernoulli trials, its mean is simply $n$ times the mean of a single trial ($n\times p$), and its variance is $n$ times the variance of a single trial ($n\times p(1-p)$). This is a great example of how simpler concepts build into more complex ones.

3. Counting Events in an Interval: The Poisson Distribution

Sometimes, we're not counting successes out of a fixed number of trials, but rather the number of times an event occurs over a fixed interval of time or space. For example:

• The number of emails you receive in an hour.
• The number of typos on a page of a book.
• The number of customers arriving at a store in a day.

For these scenarios, we use the Poisson distribution. It's defined by a single parameter, $\lambda$ (lambda).

• $\lambda$: The average number of events in the specified interval.

The probability of observing exactly $k$ events in an interval is given by the PMF:

$P(k|\lambda )=\frac{{\lambda }^k{e}^{-\lambda }}{k!}$

Where $e$ is the base of the natural logarithm ($\approx 2.718$).

A key property of the Poisson distribution is that its mean and variance are both equal to $\lambda$.

• Expected Value: $E[X]=\lambda$
• Variance: $Var(X)=\lambda$

The final section of the first video introduces the Poisson distribution and its formula.

The Connection Between Binomial and Poisson

You might wonder how the Binomial and Poisson distributions are related. Given your engineering background, you'll appreciate that the Poisson distribution can be derived as a limiting case of the Binomial distribution.

Imagine a Binomial scenario where the number of trials $n$ is very large, but the probability of success $p$ is very small. For instance, the number of cars passing a certain point on a highway in an hour ($n$ is huge, as every passing moment is a "trial") and the probability of a car being a rare classic model ($p$ is tiny). In this situation, the Binomial distribution starts to look just like a Poisson distribution, where $\lambda =n\times p$.

The following video provides a brilliant, intuitive explanation of this connection.

4. The Distributions in Practice

To see these distributions in a machine learning context and understand how to work with them in Python, the Neuromatch Academy tutorial is an excellent resource. It uses examples like a rat in a maze (Binomial) and neural spiking (Poisson).

Conclusion

Today we've covered the three most important discrete probability distributions for modeling data. By understanding their assumptions and parameters, you can now choose the right tool for modeling binary events and counts.

Key Takeaways:

Next Steps:
We've now covered the essential distributions for discrete data (counts). In our next lesson, we will shift our focus to continuous data—measurements that can take any value within a range. We'll explore the two most important continuous distributions: the Normal (Gaussian) distribution for modeling symmetric, bell-shaped data, and the Exponential distribution for modeling waiting times between events.