Bernoulli to Logistic Regression

Hello! Welcome to your next lesson on logistic regression.

In our last session, we conducted a deep dive into the logistic (or sigmoid) function, $\sigma (z)=1/(1+e^{-z})$. We rigorously derived it from three different perspectives, concluding that it is the ideal function to model the conditional probability of a positive outcome, $P(Y=1|\mathbf{x})$, by mapping the unbounded linear predictor ${\mathbf{x}}^T\mathbf{\theta }$ to the required $[0,1]$ interval.

Today, we will build directly on that foundation. Our learning outcome is to formulate the logistic regression model from a probabilistic (Bernoulli) perspective. We will formalize the relationship between our binary outcome variable, $Y$, and the sigmoid function. This will allow us to construct a complete probabilistic model for a single data point and, from there, for our entire dataset. This formulation is the theoretical cornerstone upon which the optimization process, which we will study next, is built.

1. The Probabilistic Assumption: A Bernoulli Outcome

In binary classification, the target variable $Y$ can only take on one of two values, which we label as 0 (for the negative class) and 1 (for the positive class). A random variable that can only result in one of two outcomes is perfectly described by the Bernoulli distribution.

Recall from your foundational studies in probability that a Bernoulli random variable has a single parameter, $p$, which represents the probability of "success" (i.e., the outcome being 1). The probability mass function (PMF) is:

• $P(Y=1)=p$
• $P(Y=0)=1-p$

In the context of logistic regression, we are not dealing with a single, fixed probability $p$. Instead, we hypothesize that the probability of the outcome depends on the input features $\mathbf{x}$. This is where our work from the previous lesson comes into play. We model the parameter $p$ of the Bernoulli distribution as the output of our logistic function:

$p=P(Y=1|\mathbf{x};\mathbf{\theta })=h_{\mathbf{\theta }}(\mathbf{x})=\sigma ({\mathbf{x}}^T\mathbf{\theta })$

Therefore, the core of the logistic regression model is the assumption that the response variable $Y$, conditioned on the features $\mathbf{x}$, follows a Bernoulli distribution whose success parameter is given by $h_{\mathbf{\theta }}(\mathbf{x})$. This leads to the following two conditional probabilities for a single data point $(\mathbf{x},y)$:

1. The probability of the class being 1 is: $P(Y=1|\mathbf{x};\mathbf{\theta })=h_{\mathbf{\theta }}(\mathbf{x})$
2. The probability of the class being 0 is: $P(Y=0|\mathbf{x};\mathbf{\theta })=1-h_{\mathbf{\theta }}(\mathbf{x})$

2. A Compact Formulation for the PMF

While the two separate equations above are correct, they are cumbersome to work with mathematically, especially when we want to write a function for the entire dataset. Fortunately, there is an elegant way to combine them into a single expression.

As explained in the video, because our label $y$ can only be 0 or 1, we can write a single equation for the probability of observing a specific outcome $y$ given $\mathbf{x}$:

$P(y|\mathbf{x};\mathbf{\theta })=(h_{\mathbf{\theta }}(\mathbf{x}){)}^y(1-h_{\mathbf{\theta }}(\mathbf{x}){)}^{1-y}$

This equation is a concise representation of the Bernoulli PMF where the parameter is $h_{\mathbf{\theta }}(\mathbf{x})$. Let's verify this for ourselves, which is a crucial step in ensuring rigorous understanding:

• Case 1: The true label is $y=1$
  Plugging $y=1$ into the equation gives:
  $P(y=1|\mathbf{x};\mathbf{\theta })=(h_{\mathbf{\theta }}(\mathbf{x}){)}^1(1-h_{\mathbf{\theta }}(\mathbf{x}){)}^{1-1}=(h_{\mathbf{\theta }}(\mathbf{x}){)}^1(1-h_{\mathbf{\theta }}(\mathbf{x}){)}^0=h_{\mathbf{\theta }}(\mathbf{x})$
  This correctly retrieves our first probability rule.

• Case 2: The true label is $y=0$
  Plugging $y=0$ into the equation gives:
  $P(y=0|\mathbf{x};\mathbf{\theta })=(h_{\mathbf{\theta }}(\mathbf{x}){)}^0(1-h_{\mathbf{\theta }}(\mathbf{x}){)}^{1-0}=1\cdot (1-h_{\mathbf{\theta }}(\mathbf{x}){)}^1=1-h_{\mathbf{\theta }}(\mathbf{x})$
  This correctly retrieves our second probability rule.

This compact form is not just a notational convenience; it is the mathematical object we will use to build the objective function for training the entire model.

3. From a Single Point to the Entire Dataset: The Likelihood Function

We have successfully formulated the probability for a single data point. The goal of training a model, however, is to find parameters $\mathbf{\theta }$ that are "best" for the entire training dataset $D=\{({\mathbf{x}}^{(i)},y^{(i)}){\}}_{i=1}^n$.

The principle we will use is Maximum Likelihood Estimation (MLE), which you have encountered in Module 8. The core idea of MLE is to choose the parameters $\mathbf{\theta }$ that maximize the probability of observing the data that we have actually observed. This probability is called the likelihood function, $L(\mathbf{\theta })$.

To construct this function, we make the standard assumption that our training examples are independent and identically distributed (i.i.d.). The "independent" part is key here: it means the probability of the entire set of outcomes is the product of the probabilities of each individual outcome.

The following resource provides a formal, academic description of this step.

Following the logic from the text, the likelihood of the parameters $\mathbf{\theta }$ given the dataset $(\mathbf{X},\mathbf{y})$ is the product of the individual probabilities for each data point $i=1,...,n$:

$L(\mathbf{\theta })=P(\mathbf{y}|\mathbf{X};\mathbf{\theta })={\prod }_{i=1}^{n}P(y^{(i)}|{\mathbf{x}}^{(i)};\mathbf{\theta })$

Now, we substitute our compact Bernoulli PMF into this product:

$L(\mathbf{\theta })={\prod }_{i=1}^n(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{y^{(i)}}(1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}$

This equation, the likelihood function, is the complete probabilistic formulation of the logistic regression model for a given dataset. It quantifies how "likely" our observed labels $\mathbf{y}$ are for a given set of parameters $\mathbf{\theta }$. Our goal in training will be to find the $\mathbf{\theta }$ that makes this likelihood as large as possible.

Conclusion

In this lesson, we have formally defined the logistic regression model from a rigorous probabilistic standpoint.

Key Takeaways:

• Logistic regression models the binary outcome $Y\in \{0,1\}$ as a Bernoulli random variable.
• The parameter of this Bernoulli distribution (the probability of success, $P(Y=1)$) is not constant; it is modeled by the sigmoid function applied to a linear combination of features: $h_{\mathbf{\theta }}(\mathbf{x})=\sigma ({\mathbf{x}}^T\mathbf{\theta })$.
• The probability mass function for a single observation can be written in the compact form: $P(y|\mathbf{x};\mathbf{\theta })=(h_{\mathbf{\theta }}(\mathbf{x}){)}^y(1-h_{\mathbf{\theta }}(\mathbf{x}){)}^{1-y}$.
• Assuming i.i.d. data, the likelihood function for the entire dataset is the product of these individual probabilities, giving us a complete objective function to be maximized.

Preview of the Next Lesson:
We have successfully derived the likelihood function $L(\mathbf{\theta })$. In our next lesson, we will tackle the problem of maximizing it. You will find that working with a product of many terms is computationally difficult and numerically unstable. We will see that maximizing the likelihood is equivalent to maximizing the log-likelihood, which converts the product into a sum. This will lead us directly to the derivation of the cross-entropy loss function, the standard cost function for logistic regression.