Perceptron vs. Logistic Regression: A GLM Perspective

Welcome to the first lesson of your course, "AI from Sequence to Sequence." Your goal is to build a robust theoretical foundation for modern AI, starting from the precursors to today's complex models. Given your proficiency in classical machine learning, our journey begins by reframing concepts you're already familiar with—specifically, logistic regression and the perceptron—as the fundamental building blocks of neural networks.

In this lesson, we will compare the perceptron and logistic regression models, formalizing their relationship through the powerful framework of Generalized Linear Models (GLMs). This will not only provide a deeper, more unified perspective on these algorithms but also establish the core "neuron" abstraction that we will build upon throughout this course, from simple networks to the advanced transformer architectures at the frontier of AI.

From Linear Models to a Neuron

You're already familiar with linear and logistic regression. Both models compute a weighted sum of the input features, often written as a dot product plus a bias: $\eta ={\mathbf{w}}^T\mathbf{x}+b$. This term, $\eta$, is the linear predictor or net input.

The key difference between these models lies in what they do after computing this linear predictor.

• Linear Regression uses this value directly as its output, aiming to predict a continuous target.
• Logistic Regression passes this value through a sigmoid (or logistic) function, $\sigma (\eta )$, to squash the output into the range $(0,1)$, interpreting it as a probability.
• The Perceptron, one of the earliest models of a neuron, takes a simpler approach: it passes the linear predictor through a threshold (or Heaviside step) function. If the input is positive, it outputs 1; otherwise, it outputs 0.

So, all three models share a common core: a linear transformation. The difference is the activation function applied to the result. For linear regression it's the identity function ($f(\eta )=\eta$), for logistic regression it's the sigmoid function, and for the perceptron it's the step function.

To formalize this comparison, let's turn to the Andrew Ng's CS229 lecture notes, which you'll find a useful reference throughout our early lessons.

This simple change—from a smooth sigmoid to a hard threshold—has significant implications. As the notes mention, it becomes difficult to assign a meaningful probabilistic interpretation to the perceptron's output. Logistic regression, on the other hand, has a very elegant probabilistic foundation, which is best understood through the lens of Generalized Linear Models.

A Unifying Framework: Generalized Linear Models (GLMs)

Generalized Linear Models provide a framework that unifies various regression models, including linear and logistic regression. Understanding GLMs will make it clear why the sigmoid function isn't just an arbitrary choice for logistic regression but rather a natural consequence of its underlying statistical assumptions.

A GLM is defined by three components:

1. Random Component: Specifies the probability distribution of the response variable $y$, given the features $\mathbf{x}$. This distribution must be a member of the exponential family (e.g., Normal, Bernoulli, Poisson).
2. Systematic Component: A linear predictor, which is a linear combination of the input features: $\eta ={\mathbf{w}}^T\mathbf{x}$.
3. Link Function: A function $g$ that connects the expected value of the random component, $\mu =E[y|\mathbf{x}]$, to the systematic component: $g(\mu )=\eta$. The inverse of the link function, $g^{-1}$, is called the response function.

The following reading from Andrew Ng's notes provides the theoretical backbone for GLMs, showing how common distributions fit into the exponential family and laying out the three assumptions for constructing a GLM. Your strong math background will be an asset here.

Deriving Logistic Regression as a GLM

With the GLM framework established, we can now derive logistic regression from first principles. This derivation is a perfect example of the kind of theoretical intuition this course aims to build.

We follow the three GLM assumptions:

1. Random Component: For binary classification, the outcome $y$ is either 0 or 1. The natural distribution to model this is the Bernoulli distribution. So, we assume $y|\mathbf{x};\mathbf{w}\sim \text{Bernoulli}(\varphi )$, where $\varphi =P(y=1|\mathbf{x})$.
2. Systematic Component: We assume a linear relationship between the parameters and the features: $\eta ={\mathbf{w}}^T\mathbf{x}$.
3. Link Function: Our goal is to predict the expected value of $y$. For a Bernoulli distribution, the expectation is simply the probability of success: $E[y|\mathbf{x}]=\varphi$. The canonical link function for the Bernoulli distribution is the logit function, which you saw in the derivation showing Bernoulli is in the exponential family: $$g(\varphi )=\log \left(\frac{\varphi }{1-\varphi }\right)=\eta$$ This term, the log of the odds, is the logit.

Now, by setting the link function equal to the systematic component, we get:

$$\log \left(\frac{\varphi }{1-\varphi }\right)={\mathbf{w}}^T\mathbf{x}$$

To find our hypothesis $h_{\mathbf{w}}(\mathbf{x})=\varphi$, we simply solve for $\varphi$:

$$\begin{gathered} \frac{\varphi }{1-\varphi }=e^{{\mathbf{w}}^T\mathbf{x}} \\ \varphi =(1-\varphi )e^{{\mathbf{w}}^T\mathbf{x}} \\ \varphi (1+e^{{\mathbf{w}}^T\mathbf{x}})=e^{{\mathbf{w}}^T\mathbf{x}} \\ \varphi =\frac{e^{{\mathbf{w}}^T\mathbf{x}}}{1+e^{{\mathbf{w}}^T\mathbf{x}}}=\frac{1}{1+e^{-{\mathbf{w}}^T\mathbf{x}}} \end{gathered}$$

And there it is: the sigmoid function emerges not as an ad-hoc choice, but as the canonical response function when modeling a Bernoulli-distributed outcome within the GLM framework.

The following brief reading confirms this derivation.

From GLMs to Neural Network Units

We can now see logistic regression and the perceptron as two variations on a single theme. They are both simple computational "units" that take an input vector $\mathbf{x}$, compute a linear combination $\eta ={\mathbf{w}}^T\mathbf{x}+b$, and apply an activation function.

• Logistic Regression Unit: $y=\sigma ({\mathbf{w}}^T\mathbf{x}+b)$
• Perceptron Unit: $y=\text{step}({\mathbf{w}}^T\mathbf{x}+b)$

This structure—a linear transformation followed by a non-linear activation—is the fundamental building block of a neural network. A single logistic regression classifier is a neural network with a single neuron, an input layer, and a sigmoid activation function.

The following reading and video provide a clear summary of this perspective, placing linear and logistic regression within the GLM framework and viewing them as simple neural networks.

By reframing what you know about linear models in this way, you've already taken the first crucial step. You now understand the basic computational unit that, when scaled up and composed in layers, forms the powerful deep learning models we will be studying.

Conclusion

In this lesson, we established a crucial bridge between classical machine learning and neural networks.

Key Takeaways:

• The Perceptron and Logistic Regression models both consist of a linear transformation of inputs ($\eta ={\mathbf{w}}^T\mathbf{x}+b$) followed by an activation function.
• The Perceptron uses a non-probabilistic threshold (step) function, while Logistic Regression uses a smooth sigmoid function.
• The Generalized Linear Model (GLM) framework explains why logistic regression uses the sigmoid function. It is the natural result of assuming a Bernoulli distributed outcome and using the canonical logit link function.
• This "linear combination + activation function" structure defines a single neuron, the fundamental building block of neural networks. A logistic regression classifier can be viewed as a single-neuron neural network.

This perspective reveals that the foundations of deep learning are rooted in well-established statistical principles. However, the power of neural networks comes not from a single neuron, but from combining them. A single neuron, like logistic regression, can only learn a linear decision boundary. In our next lesson, we will investigate the limitations of this by exploring the classic XOR problem, which a single neuron cannot solve. We will then see how stacking these neurons into a Multi-Layer Perceptron (MLP) overcomes this limitation and unlocks the ability to learn complex, non-linear functions.