Building Feedforward Networks: Affine Transformations and Activations

Introduction

In our last lesson, we equipped ourselves with the fundamental building blocks of modern neural networks: differentiable activation functions like Sigmoid, Tanh, and ReLU. We established that these non-linear functions are the crucial ingredient that allows a model to learn patterns beyond the simple linear boundaries of the perceptron. We now have two primary components at our disposal:

1. Affine Transformation: A linear operation on an input $\mathbf{x}$ defined by $\mathbf{z}=\mathbf{Wx}+\mathbf{b}$, where $\mathbf{W}$ is a matrix of weights and $\mathbf{b}$ is a vector of biases. Geometrically, this corresponds to a rotation, scaling, and shear followed by a translation.
2. Non-linear Activation: An element-wise function $g(\mathbf{z})$ that "warps" or "squashes" the transformed space, introducing the complexity needed to model real-world data.

This lesson is about putting these two pieces together. We will construct a complete, multi-layer feedforward neural network by treating it as a composition of functions. We will explore how organizing neurons into layers allows for highly efficient, vectorized computation, and we will develop a geometric intuition for what the network is actually learning. This step moves us from isolated components to a complete, expressive model, setting the stage for understanding how information flows through it.

From Pixels to Concepts: A High-Level View

Before diving into the mathematical formalism, it's helpful to build an intuitive picture of what we're aiming for. A neural network is designed to learn hierarchical features from data. For a task like recognizing handwritten digits, the first layer might learn to detect simple edges from raw pixels. The next layer might combine these edges to form loops and lines. A final layer could then combine these more abstract components to recognize a "9" (a loop on top of a line) or an "8" (two loops).

The following video from 3Blue1Brown provides an excellent visualization of this concept.

The Anatomy of a Layer

As the video illustrates, a neural network is composed of layers of neurons. Let's formalize this. A single neuron, as depicted below, takes multiple inputs, computes a weighted sum, adds a bias, and then applies a non-linear activation function.

While we could model a network as a collection of individual neurons, this is computationally inefficient. Your background in performance-oriented programming will appreciate that we can achieve massive speedups by processing all neurons in a layer simultaneously using vectorized operations.

A fully-connected layer (or dense layer) is a set of neurons where every neuron receives the same input vector. If we have an input vector $\mathbf{x}\in {\mathbb{R}}^n$ and a layer with $m$ neurons, we can represent the weights for the entire layer as a matrix $\mathbf{W}\in {\mathbb{R}}^{m\times n}$ and the biases as a vector $\mathbf{b}\in {\mathbb{R}}^m$.

The computation for the entire layer then becomes:

1. Affine Transformation: $\mathbf{z}=\mathbf{Wx}+\mathbf{b}$
2. Non-linear Activation: $\mathbf{a}=g(\mathbf{z})$

Here, $\mathbf{z}$ is the vector of "pre-activation" values, and $\mathbf{a}$ is the vector of "activations" or outputs from the layer. The activation function $g$ is applied element-wise to the vector $\mathbf{z}$. This matrix-vector multiplication is far more efficient than iterating through $m$ separate dot products, especially on hardware like GPUs that are optimized for linear algebra.

The article below provides a clear definition of these core components.

Stacking Layers: Function Composition

A "deep" neural network is created by stacking these layers, where the output of one layer, ${\mathbf{a}}^{[l]}$, becomes the input to the next layer, ${\mathbf{x}}^{[l+1]}$. This is a direct application of function composition.

Consider a simple two-layer network (one hidden layer, one output layer). Let's trace the flow of data:

• The input to the network is $\mathbf{x}$.
• The first hidden layer, with weights ${\mathbf{W}}^{[1]}$ and bias ${\mathbf{b}}^{[1]}$, computes its output: $${\mathbf{a}}^{[1]}=g^{[1]}({\mathbf{W}}^{[1]}\mathbf{x}+{\mathbf{b}}^{[1]})$$
• This output, ${\mathbf{a}}^{[1]}$, becomes the input to the second (output) layer, which has its own weights ${\mathbf{W}}^{[2]}$ and bias ${\mathbf{b}}^{[2]}$: $$\hat{\mathbf{y}}={\mathbf{a}}^{[2]}=g^{[2]}({\mathbf{W}}^{[2]}{\mathbf{a}}^{[1]}+{\mathbf{b}}^{[2]})$$

By substituting the first equation into the second, we can see the entire network as a single, complex composite function:

$$\hat{\mathbf{y}}=f(\mathbf{x})=g^{[2]}({\mathbf{W}}^{[2]}(g^{[1]}({\mathbf{W}}^{[1]}\mathbf{x}+{\mathbf{b}}^{[1]}))+{\mathbf{b}}^{[2]})$$

This brings us to a critical "first principles" question. In the previous lesson, we established the need for non-linear activation functions. What would happen if we decided to use a simple linear activation, $g(z)=z$, in our hidden layers? What would the two-layer network above simplify to?

Hint

Substitute $g^{[1]}(\mathbf{z}) = \mathbf{z}$ into the composite function and expand the expression. What form does the final equation take?

The answer reveals a fundamental truth about neural networks. The following reading walks through the exact algebra.

As you've seen, a stack of linear layers is mathematically equivalent to a single linear layer. The composition of two affine transformations is just another affine transformation. Without non-linearity, a "deep" linear network has no more expressive power than a shallow one. The non-linear activation function is what allows each layer to add a new level of complexity to the representation.

A Geometric View: Warping Space

For someone with an interest in computer graphics, there is a powerful geometric interpretation of what a neural network does. Each layer applies a transformation to the space its input data lives in.

1. The affine transformation ($\mathbf{Wx}+\mathbf{b}$) performs a linear transformation (rotation, scaling, shearing) and a translation of the space.
2. The non-linear activation ($g(\mathbf{z})$) then "warps" and "folds" this transformed space.

A deep neural network learns a sequence of these transformations to morph the initial data distribution into one that is more easily handled by the final layer. For a classification problem, the network tries to warp the space until the different classes are linearly separable.

This space-warping behavior is beautifully demonstrated in the interactive visualization from Andrej Karpathy's lecture.

The Computation Graph

As our networks grow more complex, representing them with a single nested mathematical equation becomes unwieldy. A more structured and computationally-minded approach is to represent the network as a computation graph. This is a directed acyclic graph (DAG) where nodes represent either variables (inputs, parameters) or operations (matrix multiplication, addition, activation functions).

This perspective is essential for two reasons:

1. Modularity: It allows us to think of the network as a set of connected, reusable modules (layers). Deep learning frameworks are built around this idea.
2. Gradient Flow: It provides the structure needed to systematically compute gradients for all parameters using the chain rule. This process, called backpropagation, involves passing gradients backward through the graph.

We will explore backpropagation in great detail in a future lesson, but it is crucial to understand now that the computation graph is the data structure that makes it possible.

Conclusion

In this lesson, we have assembled our building blocks into a complete feedforward neural network. We have formally defined the network as a composition of functions, each corresponding to a layer that performs an affine transformation followed by a non-linear activation.

Key Takeaways:

• A feedforward neural network is a composite function built by stacking layers.
• Each layer's computation, $\mathbf{a}=g(\mathbf{Wx}+\mathbf{b})$, can be efficiently implemented using vectorized operations.
• Non-linearity is essential. Without it, a deep network collapses into a single, shallow linear model, losing its expressive power.
• Geometrically, a network learns to warp the input space through a series of transformations, aiming to make the data more easily separable.
• The computation graph is a powerful formalism for representing networks that is crucial for both conceptual clarity and the practical implementation of training algorithms.

Preview of the Next Lesson:

We now understand the structure of a neural network. The next logical step is to understand its operation. In the next lesson, we will perform a forward pass: taking a concrete input vector and propagating it through the layers of the network to produce a final prediction. This is the first of the two key processes in a neural network's life cycle, the second being the backward pass for learning.