Forward Propagation: Generating Predictions

What is Forward Propagation?

At its core, forward propagation is the computational process of passing input data sequentially through a neural network to produce an output. It is a one-way street, always moving from the input layer towards the output layer.

To get an intuitive feel for this flow, let's start with a short, conceptual animation.

This visualization shows the what of forward propagation. Now, let's define it more formally before we dive into the how.

As the article states, forward propagation is the prerequisite for understanding how networks learn. It's the prediction step that must happen before a network can evaluate its own performance and make adjustments.

The Algorithm of a Forward Pass

The process you just saw visually is, under the hood, a sequence of calculations. It's an algorithm where the output of each layer becomes the input for the next.

Let's formalize this. For a network with $L$ layers:

1. The input data $\mathbf{x}$ is the activation of the "0th" layer, so ${\mathbf{a}}^{[0]}=\mathbf{x}$.
2. For each subsequent layer $\ell$ from 1 to $L$:
   • Calculate the pre-activation value, ${\mathbf{z}}^{[\ell ]}$, using the activations from the previous layer, ${\mathbf{a}}^{[\ell -1]}$: $${\mathbf{z}}^{[\ell ]}={\mathbf{W}}^{[\ell ]}{\mathbf{a}}^{[\ell -1]}+{\mathbf{b}}^{[\ell ]}$$
   • Calculate the activation for the current layer, ${\mathbf{a}}^{[\ell ]}$, by applying the layer's activation function ${\varphi }^{[\ell ]}$ to ${\mathbf{z}}^{[\ell ]}$: $${\mathbf{a}}^{[\ell ]}={\varphi }^{[\ell ]}({\mathbf{z}}^{[\ell ]})$$
3. The final activation vector, ${\mathbf{a}}^{[L]}$, is the network's prediction, often denoted as $\hat{\mathbf{y}}$.

This algorithm is elegantly captured in the following diagram.

A Step-by-Step Numerical Example

Theory and equations are essential, but seeing the numbers flow through the network solidifies understanding. The following video provides an excellent, detailed walkthrough of a forward pass with concrete values.

Let's recap the key calculation for a single neuron from that video. To get the activation of the first hidden neuron, g1:

1. Calculate the pre-activation z1:

   • Inputs: $x_1=60,x_2=80,x_3=5$
   • Weights: $w_1=0.1,w_2=0.1,w_3=0.1$ (Note: video uses a different weight naming)
   • Bias: $b_1=-15$
   • $z_1=(w_1{x}_1+w_2{x}_2+w_3{x}_3)+b_1$
   • $z_1=(0.1\cdot 60+0.1\cdot 80+0.1\cdot 5)+(-15)=(6+8+0.5)-15=-0.5$

2. Apply the activation function g1:

   • The video uses the sigmoid function: $\sigma (z)=\frac{1}{1+e^{-z}}$
   • $g_1=\sigma (z_1)=\sigma (-0.5)=\frac{1}{1+e^{-(-0.5)}}\approx 0.3775$

This exact process is repeated for every neuron in the layer, and then the entire procedure is repeated for the next layer, using the activations from the current layer as inputs.

A Deeper View: Forward Propagation as Geometric Transformation

Given your linear algebra background, we can appreciate forward propagation from a more profound perspective. It's not just a series of arithmetic steps; it's a sequence of geometric transformations. Each layer warps the data's "space."

This interpretation is powerful. A neural network learns a complex function by composing a series of simpler functions: an affine transformation followed by a non-linear "squashing." Layer by layer, it manipulates the geometry of the data until a simple decision boundary in the final layer's space is sufficient to solve the problem. This is a core reason for the "depth" in deep learning.

Conclusion

In this lesson, we demystified the process of making a prediction with a neural network. You've learned that forward propagation is the orderly, layer-by-layer computation that transforms an input into a final output.

Key Takeaways:

• Forward propagation is the process of feeding input data through a network, layer by layer, to generate a prediction.
• The process is algorithmic: the output activations of layer $\ell -1$ serve as the input for layer $\ell$.
• For each neuron, the core operation is calculating a weighted sum of inputs, adding a bias (the pre-activation $\mathbf{z}$), and then applying a non-linear activation function ($\mathbf{a}=\varphi (\mathbf{z})$).
• From a geometric viewpoint, each layer performs an affine transformation ($\mathbf{Wx}+\mathbf{b}$) followed by a non-linear warping of the data's feature space.

Preview of the Next Lesson:
We now know how a network with a given set of weights and biases makes a prediction. But this prediction is likely wrong, especially at first. How does the network learn from its errors? This brings us to the other half of the training cycle. In the next lesson, we will explore the concept of backpropagation, the ingenious algorithm that calculates how to adjust the network's 13,000+ parameters (as mentioned in the 3Blue1Brown video) to improve its predictions.