Neural Network Fundamentals

1. The Big Picture: What is a Neural Network?

Before diving into the math, let's build a strong intuition for what neural networks are and why they are so powerful. They are loosely inspired by the structure of the human brain, using a layered network of "neurons" to find patterns in data. A task that is trivial for a human, like recognizing a handwritten digit, is incredibly complex to program explicitly. Neural networks provide a way for computers to learn these patterns on their own.

The following video from the excellent 3Blue1Brown series provides a beautiful, high-level introduction to the concept.

As you saw, a neural network is essentially a sophisticated function that maps inputs to outputs. The magic lies in how it's structured to break down a complex problem (like identifying a '9') into a hierarchy of simpler ones (identifying a top loop and a vertical line, which are in turn composed of smaller edges).

2. The Artificial Neuron: The Atom of the Network

Let's zoom in on the most fundamental building block of this entire structure: the artificial neuron (or "node"). A single neuron is a simple computational unit that takes in information, processes it, and passes it on.

Here are its key components:

1. Inputs ($x_1,x_2,...,x_n$): These are the numerical values fed into the neuron. For the first layer, this is the raw data (e.g., pixel values). For subsequent layers, it's the output from neurons in the previous layer.
2. Weights ($w_1,w_2,...,w_n$): Each input has a corresponding weight. The weight determines the importance of that input. A larger weight means the input has a stronger influence on the neuron's output. These are the main parameters the network learns during training.
3. Bias ($b$): This is an extra, learnable parameter that acts like an offset. The bias allows the neuron to shift its output up or down, independent of its inputs. You can think of it as controlling how easily the neuron "fires".
4. Summation: The neuron computes a weighted sum of its inputs and adds the bias: $z=(w_1{x}_1+w_2{x}_2+...+w_n{x}_n)+b$. In vector notation, this is simply $z=\mathbf{w}\cdot \mathbf{x}+b$.
5. Activation Function ($f$): The result of the summation, $z$, is then passed through an activation function to produce the final output: $a=f(z)$. We'll explore this crucial component next.

To see this process broken down step-by-step with a simple example, watch the following segment from StatQuest.

3. Activation Functions: Introducing Non-Linearity

Why do we need that final step, the activation function? Why not just use the weighted sum $z$?

The reason is crucial: to introduce non-linearity. If we didn't have activation functions (or just used a linear one), the entire neural network, no matter how many layers, would behave like a single, simple linear model. It would only be able to learn linear relationships, which is too limited for complex tasks like image or speech recognition.

Activation functions are the "bent or curved lines" that allow neural networks to fit the complex "squiggles" of real-world data.

Let's explore this concept and a few of the most important activation functions.

To summarize the most common activation functions:

• Sigmoid: f(z) = 1 / (1 + e^-z)

  • Squashes any input into a range between 0 and 1.
  • Historically important and useful in the output layer for binary classification, where the output can be interpreted as a probability.
  • Largely replaced in hidden layers because it can lead to training problems (the "vanishing gradient" problem, which we'll discuss later).

• Hyperbolic Tangent (Tanh): f(z) = (e^z - e^-z) / (e^z + e^-z)

  • Similar to sigmoid but squashes inputs to a range between -1 and 1.
  • Being zero-centered often helps the network learn more efficiently than sigmoid, but it also suffers from the vanishing gradient problem.

• Rectified Linear Unit (ReLU): f(z) = max(0, z)

  • Very simple: if the input is positive, it returns the input; otherwise, it returns 0.
  • This is the most popular activation function for hidden layers in modern neural networks. It is computationally very efficient and generally avoids the vanishing gradient problem, leading to faster training.

The 3Blue1Brown video you watched earlier also includes a short, valuable discussion on why ReLU is often preferred over Sigmoid in modern networks.

4. Network Architecture: Building with Layers

Now that we understand the neuron, building a network is straightforward: we stack neurons into layers and then stack the layers themselves.

A typical feedforward neural network has three types of layers:

1. Input Layer: This isn't really a computational layer. It simply holds the raw input data. The number of neurons in this layer is equal to the number of features in your dataset (e.g., 784 neurons for a 28x28 pixel image).
2. Hidden Layers: These are the layers between the input and output. Each neuron in a hidden layer is connected to all neurons in the previous layer (this is called a "fully connected" or "dense" layer). This is where the network does most of the feature extraction and learns hierarchical representations of the data. A network with one or more hidden layers is a "deep" neural network.
3. Output Layer: The final layer that produces the network's prediction. The number of neurons and the activation function used here depend on the task:
   • Regression: One neuron with a linear activation function.
   • Binary Classification: One neuron with a sigmoid activation function.
   • Multi-class Classification: N neurons (for N classes) with a Softmax activation function, which turns the outputs into a probability distribution across the classes.

The process of passing the input data through the layers from start to finish to get a prediction is called Forward Propagation.

For a more formal and mathematical look at how layers are constructed and how data flows through them, the following resource is excellent. Given your interest in the theoretical underpinnings, you'll find the move to matrix notation particularly satisfying.

Conclusion

In this lesson, we've laid the groundwork for your entire journey into deep learning. You now understand the fundamental anatomy of a neural network.

Key Takeaways:

• A neural network is a layered structure of artificial neurons inspired by the brain.
• Each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation function.
• Weights control the strength of connections, and the bias acts as a tunable threshold. These are the parameters learned during training.
• Activation functions (like ReLU) are essential for introducing non-linearity, allowing networks to learn complex patterns.
• The network architecture consists of an input layer, one or more hidden layers, and an output layer, with the design tailored to the specific task.

Preview of the Next Lesson:
We've answered the "what" — what is a neural network's structure? The next logical question is "how" — how does it learn the correct values for its thousands or millions of weights and biases? In the next lesson, we will dive into the core mechanism of learning in neural networks: backpropagation and gradient descent.