Neural Network Essentials: Neurons, Layers, and Activation

Hello! Welcome to your first lesson in our journey into Large Language Models and AI.

Our goal in this course is to build a systematic understanding of how modern AI systems work, from their foundational concepts to practical applications. We'll start at the very beginning, with the building blocks of almost all modern AI: neural networks.

Today's lesson addresses the first learning outcome of our course: Explain the components of a neural network (neurons, layers, and activation functions). Understanding these fundamentals is essential because even the most sophisticated LLMs, like GPT-4, are built upon these core principles, just at an immense scale and with a specific, powerful architecture we will explore soon.

1. The Big Picture: A Network of Layers

At its heart, a neural network is a computational model inspired by the structure of the brain, designed to recognize patterns in data. It's composed of interconnected nodes, or "neurons," organized into layers.

To build an initial intuition, let's watch a segment from a classic video on the topic. It uses the problem of recognizing handwritten digits to introduce the concept of a neural network and its layered structure.

As the video explains, we can think of a neural network as being organized into three types of layers:

• Input Layer: This is the network's entry point. Each neuron in this layer represents a single feature of the input data. In the video's example, there's one input neuron for each of the 784 pixels in the 28x28 image. For a software engineer, this is analogous to the input parameters of a function.
• Hidden Layers: These are the intermediate layers between the input and output. They are "hidden" because their values aren't directly provided as input or read out as the final output. This is where the majority of the complex pattern recognition happens. The network can have one or many hidden layers.
• Output Layer: This is the final layer that produces the result. In the digit recognition example, it has 10 neurons, each corresponding to a digit from 0 to 9. The neuron with the highest activation value represents the network's prediction. This is the return value of our function.

This layered structure is the basic architecture of a neural network.

2. The Building Block: The Neuron

Now, let's zoom in on the individual units that make up these layers: neurons (also called nodes). A neuron is a simple computational unit that does two main things:

1. It receives inputs from all the neurons in the previous layer.
2. It calculates an output value to pass on to the neurons in the next layer.

This calculation happens in a few steps. For each neuron:

• Each incoming connection has a weight associated with it. This weight represents the importance of that connection—a higher weight means the input from that connection has more influence.
• The neuron calculates a weighted sum of all its inputs.
• A bias value is then added to this weighted sum. The bias acts as an offset, making it easier or harder for the neuron to produce a high output value. You can think of it as a tunable threshold.

From your perspective as a developer, you can think of the weights and biases as parameters that a function uses. The core process of "learning" in a neural network is simply the process of finding the optimal values for all these weight and bias parameters.

The following very short clip gives a clear definition of these two key terms.

The calculation inside a neuron can be expressed simply as:
$\text{output}=({\text{input}}_1\times {\text{weight}}_1)+({\text{input}}_2\times {\text{weight}}_2)+\cdots +\text{bias}$

To see this written as a piece of code, which might feel more familiar, take a quick look at the following article.

So far, the neuron's calculation is purely linear (a series of multiplications and additions). If this were all, a multi-layered network would be no more powerful than a single layer. The final component is what gives neural networks their power.

3. Adding Complexity: The Activation Function

After calculating the weighted sum + bias, the neuron passes this result through a final component: an activation function.

The activation function introduces non-linearity into the network. This is crucial because it allows the network to learn and model complex, non-linear relationships in the data. Without it, a neural network could only learn linear patterns.

Let's return to the 3Blue1Brown video to see the first type of activation function, the Sigmoid function.

The Sigmoid function, $\sigma (z)=\frac{1}{1+e^{-z}}$, maps any input value to an output between 0 and 1. This is useful, for example, in the output layer for a binary classification task where the output represents a probability.

While Sigmoid is historically important, most modern neural networks use other activation functions for better performance during training. The most common is the ReLU (Rectified Linear Unit). It's computationally very simple: it returns the input if it's positive, and 0 otherwise.
$\text{ReLU}(z)=\max (0,z)$

This article gives a quick, clear overview of these common activation functions.

Conclusion

In this lesson, we have broken down a neural network into its three fundamental components:

• Layers: The structure that holds and organizes neurons, consisting of an Input Layer, one or more Hidden Layers, and an Output Layer.
• Neurons: The basic computational units within each layer. The output of a neuron is determined by the weights of its connections and an internal bias.
• Activation Functions: Non-linear functions (like ReLU or Sigmoid) applied by each neuron to its output. They enable the network to learn complex patterns that go beyond simple linear relationships.

Conceptually, you can think of the entire network as one giant, complex function. The input data goes in one end, and a prediction comes out the other. The millions (or billions, in the case of LLMs) of weights and biases are the parameters of this function.

This brings us to the natural next question: how does the network find the right values for all those parameters? In our next lesson, we will explore this "learning" process, covering how neural networks learn by minimizing a loss function via backpropagation, at a conceptual level.