The Anatomy of an Artificial Neuron

Introduction

Welcome to the exciting world of Neural Networks! In the previous module, we explored unsupervised learning, using algorithms like K-means and PCA to discover hidden structures in data without predefined labels. You learned how to find clusters and visualize high-dimensional data, which are powerful techniques for data exploration.

Now, we pivot to a new and transformative area of AI. This module, Neural Network Fundamentals, will introduce you to the models that power many of today's most advanced applications, from image recognition to natural language processing. Your goal of building transformative technology, much like the "Tony Stark" archetype you admire, relies heavily on understanding these systems.

In this first lesson, we will start with the most fundamental building block. Our learning outcome is to describe the structure of a single artificial neuron, including weights, bias, and the activation function. Think of it as learning about a single transistor before building a complex computer chip. By the end of this lesson, you'll understand exactly how this basic computational unit works.

The Artificial Neuron: A Biological Analogy

Artificial neural networks are, as the name suggests, inspired by the structure of the human brain. While the analogy isn't perfect, it provides a powerful intuition. A biological neuron receives signals through its dendrites, processes them, and if a certain threshold is met, fires a signal down its axon to other neurons.

An artificial neuron models this basic behavior. It takes numerical inputs, performs a calculation, and produces a single numerical output.

To get a feel for this, let's start with a high-level, intuitive overview from the acclaimed 3Blue1Brown series on neural networks.

But what is a neural network? | Deep learning chapter 1

This first video, 'But what is a neural network?', will introduce the concept of a neuron as a simple 'thing that holds a number' and place it within the context of a larger network.

Watch the segment from 02:43 to 04:28. Focus on how a neuron is defined as holding a value called an 'activation' and how neurons are organized into layers (input, hidden, and output).

As you saw, a neuron's activation is just a number. For now, we'll think of it as a value between 0 and 1, representing how "active" the neuron is. Now, let's break down the mechanics of how a neuron calculates this activation value.

Deconstructing a Single Neuron

A neuron is a simple computational unit. It takes one or more inputs, processes them, and produces an output. This process involves three key components: weights, a bias, and an activation function.

The following blog post by Victor Zhou provides an exceptionally clear, step-by-step breakdown. We will use it as our primary guide.

Machine Learning for Beginners: An Introduction to Neural Networks

The article 'Machine Learning for Beginners: An Introduction to Neural Networks' clearly explains the components of a neuron. We will focus on the first section.

Please read the section '1. Building Blocks: Neurons'. Pay close attention to the diagram of the 2-input neuron and the three steps it describes: multiplying inputs by weights, adding a bias, and applying an activation function.

Let's unpack the concepts from the article one by one.

1. Inputs and Weights

A neuron receives one or more inputs, which we can represent as a vector $x = [x_{1}, x_{2}, \dots, x_{n}]$ . Each input is connected to the neuron, and each connection has a weight, which we can represent as $w = [w_{1}, w_{2}, \dots, w_{n}]$ .

The weights are the most important parameters a neuron has. A weight determines the influence of a corresponding input on the neuron's output.
A large positive weight means the input has a strong excitatory effect (it tends to make the neuron activate).
A large negative weight means the input has a strong inhibitory effect (it tends to prevent the neuron from activating).
A weight close to zero means the input has little effect on the neuron's output.

You can think of weights as being analogous to the coefficients in a linear regression model, where each coefficient scales the importance of a feature.

The first step in the neuron's calculation is to compute a weighted sum of its inputs. Given your background in linear algebra, you'll recognize this is simply the dot product of the weight and input vectors:

Weighted Sum = w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n} = w \cdot x

2. The Bias

After calculating the weighted sum, a bias term ( $b$ ) is added. The bias is a single number, and it is another trainable parameter of the neuron.

z = (w \cdot x) + b

The bias's job is to shift the neuron's activation. Think of it as a measure of how easy it is for the neuron to fire. A large positive bias means the neuron needs less stimulation from its weighted inputs to activate, while a large negative bias means it requires a very strong signal to activate. This is similar to the y-intercept in a linear equation, which shifts the line up or down.

The StatQuest video below gives a great explanation of how weights and biases work together to transform a neuron's inputs.

The Essential Main Ideas of Neural Networks

In 'The Essential Main Ideas of Neural Networks', Josh Starmer explains weights and biases from a data-fitting perspective.

First, watch from 08:25 to 08:57. This part shows how an input is multiplied by a weight and has a bias added. Then, jump to 17:04 - 17:41, where he explicitly defines the terms 'weights' and 'biases' and summarizes their role.

3. The Activation Function

The result of the weighted sum plus bias, $z$ , can be any real number. To produce a controlled, predictable output, this value is passed through an activation function, often denoted as $f (z)$ or $σ (z)$ .

output = y = f (z) = f ((w \cdot x) + b)

The primary purpose of the activation function is to introduce non-linearity into the network. This is crucial because real-world data is complex and rarely follows simple linear patterns. By adding non-linearity, neural networks can learn to approximate incredibly complex functions.

Here is a visual summary of the entire structure:

Perceptron Structure — This diagram shows the complete structure of a single neuron (also called a Perceptron). It receives inputs ($x$), multiplies them by weights ($w$), adds a bias ($b$), and passes the result ($z$) through an activation function ($g$) to produce the final output ($\hat{y}$). The mathematical formula is concisely written as $\hat{y} = g(w^Tx + b)$.

Common Activation Functions

There are several common activation functions, each with different properties. Let's explore the most important ones.

A Quick Introduction to Neural Networks

The article 'A Quick Introduction to Neural Networks' by Ujjwal Karn provides a concise overview of the most common activation functions.

Read the part of the 'A Single Neuron' section that lists and describes the Sigmoid, tanh, and ReLU activation functions. Note their formulas and the range of their outputs.

Let's summarize the key ones:

Sigmoid:
$σ (z) = \frac{1}{1 + e ^{- z}}$
This function "squashes" any real number into the range (0, 1). It was historically popular because the output could be interpreted as a probability or the firing rate of a neuron. However, it is less common in modern deep networks.
ReLU (Rectified Linear Unit):
$f (z) = max (0, z)$
This is currently the most popular activation function. Its operation is simple: if the input $z$ is negative, the output is 0; otherwise, the output is $z$ . Despite its simplicity, it works extremely well in practice and helps mitigate some technical problems that arise when training deep networks.

The 3Blue1Brown video also touches on the practical difference between Sigmoid and ReLU.

But what is a neural network? | Deep learning chapter 1

Let's revisit the 3Blue1Brown video for a quick practical comparison between Sigmoid and the more modern ReLU function.

First, re-watch the segment from 10:24 to 10:57, which explains the Sigmoid function. Then, watch the segment from 17:15 to 18:25, which discusses ReLU and why it is often preferred in modern networks.

A Complete Example

Let's walk through the numerical example from Victor Zhou's article to solidify your understanding.

Neuron Parameters:

Inputs: $x = [2, 3]$
Weights: $w = [0, 1]$
Bias: $b = 4$
Activation Function: Sigmoid

Calculation Steps:

Compute the weighted sum (dot product):
$w \cdot x = (0 \times 2) + (1 \times 3) = 0 + 3 = 3$
Add the bias:
$z = (w \cdot x) + b = 3 + 4 = 7$
Apply the activation function:
$y = σ (z) = σ (7) = \frac{1}{1 + e ^{- 7}} \approx \frac{1}{1 + 0.00091} \approx 0.999$

The neuron's final output is approximately 0.999. This entire process of passing inputs through the neuron to get an output is called feedforward.

Finally, let's see how this is implemented in code. Victor Zhou's article concludes with a small Python snippet using NumPy that perfectly translates this math into a program.

Machine Learning for Beginners: An Introduction to Neural Networks

Let's look at how to code this neuron. This will connect the mathematical theory directly to practical implementation.

Review the 'Coding a Neuron' subsection within Section 1. You don't need to run the code right now, but read through it and see how each line corresponds to the mathematical steps we just performed.

Conclusion

In this lesson, you have dissected the fundamental component of all neural networks. You now understand that an artificial neuron is a simple information-processing unit that is more about straightforward math than biological mystery.

Key Takeaways:

An artificial neuron calculates an output based on its inputs, weights, a bias, and an activation function.
Weights scale the importance of each input.
The bias acts as a trainable offset, making it easier or harder for the neuron to activate.
The activation function introduces non-linearity, allowing networks of neurons to learn complex patterns.
The entire calculation for a single neuron's output is $y = f ((w \cdot x) + b)$ .

Preview of the Next Lesson:

A single neuron can only do so much. The true power of neural networks comes from connecting many of them together. In the next lesson, "Describe the architecture of a multi-layer feedforward neural network," we will see how these individual neurons are organized into layers to create the powerful models that are at the heart of modern AI.

Recommended resources