Single Neuron: Structure and Output Calculation

Hello! Welcome to the second module of our course, Neural Networks from Scratch.

In the previous module, you built a strong foundation in practical machine learning using Scikit-learn. You mastered the entire workflow, from data preprocessing to hyperparameter tuning, and learned how to encapsulate these steps into robust, reproducible Pipelines. This is a critical skill for any machine learning practitioner.

Now, we're going to shift our focus from using pre-built models to understanding what goes on inside them. We will open the "black box" of deep learning and build our own neural network using only Python and NumPy. This will provide you with a deep, first-principles understanding that is essential for becoming a proficient AI engineer.

This first lesson begins with the most fundamental building block of any neural network: the artificial neuron. By the end of our session, you will be able to explain the structure of a single neuron and compute its output given a set of inputs, weights, bias, and an activation function.

1. From Linear Models to a Neuron

You're already familiar with linear models from your machine learning theory background. A simple linear regression equation for multiple inputs looks like this:

This exact calculation—a weighted sum of inputs plus an intercept (or bias)—forms the core of an artificial neuron. Let's formalize this structure.

2. The Anatomy of an Artificial Neuron

An artificial neuron (also known as a perceptron) is a simple computational unit that takes inspiration from its biological counterpart. It processes information through a few key steps.

As the diagram shows, a neuron has the following components:

• Inputs ($x_1,x_2,\dots ,x_n$): These are the features from your dataset that are fed into the neuron. For the very first layer of a network, this could be the pixel values of an image or columns from a tabular dataset.
• Weights ($w_1,w_2,\dots ,w_n$): Each input has an associated weight. This weight determines the importance of that input. A larger weight means the input has more influence on the neuron's output. These are the main parameters the network "learns" during training.
• Bias ($b$): This is a single, extra parameter that is added to the weighted sum. The bias allows the neuron to shift the output of the activation function up or down. A key role of the bias is to allow the neuron to produce a non-zero output even when all inputs are zero.
• Activation Function ($f$): The total sum ($\sum w_i{x}_i+b$) is passed through an activation function. This function introduces non-linearity, which is crucial. Without it, a multi-layer network would be mathematically equivalent to a single, simple linear model, and would be unable to learn the complex patterns required for tasks like image recognition.

3. Calculating the Neuron's Output

The process of calculating a neuron's output from its inputs is called feedforward. It's a two-step calculation.

To dive into this process, let's go through a clear, step-by-step explanation.

As you've just read, the calculation unfolds as follows:

1. Calculate the Net Input (Weighted Sum): First, we compute the dot product of the input vector $\mathbf{x}$ and the weight vector $\mathbf{w}$, and then add the bias $b$. This value is often denoted by $z$.

   $$z=(\mathbf{w}\cdot \mathbf{x})+b=(w_1{x}_1+w_2{x}_2+\cdots +w_n{x}_n)+b$$

2. Apply the Activation Function: Next, we pass the net input $z$ through an activation function, $f$, to get the final output $y$. The sigmoid function, $\sigma (z)$, which squashes any real number into the range (0, 1), is a common choice for introductory examples.

   $$y=f(z)=\sigma (z)=\frac{1}{1+e^{-z}}$$

The video below offers a great visual walkthrough of this exact process.

4. Practice: Your Turn to Calculate

Now it's time to put this knowledge into practice. This will confirm you've met today's learning objective.

Problem:
Consider a neuron with 3 inputs that uses the sigmoid activation function. Its parameters are:

• Weights: $\mathbf{w}=[0.5,-1.0,0.2]$
• Bias: $b=-0.1$

If the neuron receives the input vector $\mathbf{x}=[4,2,5]$, what is its final output?

Take a moment to calculate it yourself before checking the solution below.

Conclusion

In this lesson, you've taken the first crucial step into the world of deep learning. You've deconstructed the artificial neuron, the fundamental unit that powers even the most complex models like GPT-4.

Key Takeaways:

• An artificial neuron is a simple computational unit that calculates a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation function.
• The neuron's adjustable parameters, which are "learned" during training, are its weights and bias.
• The activation function is essential for introducing non-linearity, enabling neural networks to learn complex relationships in data.
• The process of computing an output from a set of inputs is called feedforward.

We focused on the sigmoid function today for its simplicity. However, there are several other activation functions, each with unique properties. In our next lesson, we will explore and compare the most common ones—Sigmoid, Tanh, and ReLU—and understand when and why to use each. This will be our next step toward building a complete neural network layer.