Introduction to Artificial Neural Network Architecture

1. From Logistic Regression to a Single Neuron

At its core, an Artificial Neural Network is a computational model composed of interconnected processing units. Before we look at a full network, let's start with its most basic building block: a single neuron, also known as a perceptron.

You might be surprised to learn that a simple logistic regression model, which you may have encountered in your statistics or finance studies, can be viewed as a one-neuron network. It takes several inputs, computes a weighted sum, and passes the result through a function (the sigmoid function) to produce a probability.

The following video from MIT OpenCourseWare provides an excellent demonstration of this idea, reframing a familiar concept into the language of neural networks.

As the video illustrates, a neuron is a simple computational unit that performs two main steps:

1. Aggregation: It calculates a weighted sum of its inputs and adds a bias. The weights determine the influence of each input, much like coefficients in a regression model. The bias acts like an intercept, allowing the neuron to adjust its output independently of its inputs.
2. Activation: The result of the aggregation is then passed through an activation function. This function transforms the aggregated value into the neuron's final output.

The diagram below shows this process. Inputs $x_1,x_2,x_3$ are multiplied by their respective weights $w_1,w_2,w_3$, summed together with a bias $b$, and then passed through an activation function $f(z)$ to produce the output $a$.

For a formal definition, the following resource provides a concise explanation.

2. Building the Network: Input, Hidden, and Output Layers

A single neuron is limited in what it can learn. The true power of ANNs comes from connecting many of these neurons together in a structured way, forming a network of layers.

A simple feedforward neural network, the most common type you'll see on an exam, has three types of layers:

• Input Layer: This is the entry point for your data. It consists of one neuron for each feature in your dataset. For example, if you were predicting a company's stock performance based on its P/E ratio, revenue, and debt level, your input layer would have three neurons. This layer doesn't perform any computation; it simply passes the data to the next layer.
• Hidden Layer(s): These are the layers between the input and output layers. A network can have one or more hidden layers. This is where the magic happens—each layer learns to detect progressively more complex features and patterns from the data passed on by the previous layer. The number of hidden layers and the number of neurons in each are key design choices that define the network's "architecture."
• Output Layer: This is the final layer that produces the model's prediction. The number of neurons in the output layer depends on the task. For a binary classification problem (e.g., will a customer default? Yes/No), you typically have one output neuron. For a regression problem (e.g., predicting a house price), you also have one neuron.

The connections are typically fully connected or dense, meaning every neuron in one layer is connected to every neuron in the next layer.

The StatQuest video below provides a very intuitive visual overview of how these layers fit together.

3. The "On" Switch: The Role of Activation Functions

We've mentioned activation functions, but why are they so important? Their crucial role is to introduce non-linearity into the network.

Imagine a network without activation functions (or with a simple linear one). Each layer would just be performing a linear transformation (like matrix multiplication). No matter how many layers you stack, the combination of multiple linear functions is still just one big linear function. Such a network could only learn linear relationships, just like a simple linear regression model. It couldn't capture complex, "squiggly" patterns in the data.

Activation functions are the non-linear "switches" that allow the network to bend and shape its decision boundary, enabling it to learn far more complex patterns. For your exam, you should be familiar with a couple of the most common ones:

• Sigmoid Function: This function squashes any input value into a range between 0 and 1. This makes it a natural choice for the output layer in a binary classification problem, where the output can be interpreted as a probability. Its formula is $\sigma (z)=\frac{1}{1+e^{-z}}$.
• ReLU (Rectified Linear Unit): This is the most popular activation function for hidden layers. It is very simple: it outputs the input directly if it's positive, and outputs zero otherwise. Its formula is $f(z)=\max (0,z)$. This simplicity makes it computationally very efficient.

The MIT lecture provides a fantastic explanation of why we need non-linearity and introduces these key functions.

This table from the "Neural Network Architecture Guide" provides a great summary of the components we've discussed.

Conclusion

In this lesson, you've learned the fundamental architectural components of an Artificial Neural Network. You are now prepared to define and describe these concepts in an exam setting.

Here are the key takeaways:

• Neuron: The basic building block of an ANN, which computes a weighted sum of its inputs, adds a bias, and passes the result through an activation function.
• Layers: Neurons are organized into an input layer (for data), one or more hidden layers (for pattern detection), and an output layer (for predictions).
• Activation Functions: These introduce essential non-linearity into the network, allowing it to model complex relationships. ReLU is standard for hidden layers, while Sigmoid is often used for binary classification outputs.

We have now established the what—the static structure of a neural network. In our next lesson, we will explore the how. We will cover Backpropagation, the core algorithm that allows the network to learn from data by adjusting its weights and biases to make better predictions.