Understanding Scalars, Vectors, and Matrices

Welcome to the first lesson of your machine learning journey! We are starting at the very beginning of your roadmap: building the mathematical foundations necessary to understand how machine learning algorithms work.

In this lesson, we will focus on the fundamental building blocks used to represent data in linear algebra. You will learn to define scalars, vectors, and matrices, and—most importantly—how to identify their dimensions. These concepts are the "alphabet" of data science and machine learning. Understanding them is the first step toward representing datasets, implementing algorithms in Python, and eventually building complex neural networks with PyTorch.

The Language of Data

Before we dive into definitions, it's helpful to understand why we need these concepts. Machine learning models, at their core, are mathematical functions that find patterns in numerical data. To perform these mathematical operations, we need a standardized way to structure that data. This is where linear algebra comes in.

To get a great visual and conceptual overview of these structures and their relevance to machine learning, please watch the following video.

Scalars, Vectors, Matrices, Tensors, etc explained | Linear Algebra Basics for Machine Learning

This video, "Scalars, Vectors, Matrices, Tensors, etc explained" from the Galaxy Inferno Codes channel, provides a clear and concise introduction to the basic objects of linear algebra and why they are crucial for machine learning.

Please watch the video from the beginning to the end. Pay close attention to: The motivation for using linear algebra in machine learning, especially in neural networks (introductory part). The definition and notation for a scalar. The definition of a vector and its representation as a column of numbers. The important difference between mathematical and programming notation for accessing vector elements. This is a key point for you as you begin your programming journey. The definition of a matrix as a two-dimensional array. The brief introduction to tensors, which generalizes these ideas to more dimensions and will be very important when we get to PyTorch.

As the video explained, we can think of these data structures in a hierarchy of complexity. Let's break them down one by one.

1. Scalars: The Simplest Form

A scalar is simply a single number. It has magnitude but no direction. Think of it as the most basic piece of information you can have. In terms of dimensionality, a scalar is considered a zero-dimensional (0D) object.

Examples of scalars include:

The temperature: 25.5°C
The age of a person: 34
The price of an item: $19.99
The value of a single pixel in a grayscale image.

In machine learning, you will often encounter scalars when dealing with performance metrics (like an accuracy score of 0.92) or configuration parameters (like a learning rate of 0.001).

2. Vectors: Data with Dimension

A vector is an ordered list of numbers. You can think of it as a row or, more commonly in mathematics, a column of scalars. Because it's a list, a vector is a one-dimensional (1D) object. The "dimension" of a vector refers to the number of elements it contains.

For example, this is a 4-dimensional vector because it contains four elements:

v = 7 - 2 50

Vectors are fundamental in machine learning because they are used to represent a single data point with multiple attributes or features. For instance, if you were trying to predict a house price, a single house might be represented by a vector containing its features:

house = area (sq. ft.) number of bedrooms age (years) = 1500310

This would be a 3-dimensional vector. In formal notation, we say this vector belongs to $R^{3}$ , which means it is a vector with 3 elements, where each element is a real number. A vector with $n$ elements is said to be in $R^{n}$ .

3. Matrices: Organizing Data in 2D

A matrix is a rectangular grid of numbers arranged in rows and columns. It is a two-dimensional (2D) object. You can also think of a matrix as a collection of vectors stacked together.

The dimensions of a matrix are given by its number of rows and columns, written as $m \times n$ , where $m$ is the number of rows and $n$ is the number of columns.

For example, here is a $2 \times 3$ matrix (2 rows, 3 columns):

A = [1450 3 - 2]

In machine learning, matrices are the standard way to represent an entire dataset. By convention, each row represents a single data point (like one house), and each column represents a specific feature (like area, number of bedrooms, etc.).

So, a dataset of four houses could be represented by a $4 \times 3$ matrix:

dataset = 15002100120018003423105252

To solidify these definitions and see some more examples, please read the following sections from the provided resources.

Introduction to Linear Algebra for Applied Machine Learning ...

This text provides a slightly more formal, mathematical perspective which will be helpful given your university math background. It introduces standard notation you will see in ML literature.

Focus on these key parts: Under "Elements of R", read the definition of a vector in R^n. In the next section, read about vector dimensions and how they map to coordinate systems. Jump to the "Matrices" section and read the formal definition of a matrix and its m \times n notation. Finally, under "Special matrices", quickly read the definitions of a rectangular matrix and a square matrix. This will help reinforce the concept of matrix dimensions.

Difference Between Scalar, Vector, Matrix and Tensor

This article from GeeksforGeeks offers very clear, point-form definitions and use cases that are great for review.

Please read the following sections to reinforce what we've covered. Don't worry about the Python code examples for now; we'll get to that in a later module. Read section 1. Scalar. Read section 2. Vector. Read section 3. Matrix. Finally, review the comparison table at the end. It's an excellent summary of the key differences.

From Scalar to Tensor: A Visual Hierarchy

The relationship between these concepts can be visualized as a hierarchy of increasing dimensions. A scalar is a single point. A vector is a line of scalars. A matrix is a grid of vectors. This progression can continue to even higher dimensions.

This image illustrates the hierarchy of data structures in linear algebra. It starts with a 0D scalar ([1]), progresses to a 1D vector of two elements, then a 2D matrix of 2x2 elements, and finally a 3D tensor composed of two 2x2 matrices.

As the video and image suggest, a tensor is a generalization of these concepts to any number of dimensions.

A 0D tensor is a scalar.
A 1D tensor is a vector.
A 2D tensor is a matrix.
A 3D tensor can be visualized as a cube of numbers (e.g., a color image with height, width, and color channels).

You don't need to master tensors right now, but it's valuable to know the term. When you start working with deep learning libraries like PyTorch, "tensor" will be the primary data structure you use for everything.

Conclusion

In this lesson, we established the foundational vocabulary for representing data in machine learning. You learned how to define scalars, vectors, and matrices and, crucially, how to describe their dimensions.

Here are the key takeaways:

Scalar: A single numerical value (0-dimensional).
Vector: An ordered list of numbers, representing a single data point with multiple features (1-dimensional). Its dimension is its length.
Matrix: A rectangular grid of numbers, representing an entire dataset with multiple data points (rows) and features (columns) (2-dimensional). Its dimensions are given as $m \times n$ (rows by columns).
Tensor: The general form for an n-dimensional array of numbers.

In our next lesson, we will build directly on this foundation by learning how to perform fundamental operations on these objects, such as vector addition and scalar multiplication. This will be our first step into manipulating the data structures we've just defined.