Dot Products and Matrix Multiplication

1. The Vector Dot Product: Measuring Similarity

We know how to add and scale vectors, but how do we quantify the relationship between two vectors? For example, do they point in similar directions? Are they perpendicular? The dot product provides the answer.

Algebraic and Geometric Definitions

The dot product has two equivalent definitions.

1. Algebraic Definition:
The dot product (also called the inner product) of two vectors $\mathbf{u}$ and $\mathbf{v}$ in ${\mathbb{R}}^n$ is the sum of the products of their corresponding components. The result is a single number (a scalar).

For example, if $\mathbf{u}=[1,2,3]$ and $\mathbf{v}=[4,-5,6]$, their dot product is:
$\mathbf{u}\cdot \mathbf{v}=(1)(4)+(2)(-5)+(3)(6)=4-10+18=12$.

2. Geometric Definition:
Geometrically, the dot product is defined as the product of the magnitudes (lengths) of the two vectors and the cosine of the angle $\theta$ between them.

Interpreting the Dot Product

This geometric formula is key to interpretation. The value of $\cos (\theta )$ tells us about the alignment of the vectors:

• If $\theta =0^{\circ }$, the vectors are parallel and point in the same direction. $\cos (0^{\circ })=1$, so $\mathbf{u}\cdot \mathbf{v}=\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert$. The dot product is at its maximum positive value.
• If $\theta =90^{\circ }$, the vectors are orthogonal (perpendicular). $\cos (90^{\circ })=0$, so $\mathbf{u}\cdot \mathbf{v}=0$. A zero dot product is a crucial indicator of orthogonality.
• If $\theta =180^{\circ }$, the vectors are parallel and point in opposite directions. $\cos (180^{\circ })=-1$, so $\mathbf{u}\cdot \mathbf{v}=-\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert$. The dot product is at its maximum negative value.

In short, the dot product measures the "similarity" or "agreement" between two vectors. A large positive value means they are long and point in a similar direction, while a large negative value means they are long and point in opposite directions.

Another powerful interpretation is that of projection. The term $\Vert \mathbf{u}\Vert \cos (\theta )$ is the length of the orthogonal projection of vector $\mathbf{u}$ onto vector $\mathbf{v}$. So, the dot product can be seen as the length of the projection of $\mathbf{u}$ onto $\mathbf{v}$, scaled by the length of $\mathbf{v}$.

To dive deeper into these interpretations, and to see the derivation that connects the algebraic and geometric definitions (which I think you'll find interesting given your preference for first principles), let's turn to a curated discussion.

The derivation you just read is a beautiful piece of mathematics. It shows that the simple algebraic operation of $\sum u_i{v}_i$ inherently contains geometric information about the angle between the vectors, all without ever explicitly calculating an angle.

The Dot Product in Python

In NumPy, the @ operator is used for dot products (and matrix multiplication, as we'll see).

import numpy as np

u = np.array([1, 2, 3])
v = np.array([4, -5, 6])

# Calculate the dot product
dot_product = u @ v  # or np.dot(u, v)
print(f"The dot product of u and v is: {dot_product}")

2. Matrix Multiplication: Composing Transformations

Just as the dot product was not an element-wise operation, neither is matrix multiplication. Its definition is specifically designed to represent the composition of linear transformations.

Imagine a vector $\mathbf{v}$. Applying a matrix $\mathbf{B}$ to it gives a new vector $\mathbf{u}=\mathbf{Bv}$. If we then apply another matrix $\mathbf{A}$ to $\mathbf{u}$, we get a final vector $\mathbf{w}=\mathbf{Au}=\mathbf{A}(\mathbf{Bv})$. Matrix multiplication defines a single matrix $\mathbf{C}=\mathbf{AB}$ that accomplishes this entire two-step transformation in one go: $\mathbf{w}=\mathbf{Cv}$.

Matrix-Vector Multiplication

Let's start with multiplying a matrix $\mathbf{A}$ (with $m$ rows, $n$ columns) by a vector $\mathbf{v}$ (with $n$ elements). The result is a new vector $\mathbf{w}$ with $m$ elements. There are two vital ways to view this:

1. Row View: Each element of the output vector $\mathbf{w}$ is the dot product of the corresponding row of matrix $\mathbf{A}$ with the vector $\mathbf{v}$. $$w_i=(\text{row }i\text{ of }\mathbf{A})\cdot \mathbf{v}$$
2. Column View: The output vector $\mathbf{w}$ is a linear combination of the columns of matrix $\mathbf{A}$, where the weights are the elements of vector $\mathbf{v}$. $$\mathbf{w}=v_1(\text{col }1\text{ of }\mathbf{A})+v_2(\text{col }2\text{ of }\mathbf{A})+\cdots +v_n(\text{col }n\text{ of }\mathbf{A})$$

The row view is useful for computation, but the column view is arguably more powerful for interpretation, as it shows how the matrix transforms the input vector into an output vector within the "space" defined by its columns.

Matrix-Matrix Multiplication

Now we can extend this to multiplying two matrices, $\mathbf{A}$ ($m\times n$) and $\mathbf{B}$ ($n\times p$). The product $\mathbf{C}=\mathbf{AB}$ will be an $m\times p$ matrix.

Rule: The element at row $i$ and column $j$ of the product matrix $\mathbf{C}$ is the dot product of the $i$-th row of $\mathbf{A}$ and the $j$-th column of $\mathbf{B}$.
$C_{ij}=(\text{row }i\text{ of }\mathbf{A})\cdot (\text{column }j\text{ of }\mathbf{B})$

Notice the dimension constraint: For the product $\mathbf{AB}$ to be defined, the number of columns in $\mathbf{A}$ must match the number of rows in $\mathbf{B}$.

To solidify these concepts, let's read a clear, geometrically-motivated explanation.

Properties and Python Implementation

One of the most important properties of matrix multiplication is that it is not commutative: in general, $\mathbf{AB}\ne \mathbf{BA}$. The order matters, just as the order of applying transformations matters.

In NumPy, the @ operator also performs matrix multiplication.

A = np.array([[1, 2],
              [3, 4]])

B = np.array([[5, 6],
              [7, 8]])

# Matrix Multiplication
C = A @ B
print("A @ B =\n", C)

# Let's check the first element C[0, 0] manually
# It's the dot product of row 0 of A and column 0 of B
# C[0, 0] = (1 * 5) + (2 * 7) = 5 + 14 = 19
# This matches the output.

# Let's check non-commutativity
D = B @ A
print("\nB @ A =\n", D)
# As you can see, A @ B is not the same as B @ A.

Conclusion

Fantastic work today! You've tackled two of the most fundamental operations in linear algebra, moving beyond simple element-wise arithmetic into more abstract and powerful concepts.

Here are the key takeaways:

• The dot product of two vectors can be computed algebraically ($\sum u_i{v}_i$) or geometrically ($\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert \cos (\theta )$). It measures the similarity or alignment between vectors and is zero for orthogonal vectors.
• Matrix-vector multiplication can be viewed as applying dot products between rows and the vector, or as creating a linear combination of the matrix's columns.
• Matrix-matrix multiplication is defined by dot products between the rows of the first matrix and columns of the second. It represents the composition of linear transformations and is not commutative.

Preview of the Next Lesson:
Now that we understand how matrices transform vectors, a natural question arises: can we "undo" a transformation? This leads to the concepts of the determinant and inverse of a matrix. You will learn how to calculate these values and understand what they tell us about the properties of a matrix and the transformation it represents.