Output Shapes in Neural Network Vector and Matrix Multiplication

Hello! This is the first lesson in the mathematical foundations of the course. Your introductory machine-learning work has already involved data with rows of examples and columns of features. Neural networks use that same kind of data, but pass it through many matrix operations. To follow those operations—and eventually the equations in Attention Is All You Need—you need to know what shape each result will have before calculating any numbers.

Today’s focus is shape reasoning: deciding whether a vector or matrix multiplication is valid, then finding the output shape. Allow about 35–40 minutes, including the short videos and reading.

---

The rule that governs matrix multiplication

A matrix’s shape is written rows by columns. A matrix with two rows and four columns has shape $2\times 4$, regardless of the values in its entries.

Suppose $A$ has shape $m\times n$ and $B$ has shape $n\times p$. Their product has shape

$$(m\times n)(n\times p)=(m\times p).$$

The inner dimensions must match: $A$ must have as many columns as $B$ has rows. The outer dimensions give the result: it has $A$’s number of rows and $B$’s number of columns. If the inner dimensions differ, this matrix product is not defined.

Watch Khan Academy establish the condition and show how entries of the result are formed:

Why does the inner dimension have to match? Each output entry pairs a row of $A$ with a column of $B$, multiplying corresponding values and adding them. Those two groups need the same number of values. There is one output entry for every choice of an $A$ row and a $B$ column, which explains the $m\times p$ result.

In the illustrated example, the top-left output entry is $50$: the first row of $A$ is paired with the first column of $B$. You do not have to calculate the other five entries to know that the product has shape $2\times 3$. In a neural network, checking shapes first can save you from doing substantial arithmetic—or debugging—with incompatible inputs.

---

Vectors follow the same rule

A vector can be written as a column vector, with shape $n\times 1$, or a row vector, with shape $1\times n$. Its orientation matters when you multiply it by a matrix.

If $A$ has shape $m\times n$ and $x$ is a column vector of shape $n\times 1$, then

$$Ax:(m\times n)(n\times 1)=(m\times 1).$$

The result is a column vector with $m$ entries. For example, a matrix that turns four input features into three outputs has shape $3\times 4$. Multiplying it by one $4\times 1$ input gives a $3\times 1$ output.

The row-vector version puts the input on the left. If $x$ has shape $1\times n$ and $W$ has shape $n\times p$, then

$$xW:(1\times n)(n\times p)=(1\times p).$$

Here, one row of $n$ input features becomes one row of $p$ outputs. Neither layout is inherently more correct; the weight matrix must be arranged to match the chosen layout.

Two other vector products reveal why orientation cannot be ignored. A $1\times n$ row vector times an $n\times 1$ column vector produces a $1\times 1$ result—a single value. In the opposite order, an $n\times 1$ column vector times a $1\times n$ row vector produces an $n\times n$ matrix. The entries could involve the same vector values, yet the order changes the output shape. We will explore the meaning of the single-value product in the next lesson.

One notation detail will matter when we reach code: a one-dimensional array may have shape written as $(n,)$, rather than explicitly $(1,n)$ or $(n,1)$. It has $n$ elements but no visible row or column axis. For now, use explicit row and column shapes when reasoning on paper; later, we will see how PyTorch handles one-dimensional tensors.

---

Recognizing the operation inside a neural network

A neural network layer often takes several input features and produces several output features. Matrix multiplication lets it perform those calculations together. This is a move beyond an introductory workflow that treats a model as a function accepting a feature table: we will track the dimensions inside that function.

Watch the neural-network portion of StatQuest’s explanation. You need only the transformation shapes here, not the details of training or activation functions.

In that example, the first multiplication takes two input features to two intermediate values: its output shape is $1\times 2$. A later multiplication takes those two values to three class scores: its output shape is $1\times 3$. The shapes tell you how many values exist at each stage, even if you do not know the weights.

Stanford’s CS231n notes provide the same idea using column vectors. Read the worked network example to see why its multiplication order looks different from the video’s while following the same rule:

CS231n starts with a $3\times 1$ input column. Its first weight matrix is $4\times 3$, so the first layer returns a $4\times 1$ column:

$$(4\times 3)(3\times 1)=(4\times 1).$$

The next weight matrix is $4\times 4$, so the feature count stays at four. The last is $1\times 4$, producing a $1\times 1$ output. At each layer, the output feature count becomes the required input feature count of the next layer. That is the shape relationship that makes a chain of layers possible.

---

From one example to a batch

Models commonly process multiple examples at once. Put $b$ examples into a matrix $X$, one example per row, with $d_{\text{in}}$ features per example. Then $X$ has shape $b\times d_{\text{in}}$. To produce $d_{\text{out}}$ values per example, use a weight matrix $W$ with shape $d_{\text{in}}\times d_{\text{out}}$:

$$XW:(b\times d_{\text{in}})(d_{\text{in}}\times d_{\text{out}})=(b\times d_{\text{out}}).$$

For instance, five examples with four features each can be transformed into three scores each:

$$(5\times 4)(4\times 3)=(5\times 3).$$

The batch still contains five examples. Each example now has three outputs instead of four input features. This is a useful way to read a result shape: the first dimension counts examples; the second counts the values produced for each example.

CS231n briefly describes a different batch layout: examples can instead be placed in columns. In that convention, the weight matrix goes on the left. When reading neural-network notation, do not memorize a single multiplication order without checking it. Identify what each dimension counts, confirm the inner dimensions match, and then read the outer dimensions as the output.

That habit will remain useful when the matrices in the Transformer paper become larger and acquire names such as queries and keys. Their roles come later; their multiplication shapes obey exactly the rule learned here.

---

Key takeaways

For a matrix product, check the inner dimensions to decide whether it is valid and keep the outer dimensions to find its shape. Treat a row vector as $1\times n$ and a column vector as $n\times 1$; their orientation can change the result from one value to an entire matrix. In a neural-network layer, follow what each axis counts—examples, input features, or output features—rather than relying on memorized symbols.

Next, we will examine the dot product: the single-value calculation inside matrix multiplication, and what it can tell us about the alignment of two vectors.