Computing Weighted Sums of Vectors

Recommended resources

The freely available resources this lesson will direct you to.

Hello again. Last lesson, a dot product combined two vectors to produce one number. This lesson uses numbers in a different role: as weights that scale several vectors before we add them. The result is another vector.

This is one of the operations you will eventually recognize in Attention Is All You Need. For now, we will take the weights as given and concentrate on calculating the resulting vector. Allow about 35–40 minutes, including the video and reading.

Scaling and adding vectors

Suppose two vectors describe different directions. Multiplying a vector by a scalar changes its length; a negative scalar also reverses its direction. Adding the scaled vectors produces a new vector. We call this a weighted sum, or a linear combination:

y = w_{1} v_{1} + w_{2} v_{2} + \dots + w_{m} v_{m} .

Here each $w_{i}$ is a scalar weight, and each $v_{i}$ is a vector. There must be one weight for each vector, and all the vectors must have the same number of components so that they can be added.

Watch selected parts of 3Blue1Brown’s Linear combinations, span, and basis vectors for a visual account of this operation. Its coordinate-axis vectors are simply the vectors being scaled; you do not need to study the later discussion of span or basis yet.

Linear combinations, span, and basis vectors | Chapter 2, Essence of linear algebra

In 3Blue1Brown’s Linear combinations, span, and basis vectors, the animation makes the two operations in a weighted sum—scaling and addition—visible.

Watch scaled vectors to see how two numbers scale two vectors and their results add. Then jump to the name for this operation, and three vectors to see that adding a third vector does not change the rule. Focus on what the numbers do to the vectors, rather than on the terminology introduced elsewhere in the video.

The supplied diagram illustrates the same idea with two weights:

The diagram labels the orange resultant as $\mathbf v=2\mathbf u_1+1.5\mathbf u_2$: each original vector receives its own scalar weight before the contributions are combined.

The picture helps with direction, but components are the reliable way to calculate an exact answer.

Calculating component by component

Take three two-component vectors and three weights:

v_{1} v_{2} v_{3} = (2, 0), = (0, 4), = (- 2, 2), w_{1} w_{2} w_{3} = 0.5, = 0.25, = 0.25.

First multiply every component of each vector by its weight. Then add the resulting vectors component by component:

y = 0.5 (2, 0) + 0.25 (0, 4) + 0.25 (- 2, 2) = (1, 0) + (0, 1) + (- 0.5, 0.5) = (0.5, 1.5) .

There are three input vectors, but the output still has two components, because every vector being added has two components. Looking at one output coordinate at a time gives the same calculation:

y_{1} = 0.5 (2) + 0.25 (0) + 0.25 (- 2) = 0.5,

y_{2} = 0.5 (0) + 0.25 (4) + 0.25 (2) = 1.5.

The weights in this example are nonnegative and sum to one. That makes the result a weighted average of the vectors: each vector contributes a share of the output. But weighted sums do not generally require those conditions. For instance,

2 (1, - 1) - 1 (3, 2) = (2, - 2) + (- 3, - 2) = (- 1, - 4) .

A negative weight reverses a vector’s contribution; a zero weight removes it. Neither case changes the procedure.

Notice the contrast with last lesson’s dot product. In $a \cdot b$ , paired components are multiplied and added to obtain a scalar. In a weighted sum, each scalar multiplies an entire vector, and adding those scaled vectors produces a vector.

The matrix form is the same calculation

Matrices provide a compact way to arrange the vectors and weights. Place the vectors as columns of a matrix $A$ , and put their weights in a column vector $w$ . Matrix-vector multiplication then gives their weighted sum:

A = ∣ v_{1} ∣ ∣ v_{2} ∣ ∣ v_{3} ∣, w = w_{1} w_{2} w_{3}, A w = w_{1} v_{1} + w_{2} v_{2} + w_{3} v_{3} .

In our worked example, the three two-component vectors make $A$ a $2 \times 3$ matrix:

[2004 - 2 2] 0.5 0.25 0.25 = [0.5 1.5] .

This also connects to the shape rules you learned two lessons ago. The matrix has one column per weight and one row per output component. Thus a $2 \times 3$ matrix times a $3 \times 1$ weight vector produces a $2 \times 1$ result.

Read the worked example and definition in the following LibreTexts section. Its numbers differ from ours, but the operation is identical.

2.2: Matrix multiplication and linear combinations

LibreTexts’ Matrix multiplication and linear combinations shows why multiplying a matrix by a vector means forming a weighted sum of the matrix’s columns. Read it to connect the component calculation to matrix notation and output shape.

In Section 2.2.2, “Matrix-vector multiplication and linear combinations,” begin at Example 2.2.1 with the sentence “Suppose we have the matrix” and read the example and definition through Definition 2.2.2, stopping before Activity 2.2.2. Track which numbers are weights, which vectors are columns, and why the output has three components. You can skip the activity and the later material on solving linear systems.

There is another common layout: store the same vectors as rows instead. Then a $1 \times 3$ row of weights multiplying a $3 \times 2$ matrix produces a $1 \times 2$ weighted sum. The layout changes, but the component-by-component calculation does not.

Why this matters for attention

Recall that the dot products mentioned last lesson will eventually help an attention mechanism assign scores. After a further step turns scores into weights, attention uses those weights to combine vectors—often called value vectors—with precisely the operation learned here.

For example, if three value vectors receive weights $0.5$ , $0.25$ , and $0.25$ , the output is half of the first vector plus one quarter of each remaining vector, calculated component by component. A larger weight gives a vector more influence on that output; it does not replace the other vectors unless their weights are zero. We will examine where attention’s weights come from later.

Key takeaways

To compute a weighted sum, multiply each vector by its corresponding scalar and add the scaled vectors component by component. The output has the same number of components as each input vector. Writing the vectors as matrix columns makes matrix-vector multiplication a compact expression for exactly this calculation.

Next lesson, you will meet softmax, a way to turn a list of scores into nonnegative weights that sum to one.