Welcome to the next step in our journey to build a high-performance vector search engine. In the last lesson, you successfully vectorized the Euclidean distance calculation, replacing a sequence of scalar operations with powerful AVX2 instructions. You learned how to load, process, and reduce 256-bit vectors, dramatically cutting down the instruction count needed for the computation.
Today, we will build on that foundation to implement the dot product, which is the heart of the Cosine Similarity metric. This lesson introduces one of the most important instructions in modern CPUs for high-performance computing: the Fused Multiply-Add (FMA). Mastering FMA is essential for pushing the boundaries of computational throughput, a skill highly relevant to your interests in low-level performance and systems programming.
The Power of Fusing Operations
At its core, a dot product is a sum of products: . A naive implementation would loop through the vectors, performing a multiplication and then an addition in each iteration. However, modern CPUs can do better. The Fused Multiply-Add instruction combines these two steps, (a * b) + c, into a single hardware operation.
The benefits of FMA extend beyond just reducing cycles. The following video provides an excellent deep dive into why FMA is so powerful.
SIMD and vectorization using AVX intrinsic functions (Tutorial)
This segment from Joel Carpenter's tutorial explains the technical advantages of using Fused Multiply-Add operations.
Watch the portion of the video discussing multiply-accumulate operations. Pay close attention to the two key advantages he describes: Throughput and Latency: FMA offers two floating-point operations (a multiply and an add) for the price of one, potentially doubling your raw FLOPS. Accuracy: By performing the a * b part with higher internal precision and only rounding once after the addition to c, FMA reduces numerical error compared to two separate operations with two separate rounding steps.
The intrinsic function for a packed single-precision FMA operation on 256-bit vectors is _mm256_fmadd_ps(a, b, c). To use it, you'll need to ensure your compiler is instructed to generate code for a CPU that supports this instruction set. This is typically done by adding the -mfma flag during compilation, or by using a broader flag like -march=native which enables all instruction sets supported by your machine, including FMA.
Understanding SIMD Performance: A Developer's Introduction with ...
This article by Necati Demir provides a quick reference for the FMA intrinsic and the necessary compiler flags.
Quickly review the list of key intrinsics in the section on AVX2 fundamentals to see _mm256_fmadd_ps. Then, check the compilation requirements in the FAQ sections that mention the -mfma flag and reiterate its importance.
Implementing Dot Product with FMA and Loop Unrolling
Now, let's construct the dot product function. The logic sum = sum + (a_i * b_i) is a perfect fit for _mm256_fmadd_ps. The core of our function will be a loop where we repeatedly update an accumulator vector:accumulator = _mm256_fmadd_ps(vec_a, vec_b, accumulator);
However, there's a subtlety here related to performance. Modern CPUs can execute multiple instructions at once (superscalar execution), but an instruction like our FMA has a certain latency—it takes a few clock cycles from when the instruction starts to when its result is available. If we use only one accumulator, each loop iteration has to wait for the previous one to complete, creating a dependency chain that stalls the pipeline.
The solution is a technique called loop unrolling with multiple accumulators. Instead of one running sum, we maintain several (e.g., four). In each iteration of our unrolled loop, we perform FMA operations on all four accumulators with different chunks of data. This hides the latency of any single FMA operation, as the CPU can work on the others while the first is still "in flight".
This is a professional-grade optimization technique. The following resource provides a fantastic, complete C++ implementation that uses this exact approach.
c++ - AVX2: Computing dot product of 512 float arrays - Stack Overflow
This Stack Overflow answer by a user named Soonts is a canonical example of a high-performance SIMD dot product. We will focus on the first version of the code provided in the answer.
Read the complete code block in the accepted answer. Don't worry about the template functions at the top for now; focus on the logic inside the dotProductFma function. As you read, identify these key sections: Initialization: Notice the use of four separate __m256 accumulators: dot0, dot1, dot2, and dot3. The Main Loop: See how it processes 32 floats (4 vectors of 8 floats) in every single iteration by updating each of the four accumulators. This is loop unrolling in action. Horizontal Reduction: After the loop, observe how the four accumulator vectors are first added together (_mm256_add_ps) into a single vector. This final vector is then reduced to a single scalar float using the same horizontal sum technique you saw in the previous lesson.
The code you just studied is a significant step up. It's not just a direct translation of the math; it's an implementation that is carefully structured to respect the realities of the underlying hardware pipeline. By using four accumulators, the code breaks the dependency chain and allows the CPU's execution units to stay saturated with useful work.
After summing the four accumulators into a final vector, the reduction logic is identical to what you saw before for Euclidean distance. It collapses the 8 partial sums within the vector into a single scalar value which is our final dot product result.
Note that the provided code assumes the vector length is a multiple of 32. The author later provides an updated version that handles arbitrary lengths using masked loads. While this is important for a robust library, for this lesson, we are focused on mastering the core FMA and unrolling concepts.
Conclusion
In this lesson, you've added a powerful new tool, Fused Multiply-Add, to your performance engineering arsenal. You've gone beyond a simple translation of a formula into SIMD and implemented a more advanced, latency-aware optimization with loop unrolling. This approach is fundamental to achieving near-theoretical peak performance in computationally-bound numerical code.
Key Takeaways:
- Fused Multiply-Add (FMA) executes
(a * b) + cas a single, highly efficient instruction, boosting throughput and improving numerical accuracy. - The
_mm256_fmadd_psintrinsic is the function for performing FMA on 256-bit vectors of floats. - Loop unrolling with multiple accumulators is a critical technique to hide instruction latency and exploit the instruction-level parallelism of modern CPUs.
- A high-performance dot product can be implemented as an unrolled loop of FMA instructions followed by a final reduction of the accumulator registers.
You are now very close to having a complete, accelerated Cosine Similarity function. In the next lesson, we will implement the final missing piece: calculating the squared L2 norm of a vector. As you might guess, this is simply the dot product of a vector with itself, so you will be able to apply today's knowledge directly.