Hybrid Search: Combining Sparse and Dense Retrieval

Introduction

In our last lesson, you successfully built a complete RAG pipeline using LangChain. We focused on dense retrieval, using semantic search to find documents based on their conceptual meaning. This is a powerful technique, but it has its blind spots.

Today, we'll address those limitations by exploring the other major family of retrieval methods. This lesson is designed to help you master the learning outcome: Explain sparse retrieval methods (e.g., BM25) and implement a hybrid search approach combining dense and sparse retrievers.

We'll start by understanding why dense retrieval isn't always enough, dive into the mechanics of the classic keyword-based algorithm BM25, and then combine the two approaches into a robust hybrid system that leverages the strengths of both.

Part 1: The Limits of Dense Retrieval and the Rise of Sparse Methods

Dense retrieval, powered by embeddings, is excellent at understanding semantic similarity. If you search for "ways to make a company bigger," it can easily find a document that talks about "strategies for business growth." However, it can struggle with queries where specific, exact words are critical.

To see why, let's explore the inherent limitations of vector search when dealing with keywords, jargon, and numerical identifiers.

Why vector search is not enough and we need BM25

The video 'Why vector search is not enough and we need BM25' from Diffbot provides an excellent explanation of why dense embeddings can fail when precision is required.

Watch the first part of this video (from 00:00 to 04:16). Focus on how embeddings handle: Numerical values: Notice they are interpreted contextually, not mathematically. Time-related terms: Words like 'latest' and 'recent' are grouped by context, not a timeline. Exact keyword matches: Why a search for 'apple pie' might return 'cherry pie'.

This is where sparse retrieval comes in. Unlike dense retrieval which uses dense vectors (where most values are non-zero), sparse retrieval methods rely on sparse vectors (where most values are zero). These methods are fundamentally about keyword matching.

The most prominent and widely used sparse retrieval algorithm today is Okapi BM25 (Best Matching 25). It's a sophisticated evolution of the classic TF-IDF (Term Frequency-Inverse Document Frequency) algorithm and serves as the default search algorithm in systems like Elasticsearch.

BM25 calculates a relevance score for each document based on the query terms. It does this by intelligently balancing three key factors:

Term Frequency (TF): How often does a query term appear in a document? BM25 includes a saturation point (k1 parameter), meaning that after a certain number of mentions, repeating the term adds diminishing returns. This prevents "keyword stuffing."
Inverse Document Frequency (IDF): How rare is the term across all documents? Rare and important terms (like "RoPE" or "GQA") are given more weight than common words (like "the" or "model").
Document Length Normalization: How long is the document? BM25 penalizes very long documents that mention a term once, assuming the term is less relevant than in a short, concise document where it's mentioned once (b parameter).

The same video provides a clear, conceptual breakdown of how these components work together.

Why vector search is not enough and we need BM25

Let's continue with the Diffbot video to understand the mechanics of BM25.

Watch the segment from 05:46 to 08:05. Pay attention to the explanation of each BM25 component: term frequency saturation, inverse document frequency, and document length normalization.

Dense vs. Sparse: A Summary

Feature	Dense Retrieval (e.g., Semantic Search)	Sparse Retrieval (e.g., BM25)
Core Mechanism	Vector similarity (e.g., cosine similarity)	Weighted keyword counting
Strengths	Understands synonyms, paraphrasing, concepts	Precise keyword matching, acronyms, codes, specific jargon
Weaknesses	Can miss critical keywords, poor with numbers/IDs	No semantic understanding, fails if exact words aren't present
Computational	Expensive embedding process, fast ANN search	Fast indexing, generally very fast search
"When to use it"	General questions, conceptual search	Code search, legal documents, product catalogs, fact-checking

Test your understanding!

You are building a RAG system for an e-commerce company. Which retrieval method (Dense, Sparse, or both) would be most critical for handling the following queries, and why?

"Show me stylish jackets for cold weather"
"Find product with SKU XZ-48-B-V2"
"Are there any customer reviews that mention 'faulty zipper' for the 'Alpine Parka'?"

Show answer

Dense Retrieval: This query is conceptual. "Stylish" and "cold weather" are subjective and can be described in many ways. Dense retrieval is needed to match these concepts to product descriptions like "fashionable winter coat" or "elegant outerwear."
Sparse Retrieval: This query requires an exact match for the product stock-keeping unit (SKU) XZ-48-B-V2. Dense retrieval would likely fail as the embedding for this specific code might not be meaningful or distinct. BM25 is designed for this kind of precise, keyword-based lookup.
Both (Hybrid): This query has two parts. "faulty zipper" is a very specific keyword phrase that sparse retrieval would excel at finding. "Alpine Parka" is also a specific name. However, the overall context of a customer review is unstructured text where semantic understanding could be beneficial. A hybrid approach ensures you find the exact phrase "faulty zipper" while also understanding the context around the product "Alpine Parka".

Part 2: Hybrid Search - The Best of Both Worlds

Since dense and sparse retrievers have complementary strengths, the logical next step is to combine them. This is called hybrid search. A hybrid system runs a query through both a dense retriever and a sparse retriever in parallel and then intelligently merges the results.

Hybrid Search in RAG Production Systems — This diagram illustrates a typical hybrid search pipeline in a RAG system. The user's query is sent to both a sparse (keyword) index and a dense (vector) index. The results from both are then combined by a fusion algorithm before being passed to the LLM.

The most challenging part of hybrid search is the fusion step: how do you combine two lists of results that have completely different scoring systems? BM25 scores can be unbounded, while cosine similarity is typically between -1 and 1. There are two common fusion methods.

Top 3 RAG Retrieval Strategies: Sparse, Dense, & Hybrid Explained

The video 'Top 3 RAG Retrieval Strategies' from IBM Technology gives a great overview of the different strategies, culminating in hybrid search and its fusion algorithms.

Watch the section on hybrid retrieval from 05:10 to 07:15. Focus on the two fusion methods discussed: Weighted Sum: A simple, weighted average of normalized scores. Reciprocal Rank Fusion (RRF): A more robust method that uses ranks instead of scores.

Reciprocal Rank Fusion (RRF) is often preferred because it's parameter-free and robust. It doesn't care about the actual scores, only the ranks. The formula for a document's RRF score is:

$RRF_Score = \sum_{i \in retrievers} \frac{1}{k + rank _{i}}$

Where rank_i is the rank of the document from retriever i, and k is a constant (usually set to 60) that dampens the influence of high ranks.

Reciprocal Rank Fusion Example — This image visually demonstrates how RRF works. It calculates a final score for each document by summing the reciprocal of its rank from each retriever (BM25 and Vector), providing a score-agnostic way to merge results.

Part 3: Implementing Hybrid Search

Now, let's put theory into practice. Given your background, seeing the underlying code will be most valuable. We'll first look at a from-scratch implementation to understand the mechanics, and then see how to do it with a high-level framework.

Implementation from First Principles

The following resource provides a clear, self-contained Python implementation of a BM25Retriever and a HybridRetriever. This will allow you to see exactly how the scoring and fusion logic works.

Dense vs Sparse Retrieval: Mastering FAISS, BM25, and Hybrid Search

The article 'Dense vs Sparse Retrieval' provides excellent, commented Python classes for both BM25 and Hybrid Search. Studying this code will solidify your understanding of the algorithms themselves.

Please review the Python code in these two sections: BM25: The Industry Standard: Examine the BM25Retriever class. Pay close attention to the _score_document method, where you can see the TF, IDF, and length normalization components of the formula being calculated. Hybrid Search: Best of Both Worlds: Study the HybridRetriever class. Focus on the _reciprocal_rank_fusion method. Trace how it takes the two ranked lists and calculates the final combined RRF score for each document.

By reviewing this code, you're not just using an API; you're seeing the direct implementation of the concepts we've discussed, fulfilling a key part of our learning outcome.

Implementation with a Framework (LlamaIndex)

While building from scratch is insightful, in a production environment you'll likely use a framework like LlamaIndex or LangChain. These frameworks provide pre-built, optimized components for sparse and hybrid retrieval.

LlamaIndex, for example, offers a QueryFusionRetriever that makes implementing hybrid search straightforward.

BM25 Retriever | LlamaIndex Python Documentation

This LlamaIndex documentation shows how to combine a BM25 retriever with a vector store (Chroma) to create a hybrid system.

Skim the section 'Hybrid Retriever with BM25 + Chroma'. You don't need to run the code, but observe the pattern. Notice how a QueryFusionRetriever is initialized with a list of individual retrievers (a dense one from the index and a BM25Retriever). The framework handles the fusion process internally.

This framework-based approach abstracts away the fusion logic we saw in the pure Python code, but the underlying principle is the same. It takes a list of retrievers, runs the query against each, and merges the results.

Conclusion

You have now moved beyond relying on a single retrieval method. By understanding and implementing both sparse and hybrid search, you've added a critical layer of robustness to your RAG systems.

Key Takeaways:

Dense and Sparse are Complementary: Dense retrieval captures semantic meaning, while sparse retrieval excels at matching keywords, IDs, and jargon.
BM25 is the Keyword Standard: It improves on TF-IDF by adding term frequency saturation and document length normalization, making it a powerful baseline for sparse retrieval.
Hybrid Search Combines Strengths: By running both retrievers and fusing the results, hybrid search provides significantly better recall and precision than either method alone.
Reciprocal Rank Fusion (RRF) is a Robust Fusion Strategy: It merges ranked lists without needing to worry about incompatible score scales, making it a go-to choice for hybrid systems.

What's Next?

We've now explored several retrieval strategies: pure dense, pure sparse, and hybrid. This raises an important question: how do we know which one is best for our specific use case? To answer that, we need a systematic way to measure their performance.

In the next lesson, we will dive into evaluation. You will learn how to evaluate a RAG system using retrieval metrics (e.g., hit rate, Mean Reciprocal Rank) and end-to-end generation quality metrics. This will equip you to make data-driven decisions about your RAG architecture.

Recommended resources