Hybrid Search: Dense Vectors & Keywords

In our last lesson, you built a functional vector search system that retrieves information based on semantic meaning. This is a huge step, but as you may have noticed, it's not perfect. Purely semantic systems can sometimes struggle with queries that contain specific keywords, product codes, or names, as the meaning of the surrounding text might dilute their importance.

Today, we will address this limitation head-on. Our goal is to implement hybrid search by combining dense vector retrieval with keyword-based search (e.g., BM25). This approach creates a more robust and reliable retrieval system by leveraging the strengths of both semantic understanding and lexical precision. For an AI Engineer, mastering hybrid search is a key skill for building production-grade RAG systems that deliver consistently relevant results across a wide range of user queries.

Why Not Just One Search Method?

Vector search is powerful for understanding the intent behind a query, but it can be a blunt instrument. Keyword search is precise but lacks contextual understanding. A truly effective search system needs both.

Consider this example:

• Query: "debugging python memory leaks"
• Vector Search might find: "How to track down RAM problems in Python code" (Good semantic match)
• Keyword Search might find: "A guide to SKU-PY-MEM-LEAK-01" (Poor match, but has keywords)

Now consider this query:

• Query: "fix for HTTP 502 error"
• Vector Search might find: "Troubleshooting common web server issues" (Too broad)
• Keyword Search will find: "Guide to resolving HTTP 502 error in Nginx" (Perfect match)

Hybrid search runs both types of searches in parallel and intelligently merges the results, ensuring you get the conceptual relevance of vector search and the precision of keyword search.

The overall architecture involves running two search processes in parallel and then fusing their results, as shown in the diagram below.

Component 1: Keyword Search with BM25

While your vector database handles semantic search, you need a separate mechanism for keyword search. The industry standard algorithm for this is BM25 (Best Matching 25). It's a sophisticated version of TF-IDF (Term Frequency-Inverse Document Frequency) and is the engine behind many popular search systems like Elasticsearch.

Conceptually, BM25 scores documents based on:

• The terms from your query that appear in the document.
• How frequently those terms appear (term frequency).
• How rare those terms are across all documents (inverse document frequency).
• The length of the document.

You don't need to implement the BM25 algorithm from scratch. We can use a well-established Python library called rank-bm25.

To use it, we first need to install the library:
pip install rank_bm25

Next, we'll implement the keyword search component. This involves taking our document chunks, tokenizing them (splitting them into words), and building a BM25 index.

Component 2: Fusing the Results with Reciprocal Rank Fusion (RRF)

Now for the most critical part: combining the two sets of results. The vector search gives you a list of documents ranked by semantic similarity, and the BM25 search gives you a different list ranked by keyword relevance. Their scores are on completely different scales, so you can't just add them up.

The most common and robust method for this is Reciprocal Rank Fusion (RRF). RRF is elegant because it disregards the actual scores and only considers the rank of each document in the different lists.

The formula for each document's RRF score is:
$\text{RRF Score}={\sum }_{i\in \text{lists}}\frac{1}{k+{\text{rank}}_i}$

Where:

• rank_i is the rank of the document in list i (e.g., 1st, 2nd, 3rd).
• k is a constant (a common value is 60) that dampens the influence of lower-ranked items.

Let's see how this works in practice.

The beauty of RRF is that documents that appear in both result sets get a natural boost, and it's completely agnostic to the underlying scoring mechanisms. Now, let's look at a Python implementation.

Hands-On: Implementing Hybrid Search

Let's integrate these components into your RAG system. The following script builds upon the code from our previous lesson. It will:

1. Load the existing ChromaDB collection.
2. Create a BM25 index from the same documents.
3. Perform both vector and keyword searches for a given query.
4. Use the RRF function to fuse the results into a single, superior ranked list.

This script assumes you have the my_vector_db directory from the previous lesson and have installed rank-bm25.

import chromadb
import os
from openai import OpenAI
import pprint
import re
from collections import defaultdict
from rank_bm25 import BM25Okapi




# --- Setup from previous lessons ---
client = OpenAI() # Assumes OPENAI_API_KEY is set in your environment
chroma_client = chromadb.PersistentClient(path="./my_vector_db")
collection_name = "rag_system_docs"
collection = chroma_client.get_collection(name=collection_name)

def get_openai_embeddings(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    if not texts: return []
    response = client.embeddings.create(model=model, input=texts)
    sorted_data = sorted(response.data, key=lambda x: x.index)
    return [item.embedding for item in sorted_data]




# --- RRF Implementation (from resource d0a91) ---
def reciprocal_rank_fusion(ranked_lists: list[list[dict]], k: int = 60, id_field: str = "id") -> list[dict]:
    rrf_scores: defaultdict[any, float] = defaultdict(float)
    doc_data: dict[any, dict] = {}

    for ranked_list in ranked_lists:
        for rank, doc in enumerate(ranked_list, start=1):
            doc_id = doc[id_field]
            rrf_scores[doc_id] += 1.0 / (k + rank)
            if doc_id not in doc_data:
                doc_data[doc_id] = doc
    
    sorted_ids = sorted(rrf_scores.keys(), key=lambda x: rrf_scores[x], reverse=True)
    
    results = []
    for doc_id in sorted_ids:
        doc = doc_data[doc_id].copy()
        doc["rrf_score"] = rrf_scores[doc_id]
        results.append(doc)
    return results




# --- BM25 Keyword Search Implementation ---
def preprocess_text_for_bm25(text: str) -> list[str]:
    text = re.sub(r'[^a-zA-Z0-9\s]', ' ', text.lower())
    tokens = [token for token in text.split() if token.strip()]
    return tokens




# 1. Get all documents from the collection to build the BM25 index
all_docs = collection.get() # This retrieves all documents, embeddings, metadatas, etc.
doc_texts = all_docs['documents']
doc_ids = all_docs['ids']




# 2. Preprocess and tokenize the documents for BM25
tokenized_corpus = [preprocess_text_for_bm25(doc) for doc in doc_texts]
bm25 = BM25Okapi(tokenized_corpus)




# --- Hybrid Search Function ---
def hybrid_search(query: str, n_results: int = 4):



    # a. Vector Search
    print("--- Performing Vector Search ---")
    query_embedding = get_openai_embeddings([query])[0]
    vector_results_raw = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results * 2 # Fetch more to give RRF more to work with
    )



    # Format for RRF
    vector_results = [
        {'id': vector_results_raw['ids'][0][i], 'content': vector_results_raw['documents'][0][i]}
        for i in range(len(vector_results_raw['ids'][0]))
    ]
    print("Vector search found:", [res['id'] for res in vector_results])





    # b. Keyword Search
    print("\n--- Performing Keyword Search ---")
    query_tokens = preprocess_text_for_bm25(query)
    bm25_scores = bm25.get_scores(query_tokens)
    



    # Get top N results for BM25
    top_n_indices = sorted(range(len(bm25_scores)), key=lambda i: bm25_scores[i], reverse=True)[:n_results * 2]
    



    # Format for RRF
    keyword_results = [
        {'id': doc_ids[i], 'content': doc_texts[i]}
        for i in top_n_indices if bm25_scores[i] > 0
    ]
    print("Keyword search found:", [res['id'] for res in keyword_results])
    



    # c. Fusion with RRF
    print("\n--- Fusing results with RRF ---")
    fused_results = reciprocal_rank_fusion([vector_results, keyword_results])
    
    return fused_results[:n_results]




# --- Run the search ---
user_query = "What is a fast algorithm for vector search?"
final_results = hybrid_search(user_query)

print("\n\n--- Final Hybrid Search Results ---")
pprint.pprint(final_results)

When you run this code, observe the outputs from the vector search and keyword search steps. You'll likely see some overlap but also some unique documents from each. The final RRF output will be a re-ranked list that intelligently combines these two sets of candidates. A document like "Vector databases use indexing for fast retrieval," which contains both strong semantic and keyword matches, should rank very highly.

Conclusion

You have now built a significantly more powerful and resilient retrieval system. By combining semantic and keyword search, you've created a hybrid engine that can handle a much wider variety of queries, from conceptual questions to searches for specific terms. This is a standard pattern in modern, production-level RAG applications.

Key Takeaways:

• Hybrid search mitigates the weaknesses of using either vector or keyword search alone.
• BM25 is the standard algorithm for keyword search, focusing on term frequency and rarity.
• Reciprocal Rank Fusion (RRF) is a score-agnostic method to intelligently merge multiple ranked lists based only on the rank of items.
• The implementation involves running both searches in parallel and then applying a fusion algorithm to the results to produce a final, re-ranked list.

Our retrieval system is now quite sophisticated. However, the quality of retrieval is still highly dependent on the quality of the user's initial query. In our next lesson, we will enter the "Advanced RAG" module and tackle this problem by transforming user queries into optimized search queries using an LLM, further improving the relevance of our retrieved context.