Edvaldo Melo

Traditional RAG, which I covered in my previous post, uses dense vector search to retrieve documents that are semantically similar to the user’s query. This approach is especially useful for open-ended or natural language questions, where users describe their intent without necessarily knowing or using the exact terms found in the documents. However, relying solely on vector search may not be sufficient in systems that require exact term matching, especially in technical, legal, or medical domains, where specific identifiers such as product codes, document numbers, or rare acronyms can be essential for finding the correct information.

Hybrid search is particularly useful in these types of cases because it combines the best of both worlds: dense vector search and keyword-based (sparse) search, which is highly precise at matching exact terms and uses statistics such as term frequency and rarity to assign greater relevance to documents containing rare and specific terms.

The true power of hybrid search lies in its ability to combine the results of both search approaches through rank fusion techniques, such as Reciprocal Rank Fusion (RRF). After both searches are performed, the resulting rankings can be combined to produce a final list that captures both the user’s semantic intent and lexical precision. I will cover rank fusion and re-ranking algorithms in a future post.

In this post, I will cover:

• Understanding the BM25 Algorithm
• Dataset Used in This Post
• Environment Setup
• Building the Hybrid Search System
• How to Run the System
• Results Analysis

Understanding the BM25 Algorithm

Best Matching 25 (BM25) is one of the most widely used keyword search algorithms. To better understand how it works, it is first necessary to understand the TF-IDF algorithm.

First, TF, which stands for Term Frequency, measures the frequency of a term within a document, that is, how many times a word occurs in that document. One way to calculate TF is to normalize this count by the total number of terms in the document:

where f(t,d) represents the number of occurrences of term t in document d, and |d| represents the total number of terms in the document.

IDF (Inverse Document Frequency) identifies how common a term is across documents, with the goal of reducing the weight of terms that occur very frequently in the collection, such as stop words (“of”, “the”, “as”, etc.), while increasing the weight of less frequent terms:

where D represents the document collection and N is the total number of documents in D. In the formula, the denominator represents the number of documents containing the term t, while the logarithm compresses this ratio, preventing extremely rare terms from receiving disproportionately high weights. Therefore, the more frequent a term is across the collection, the larger the denominator and the lower its IDF. Conversely, the rarer a term is, the smaller the denominator and the higher its IDF.

The main intuition behind TF-IDF is that a term tends to be more relevant to a document when it appears frequently in that document but is relatively uncommon across the collection as a whole. Therefore, the higher the TF-IDF score of a term in a document, the more relevant that term tends to be to the document:

One of the problems with TF-IDF is the lack of keyword saturation, since linear TF assigns the same increase in importance to each additional occurrence of a term, regardless of how many times it has already appeared in the document. For example, imagine a collection of 4 documents, each containing 1,000 terms. A given term appears 4 times in document A, 7 times in document B, 60 times in document C, and 63 times in document D. Since TF is calculated directly from the term frequency and then multiplied by IDF, each additional occurrence increases the score linearly. In this example, because all documents have the same length and we are analyzing the same term, both the TF denominator and the IDF remain constant. Therefore, adding 3 occurrences produces exactly the same increase in the score, whether from 4 → 7 or from 60 → 63. In the first case, increasing from 4 to 7 occurrences may represent a significant increase in the document’s relevance to the user’s search. In the second case, however, increasing from 60 to 63 occurrences should not make the document significantly more relevant to the search, since the documents are already saturated with the term.

The BM25 algorithm addresses this problem by modifying how term frequency contributes to the score, introducing a tunable parameter k1, which controls the saturation rate:

The smaller the value of k1, the faster saturation occurs and, consequently, the smaller the impact of additional occurrences of the term. The larger the value of k1, the more slowly this saturation occurs. Values between 1.2 and 2.0 are frequently used. However, since k1 is a tunable parameter, the best value depends on the dataset. The k1+1 term in the numerator acts as a normalization factor: when the term appears only once, the value of the expression is exactly 1. As the term frequency increases, each additional occurrence produces a progressively smaller gain, causing the score to approach k1+1.

Another improvement introduced by BM25 is taking document length into account. The intuition is that the same number of occurrences of a term can have different meanings depending on the length of the document. For example, a term appearing 5 times in a short document may be stronger evidence of relevance than the same term appearing 5 times in a much longer document. However, not every problem should assign the same importance to document length. For this reason, the tunable parameter b is introduced to control the impact of this normalization on the score:

where avgdl represents the average document length in the collection. The parameter b ranges from 0 to 1: when b=0, document length is completely ignored; when b=1, document length normalization is fully applied. Like k1, b is a tunable parameter, and the best value depends on the dataset and can be determined experimentally

Regarding IDF, BM25 also introduces changes to how the rarity of a term is calculated. The main idea remains the same: assigning greater weight to terms that appear in fewer documents and lower weight to those present in a large portion of the collection:

where df(t) represents the number of documents containing the term t. The 0.5 values act as smoothing factors, helping to avoid extreme behavior. The +1 inside the logarithm ensures that, in this IDF variant, its value remains positive. Without it, terms appearing in more than half of the documents could produce negative values, which may be undesirable in retrieval systems that use the score to rank the most relevant documents for the user’s query.

Thus, the complete BM25 formula can be represented as:

where q represents the user’s query and d the document being evaluated.

It is worth noting that BM25 has numerous variants. Therefore, the formula presented above represents only one of the possible variants of the algorithm.

Dataset Used in This Post

For this post, an e-commerce dataset from The Home Depot, a retail company specializing in construction, home improvement, maintenance, and home-related products, was used. The dataset is available on Kaggle and contains 2,551 records and 13 columns. However, for the purpose of this post, only the “title”, “description”, “brand”, and “price” columns were used.

The goal of using this dataset is to build an assistant capable of answering questions and providing product recommendations based on the context retrieved through Hybrid Search.

Environment Setup

The project file structure is organized as follows:

.
├── dataset
│   └── home_depot_data_1_2021_12.csv
├── src
│   ├── services
│   │   ├── embeddings.py
│   │   ├── llm.py
│   │   └── vector_db.py
│   ├── ingestion.py
│   └── main.py
├── .env
├── docker-compose.yml
└── requirements.txt

Since the project uses OpenAI models, you need to provide your OpenAI API key in the .env file:

OPENAI_API_KEY=your_openai_api_key

Additionally, the requirements.txt file available below contains the dependencies required to run the project:

Building the Hybrid Search System

The goal of this section is to explain how the RAG system with hybrid search was implemented. To begin, the files in the services directory will be presented. Below is the embeddings.py file:

from sentence_transformers import SentenceTransformer
class EmbeddingsService:
    def __init__(self):
        self.model = SentenceTransformer("all-MiniLM-L6-v2")
    def get_embedding(self, text: str) -> list[float]:
        return self.model.encode(text).tolist()

The EmbeddingsService class loads the all-MiniLM-L6-v2 embedding model in its constructor and uses the get_embedding method to convert the text received as a parameter into an embedding vector.

Next, we have the vector database service (vector_db.py), which is responsible for creating the collection, inserting the data, and performing the hybrid search:

import weaviate
import weaviate.classes.config as wvc
from src.services.embeddings import EmbeddingsService
class VectorService:
    def __init__(self):
        self.client = weaviate.connect_to_local()
        self.embeddings_service = EmbeddingsService()
    def create_collection(self, collection_name: str):
        print(f"Creating collection '{collection_name}'...")
        if self.client.collections.exists(collection_name):
            self.client.collections.delete(collection_name)
        
        self.client.collections.create(
            name=collection_name,
            properties=[
                wvc.Property(name="title", data_type=wvc.DataType.TEXT),
                wvc.Property(name="description", data_type=wvc.DataType.TEXT),
                wvc.Property(name="brand", data_type=wvc.DataType.TEXT),
                wvc.Property(name="price", data_type=wvc.DataType.NUMBER),
            ],
            vector_config=wvc.Configure.Vectors.self_provided()
        )
        print("✅ Collection created successfully.")
    
    def insert_data(self, collection_name: str, data: list[dict]):
        print(f"🚀 Starting data insertion into collection '{collection_name}'...")
        collection = self.client.collections.get(collection_name)
        with collection.batch.dynamic() as batch:
            for item in data:
                text = f"{item['title']}: {item['description']}"
                vector = self.embeddings_service.get_embedding(text)
                batch.add_object(
                    properties={
                        'title': item['title'],
                        'description': item['description'],
                        'brand': item['brand'],
                        'price': item['price'],
                    },
                    vector=vector
                )
        print(f"✅ Insertion of {len(data)} records completed successfully.")
    
    def hybrid_search(
        self, query: str, collection_name: str, top_k: int, alpha: float = 0.5
    ) -> list:
        collection = self.client.collections.get(collection_name)
        query_vector = self.embeddings_service.get_embedding(query)
        response = collection.query.hybrid(
            query=query,
            vector=query_vector,
            alpha=alpha,
            limit=top_k
        )
        return response.objects
    def close_connection(self):
        self.client.close()

First, it is important to highlight that the vector database used in this project is Weaviate. The VectorService class was created to centralize the operations related to the vector database. Its constructor establishes a local connection to Weaviate through the connect_to_local() method and initializes the embedding service using the EmbeddingsService class created in embeddings.py. In addition, VectorService contains the following methods:

• create_collection: responsible for creating a collection in Weaviate using the name received through the collection_name parameter. If the collection already exists, it is deleted, allowing it to be created again through the create() method, where the collection name, properties, and vector configuration are defined. The properties define the fields stored in each object of the collection, which in this case are title, description, brand, and price. The vector_config=wvc.Configure.Vectors.self_provided() configuration specifies that the vectors will be provided by the application itself rather than automatically generated by Weaviate.
• insert_data: responsible for inserting data into the collection based on the collection name and data, a list of dictionaries containing the dataset used in the project. First, the collection is retrieved through the get() method. The objects are then inserted using batch.dynamic(), which performs batch imports and dynamically adjusts how many objects are sent at a time based on import performance. For each object, title and description are concatenated and converted into a vector through the get_embedding() method of EmbeddingsService. Finally, add_object() stores the product properties along with the generated vector.
• hybrid_search: responsible for retrieving the most relevant objects for the user’s query through hybrid search. First, the collection is retrieved and the query is converted into a vector through the get_embedding() method. Then, query.hybrid() receives both the original text query, used for keyword search, and query_vector, used for vector search. The limit parameter defines the maximum number of objects returned, while alpha, which ranges from 0 to 1, controls the weight between the two search approaches: alpha=1 uses only vector search, alpha=0 uses only keyword search, and alpha=0.5 assigns equal weight to both components.
• close_connection: closes the connection to Weaviate through the close() method.

It is worth noting that, under the hood, Weaviate uses the BM25 algorithm to perform keyword search. By default, the k1 and b parameters are set to 1.2 and 0.75, respectively (click here).

Finally, the last file in the services directory is llm.py, which is responsible for using an LLM to generate contextualized responses based on the retrieved documents.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from dotenv import load_dotenv
load_dotenv()
class LLMService:
    def __init__(self):
        self.llm = ChatOpenAI(
            model="gpt-4.1-mini",
            temperature=0,
        )
    
    def generate_response(self, query: str, objects: list) -> str:
        context = "\n\n".join(
            f"Title: {obj.properties['title']}\n"
            f"Description: {obj.properties['description']}\n"
            f"Brand: {obj.properties['brand']}\n"
            f"Price: {obj.properties['price']}"
            for obj in objects
        )
        
        system_prompt = (
            "You are an AI assistant for product search.\n"
            "Help users find and understand products based on their needs and preferences.\n"
            "Answer the user's question using only the product information below.\n"
            "Include relevant product details when they help answer the question.\n"
            "Do not make assumptions, infer unsupported details, or invent information.\n"
            "If you cannot answer the question, respond naturally and briefly that you do not know or "
            "do not have that information, without explaining why or referring to the context, retrieved "
            "results, sources, documents, dataset, or internal system behavior.\n"
            "For example, say 'I’m sorry, but I don’t have that information.' instead of saying "
            "'The provided product information does not include that information.'\n"
            "If the question is unrelated to products, politely explain that you can only answer "
            "questions about products.\n\n"
            "Product information:\n"
            "{context}"
        )
        user_prompt = "Question: {query}"
        messages = ChatPromptTemplate.from_messages([
            ("system", system_prompt),
            ("human", user_prompt),
        ]).format_messages(
            context=context,
            query=query,
        )
        response = self.llm.invoke(input=messages)
        return response.content

In the constructor of the LLMService class, the model is instantiated through LangChain’s ChatOpenAI class. The model used is gpt-4.1-mini with a temperature of 0, aiming to reduce randomness and generate more consistent responses. The generate_response method receives the user’s query and the objects returned by the hybrid_search method of VectorService. First, the context that will be sent to the LLM is created, containing the title, description, brand, and price of each retrieved object. Then, the system_prompt is defined with the instructions for the model and the retrieved context, while the user_prompt contains the user’s query. The messages variable is then created from both prompts and sent to the LLM through the invoke() method, which generates the response whose content is returned by the method.

Next, the code for the ingestion.py file is shown below, which is responsible for creating the collection and inserting the data into Weaviate:

from pandas import read_csv
from src.services.vector_db import VectorService
if __name__ == "__main__":
    vector_service = VectorService()
    collection_name = "HomeDepot"
    try:
        df = read_csv(
            "dataset/home_depot_data_1_2021_12.csv", 
            usecols=["title", "description", "brand", "price"]
        )
        data = df.to_dict(orient="records")
        vector_service.create_collection(collection_name)
        vector_service.insert_data(collection_name, data)
    finally:
        vector_service.close_connection()

First, the vector_service object, used to access the vector database service methods, and collection_name, set to HomeDepot, are defined. Next, the dataset is loaded using Pandas, considering only the four columns used in the project: title, description, brand, and price. The DataFrame is then converted into a list of dictionaries and stored in the data variable. Finally, the collection is created through the create_collection method, the data is inserted using the insert_data method, and the database connection is closed through the close_connection method.

Additionally, the main.py file is shown below. It is responsible for the main execution of the application, allowing the assistant to answer the user’s questions.

from src.services.vector_db import VectorService
from src.services.llm import LLMService
if __name__ == "__main__":
    vector_service = VectorService()
    llm_service = LLMService()
    collection_name = "HomeDepot"
    print("🛍️ Product Search Assistant")
    print("⚠️ Type 'exit' to close the application.\n\n")
    try:
        while True:
            question = input("Question: ").strip()
            if question.lower() == "exit":
                break
            if not question:
                continue
            objects = vector_service.hybrid_search(
                question,
                collection_name,
                top_k=5,
                alpha=0.5
            )
            response = llm_service.generate_response(question, objects)
            print(f"💬 Response: {response}\n")
    finally:
        vector_service.close_connection()

First, the vector_service and llm_service objects are created, along with the name of the collection being used. Next, a loop is executed to simulate user interactions until the user types exit. For each question, the objects most relevant to the user’s question are retrieved through the hybrid_search method. Note that alpha=0.5, assigning equal weight to the vector search and keyword search components during hybrid search. The retrieved objects and the user’s question are then passed to the generate_response method, which generates the assistant’s response and presents it to the user. Finally, when the loop ends, the database connection is closed.

Finally, the docker-compose.yml file is shown below, which is responsible for running the Weaviate database locally using Docker.

services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.34.7
    ports:
    - 8080:8080
    - 50051:50051
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'none'
      ENABLE_MODULES: ''

Ports 8080 and 50051 are used for communication with Weaviate. Port 8080 exposes the HTTP interface, allowing the database to be accessed through HTTP requests, while port 50051 exposes the gRPC (Google Remote Procedure Call) interface, which is used by the Weaviate client to perform certain communication operations with the database more efficiently. Regarding the environment variables defined, we have:

• QUERY_DEFAULTS_LIMIT: defines the default maximum number of objects returned by a query when no limit is specified. In this case, the value is set to 25.
• AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: when set to true, allows access to Weaviate without authentication, meaning that no credentials are required.
• PERSISTENCE_DATA_PATH: defines the path inside the container where Weaviate stores its persistent data. In this case, /var/lib/weaviate.
• DEFAULT_VECTORIZER_MODULE: when set to none, indicates that Weaviate will not use a default module to generate embeddings. In this project, the vectors are generated by the Python application and sent to Weaviate.
• ENABLE_MODULES: defines which additional Weaviate modules will be enabled. Since it is empty, no additional modules are enabled.

How to Run the System

The goal of this section is to explain, step by step, how to run the system. First, to start the Weaviate database, open a terminal, navigate to the project directory where the docker-compose.yml file is located, and run:

docker compose up -d

Next, it is necessary to create the HomeDepot collection and insert the data. To do this, simply run the ingestion.py file, which inserts the 2,551 records, as shown below:

Creating collection 'HomeDepot'...
✅ Collection created successfully.
🚀 Starting data insertion into collection 'HomeDepot'...
✅ Insertion of 2551 records completed successfully.

Finally, we can run the main.py file, which is responsible for starting the interaction with the assistant. Below is a simple example using only Hello and then exit to close the application and demonstrate the complete execution of the loop:

🛍️ Product Search Assistant
⚠️ Type 'exit' to close the application.
Question: Hello
💬 Response: Hello! How can I assist you with products today?
Question: exit

Results Analysis

The goal of this section is to analyze the effectiveness of hybrid search. First, a simple test will be presented. Next, two tests will be presented where pure vector search fails but hybrid search succeeds. Finally, a case where hybrid search also fails will be presented.

The first test uses the question: “Do you have a DEWALT cutting wheel for stainless steel?”. This is a simple query that combines semantic and lexical characteristics, as it describes the type and intended use of the product being searched for while also containing specific terms such as DEWALT and stainless steel. The assistant’s response is shown below:

Question: Do you have a DEWALT cutting wheel for stainless steel?
💬 Response: Yes, there is a DEWALT cutting wheel suitable for stainless
steel. It is the 7/8 in. x 4-1/2 in. x 0.045 in. Metal and Stainless
Cutting Wheel (50-Pack). These DEWALT High Performance cutting wheels
have a thin 0.045 in. cutting edge for fast, burr-free cutting and are
designed for high performance cutting in all types of ferrous metals and
stainless steel. The pack of 50 is priced at $44.97.

Now, the goal is to demonstrate a scenario where purely semantic search may fail, while hybrid search is able to retrieve the necessary context for the assistant to answer correctly. To perform this comparison, the alpha parameter in the main.py file will initially be changed to 1, causing only vector search to be used. Then, the same question will be executed with alpha=0.5, combining vector search with keyword search. The results for both configurations are shown below using the question “What are the main features of the paint with code HDGV43U?”.

To reinforce the scenario presented previously, a second test is performed using another specific identifier, this time with the question “I have a 290-409 part. What is it and what equipment is it compatible with?”.

Finally, the last test demonstrates that hybrid search also has limitations and may fail to retrieve the necessary context for the assistant to answer correctly. For this test, the following question was used: “I have an A32RF01N replacement filter. What product is it compatible with?”.

Question: I have an A32RF01N replacement filter. What product is it
compatible with?
💬 Response: I’m sorry, but I don’t have information about the
compatibility of the A32RF01N replacement filter.

Therefore, we can conclude that hybrid search can be effective in scenarios where pure vector search fails, especially when the query contains specific terms that benefit from lexical matching. However, hybrid search also has limitations and does not guarantee that the necessary context will always be retrieved. For this use case, we could evaluate different system configurations, such as adjusting the alpha parameter, tuning the BM25 parameters k1 and b, increasing the number of retrieved documents (top_k), or using a different embedding model.