Edvaldo Melo

In real-world applications, semantic similarity search alone is not always sufficient to meet information retrieval requirements. Beyond semantic relevance, results often need to satisfy specific constraints, such as belonging to a particular category or excluding certain values. When performing similarity search across the entire dataset, the retrieved results may be semantically relevant to the query but may not necessarily satisfy user-defined filters or the application’s business rules.

Modern vector databases allow metadata to be associated with stored vectors, including descriptive attributes such as brand, price, and category. This metadata can be used as filtering criteria to restrict the candidate set based on specific conditions. In pre-filtering, these constraints are applied before semantic similarity search, ensuring that only vectors satisfying the specified filters are considered during retrieval.

An important aspect of pre-filtering is indexing, the process of building auxiliary data structures, called indexes, that accelerate the identification of records satisfying specific filters. Consider, for example, a filter requiring the category to be equal to “news”. Without an index on this field, a conventional approach would be to scan the metadata of all records stored in the vector database, individually checking which ones satisfy the condition. As the collection grows to millions of records, this linear scan can significantly increase computational cost and query latency. With indexing, on the other hand, it becomes possible to identify records associated with specific values more efficiently, exclude those that do not satisfy the filters, and combine conditions through efficient operations, reducing computational cost and, consequently, filtering latency.

There are different data indexing structures, such as inverted indexes and bitmap indexes. In this post, we’ll use pinecone, a vector database that employs roaring bitmaps for metadata indexing (read more about its architecture here). In addition to exploring how roaring bitmaps work, we’ll build a book recommendation system using RAG with pre-filtering, where pinecone will be responsible for storing book embeddings and metadata, as well as retrieving the context provided to the AI assistant.

Therefore, this post will cover:

• Understanding Roaring Bitmaps
• Dataset Overview
• Environment Setup
• Building the Book Recommendation System
• How to Run the System
• Results Analysis

Understanding Roaring Bitmaps

Although traditional structures, such as conventional bitmaps, are efficient for representing sets of integers, they can consume significant storage space when values are sparsely distributed across a large range. In this context, roaring bitmap emerged as a compressed data structure designed to represent sets of integers in a space-efficient manner while supporting fast operations on these sets.

Two of the main data structures used by roaring bitmaps are arrays and conventional bitmaps, organized into structures called containers. To understand their differences, consider the set of integers $\{9, 2, 15, 5\}$. In an array container, values are stored directly in an array sorted in ascending order, regardless of insertion order:

In a bitmap container, each value corresponds to a bit position, which is set to 1 to indicate its presence in the set:

Now, consider storing the integers $\{5, 100, 70000, 1000000\}$. In a conventional bitmap, 1,000,001 bits would be required to represent just four integers, equivalent to approximately 125 KB of storage. But how does roaring ritmap handle this scenario? To understand how it works, we’ll use the 32-bit version of Roaring Bitmap as an example.

The first step is to split each 32-bit integer into two 16-bit parts. The 16 most significant bits identify the container to which the integer belongs, while the 16 least significant bits represent its value within that container. These two parts are referred to as high and low, respectively:

Thus, given an integer $x$, bitwise operations are used to identify its container and the corresponding value within it. To obtain the high, a 16-bit right shift is applied, equivalent to integer division by $2^{16}$. To obtain the low, a bitwise AND operation is applied using the $0xFFFF$ mask, which preserves the 16 least significant bits and produces the same result as the remainder of division by $2^{16}$. These operations can be represented by the following expressions:

Since $2^{16} = 65,536$, consider the integer $x = 1,000,000$. Performing integer division by 65,536 gives 15, indicating that this integer belongs to the container identified by $high=15$. The remainder of this division is 16,960, corresponding to $low = 16,960$ within that container.

To better understand how containers are organized, consider that each one represents a range of 65,536 possible values. For example, when $high(x) = 0$, we have $x \in [0, 65535]$; when $high(x) = 1$, we have $x \in [65536, 131071]$, and so on. Although each container corresponds to a different range, low values always range from 0 to 65,535, as they represent the relative position of each integer within the range associated with its container. Thus, the original integer can be reconstructed by combining its high and low values. The table below shows some integers and their corresponding high and low values:

But how do we choose between an array container and a bitmap container? Each container represents a range of 65,536 possible values. In an array container, only the values present in the set are stored. In a bitmap container, all 65,536 positions are represented by bits, indicating the presence or absence of each value. Thus, the storage consumption of an array container depends on the number of values stored, while a bitmap container has a fixed size of 8 KiB. Since each element in an array container occupies 2 bytes, we can express the storage consumption of both containers as follows:

where $C$ represents the number of distinct integers present in the container. By equating the two storage requirements, we can determine the point at which both representations occupy the same amount of space:

In other words, for up to 4,096 elements, an array container consumes less than or the same amount of space as a bitmap container. When the number of elements exceeds 4,096, the Bitmap Container becomes more space-efficient.

But how does searching work? To better understand this, consider the contains(x) operation, which checks whether a given integer is present in the set. Suppose $x = 70000$. The first step is to calculate its high and low values, which in this case are 1 and 4,464, respectively. Next, we check whether the container identified by $high(x) = 1$ exists. If it does not, the integer is not present in the set. Otherwise, the search continues within that container. Assuming it is an array container, since its values are maintained in ascending order, we can use binary search, with a time complexity of $O(\log C),$ to check whether $low(x) = 4464$ is present, where $C$ represents the number of elements stored in the container.

But what if it is a bitmap container? Modern processors can efficiently perform bitwise operations on 64-bit words. Therefore, a bitmap container is organized into 1,024 64-bit words, totaling 65,536 bits. To identify which word and bit to check, we use the following expressions:

In this example, $low(x) = 4464$, resulting in $word(x) = 69$ and $bit(x) = 48$. This means we need to check bit index 48 in word index 69, assuming both indices start at zero. If this bit is set to 1, the integer is present in the set; otherwise, it is absent. Since this operation only involves direct access to the word and checking a single bit, its time complexity is constant, $O(1)$.

Beyond searching for a specific element, roaring bitmap also supports set operations such as AND, OR, and XOR, involving different container combinations, including Array × Array, Array × Bitmap, and Bitmap × Bitmap. Although these operations will not be covered in this post, I encourage you to explore how they can be performed based on the structure’s behavior explained so far.

To consolidate our understanding of roaring bitmap, consider the example below. On the left, we have a database table containing one million records and the columns country, plan, and age. On the right, a representation of how these data can be indexed using roaring bitmaps. For the country and plan columns, each distinct value has its own index, while for age, two ranges have been defined: [18–29] and [30–49]. Each index represents the set of record IDs that satisfy its corresponding condition. Internally, these IDs are split into high and low, which identify the corresponding container and the value within it. The ellipses indicate omitted records and portions of the indexes to simplify the visualization.

Finally, it is worth noting that relational and vector databases using roaring bitmaps may adopt different implementations and optimizations. Nevertheless, the fundamental principles of the data structure discussed throughout this section remain applicable.

Dataset Overview

The goal of this post is to build a RAG-based book recommendation system. To do so, we’ll use a dataset containing approximately 10,000 books across different genres, with information collected from Goodreads.

The dataset includes attributes such as title (Book), author, description, genres, average rating, number of ratings, and URL. The dataset is available on Kaggle.

Environment Setup

The project’s file structure is organized as follows:

.
├── dataset/
│   └── goodreads_data.csv
├── src/
│   ├── services/
│   │   ├── embeddings.py
│   │   ├── llm.py
│   │   └── vector_db.py
│   ├── ingestion.py
│   └── main.py
├── .env
└── requirements.txt

Since the project uses OpenAI models and the pinecone vector database, the corresponding API keys must be configured in the .env file.

OPENAI_API_KEY=your_openai_api_key
PINECONE_API_KEY=your_pinecone_api_key

Finally, the requirements.txt file contains the dependencies required to run the project:

Building the Book Recommendation System

In this section, we’ll walk through the implementation of the RAG-based book recommendation system. We’ll start with the embeddings.py file, which is responsible for generating the embeddings used in the project:

from openai import OpenAI
class EmbeddingsService:
    def __init__(self):
        self.client = OpenAI()
        self.model = "text-embedding-3-small"
        self.dimension = 1536
    def get_embeddings(self, texts: list[str]) -> list[list[float]]:
        response = self.client.embeddings.create(
            model=self.model,
            input=texts
        )
        return [item.embedding for item in response.data]

The EmbeddingsService class uses OpenAI’s text-embedding-3-small model, which generates 1,536-dimensional vectors by default. The get_embeddings method takes a list of texts and uses the OpenAI API to generate and return their corresponding embeddings.

Next, we’ll look at the vector_db.py file, which handles operations related to the pinecone vector database.

from uuid import uuid4
from pinecone import Pinecone, ServerlessSpec
from src.services.embeddings import EmbeddingsService
class VectorService:
    def __init__(self):
        self.pinecone = Pinecone()
        self.index = None
        self.index_name = None
        self.embeddings_service = EmbeddingsService()
    def create_index(
        self,
        index_name: str,
        namespace_name: str,
        filterable_fields: list[str]
    ) -> None:
        print(f"Creating index '{index_name}'...")
        if self.pinecone.has_index(index_name):
            self.pinecone.delete_index(name=index_name)
        self.pinecone.create_index(
            name=index_name,
            dimension=self.embeddings_service.dimension,
            metric="cosine",
            spec=ServerlessSpec(
                cloud="aws",
                region="us-east-1"
            ),
            vector_type="dense"
        )
        self._get_index(index_name)
        self.index.create_namespace(
            name=namespace_name,
            schema={
                "fields": {
                    field: {"filterable": True}
                    for field in filterable_fields
                }
            }
        )
        print("✅ Index created successfully.")
    
    def insert_data(
        self,
        index_name: str,
        namespace_name: str,
        data: list[dict],
        batch_size: int = 100
    ) -> None:
        print(f"🚀 Starting data insertion into index '{index_name}'...")
        self._get_index(index_name)
        total_batches = (len(data) + batch_size - 1) // batch_size
        for i in range(0, len(data), batch_size):
            batch = data[i: i + batch_size]
            
            texts = [
                f"{item['Book']}: {item['Description']}"
                for item in batch
            ]
            embeddings = self.embeddings_service.get_embeddings(texts)
            vectors = [
                {
                    "id": str(uuid4()),
                    "values": embedding,
                    "metadata": {
                        "book": book["Book"],
                        "author": book["Author"],
                        "description": book["Description"],
                        "genres": book["Genres"],
                        "avg_rating": book["Avg_Rating"],
                        "num_ratings": book["Num_Ratings"]
                    }
                }
                for book, embedding in zip(batch, embeddings)
            ]
            self.index.upsert(vectors=vectors, namespace=namespace_name)
            current_batch = i // batch_size + 1
            print(f"\r📦 Inserting batches: {current_batch}/{total_batches}", end="", flush=True)
        
        print(f"\n✅ Insertion of {len(data)} records completed successfully.")
    
    def search(self, query: str, index_name: str, namespace_name:str, top_k: int, filters: dict):
        filter_conditions = {}
        if filters:
            if "genres" in filters:
                genres = [genre.lower() for genre in filters["genres"]]
                filter_conditions["genres"] = {"$in": genres}
            
            if "avg_rating" in filters:
                min_value, max_value = filters["avg_rating"]
                filter_conditions["avg_rating"] = {}
                if min_value is not None:
                    filter_conditions["avg_rating"]["$gte"] = min_value
                
                if max_value is not None:
                    filter_conditions["avg_rating"]["$lte"] = max_value
            if "num_ratings" in filters:
                min_value, max_value = filters["num_ratings"]
                filter_conditions["num_ratings"] = {}
                if min_value is not None:
                    filter_conditions["num_ratings"]["$gte"] = min_value
                
                if max_value is not None:
                    filter_conditions["num_ratings"]["$lte"] = max_value
            
            if "author" in filters:
                filter_conditions["author"] = {"$eq": filters["author"].lower()}
        self._get_index(index_name)
        query_vector = self.embeddings_service.get_embeddings([query])[0]
        results = self.index.query(
            namespace=namespace_name,
            vector=query_vector,
            top_k=top_k,
            filter=filter_conditions if filter_conditions else None,
            include_metadata=True,
            include_values=False
        )
        return results
    
    def close_connection(self):
        try:
            if self.index is not None:
                self.index.close()
        finally:
            self.index = None
            self.index_name = None
            self.pinecone.close()
    
    def _get_index(self, index_name: str) -> None:
        if self.index_name != index_name:
            if self.index is not None:
                self.index.close()
            self.index = self.pinecone.Index(index_name)
            self.index_name = index_name

The VectorService class constructor initializes the pinecone and embeddings_service objects, along with the variables used to manage access to pinecone indexes, which store embeddings and their associated metadata. The class also includes a private method, _get_index, which uses pinecone.Index to obtain a reference to the specified index.

The remaining methods of the VectorService class are described below:

• create_index: responsible for creating an index in pinecone. The method takes the index name, namespace, and a list of metadata fields to be used for filtering. In pinecone, a namespace represents a logical partition within an index, allowing sets of vectors and their associated metadata to be organized and isolated. Initially, if the index already exists, it is deleted and recreated from scratch. This approach was adopted to simplify the ingestion of approximately 10,000 records from the dataset, which is performed separately from the execution of the RAG system. Next, the index is created using pinecone.create_index, specifying its name, embedding dimensionality, similarity metric, deployment infrastructure through the spec parameter, and vector type. Finally, the create_namespace method creates the namespace and configures the metadata fields to be indexed for filtering. In this project, we’ll use the author, genres, avg_rating, and num_ratings fields. As discussed earlier, pinecone uses roaring bitmaps in its metadata indexing architecture.
• insert_data: responsible for inserting embeddings and their associated metadata into pinecone. The method takes the index name, namespace, data organized as a list of dictionaries, and the batch size used during ingestion. For each book, the text used to generate its embedding is formed by combining the title (Book column) with its corresponding description (Description column). The embeddings are then generated using embeddings_service.get_embeddings. The vectors variable stores a list of dictionaries, each containing a unique UUID identifier, the generated embedding, and the book’s metadata. Finally, the upsert method inserts these records into the specified namespace. During ingestion, pinecone internally manages the indexing of metadata fields configured as filterable, allowing them to be used in subsequent queries.
• search: responsible for performing vector similarity search. The method takes the user’s query, index and namespace names, the maximum number of results (top_k), and metadata filters. First, the filters are constructed using pinecone’s filtering syntax. The $in operator selects books whose genres match at least one of the specified values. The $gte and $lte operators define minimum and maximum thresholds for avg_rating and num_ratings, while $eq checks whether the author field matches the requested author. Next, the user’s query is converted into an embedding. The search is performed using index.query, specifying the namespace, query vector, top_k, and the defined filters. Additionally, include_metadata=True returns the metadata associated with the results, while include_values=False prevents the embedding vectors themselves from being returned.
• close_connection: responsible for releasing resources associated with the pinecone index and client, as well as resetting the internal references maintained by the class.

Next, we’ll look at the llm.py file, which is responsible for generating book recommendations using an LLM.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
class LLMService:
    def __init__(self):
        self.llm = ChatOpenAI(
            model="gpt-5.4-mini",
            temperature=0,
        )
    
    def generate_response(self, query: str, search_results, filters: dict) -> str:
        context = "\n\n".join(
            f"Book: {result.metadata['book']}\n"
            f"Description: {result.metadata['description']}\n"
            f"Author: {result.metadata['author']}\n"
            f"Genres: {result.metadata['genres']}\n"
            f"Avg Rating: {result.metadata['avg_rating']}\n"
            f"Num Ratings: {result.metadata['num_ratings']}"
            for result in search_results.matches
        )
        system_prompt = (
            "You are an AI book recommendation assistant.\n"
            "Your goal is to recommend books that satisfy both the user's reading "
            "preferences and their selected filters.\n\n"
            "Follow these rules:\n"
            "1. Use only the book information provided in the context. "
            "Never recommend books that are not included in the context.\n"
            "2. Recommend only books that satisfy BOTH the user's reading preferences "
            "and ALL selected filters (genre, author, average rating, and number of ratings). "
            "Treat both as equally important requirements. "
            "If a book fails either one, do not recommend it, regardless of how well "
            "it matches the other. If no books qualify, say so.\n"
            "3. For each recommended book, include its title, author, genres, "
            "average rating, and number of ratings.\n"
            "4. Briefly explain why each book matches the user's preferences, "
            "using only its description and metadata. "
            "Do not exaggerate or force connections to the user's query.\n"
            "5. Do not invent plot details, character traits, ratings, genres, "
            "or any other information not supported by the context.\n"
            "6. Be concise, natural, and helpful.\n"
            "7. Format your response using Markdown.\n\n"
            "Book information:\n"
            "{context}"
        )
        user_prompt = (
            "Question: {query}\n\nFilters:\n{filters}"
        )
        messages = ChatPromptTemplate.from_messages([
            ("system", system_prompt),
            ("human", user_prompt),
        ]).format_messages(
            context=context,
            query=query,
            filters=filters
        )
        response = self.llm.invoke(input=messages)
        return response.content

The LLMService class initializes the gpt-5.4-mini model using ChatOpenAI, with the temperature=0 parameter. The generate_response method, responsible for generating recommendations, takes the user’s query, the vector search results from pinecone (search_results), and the selected metadata filters. Based on the retrieved results, a context is constructed containing information about each book, such as title, description, author, genres, average rating, and number of ratings. This context is incorporated into the system_prompt, which also defines the rules the LLM must follow. The user_prompt, in turn, contains the user’s query and selected filters. Both prompts are used by ChatPromptTemplate.from_messages to construct the messages sent to the model. Finally, llm.invoke calls the LLM to generate the recommendations, and the resulting content is returned through response.content.

Next, we’ll look at the ingestion.py file, which handles data ingestion into pinecone, performed separately from the execution of the RAG system.

import ast
from pandas import read_csv
from dotenv import load_dotenv
from src.services.vector_db import VectorService
load_dotenv()
if __name__ == "__main__":
    vector_service = VectorService()
    index_name = "goodreads"
    namespace_name = "books"
    filterable_fields = ["author", "genres", "avg_rating", "num_ratings"]
    try:
        df = read_csv(
            "dataset/goodreads_data.csv",
            converters={"Genres": ast.literal_eval},
            dtype={"Num_Ratings": int},
            thousands=","
        )
        df = df.drop_duplicates(subset=["Book", "Author"])
        df["Description"] = df["Description"].fillna("No description available.")
        df["Author"] = df["Author"].str.lower()
        df["Genres"] = df["Genres"].apply(
            lambda genres: [genre.lower() for genre in genres]
        )
        data = df.to_dict(orient="records")
        
        vector_service.create_index(index_name, namespace_name, filterable_fields)
        vector_service.insert_data(index_name, namespace_name, data)
    finally:
        vector_service.close_connection()

First, the index name (goodreads), namespace (books), and filterable_fields list are defined. This list contains the author, genres, avg_rating, and num_ratings fields, which will be used for metadata filtering. Next, the CSV dataset is loaded using pandas.read_csv. During loading, ast.literal_eval is used to convert genres into lists, while the Num_Ratings column is interpreted as integer values. After loading, duplicate records are removed based on the Book and Author columns. Additionally, missing descriptions are filled with “No description available.”, while author names and genres are converted to lowercase to standardize the values used for filtering. The DataFrame is then converted into a list of dictionaries using df.to_dict(orient="records"), where each dictionary represents a book. Finally, the create_index method of the VectorService class is used to create the index and namespace in pinecone, while insert_data generates and inserts the embeddings along with their associated metadata. Regardless of whether the process completes successfully or encounters an error, the close_connection method is executed to release the resources associated with pinecone.

Finally, we’ll look at the main.py file, which is responsible for running the book recommendation RAG system.

import argparse
from dotenv import load_dotenv
from src.services.vector_db import VectorService
from src.services.llm import LLMService
load_dotenv()
def _get_example(option: int) -> dict:
    examples = {
        1: {
            "query": (
                "I'm looking for a psychological suspense or horror book with a clever, "
                "unpredictable plot and morally complex characters. "
                "I prefer stories that make me think rather than relying too much on action."
            ),
            "filters": {
                "genres": ["Suspense", "Horror"],
                "avg_rating": [4.0, 5.0],
                "num_ratings": [50000, None]
            }
        },
        2: {
            "query": (
                "I'm looking for a dark, suspenseful story that explores fear, isolation, "
                "and the psychological struggles of its characters. "
                "I enjoy unsettling atmospheres and stories where "
                "ordinary people are pushed to their limits."
            ),
            "filters": {
                "author": "Stephen King"
            }
        }
    }
    return examples[option]
if __name__ == "__main__":
    vector_service = VectorService()
    llm_service = LLMService()
    parser = argparse.ArgumentParser(
        description="Book recommendation with RAG and pre-filtering"
    )
    parser.add_argument(
        "--example",
        type=int,
        choices=[1, 2],
        required=True,
        help="Search example to execute"
    )
    args = parser.parse_args()
    example = _get_example(args.example)
    query = example["query"]
    filters = example["filters"]
    try:
        search_results = vector_service.search(
            query=query,
            index_name="goodreads",
            namespace_name="books",
            top_k=5,
            filters=filters
        )
    finally:
        vector_service.close_connection()
    response = llm_service.generate_response(query, search_results, filters)
    print(f"💬 Response: {response}\n")

To simplify the pre-filtering demonstration, two examples were defined in the _get_example function, each containing a query and its corresponding metadata filters. These examples can be selected through the command-line interface (CLI), as demonstrated in the next section. For this purpose, the argparse library was used to parse command-line arguments. The --example argument specifies which of the two examples will be executed, while the parse_args method processes the arguments and makes their values available through the args variable. Once an example is selected, its query and corresponding filters are passed to the search method of the VectorService class. This method performs a vector search in pinecone while applying the specified metadata filters. With top_k=5, up to five books with the highest vector similarity to the query are returned from those that satisfy the applied filters. Finally, the search results, along with the query and selected filters, are passed to the generate_response method of the LLMService class. The LLM uses this information to generate book recommendations based on the user’s preferences, following the instructions defined in the prompt.

How to Run the System

As mentioned earlier, data ingestion into pinecone is performed independently of the RAG system’s execution. Therefore, the ingestion.py file must be run first. Once the process is complete, the following messages will be displayed:

✅ Index created successfully.
🚀 Starting data insertion into index 'goodreads'...
📦 Inserting batches: 99/99
✅ Insertion of 9896 records completed successfully.

It is worth noting that the original dataset contains approximately 10,000 records. However, after removing duplicates based on the combination of book title and author, 9,896 books were inserted into pinecone.

With data ingestion complete, the RAG system is ready to run. As explained in the previous section, two examples were defined to demonstrate the application of different metadata filters. To run each example from the command line, use:

python -m src.main --example 1

python -m src.main --example 2

Results Analysis

This section presents the assistant’s responses for each of the defined examples, along with an analysis of how pre-filtering helps retrieve context that is better aligned with user preferences in a RAG system.

First, the assistant’s response for example 1 is shown below, rendered in markdown. In this example, we search for psychological suspense or horror books with an average rating between 4 and 5 and at least 50,000 ratings.

With pre-filtering applied, the candidate set for vector similarity search was reduced from 9,896 to just 215 books that satisfy the metadata filters. This represents an approximately 97.8% reduction in the search space. In addition to narrowing the candidate set, pre-filtering ensures that the retrieved books satisfy the objective criteria defined by the user, such as genre, average rating, and number of ratings. As a result, vector similarity search focuses on identifying, among the books that meet these criteria, those that are semantically closest to the preferences expressed in the query.

Next, the assistant’s response for example 2 is presented. In this example, only the author filter was applied, restricting the search to books by Stephen King.

With the author filter applied, the candidate set for vector similarity search was reduced from 9,896 to just 56 books, representing an approximately 99.4% reduction in the search space. As a result, vector similarity search focuses exclusively on Stephen King’s books, allowing the retrieval of those that are semantically closest to the preferences expressed by the user.

To conclude, consider a hypothetical scenario with 1 million books stored in a vector database. Without pre-filtering, the similarity search would have this entire collection as its candidate set, although algorithms such as HNSW, whose workings were explained in another post, enable approximate searches without comparing the query against every stored vector. With pre-filtering, this candidate set can be significantly narrowed to books that satisfy the metadata filters defined by the user. In the author-filtering example, we observed an approximately 99.4% reduction in the candidate set. If the same proportion were maintained in a hypothetical scenario with 1 million books, approximately 6,000 candidates would remain eligible for similarity search. This example illustrates the potential of pre-filtering in RAG systems. Instead of performing retrieval over a broad collection of data, the process can be directed toward records that meet the user’s objective criteria. In this way, pre-filtering not only reduces the search space but also helps ensure that the context provided to the LLM is better aligned with the specified requirements, making information retrieval more targeted and relevant.