Edvaldo Melo

Sharing my knowledge and insights on AI and Data Science.

Category: AI Engineering

  • RAG: Pre-Filtering

    In real-world applications, semantic similarity search alone is not always sufficient to meet information retrieval requirements. Beyond semantic relevance, results often need to satisfy specific constraints, such as belonging to a particular category or excluding certain values. When performing similarity search across the entire dataset, the retrieved results may be semantically relevant to the query…

  • Vector Databases: HNSW Algorithm

    To perform a search in a vector database containing millions or even billions of high-dimensional vectors, it would be necessary to compute the similarity between the query vector and every stored vector and then return the top-k nearest vectors. This approach is known as exact search and, although it guarantees exact results, it becomes computationally…

  • RAG: Hybrid Search

    Traditional RAG, which I covered in my previous post, uses dense vector search to retrieve documents that are semantically similar to the user’s query. This approach is especially useful for open-ended or natural language questions, where users describe their intent without necessarily knowing or using the exact terms found in the documents. However, relying solely…

  • RAG: Getting Started

    Although an LLM is trained on a vast amount of data and is capable of generating high-quality text, it has inherent limitations, such as being restricted to the knowledge available up to its training cutoff and lacking access to domain-specific or private knowledge bases. Retrieval-Augmented Generation (RAG) is widely used to overcome these limitations by…

  • Vector Databases: Getting Started

    A vector database is a system designed to store and retrieve high-dimensional vectors through similarity search. These vectors are numerical representations of different types of data, such as text, images, audio, and video. In practice, vector databases are primarily used to store embeddings, which are numerical representations generated by machine learning models that preserve the…

  • Embeddings: Getting Started

    An embedding is a numerical representation of a word, sentence, image, or audio sample in the form of a high-dimensional vector, designed to capture relevant characteristics of the data, such as its semantic meaning. Embeddings are typically generated by machine learning models trained on large amounts of data. During training, these models learn to organize…

  • Tokenization: BPE Algorithm

    NLP models and LLMs do not process raw text directly, but instead operate on numerical representations. In this context, tokenization is the process of converting a sequence of characters (a string) into a sequence of tokens, smaller units of text. These tokens are then mapped to numerical identifiers (integers), which correspond to positions in a…

  • LangChain: Chatbot with Streamlit

    The goal of this post is to build a web interface for the chatbot developed in the previous post, using Streamlit, a free framework that makes it easy and intuitive to create web applications directly with Python. Since this post is a direct continuation of the previous one, it is strictly necessary that you read…

  • LangChain: Observability with LangSmith

    The goal of this post is to enhance the chatbot developed in the previous post by introducing an agent capable of using a tool to retrieve information from the internet whenever needed. In addition, LangSmith is presented as an observability layer, enabling the tracking of agent executions and the analysis of each step of the…

  • LangChain: Chatbot

    LangChain is an open-source framework designed to simplify the development of applications based on Large Language Models (LLMs). It provides a modular architecture that makes it easy to integrate language models, prompts, memory mechanisms, external APIs, and diverse data sources, enabling the creation of organized, flexible, and extensible solutions. The goal of this post is…