RECENT POSTS
Vector Databases: Getting Started
A vector database is a system designed to store and retrieve high-dimensional vectors through similarity search. These vectors are numerical representations of different types of data, such as text, images, audio, and video. In practice, vector databases are primarily used to store embeddings, which are numerical representations generated by machine learning models that preserve the…
Embeddings: Getting Started
An embedding is a numerical representation of a word, sentence, image, or audio sample in the form of a high-dimensional vector, designed to capture relevant characteristics of the data, such as its semantic meaning. Embeddings are typically generated by machine learning models trained on large amounts of data. During training, these models learn to organize…
Tokenization: BPE Algorithm
NLP models and LLMs do not process raw text directly, but instead operate on numerical representations. In this context, tokenization is the process of converting a sequence of characters (a string) into a sequence of tokens, smaller units of text. These tokens are then mapped to numerical identifiers (integers), which correspond to positions in a…
Edvaldo Melo 