We have seen recently a surge in vector databases in this era of generative AI. The idea behind vector databases is to index the data with vectors that relate to that data. Vector databases are often used for recommender engines where we learn vector representations of users and items we want to recommend. This allows us to quickly find similar items by using an approximate nearest neighbors search.
We are going to cover why we need Vector databases in the context of LLMs and why we need to index the data. We are covering 3 indexing techniques:
Product Quantization,
Locality-sensitive hashing,
and Hierarchical Navigable Small World.
We also cover the Maximal marginal relevance algorithm to ensure diversity in the information retrieved from vector databases.
The video in this session is mostly the video version of the following Newsletter:
Deep Dive: How do Vector Databases Work
Today we dive into the subject of vector databases. Those databases are often use in search engines by using the vector representations of the items we are trying to search. We dig into the different algorithms that allow us to search for vectors among billions or trillions of documents. We cover:
Share this post