We have seen recently a surge in vector databases in this era of generative AI. The idea behind vector databases is to index the data with vectors that relate to that data. Vector databases are often used for recommender engines where we learn vector representations of users and items we want to recommend. This allows us to quickly find similar items by using an approximate nearest neighbors search.
We are going to cover why we need Vector databases in the context of LLMs and why we need to index the data. We are covering 3 indexing techniques:
Product Quantization,
Locality-sensitive hashing,
and Hierarchical Navigable Small World.
We also cover the Maximal marginal relevance algorithm to ensure diversity in the information retrieved from vector databases.
The video in this session is mostly the video version of the following Newsletter:
Introduction to LangChain: Vector Database Basics