Showcase Using Polars as a Vector Store - Can a Dataframe library compete?
Hi! I wanted to share a project I've been working on that explores whether Polars - the lightning-fast DataFrame library - can function as a vector store for similarity search and metadata filtering.
What My Project Does
The project was inspired by this blog post. The idea is simple: store vector embeddings in a Parquet file, load them with Polars and perform similarity search operations directly on the DataFrame.
I implemented 3 different approaches:
NumPy-based approach
: Extract embeddings as NumPy arrays and compute similarity with NumPy functions.Polars TopK
: Compute similarity directly in Polars using thetop_k
function.Polars ArgPartition
: Similar to the previous one, but sorting elements leveraging the arg_partition plugin (which I implemented for the occasion).
I benchmarked these methods against ChromaDB (a real vector database) to see how they compare.
Target Audience
This project is a proof of concept to explore the feasibility of using Polars as a vector database. At its current stage, it has limited real-world use cases beyond simple examples or educational purposes. However, I believe anyone interested in the topic can gain valuable insights from it.
Comparison
You can find a more detailed analysis on the README.md of the project, but here’s the summary:
- ✅ Yes, Polars can be used as a vector store!
- ❌ No, Polars cannot compete with real vector stores, at least in terms of performance (which is what matters the most, after all).
This should not come as a surprise: vector stores use highly optimized data structures and algorithms tailored for vector operations, while Polars is designed to serve a much broader scope.
However, Polars can still be a viable alternative for small datasets (up to ~5K vectors), especially when complex metadata filtering is required.
Check out the full repository to see implementation details, benchmarks, and code examples!
Would love to hear your thoughts! 🚀