Developers are always winners in an open-source environment. And we love to see it.
We tested pgvector’s implementation of HNSW for you and compared it to pg_embbeding’s. This covers why this milestone is important for Postgres and why Neon is committed to supporting pgvector.
A few weeks ago, we released and open-sourced pg_embedding, a vector search extension that implements the Hierarchical Navigable Small Worlds (HNSW) index for 20x faster vector similarity search. Recognizing its performance improvements over the Inverted File (IVFFlat) index, the community urged pgvector maintainers to implement the HNSW index to the extension, and we are very happy to see pgvector’s support for HNSW in their coming 0.5.0 release.
Neon is open-source at its core, and we are committed to helping the community leverage the best tools for their applications. This is why we are excited about pgvector’s support for HNSW and are committed to having day-one support for its coming 0.5.0 version.
Note that as per the maintainers’ recommendations, the current version of pgvector’s HNSW index might be subject to change and is not ready for production.
This update to the most popular vector search extension proves that Postgres is all you need and is a scalable solution for building Large Learning Models (LLM) applications.
Why is HNSW index for Postgres a big deal?
We first introduced the HNSW index via our open-source pg_embedding extension for scalable vector similarity search in Postgres for million-row datasets, where we compared it against the IVFFlat index of pgvector in our benchmark study, “20x faster than pgvector”.
It was immediately apparent to us that it was the missing piece in Postgres’s story with vector search.
Before pg_embedding, the only way to do vector search in Postgres was using pgvector and the IVFFlat index. However, ANN-benchmarks which compare approximate nearest neighbor algorithms search, exposed IVFFlat’s limitations at scale, making it hard for the community to adopt Postgres for million-plus-rows LLM applications.
I'll pay $2k to anyone who fixes this, e.g. merged PR to convert pgvector to hnswlib. Anyone else want to pile on? https://t.co/eBA4Rn61QM
— Alex Graveley (@alexgraveley) June 30, 2023
Observing those limitations around IVFFlat, pgvector users asked the maintainers to implement the index, and today, we’re pleased to see the HNSW index making its debut in pgvector.
pgvector (HNSW) vs. pg_embedding
The addition of HNSW unquestionably improves pgvector’s performance. We were curious to see how pgvector now stacks up against pg_embedding, so we ran a benchmark on Postgres hosted on a 32GB of memory and 16 CPUs c5a.4xlarge AWS EC2 instance.
We tested the extensions based on the following criteria:
- HNSW index build time
- vector search execution time
- recall as well as query per second (QPS) for both extensions using NirantK 1M OpenAI embeddings dataset and a test set of 100 randomly generated OpenAI embeddings.
You can see the improvements in our recent comparison using Neon:
So, what does this mean for choosing between pgvector and pg_embedding?
Even though HNSW brings both extensions at the same search execution time level, the index size remains smaller on pg_embedding. The reason index size matters is that it impacts your overall storage size but also your query performance.
One way to speed up index scans is by uploading them to the `shared_buffers` using the extension `pg_prewarm`. This means that you need to allocate enough memory in your Postgres instance to fit your HNSW index in the `shared_buffers`.
Index size may not be a deal breaker for many LLM applications with small datasets. This is why we believe that both extensions can coexist. pg_embedding enables us to experiment quicker, which will potentially help us find ways to improve the Postgres ecosystem for all developers.
Additionally, we designed and open-sourced pg_embedding with a vision to allow LLM applications to scale to billions of rows. With a dedicated team of engineers and Postgres contributors, we’re more committed than ever to helping you harness the power of Postgres to scale your AI applications.
