README.md

August 23, 2026 · View on GitHub

Radar chart

Vector Index Benchmark for Embeddings (VIBE) is an extensible benchmark for approximate nearest neighbor search methods, or vector indexes, using modern embedding datasets.

Overview

📊 Modern vector index benchmark with embedding datasets
🎯 Includes datasets for both in-distribution and out-out-distribution settings
🏆 Includes the most comprehensive collection of state-of-the-art vector search algorithms
💎 Support for quantized datasets in both 8-bit integer and binary precision
🖥️ Support for HPC environments with Slurm and NUMA
🚀 Support for GPU algorithms

Results

The current VIBE results can be viewed on our website:

https://vector-index-bench.github.io

The website also features several other tools and visualizations to explore the results.

The results are run on Intel Xeon Gold 6230 (Cascade Lake) CPUs with support for AVX-512 instructions. All algorithms are benchmarked using a single core. The GPU algorithms are run using an NVIDIA V100 (32 GB). The next results update will use AMD Turin 9965 CPUs, while GPU algorithms will be run using NVIDIA GH200 (96 GB).

Publication

E. Jääsaari, V. Hyvönen, M. Ceccarello, T. Roos, M. Aumüller. VIBE: Vector Index Benchmark for Embeddings. Journal of Data-centric Machine Learning Research, 2026.

Authors

VIBE is maintained by Elias Jääsaari, Matteo Ceccarello, and Martin Aumüller.

Alternative Benchmarks

Please check out big-ann-benchmarks (NeurIPS 2021/2023) for the state-of-the-art in billion-scale ANN and constrained ANN, such as ANN under filtered or sparse workloads.

Credits

The evaluation code and some algorithm implementations in VIBE are based on the ann-benchmarks project.

License

VIBE is available under the MIT License (see LICENSE). The pyyaml library is also distributed in the vibe folder under the MIT License.

Getting started

Requirements

Linux
Apptainer (or Singularity)
Python 3.6 or newer

For example, to install Apptainer on Ubuntu:

sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

Some algorithms may require that the CPU supports AVX-512 instructions and some algorithms may require an Intel CPU due to a dependency on Intel MKL. The GPU algorithms assume that an NVIDIA GPU is available.

Important

For accurate benchmarking, it is recommended to disable SMT/hyperthreading:

echo off | sudo tee /sys/devices/system/cpu/smt/control

On hybrid architectures (e.g., Intel Raptor Lake), it is recommended to disable efficiency (E) cores.

If not running in an HPC or cloud environment, it is also recommended to set the performance governor

sudo cpupower frequency-set -g performance

and to check that transparent huge pages are set to madvise or never:

cat /sys/kernel/mm/transparent_hugepage/enabled

Building library images

Building all library images can be done using

./install.sh

Use ./install.sh --skip-gpu if you don't need GPU methods. To build an image for a single library:

./install.sh --algorithm hnswlib

Tip

install.sh takes an argument --build-dir that specifies the temporary build directory. For example, to speed up the build in a cluster environment, you can set the build directory to a location on an SSD while the project files are on a slower storage medium.

Tip

See an example Slurm job for building the libraries using Slurm.

Running benchmarks

The benchmarks for a single dataset can be run using run.py. For example:

python3 run.py --dataset agnews-mxbai-1024-euclidean

The run.py script does not depend on any external libraries and can therefore be used without a container or a virtual environment.

Common options for run.py:

--parallelism n: Use n processes for benchmarking.
--module mod: Run the benchmark only for algorithms in module (library) mod.
--algorithm algo: Run the benchmark for only algorithm algo.
--count k: Run the benchmarks using k nearest neighbors (default 100).
--gpu: Run the benchmark in GPU mode.

For all options, see

python3 run.py --help

The benchmark should take less than 24 hours to run for a given dataset using parallelism > 12. We recommend having at least 16 GB of memory per used core.

Tip

See an example Slurm job for running the benchmark using Slurm.

Plotting results

You should first build the plot.sif image:

singularity build plot.sif plot.def

Before plotting, the current results must first be exported:

./export_results.sh --parallelism 8

The results for a dataset can then plotted with e.g.:

./plot.sh --dataset agnews-mxbai-1024-euclidean

To plot the radar chart above, use:

./plot.sh --plot-type radar

For all available options, see:

./plot.sh --help

Tip

You can also use uv to directly run export_results.py and plot.py without building the container image if preferable. The arguments for these scripts are the same as above.

Creating datasets from scratch

The benchmark code downloads precomputed embedding datasets. However, the datasets can also be recreated from scratch, and it is also possible to create new datasets by modifying the datasets.py file.

Creating the datasets can be done using create_dataset.sh. It first requires that dataset.sif is built:

singularity build dataset.sif dataset.def

The VIBE_CACHE environment variable should be set to a cache directory with at least 200 GB of free space when creating image embeddings using the Landmark or ImageNet datasets. Datasets can then be created using the --dataset argument (the --nv argument specifies that an available GPU can be used):

export VIBE_CACHE=$LOCAL_SCRATCH
./create_dataset.sh --singularity-args "--bind $LOCAL_SCRATCH:$LOCAL_SCRATCH --nv" --dataset agnews-mxbai-1024-euclidean

Tip

See an example Slurm job for creating datasets using Slurm.

Adding a new method to the benchmark

VIBE is an on-going effort and we actively welcome new additions to the benchmarks.

See the new method guide. In short, add your algorithm in the folder vibe/algorithms/{METHOD}/ by providing:

Python wrapper in module.py
Singularity container definition in image.def
Hyperparameter grid in config.yml

Please refer to e.g. the hnswlib module for a reference implementation.

Name	Type	n	d	Distance
agnews-mxbai-1024-euclidean	Text	769,382	1024	euclidean
arxiv-nomic-768-normalized	Text	1,344,643	768	any
dpr-jina-768-normalized	Text	20,969,760	768	any
glove-200-cosine	Word	1,192,514	200	cosine
gooaq-distilroberta-768-normalized	Text	1,475,024	768	any
imagenet-clip-512-normalized	Image	1,281,167	512	any
inaturalist-resnet-2048-cosine	Image	499,000	2048	cosine
landmark-dino-768-cosine	Image	760,757	768	cosine
landmark-nomic-768-normalized	Image	760,757	768	any
msmarco-qwen-1024-normalized	Text	8,840,823	1024	any
yahoo-minilm-384-normalized	Text	677,305	384	any

Out-of-distribution datasets

Name	Type	n	d	Distance
hotpotqa-harrier-640-normalized	Text	5,233,329	640	any
imagenet-align-640-normalized	Text-to-Image	1,281,167	640	any
laion-clip-512-normalized	Text-to-Image	1,000,448	512	any
yandex-200-cosine	Text-to-Image	1,000,000	200	cosine
cqadupstack-lemur-2048-ip	Multi-vector	457,149	2048	IP
cqadupstack-muvera-5120-ip	Multi-vector	457,149	5120	IP
yi-128-ip	Attention	187,843	128	IP
llama-128-ip	Attention	256,921	128	IP

Deprecated datasets

Deprecated datasets will remain available, but their benchmark results will not be updated in the future.

Name	Type	n	d	Distance
ccnews-nomic-768-normalized	Text	495,328	768	any
celeba-resnet-2048-cosine	Image	201,599	2048	cosine
coco-nomic-768-normalized	Text-to-Image	282,360	768	any
codesearchnet-jina-768-cosine	Code	1,374,067	768	cosine
simplewiki-openai-3072-normalized	Text	260,372	3072	any

Algorithms

Method	Version
ANNOY	1.17.3
FALCONN++	git+5fd3f17
FlatNav	0.1.2
CAGRA	26.04.00
GGNN	0.9
Glass	git+d2296ec
HNSW	0.8.0
HNSW-RaBitQ	git+5ea4df0
IVF (Faiss)	1.14.3
IVF-PQ (Faiss)	1.14.3
IVF-RaBitQ	git+5ea4df0
Jasper	git+23647b9
LVQ (SVS)	0.4.0
LeanVec (SVS)	0.4.0
LoRANN	0.5
MLANN	git+de8f9d6
MRPT	2.0.4
NGT-ONNG	2.7.4
NGT-QG	2.7.4
NSG	1.14.3
PAG	git+ee34ed7
PDX	git+93531b9
PUFFINN	git+fd86b0d
PyNNDescent	0.6.0
RoarGraph	git+f2b49b6
ScaNN	1.4.2
SymphonyQG	git+32a0019
Vamana (DiskANN)	0.7.0