ELIPS features an automated, reproducible benchmarking framework in C++ and CLI to evaluate single-thread and multi-thread vector search throughput (QPS), p50/p95/p99 latency, index construction times, recall accuracy, and GPU memory bandwidth scaling.
Overview
The performance of embedded vector databases depends on index graph quality, cache locality, SIMD vectorization (AVX-512, ARM Neon), and lock contention during concurrent queries.
Key Performance Metrics
- QPS (Queries Per Second): Total query throughput evaluated across 1, 4, 16, and 64 worker threads.
- Latency Distribution: Microsecond latency bounds measured at p50, p95, and p99 percentiles.
- Recall@K: Proportion of top-$K$ approximate graph search results matching ground-truth exact brute-force Euclidean/Cosine distance results ($K=1, 10, 100$).
- Memory Footprint: Bytes allocated per vector node (including vector data, payload map headers, and bi-directional graph link lists).
HNSW Parameter Trade-offs
The graph index parameters M, ef_construction, and ef_search directly govern the trade-off between recall accuracy, insertion throughput, query latency, and RAM usage.
| Configuration | M | ef_construction | ef_search | Recall@10 | Latency (p95) | QPS / Core |
|---|---|---|---|---|---|---|
| Low-Latency / Fast | 12 | 100 | 16 | 0.942 | 0.12 ms | 8,300 |
| Balanced Default | 16 | 200 | 50 | 0.985 | 0.35 ms | 2,850 |
| High Precision | 32 | 400 | 128 | 0.998 | 0.89 ms | 1,120 |
CPU vs. GPU Benchmark Results
Evaluated on 1,000,000 vectors of dimension 1,536 (Ada-002 embeddings) using Cosine distance:
- CPU (Apple M3 Max 16-Core / 64GB UMA): 14,200 QPS (p95 latency: 0.28ms).
- GPU (NVIDIA RTX 4090 / CUDA GpuIVFPQIndex): 185,000 QPS (p95 latency: 0.04ms).
- GPU Batch Mode (DynamicBatcher batch_size=256): 420,000 QPS.
Durability Policy Throughput
Write throughput evaluated on persistent disk storage (PCIe Gen4 NVMe SSD):
durability = "sync_on_commit"— 4,200 writes/sec (fsync bound per commit).durability = "wal_only"— 95,000 writes/sec (buffered OS page cache WAL).durability = "in_memory"— 340,000 writes/sec (pure RAM memory graph build).
Running Benchmarks (`elips bench`)
Run the built-in benchmarking tool directly from the command line:
# Run a 100,000 vector benchmark with 1536 dimensions
elips bench --vectors 100000 --dim 1536 --threads 16 --metric cosine
# Benchmark GPU acceleration
elips bench --vectors 1000000 --dim 384 --gpu --batch-size 128