ELIPS ships a reproducible benchmark harness (benchmarks/bench_ann.py) that measures recall@k and QPS against an exact-Flat sanity gate, with faiss as the reference. The numbers below are from that harness — reported as measured, wins or losses.
Overview
Environment: Apple M4 (arm64), macOS, Python 3.13, ELIPS 1.1.0 (GPU bindings on), faiss-cpu 1.15.1. Settings: dim=128, nb=100000, nq=1000, k=10, metric euclidean (L2). Ground truth is exact numpy L2 brute force; the exact-Flat sanity gate passes. QPS and p50/p99 are from single-query (non-batched) search unless noted.
Key metrics
- Recall@10: fraction of the top-10 approximate results that match the exact L2 ground truth.
- QPS: queries per second (single-query search unless a row says batched).
- p50 / p99: per-query latency percentiles in milliseconds.
- Build s: index construction time.
ANN results (Apple M4)
| Config | Recall@10 | QPS | p50 ms | p99 ms |
|---|---|---|---|---|
| ELIPS IndexFlat (cpu) | 1.000 | 489 | 1.981 | 3.336 |
| ELIPS bf_knn (metal, 1-query) | 1.000 | 24 | 41.315 | 69.498 |
| ELIPS bf_knn (metal, batched) | 1.000 | 914 | 1.094 | 1.094 |
| ELIPS IVFFlat nlist=316 nprobe=128 (cpu) | 0.801 | 1,616 | 0.617 | 0.650 |
| faiss IndexFlatL2 (cpu) | 1.000 | 989 | 0.926 | 1.432 |
| faiss IVFFlat nlist=316 nprobe=128 (cpu) | 0.793 | 2,587 | 0.385 | 0.416 |
ELIPS and faiss both use the FAISS-style Index API (IndexFlat / IndexIVFFlat, metric="l2"); labels are row indices, so recall is measured against the same ground truth.
IVF nprobe trade-offs
For IndexIVFFlat with nlist=316, raising nprobe trades QPS for recall (ELIPS, cpu, M4):
| nprobe | Recall@10 | QPS |
|---|---|---|
| 1 | 0.030 | 46,614 |
| 8 | 0.152 | 14,123 |
| 32 | 0.381 | 6,273 |
| 128 | 0.801 | 1,616 |
CPU vs. Metal GPU
The only implemented GPU backend is Metal (Apple silicon); it accelerates the exact brute-force path, elips.bf_knn(..., device="metal"). Batching matters: a single query over Metal is latency-bound (24 QPS), but amortised over one batched call it reaches 914 QPS at recall 1.000 — comparable to the CPU exact path (IndexFlat, 489 QPS) while leaving the CPU free. CUDA and HIP backends are experimental and not benchmarked here.
Running benchmarks (elips bench)
Run the built-in benchmarking tool directly from the command line:
# Reproduce the ANN suite
python benchmarks/bench_ann.py
# Quick engine micro-benchmark via the CLI
elips bench /path/to/db --count 100000 --dim 128