ELIPS treats the GPU as an optional engine — never a runtime dependency. Domain code talks to Index; when GPU is built in, the distance and top-k stage happens to run on the device. Everything else — selection, allocation, batching, fallback — is out of band, behind narrow ports.

Thesis

Three commitments shape every header under include/elips/gpu_engine/:

Interface Segregation. Four ports (compute, memory, kernel, stream) instead of one giant facade.
Failure is expected. Every GPU operation returns std::expected<T, GpuError> — device-lost, OOM, and unsupported metric are values, not exceptions.
Optional, additive. The CPU index and serving path stand on their own. GPU is a build-time switch (-DELIPS_GPU_ENABLED=ON), not a runtime requirement.

Layered architecture

Surface → orchestration → ports → backends. Each layer only knows the layer directly beneath it.

Ports — interface segregation

The engine is decomposed across five virtual interfaces so each caller depends only on the slice it actually needs.

cpp

// GpuBackend.hpp — the umbrella port (init + compute + top_k)
class GpuBackend {
public:
    virtual std::expected<void, GpuError>      initialize(const GpuConfig&) = 0;
    virtual void                               shutdown() noexcept = 0;
    virtual GpuDeviceInfo                      device_info() const noexcept = 0;

    virtual std::expected<GpuBuffer, GpuError> allocate_device(size_t bytes) = 0;
    virtual void                               free_device(GpuBuffer) noexcept = 0;
    virtual std::expected<void, GpuError>      upload(const void*, GpuBuffer&, size_t) = 0;
    virtual std::expected<void, GpuError>      download(const GpuBuffer&, void*, size_t) = 0;

    virtual std::expected<void, GpuError>      compute_distances_batch(
        std::span<const float> queries, std::span<const float> database,
        std::span<float> out, size_t nq, size_t nb, size_t dim, elips::Metric) = 0;

    virtual std::expected<void, GpuError>      top_k(
        std::span<const float> distances,
        std::span<uint32_t> indices_out, std::span<float> values_out,
        size_t nq, size_t nb, size_t k) = 0;

    virtual void synchronize() = 0;
    virtual bool is_idle() const noexcept = 0;
};

The dedicated ports keep call sites honest:

GpuMemoryPort — allocate / deallocate / pinned host buffers / bytes_used / available / peak.
GpuKernelPort — explicit metric-typed kernels: cosine_fp32, euclidean_fp32, dot_product_fp32.
GpuStreamPort — fine-grained synchronisation surface for callers that schedule their own streams.

There is no GPU-resident ANN index. The device accelerates the exact distance + top-k stage; the exact brute-force path is surfaced to Python as elips.bf_knn(xq, xb, k, device="metal"). HNSW and IVF indexes stay on the CPU.

GpuConfig

cpp

enum class GpuPolicy    { Auto, PreferGpu, RequireGpu, CpuOnly, Specific };
enum class GpuPrecision { FP32, FP16, Auto };

struct GpuConfig {
    GpuPolicy    policy{GpuPolicy::Auto};
    std::string  preferred_backend;        // "metal" | "cuda" | "hip" | ""
    int32_t      device_index{-1};         // -1 = let the selector decide

    size_t       device_memory_pool_bytes{0};            // 0 = auto-size
    size_t       pinned_host_pool_bytes{256UL * 1024 * 1024};
    bool         use_unified_memory{false};              // Apple Silicon

    size_t       query_batch_window_us{500};
    size_t       query_batch_max_size{256};
    GpuPrecision search_precision{GpuPrecision::Auto};
    bool         enable_fp16_search{false};

    bool         enable_profiling{false};
    bool         emit_kernel_timings{false};
};

policy drives the selector: Auto tries GPU and falls back; PreferGpu warns but still falls back; RequireGpu errors if no device is usable; CpuOnly short-circuits the engine entirely; Specific pins to preferred_backend / device_index. From Python this is reached with Client(..., device="metal", fallback=False) or gpu=GpuConfig(...) — never gpu=True.

Backends & maturity

Three concrete backends live under src/gpu_engine/backends/:

Metal (backends/metal/) — Apple Silicon, the implemented path (Beta, validated on Apple M4). Leans on unified memory (has_unified_memory in GpuDeviceInfo).
CUDA (backends/cuda/) — NVIDIA path, ships cosine_fp32.cu kernel. Experimental.
HIP / ROCm (backends/hip/) — AMD path, parallel cosine_fp32.hip kernel. Experimental.

Backend	Status	Notes
Metal	Production	Default ON on Apple; distance + top-k kernels.
CUDA	Experimental	NVIDIA; opt-in.
HIP / ROCm	Experimental	AMD; opt-in.
CPU	Always	Fallback and default when GPU is not built in.

Each backend implements GpuBackend, registers its capabilities in GpuDeviceInfo (FP16 support, unified memory, batching) and is ranked by BackendSelector against the accelerators ELIPS enumerates. From Python, that list is elips.accelerators():

python

import elips

for acc in elips.accelerators():          # list[AcceleratorSpec]
    print(acc.name, acc.backend, acc.unified_memory, acc.supports_fp16)

print(elips.has_gpu)                       # bool
best = elips.accelerator()                 # AcceleratorSpec | None

# pin a device; raise BackendUnavailableError if absent and fallback is off
client = elips.PersistentClient("/tmp/db", dimension=384, device="metal", fallback=False)
print(client.native.gpu_info())

Distance & top-k path

The GPU does one job: the exact distance computation and the top-k selection over a batch of queries. There is no GPU-resident ANN structure — no CAGRA, no device-side IVF — so the kernels stay small and portable across Metal, CUDA, and HIP. The exact brute-force path is exposed to Python as bf_knn, and ANN continues to run on the CPU indexes (IndexHNSW, IndexIVFFlat, …).

python

import numpy as np, elips

xb = np.random.rand(100000, 128).astype("float32")
xq = np.random.rand(1000, 128).astype("float32")

# exact k-NN on the Apple GPU, no index build
D, I = elips.bf_knn(xq, xb, 10, metric="l2", device="metal")

On the shipped Apple M4 benchmark (dim 128, nb 100000, nq 1000, k 10, L2, ELIPS 1.1.0), bf_knn on Metal reaches recall 1.000 at 914 QPS batched — versus 24 QPS when queries are issued one at a time, which is exactly what the query batcher below is for.

Query batcher

Concurrent single-query searches coalesce into one kernel launch. QueryBatcher opens a window of query_batch_window_us µs (default 500), buffers up to query_batch_max_size queries (default 256), then flushes — either when the window closes or the buffer fills.

A short batching window converts many small queries into a few large kernel launches. Each query still resolves through its own std::future.

cpp

std::future<std::vector<SearchResult>>
QueryBatcher::enqueue(std::span<const float> q, size_t k);

struct BatchStats {
    size_t queries_coalesced{0};
    size_t kernel_launches{0};
    float  avg_batch_size{0.0f};
    float  p99_latency_us{0.0f};
};

Memory

GpuMemoryManager implements GpuMemoryPort: a slab-allocator over a device-side pool sized by device_memory_pool_bytes, plus a pinned host pool sized by pinned_host_pool_bytes for fast H2D / D2H transfers. GpuMemoryPool provides a simpler in-pool allocator used by the search pipeline for scratch buffers.

On Apple Silicon, use_unified_memory avoids redundant copies — the same allocation is visible to host and GPU. See ADR-GPU-006.

Error model

cpp

enum class GpuError {
    DeviceNotFound, InsufficientMemory, KernelLaunchFailed,
    TransferFailed, IndexBuildFailed, UnsupportedMetric,
    InitializationFailed, BackendUnavailable,
};

[[nodiscard]] std::expected<...> op(...);   // every port method

Errors flow up as values; the caller decides whether to fall back to CPU, retry on another device, or surface. The fallback chain is codified in ADR-GPU-008.

Metrics & profiling

client.native.gpu_info() (on the native Database handle) returns a GpuMetricsSnapshot:

cpp

struct GpuMetricsSnapshot {
    std::string backend, device_name;
    size_t device_memory_used_bytes, device_memory_total_bytes;
    size_t search_kernel_launches_total;
    size_t search_p50_latency_us, search_p99_latency_us;
    float  batch_avg_size, batch_coalescing_ratio;
    bool   fp16_search_enabled;
    size_t fallback_events_total;
    size_t kernel_errors_total;
    size_t pinned_memory_pool_used_bytes;
};

Per-kernel timing is opt-in via GpuConfig::emit_kernel_timings; recent samples are retrievable from GpuProfiler::recent_timings(n).

Build flags

bash

# CPU-only — the default
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release

# GPU-enabled — backends auto-detected from the host toolchain
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DELIPS_GPU_ENABLED=ON

cmake --build build -j
ctest --test-dir build --output-on-failure

Reference: ADR-GPU-001 through ADR-GPU-008 in Design decisions.