GPU Engine Architecture

ELIPS features a unified, multi-backend GPU acceleration engine capable of offloading matrix distance calculations, IVF-PQ quantization, graph traversals, and dynamic batch queries to hardware accelerators.

Overview

The GPU engine provides ultra-high throughput vector search by exploiting massively parallel CUDA cores (NVIDIA), HIP streams (AMD), and Metal Performance Shaders (Apple Silicon).

Backend Support (CUDA, HIP, Metal)

• CUDA Backend: Native CUDA C++ kernels targeting NVIDIA Volta, Ampere, Hopper, and Blackwell architectures using async streams and Tensor Cores.
• HIP Backend: AMD ROCm acceleration for Instinct MI200/MI300 series accelerators.
• Metal Backend: Hardware-accelerated matrix multiplication on Apple M1/M2/M3/M4 unified memory architectures.

GpuPort & Device Abstractions

namespace elips::gpu {
    class GpuPort {
    public:
        virtual ~GpuPort() = default;
        virtual GpuDeviceInfo device_info() const = 0;
        virtual GpuMetricsSnapshot metrics() const = 0;
        virtual void compute_distances(
            const float* queries, std::size_t num_queries,
            const float* database, std::size_t num_vectors,
            std::size_t dim, float* distances
        ) = 0;
    };
}

GPU Index Family

ELIPS implements dedicated GPU index ports under include/elips/gpu_engine/:

• GpuBruteForceIndex — High-throughput exact k-NN distance computation.
• GpuIVFFlatIndex — Inverted File index with GPU cluster lookup.
• GpuIVFPQIndex — Product Quantization with GPU centroid lookup table decoding.
• GpuGraphIndex — GPU-assisted HNSW graph entry point evaluation.
• GpuHybridIndex — Concurrent GPU vector similarity and sparse text relevance ranking.

Memory Management & Pools

To avoid high latency associated with device memory allocation (`cudaMalloc`), ELIPS uses custom allocators:

• GpuMemoryPool — Pre-allocates slab chunks on device memory.
• PinnedBuffer — Host page-locked memory for zero-copy DMA transfers.
• UnifiedBuffer — Unified Virtual Memory (UVM) bridging CPU and GPU pointers.

Dynamic Batcher & Pipelines

namespace elips::gpu {
    class DynamicBatcher;
    class GpuIngestionPipeline;
    class GpuSearchPipeline;
}

The DynamicBatcher accumulates incoming single vector queries from multiple CPU worker threads into contiguous GPU batch matrices, maximizing Tensor Core compute efficiency.

Python API Integration

import elips

# Initialize GPU accelerator handle
with elips.connect(":memory:", dimension=1536, gpu=True) as engine:
    print(engine.gpu_info())
    # {'device_name': 'NVIDIA RTX 4090', 'total_memory_mb': 24576, 'backend': 'cuda'}