Database

Overview

Database is the top-level handle to an ELIPS instance. In the C++ layer it is ElipsInstance; the Python binding exposes it under the name Database. Every operation — inserting vectors, searching, issuing EQL queries, managing transactions — flows through this object.

A Database is opened once and kept alive for the lifetime of your process (or for as long as you need it). It is safe to share across threads; an internal instance mutex guards the vault registry, while each individual Vault carries its own reader-writer lock that allows concurrent searches without serialisation.

import elips

# Minimal — in-memory ephemeral database
db = elips.open(":memory:", dimension=128, metric="cosine")

# Persistent on disk
db = elips.open(
    "/var/lib/myapp/vectors.elips",
    dimension=768,
    metric="cosine",
    durability="standard",
)

Opening a Database — elips.open()

The module-level elips.open() function is the primary entry point. It constructs a Config from keyword arguments, validates it against any existing on-disk metadata, and returns a fully initialised Database.

Signature

elips.open(
    path: str,
    *,
    dimension: int                           = 0,
    metric: str                              = "cosine",
    durability: str                          = "standard",
    access_mode: str                         = "read_write",
    index: str                               = "graph",
    segmented_storage: bool                  = False,
    metadata_acceleration: bool              = True,
    embedder: callable | None                = None,
    **kwargs,
) -> Database

Parameters

• path — Filesystem path to the database directory. Pass ":memory:" for a fully in-memory, non-persistent database. The directory is created if it does not exist (for read-write mode).
• dimension — Number of dimensions for every vector stored in this database. Required when creating a new database. Ignored (but validated) when reopening — ELIPS locks the dimension to the value stored in the persisted metadata; passing a mismatching value raises ConfigError.
• metric — Distance metric. One of "cosine", "euclidean", or "dot_product". Like dimension, this is a persisted identity field and cannot be changed after the first open.
• durability — WAL flush strategy. See Config for the full breakdown of each level. Defaults to "standard".
• access_mode — "read_write" (default) or "read_only". Read-only mode acquires no file locks beyond a shared reader lock and disables all mutation APIs.
• index — Index strategy: "graph" (HNSW, default) or "exact" (brute-force, no index overhead, scales poorly beyond ~100 k vectors).
• segmented_storage — When True, vector data is stored across multiple memory-mapped segments, enabling databases larger than the available virtual address space on 32-bit platforms (rare). Defaults to False.
• metadata_acceleration — Maintains a secondary in-memory hash map of record metadata for O(1) filter evaluation during search. Disable only if memory is extremely constrained. Defaults to True.
• embedder — A callable (text: str) -> list[float] that ELIPS calls automatically when you use text-based APIs (seek_text, place_document). Equivalent to calling Config.text_embedder(fn, ...).

Return value

A Database instance. The connection to the storage engine is live immediately; no further initialisation call is needed.

Exceptions

• ConfigError — dimension/metric/index conflict with existing database, or an invalid parameter value was supplied.
• IOError / OSError — path is not accessible or the database files are corrupted.
• LockConflict — another process holds an exclusive lock on the database directory.

import elips

# New database — dimension is set for the first time
db = elips.open(
    "/data/products.elips",
    dimension=1536,
    metric="cosine",
    durability="paranoid",
    index="graph",
    metadata_acceleration=True,
)

# Reopen the same database — dimension/metric are read from disk
db2 = elips.open("/data/products.elips")

# Wrong dimension → ConfigError
try:
    bad = elips.open("/data/products.elips", dimension=512)
except elips.ConfigError as e:
    print(e)  # Dimension mismatch: expected 1536, got 512

elips.open_with_config(path, config)

A lower-level alternative to elips.open() that accepts a pre-built Config object. Useful when you construct Config programmatically — for example, from a TOML file or environment variables — and want to avoid keyword argument forwarding.

import elips
from elips import Config, GraphParams

cfg = (
    Config()
    .dimension(768)
    .metric("cosine")
    .index("graph")
    .graph_params(GraphParams(max_connections=32, ef_construction=400, ef_search=100))
    .durability("standard")
    .metadata_acceleration(True)
)

db = elips.open_with_config("/data/embeddings.elips", cfg)

The function signature is:

elips.open_with_config(path: str, config: Config) -> Database

Database.vault(name)

Returns a Vault — a named namespace within the database that stores its own set of vectors and associated records. Vaults are created lazily: the first call to vault("name") for a given name creates and registers the vault; subsequent calls return the same object.

elips.Database.vault(name: str) -> elips.Vault

Parameters

• name — A non-empty string. Vault names are case-sensitive. The name is persisted and must remain stable across restarts.

Notes

• The vault registry is protected by the instance mutex, so concurrent calls to vault() from multiple threads are safe — the first caller creates the vault, subsequent callers receive the cached reference.
• All vaults within a database share the same dimension and metric (they are database-level identity fields).
• A vault with a given name that already exists on disk is loaded; a new name results in an empty vault being created in memory and flushed on first write.

db = elips.open("/data/shop.elips", dimension=384, metric="cosine")

products = db.vault("products")
reviews  = db.vault("reviews")
# Calling vault() again returns the same object
assert db.vault("products") is products

Database.list_vaults()

Returns the names of all vaults currently registered in the database, including those that exist on disk but have not yet been accessed in this process session.

Database.list_vaults() -> list[str]

names = db.list_vaults()
print(names)  # ['products', 'reviews', 'sessions']

Database.begin_transaction()

Returns a Transaction object that groups multiple mutations across one or more vaults into a single atomic operation. Prefer using it as a context manager so that rollback is automatic on error.

Database.begin_transaction() -> Transaction

See the dedicated Transaction reference for full details on commit, rollback, WAL framing, and crash-safety guarantees.

with db.begin_transaction() as txn:
    v = txn.vault("products")
    v.place([0.1] * 768, data={"sku": "ABC-001"})
    v.place([0.2] * 768, data={"sku": "ABC-002"})
# Committed atomically — both records appear or neither does

Database.query(eql, bindings=)

Executes an ELIPS Query Language (EQL) statement against the database and returns the results as a list of Result objects.

Database.query(eql: str, bindings: dict = {}) -> list[Result]

Parameters

• eql — A valid EQL statement string. EQL supports SEEK, SCAN, PLACE, ERASE, and FETCH operations with a SQL-inspired syntax.
• bindings — A mapping from placeholder names to Python values. Use $name placeholders in the EQL string and supply their values here to avoid injection vulnerabilities.

# Simple seek via EQL
results = db.query(
    "SEEK $vec TOP 10 IN products",
    bindings={"vec": embedding_vector},
)
for r in results:
    print(r.id, r.distance, r.data)

# EQL with a metadata filter
results = db.query(
    "SEEK $vec TOP 5 IN products WHERE category = $cat AND price < $max_price",
    bindings={
        "vec": embedding_vector,
        "cat": "electronics",
        "max_price": 999.0,
    },
)

# SCAN
records = db.query("SCAN products LIMIT 100")

Maintenance Operations

Database.checkpoint()

Flushes all pending WAL entries to the main data files, then truncates the WAL. This reduces WAL file size and shortens recovery time on the next open. Calling checkpoint() does not compact or rebuild the graph index.

Database.checkpoint() -> None

Database.compact()

Runs a full compaction pass: checkpoints the WAL, rewrites the data segment to remove dead space from erased records, and rebuilds the HNSW graph index for each vault whose tombstone ratio exceeds its configured compaction_ratio. This is an expensive, blocking operation — schedule it during off-peak periods.

Database.compact() -> None

Database.vacuum()

Reclaims disk space by physically removing the storage files for any vaults that have been deleted, and truncating free-list space in the main data file. Unlike compact(), it does not rebuild graph indexes. Typically run after a large number of erases.

Database.vacuum() -> None

Database.close()

Explicitly closes the database: flushes all pending writes, writes a checkpoint (for persistent databases), releases all file locks, and invalidates all Vault handles obtained from this instance. After calling close(), any further method call on theDatabase or its Vault objects raises RuntimeError.

You do not need to call close() explicitly if you use the database as a context manager, or if you allow normal Python garbage collection to destroy the object (the destructor checkpoints and closes).

db.close()

# Or, using Database as a context manager (preferred for scripts):
with elips.open("/data/vectors.elips", dimension=128, metric="cosine") as db:
    vault = db.vault("default")
    vault.place([0.5] * 128, data={"label": "test"})
# Checkpoint and close happen automatically here

Properties

Database.path: str

The filesystem path that was passed to open(). Returns ":memory:" for ephemeral databases.

Database.config: Config

The fully-resolved Config in effect for this database. Includes values read back from persisted metadata (so db.config.dimension_val is always accurate even if you did not pass dimension= at open time).

print(db.config.dimension_val)  # e.g. 768
print(db.config.metric_val)     # 'cosine'
print(db.config.index_val)      # 'graph'

Database.is_persistent: bool

True if the database is backed by disk storage (i.e., path != ":memory:" and durability is not "ephemeral"). Persistent databases are checkpointed on destruction.

Database.wal

Low-level access to the Write-Ahead Log handle. Exposes WAL position counters and flush statistics. Intended for monitoring and debugging, not normal application code. The exact API is an implementation detail and subject to change between minor versions.

Thread Safety

ELIPS has a layered concurrency model:

• Instance mutex — A single mutex guards the vault registry. Calls to vault() and list_vaults() lock this mutex briefly. It is not held during search or insert.
• Per-vault SharedMutex — Each Vault holds a reader-writer lock. Multiple threads can execute seek() simultaneously on the same vault without serialisation. Write operations (place(), erase(), committed transactions) acquire an exclusive write lock.
• Database itself is thread-safe — you may share one Database object across threads freely.
• Transaction is NOT thread-safe — do not share a Transaction between threads. Use one transaction per thread.

import threading, elips

db = elips.open("/data/vectors.elips", dimension=384, metric="cosine")
vault = db.vault("items")

def search_worker(query_vec):
    # Safe — concurrent seeks do not serialize
    results = vault.seek(query_vec, top=10)
    return results

threads = [threading.Thread(target=search_worker, args=([0.1]*384,)) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

Lifecycle & Destruction

When a Database object is garbage-collected or its __del__ runs, ELIPS performs the following sequence for persistent databases:

1. Acquire the instance mutex.
2. Flush all dirty in-memory pages and pending WAL records to disk.
3. Write a WAL checkpoint record.
4. Release all file locks.
5. Close memory-mapped file handles.

For ephemeral (":memory:" or durability="ephemeral") databases, the destructor simply discards all in-memory data — no disk I/O occurs.

Best practice: in long-running server processes, keep a single Database alive for the process lifetime. Avoid repeated open/close cycles — each open replays the WAL from disk, which adds latency proportional to WAL size.

Full Examples

Server application pattern

import elips
import numpy as np

# Module-level singleton — opened once at startup
db = elips.open(
    "/var/lib/myapp/vectors.elips",
    dimension=1536,
    metric="cosine",
    durability="standard",
    metadata_acceleration=True,
)

products = db.vault("products")
sessions = db.vault("sessions")

def index_product(sku: str, embedding: list[float], meta: dict) -> str:
    return products.place(embedding, data={"sku": sku, **meta})

def find_similar(embedding: list[float], n: int = 10):
    return products.seek(embedding, top=n)

# Periodic maintenance — e.g., called from a cron job
def nightly_maintenance():
    db.compact()

Ephemeral / test database

import elips

def test_vector_search():
    db = elips.open(":memory:", dimension=4, metric="cosine")
    v = db.vault("test")
    v.place([1.0, 0.0, 0.0, 0.0], data={"label": "x-axis"})
    v.place([0.0, 1.0, 0.0, 0.0], data={"label": "y-axis"})
    results = v.seek([1.0, 0.1, 0.0, 0.0], top=1)
    assert results[0].data["label"] == "x-axis"
    db.close()

Read-only replica

# A secondary process can open the same path read-only
reader = elips.open(
    "/var/lib/myapp/vectors.elips",
    access_mode="read_only",
)
results = reader.vault("products").seek(query_vec, top=5)