Overview

Maintaining the performance and health of your ELIPS database is critical for long-running deployments. This page documents the APIs available for managing vault and database maintenance in Python, including managing HNSW tombstones, reclaiming disk space, and monitoring the lifecycle of your database.

HNSW Compaction & Tombstones

When you delete a record in ELIPS using Vault.erase(), the node is not immediately removed from the HNSW graph to avoid heavy lock contention. Instead, it is marked as a tombstone.

The search algorithm uses an adaptive ef beam: when tombstones build up, search() dynamically scales the search width by the ratio of total nodes to live nodes. This ensures that the search beam still yields enough live hits, allowing recall to stay within 0.15 of baseline even at a 50% delete ratio, while the graph size remains bounded.

You can check how many tombstones are pending removal using the Vault.pending_removals property.

Vacuum

To reclaim the space held by tombstones in the index, use vacuum(). It performs a self-compaction of the graph index without rewriting the on-disk snapshot, making it cheap and safe to use even on in-memory databases.

Vault.vacuum() — Vacuums a specific vault. It's a cheap no-op if nothing is tombstoned.
Database.vacuum() — Reclaims tombstoned index space across every vault in the database.

python

import elips

db = elips.open("/data/vec", dimension=128)
vault = db.vault("docs")

# Ingest then delete a batch
ids = [vault.place([0.0]*128, {"i": i}) for i in range(1000)]
for rid in ids[:200]:
    vault.erase(rid)

# Check tombstone pressure
print(vault.pending_removals)  # 200

# Vacuum: cheap, works on in-memory DBs too
vault.vacuum()
print(vault.pending_removals)  # 0

# Database-level vacuum across all vaults
db.vacuum()

Rebuild Index

Unlike vacuum(), Vault.rebuild_index() completely reconstructs the backing index from stored records. It is more expensive but creates a perfectly fresh HNSW graph without the fragmentation that might accumulate after millions of updates.

Graph Params

The compaction_ratio parameter controls the tombstone fraction that triggers auto-compaction. By default, it is set to 0.2 (20%). Setting it to 0.0 disables auto-compaction.

python

# GraphParams with compaction_ratio
params = elips.GraphParams(
    max_connections=16,
    ef_construction=200,
    ef_search=50,
    compaction_ratio=0.15,  # compact when 15% tombstoned
)
config = elips.Config().dimension(128).graph_params(params)
db = elips.open_with_config("/data/vec", config)

Read-Only Mode & Sealing

You can explicitly control the mutability of your vaults:

Vault.read_only: Returns True when the vault is in read-only mode and refuses mutations.
Vault.set_read_only(read_only: bool): Toggles runtime read-only mode. Further mutations will raise a StorageError.
Vault.sealed: Returns True once the owning database is closed. Writes to a sealed vault will immediately raise an error rather than silently failing to persist data to memory that will never be checkpointed.

Records Snapshot

You can snapshot every stored record in a vault using Vault.records(), which returns a copy of the live map (safely guarded by a mutex) as a list of StoredRecord (a TypedDict).

python

records = vault.records()  # list[StoredRecord]
print(len(records))  # 800

Note: For large vaults, prefer using scan() with a filter and limit instead to avoid copying the entire dataset into memory at once.

Database Lifecycle

The Database object exposes several maintenance and lifecycle methods:

Database.compact() — Rebuilds every vault index from scratch and checkpoints the state. Ideal for periodic cleanup on on-disk databases (unlike vacuum, it does not work on in-memory DBs).
Database.checkpoint() — Flushes current state and WAL to disk for durability.
Database.close() — Triggers a final checkpoint, releases locks, and seals every vault.

Lifecycle properties:

Database.path — The filesystem directory or ":memory:" for transient databases.
Database.persistent — Returns False for in-memory databases.
Database.closed — Returns True once close() or abandon() has run.

Modern Engine Maintenance

The modern Engine API exposes analogous maintenance tools:

Engine.vacuum() — Vacuums all arenas.
Engine.vault_names() -> list[str] — Lists all vault/arena names.
Engine.pending_writes() -> list[WalRecord] — Returns all pending WAL entries.

Arena Health

For detailed diagnostics, Arena.health() returns an ArenaHealth TypedDict which includes metrics like the tombstone_ratio: float. This is useful for building operational dashboards and determining when to trigger a manual vacuum.

Operational Patterns

For production applications, follow this checklist to ensure consistent performance:

Monitor tombstones: Periodically check Vault.pending_removals or Arena.health()['tombstone_ratio'].
Vacuum proactively: Call vacuum() when the tombstone_ratio exceeds your configured compaction_ratio threshold, or rely on auto-compaction.
Use checkpoints: Periodically call checkpoint() in long-running serving loops to ensure regular durability without pausing queries.
Scheduled compactions: Call compact() before planned long downtime windows to flush the WAL and completely rebuild indices on disk.

Common Pitfalls

Never call Vault.rebuild_index() on a hot vault under active concurrent search load unless you have a second replica serving the traffic.
Don't expect compact() to work on an in-memory database; use vacuum() instead.
Writing to a vault after calling Database.close() will raise an error because the vault is sealed. Be sure to stop ingest traffic before closing the database.

See also: Python SDK, Algorithms, Storage.