Overview

Maintaining the performance and health of your ELIPS database is critical for long-running deployments. This page documents the APIs available for managing collection and client maintenance in Python, including managing HNSW tombstones, reclaiming disk space, and monitoring the lifecycle of your database.

HNSW Compaction & Tombstones

When you delete a record in ELIPS using collection.delete(), the node is not immediately removed from the HNSW graph to avoid heavy lock contention. Instead, it is marked as a tombstone.

The search algorithm uses an adaptive ef beam: when tombstones build up, query() dynamically scales the search width by the ratio of total nodes to live nodes. This ensures that the search beam still yields enough live hits, allowing recall to stay within 0.15 of baseline even at a 50% delete ratio, while the graph size remains bounded.

You can check how many tombstones are pending removal using the collection.pending_removals property.

Vacuum

To reclaim the space held by tombstones in the index, use vacuum(). It performs a self-compaction of the graph index without rewriting the on-disk snapshot, making it cheap and safe to use even on in-memory databases.

collection.vacuum() — Vacuums a specific collection. It's a cheap no-op if nothing is tombstoned.
client.vacuum() — Reclaims tombstoned index space across every collection in the database.

python

import elips

client = elips.PersistentClient("/data/vec", dimension=128)
docs = client.get_or_create_collection("docs")

# Ingest then delete a batch
ids = [docs.add(embedding=[0.0] * 128, metadata={"i": i}) for i in range(1000)]
docs.delete(ids[:200])

# Check tombstone pressure
print(docs.pending_removals)  # 200

# Vacuum: cheap, works on in-memory DBs too
docs.vacuum()
print(docs.pending_removals)  # 0

# Client-level vacuum across all collections
client.vacuum()

Rebuild

Unlike vacuum(), collection.rebuild() completely reconstructs the backing index from stored records. It is more expensive but creates a perfectly fresh HNSW graph without the fragmentation that might accumulate after millions of updates.

For quantized collections, collection.compress() (alias train) trains the codebook from the stored vectors so compressed search can be served.

HNSW Params

The compaction_ratio parameter controls the tombstone fraction that triggers auto-compaction. By default, it is set to 0.2 (20%). Setting it to 0.0 disables auto-compaction.

python

# HnswParams with compaction_ratio
params = elips.HnswParams(
    max_connections=16,
    ef_construction=200,
    ef_search=50,
    compaction_ratio=0.15,  # compact when 15% tombstoned
)
client = elips.PersistentClient("/data/vec", dimension=128, index="graph", hnsw=params)

Read-Only Mode & Sealing

You can explicitly control the mutability of your collections:

collection.read_only: Returns True when the collection is in read-only mode and refuses mutations.
collection.freeze(frozen=True): Freezes the collection so further mutations are rejected. Pass frozen=False to unfreeze.
collection.sealed: Returns True once the owning client is closed. Writes to a sealed collection immediately raise an error rather than silently failing to persist data to memory that will never be checkpointed.

Inspecting Records

To look at stored records without running a similarity search, use collection.peek(n) (a quick sample) or collection.get(...) (by ids or a where filter). Both return Row objects.

python

sample = docs.peek(10)        # list[Row]
print(len(sample))            # up to 10
print(docs.count())           # total live records

# Scan a filtered page without loading everything
rows = docs.get(where=elips.Filter().field("kind").equals("design"), limit=100)

Note: For large collections, prefer get() with a filter and limit instead of fetching every record, to avoid copying the entire dataset into memory at once.

Client Lifecycle

The Client / PersistentClient object exposes several maintenance and lifecycle methods:

client.compact() — Rebuilds every collection index from scratch and checkpoints the state. Ideal for periodic cleanup on on-disk databases (unlike vacuum, it does not work on in-memory DBs).
client.checkpoint() — Flushes current state and WAL to disk for durability.
client.close() — Triggers a final checkpoint, releases locks, and seals every collection.

Use the client as a context manager so it is always closed cleanly:

python

with elips.PersistentClient("/data/vec", dimension=128) as client:
    docs = client.get_or_create_collection("docs")
    # ... work ...
    client.checkpoint()
# close() runs automatically on exit

The low-level native handle is available via client.native, and the resolved settings via client.config.

Client Maintenance

The client exposes database-wide maintenance helpers:

client.vacuum() — Vacuums all collections.
client.collection_names() -> list[str] — Lists all collection names (alias list_collections()).
client.pending_writes() -> list[WalRecord] — Returns all pending WAL entries.

Collection Health

For detailed diagnostics, collection.health() returns a CollectionHealth dataclass which includes metrics like the tombstone_ratio property. This is useful for building operational dashboards and determining when to trigger a manual vacuum.

Operational Patterns

For production applications, follow this checklist to ensure consistent performance:

Monitor tombstones: Periodically check collection.pending_removals or collection.health().tombstone_ratio.
Vacuum proactively: Call vacuum() when the tombstone_ratio exceeds your configured compaction_ratio threshold, or rely on auto-compaction.
Use checkpoints: Periodically call checkpoint() in long-running serving loops to ensure regular durability without pausing queries.
Scheduled compactions: Call compact() before planned long downtime windows to flush the WAL and completely rebuild indices on disk.

Common Pitfalls

Never call collection.rebuild() on a hot collection under active concurrent search load unless you have a second replica serving the traffic.
Don't expect compact() to work on an in-memory database; use vacuum() instead.
Writing to a collection after calling client.close() will raise an error because the collection is sealed. Be sure to stop ingest traffic before closing the client.

See also: Python SDK, Algorithms, Storage.