Engine

Engine is the high-level entry point for the modern ELIPS Python API. It wraps a low-level Database handle and adds typed arena management, WAL introspection, and an idiomatic context-manager protocol. Most applications should open a database via elips.connect() rather than constructing Engine directly.

Overview

ELIPS ships two Python surfaces over the same C++ core. The low-level surface (open / Database / Vault) mirrors the runtime exactly. The modern surface (connect / Engine / Arena) layers typed, text-first ergonomics on top. Engine is the bridge: it holds the Database handle, carries an optional default embedder, and manufactures Arena wrappers on demand via engine.arena(name).

Vaults (arenas) are created lazily — calling engine.arena("documents") is sufficient to bring a vault into existence the moment the first record is written to it. No explicit schema creation step is required.

import elips

# Minimal in-memory database
with elips.connect(":memory:", dimension=128) as engine:
    arena = engine.arena("documents")
    key = arena.write(text="Hello, ELIPS!", meta={"source": "readme"})
    hits = arena.probe_text("Hello", top=3)
    print(hits[0].text)   # Hello, ELIPS!

Constructor

class Engine:
    def __init__(
        self,
        db: Database,
        *,
        default_embedder: Embedder | None = None,
    ) -> None: ...

Direct construction is rarely needed. Prefer elips.connect() or elips.connect_with_config() which build the underlying Config, open the database, and return a fully configured Engine.

• db — An open elips.Database handle obtained via elips.open() or elips.open_with_config().
• default_embedder — Optional Python batch embedder (a callable matching the Embedder protocol). Passed down to every arena that does not supply its own override. Used when the database has no native text embedder configured.

connect()

def connect(
    path: str,
    *,
    dimension: int = 0,
    metric: "cosine" | "euclidean" | "dot_product" = "cosine",
    index: "graph" | "exact" = "graph",
    access_mode: "read_write" | "read_only" = "read_write",
    segmented_storage: bool = True,
    metadata_acceleration: bool = True,
    embedder: Embedder | LocalEmbedderConfig | None = None,
    embedder_provider: str = "python",
    embedder_model: str = "callable",
    embedder_revision: str = "",
    use_default_text_embedder: bool = True,
    gpu: GpuConfig | None = None,
) -> Engine

The canonical way to open an ELIPS database with the modern API. Builds a Config, opens the database, attaches the embedder, and returns a ready-to-use Engine.

• path — Filesystem directory path or ":memory:" for a transient in-memory database.
• dimension — Vector dimension for new databases. Existing persistent databases restore their dimension from the manifest automatically; passing 0 is safe. In-memory databases always need a non-zero dimension.
• metric — Similarity metric: "cosine" (default), "euclidean", or "dot_product".
• index — Index backend: "graph" (HNSW, default) or "exact" (brute-force).
• access_mode — "read_write" acquires an exclusive advisory lock; "read_only" takes a shared lock and refuses all mutations.
• segmented_storage — Whether to use the segmented persistence layout (elips.manifest + per-vault segment files). Almost always True.
• metadata_acceleration — Enables the MetadataIndex for equality and set-membership filters. Greatly speeds up filtered searches at a small memory cost.
• embedder — A Python callable or a LocalEmbedderConfig. When a callable is supplied, metadata about it is persisted but the callable itself is not. Reopening the database without the same callable leaves text-first calls raising ValueError.
• embedder_provider / embedder_model / embedder_revision — Metadata stored alongside a Python callable embedder for traceability in EmbeddingLineage.
• use_default_text_embedder — When True(default), a new database automatically provisions the built-in local text embedder. Set to False when you supply your own embedder or want a vector-only database.
• gpu — Optional GpuConfig for GPU-accelerated index builds. Requires an ELIPS build with GPU support.

import elips

# Persistent, cosine, HNSW — typical production setup
engine = elips.connect(
    "/var/lib/myapp/vectors",
    dimension=768,
    metric="cosine",
    index="graph",
)

# In-memory — tests and notebooks
engine = elips.connect(":memory:", dimension=128)

# Bring your own embedder (sentence-transformers example)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def embed(texts):
    return model.encode(texts, normalize_embeddings=True).tolist()

engine = elips.connect(
    "/var/lib/myapp/vectors",
    dimension=384,
    embedder=embed,
    embedder_model="all-MiniLM-L6-v2",
    use_default_text_embedder=False,
)

engine.close()

connect_with_config()

def connect_with_config(
    path: str,
    config: Config,
    *,
    embedder: Embedder | LocalEmbedderConfig | None = None,
    embedder_provider: str = "python",
    embedder_model: str = "callable",
    embedder_revision: str = "",
) -> Engine

Use this when you need fine-grained control over Config options that connect() does not expose directly — for example, a LocalEmbedderConfig with a custom model path, or GPU tuning parameters that go into the config builder.

If config already contains a text embedder, embedder is ignored unless it is a non-local callable, in which case it becomes the runtime embedder for Python-side embedding fallback.

import elips

config = (
    elips.Config()
    .dimension(768)
    .metric("cosine")
    .segmented_storage(True)
    .metadata_acceleration(True)
    .auto_text_embedder(False)   # we bring our own
)

engine = elips.connect_with_config(
    "/var/lib/myapp/vectors",
    config,
    embedder=embed,
    embedder_model="all-MiniLM-L6-v2",
)
engine.close()

Properties

engine.raw → Database

Returns the underlying low-level Database handle. Useful when you need a capability that Engine does not wrap, such as db.begin_transaction(), db.query(eql), or db.gpu_info().

engine = elips.connect(":memory:", dimension=2)

# Drop to low-level for a transaction
with engine.raw.begin_transaction() as txn:
    txn.vault("logs").place([1.0, 0.0], {"msg": "start"})
    txn.vault("logs").place([0.0, 1.0], {"msg": "end"})

engine.close()

engine.config → Config

Returns the effective Config as resolved by the runtime — including persisted dimension, metric, index type, and embedder metadata. Read-only; modifying the returned object has no effect.

engine = elips.connect(":memory:", dimension=128, metric="euclidean")
cfg = engine.config
print(cfg.dimension_val)          # 128
print(cfg.metric_val)             # euclidean
print(cfg.has_text_embedder)      # True (default embedder attached)
engine.close()

arena()

engine.arena(
    name: str,
    *,
    embedder: Embedder | None = None,
    text_slot: str = "__elips_text__",
) -> Arena

Create a typed Arena wrapper for the named vault. The vault is created lazily — it materialises on the first write, not on this call.

• name — Vault name. Any string is valid; convention is lowercase with hyphens (e.g. "documents", "product-chunks").
• embedder — Arena-level embedder override. Takes precedence over the engine's default_embedder. Useful when different arenas embed in different vector spaces or use different models.
• text_slot — Reserved backward-compat argument. The current runtime stores text on DocumentAttachment rather than mirroring it into metadata. Leave as default.

engine = elips.connect(":memory:", dimension=2)

# Two arenas, same database
docs   = engine.arena("documents")
images = engine.arena("image-captions")

docs.write(text="A design document", meta={"kind": "design"})
images.write(vector=[0.5, 0.5], meta={"caption": "hero banner"})

print(engine.vault_names())   # ['documents', 'image-captions']
engine.close()

Lifecycle methods

engine.checkpoint() → None

Flush the current in-memory database state to durable storage. Writes the manifest and segment files (or a snapshot) and truncates the WAL. After a checkpoint, engine.pending_writes() returns an empty list.

close() always checkpoints before releasing locks, so manual checkpointing is only needed for long-running write sessions where you want to reduce WAL size or guarantee durability mid-session.

import tempfile, elips

path = tempfile.mkdtemp()
engine = elips.connect(path, dimension=2)
arena = engine.arena("docs")
for i in range(1000):
    arena.write(vector=[float(i), 0.0])

engine.checkpoint()           # flush now; WAL is truncated
print(engine.pending_writes())   # []
engine.close()

engine.compact() → None

Rebuild every vault index from scratch and then checkpoint. Compaction produces a higher-quality HNSW graph than incremental inserts, which trade quality for throughput. Run after a large bulk load to recover optimal recall.

Compaction is CPU-intensive and holds the write lock for its full duration. Schedule it during low-traffic windows.

# After a large bulk import:
keys = arena.ingest(texts=corpus_texts, meta=corpus_meta)
engine.compact()   # rebuild index, then checkpoint

engine.vacuum() → None

Reclaim index space held by deleted records across every arena. Deletes leave tombstones in the HNSW graph so that live neighbours remain reachable; the index widens its beam to bypass them. Tombstones are reclaimed automatically once they reach the arena's compaction_ratio (default 0.2), but after a bulk delete it is worth reclaiming immediately.

Unlike compact(), vacuum() does not rewrite the on-disk snapshot and works on in-memory databases.

engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("documents")
keys = [arena.write(vector=[float(i), 1.0]) for i in range(50)]

# Delete 30 records
arena.discard(keys[:30])
print(arena.pending_removals)   # may be up to 30

engine.vacuum()
print(arena.pending_removals)   # 0
engine.close()

engine.close() → None

Checkpoint, release the cross-process advisory lock, and seal every arena. After close(), writes to any arena raise elips.StorageError rather than silently failing to persist. Calling close() more than once is safe (idempotent).

engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("documents")
arena.write(vector=[1.0, 0.0])
engine.close()

# Further writes raise StorageError:
try:
    arena.write(vector=[0.0, 1.0])
except elips.StorageError as exc:
    print("sealed:", exc)

vault_names() → list[str]

Return the names of every vault that currently exists in the database. An arena that has been obtained via engine.arena() but never written to does not appear here — vaults are created on first write.

engine = elips.connect(":memory:", dimension=2)
print(engine.vault_names())   # []

engine.arena("alpha").write(vector=[1.0, 0.0])
engine.arena("beta").write(vector=[0.0, 1.0])
print(engine.vault_names())   # ['alpha', 'beta']
engine.close()

pending_writes() → list[WalRecord]

Read the database's write-ahead log without mutating anything. Returns every acknowledged record in log order. Transaction markers are resolved during replay and never surface here; records inside an unterminated transaction are omitted; a corrupt tail is dropped silently.

Returns an empty list for ":memory:" databases (no WAL file) and after checkpoint() (WAL is truncated). Each item is a WalRecord.

import tempfile, elips

path = tempfile.mkdtemp()
engine = elips.connect(path, dimension=2)
arena = engine.arena("docs")
k1 = arena.write(vector=[1.0, 0.0], meta={"rev": 1})
k2 = arena.write(vector=[0.5, 0.5], meta={"rev": 2})

records = engine.pending_writes()
print(len(records))                  # 2
print(records[0].op)                 # insert
print(records[0].arena)              # docs
print(records[0].key == k1)          # True

# After checkpoint, WAL is empty
engine.checkpoint()
print(engine.pending_writes())       # []
engine.close()

This is the same data that elips.replay_wal() reads at the file level, wrapped in typed WalRecord objects.

Context manager

Engine implements __enter__ and __exit__, making it safe to use with Python's with statement. __exit__ always calls close() regardless of whether the block raised an exception.

import elips

# The context manager is the idiomatic production pattern
with elips.connect("/var/lib/myapp/vectors", dimension=768) as engine:
    arena = engine.arena("documents")
    arena.write_many([
        elips.RecordInput(text="First doc",  meta={"id": 1}),
        elips.RecordInput(text="Second doc", meta={"id": 2}),
    ])
    hits = arena.probe_text("first", top=5)
    for h in hits:
        print(h.key, h.distance, h.text)
# engine.close() is called automatically here

For long-lived server processes, hold the Engine for the lifetime of the process and call close() in a shutdown hook, rather than using a context manager.

Thread safety

Engine itself is a thin Python wrapper and carries no thread-local state. The underlying C++ Database serializes concurrent writes via an internal mutex. Concurrent reads are safe across threads. Concurrent writes from multiple Python threads to the same Engine are safe but may contend on the C++ mutex.

For multi-process deployments: only one process may hold the read_write lock at a time. Use access_mode="read_only" in reader processes (see Connect guide).

import threading, elips

engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("shared")

def worker(i: int) -> None:
    arena.write(vector=[float(i), 0.0], meta={"worker": i})

threads = [threading.Thread(target=worker, args=(i,)) for i in range(8)]
for t in threads: t.start()
for t in threads: t.join()

print(arena.count())  # 8
engine.close()

Common mistakes

• Writing after close. Obtaining an Arena and keeping a reference to it after engine.close() will cause the next write to raise StorageError. Always close the engine after all writes are done.
• Forgetting dimension for in-memory databases. elips.connect(":memory:") requires a non-zero dimension every time it is called — there is no manifest to restore from.
• Multiple read-write openers. Opening the same database directory with two "read_write" processes raises LockConflict. Readers should use access_mode="read_only".
• Calling arena() but never writing. An arena obtained via engine.arena("name") that never receives a write does not appear in vault_names(). The vault is created on first write.
• Reopening without a callable embedder. If the database was opened with a Python callable embedder, reopening it without supplying the same callable leaves probe_text and write (text-only) raising ValueError.