elips/docs
Reference · Python

Arena

Arena is the typed high-level wrapper around a single ELIPS vault. Obtain one via engine.arena(name). It handles writing, searching, fetching, scanning, and deleting records, and automatically routes between native C++ text embedding and Python-side embedding depending on how the database was configured.

Overview

An Arena wraps exactly one Vault. All operations go through the same WAL-backed record store and the same HNSW or exact index. The arena adds:

  • Typed RecordInput / Row / Hit data classes instead of raw dicts.
  • Automatic batch embedding: records without an explicit vector are embedded in a single batch call before insertion.
  • Transparent routing between Vault.seek_text (native embedder path) and Vault.seek_hybrid (Python embedder fallback).
  • Column-oriented bulk ingestion via ingest() for compatibility with pipeline-style data frames.
python
import elips

with elips.connect(":memory:", dimension=2) as engine:
    arena = engine.arena("articles")

    # Write records
    keys = arena.write_many([
        elips.RecordInput(text="The quick brown fox", meta={"lang": "en"}),
        elips.RecordInput(text="Le renard brun rapide", meta={"lang": "fr"}),
    ])

    # Vector search
    hits = arena.probe([1.0, 0.0], top=5)

    # Text search (uses native embedder)
    hits = arena.probe_text("quick fox", top=3)

    # Fetch by key
    rows = arena.pull(keys, include_vectors=True)
    print(rows[0].text)   # The quick brown fox

    # Delete
    removed = arena.discard(keys[:1])
    print(removed)        # 1

Properties

arena.name → str

The vault name this arena was opened against.

python
arena = engine.arena("documents")
print(arena.name)  # documents

arena.raw → Vault

The underlying low-level Vault handle. Drop to this when you need a capability the arena does not expose — for example, vault.place_many(), vault.info(), or direct EQL targeting.

python
arena = engine.arena("documents")
info = arena.raw.info()
print(info.dimension, info.metric)  # 128  cosine

arena.read_only → bool

True if this arena currently refuses writes — either because the database was opened with access_mode="read_only" or because arena.freeze(True) was called.

arena.sealed → bool

True once the owning engine has been closed. Writes to a sealed arena raise elips.StorageError.

arena.pending_removals → int

Number of deleted records that have been tombstoned in the HNSW graph but not yet reclaimed. Tombstones act as routing waypoints so live neighbours stay reachable, but they consume memory and widen the search beam. Call arena.vacuum() to reclaim them immediately, or let the arena auto-compact when they cross its compaction_ratio (default 0.2).

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("documents")
keys = [arena.write(vector=[float(i), 1.0]) for i in range(20)]
arena.discard(keys[:2])           # 2 / 20 = 0.10 — below auto-compact
print(arena.pending_removals)     # 2
engine.close()

count() → int

Return the number of live (non-tombstoned) records. Does not count deleted records awaiting compaction.

python
arena = engine.arena("documents")
print(arena.count())   # 0
arena.write(vector=[1.0, 0.0])
print(arena.count())   # 1

write() → str

python
arena.write(
    record: RecordInput | dict | None = None,
    /,
    *,
    vector: Sequence[float] | None = None,
    text: str | None = None,
    meta: dict | None = None,
    key: str | None = None,
    document: DocumentAttachment | None = None,
    chunk: ChunkInfo | None = None,
    lineage: EmbeddingLineage | None = None,
) -> str

Write a single record. Returns the assigned record key (UUIDv7 hex unless key was supplied).

You can pass either a positional structured record (RecordInput or a dict with the right shape) or keyword arguments — never both.

  • Vector path: supply vector. If text or document is also supplied, the text is stored as a DocumentAttachment for hybrid search but the embedding is taken from vector.
  • Text path: supply text (or document with text). The arena embeds via the native embedder if present, otherwise via the configured Python embedder.
  • key — Optional caller-supplied identifier. Must be a valid ELIPS ID (use elips.generate_id() if you need a pre-allocated key).
  • document — An elips.DocumentAttachment that carries text, a URI, and a MIME type. Use this when you want to attach a URI or a non-text/plain MIME type; otherwise a plain text="..." is sufficient.
  • chunk — A ChunkInfo describing where in a parent document this chunk came from.
  • lineage — An EmbeddingLineage recording the embedding provider and model.
python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")

# Keyword form — most common
key = arena.write(text="Alpha design note", meta={"kind": "design"})

# Structured form
record = elips.RecordInput(
    vector=[1.0, 0.0],
    text="Beta runbook",
    meta={"kind": "ops"},
)
key = arena.write(record)

# Dict form (same as RecordInput.from_mapping)
key = arena.write({"text": "Gamma spec", "meta": {"kind": "spec"}})

# With explicit document attachment (URI + MIME type)
doc = elips.DocumentAttachment(
    text="# Proposal

See attached.",
    uri="proposals/q4.md",
    mime_type="text/markdown",
)
chunk = elips.ChunkInfo()
chunk.document_key = "doc-q4"
chunk.ordinal = 0
chunk.char_start = 0
chunk.char_end = 27
key = arena.write(vector=[0.8, 0.2], document=doc, chunk=chunk)

engine.close()

write_many() → list[str]

python
arena.write_many(
    records: Sequence[RecordInput | dict],
) -> list[str]

Write a batch of records. Returns assigned keys in input order. At least one record is required.

Records without a vector are collected, their texts are passed to the embedder in a single batch call, and the resulting vectors are distributed back. This avoids per-record embedding overhead. Records that already have a vector bypass the embedder entirely.

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")

keys = arena.write_many([
    elips.RecordInput(text="Alpha",    meta={"order": 1}),
    elips.RecordInput(text="Beta",     meta={"order": 2}),
    elips.RecordInput(vector=[1.0, 0.0], meta={"order": 3}),
    {"text": "Delta", "meta": {"order": 4}},   # dict form
])

print(len(keys))   # 4
engine.close()

Performance note: for bulk imports, prefer write_many() or ingest() over repeated write() calls. The embedder batch is amortized across the whole list, and the WAL absorbs appends efficiently.

ingest() / merge() → list[str]

python
# Structured form — same as write_many
arena.ingest(records: Sequence[RecordInput | dict]) -> list[str]

# Column-oriented form — pipeline / dataframe style
arena.ingest(
    *,
    vectors:   Sequence[Sequence[float] | None] | None = None,
    texts:     Sequence[str | None] | None = None,
    meta:      Sequence[dict | None] | None = None,
    keys:      Sequence[str | None] | None = None,
    documents: Sequence[DocumentAttachment | None] | None = None,
    chunks:    Sequence[ChunkInfo | None] | None = None,
    lineages:  Sequence[EmbeddingLineage | None] | None = None,
) -> list[str]

ingest() is the column-oriented bulk ingestion API. It accepts either a list of structured records (identical to write_many()) or parallel column sequences. All column sequences must have the same length.

merge() is a compatibility alias for ingest() with identical semantics.

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")

# Column-oriented — natural for data-pipeline output
keys = arena.ingest(
    texts=["Alpha note", "Beta note", "Gamma note"],
    meta=[{"category": "A"}, {"category": "B"}, {"category": "C"}],
)
print(len(keys))   # 3

# Column-oriented with explicit vectors for some rows
keys = arena.ingest(
    vectors=[[1.0, 0.0], None, [0.0, 1.0]],
    texts=[None, "only text", None],
    meta=[{"v": True}, {"t": True}, {"v": True}],
)
# Row 1: vector provided directly
# Row 2: embedded via configured embedder
# Row 3: vector provided directly

engine.close()

When to prefer ingest() over write_many(): when your upstream pipeline already produces column-oriented outputs (e.g., NumPy arrays, Pandas DataFrames sliced into lists). The structured write_many() form is recommended for new code because it keeps related fields together and is easier to type-check.

probe() → list[Hit]

python
arena.probe(
    vector: Sequence[float],
    *,
    top: int = 10,
    where: Filter | None = None,
    max_distance: float | None = None,
    include_vectors: bool = False,
) -> list[Hit]

Approximate nearest-neighbour search. Returns hits sorted by distance (ascending).

  • vector — Query vector. Must match the database dimension.
  • top — Maximum number of hits returned.
  • where — Optional Filter to narrow the candidate set.
  • max_distance — Hard upper bound on distance. Hits beyond this threshold are dropped even if fewer than top remain.
  • include_vectors — Whether to hydrate the stored embedding on each hit. Incurs one record-store fetch per hit; leave False unless you need the vectors.
python
import elips

engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
arena.write(vector=[1.0, 0.0], text="Alpha", meta={"kind": "a"})
arena.write(vector=[0.0, 1.0], text="Beta",  meta={"kind": "b"})

# Plain ANN
hits = arena.probe([1.0, 0.0], top=1)
print(hits[0].text)      # Alpha
print(hits[0].distance)  # ≈ 0.0

# Filtered ANN
f = elips.Filter().field("kind").equals("b")
hits = arena.probe([0.0, 1.0], top=5, where=f)
print(len(hits))         # 1
print(hits[0].text)      # Beta

# With distance threshold
hits = arena.probe([1.0, 0.0], top=10, max_distance=0.5)

# With stored vectors hydrated
hits = arena.probe([1.0, 0.0], top=1, include_vectors=True)
print(hits[0].vector)    # (1.0, 0.0)

engine.close()

probe_text() → list[Hit]

python
arena.probe_text(
    text: str,
    *,
    top: int = 10,
    where: Filter | None = None,
    max_distance: float | None = None,
    include_vectors: bool = False,
    lexical_weight: float = 0.25,
) -> list[Hit]

Text-first retrieval. The call routes itself based on the database configuration:

  • Native text embedder present (i.e., the database was opened with the built-in local embedder or a LocalEmbedderConfig): uses Vault.seek_text() — the C++ core embeds the query and runs ANN in one step.
  • Python embedder configured: embeds the query with the Python callable and then calls Vault.seek_hybrid() with the lexical_weight blending factor.
  • Neither: raises ValueError. ELIPS never silently falls back to lexical-only retrieval.
python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
arena.write(text="The quick brown fox", meta={"lang": "en"})
arena.write(text="Le renard brun rapide", meta={"lang": "fr"})

# Text-first, no filter
hits = arena.probe_text("quick fox", top=5)
print(hits[0].text)          # The quick brown fox

# Filtered by language
f = elips.Filter().field("lang").equals("fr")
hits = arena.probe_text("renard", top=3, where=f)
print(hits[0].meta["lang"])  # fr

engine.close()

lexical_weight is only used on the Python-embedder path. It controls how much the BM25-style lexical overlap score contributes to the fused ranking. 0.0 = pure ANN; 1.0 = pure lexical; the default 0.25 gives a modest lexical boost.

probe_hybrid() → list[Hit]

python
arena.probe_hybrid(
    vector: Sequence[float],
    text: str,
    *,
    top: int = 10,
    where: Filter | None = None,
    max_distance: float | None = None,
    lexical_weight: float = 0.25,
    include_vectors: bool = False,
) -> list[Hit]

Explicit hybrid retrieval: you supply both the query vector and the query text. The core fuses ANN distance with lexical overlap from stored documents according to lexical_weight. Use this when you have already embedded the query and want control over the blend rather than leaving routing to probe_text().

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
arena.write(vector=[1.0, 0.0], text="Alpha design note")
arena.write(vector=[0.0, 1.0], text="Beta security note")

hits = arena.probe_hybrid(
    [1.0, 0.0],    # query vector
    "alpha",       # query text for lexical overlap
    top=5,
    lexical_weight=0.3,
)
print(hits[0].text)   # Alpha design note
engine.close()

explain() → QueryPlan

python
arena.explain(
    vector: Sequence[float],
    *,
    top: int = 10,
    where: Filter | None = None,
    max_distance: float | None = None,
    has_text_component: bool = False,
) -> QueryPlan

Return the planner's decision for a hypothetical query without executing it. Useful for debugging filter and index interactions.

The returned QueryPlan exposes:

  • strategy — One of ann_index, exact_candidates, full_scan, text_probe, hybrid_fusion.
  • metadata_accelerated — Whether the MetadataIndex was used to narrow candidates.
  • candidate_count — Estimated pre-filter candidate set size.
python
engine = elips.connect(":memory:", dimension=2, metadata_acceleration=True)
arena = engine.arena("docs")
for i in range(50):
    arena.write(vector=[float(i), 1.0], meta={"group": i % 5})

f = elips.Filter().field("group").equals(0)
plan = arena.explain([1.0, 0.0], top=10, where=f, has_text_component=False)
print(plan.strategy.name)           # ann_index or exact_candidates
print(plan.metadata_accelerated)    # True
print(plan.candidate_count)         # ≈ 10 (5 * 10 / 5 groups)
engine.close()

pull() → list[Row]

python
arena.pull(
    keys: Sequence[str],
    *,
    include_vectors: bool = True,
) -> list[Row]

Fetch records by key and return typed Row objects. Keys that no longer exist (deleted or never written) are silently skipped — the returned list may be shorter than keys.

By default include_vectors=True — the stored embedding is included in each row. Pass False if you only need metadata and text.

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")

k1 = arena.write(text="Alpha", meta={"order": 1})
k2 = arena.write(vector=[0.0, 1.0], meta={"order": 2})

rows = arena.pull([k1, k2])
print(rows[0].text)             # Alpha
print(rows[1].vector)           # (0.0, 1.0)

# Missing key is silently skipped
rows = arena.pull([k1, "nonexistent-key"])
print(len(rows))                # 1

engine.close()

sweep() → list[Row]

python
arena.sweep(
    *,
    where: Filter | None = None,
    offset: int = 0,
    limit: int | None = None,
    include_vectors: bool = False,
) -> list[Row]

Full scan of the arena, optionally filtered. Returns Row objects in storage order (UUIDv7 insert order approximately). Use for export, reindexing, or auditing — not as a substitute for probe().

  • where — Optional filter to apply during the scan.
  • offset / limit — Pagination. limit=None returns all matching records.
  • include_vectors — Hydrate stored vectors. Expensive for large arenas.
python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
for i in range(100):
    arena.write(vector=[float(i), 0.0], meta={"page": i // 10})

# All records
rows = arena.sweep()
print(len(rows))    # 100

# Paginated
page1 = arena.sweep(offset=0, limit=10)
page2 = arena.sweep(offset=10, limit=10)

# Filtered
f = elips.Filter().field("page").equals(0)
rows = arena.sweep(where=f)
print(len(rows))    # 10

engine.close()

discard() → int

python
arena.discard(
    keys: Sequence[str] | None = None,
    *,
    where: Filter | None = None,
) -> int

Delete records by key, metadata filter, or both. Returns the count of records actually removed. At least one of keys or where must be provided.

Deletes are tombstone operations — the node remains in the graph as a routing waypoint until the arena compacts. Live neighbours are unaffected; recall is maintained by a widened search beam.

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
keys = [arena.write(vector=[float(i), 0.0], meta={"group": i % 3}) for i in range(9)]

# Delete by key
removed = arena.discard([keys[0], keys[1]])
print(removed)   # 2

# Delete by filter
f = elips.Filter().field("group").equals(2)
removed = arena.discard(where=f)
print(removed)   # 3  (indices 2, 5, 8)

# Delete by both (union, no double-counting)
removed = arena.discard([keys[3]], where=elips.Filter().field("group").equals(1))
print(removed)   # up to 4 (key 3 + group-1 records 4 and 7)

engine.close()

Common mistake: calling discard() with neither keys nor where raises ValueError — there is no "delete all" shorthand. To clear an arena, sweep for all keys first.

Maintenance

arena.freeze(frozen=True) → None

Temporarily refuse writes on this arena without reopening the database read-only. Pass frozen=False to re-enable writes.

python
arena.freeze()           # reject subsequent writes
arena.freeze(False)      # allow writes again

arena.vacuum() → None

Reclaim graph nodes held by deleted records in this arena. The arena auto-compacts once tombstones pass the compaction_ratio (default 0.2). Call this after a bulk delete to free memory without waiting.

python
keys = [arena.write(vector=[float(i), 1.0]) for i in range(40)]
arena.discard(keys[:20])
print(arena.pending_removals)   # 20 (0.5 — above threshold, may already be 0)
arena.vacuum()
print(arena.pending_removals)   # 0

arena.rebuild() → None

Rebuild the HNSW index from the authoritative record store. Incremental inserts in arrival order yield a lower-quality graph than a single build over the full set. Call this after a large bulk load to maximise recall before opening to search traffic.

python
# Bulk load — insert-order graph quality is suboptimal
arena.write_many(large_batch)

# Rebuild — one pass over all records yields a better graph
arena.rebuild()

# Now compact the on-disk layout too
engine.compact()

health() → ArenaHealth

Return a point-in-time health snapshot. Useful for monitoring and capacity planning.

python
engine = elips.connect(":memory:", dimension=2)
arena = engine.arena("docs")
for i in range(10):
    arena.write(vector=[float(i), 1.0])
arena.discard([arena.sweep()[0].key])

health = arena.health()
print(health.name)              # docs
print(health.live)              # 9
print(health.pending_removals)  # 0 or 1 (auto-compacts at ratio)
print(health.tombstone_ratio)   # 0.0 – 0.1
print(health.dimension)         # 2
print(health.metric)            # cosine
print(health.read_only)         # False
print(health.sealed)            # False
engine.close()

See ArenaHealth for the full model reference.

Embedding resolution

When a record lacks an explicit vector, the arena resolves it in this priority order:

  1. Native text embedder — If engine.config.has_text_embedder is truthy, the core embeds via the C++ runtime. Vault.place_document() is called; no Python-side embedding occurs.
  2. Python embedder — If the arena or engine has a configured Python callable embedder, texts are batched and passed to it. The returned vectors are used to call Vault.place().
  3. Error — If neither is present, ValueError is raised. ELIPS never silently degrades to storing text without a vector.

Records with custom document metadata (a non-empty uri or non-text/plain MIME type) always require an explicit vector. The native place_document() path accepts only raw text, not a full DocumentAttachment.

Thread safety

Arena itself carries no locks. All thread safety is provided by the underlying C++ Vault and Database, which serialize concurrent writes. Concurrent reads and writes from multiple threads through the same Arena are safe. If you call a Python embedder from multiple threads, ensure the callable is itself thread-safe.

← PreviousEngine
Edit on GitHub →
Next →Data models