Recovery & Introspection

Overview

This page documents the advanced diagnostic and utility APIs available in the ELIPS Python client. These APIs allow you to replay the write-ahead log (WAL) for forensic analysis, parse EQL queries into abstract syntax trees (AST) for linting and security checks, introspect index snapshots and embedder configurations, and perform common vector and metric mathematical operations.

WAL Replay API

The WAL replay API enables you to read the contents of a Write-Ahead Log without opening the full database. This is primarily used for post-crash reconciliation to answer "what did the DB actually acknowledge before it died?". The replay is entirely read-only and mutates no database files. Corrupt tails are dropped silently (not raised), and unterminated transactions (begin without a commit) are omitted, matching what the database recovery mechanism would apply.

• replay_wal(path: str) -> list[WalEntry] — Parses the WAL file at the given path and returns a sequence of entries.

The returned WalEntry objects have the following properties:

• op (WalOp) — The operation type.
• vault (str) — The target vault.
• id (str) — The record ID.
• vector (tuple[float, ...]) — The vector payload (for inserts).
• data (dict) — The metadata payload.
• document (Optional[DocumentAttachment]) — Attached document data.
• chunk (Optional[ChunkInfo]) — Chunking metadata.
• lineage (Optional[EmbeddingLineage]) — Embedding traceability info.

The WalOp enum includes:

• insert
• erase
• insert_ex (insertion with attachments)
• txn_begin
• txn_commit

For a live view of what has been acknowledged but not yet checkpointed on an open database, use client.pending_writes(), which returns a typed list[WalRecord] (each with op, arena, key, and .is_delete). The replay_wal() API below is for reading a WAL file offline, without opening the database.

For details on transaction semantics, see Transaction Engine. For details on the underlying binary format, see Storage Architecture.

Crash Forensics Example

The following example demonstrates how to reconcile acknowledged writes with the live database state after a crash.

import elips
from collections import Counter

entries = elips.replay_wal("/data/vectors/wal.log")
print(Counter(e.op.name for e in entries))

acked = {e.id for e in entries if e.op != elips.WalOp.erase}
erased = {e.id for e in entries if e.op == elips.WalOp.erase}

client = elips.PersistentClient("/data/vectors")
docs = client.collection(entries[0].vault) if entries else None
missing = [rid for rid in acked - erased
           if docs is not None and not docs.get([rid])]
if missing:
    raise SystemExit(f"{len(missing)} acknowledged writes lost")
print("recovered cleanly;", len(acked - erased), "live writes")

EQL AST API

The EQL AST API lets you parse, validate, and tokenize EQL (ELIPS Query Language) statements without executing them. The parse entry points live at the top level, while the AST node types live under elips.query.

• parse_eql(source: str) -> Statement — Parses the query string and returns an AST root node. Raises ParseError on syntax failure.
• validate_eql(source: str) -> None — Validates the syntax without constructing the full AST tree.
• tokenize_eql(source: str) -> list[Token] — Lexes the source into tokens.

EQL is a small, lowercase, expression-oriented DSL — not SQL. Bindings are referenced with $name and supplied as bindings={"name": vector}.

A Token contains:

• kind (TokenKind: word, number, string, punct, or end)
• text (str)
• number (float)
• is_integer (bool)

The Statement returned by parse_eql is a union of one of the following classes (all under elips.query):

SearchStatement — verb search

• collection (str)
• query (VectorRef)
• k (Optional[int]) — result count (the k clause)
• threshold (Optional[float])
• where (Filter)
• rank_by (Optional[str])
• projection (list[str])

GetStatement — verb get from

• collection (str)
• id (str)

ScanStatement — verb scan

• collection (str)
• where (Filter)
• offset (Optional[int])
• limit (Optional[int])

AddStatement — verb add in

• collection (str)
• vector (VectorRef)
• metadata (dict)

DeleteStatement — verb delete from

• collection (str)
• id (str)

A VectorRef represents a vector literal or parameter binding. It has fields literal (list[float]) and binding (str), one of which will be non-empty.

EQL Guardrails Example

By leveraging parse_eql, you can implement multi-tenant query linters and security guardrails that inspect queries before dispatching them.

import elips
MAX_K = 100

def check(query: str) -> None:
    stmt = elips.parse_eql(query)  # raises ParseError on bad syntax
    if isinstance(stmt, elips.query.SearchStatement):
        if stmt.k is None or stmt.k > MAX_K:
            raise ValueError(f"search needs k <= {MAX_K}")
        if stmt.where.matches_all():
            raise ValueError("search needs a where clause")
        if stmt.where.exact_constraints() is None:
            raise ValueError("where clause is not index-accelerable")
    elif isinstance(stmt, elips.query.ScanStatement):
        if stmt.limit is None:
            raise ValueError("scan needs a limit")

check('search in docs nearest $q k 10 where tenant = "acme" yield')

Index Snapshots API

Index snapshots provide read-only introspective access to internal index layouts.

The IndexSnapshotKind enum defines the hardware and layout target: unknown, exact, graph, gpu_brute_force, gpu_ivf_flat, gpu_ivf_pq, gpu_graph, gpu_hybrid, gpu_distributed.

The IndexSnapshot object exposes the following properties:

• kind (IndexSnapshotKind)
• metric (Metric)
• dimension (int)
• ids (list[str])
• vectors (list[float]) (if stored natively)
• ivf (Optional[IvfSnapshot])
• pq (Optional[PqSnapshot])
• __len__() returns the number of items.

IvfSnapshot represents Inverted File structures with properties: n_lists, n_probe, centroids (list[float]), and assignments (list[int]).

PqSnapshot represents Product Quantization structures with properties: pq_dim, pq_bits, codebook (list[float]), and codes (list[int]).

Embedder Introspection API

You can inspect how the Python client resolves and configures embedding models using describe_local_embedder.

• describe_local_embedder(config=..., fallback_dimension=0, auto_attached=False) -> TextEmbedderInfo

The returned TextEmbedderInfo exposes properties such as: kind (TextEmbedderKind), provider, model, revision, backend, dimension, fingerprint, storage_path, rehydratable, loaded, and auto_attached.

TextEmbedderKind specifies whether the model is external (e.g. remote API) or local_builtin (running directly within ELIPS).

Vector Utilities

These utilities are commonly used for manipulating vectors and IDs.

• generate_id() -> str — Returns a fresh UUIDv7 hex string. This is useful when the caller must know the ID prior to the write operation (e.g., to publish to an event queue within the same transaction).
• is_valid_id(id: str) -> bool — Validates the format of a record ID.
• normalize(vector) -> tuple[float, ...] — L2-normalizes the given vector (zero vectors are returned unchanged).
• magnitude(vector) -> float — Computes the Euclidean L2 norm of the vector.

Metric Utilities

• distance(metric, a, b) -> float — Computes the ordering-normalized distance between two vectors according to the specified metric ("cosine", "euclidean", or "dot_product"). A smaller value always means "closer", matching how Hit.distance is ranked.