Overview

This page documents the advanced diagnostic and utility APIs available in the ELIPS Python client. These APIs allow you to replay the write-ahead log (WAL) for forensic analysis, parse EQL queries into abstract syntax trees (AST) for linting and security checks, introspect index snapshots and embedder configurations, and perform common vector and metric mathematical operations.

WAL Replay API

The WAL replay API enables you to read the contents of a Write-Ahead Log without opening the full database. This is primarily used for post-crash reconciliation to answer "what did the DB actually acknowledge before it died?". The replay is entirely read-only and mutates no database files. Corrupt tails are dropped silently (not raised), and unterminated transactions (begin without a commit) are omitted, matching what the database recovery mechanism would apply.

replay_wal(path: str) -> list[WalEntry] — Parses the WAL file at the given path and returns a sequence of entries.

The returned WalEntry objects have the following properties:

op (WalOp) — The operation type.
vault (str) — The target vault.
id (str) — The record ID.
vector (tuple[float, ...]) — The vector payload (for inserts).
data (dict) — The metadata payload.
document (Optional[DocumentAttachment]) — Attached document data.
chunk (Optional[ChunkInfo]) — Chunking metadata.
lineage (Optional[EmbeddingLineage]) — Embedding traceability info.

The WalOp enum includes:

insert
erase
insert_ex (insertion with attachments)
txn_begin
txn_commit

For details on transaction semantics, see Transaction Engine. For details on the underlying binary format, see Storage Architecture.

Crash Forensics Example

The following example demonstrates how to reconcile acknowledged writes with the live database state after a crash.

python

import elips
from collections import Counter

entries = elips.replay_wal("/data/vectors/wal.log")
print(Counter(e.op.name for e in entries))

acked = {e.id for e in entries if e.op != elips.WalOp.erase}
erased = {e.id for e in entries if e.op == elips.WalOp.erase}

db = elips.open("/data/vectors")
vault = db.vault(entries[0].vault) if entries else None
missing = [rid for rid in acked - erased
           if vault is not None and vault.fetch(rid) is None]
if missing:
    raise SystemExit(f"{len(missing)} acknowledged writes lost")
print("recovered cleanly;", len(acked - erased), "live writes")

EQL AST API

The EQL AST API allows you to parse, validate, and tokenize EQL (ELIPS Query Language) statements without executing them.

parse_eql(source: str) -> Statement — Parses the query string and returns an AST root node. Raises ParseError on syntax failure.
validate_eql(source: str) -> None — Validates the syntax without constructing the full AST tree.
tokenize_eql(source: str) -> list[Token] — Lexes the source into tokens.

A Token contains:

kind (TokenKind: word, number, string, punct, or end)
text (str)
number (float)
is_integer (bool)

The Statement returned by parse_eql is a union of one of the following classes:

SearchStatement

vault (str)
query (VectorRef)
top (Optional[int])
threshold (Optional[float])
where (Filter)
rank_by (Optional[str])
projection (list[str])

FetchStatement

vault (str)
id (str)

ScanStatement

vault (str)
where (Filter)
offset (Optional[int])
limit (Optional[int])

InsertStatement

vault (str)
vector (VectorRef)
data (dict)

DeleteStatement

vault (str)
id (str)

A VectorRef represents a vector literal or parameter binding. It has fields literal (list[float]) and binding (str), one of which will be non-empty.

EQL Guardrails Example

By leveraging parse_eql, you can implement multi-tenant query linters and security guardrails that inspect queries before dispatching them.

python

import elips
MAX_TOP = 100

def check(query: str) -> None:
    stmt = elips.parse_eql(query)  # raises ParseError on bad syntax
    if isinstance(stmt, elips.SearchStatement):
        if stmt.top is None or stmt.top > MAX_TOP:
            raise ValueError(f"seek needs top <= {MAX_TOP}")
        if stmt.where.matches_all():
            raise ValueError("seek needs a where clause")
        if stmt.where.exact_constraints() is None:
            raise ValueError("where clause is not index-accelerable")
    elif isinstance(stmt, elips.ScanStatement):
        if stmt.limit is None:
            raise ValueError("scan needs a limit")

check('seek in docs nearest $q top 10 where tenant = "acme" yield')

Index Snapshots API

Index snapshots provide read-only introspective access to internal index layouts.

The IndexSnapshotKind enum defines the hardware and layout target: unknown, exact, graph, gpu_brute_force, gpu_ivf_flat, gpu_ivf_pq, gpu_graph, gpu_hybrid, gpu_distributed.

The IndexSnapshot object exposes the following properties:

kind (IndexSnapshotKind)
metric (Metric)
dimension (int)
ids (list[str])
vectors (list[float]) (if stored natively)
ivf (Optional[IvfSnapshot])
pq (Optional[PqSnapshot])
__len__() returns the number of items.

IvfSnapshot represents Inverted File structures with properties: n_lists, n_probe, centroids (list[float]), and assignments (list[int]).

PqSnapshot represents Product Quantization structures with properties: pq_dim, pq_bits, codebook (list[float]), and codes (list[int]).

Embedder Introspection API

You can inspect how the Python client resolves and configures embedding models using describe_local_embedder.

describe_local_embedder(config=..., fallback_dimension=0, auto_attached=False) -> TextEmbedderInfo

The returned TextEmbedderInfo exposes properties such as: kind (TextEmbedderKind), provider, model, revision, backend, dimension, fingerprint, storage_path, rehydratable, loaded, and auto_attached.

TextEmbedderKind specifies whether the model is external (e.g. remote API) or local_builtin (running directly within ELIPS).

Vector Utilities

These utilities are commonly used for manipulating vectors and IDs.

generate_id() -> str — Returns a fresh UUIDv7 hex string. This is useful when the caller must know the ID prior to the write operation (e.g., to publish to an event queue within the same transaction).
is_valid_id(id: str) -> bool — Validates the format of a record ID.
normalize(vector) -> tuple[float, ...] — L2-normalizes the given vector (zero vectors are returned unchanged).
magnitude(vector) -> float — Computes the Euclidean L2 norm of the vector.

Metric Utilities

distance(metric, a, b) -> float — Computes the ordering-normalized distance between two vectors according to the specified metric.
requires_normalization(metric) -> bool — Returns True only for the cosine metric.
metric_to_string(metric) -> str — Returns the string name for a given Metric.
metric_from_string(name) -> Metric — Returns the corresponding Metric enum from its string name.