elips/docs
PYTHON API · LOW-LEVEL

Transaction

Overview

A Transaction lets you group multiple mutations — spanning one or more vaults — into a single atomic operation. Either every pending operation in the transaction is applied, or none of them are. This guarantee holds across process crashes: the WAL framing ensures that a partially written transaction is never replayed on recovery.

Transactions are obtained from a Database object and are most conveniently used as Python context managers.

python
with db.begin_transaction() as txn:
    products = txn.vault("products")
    sessions = txn.vault("sessions")

    products.place([0.1] * 768, data={"sku": "BOOT-42"})
    sessions.erase("session:expired-001")
# Both ops committed atomically

Database.begin_transaction()

python
Database.begin_transaction() -> Transaction

Allocates a new Transaction context against the database. This call writes a txn_begin marker to the WAL immediately. No vault-level write lock is acquired until the first mutating operation is enqueued or commit is called.

You may call begin_transaction() from any thread; however, the returned Transaction object must not be shared across threads (see Thread Safety).

Transaction.vault(name) -> TransactionVault

python
Transaction.vault(name: str) -> TransactionVault

Returns a TransactionVault — a thin proxy over the named vault that buffers all mutations into the transaction's pending-operation log. The underlying vault is created if it does not exist (same lazy semantics as Database.vault()).

Multiple calls to txn.vault("same-name") within the same transaction return the same TransactionVault proxy (idempotent).

TransactionVault.place()

python
TransactionVault.place(
    vector: list[float] | np.ndarray,
    data:   dict        = {},
    id:     str | None  = None,
) -> str

Enqueues a place (insert/upsert) operation into the transaction log. The record is not visible to other readers until the transaction is committed. Returns the record ID (auto-generated if id is None).

TransactionVault.erase()

python
TransactionVault.erase(id: str) -> None

Enqueues a delete operation into the transaction log. The record remains visible to readers outside this transaction until commit.

Transaction.commit()

python
Transaction.commit() -> None

Applies all pending operations atomically:

  1. Acquires exclusive write locks on every vault touched by the transaction (in a deterministic order to avoid deadlocks).
  2. Replays the pending-operation buffer against each vault's in-memory state.
  3. Writes a txn_commit marker to the WAL, then flushes according to the database's durability setting.
  4. Releases all vault write locks.

If any step fails (e.g., a WAL write error, a dimension mismatch discovered during replay), ELIPS automatically rolls back the in-memory state to its pre-transaction snapshot using the undo log before propagating the exception. The transaction is then invalid and must be discarded.

After a successful commit(), calling any method on theTransaction raises RuntimeError.

python
txn = db.begin_transaction()
v = txn.vault("items")
v.place([0.5] * 768, data={"label": "a"})
v.place([0.6] * 768, data={"label": "b"})

try:
    txn.commit()
except Exception as e:
    # In-memory state has already been restored.
    # No partial data is visible to readers.
    print(f"Commit failed: {e}")

Transaction.rollback()

python
Transaction.rollback() -> None

Discards all pending operations without writing anything to the vaults or WAL. A txn_rollback marker is written to the WAL so that WAL recovery skips any prior txn_begin for this transaction. After rollback, the Transaction is invalid.

python
txn = db.begin_transaction()
v = txn.vault("items")
v.place([0.1] * 768, data={"label": "draft"})

# Decided not to proceed
txn.rollback()
# The record is never visible to anyone

Context Manager

Transaction implements the Python context manager protocol (__enter__ / __exit__):

  • Clean exit (__exit__ with no exception) — automatically calls commit().
  • Exception exit — automatically calls rollback(), then re-raises the original exception.

This is the recommended usage pattern for all application code.

python
# Clean exit → auto-commit
with db.begin_transaction() as txn:
    txn.vault("items").place([0.1] * 768, data={"x": 1})
    txn.vault("items").place([0.2] * 768, data={"x": 2})
# commit() called here automatically

# Exception → auto-rollback
try:
    with db.begin_transaction() as txn:
        txn.vault("items").place([0.3] * 768, data={"x": 3})
        raise ValueError("something went wrong")
except ValueError:
    pass  # rollback() was called; the record is not in the vault

WAL Framing

Every transaction writes a pair of WAL markers that bracket all operation records:

  1. txn_begin(txn_id, timestamp) — written by begin_transaction().
  2. Individual operation records (op_place, op_erase) — written during commit replay.
  3. txn_commit(txn_id) — written as the last record of a successful commit.

During WAL recovery (after a crash), ELIPS scans the WAL and only applies transactions for which it finds a matching txn_commit marker. Any transaction with a txn_begin but no corresponding txn_commit is silently skipped — no partial data is ever applied.

bash
# Conceptual WAL layout:
# [txn_begin:42]
#   [op_place vault=products id=abc ...]
#   [op_place vault=products id=def ...]
#   [op_erase vault=sessions id=xyz]
# [txn_commit:42]       ← only if this is present does recovery apply the above

Crash Safety Guarantee

ELIPS provides the following crash-safety guarantee for transactions:

  • Before commit — A crash at any point before txn_commit is flushed to disk means the transaction is completely absent after recovery. Zero data is applied.
  • During commit flush — If the process crashes while flushing txn_commit to the WAL, the incomplete record is detected by a checksum mismatch during recovery and the transaction is skipped.
  • After commit flush — The transaction is fully durable (subject to the database's durability setting). With "paranoid" durability, an fsync is issued before commit() returns, guaranteeing the data survives even a hard power loss.

Undo Log

Internally, a Transaction maintains two structures:

  • PendingOp buffer — An ordered list of operations to apply at commit time. Each entry is a discriminated union of Place or Erase variants.
  • UndoEntry list — A snapshot of the pre-commit state for every record that will be mutated. Built during commit replay just before each mutation is applied. If any mutation fails mid-batch, the undo entries are replayed in reverse order via their restore_for_undo() mechanism, returning every affected vault to its exact pre-transaction state.

This means a failed commit is always clean: you never observe a vault that has some-but-not-all of the transaction's operations applied.

LockConflict

elips.LockConflict (subclass of RuntimeError) is raised when a transaction's commit cannot acquire a required vault write lock within the configured timeout. This can happen if:

  • Another long-running transaction holds the write lock on the same vault.
  • A rebuild_index() or vacuum() call is in progress on the vault.

When LockConflict is raised, the transaction is automatically rolled back — no partial mutations have been applied. You may retry the transaction.

python
import time, elips

def place_with_retry(db, vault_name, vector, data, max_retries=3):
    for attempt in range(max_retries):
        try:
            with db.begin_transaction() as txn:
                txn.vault(vault_name).place(vector, data=data)
            return  # success
        except elips.LockConflict:
            if attempt == max_retries - 1:
                raise
            time.sleep(0.05 * (2 ** attempt))  # exponential back-off

Thread Safety

Transaction objects are NOT thread-safe. Do not share a single Transaction across threads. The correct pattern is one transaction per thread:

python
import threading, elips

db = elips.open("/data/v.elips", dimension=384, metric="cosine")

def worker(items):
    # Each thread creates its own transaction
    with db.begin_transaction() as txn:
        v = txn.vault("items")
        for vec, meta in items:
            v.place(vec, data=meta)

threads = [
    threading.Thread(target=worker, args=(batch,))
    for batch in batches
]
for t in threads: t.start()
for t in threads: t.join()

The Database object itself is thread-safe and can be shared freely. Concurrent calls to begin_transaction() from different threads each produce independent Transaction objects with separate pending-op buffers.

Examples

Minimal example

python
import elips

db = elips.open("/data/v.elips", dimension=128, metric="cosine")

with db.begin_transaction() as txn:
    v = txn.vault("default")
    v.place([0.1] * 128, data={"label": "hello"}, id="rec:1")
    v.place([0.2] * 128, data={"label": "world"}, id="rec:2")

# Both records are now visible
print(db.vault("default").count)  # 2

Multi-vault atomic update

python
"""
Atomically move a record from 'staging' to 'production' and
record the transition in an 'audit' vault.
"""
import elips, time

db = elips.open("/data/pipeline.elips", dimension=768, metric="cosine")

def promote(record_id: str, embedding: list[float], meta: dict):
    with db.begin_transaction() as txn:
        staging    = txn.vault("staging")
        production = txn.vault("production")
        audit      = txn.vault("audit")

        staging.erase(record_id)
        production.place(embedding, data=meta, id=record_id)
        audit.place(
            embedding,
            data={
                "action":    "promoted",
                "record_id": record_id,
                "at":        time.time(),
            },
        )
    # All three operations committed atomically, or none

Error handling example

python
import elips

db = elips.open("/data/v.elips", dimension=128, metric="cosine")

def safe_batch_insert(records: list[dict]) -> bool:
    """Returns True on success, False if the transaction could not be applied."""
    try:
        with db.begin_transaction() as txn:
            v = txn.vault("items")
            for r in records:
                v.place(r["vector"], data=r.get("data", {}), id=r.get("id"))
        return True
    except elips.LockConflict as e:
        print(f"Lock conflict — will retry: {e}")
        return False
    except elips.DimensionMismatch as e:
        print(f"Bad vector dimension in batch: {e}")
        return False
    except Exception as e:
        print(f"Unexpected error (transaction rolled back): {e}")
        return False

Common Mistakes

Sharing a Transaction across threads

python
# ❌ WRONG — TransactionVault is not thread-safe
txn = db.begin_transaction()
v = txn.vault("items")

def bad_worker(vec, meta):
    v.place(vec, data=meta)  # Race condition!

threads = [threading.Thread(target=bad_worker, args=(vec, {})) for vec in vecs]

# ✅ CORRECT — one transaction per thread
def good_worker(vecs_metas):
    with db.begin_transaction() as txn:
        v = txn.vault("items")
        for vec, meta in vecs_metas:
            v.place(vec, data=meta)

Using a Transaction after commit or rollback

python
# ❌ WRONG
txn = db.begin_transaction()
txn.vault("items").place([0.1] * 128)
txn.commit()
txn.vault("items").place([0.2] * 128)  # RuntimeError: transaction already finalised

# ✅ CORRECT — use the context manager or open a new transaction
with db.begin_transaction() as txn:
    txn.vault("items").place([0.1] * 128)

with db.begin_transaction() as txn:
    txn.vault("items").place([0.2] * 128)

Forgetting to handle LockConflict

python
# ❌ FRAGILE — no retry on lock contention
with db.begin_transaction() as txn:
    txn.vault("items").place(vec, data={})

# ✅ ROBUST — wrap with retry logic for high-concurrency scenarios
import time

def insert_with_retry(db, vault_name, vec, data, retries=5):
    for i in range(retries):
        try:
            with db.begin_transaction() as txn:
                txn.vault(vault_name).place(vec, data=data)
            return
        except elips.LockConflict:
            time.sleep(0.02 * (2 ** i))
    raise RuntimeError(f"Failed to insert after {retries} attempts")
← PreviousConfig & GraphParams
Edit on GitHub →
Next →Filtering