Overview

ELIPS provides strong guarantees around data persistence and atomicity, allowing you to tune performance versus crash safety according to your application's needs. With the introduction of the F1-F4 storage engine updates, Python developers now have access to robust Write-Ahead Log (WAL) behaviors, reliable undo logs, and secure cross-process access controls.

Durability Levels

Durability levels determine how aggressively ELIPS forces the operating system to flush writes to physical media. Durability is a native Config/Durability concept — it is configured on a Config object, not as a keyword argument to the client constructor:

paranoid: F_FULLFSYNC on macOS, fdatasync on Linux before every acknowledgment. Provides maximum crash safety, suitable for financial or medical data where every single write must survive power loss.
standard: OS-synced writes (fsync at checkpoint). This is the default setting and is suitable for most production uses.
relaxed: Writes directly to the OS page cache. Data will survive a process crash but not an OS crash or power loss. Ideal for staging or batch ingest pipelines where you can afford to re-ingest data if a system goes down.
ephemeral: No persistence at all (in-memory mode, no WAL). Suitable for unit tests, CI, and in-process search over preloaded data.

python

import elips

# paranoid = strongest durability (sync every ack)
config = elips.Config().durability("paranoid")

# standard = default
config = elips.Config().durability("standard")

# relaxed = throughput-optimized (page cache)
config = elips.Config().durability("relaxed")

# ephemeral = no persistence
config = elips.Config().durability("ephemeral")

You can inspect the currently active durability via the config object's durability_val property.

python

import elips

# Throughput-optimized (relaxed) — apply a full Config via the
# legacy connect_with_config path:
config = elips.Config().dimension(128).durability("relaxed")
client = elips.connect_with_config("/data/staging", config)  # legacy Config-object open

# In-memory, no WAL at all:
client = elips.Client(dimension=128)

Storage Improvements (F1-F4)

The F1 through F4 patches significantly hardened the ELIPS storage engine, resolving critical issues around atomicity and corruption recovery. It is important for users to understand what these fixes guarantee:

F1: True Sync Guarantees. The Write-Ahead Log (WAL) now correctly fsyncs (or F_FULLFSYNC on macOS) before returning from an append. Additionally, snapshot, segment, and manifest writes are published via a durable rename, which syncs both the file and directory metadata. Previously, acknowledged writes could be lost on OS crash or power loss even under the "standard" durability setting.
F2: Bounded Allocations. Length-prefixed payloads (like strings or vectors) are now bounded by the remaining stream length before allocation. This means a corrupt WAL can no longer cause the system to attempt unbounded memory allocations.
F3: Faster Recovery. WAL replay time was optimized from O(n*k) to O(n) by avoiding unnecessary tail-copying per record.
F4: Robust Atomicity. The commit() process now pre-checks writability and maintains an undo log. A failed WAL write automatically restores the prior state. Commits on a read-only vault throw an error before applying anything. Furthermore, the WAL brackets batches with txn_begin and txn_commit, ensuring that replay completely drops unterminated transaction windows.

Transaction Semantics

Following the F4 updates, transactions are strictly all-or-nothing:

All-or-nothing: A failure partway through a transaction uses the undo log to revert any operations that were already applied.
WAL framing: Transactions are bracketed with txn_begin and txn_commit markers in the WAL. If a process dies mid-commit, the replay engine will drop the unterminated window.
Pre-checks: commit() verifies that a vault is not read-only before attempting to apply operations.
Safe Rollbacks: Calling rollback() is always safe, even after a failed commit(), since the state will have already been restored by the undo log.

Transaction Examples

python

import elips

# Configure maximum durability on a native Config
config = (
    elips.Config()
    .dimension(384)
    .metric("cosine")
    .durability("paranoid")
)
client = elips.connect_with_config("/data/critical", config)  # legacy Config-object open

# Native transaction - all or nothing using a context manager
with client.native.begin_transaction() as txn:
    v = txn.vault("docs")
    v.place([1.0, 0.0], {"title": "A"})
    v.place([0.0, 1.0], {"title": "B"})
    # Clean exit: both writes committed atomically
    # Exception: both writes rolled back

# Failed commit restores state automatically
txn = client.native.begin_transaction()
v = txn.vault("docs")
v.place([1.0, 0.0], {"title": "C"})
try:
    txn.commit()
except elips.StorageError:
    # State restored. Safe to retry or rollback explicitly.
    txn.rollback()  # Also safe; state already restored

Access Modes & Locking

ELIPS uses POSIX advisory flock to manage cross-process locking. There are two primary modes for opening a database:

read_write (default): Acquires an exclusive writer lock. Only one process can open the database in this mode at a time.
read_only: Acquires a shared advisory lock. Multiple processes can open the database in read_only mode. Write operations (like place, erase, rebuild, or compaction) will raise a StorageError.

If a second process attempts to open a database in read_write mode while another process already holds the exclusive lock, ELIPS raises a LockConflict exception. To read from a database while it is being written to by another process, use read_only mode.

Read-Only Serving

The shared-reader pattern allows you to scale read traffic without interfering with a single writer process.

python

# Read-only serving (shared readers)
# A separate process might hold the exclusive writer lock
reader = elips.PersistentClient("/data/critical", access_mode="read_only")
results = reader.collection("docs").query(query_embeddings=[1.0, 0.0], n_results=5)

Monitoring Durability

You can inspect operations that are currently in the WAL but not yet checkpointed using client.pending_writes(). This returns a typed list of WalRecord objects, which is extremely useful for monitoring durability lag and deciding when to force a checkpoint.

Crash Recovery

Recovery after an unclean shutdown happens automatically. When you reopen a database with elips.PersistentClient(), ELIPS replays the WAL to recover any committed transactions that were not yet checkpointed.

python

# After an unclean shutdown (crash or hard kill), the WAL is left on disk.
# Reopening replays it automatically — no special call is needed:
client = elips.PersistentClient("/data/critical")

Common Pitfalls

LockConflict in multiprocessing: Remember that only one process can have a read_write handle. Use read_only for read replicas in secondary processes.
Unnecessary durability: Using paranoid durability introduces significant latency (often 10-20ms per transaction due to syncing physical media). Only use it if losing a single write during power loss is unacceptable.
Leaving transactions open: Long-running transactions delay checkpointing. Keep transactions scoped tightly around the required batch of operations.

Learn more: Check out the Transaction Engine, Storage format, and Lock Manager documentation.