Overview

ELIPS provides strong guarantees around data persistence and atomicity, allowing you to tune performance versus crash safety according to your application's needs. With the introduction of the F1-F4 storage engine updates, Python developers now have access to robust Write-Ahead Log (WAL) behaviors, reliable undo logs, and secure cross-process access controls.

Durability Levels

Durability levels determine how aggressively ELIPS forces the operating system to flush writes to physical media. You can configure this using the durability() builder on the Config object:

paranoid: F_FULLFSYNC on macOS, fdatasync on Linux before every acknowledgment. Provides maximum crash safety, suitable for financial or medical data where every single write must survive power loss.
standard: OS-synced writes (fsync at checkpoint). This is the default setting and is suitable for most production uses.
relaxed: Writes directly to the OS page cache. Data will survive a process crash but not an OS crash or power loss. Ideal for staging or batch ingest pipelines where you can afford to re-ingest data if a system goes down.
ephemeral: No persistence at all (in-memory mode, no WAL). Suitable for unit tests, CI, and in-process search over preloaded data.

python

import elips

# paranoid = strongest durability (sync every ack)
config = elips.Config().durability("paranoid")

# standard = default
config = elips.Config().durability("standard")

# relaxed = throughput-optimized (page cache)
config = elips.Config().durability("relaxed")

# ephemeral = no persistence
config = elips.Config().durability("ephemeral")

You can inspect the currently active durability via properties on the config object: config.durability_enum and config.durability_val.

python

# Throughput-optimized (relaxed)
config = elips.Config().dimension(128).durability("relaxed")
db = elips.open_with_config("/data/staging", config)

# In-memory (no WAL at all)
db = elips.open(":memory:", dimension=128)
print(db.persistent)  # False

Storage Improvements (F1-F4)

The F1 through F4 patches significantly hardened the ELIPS storage engine, resolving critical issues around atomicity and corruption recovery. It is important for users to understand what these fixes guarantee:

F1: True Sync Guarantees. The Write-Ahead Log (WAL) now correctly fsyncs (or F_FULLFSYNC on macOS) before returning from an append. Additionally, snapshot, segment, and manifest writes are published via a durable rename, which syncs both the file and directory metadata. Previously, acknowledged writes could be lost on OS crash or power loss even under the "standard" durability setting.
F2: Bounded Allocations. Length-prefixed payloads (like strings or vectors) are now bounded by the remaining stream length before allocation. This means a corrupt WAL can no longer cause the system to attempt unbounded memory allocations.
F3: Faster Recovery. WAL replay time was optimized from O(n*k) to O(n) by avoiding unnecessary tail-copying per record.
F4: Robust Atomicity. The commit() process now pre-checks writability and maintains an undo log. A failed WAL write automatically restores the prior state. Commits on a read-only vault throw an error before applying anything. Furthermore, the WAL brackets batches with txn_begin and txn_commit, ensuring that replay completely drops unterminated transaction windows.

Transaction Semantics

Following the F4 updates, transactions are strictly all-or-nothing:

All-or-nothing: A failure partway through a transaction uses the undo log to revert any operations that were already applied.
WAL framing: Transactions are bracketed with txn_begin and txn_commit markers in the WAL. If a process dies mid-commit, the replay engine will drop the unterminated window.
Pre-checks: commit() verifies that a vault is not read-only before attempting to apply operations.
Safe Rollbacks: Calling rollback() is always safe, even after a failed commit(), since the state will have already been restored by the undo log.

Transaction Examples

python

import elips

# Configure maximum durability
config = (
    elips.Config()
    .dimension(384)
    .metric("cosine")
    .durability("paranoid")
)
db = elips.open_with_config("/data/critical", config)

# Transaction - all or nothing using a context manager
with db.begin_transaction() as txn:
    v = txn.vault("docs")
    v.place([1.0, 0.0], {"title": "A"})
    v.place([0.0, 1.0], {"title": "B"})
    # Clean exit: both writes committed atomically
    # Exception: both writes rolled back

# Failed commit restores state automatically
txn = db.begin_transaction()
v = txn.vault("docs")
v.place([1.0, 0.0], {"title": "C"})
try:
    txn.commit()
except elips.StorageError:
    # State restored. Safe to retry or rollback explicitly.
    txn.rollback()  # Also safe; state already restored

Access Modes & Locking

ELIPS uses POSIX advisory flock to manage cross-process locking. There are two primary modes for opening a database:

read_write (default): Acquires an exclusive writer lock. Only one process can open the database in this mode at a time.
read_only: Acquires a shared advisory lock. Multiple processes can open the database in read_only mode. Write operations (like place, erase, rebuild, or compaction) will raise a StorageError.

If a second process attempts to open a database in read_write mode while another process already holds the exclusive lock, ELIPS raises a LockConflict exception. To read from a database while it is being written to by another process, use read_only mode.

Read-Only Serving

The shared-reader pattern allows you to scale read traffic without interfering with a single writer process.

python

# Read-only serving (shared readers)
# A separate process might hold the exclusive writer lock
reader = elips.open("/data/critical", access_mode="read_only")
results = reader.vault("docs").seek([1.0, 0.0], top=5)

Monitoring Durability

You can inspect operations that are currently in the WAL but not yet checkpointed using vault.pending_writes(). This returns a typed list of WalRecord objects, which is extremely useful for monitoring durability lag and deciding when to force a checkpoint.

Crash Recovery

Recovery after an unclean shutdown happens automatically. When you call elips.open(), ELIPS replays the WAL to recover any committed transactions that were not yet checkpointed.

python

# Simulate unclean shutdown by abandoning the handle
db.abandon()  # Leaves WAL on disk, bypassing clean shutdown

# Next open() replays the WAL automatically
db2 = elips.open("/data/critical")

Common Pitfalls

LockConflict in multiprocessing: Remember that only one process can have a read_write handle. Use read_only for read replicas in secondary processes.
Unnecessary durability: Using paranoid durability introduces significant latency (often 10-20ms per transaction due to syncing physical media). Only use it if losing a single write during power loss is unacceptable.
Leaving transactions open: Long-running transactions delay checkpointing. Keep transactions scoped tightly around the required batch of operations.

Learn more: Check out the Transaction Engine, Storage format, and Lock Manager documentation.