Persistence

How kevy keeps data across restarts — the AOF, snapshots, fsync policies, rewrite/compaction, crash recovery, and the introspection that lets you watch all of it.

When you need this

Reach for this doc when you are:

• Choosing a durability policy (zero-loss vs throughput) for a production deployment.
• Sizing disk usage and replay-time budgets for a write-heavy workload.
• Debugging an unexpected on-disk artifact — a quarantine file, a stale .rewrite temp, a .premigration.* backup.
• Wiring an embedded kevy_embedded::Store into a host application and want to know what survives a process crash, what doesn't, and how to observe it from inside the host.
• Looking at a key whose TTL behaves oddly across restarts.

If you only want a quick "does it survive kill -9?" answer: yes, with at most one second of writes lost under the default policy.

Core idea

Every shard owns two files in the persistence directory: an append-only log of mutating commands (aof-<id>.aof) and an optional binary snapshot (dump-<id>.rdb). The AOF alone is a complete durable record; the snapshot exists only to bound replay time. On boot kevy loads the snapshot if present, then replays the AOF; on a successful snapshot the AOF is reset so the two files together cover the full history exactly once.

The directory itself is created at boot if missing (v3.17); a path that cannot be created is one named startup error, not a bare ENOENT from whichever subsystem touches it first.

Worked examples

Server mode

Drop this into kevy.toml and launch with kevy --config kevy.toml:

# kevy.toml
[server]
data_dir = "/var/lib/kevy"
port     = 6379
threads  = 4

[persistence]
aof = true
# AOF durability — see the knobs table below for the full set.
appendfsync                 = "everysec"   # always | everysec | no
auto_aof_rewrite_percentage = 100          # rewrite when the AOF doubles since the last rewrite
auto_aof_rewrite_min_size   = "64mb"       # …and is at least this big

Operate it with standard Redis-style commands over RESP:

$ redis-cli -p 6379 BGSAVE
Background saving started

$ redis-cli -p 6379 BGREWRITEAOF
Background append only file rewriting started

$ redis-cli -p 6379 INFO persistence
aof_enabled:1
appendfsync:everysec
aof_rewrite_in_progress:0
aof_rewrites_total:3

CONFIG SET appendfsync always retunes the policy live without a restart.

Embedded mode

Add the crate to Cargo.toml:

[dependencies]
kevy-embedded = "*"

Then in main.rs:

use std::time::Duration;
use kevy_embedded::{AppendFsync, Config, KevyMetric, Store};

fn main() -> kevy_embedded::KevyResult<()> {
    let cfg = Config::default()
        .with_persist("/var/lib/myapp/kevy")
        .with_appendfsync(AppendFsync::EverySec)
        .with_auto_aof_rewrite(100, 64 * 1024 * 1024)
        .with_metric_sink(|m| match m {
            KevyMetric::Replay { commands, bytes, elapsed_ms } => {
                eprintln!("kevy replay: {commands} cmds / {bytes} B in {elapsed_ms} ms");
            }
            KevyMetric::Rewrite { keys, before_bytes, after_bytes, elapsed_ms } => {
                eprintln!(
                    "kevy rewrite: {keys} keys, {before_bytes} -> {after_bytes} B in {elapsed_ms} ms"
                );
            }
            _ => {}
        });

    let store = Store::open(cfg)?;

    store.set(b"hello", b"world")?;
    store.expire(b"hello", Duration::from_secs(300))?;

    // Point-in-time snapshot. Returns after the file is on disk; per-shard
    // locks are held only for the view freeze and the final rename.
    store.save_snapshot()?;

    // On-demand AOF compaction. Same lock discipline as save_snapshot.
    let _stats = store.rewrite_aof()?;

    // Live introspection.
    let info = store.info();
    println!("{} keys, {} bytes AOF", info.keys, info.aof_bytes);

    Ok(())
}

A fresh embedded store with the default config writes only the AOF — no snapshot file appears until save_snapshot runs. That is expected; the AOF on its own is enough to rebuild the keyspace.

Configuration knobs

Durability and AOF growth

Trigger surface

fsync policy semantics

Trade-offs and limits

Per-policy throughput vs data loss. Always blocks each reply on fsync; it is the only policy that survives kill -9 with zero command loss, and it cuts SET-heavy throughput roughly in half on typical NVMe. EverySec runs a background flush every second and loses up to that window on a crash — the default precisely because it matches the Redis trade and the lost window is usually tolerable. No lets the kernel decide; throughput is highest but a crash can lose anything still in pagecache, potentially many seconds.

AOF replay cost vs snapshot load cost. Without a snapshot, boot time grows linearly with the AOF byte count: a 4 GiB AOF replays in a few seconds on local NVMe, a 40 GiB one in a minute or more. A snapshot caps that — load is one streaming read plus a short tail of post-snapshot AOF — but costs a transient view freeze (O(keys), nanoseconds per key, because collection values are refcount-shared) plus a one-time copy of any collection first mutated while the snapshot is in flight. For write-heavy workloads, prefer leaning on auto-rewrite to keep the AOF bounded rather than running periodic BGSAVEs: rewrite gives you the same boot-time bound with no second file to manage.

Background-job concurrency. Each shard runs at most one background save or rewrite at a time. A duplicate request that arrives mid-job is skipped with a log line, never queued.

TTL persistence. TTLs are written as absolute Unix-millisecond deadlines (PEXPIREAT in the AOF, an absolute field in the snapshot format), so a key keeps its original expiry instant across any number of restarts and the time the process spent down is subtracted correctly. Older AOFs that recorded relative remaining time still load (treated as relative on entry); new writes are always absolute. EXPIREAT and PEXPIREAT are exposed as client commands.

Shard-layout changes are crash-idempotent. Changing --threads / shards writes new snapshots under .reshard temp names, commits via a durable reshard.journal, and rolls an interrupted migration forward on the next start. Source files survive as .premigration.<unix_ts> backups; the journal is the commit point and must never be deleted by hand.

What is not persisted. Pub/sub channels, subscriptions, and undelivered messages live only in memory. Blocking-command waiters such as BLPOP and blocking XREAD are connection state, not data. Neither is written to the AOF or snapshot, and neither is replayed.

FAQ

My AOF file is growing — how do I compact it?

Run BGREWRITEAOF on the server or Store::rewrite_aof() in embedded mode. Rewrite rebuilds the log as the minimal command set that reconstructs the current keyspace — one SET / HSET / etc. per key, plus a PEXPIREAT for TTL'd keys — and atomically swaps the new file in. Ten thousand overwrites of hot collapse to a single SET hot <latest>.

For unattended ops, leave auto-rewrite at its defaults — 100% growth over the previous rewrite size, with a 64 MiB floor — and the reactor will fire compaction on its own. Set auto_aof_rewrite_percentage = 0 to disable it and drive rewrite entirely by hand.

Rewrite is non-blocking for the keyspace: serialization and fsync run with reads and writes flowing, and any writes that land during the rewrite are tee'd into a diff buffer that gets appended to the compacted image. If a rewrite crashes midway the original AOF is untouched (the swap is an atomic rename) and the leftover aof-<id>.aof.rewrite temp is safe to delete.

Can I disable persistence entirely?

Yes, in two ways:

• Server: set appendonly = false in kevy.toml (or omit --dir). The server runs as a pure in-memory cache; no aof-* or dump-* files are created.
• Embedded: build a Config without calling with_persist(...). Store::open runs the keyspace entirely in memory; save_snapshot and rewrite_aof become no-ops at the API surface (or surface an error indicating no persistence directory is configured).

If you want persistence but no AOF growth at all between snapshots, that combination is not supported — kevy's durability model is AOF-first, and the snapshot exists to bound AOF replay, not to replace the AOF.

What is the cost of a snapshot during high write load?

The blocking portion is tiny. A per-shard freeze of the keyspace is O(keys), not O(bytes), because collection values are reference-counted and shared with the live store; on a million-key shard the freeze takes single-digit milliseconds. Serialization itself runs with the keyspace live — writes are not paused.

The transient cost you pay is memory. Any collection (list, hash, set, sorted-set) that gets mutated while the snapshot is being written gets cloned once, so the live store can move on without disturbing the frozen view. For workloads dominated by SET on plain string keys the extra memory is negligible; for workloads dominated by HSET / LPUSH on a small number of huge collections it can briefly double the resident size of those specific collections.

A successful snapshot also resets the AOF — the snapshot now carries everything the log used to, and the log restarts with only writes that landed after the freeze. A restart then loads snapshot + log without ever double-applying history.

How is recovery sequenced on the next boot?

For each shard, in order:

1. Load the snapshot. If dump-<id>.rdb exists, stream it into the keyspace. Expired TTLs are dropped during load.
2. Replay the AOF. Read aof-<id>.aof from the front and apply each frame.
3. Handle the tail. A clean file applies in full. A truncated tail (crash mid-append) drops the partial trailing frame and applies the prefix. A corrupt frame moves the bad bytes aside to aof-<id>.aof.panic-quarantine.<unix_ts> so they don't block future starts, then applies the prefix. The quarantined tail is never re-applied; inspect it by hand if you need to recover anything from it.
4. Log a one-line summary including wall-clock time:

``text kevy: AOF /data/kevy/aof-0.aof replayed 145313 commands from 418261733 bytes in 247 ms (clean) ``

1. Roll forward any interrupted shard-layout migration by replaying reshard.journal.

Watch the replay-time line and use auto-rewrite to keep it bounded — replay time grows linearly with the unrewritten AOF size.

How do I monitor persistence from inside an embedded host process?

Two surfaces.

Polling. store.info() returns a KevyInfo struct with keys, used_memory, aof_bytes, expire_pending, evictions, expired_keys. Finer-grained helpers cover the same ground:

store.dbsize();                 // live key count
store.ttl(key);                 // Option<Duration> (None = no key / no TTL)
store.ttl_ms(key);              // Redis PTTL semantics: -2 no key, -1 no TTL, else ms
store.expire_pending_count();   // live keys carrying a TTL
store.used_memory();            // resident-bytes estimate
store.expired_keys_total();     // total expired (lazy + reaper)
store.evictions_total();        // total evicted by maxmemory

expire_pending_count() == 0 when you expected TTLs is the classic tell that the TTL subsystem didn't register your keys.

Push. Register Config::with_metric_sink(...) and receive KevyMetric events on AOF replay (startup) and each AOF rewrite (compaction). The sink runs synchronously on the emitting thread (the reaper for background rewrites), so keep the callback fast. KevyMetric is #[non_exhaustive] — always match a _ arm to stay forward-compatible.

What is every file in the persistence directory?

Durability contract (v2.1)

What "the call returned OK" guarantees, per appendfsync × write path. "Durable" = on stable storage (fdatasync completed); "windowed" = in the OS page cache, lost only if the machine (not just the process) dies inside the window.

Store::fsync_aof() is the per-write durability escape hatch (Postgres synchronous_commit-per-transaction genre): run a deployment on everysec for throughput, and place the barrier after the few writes that must survive a machine crash the moment they are acknowledged. Cost: one fdatasync per dirty shard.

Process crash (SIGKILL) never loses acknowledged writes under always and loses at most the fsync window otherwise; the AOF tail is replayed on the next open, and a torn final frame is quarantined (panic-quarantine), never silently applied.

An orderly stop (SHUTDOWN or SIGTERM) loses nothing under any policy: the drain force-fsyncs the AOF tail before exit, so the everysec window that a crash can lose does not apply to a clean shutdown.

Atomicity charter (embedded serving-store, v2.1)

• Store::atomic(body) — single-shard transaction: takes that shard's write lock for the closure, reads inside see the closure's own writes, AOF appends are deferred and committed with one fsync at commit (under always). Every key touched must hash to the same shard — the blessed serving-store config is therefore 1 shard when your write patterns span arbitrary keys: you keep full atomicity and pay no cross-shard coordination. The ceiling of the 1-shard config is single-core write throughput; measured numbers live in bench/REPORT.md.
• Store::atomic_all_shards(body) — multi-shard transaction: acquires every shard's write lock in shard-index order (deterministic order = no deadlock), commits per-shard AOF batches on return. Cost: blocks all other readers + writers for the closure's duration — use it for cross-shard invariants, not as the default write path.
• Store::pipeline() — NOT atomic: each op takes its own lock; other writers interleave. It batches fsyncs (N ops → ≤ shard-count fsyncs), nothing more.
• Both atomic forms log the effect of conditional ops (ZADD GT, SPOP) as unconditional verbs, so replay and replica-apply are deterministic by construction.

Recovery points (v2.3)

With the change feed enabled ([feed] enabled = true, see cdc.md), every snapshot records the feed cursor it was taken at — frozen in the same no-append window as the snapshot data itself. That yields the recovery-point contract:

snapshot S + the feed frames from S's recorded cursor = the exact state at any later cursor.

kevy_persist::read_snapshot_cursor(path) reads the cursor back (None for pre-v2.3 snapshots — format v4 and older carry no cursor and remain fully loadable). The executable form of the contract is bench/restore-drill.sh, run as a diskgate line: write → SAVE → write more → kill → restore the dumps alone → replay the captured feed frames → byte-exact key-by-key verification.

Scope note: the feed window is the in-memory backlog. Frames older than the window are gone — a snapshot older than the window's reach is a plain snapshot restore (state at S), not a PITR base. Take snapshots at least as often as the window turns over if you rely on exact-point recovery.

The snapshot on the replication wire (v3.15)

The same snapshot format is what a primary in-line-ships to a replica that has fallen past the backlog window (see replication.md). One semantic worth knowing: a shipped snapshot replaces the replica's local state, it does not merge — the replica flushes its keyspace before loading. That is deliberate: when a rejoining ex-primary carries a forked suffix (writes that were never replicated), the resync must genuinely discard the fork rather than leave it as residue under an upsert-only load.