Persistence

How kevy keeps data across restarts — the AOF, snapshots, fsync policies, rewrite/compaction, crash recovery, and the introspection that lets you watch all of it.

When you need this

Reach for this doc when you are:

• Choosing a durability policy (zero-loss vs throughput) for a production deployment.
• Sizing disk usage and replay-time budgets for a write-heavy workload.
• Debugging an unexpected on-disk artifact — a quarantine file, a stale .rewrite temp, a .premigration.* backup.
• Wiring an embedded kevy_embedded::Store into a host application and want to know what survives a process crash, what doesn't, and how to observe it from inside the host.
• Looking at a key whose TTL behaves oddly across restarts.

If you only want a quick "does it survive kill -9?" answer: yes, with at most one second of writes lost under the default policy.

Core idea

Every shard owns two files in the persistence directory: an append-only log of mutating commands (aof-<id>.aof) and an optional binary snapshot (dump-<id>.rdb). The AOF alone is a complete durable record; the snapshot exists only to bound replay time. On boot kevy loads the snapshot if present, then replays the AOF; on a successful snapshot the AOF is reset so the two files together cover the full history exactly once.

The directory itself is created at boot if missing (v3.17); a path that cannot be created is one named startup error, not a bare ENOENT from whichever subsystem touches it first.

Worked examples

Server mode

Drop this into kevy.toml and launch with kevy --config kevy.toml:

# kevy.toml
[server]
data_dir = "/var/lib/kevy"
port     = 6379
threads  = 4

[persistence]
aof = true
# AOF durability — see the knobs table below for the full set.
appendfsync                 = "everysec"   # always | everysec | no
auto_aof_rewrite_percentage = 100          # rewrite when the AOF doubles since the last rewrite
auto_aof_rewrite_min_size   = "64mb"       # …and is at least this big

Operate it with standard Redis-style commands over RESP:

$ redis-cli -p 6379 BGSAVE
Background saving started

$ redis-cli -p 6379 BGREWRITEAOF
Background append only file rewriting started

$ redis-cli -p 6379 INFO persistence
aof_enabled:1
appendfsync:everysec
aof_rewrite_in_progress:0
aof_rewrites_total:3

CONFIG SET appendfsync always retunes the policy live without a restart.

Embedded mode

Add the crate to Cargo.toml:

[dependencies]
kevy-embedded = "*"

Then in main.rs:

use std::time::Duration;
use kevy_embedded::{AppendFsync, Config, KevyMetric, Store};

fn main() -> kevy_embedded::KevyResult<()> {
    let cfg = Config::default()
        .with_persist("/var/lib/myapp/kevy")
        .with_appendfsync(AppendFsync::EverySec)
        .with_auto_aof_rewrite(100, 64 * 1024 * 1024)
        .with_metric_sink(|m| match m {
            KevyMetric::Replay { commands, bytes, elapsed_ms, dropped_bytes, corrupt } => {
                eprintln!("kevy replay: {commands} cmds / {bytes} B in {elapsed_ms} ms");
                if dropped_bytes > 0 || corrupt {
                    eprintln!("kevy replay dropped {dropped_bytes} B (corrupt: {corrupt}) — ALERT");
                }
            }
            KevyMetric::Rewrite { keys, before_bytes, after_bytes, elapsed_ms } => {
                eprintln!(
                    "kevy rewrite: {keys} keys, {before_bytes} -> {after_bytes} B in {elapsed_ms} ms"
                );
            }
            _ => {}
        });

    let store = Store::open(cfg)?;

    store.set(b"hello", b"world")?;
    store.expire(b"hello", Duration::from_secs(300))?;

    // Point-in-time snapshot. Returns after the file is on disk; per-shard
    // locks are held only for the view freeze and the final rename.
    store.save_snapshot()?;

    // On-demand AOF compaction. Same lock discipline as save_snapshot.
    let _stats = store.rewrite_aof()?;

    // Live introspection.
    let info = store.info();
    println!("{} keys, {} bytes AOF", info.keys, info.aof_bytes);

    Ok(())
}

A fresh embedded store with the default config writes only the AOF — no snapshot file appears until save_snapshot runs. That is expected; the AOF on its own is enough to rebuild the keyspace.

Configuration knobs

Durability and AOF growth

Transactions replay all-or-nothing, at any size. An embedded atomic() / atomic_all_shards() block is bracketed in the AOF by a begin and a commit marker. Replay holds every frame after a begin and applies the batch only when it reaches the matching commit; a log that ends mid-transaction discards it. Rejecting the block (returning Err) appends nothing at all.

The markers are what make this size-independent, and the distinction matters. Group commit alone only defers the fsync — the AOF writes through a 256 KiB buffer, so a longer transaction still hands whole, valid frames to the kernel as that buffer fills, and kill -9 leaves them there. Measured before the markers existed: a 20,000-mutation block killed mid-commit replayed 6,393 of them. No appendfsync setting fixes that, because the partial state comes from the shape of the commit loop, not from when it syncs.

appendfsync governs something different: the power-loss window for data already written. It is orthogonal to whether a transaction is all-or-nothing.

Both are true as of 4.0. Earlier versions documented group commit but never enabled it, so every mutation in a transaction was synced separately AND replayed independently.

v1-format logs have no record envelope and cannot express a transaction boundary; they keep the old behaviour until their first rewrite promotes them to v2.

The markers belong to transactions and to nothing else. A pipelined read batch is not a transaction — Redis pipelining is explicitly non-atomic — so reactor batches carry no markers (they only share one fsync under always; briefly in the 4.x line every single-command batch paid the ~65 B marker pair, which a disk gate caught). On the server, MULTI/EXEC brackets its queued commands on the connection's shard; commands that fan out to other shards land in those shards' own logs individually. Cross-shard EXEC atomicity under crash is therefore per-shard, not global — same spirit as Redis's own rule that a runtime error inside EXEC does not undo the other commands. The all-or-nothing guarantee above is the embedded atomic() family's, whose writes are single-shard by construction.

Trigger surface

fsync policy semantics

Trade-offs and limits

Per-policy throughput vs data loss. Always blocks each reply on fsync; it is the only policy that survives kill -9 with zero command loss, and it cuts SET-heavy throughput roughly in half on typical NVMe. EverySec runs a background flush every second and loses up to that window on a crash — the default precisely because it matches the Redis trade and the lost window is usually tolerable. No lets the kernel decide; throughput is highest but a crash can lose anything still in pagecache, potentially many seconds.

What AppendFsync does and does not govern. It sets the power-loss window for individual commands. It has never had anything to do with whether an atomic block is all-or-nothing — that is transaction markers in the log (see crash-consistency), and it holds under every fsync policy as of 4.0.

This is worth saying plainly because the names invite the opposite reading. A consumer storing financial data selected Always on first contact, reasoning that acknowledged writes must not be lost, and got a setting that costs the most and — before 4.0 — bought nothing at all for the block atomicity they actually needed. Pick Always when you cannot lose a single acknowledged command; pick EverySec when a one-second window is tolerable. Neither choice affects transactions.

AOF replay cost vs snapshot load cost. Without a snapshot, boot time grows linearly with the AOF byte count: a 4 GiB AOF replays in a few seconds on local NVMe, a 40 GiB one in a minute or more. A snapshot caps that — load is one streaming read plus a short tail of post-snapshot AOF — but costs a transient view freeze (O(keys), nanoseconds per key, because collection values are refcount-shared) plus a one-time copy of any collection first mutated while the snapshot is in flight. For write-heavy workloads, prefer leaning on auto-rewrite to keep the AOF bounded rather than running periodic BGSAVEs: rewrite gives you the same boot-time bound with no second file to manage.

Background-job concurrency. Each shard runs at most one background save or rewrite at a time. A duplicate request that arrives mid-job is skipped with a log line, never queued.

AOF writes and the reactor thread. Appends and fsyncs stay off the reactor on every reactor: with io_uring they ride the shard's own ring as queued operations; on the epoll/kqueue reactors a per-shard writer thread drains the same queue with sequential appends (byte-identical on disk). Either way the reactor never blocks in write(2) or fsync(2) on the hot path — which is what used to park it for seconds under GB/s ingest (the 5.0 tail-latency work; measured end-to-end in bench/'s finding documents). Durability is unchanged: the everysec crash window is still ≤ 1 s, and under always a write's reply is held until the fsync covering it completes — the reply itself still guarantees durability, the reactor just no longer waits alongside it, and concurrent connections share fsync rounds (group commit). KEVY_AOF_OFFLOAD=0 restores the classic synchronous path on any reactor.

Rewrites defer under saturating ingest (5.0). A rewrite must fold the writes that land while it runs; when the append rate provably outruns that fold, 5.0 defers the rewrite instead of paying an unbounded stall: the growth rule re-anchors at the current size and retries after the next growth factor. Explicit BGREWRITEAOF is never gated. The observable trade: under sustained write saturation the AOF grows past its usual rewrite point and shrinks once pressure eases — disk is refundable, a stall is not. Around a rewrite you may briefly see <aof>.rewrite (the image under construction) and <aof>.trashN (a hardlink that moves the old log's multi-gigabyte free off the serving thread); both clean themselves up, and orphans from a crash are reclaimed by the next rewrite. Exclude *.rewrite / *.trash* from backups.

TTL persistence. TTLs are written as absolute Unix-millisecond deadlines (PEXPIREAT in the AOF, an absolute field in the snapshot format), so a key keeps its original expiry instant across any number of restarts and the time the process spent down is subtracted correctly. Older AOFs that recorded relative remaining time still load (treated as relative on entry); new writes are always absolute. EXPIREAT and PEXPIREAT are exposed as client commands.

One consequence worth knowing, reported from a consumer's gate: because the persisted deadline is wall-clock time, a system clock step (NTP correction) between the write and a later replay moves the remaining TTL by exactly the step, in either direction — a backwards step reads back a longer remainder, a forwards step a shorter one. This is inherent to absolute wall deadlines (Redis has the same property, in-memory too); kevy's live TTLs run on the monotonic clock and are immune — only the restart boundary converts through wall time. For short rate-limiter TTLs, treat the remainder as accurate to within your host's clock discipline across a restart.

Shard-layout changes are crash-idempotent. Changing --threads / shards writes new snapshots under .reshard temp names, commits via a durable reshard.journal, and rolls an interrupted migration forward on the next start. Source files survive as .premigration.<unix_ts> backups; the journal is the commit point and must never be deleted by hand.

Giant single collections vs the rewrite window. Lists, hashes, sets, and sorted sets past ~16K elements use element-granular (segmented) copy-on-write: a write while a rewrite or snapshot view pins the value clones only the ~16K-element segment it touches, so the cost is about a millisecond and independent of collection size (box-measured 0.4-2.1 ms first-write at 20M elements, where the previous whole-value clone paid 0.35-9.5 s and briefly doubled the collection's resident memory). Streams are the remaining exception: a stream value still clones whole under a pinned view, so an unbounded single-key stream at multi-GB scale can see rewrite-window latency spikes on its shard — trim with XTRIM if you keep one.

What is not persisted. Pub/sub channels, subscriptions, and undelivered messages live only in memory. Blocking-command waiters such as BLPOP and blocking XREAD are connection state, not data. Neither is written to the AOF or snapshot, and neither is replayed.

FAQ

My AOF file is growing — how do I compact it?

Run BGREWRITEAOF on the server or Store::rewrite_aof() in embedded mode. Rewrite rebuilds the log as the minimal command set that reconstructs the current keyspace — one SET / HSET / etc. per key, plus a PEXPIREAT for TTL'd keys — and atomically swaps the new file in. Ten thousand overwrites of hot collapse to a single SET hot <latest>.

For unattended ops, leave auto-rewrite at its defaults — 100% growth over the previous rewrite size, with a 64 MiB floor — and the reactor will fire compaction on its own. Set auto_aof_rewrite_percentage = 0 to disable it and drive rewrite entirely by hand.

Rewrite is non-blocking for the keyspace: serialization and fsync run with reads and writes flowing, and any writes that land during the rewrite are tee'd into a diff buffer that gets appended to the compacted image. If a rewrite crashes midway the original AOF is untouched (the swap is an atomic rename) and the leftover aof-<id>.aof.rewrite temp is safe to delete.

Can I disable persistence entirely?

Yes, in two ways:

• Server: set appendonly = false in kevy.toml (or omit --dir). The server runs as a pure in-memory cache; no aof-* or dump-* files are created.
• Embedded: build a Config without calling with_persist(...). Store::open runs the keyspace entirely in memory; save_snapshot and rewrite_aof become no-ops at the API surface (or surface an error indicating no persistence directory is configured).

If you want persistence but no AOF growth at all between snapshots, that combination is not supported — kevy's durability model is AOF-first, and the snapshot exists to bound AOF replay, not to replace the AOF.

What is the cost of a snapshot during high write load?

The blocking portion is tiny. A per-shard freeze of the keyspace is O(keys), not O(bytes), because collection values are reference-counted and shared with the live store; on a million-key shard the freeze takes single-digit milliseconds. Serialization itself runs with the keyspace live — writes are not paused.

The transient cost you pay is memory. A collection mutated while the snapshot is being written clones the parts the writes touch, so the live store can move on without disturbing the frozen view. For plain string keys and for lists/hashes/sets/sorted-sets (which clone per ~16K-element segment) the extra memory is a few megabytes at worst; only a stream mutated under a pinned view still clones whole and can briefly double that stream's resident size.

A successful snapshot also resets the AOF — the snapshot now carries everything the log used to, and the log restarts with only writes that landed after the freeze. A restart then loads snapshot + log without ever double-applying history.

How is recovery sequenced on the next boot?

For each shard, in order:

1. Load the snapshot. If dump-<id>.rdb exists, stream it into the keyspace. Expired TTLs are dropped during load.
2. Replay the AOF. Read aof-<id>.aof from the front and apply each frame.
3. Handle the tail. A clean file applies in full. A torn or corrupt frame stops the replay at the last complete frame before it; the open then copies the dropped region to aof-<id>.aof.corrupt-quarantine.<unix_ts> (fsynced — after a mid-file corrupt frame that region is mostly well-formed frames, and the copy is the only way back to them) and truncates the file to the last complete frame before the first new append, so new writes stay contiguous with the replayable prefix instead of landing behind the bad bytes (where the next replay would stop again and silently orphan them). Quarantined bytes are never re-applied; inspect or salvage them by hand. If the quarantine copy itself fails (e.g. disk full), the open fails with the file intact — kevy never destroys the only copy of your bytes.
4. Log a one-line summary including wall-clock time:

``text kevy: AOF /data/kevy/aof-0.aof replayed 145313 commands from 418261733 bytes in 247 ms (clean) ``

1. Roll forward any interrupted shard-layout migration by replaying reshard.journal.

Watch the replay-time line and use auto-rewrite to keep it bounded — replay time grows linearly with the unrewritten AOF size.

How do I monitor persistence from inside an embedded host process?

Two surfaces.

Polling. store.info() returns a KevyInfo struct with keys, used_memory, aof_bytes, expire_pending, evictions, expired_keys. Finer-grained helpers cover the same ground:

store.dbsize();                 // live key count
store.ttl(key);                 // Option<Duration> (None = no key / no TTL)
store.ttl_ms(key);              // Redis PTTL semantics: -2 no key, -1 no TTL, else ms
store.expire_pending_count();   // live keys carrying a TTL
store.used_memory();            // resident-bytes estimate
store.expired_keys_total();     // total expired (lazy + reaper)
store.evictions_total();        // total evicted by maxmemory

expire_pending_count() == 0 when you expected TTLs is the classic tell that the TTL subsystem didn't register your keys.

Push. Register Config::with_metric_sink(...) and receive KevyMetric events on AOF replay (startup) and each AOF rewrite (compaction). The sink runs synchronously on the emitting thread (the reaper for background rewrites), so keep the callback fast. KevyMetric is #[non_exhaustive] — always match a _ arm to stay forward-compatible.

What is every file in the persistence directory?

Durability contract (v2.1)

What "the call returned OK" guarantees, per appendfsync × write path. "Durable" = on stable storage (fdatasync completed); "windowed" = in the OS page cache, lost only if the machine (not just the process) dies inside the window.

Store::fsync_aof() is the per-write durability escape hatch (Postgres synchronous_commit-per-transaction genre): run a deployment on everysec for throughput, and place the barrier after the few writes that must survive a machine crash the moment they are acknowledged. Cost: one fdatasync per dirty shard.

Process crash (SIGKILL) never loses acknowledged writes under always and loses at most the fsync window otherwise; the AOF tail is replayed on the next open, and a torn final frame is truncated away on open, never silently applied (see the crash-consistency contract below for the full state machine).

An orderly stop (SHUTDOWN or SIGTERM) loses nothing under any policy: the drain force-fsyncs the AOF tail before exit, so the everysec window that a crash can lose does not apply to a clean shutdown.

Crash-consistency contract (v4)

The open path is a fixed state machine per shard — open → verify → replay → verdict → repair → append — and each verdict carries a hard loss bound. crashgate (bench/crashgate.sh) executes this table: a SIGKILL matrix (mid-append, mid-rewrite, mid-snapshot, mid-feed-emit × fsync policies × shard counts) plus injected torn-tail / mid-file / payload damage.

The no-black-hole invariant (the 3.18 incident's fix, held by crashgate): the truncate-on-open happens before the first append, so the replay stop point never regresses across restarts — a post-crash restart's writes always survive the next restart.

Multi-shard skew. Each shard owns an independent aof-<id>.aof and repairs independently, so after a crash different shards may recover to slightly different moments (each within its own loss bound). kevy makes no cross-shard atomicity promise for independent writes — atomic/atomic_all_shards blocks fsync per touched shard on commit and are the tool when a group of writes must land together.

The feed (CDC) is memory-only and runs ahead of the disk. The feed backlog is not rebuilt from the AOF at open; only its (generation, offset) cursor survives a restart. Frames are emitted at apply time, before the AOF bytes recording the same write are fsynced — under everysec a consumer can observe up to ~1 s of writes that a crash will roll back (always: zero; no: unbounded). A crash bumps the feed generation, so every pre-crash cursor gets -FEEDRESYNC / FeedError::Resync and the consumer must rebuild from a scan of the recovered store. Treat a delivered frame as a durable fact only once the fsync window covering it has closed — side effects taken on not-yet-durable frames cannot be recalled by the resync.

Replicas hold no durable claim over un-fsynced frames. After an unclean primary restart the primary rolls back by its un-fsynced suffix and bumps the feed generation; a replica that applied the rolled-back writes is ahead, and on reconnect its forked history is discarded via a full snapshot resync. The reconnect handshake carries the replica's generation (v4), and the primary refuses to serve offset continuity across a generation mismatch — a replica can no longer be silently fed the new history's same-numbered offsets, no matter how long it waits to reconnect (see replication.md).

The AOF record format (v2, KEVYAOF2)

Since 4.0, new AOF files open with a KEVYAOF2\n magic and carry every command as a checksummed record:

[payload_len: u32 LE][crc32c: u32 LE][payload: RESP multibulk command]

What the envelope buys, each one a class of incident the v1 format could not handle:

• Integrity. crc32c (Castagnoli, hardware-accelerated on aarch64 and SSE4.2 x86-64) covers the payload. A flipped bit — disk rot, a bad cable, a truncated-then-overwritten page — fails the check and the record is refused instead of replaying a tainted value. v1 had no integrity check at all: bit-rot replayed silently.
• Deterministic record boundaries. The length prefix frames the log without parsing RESP, so a torn tail is detected by arithmetic (fewer bytes than the header promises) rather than by a parser happening to choke.
• Deterministic resync. After a corrupt region, the next record boundary can be re-found and verified (length + CRC + exactly one well-formed command must all agree) — the basis of the resync replay below.

Compatibility contract:

• A 4.0 binary reads v1 (KEVYAOF1) files forever. A 3.x data dir opens with zero migration work.
• Formats never mix within one file: appends to an existing v1 file stay v1.
• New files, truncations, and every rewrite output are v2. The first rewrite (auto or BGREWRITEAOF) therefore upgrades a v1 file to v2 — after which 3.x binaries can no longer read it. Downgrade windows and the operator sequence live in UPGRADING.md.

The per-record overhead is 8 bytes; for the mailrs-shaped workload (mixed small commands) the disk cost measures in the low single-digit percent, and the CRC rides the hardware instruction on both server architectures.

Internal frames — and what "the truth set" means

Two kinds of record carry a NUL-prefixed name no client-typed RESP verb can collide with:

• Transaction brackets (\0KEVYTXNBEGIN / \0KEVYTXNCOMMIT) — make a group of appends replay all-or-nothing.
• The segment stitch (\0KEVYSEGMENTED <file>) — records that a batch of rows was evicted from the hot layer into a cold segment file under segs-<shard>/. Replay re-does that eviction: the rows' write frames precede the stitch in the log, replay brings them in and the stitch asks them back out. A write frame after the stitch is a revival and simply stays (hot-first reads shadow the segment copy).

Once a data dir contains segs-<shard>/ directories, the AOF alone is no longer the complete truth. The truth set is: the snapshots (dump-*.rdb) + the AOFs (aof-*.aof) + the segment directories (segs-*/, each holding its segment files and the segs.manifest ledger that makes them real). A backup must copy all three; segments are immutable once sealed, so incremental backup is "copy the new segment files and the manifest".

The manifest — not the stitch frame — is the segment set's source of truth (it is fsynced before the frame is logged). Two consequences: a stitch frame naming a segment the manifest does not hold means the segment directory was damaged after the fact, and startup refuses by name rather than silently dropping the rows whose only durable copy is unreachable; conversely a stitch frame lost to a snapshot truncation or an AOF rewrite is harmless — by then the hot layer no longer holds those rows, or (if a rewrite view froze mid-eviction) the rows survive in both tiers and hot-first reads shadow the segment copy.

Resync replay — recovering the good tail

The default (strict) replay stops at the first corrupt record: the prefix is applied, everything after — mostly well-formed records — is quarantined and dropped from the live file. That is the honest default (a corrupt region means something went wrong; refusing to guess is safest), but for a large log with one small damaged region it surrenders data that is provably intact.

replay_resync opts into recovering it:

[persistence]
replay_resync = true

(or Config::with_replay_resync(true) embedded, or Runtime::with_replay_resync(true) on a hand-built runtime; the setting is boot-time only — replay happens before the first live config tick).

Under resync, replay hops the corrupt region: it scans forward for the next position where the length prefix, the CRC, and a well-formed single-command parse all agree (a false accept requires defeating all three — about 2⁻³² per candidate offset), then resumes applying from there. Every skipped range is reported — ReplayReport::resynced_ranges at the persist layer, OpenReport::resynced_bytes on Store::open_report() — and the corrupt flag stays raised: resync recovers data, it does not declare the file healthy.

Repair semantics differ from strict mode: the on-disk file is truncated (and the tail quarantined) only past the last recoverable record; interior corrupt regions stay in place — hopped again on every boot — until the next rewrite compacts the file. The recovered-good-tail guarantee is executable: crashgate's mid-file splice cell (the mailrs damage shape, 8 bytes spliced out of a 231 MB-class log) must report every post-damage record recovered.

When to leave it off: if your host treats any corruption as a reason to fail over to a replica or restore from backup, strict mode gives you the loudest, earliest stop. When to turn it on: single-node deployments where the AOF is the only copy — the mailrs posture — and losing three days of writes to an 8-byte splice is the worse outcome.

Atomicity charter (embedded serving-store, v2.1)

• Store::atomic(body) — single-shard transaction: takes that shard's write lock for the closure, reads inside see the closure's own writes, AOF appends are deferred and committed with one fsync at commit (under always). Every key touched must hash to the same shard — the blessed serving-store config is therefore 1 shard when your write patterns span arbitrary keys: you keep full atomicity and pay no cross-shard coordination. The ceiling of the 1-shard config is single-core write throughput; measured numbers live in bench/REPORT.md.
• Store::atomic_all_shards(body) — multi-shard transaction: acquires every shard's write lock in shard-index order (deterministic order = no deadlock), commits per-shard AOF batches on return. Cost: blocks all other readers + writers for the closure's duration — use it for cross-shard invariants, not as the default write path.
• Store::pipeline() — NOT atomic: each op takes its own lock; other writers interleave. It batches fsyncs (N ops → ≤ shard-count fsyncs), nothing more.
• Both atomic forms log the effect of conditional ops (ZADD GT, SPOP) as unconditional verbs, so replay and replica-apply are deterministic by construction.

Recovery points (v2.3)

With the change feed enabled ([feed] enabled = true, see cdc.md), every snapshot records the feed cursor it was taken at — frozen in the same no-append window as the snapshot data itself. That yields the recovery-point contract:

snapshot S + the feed frames from S's recorded cursor = the exact state at any later cursor.

kevy_persist::read_snapshot_cursor(path) reads the cursor back (None for pre-v2.3 snapshots — format v4 and older carry no cursor and remain fully loadable). The executable form of the contract is bench/restore-drill.sh, run as a diskgate line: write → SAVE → write more → kill → restore the dumps alone → replay the captured feed frames → byte-exact key-by-key verification.

Scope note: the feed window is the in-memory backlog. Frames older than the window are gone — a snapshot older than the window's reach is a plain snapshot restore (state at S), not a PITR base. Take snapshots at least as often as the window turns over if you rely on exact-point recovery.

The snapshot on the replication wire (v3.15)

The same snapshot format is what a primary in-line-ships to a replica that has fallen past the backlog window (see replication.md). One semantic worth knowing: a shipped snapshot replaces the replica's local state, it does not merge — the replica flushes its keyspace before loading. That is deliberate: when a rejoining ex-primary carries a forked suffix (writes that were never replicated), the resync must genuinely discard the fork rather than leave it as residue under an upsert-only load.

One consequence of that replace-not-merge semantic for a replica running with its own AOF: the flush + snapshot load bypass the commit path, so after the load the replica synchronously rewrites its local AOF from the post-resync keyspace (4.0). Without that, the local log would still describe the pre-resync history, and a restart with the primary unreachable would serve a state assembled from the wrong base.

Operator runbook

The incident-shaped checklist. Every line here is backed by a gate (crashgate, repligate, diskgate) or a test named in this doc.

Stopping kevy. Prefer an orderly stop — SHUTDOWN / SIGTERM (server) or dropping the last Store clone / Store::shutdown() (embedded). The drain force-fsyncs every AOF tail, so an orderly stop loses nothing under any fsync policy, and the feed writes its clean-shutdown marker so consumers and replicas resume without a generation bump. kill -9 is survivable (that is crashgate's whole matrix) but spends the fsync window and breaks feed/replica continuity (generation bump → consumers rebuild, replicas snapshot-resync).

After every (re)start, read the verdict. Three equivalent surfaces; wire at least one into your health checks:

• The boot log line per shard — kevy: AOF … replayed N commands from M bytes in T ms (clean). Anything other than (clean) comes with a WARN naming dropped bytes and the quarantine path.
• INFO persistence on the server: aof_last_open_dropped_bytes and aof_last_open_corrupt — nonzero means this boot recovered less than the files held. Alert on it; the 3-day silent-loss incident was exactly this signal living only in stderr.
• Store::open_report() embedded (or kevy_open_report over the C ABI): dropped_bytes, corrupt, quarantine_paths, resynced_bytes as data — turn a bad boot into a refused deployment instead of a log line nobody reads.

If a boot reports drops. The dropped region is in aof-<id>.aof.corrupt-quarantine.<unix_ts>, byte-exact. Decide in this order: (1) if a replica or backup has the writes, restore from there; (2) if not, consider a one-time boot with replay_resync = true — on a mid-file corruption it recovers the good tail and reports the skipped ranges; (3) hand-salvage the quarantine file (it is mostly well-formed records / RESP). Keep the quarantine file until the incident is closed; kevy never deletes it.

Rewrite cadence. Auto-rewrite is the boot-time and incident bound: replay time, quarantine blast radius, and resync hop cost all scale with unrewritten log size. The growth pair (auto_aof_rewrite_percentage / min_size) is the default; add the absolute cap (auto_aof_rewrite_bytes) when disk or replay budgets are hard, and the staleness trigger (auto_aof_rewrite_interval_secs) for write-light deployments whose logs never double but carry weeks of history. All three are independent; first to fire wins.

Disk-full and quarantine failures fail loudly. If the open cannot write the quarantine copy (e.g. disk full), it fails with the AOF intact rather than truncating the only copy of your bytes. Free space, then reopen.