kevy's cluster surface has two independent layers — single-node multi-shard exposure (one process, every shard speaks Redis Cluster) and multi-node replication + scoped multi-writer (primaries, replicas, embeds, quorum failover) — and you can run either, both, or neither.

The two layers at a glance

Single-node cluster mode. One kevy process partitions its keyspace across N shards and exposes each shard as a virtual cluster node on a deterministic per-shard port. CLUSTER SLOTS / SHARDS / NODES report the real CRC16 partition; key-aware clients (redis-cli -c, redis-benchmark --cluster, stock cluster-aware libraries, and the bundled ClusterClient) hash each key and connect straight to the owning shard. The win is mechanical — removing the server-side cross-shard hop translates directly into higher throughput and lower tail latency.

Multi-node cluster. A kevy server can act as a primary streaming a write log to one or more replicas (either kevy servers or in-process kevy-embedded stores). A primary can also delegate scoped writes by prefix: [cluster] scopes declares which node owns writes for app:billing:*, which owns app:auth:*, etc.; writes that land on the wrong node receive -MISDIRECTED writer is <host:port> so the client follows. kevy-elect provides the quorum control plane: it flags a dead node DOWN, elects a replacement primary for the replication topology (v3.15), and promotes a scope's declared fallback. Operator-issued MOVE-SCOPE migrates a prefix under a quiesce window.

When you need this

The two layers compose: a primary in cluster mode advertises N shards, each replica also runs N shards, and a routing client wires them together.

Layer 1 — Single-node cluster mode

Core idea

A normal kevy process accepts every command on a single port and internally forwards mis-routed keys to the shard that owns them. That forward is correct, but on a hot path it dominates p99 latency and caps throughput. Cluster mode exposes each shard at its own port; a key-aware client hashes the key with CRC16-XMODEM, looks up the owner shard from CLUSTER SLOTS, and connects straight there — no forward, no -MOVED.

                  ┌─────────────────────────────────────────┐
                  │            kevy process (1 host)        │
                  │                                         │
  main port  ───▶ │  6004  ── proxy: forwards or -MOVED ──▶ │
                  │                                         │
  shard ports ──▶ │  6005  ── shard 0   (slots     0– 4095) │
                  │  6006  ── shard 1   (slots  4096– 8191) │
                  │  6007  ── shard 2   (slots  8192–12287) │
                  │  6008  ── shard 3   (slots 12288–16383) │
                  └─────────────────────────────────────────┘

Shard i always binds port_base + 1 + i (override port_base via TOML). The main port keeps the proxy behaviour for clients that don't speak cluster; per-shard ports answer -MOVED <slot> <host:port> when a key arrives at the wrong owner.

Whole-keyspace commands (KEYS, SCAN, DBSIZE, FLUSHALL) stay whole-keyspace on every port — kevy fans them out internally so a client doesn't have to.

Enabling it

# kevy.toml
[server]
port = 6004

[cluster]
enabled   = true
# port_base = 6004   # defaults to `port`; shards live at port_base + 1 + i

Equivalent CLI / env:

kevy --port 6004 --threads 8 --cluster      # shard ports 6005..6012
KEVY_CLUSTER=1 kevy --port 6004 --threads 8

Switching a data directory in or out of cluster mode re-homes the keys once at startup; the prior files are backed up as *.premigration.<ts>.

Using ClusterClient from Rust

[dependencies]
kevy-client = "*"

use kevy_client::ClusterClient;

// Seed against any cluster port; topology is discovered via CLUSTER SLOTS
// and one connection is opened per shard.
let mut cc = ClusterClient::connect("127.0.0.1", 6005)?;

cc.set(b"user:42", b"alice")?;
let v = cc.get(b"user:42")?;            // routed to user:42's owner shard
let n = cc.incr(b"counter")?;

// Multi-key DEL/EXISTS — routed per key and summed.
let removed = cc.del(&[b"a", b"b", b"c"])?;
# Ok::<(), kevy_client::KevyError>(())

A runnable seed example lives at crates/kevy-client/examples/cluster.rs; a benchmark at crates/kevy-client/examples/cluster_bench.rs.

How routing removes the cross-shard hop

1. Discover. connect sends CLUSTER SLOTS to the seed, reads each shard's [start, end, host, port], and builds a 16384-entry slot → shard-index table. The table comes from the server's advertised ranges, so the client never reimplements the partitioning arithmetic.
2. Route. Every single-key command computes key_hash_slot(key) (CRC16-XMODEM over the {hashtag} if present, else the whole key) and sends straight to that slot's owner connection.
3. Fan-out where needed. dbsize, flushall, and other whole-cluster commands are handled server-side; the client issues one call.

On a 16-core Linux box with GET at concurrency 64, removing the cross-shard hop lifts measured throughput from 333 k ops/s to 533 k ops/s (1.6×) and drops p99 from 3858 µs to 260 µs (~15× lower tail). Reproduce with cargo run -p kevy-client --release --example cluster_bench.

The hop's cost only shows up under load on a clean machine. On a small co-located cloud VM the difference is buried in scheduling noise.

Cross-slot multi-key commands

Unlike Redis Cluster, kevy does not return -CROSSSLOT when a multi-key command (MGET, MSET, SUNION, transactions, blocking fan-outs) spans shards on a single-node cluster: the server fulfils the request across shards. kevy is a superset of Redis Cluster on a single machine — every Redis Cluster client works, plus the surface you would have hit -CROSSSLOT on still works. A shared {hashtag} is still the right tool when you need data co-located for atomicity, but it is no longer required for correctness.

CLUSTER commands supported on a cluster port

Falling back to raw routed helpers

// Route an arbitrary single-key command to its owner shard.
let reply = cc.request_keyed(b"mykey", &[b"STRLEN".to_vec(), b"mykey".to_vec()])?;
// Keyless commands go to any shard.
let reply = cc.request_unkeyed(&[b"PING".to_vec()])?;
# Ok::<(), kevy_client::KevyError>(())

ClusterClient wraps the common verbs across strings, hashes, lists, sets, sorted sets, pub/sub, and the multi-key DEL / EXISTS. Pub/sub is process-global: a Subscriber on any port sees every published message regardless of which shard accepted the PUBLISH.

Layer 2 — Multi-node cluster

Primary and replicas

A kevy server can be a primary streaming its write log, or a replica that mirrors one (the default role is standalone — replication dormant). The primary binds a dedicated replication listener per shard; replicas connect, hand over their last-applied offset, and apply the streamed frames onto local shards. Chain replication (replica-of-replica) is not supported.

# primary.toml
[server]
port = 6004

[replication]
role             = "primary"
listen_port_base = 16004    # optional; defaults to port + 10000

# replica.toml
[server]
port = 6004

[replication]
role     = "replica"
upstream = "primary.local:16004"

Full server-side semantics — backlog sizing, snapshot ingest, heartbeats and ACKs — live in docs/replication.md; failover (planned FAILOVER and crash elections) and the consistency ladder are docs/availability.md. The relevant fact for this document is that the same wire protocol carries cluster-mode replication: a primary running with [cluster] enabled = true streams N shards' worth of writes, and a replica running with the same shard count applies them shard-for-shard.

Embed as read-replica

A kevy-embedded store can subscribe to a primary directly and serve in-process reads with zero network hop. Writes are refused locally with READONLY.

use kevy_embedded::Store;

// In-memory replica, AOF off, default reconnect (100 ms → 5 s).
let replica = Store::open_replica("primary.local:16004")?;

let v = replica.get(b"hello")?;
assert!(replica.set(b"k", b"v").is_err());      // KevyError::ReadOnly
# Ok::<(), kevy_embedded::KevyError>(())

For tuning:

use std::time::Duration;
use kevy_embedded::{Config, Store};

let cfg = Config::default()
    .with_replica_upstream("primary.local:16004")
    .with_replica_id("backup-svc-region-a")
    .with_replica_reconnect(Duration::from_millis(50), Duration::from_secs(10));
let replica = Store::open(cfg)?;
# Ok::<(), kevy_embedded::KevyError>(())

The handshake sends REPLICATE FROM <last-applied-offset> ID <replica_id>; the primary acks the offset and streams frames. The runner thread is joined when the last Store clone drops, so the primary observes a clean FIN and frees the slot. PUBLISH is allowed locally on the embed (pub/sub is process-local), but the keyspace itself remains read-only.

Scoped multi-writer

Scoped multi-writer splits writes by key prefix across nodes. Every node knows the full ownership table from static config; writes that land on a non-owner answer -MISDIRECTED writer is <host:port> so the client retries against the right node.

# Same config block on every member.
[cluster]
node_id = "embed-billing-1"
peers   = "embed-billing-1@10.0.0.1:6004,server-eu-1@10.0.0.2:6004,reader-1@10.0.0.3:6004"

# prefix=writer[|fallback], comma-separated.
# The first `=` splits prefix from owner spec, so `app:billing:` (with `:`) is fine.
scopes  = "app:billing:=embed-billing-1|server-eu-1, app:auth:=embed-auth-1"

elect_port_base = 16100    # kevy-elect listens here

peers is a flat string of <node_id>@<host>:<port> entries — no nested structure, easy to template. scopes is parsed prefix=writer[|fallback], comma-separated. A node with no scope ownership simply forwards writes; a node that does own a scope accepts writes for it and rejects others.

Reads are independent of scope ownership — any node holding the data (typically a read-replica) can serve them. The scope mechanism is for write attribution only.

Embed as scoped writer

use kevy_embedded::{Config, Store};

let writer = Store::open(
    Config::default().with_embed_writer("0.0.0.0:6105")
)?;

// Local writes feed the embed's replication source backlog;
// readers connect to 0.0.0.0:6105 via kevy_replicate::ReplicaClient.
writer.set(b"app:billing:invoice:42", b"...")?;
# Ok::<(), kevy_embedded::KevyError>(())

The embed exposes a replication listener on the address passed to with_embed_writer. Other nodes pull the log from there exactly as they would from a server primary.

kevy-elect quorum failover

kevy-elect is the in-process quorum control plane every cluster member runs when [cluster] node_id and peers are both set. Each node binds one TCP control listener at elect_port_base (default: client port + 200) and heartbeats every peer; a peer silent past down_after (5 s) is flagged DOWN. Membership is static (the operator-declared peers table); roles are dynamic — elections move the primary around inside that table. The election timings (hb_interval 200 ms, down_after 5 s, election_timeout 3 s) are fixed constants in this release, not config keys.

It drives two failover surfaces:

Replication primary election (v3.15). When the current primary is DOWN, an eligible replica — the one holding the highest applied replication offset among alive peers, lowest node_id breaking ties — starts a candidacy and needs quorum (N/2 + 1) ACCEPTs within election_timeout. Epoch and vote are persisted to <data_dir>/elect.meta before any answer leaves the node, so a crash-restart can never double-vote. The winner opens writes; losers retarget their replication upstream automatically. Three clamps back this up: a cold start with no known primary elects after one down_after grace window; a restarted quorum member holds writes until an election settles (config role = "primary" is only a preference); and a primary that cannot see a strict majority fences its own writes within one lease window (-NOREPLICAS primary lost quorum; writes fenced). The operational walkthrough is docs/availability.md.

Scope fallback. A scope's declared fallback consults the DOWN set on every accepted write: if its writer is DOWN, the fallback treats itself as the active owner and accepts the write; the next write on every other node now MISDIRECTs to the fallback. When the original writer's HBs resume, it leaves the DOWN set and the fallback steps down implicitly on the next decision.

Manual rejoin recovery

If the original writer was DOWN long enough that the fallback accepted writes, those writes only live on the fallback. Before re-enabling the original writer for the scope: stop the writer, copy the fallback's data directory into the writer's, then restart. This stays inside the no-consensus contract — no shadow-writes, no double-acceptance.

MOVE-SCOPE

MOVE-SCOPE migrates a prefix from one writer to another under a bounded quiesce window. It is operator-issued and runs on the current writer.

MOVE-SCOPE <prefix> from <from-node-id> to <to-node-id>

Step by step:

1. The current writer flips local state for <prefix> to MIGRATING. Subsequent writes to keys under the prefix return -QUIESCED migrating to <to-host:port>. Clients back off briefly and retry.
2. The writer serialises the prefix's keyspace slice and ships it via MOVE-SCOPE-INGEST <prefix> <bulk> to the target's data port. Every value type moves with full fidelity — including streams, which carry their entries, last_id bookkeeping, consumer groups, consumers, and live pending (PEL) rows across the move (only tombstone PEL rows for already-deleted entries are dropped; no RESP verb can recreate those).
3. On +OK from the target, the writer commits the migration locally. Future writes for the prefix on the source now return -MISDIRECTED writer is <to-host:port>.
4. Other cluster members continue routing per their static scopes config until the operator pushes new config and restarts.

The two wire replies you will see during a move:

A cluster-aware client caches the per-key target on -MISDIRECTED and retries transparently; on -QUIESCED it should sleep briefly (single-digit hundreds of milliseconds) before retrying.

Aborting mid-ship reverts to the source writer; no partial-apply state is left on the target.

Configuration reference

Single-node cluster mode

Replication

TOML-only — there are no replication CLI flags or env vars:

The full key list (backlog sizing, replica_read_only, replica_max_staleness_ms, min_replicas_to_write, single_source) is in docs/replication.md.

Scoped multi-writer + elect

Election timings (heartbeat 200 ms, DOWN after 5 s, election timeout 3 s) are fixed constants, not config keys.

Trade-offs and limits

• Single-node cluster mode is a single process. It buys client-side key routing, not host-level fault tolerance. Add replicas for that.
• The proxy port stays available. It will keep working for non-cluster clients and remains correct, just with the cross-shard hop.
• Membership is static; roles are dynamic. peers and scopes are read from config at startup — a membership change is "push new config, restart", and there is no gossip, by design. Within that static table, elections move the primary role around at runtime (v3.15); failed nodes are never auto-replaced.
• MOVE-SCOPE quiesces writes for the prefix. The window is bounded by slice-ship time, which is single-digit seconds for GB-class scopes over LAN. For prefixes much larger than that, schedule during a maintenance window.
• Embed as scoped writer is sized for service-shape workloads (a billing service, an auth service), not multi-TB datasets.
• Manual rejoin recovery after fallback acceptance. Copy the fallback's data dir into the writer's before re-enabling; no automatic consensus catch-up.

Out of scope by design

• Mutating CLUSTER subcommands (RESET, SETSLOT, FORGET, MEET, …) are accepted as no-op +OK: kevy has no dynamic membership to mutate (membership is "push new config, restart"), and defensively-probing clients should proceed rather than error. The read-only subcommands always report the true static topology.
• AUTH and TLS — handled by the deployment edge (sidecar, mesh, LB), not by kevy.
• Multi-DC active-active and CRDTs.
• Raft, Paxos, or any consensus log under the keyspace.
• Gossip-based discovery — peers is static.
• Online resharding, MIGRATE, ASK redirection.
• Multi-master with overlapping ownership — every prefix has exactly one writer at a time.

These will not be added. The simplicity is the feature.

FAQ

Do I need cluster mode to use replication? No. Single-node cluster mode and the replication / multi-node layer are independent. A non-cluster primary can have non-cluster replicas; a cluster primary can have cluster replicas. They compose but neither requires the other.

Can I run a standard cluster-aware client (Lettuce, ioredis, redis-py-cluster) against a kevy in cluster mode? Yes. CLUSTER SLOTS / SHARDS / NODES advertise a real partition and -MOVED fires on a wrong-shard hit, which is the entire surface those libraries depend on. Stick to the per-shard ports (not the main proxy port) so the client's routing is what reaches the shards.

What happens to a multi-key command across shards in single-node cluster mode? It succeeds. kevy executes cross-slot MGET, MSET, SUNION, transactions, and blocking fan-outs server-side rather than returning -CROSSSLOT. {hashtag} co-location is still useful for atomicity-sensitive cases but is no longer a correctness requirement.

How do I survive a writer crash without an operator? For the replication primary: configure the [cluster] block (node_id, elect_port_base, peers) on every node — a dead primary is detected after down_after (5 s) and the most-advanced replica is elected in its place; a rejoining ex-primary demotes and resyncs automatically (see docs/availability.md). For a scoped writer: declare a fallback (prefix=writer|fallback) — when the writer misses heartbeats past down_after, the fallback starts accepting that prefix's writes, clients receive -MISDIRECTED writer is <fallback> and follow, and when the original writer comes back you run the manual rejoin recovery.

Why is gossip / Raft permanently out of scope? The cost of a consensus log under every write would erase the throughput and tail-latency advantages that make kevy worth choosing. The static-config + quorum-heartbeat design gives you the failover branch without paying for state-machine replication on the hot path. If your workload genuinely needs a consensus-backed key-value store, kevy is the wrong tool.