cache-sharding-and-replication git:20260911.37b054bA

Immutable. This exact content never changes and is served at /api/v1/blob/157b46d2359bd90b.

---
name: cache-sharding-and-replication
description: >
  Topology for a cache that no longer fits one node: client-side sharded, proxy-fronted,
  clustered, and fully replicated, compared on failure behaviour, cost and client
  complexity; and why a read after a write on a replicated cache is not read-your-writes.
  Estimates origin load when a cache node fails from its measured request share and the
  surviving copies, routing and capacity — mitigated by
  replication, warming, coalescing and admission control. Use when choosing between client
  sharding, a proxy and cluster mode, when a cache node loss or rolling restart took the
  database with it, when replicas of a cache disagree, or when deciding between sharding the
  cache and replicating all of it. Does not cover whether to cache, TTL, stampede or
  invalidation (caching-strategies), the key-to-node mapping (consistent-hashing), a single
  hot cache key (hot-partitions-and-rebalancing), entry serialisation cost
  (serialization-performance), or what a replicated read observes (consistency-models).
---

# Cache Sharding And Replication

## Purpose

Decide how a cache is laid out across nodes, and what happens when one of those nodes goes
away. This is a topology skill only: whether to cache, how long to keep an entry, and how to
invalidate it are `caching-strategies`, and everything here assumes those decisions are
already made.

The failure this prevents is the one that never looks like a cache incident. A cache node is
restarted for a routine upgrade; with N balanced nodes, consistent hashing and no replicas,
about 1/N of the keyspace loses its cached copy. After fail-fast remapping, requests for those
keys can reach the origin until refill, while survivors continue serving. The cache tier
reports a modest dip in hit rate. The database saturates. Nobody investigating the database
is looking at the cache, because the cache is up.

## Workflow

Before recommending changes, reuse the request, configuration and existing measurements: cache
product/version, relevant clients and runtimes, routing/retry configuration, replica placement,
working-set bytes, per-node request share and origin capacity at the required SLO. Ask only about
unresolved constraints that change the decision, such as permitted stale reads or replica locations;
continue independent analysis with explicit assumptions. This skill is language-independent and
declares no Java baseline or executable Java examples; do not infer support for a client feature
or authorize an upgrade. With missing measurements, provide conditional estimates and the exact
measurement needed, not a production sizing or confirmed incident diagnosis.

1. **Classify the cache first: performance or availability.** If the origin cannot serve the
   full request rate with the cache empty, the cache is an availability component. Separate this
   from durability: the cache does not become the authoritative data store by being essential.
2. **Do the node-loss arithmetic before choosing a topology.** Measure the request share owned by
   each node; `total_rate / N` is only the uniform approximation. Losing node `i` can send its
   request share to the origin, plus secondary evictions and retries. Compare rate, concurrency,
   query mix and duration to the origin's measured capacity. The worked example is
   `references/node-loss-and-origin-protection.md`.
3. **Choose sharding or full replication from the working set.** If the whole working set
   fits comfortably in one node's memory and reads dominate, replicating everything can remove a
   network hop only when the replica is process-local. It avoids key loss after a node failure if
   routing and remaining capacity work, and costs roughly `N ×` value memory plus metadata.
4. **Choose the topology** — client-sharded, proxy, or clustered — on operational cost and
   client complexity and measured end-to-end latency. The comparison is `references/topologies.md`.
5. **Set the replication factor from step 2**, not from a default. Replication exists here to
   keep the shard served when a node dies; if the arithmetic says the origin survives a node
   loss, RF=1 is a legitimate, cheaper answer. RF counts all copies including the primary;
   place them across the failure domains being protected and check promotion/quorum requirements.
6. **Exercise failure under load**—crash, partition/timeout, promotion and rejoin—and assert bounds
   on origin rate/concurrency, client errors and recovery, not only cache hit rate.
7. **Add a local L1 only for a measured reason**, and accept that invalidation now has to
   reach every instance's L1 as well as the shared tier.

## Decision block

```text
Sharded cache (each key has an owner shard, with optional replicas) when:
- the working set exceeds one node's memory, or memory cost makes N copies unattractive
- writes and invalidations are frequent enough that keeping N copies converged is work
Fully replicated cache (every node holds everything) when:
- the working set fits one node's memory with headroom, reads dominate heavily, and the
  value of surviving node loss exceeds N × memory; only process-local copies remove the hop
- typically the shape of small reference data: feature flags, rates, configuration
Replicate each shard (RF > 1) when:
- the measured node-loss arithmetic exceeds origin headroom
- or one shard is read-hot and the product can route reads to replicas within the required
  consistency model
Keep RF = 1 when:
- the origin demonstrably absorbs a node loss, and the memory is better spent on a larger
  working set; replicas can also serve reads if the product and consistency contract allow it
Prefer a proxy or a clustered cache over client-side sharding when:
- its routing/discovery protocol reduces coordination across numerous or polyglot clients;
  check actual client capabilities, because dynamic client-side membership can also avoid deploys
Prefer client-side sharding when:
- clients are few and share a runtime, and the extra network hop is a measurable share of
  the cache's own latency — the point of a cache is that it is fast
Do not add a cache node to fix a hot key:
- one key has one owner under every mapping function (hot-partitions-and-rebalancing)
```

## Rules

- **Losing a cache node is an origin-load event.** Size the origin, or the protection in
  front of it, for the loss of one cache node — that is a routine occurrence (upgrade,
  eviction, spot reclaim), not a disaster scenario.
- A rolling restart can cause repeated remapping or replica promotion. Gate each next restart
  on origin headroom, client SLOs and restored replica readiness, not just recovered hit rate.
- Stable placement limits movement: changing N in `hash(key) % N` can remap a large fraction,
  depending on the divisors and node-index mapping. The moved keys' request share, surviving
  copies and refill policy determine miss traffic; compare resulting origin demand with measured
  capacity before predicting an outage. Consistent hashing is one option; fixed slots can also
  preserve placement. The mapping function belongs to `consistent-hashing`.
- Node loss has a **second-order** cost when keys remap and refill on survivors: their
  memory did not grow, so insufficient headroom can raise evictions on previously healthy shards.
  Measure this effect rather than assuming the hit-rate dip equals the lost key share.
- **Replication alone does not give read-your-writes.** With asynchronous replication, a write
  acknowledged by one replica and a read served by another may return the old value. If the requirement is that a
  user sees their own change, route to a copy known to have applied that write, or bypass stale
  cache copies and read an origin endpoint that supplies the guarantee. Arbitrary replica affinity
  and invalidation alone are insufficient; promotion may lose an acknowledged write.
  `consistency-models` owns the guarantee, including the behavior across failover.
- Replication guarantees are product/configuration-specific. Asynchronous replicas have no useful
  convergence deadline unless lag is bounded and monitored. TTL bounds how long a missed
  invalidation can survive only if expiry forces a correct reload; it is not a consistency proof.
- Client-side sharding puts the topology in every client. Adding a node means every client
  must use compatible node lists, virtual-node counts and hashes; overlapping versions need
  an explicit migration/invalidation protocol. A
  disagreement is two clients writing the same key to two different nodes, and both of them
  may read stale. Distribute membership through one source, versioned; versioning alone does
  not make an overlapping rollout coherent.
- Replica and L1 placement must respect data-access and residency constraints. Preserve tenant
  and authorization distinctions in keys and access checks; a key prefix is not access control.
  Check whether one tenant's refill can consume the shared origin budget before widening replication.
- A proxy costs one extra network hop on the cache path, which is the path chosen for being
  fast. Measure the hop against `T_source` before rejecting it: a fraction of a millisecond
  in front of a source costing tens of milliseconds is usually the right trade, and it buys
  topology changes without client deploys.
- A clustered cache with server-owned placement moves membership out of application config.
  Cross-slot multi-key semantics vary by product; Redis Cluster rejects many such operations,
  while other systems coordinate them at extra latency/availability cost. Check the exact command
  and failure contract against the access pattern.
- **A near-cache (local L1 in front of the shared L2) is a second cache with its own
  coherence problem**, and it is per-instance. An L2 acknowledgement alone does not establish
  that every L1 is current; a product may propagate invalidations, with delivery/loss-recovery
  semantics to verify. `caching-strategies` owns that protocol and the source-age budget across
  layers; the topology consequence is up to `instances caching the key + RF` copies per key,
  excluding temporary migration copies.
- Every entry crossing the network is serialised, so the value size is a throughput decision,
  not a detail. A large value multiplied by the fan-out of a warm-up is a network incident —
  `serialization-performance` owns the format cost.

## Deliverable

Scale the result to the task. For a topology decision, return the recommendation (including
keeping the current design), the materially relevant alternative and why it loses, measured
inputs versus assumptions, node-loss origin-load estimate, replica placement/read policy, and
failure-test acceptance bounds with rollout abort criteria. State what evidence would change
the decision. Stop discovery when the remaining unknowns do not change it; otherwise name the
specific measurement or experiment needed. An incident diagnosis need not choose a replacement
topology: distinguish observed timing/counters from the cache-loss hypothesis and name evidence
that would refute it. State which checks actually ran; a paper estimate is not demonstrated
failure tolerance.

## Primary sources

- [Redis Cluster specification](https://redis.io/docs/latest/operate/oss_and_stack/reference/cluster-spec/)
- [Redis replication](https://redis.io/docs/latest/operate/oss_and_stack/management/replication/)
- [Hazelcast 5.6 Near Cache](https://docs.hazelcast.com/hazelcast/5.6/cluster-performance/near-cache) —
  an example of invalidation propagation and loss reconciliation, not a universal L1 guarantee.
- [Amazon Dynamo paper](https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

## References

- [Cache topologies](references/topologies.md) — client-side sharded, proxy-fronted,
  clustered and fully replicated compared on failure behaviour, operational cost, client
  complexity and consistency, with the near-cache layer and a decision table. Read when
  choosing or changing a topology, or when a client library's sharding is in question.
- [Node loss and origin protection](references/node-loss-and-origin-protection.md) — the miss
  storm computed from real numbers, replication factor as the lever, warming, coalescing,
  origin admission control, and the kill-a-node-under-load test with the bound it asserts.
  Read before sizing a cache tier, after any incident where the origin saturated, or when
  planning a cache upgrade or restart.