git:20261104.351f420 to git:20261105.4a85e6f

# meta-srv — Agent & Contributor Guide

Navigation aid for `src/meta-srv`. Keep it short and point to code. Paths are

relative to the repo root.

Repo-wide rules that apply here: [`.agents/architecture-invariants.md`](../../.agents/architecture-invariants.md).

## What this crate does

Metasrv is the metadata and coordination service for distributed mode: it

persists cluster/table/region metadata, runs leader election, drives the

heartbeat loop that tracks cluster topology, and orchestrates distributed

procedures (DDL, region migration, repartition). Data models, the KV backend

abstraction, election, key encoding, and the DDL manager live in `common-meta`;

`meta-srv` implements the server, state machine, and control logic on top.

## Module map

| Module | Path | Purpose |

| --- | --- | --- |

| `bootstrap` | `src/meta-srv/src/bootstrap.rs` | Wires up KV backend, election, gRPC services, procedure manager |

| `metasrv` | `src/meta-srv/src/metasrv.rs`, `src/meta-srv/src/metasrv/builder.rs` | `Metasrv` struct, `MetasrvOptions`, `MetasrvBuilder`, startup |

| `state` | `src/meta-srv/src/state.rs` | Leader/Follower state machine and transitions |

| `handler` | `src/meta-srv/src/handler/` | Heartbeat handler chain (`HeartbeatHandler`): leader check, region lease, stats, mailbox |

| `service` | `src/meta-srv/src/service/` | gRPC services (`heartbeat`, `procedure`, `cluster`, `store`) and HTTP admin |

| `procedure` | `src/meta-srv/src/procedure/` | Distributed procedures: `region_migration/`, `repartition.rs`, `wal_prune/` |

| `region` | `src/meta-srv/src/region/` | Region supervisor, lease keeper, failover triggers |

| `discovery` | `src/meta-srv/src/discovery/` | Node discovery and lease-backed node info |

| `pubsub` | `src/meta-srv/src/pubsub/` | Heartbeat topic publish/subscribe support |

| `gc` | `src/meta-srv/src/gc/` | Metadata-driven garbage-collection procedures and scheduling |

| `selector` | `src/meta-srv/src/selector/` | Region placement strategies (round-robin / load-based / lease-based) |

| `peer` | `src/meta-srv/src/peer.rs` | Peer allocation through selectors |

| `cluster` | `src/meta-srv/src/cluster.rs` | `MetaPeerClient`: internal RPC with leader fallback |

| `cache_invalidator` | `src/meta-srv/src/cache_invalidator.rs` | Pushes cache-invalidation to frontends/datanodes |

| `key` | `src/meta-srv/src/key/` | Metasrv-side KV key encoding |

## Core flows

- **Heartbeat**: `service/heartbeat.rs` receives datanode/frontend heartbeats and

runs the handler chain in `handler/`; responses carry mailbox messages (DDL

results, cache invalidation). Node information is refreshed in memory on each

heartbeat. Datanode statistics collection and flownode remapping share

`handler/utils.rs` for per-node address persistence. Datanode address

updates retain the statistics handler's per-node ordering. Address epochs are

confirmed only after a successful save; cache invalidation remains best effort.

A changed datanode address invalidates every table on that node (`__dn_table`),

not only the tables in the current heartbeat.

- **Datanode process fence**: before the handler chain, the leader's heartbeat

session checks a datanode request's non-zero `node_epoch` against the highest

one recorded under `__datanode_epoch`, read and raised through the leader

cache. A lower epoch gets a reply without running the chain; requests that

carry an epoch take a per-node lock through the check and the chain; epoch 0

(handshakes, older datanodes) is not checked. A datanode stream registers

its pusher on its first request that passes, not on the handshake; pushers

carry the epoch, and a lower one cannot replace a higher one.

- **DDL**: `service/procedure.rs` (`ddl`) is leader-only and hands off to

`common-meta`'s `DdlManager`, executed via `common-procedure`.

- **Region migration**: `procedure/region_migration/manager.rs` plus per-step

files; triggered by failover or an explicit command.

- **Leader election**: via `common-meta`'s election; transitions in `state.rs`.

## Public surface

Clients use the `meta-client` crate (`MetaClient`). On the server side the entry

points are the gRPC services under `src/meta-srv/src/service/` and the HTTP admin

API under `src/meta-srv/src/service/admin/`.

## When you change X, also touch Y

- **Heartbeat interval / leases**: timing constants live in `common-meta`

(`distributed_time_constants`); changing them affects lease and failover timing

in `handler/region_lease_handler.rs` and `region/supervisor.rs`.

- **Procedures**: state must stay persisted in the KV backend and `execute()`

must be idempotent for crash recovery.

- **Cache invalidation**: new metadata types need matching invalidation in

`cache_invalidator.rs` and the heartbeat publish handler.

- **Affinity**: `common-meta/affinity.rs` defines requirements, matching, batch

resolution, and the query allocation-context provider. The selector factory

optionally supplies a resolver shared by DDL allocation and the region

supervisor; without one, allocation skips source lookup. DDL configurators can

install a query provider before procedure loaders are registered. Repartition

and failover resolve source requirements before selection. Each selector allows

one unconstrained retry on candidate shortage when its context permits it.

Failover always permits this retry and continues without affinity when source

resolution fails or returns no requirements; DDL/repartition retain their own

policy. Candidate discovery errors still propagate. Procedure state is unchanged.

- **Selector stats fields**: keep `selector/` implementations in sync with the

stats they consume.

## Testing

```bash

cargo nextest run -p meta-srv

# mock election / KV backend:

cargo nextest run -p meta-srv --features mock

```

Helpers: `src/meta-srv/src/mocks.rs`, `src/meta-srv/src/test_util.rs`,

`src/meta-srv/src/procedure/test_util.rs`.

## Gotchas

- Most mutating operations are leader-only — guard them (see the leader check in

`handler/` and `service/procedure.rs`). Non-leaders return "not leader".

- In-memory state is not durable; anything needed after a leader change must be

persisted to the KV backend. The leader uses a cached KV backend that resets on

leader change.

- Region lease renewal timing and the supervisor's check interval must be

coordinated, or a region can appear active on two datanodes.

## Maintenance contract

Update this file when you add a procedure, change the heartbeat handler chain,

change leader-only boundaries, or alter the metasrv ↔ common-meta split.