v1.0.0 to v2.0.0
51 added, 23 removed. Audit A to A.
---
name: custody-solution-uptime-and-liveness-guarantees
- description: Quantitative custody liveness monitoring engine for tracking API uptime
- SLAs (99.9%+), MPC threshold quorum (k-of-n), and signing latency SLAs, triggering
- failover upon availability breaches.
+ description: Fail-closed custody liveness monitor tracking API uptime against a contractual
+ SLA target, MPC signing quorum (k-of-n) with early warning at zero redundancy, and
+ sample-gated P99 signing latency, recommending failover on breach, stale telemetry,
+ or no telemetry.
domain: Crypto Custody & Security
subdomain: Custody SLA & Liveness
tags:
- custody-sla
- liveness-guarantees
- mpc-quorum
- signing-latency
- fireblocks
- bitgo
- anchorage
- uptime-monitoring
brokers_frameworks:
- SOC 2 Type II
- Python Dataclasses
- version: "1.0.0"
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill in institutional crypto trading desks and automated custody systems to monitor API uptime, MPC node liveness ($k$-of-$n$), and transaction signing latency SLAs across custody providers (Fireblocks, BitGo, Coinbase Custody, Anchorage). In 24/7/365 crypto markets, a custody outage or MPC quorum failure during market volatility prevents margin top-ups or liquidation defense. This module computes rolling uptime percentages, tracks P99 signing latency, and triggers secondary custody failover when SLAs are breached ($< 99.9\%$ uptime or signing latency $> 5000\text{ ms}$).
+ Use this skill on institutional crypto desks whose automated strategies depend on a custody provider being able to *sign right now*. In 24/7 crypto markets a custody outage or an MPC quorum failure during a volatility spike blocks margin top-ups and liquidation defence, and the loss is realized before a human notices.
+ The engine consumes health-probe telemetry and answers one question — is the primary custodian live enough to sign, and if not, should we fail over? It reports:
+
+ - **Uptime** against your *contractual* target, compared without a rounding step that could mask a breach.
+ - **MPC quorum** for a k-of-n signing cluster, including a `QUORUM_AT_RISK` warning at zero remaining redundancy — before the halt, not during it.
+ - **P99 signing latency**, gated on having enough samples for a 99th percentile to mean anything.
+
+ **It fails closed.** No telemetry, stale telemetry, and malformed telemetry are all non-healthy outcomes that recommend failover. Silence is not health.
+
+ ## When NOT to Use
+
+ - **As the failover executor.** This engine sets `is_failover_recommended`. It does not move keys, re-route signing, or touch the secondary provider. Failover is a capital-moving action; keep the decision and the execution separate and human-reviewable.
+ - **As a source of SLA numbers.** Every threshold is an input, not a constant. `target_uptime_pct` and `max_signing_latency_ms` come from your executed agreement with the provider. No numeric SLA is hard-coded or implied here.
+ - **As a time-weighted availability calculator for SLA credits.** Uptime is computed as a probe-success ratio, which equals contractual availability only when probes are evenly spaced. Claims for service credits should be computed from the provider's own incident record and your contract's measurement method.
+ - **As a substitute for the provider's status page.** A probe pipeline can be healthy while the custodian is degraded in ways your probes do not exercise (for example, withdrawals halted while reads succeed).
+
## Prerequisites
- - Custody provider SLA configuration: `target_uptime_pct` (e.g. 99.9%), `max_signing_latency_ms` (e.g. 2000 ms), `mpc_threshold_k`, `mpc_total_n`.
- - Continuous health probe & signing telemetry feeds.
+ - **Contractual SLA terms** from the executed provider agreement: `target_uptime_pct`, `max_signing_latency_ms`, and the measurement window they are defined over.
+ - **Cluster shape**: `mpc_threshold_k` (shares required to sign) and `mpc_total_n` (shares provisioned). A `k`-of-`n` cluster tolerates exactly `n - k` unavailable nodes.
+ - **Continuous health-probe telemetry** carrying `timestamp_ms`, `is_api_healthy`, `signing_latency_ms`, and `active_mpc_nodes`.
+ - **A freshness bound** (`max_probe_age_ms`) and a wall-clock reference (`as_of_timestamp_ms`) if staleness is to be detected at all.
## Workflow
- 1. **Telemetry Ingestion**:
- - Collect API ping health, MPC node status, and signing latency measurements.
- 2. **Uptime & Quorum Audit**:
- - $\text{Uptime Pct} = \frac{N_{\text{healthy}}}{N_{\text{total}}} \times 100\%$.
- - Check MPC threshold: If $\text{Active Nodes} < k \implies$ Trigger `MPC_QUORUM_LOST` emergency alert.
- 3. **P99 Signing Latency Audit**:
- - Compute P99 signing time over rolling 100 transactions.
- - If $\text{P99} > \text{SLA Max} \implies$ Flag `LATENCY_SLA_BREACH`.
- 4. **Secondary Signer Failover**:
- - If $\text{Uptime} < 99.9\%$ or $\text{MPC Quorum Lost} \implies$ Trigger automated failover to secondary custody provider.
+ 1. **Configure from the contract, then validate.** `CustodyLivenessMonitorEngine(config)` rejects an unsatisfiable cluster (`k > n`), a `target_uptime_pct` outside `[0, 100]`, and non-positive latency or freshness bounds at construction. A 4-of-3 quorum is dead on arrival and should fail at startup, not at the first signing attempt.
+ 2. **Feed probes; do not pre-sort them.** `audit_liveness` orders by `timestamp_ms` itself. Do not rely on list order to identify the current state — concurrent collectors routinely deliver an older probe last, and taking the final list element as "now" is how a real quorum loss goes unseen.
+ 3. **Pass `as_of_timestamp_ms` whenever you have a clock.** Without it, freshness is *not evaluated* and the report says so explicitly. With it, a newest probe older than `max_probe_age_ms` yields `STALE_TELEMETRY` and recommends failover — a frozen collector must not read as a healthy custodian.
+ 4. **Read `breached_conditions`, not just `status`.** `status` is the single most severe condition; `breached_conditions` and `recommendations` carry all of them. A quorum halt and an uptime breach commonly occur together and the operator needs both.
+ 5. **Check `percentiles_reliable` before acting on P99.** When `latency_sample_count < min_latency_samples` the P99 is reported but explicitly not gated, because a 99th percentile drawn from a handful of samples is the maximum wearing a percentile's name. Widen the window rather than lowering the bar.
+ 6. **Decide failover deliberately.** Quorum loss, uptime breach, stale telemetry, and no telemetry all set `is_failover_recommended`. A latency breach does so only when `failover_on_latency_breach=True`, because slow signing is usually better than a custody migration mid-volatility — make that trade explicit rather than inheriting it.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Ignoring MPC Quorum Redundancy**: Relying on 2-of-3 MPC nodes without monitoring node health, suffering liveness halts when 1 node goes down for maintenance.
- - **Relying on Average Signing Latency**: Tracking average signing time (500ms) while missing 15-second P99 signing spikes during high gas spikes on Ethereum.
- - **Un-tested Custody Failover**: Defining a secondary custody provider without regularly testing automated failover routing under simulated outages.
+ - **Treating "no data" as "no problem".** An empty probe list is the signature of a dead collector, not a perfect custodian. Any monitor that returns healthy when blind will report healthy through the exact incident it exists to catch.
+ - **Rounding before comparing.** Rounding uptime to two decimals and then testing it against a 99.9% target lets a true 99.896% clear the gate. Round for display, compare on the raw value.
+ - **Trusting list order for "current" state.** `probes[-1]` is the last element, not the newest observation. Sort by timestamp before reading the current node count.
+ - **Letting NaN through.** Every `>` comparison against NaN is False, so one corrupt latency reading passes an SLA gate instead of tripping it. Reject non-finite telemetry at ingestion.
+ - **Alerting only after quorum is lost.** In a 2-of-3 cluster, two active nodes is not healthy — it is one node from a signing halt, and routine maintenance on either node causes it. Warn at zero remaining redundancy.
+ - **Gating on an under-sampled P99.** A 99th percentile needs at least 100 observations to resolve the top 1% at all; below that it is just the maximum and one unlucky request looks like an SLA breach.
+ - **Reporting one breach at a time.** An `if/elif` chain hides concurrent failures; the quorum alarm masks the uptime breach that explains it.
+ - **Reading a 99.9% figure as a standard.** SOC 2 does not set an uptime number — see `references/standards.md`. The number is contractual, and it is only meaningful alongside its measurement window.
+ - **Confusing probe-ratio uptime with time-weighted availability.** If probe frequency rises during incidents (retries, denser polling), the ratio is biased relative to the contractual measure.
## Verification
- - Instantiate `CustodyLivenessMonitorEngine`. Register `Primary_Custodial_MPC` ($k=2, n=3$, target uptime $99.9\%$, max signing $2000\text{ ms}$). Simulate 100 health probes with 99.95% uptime and P99 latency $450\text{ ms}$. Verify status is `HEALTHY`. Simulate 2 MPC nodes going offline (Active $= 1 < k=2$). Verify monitor triggers `MPC_QUORUM_LOST` and directs failover to secondary provider.
- - Run `python scripts/test_custody_solution_uptime_and_liveness_guarantees.py`.
+ - Construct `ProviderSlaConfig(target_uptime_pct=99.9, max_signing_latency_ms=2000.0, mpc_threshold_k=2, mpc_total_n=3)`. Feed 100 healthy probes at 450 ms with 3 active nodes → `HEALTHY`, `redundant_nodes == 1`, no failover.
+ - Feed 24,974 healthy probes and 26 unhealthy (99.896%) → `DEGRADED_SLA_BREACH` with failover. Rounding to two decimals would have shown 99.9% and passed.
+ - Feed probes where the newest by timestamp reports 1 active node but arrives mid-list → `QUORUM_LOST_LIVENESS_HALT`, `current_active_mpc_nodes == 1`.
+ - Feed an empty list → `UNKNOWN_NO_TELEMETRY` with `is_failover_recommended is True`.
+ - Set `mpc_threshold_k=4, mpc_total_n=3` → `ValueError` at construction.
+ ```bash
+ python -m unittest discover -s skills/custody-solution-uptime-and-liveness-guarantees/scripts
+ ```
+
## Related Skills
- `custodial-vs-non-custodial-tradeoff-assessment`
- `multi-party-computation-mpc-custody-solutions`
- ---
+ - `custody-solution-vendor-due-diligence-checklist`
+ - `broker-failover-secondary-account-routing`