v1.0.0 to v2.0.0
44 added, 22 removed. Audit A to A.
---
name: colocation-latency-budget-accounting
- description: Quantitative co-location telemetry module for decomposing tick-to-trade
- (T2T) latency budgets into nanosecond phases, detecting SLA breaches, and computing
- P99/P99.9 tail jitter statistics.
+ description: >-
+ Use when accounting for the in-host tick-to-trade (T2T) latency budget of a
+ co-located trading server: decomposes a NIC-ingress-to-NIC-egress trace into
+ nanosecond phases, audits each trace against total and per-phase SLAs, names
+ the phase most over its budget, and reports P50/P95/P99/P99.9 tail jitter.
domain: Infrastructure
subdomain: Latency Optimization
tags:
- latency-budget
- tick-to-trade
- hft
- colocation
- hardware-timestamping
- sla
- tail-jitter
+ - clock-domain
brokers_frameworks:
- Generic Infrastructure
- NumPy
- version: "1.0.0"
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill when profiling and auditing the "hot path" of high-frequency trading (HFT) servers in co-located data centers. Tick-to-trade (T2T) latency represents the elapsed time from hardware packet arrival at the NIC (`T_ingress`) to order byte transmission (`T_egress`). This module decomposes total T2T latency into discrete processing phases (NIC Ingress -> Decode -> Alpha Signal -> Risk Check -> FIX/Binary Encode -> NIC Egress) to identify software/hardware bottlenecks and track tail jitter ($P_{99}, P_{99.9}$).
+ Use this skill when profiling and auditing the "hot path" of a high-frequency trading (HFT) server in a co-located data centre. Tick-to-trade (T2T) latency is the elapsed time from hardware packet arrival at the NIC ($T_0$) to order byte transmission ($T_5$). This module decomposes T2T into discrete processing phases (NIC ingress → decode → alpha signal → pre-trade risk → binary encode → NIC egress), audits them against a total and per-phase budget, names the phase most over its budget, and reports tail jitter ($P_{99}$, $P_{99.9}$).
+ ## When NOT to Use
+
+ - **For anything outside the box.** This skill starts at NIC ingress and stops at NIC egress. Cross-venue propagation, facility choice and cross-connects belong to `co-location-provider-selection-and-network-topology`; the end-to-end measurement harness belongs to `tick-to-trade-latency-measurement`.
+ - **As the instrumentation itself.** This is the *offline accounting* half. It consumes traces that some lock-free hot-path instrumentation already captured; it does not capture timestamps, and it must never be called from the hot path.
+ - **When the timestamps are not on one clock.** $T_0/T_5$ come from the NIC's PTP hardware clock (PHC); $T_1 \dots T_4$ typically come from `CLOCK_MONOTONIC`. Convert to a single time base *before* building a `HotPathTrace` — see Pitfalls. The engine rejects traces that go backwards, but it cannot detect a constant inter-domain offset.
+ - **As a regulatory timestamping record.** MiFID II RTS 25 governs *business clock* accuracy and granularity for reportable events against UTC. `CLOCK_MONOTONIC` is not UTC-traceable, and this module's relative nanosecond deltas are not a substitute for a compliant timestamping record. See `references/standards.md`.
+ - **For sub-100 sample batches.** Percentiles finer than the batch can resolve are interpolations, not measurements (see Pitfalls).
+
## Prerequisites
- - Hardware timestamping enabled on Network Interface Cards (NICs).
- - Nanosecond or microsecond timer calls (e.g. `clock_gettime(CLOCK_MONOTONIC)` or `rdtsc` instruction).
+ - Hardware timestamping enabled on the NIC. On Linux, request `SOF_TIMESTAMPING_RX_HARDWARE` / `SOF_TIMESTAMPING_TX_HARDWARE` via `SO_TIMESTAMPING`, and confirm driver support with `ethtool -T <iface>`.
+ - A high-resolution in-host timer for $T_1 \dots T_4$: `clock_gettime(CLOCK_MONOTONIC)`, or `rdtsc`. **`rdtsc` returns cycle counts, not nanoseconds** — converting requires a calibrated invariant-TSC frequency (invariant TSC is advertised by `CPUID.80000007H:EDX[8]`), and readings are only comparable across cores when the TSC is invariant and synchronised.
+ - A documented conversion between the NIC PHC domain and the in-host timer domain.
+ - NumPy.
+ ## Units
+
+ All timestamps, phase durations and SLAs are **nanoseconds (ns)**, as Python `int`. Percentile outputs are floats rounded to 1 dp; `count` is the batch size. Nothing in this module is microseconds — a value pasted in as µs is off by 1,000x and will not be caught.
+
## Workflow
- 1. **Hot Path Instrumentation**: Log high-resolution timestamps at key processing stage boundaries:
- - $T_0$: NIC Ingress Timestamp
- - $T_1$: Data Decode Completed
- - $T_2$: Alpha Signal Evaluated
- - $T_3$: Pre-Trade Risk Check Completed
- - $T_4$: Binary Order Encoded
- - $T_5$: NIC Egress Timestamp
- 2. **Phase Breakdown**: Calculate duration for each phase: $\Delta_i = T_i - T_{i-1}$.
- 3. **SLA Audit & Bottleneck Identification**: Compare phase durations against target SLAs. If $T_5 - T_0 > \text{Total\_SLA}$, flag the execution as a breach and isolate the phase responsible for the maximum excess delay.
- 4. **Tail Latency Reporting**: Calculate $P_{50}$, $P_{95}$, $P_{99}$, and $P_{99.9}$ metrics across executions to measure jitter.
+ 1. **Hot Path Instrumentation**: record high-resolution timestamps at each stage boundary into a fixed-size lock-free buffer — $T_0$ NIC ingress, $T_1$ decode complete, $T_2$ alpha signal evaluated, $T_3$ pre-trade risk complete, $T_4$ order encoded, $T_5$ NIC egress.
+ 2. **Normalise to One Clock**: convert $T_0/T_5$ (NIC PHC) and $T_1 \dots T_4$ (`CLOCK_MONOTONIC`) onto a single monotonic nanosecond base before constructing traces.
+ 3. **Ingest with Quarantine**: build `HotPathTrace` objects off the hot path. Construction raises `ValueError` on non-monotonic timestamps — catch it **per trace**, increment a quarantine counter, and continue. A non-monotonic trace is an instrumentation or clock-domain defect, not a fast execution; do not clamp it to zero and do not let it abort the batch.
+ 4. **Configure the Budget** (`LatencyBudgetAccountingEngine`): `total_sla_ns` plus a `phase_slas_ns` map covering **every** phase in `PHASE_NAMES`. Unknown or missing keys raise — a typo must not silently give a phase a 0 ns budget. If the phase budgets sum above the total, the engine logs a warning: that configuration is legal but means a trace can breach the total with no phase over its own budget.
+ 5. **Audit** (`audit_trace`): `is_sla_breach` is a strict `>`, so a trace landing exactly on `total_sla_ns` is inside budget. On breach the report carries `phase_excess_ns` (duration − budget, per phase) and names as `primary_bottleneck_phase` the phase with the greatest excess — even when every excess is negative, in which case it is the phase closest to its limit. Ties break by hot-path order.
+ 6. **Tail Reporting** (`compute_percentiles`): $P_{50}, P_{95}, P_{99}, P_{99.9}$ plus `mean` and `count` for the total and every phase. Read `count` before trusting a tail figure.
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Using Software Timestamps**: Measuring T2T latency using standard OS `time.time()`, which suffers from 1-10 microsecond interrupt jitter. Hardware NIC timestamps (`ethtool -T`) must be used for ingress/egress.
- - **Profiling in Main Thread**: Writing log lines or computing percentiles directly in the hot-path execution thread, adding milliseconds of I/O latency to the order. Timestamps must be recorded in lock-free ring buffers and processed out-of-band.
- - **Focusing Only on Mean Latency**: Relying on average T2T latency instead of $P_{99.9}$ tail latency. A 10µs mean latency is useless if $P_{99}$ spikes to 5ms during market volatility.
+ - **Mixing Clock Domains**: NIC hardware timestamps are taken in the NIC's own PHC domain; `CLOCK_MONOTONIC` is a different clock. Subtracting one from the other yields an offset, not a duration — often large enough to be negative, sometimes small enough to look plausible. The engine rejects the negative case; the plausible case is silently wrong, which is why the conversion must be documented rather than assumed.
+ - **Using `time.time()` for Hot-Path Timestamps**: `time.time()` reads `CLOCK_REALTIME`, which NTP steps and slews — it can move backwards, so a phase can measure negative. Use `CLOCK_MONOTONIC` for in-host deltas and NIC hardware timestamps for $T_0/T_5$. A userspace timestamp also includes interrupt-handling and scheduler delay that the NIC timestamp excludes; that gap is not bounded and grows exactly when the machine is busiest.
+ - **Profiling in the Hot-Path Thread**: writing log lines or computing percentiles inside the execution thread adds I/O and allocation latency to the order it is measuring. Record into a lock-free ring buffer and run this module out-of-band.
+ - **Reading a P99.9 off a Small Batch**: a percentile finer than $1/(1-q)$ samples is interpolated between the top two observations. P99.9 from 10 samples is a restatement of the maximum with a decimal point on it. `compute_percentiles` warns and returns `count` — check it before putting the figure on a dashboard.
+ - **Believing a Breach Has No Bottleneck**: whenever the per-phase budgets sum above the total, every phase can be inside its own budget while the total breaches. A breached trace always names a phase; a *negative* `phase_excess_ns` on that phase is the signal that the budget allocation, not the code, is the problem.
+ - **Focusing on Mean Latency**: a 10 µs mean is meaningless if $P_{99.9}$ spikes to 5 ms during a volatility burst — the spike is exactly when the fill mattered. Evaluate SLAs on tails.
+ - **Silently Dropping Rejected Traces**: quarantining a bad trace without counting it biases the tail downwards, because instrumentation defects correlate with the slow path. Count and alert on the quarantine rate.
## Verification
- - Instantiate `LatencyBudgetAccountingEngine` with a total SLA of 5,000 ns. Record 1,000 execution samples where Risk Check is artificially delayed to 4,000 ns (SLA 500 ns). Verify that the engine correctly flags SLA breaches and identifies `risk_check` as the primary bottleneck.
- - Run `python scripts/test_colocation_latency_budget_accounting.py`.
+ - Construct `LatencyBudgetAccountingEngine(total_sla_ns=10000)` (default phase budgets). Audit a trace with phase durations `[1000, 1000, 8000, 1000, 4000]` ns: total is 15,000 ns, `is_sla_breach` is `True`, and `primary_bottleneck_phase` is `signal_to_risk_ns` with `phase_excess_ns["signal_to_risk_ns"] == 6500` (8,000 ns against a 1,500 ns budget).
+ - Construct `LatencyBudgetAccountingEngine(total_sla_ns=5000)` — the default phase budgets sum to 8,000 ns, so the engine warns. Audit a trace of five 1,100 ns phases: total 5,500 ns breaches, yet every phase is inside its own budget. `primary_bottleneck_phase` must still be non-`None`, with a negative excess.
+ - `HotPathTrace("BAD", 1000, 900, 2000, 3000, 4000, 5000)` must raise `ValueError`, not produce a −100 ns phase.
+ - Run `python -m unittest discover -s skills/colocation-latency-budget-accounting/scripts`.
## Related Skills
- `tick-to-trade-latency-measurement`
+ - `strategy-latency-budget-decomposition`
- `clock-drift-monitoring-alerting-thresholds`
+ - `hardware-timestamping-vs-software-timestamping-accuracy`
+ - `co-location-provider-selection-and-network-topology`