v1.0.0 to v2.0.0
149 added, 20 removed. Audit A to A.
---
name: clock-skew-correction-for-tick-timestamps
- description: Quantitative market data pipeline utility for estimating and correcting
- clock drift between venue feeds and local recorders using minimum-delay filtering
- without violating time monotonicity.
+ description: Use when captured tick data carries two timestamps — the venue's and the
+ local recorder's — and the local clock runs at a slightly different rate, to estimate
+ the drift from windowed minimum one-way delays, re-express local timestamps on the
+ venue timescale, and keep the result strictly monotonic without mistaking network
+ congestion for clock drift.
domain: Data Management
subdomain: Market Data Infrastructure
tags:
- clock-skew
- - paxson-algorithm
+ - minimum-delay-filtering
- timestamps
- market-data
- hft
- monotonicity
brokers_frameworks:
- Generic Infrastructure
- NumPy
- - Pandas
- version: "1.0.0"
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Use this skill when processing multi-venue high-frequency tick data or order book message logs captured across different hosts. Variations in local clock synchronization (NTP drift, host-level clock skew) cause measured delays to drift over time. This utility estimates the linear drift via minimum-delay lower-bound filtering (Paxson's algorithm principle) and adjusts the timestamps while strictly maintaining monotonicity.
+ Use this skill when you hold a capture of market data in which each event carries both
+ a **venue-stamped send time** and a **locally stamped receive time**, and the two clocks
+ run at measurably different rates. Uncorrected skew shows up as a delay series that
+ trends across the session: the same event appears to take longer and longer to arrive,
+ when in fact the local clock is losing or gaining time. Typical consumers:
+ - Merging captures from several hosts into one ordered event stream for order-book
+ reconstruction or backtesting.
+ - Latency analysis where a drifting recorder would otherwise produce a fake trend in
+ tick-to-trade or feed-arrival measurements.
+ - Re-stamping an archive taken while a host was running on plain NTP, or before PTP was
+ deployed.
+
+ `ClockSkewCorrector` fits `delay ≈ alpha + beta·t` to the **minimum** delay in each time
+ window, then subtracts the fitted line from the local timestamps and forces the output
+ to increase strictly.
+
+ ## When NOT to Use
+
+ - **To satisfy a clock-synchronisation obligation.** Under MiFID II RTS 25 the
+ timestamps a firm records for reportable events must come from a clock synchronised
+ and traceable to UTC — 100 µs maximum divergence and 1 µs granularity for
+ high-frequency algorithmic trading. That is achieved by disciplining the clock
+ (`clock-synchronization-ptp-for-trading-hosts`), not by regressing timestamps after
+ the fact. Statistically corrected timestamps are a research artifact; never file them
+ or present them as an RTS 25 record. See `references/standards.md`.
+ - **To measure one-way latency after removing the constant term.** From one-way data the
+ minimum transit delay and the constant clock offset are *not separable* — they enter
+ the measurement only as a sum. The default correction removes both, so
+ `corrected − exchange_ts` is excess delay above the session minimum, not latency. Use
+ `remove_constant_offset=False` for latency work, which removes drift only.
+ - **When the host is already PTP-disciplined to sub-microsecond accuracy.** Fitting
+ skew to a hardware-disciplined clock mostly fits your own measurement noise. Monitor
+ it instead (`clock-drift-monitoring-alerting-thresholds`).
+ - **On ticks merged from multiple venues or multiple capture hosts.** Skew is a property
+ of one clock *pair*. One line fitted across a mixed series describes no real pair of
+ clocks. Split by (venue, host) and fit each separately; the implementation rejects
+ input that is not ordered by exchange time, which catches the common form of this
+ mistake.
+ - **Across a clock step.** NTP *slews* small offsets but *steps* the clock once the
+ offset exceeds 128 ms by default, and a leap second is a 1 s step. A single straight
+ line fitted across a step is wrong on both sides of it — split the series at the step.
+ - **When the venue timestamp is coarser than the drift you are chasing.** A venue that
+ stamps to the millisecond cannot evidence tens of microseconds of drift over a short
+ window; you need a much longer span or a better source timestamp.
+
## Prerequisites
- - Paired timestamp data: `exchange_timestamp` (sender) and `local_timestamp` (receiver) for market events.
- - Monotonically increasing event sequence numbers.
+ - Paired `exchange_ts` and `local_ts` per event, **both** from the same venue/host pair,
+ sorted by exchange time, in a single unit (`s`, `ms`, `us`, `ns`).
+ - Timestamps as **UTC epoch counts**, never local wall-clock — a DST transition in a
+ wall-clock series is a one-hour "skew" that this model will happily fit. See
+ `daylight-saving-time-transition-handling`.
+ - For tick-resolution work, **int64 nanoseconds** with `time_unit="ns"`. Float64 POSIX
+ seconds have a ULP of ~238 ns near 1.7e9 and physically cannot hold two events 100 ns
+ apart, whatever the correction does.
+ - Enough events per window to make a minimum meaningful — the default
+ `min_points_per_window=10` drops sparser windows rather than trusting them.
+ - NumPy. No other runtime dependency.
## Workflow
- 1. **One-Way Delay Calculation**: Compute raw one-way delay: $Delay_i = T_{local, i} - T_{exchange, i}$.
- 2. **Minimum Delay Filtering**: Group time into rolling bins or expanding windows and extract the minimum delay points. Because true network latency has a hard lower bound, minimum delays reflect pure clock offset without network queueing noise.
- 3. **Linear Regression Fit**: Fit a linear model ($Offset(t) = a + b \cdot t$) to the lower-bound delay points to find the clock drift rate ($b$).
- 4. **Correction Application**: Subtract the estimated $Offset(t)$ from $T_{local, i}$.
- 5. **Monotonicity Enforcement**: Ensure $T_{corrected, i} \ge T_{corrected, i-1}$. If a correction would force a timestamp backward, clamp it to $T_{corrected, i-1} + \epsilon$.
+ 1. **Split the series by clock pair.** One fit per (venue, capture host). Sort by
+ exchange timestamp; `fit` raises rather than silently returning zero drift if you do
+ not.
+ 2. **Compute one-way delays** `local_ts − exchange_ts`, differencing in the input's
+ integer unit before scaling to seconds so the difference is not swallowed by float
+ rounding at epoch magnitudes.
+ 3. **Take the minimum per window, not the mean or median.** Queueing delay is one-sided
+ and strictly positive, so any central statistic is biased upward by congestion and
+ moves with it. Anchor each retained point at the exchange time of the sample that
+ *achieved* the minimum, not at the window's start edge.
+ 4. **Drop windows with too few samples.** The minimum of *k* draws falls as *k* grows, so
+ a window holding one tick contributes a raw jitter draw as though it were a lower
+ bound. Under a U-shaped intraday volume profile that converts tick density into fake
+ skew.
+ 5. **Fit the line and then interrogate it before believing it.** Check `diagnostics`:
+ `reliable` False means the fit fell back to zero drift, not that the clock is good.
+ A low `r_squared`, a large `residual_std_sec`, or a step warning in the log means
+ the lower envelope is not a straight line. (A step mid-series inflates the residual
+ spread too, so `fit` detects steps from jumps between consecutive window minima
+ rather than from residual outliers.) A drift beyond
+ `max_drift_ppm` (default 1000 ppm) raises — the Unix kernel discipline NTP relies on
+ slews at most 500 ppm, so a larger *sustained* rate is a data problem: mixed units,
+ mixed hosts, or a step inside the window.
+ 6. **Decide what to subtract.** Default (`remove_constant_offset=True`) puts the output
+ on the venue's timescale and origin. For latency analysis pass `False` to remove
+ drift only and keep delays positive.
+ 7. **Apply, then enforce monotonicity.** `T_final,i = max(T_corr,i, T_final,i−1 + ε)`.
+ ε is clamped up to one output tick in integer mode; in float mode the guarantee is
+ still enforced, falling back to one-ULP separation with a warning when the epsilon is
+ below what the magnitude can represent.
+ 8. **Keep the model causal in production.** `fit` records its first exchange timestamp
+ as the reference epoch and `transform` measures against *that* epoch, so calibrating
+ on a warm-up window and applying to later ticks is correct. `fit_transform` is
+ in-sample by construction — offline research only.
> Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
+ > Standards and evidence: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Mean/Median Regression**: Using ordinary least squares on *all* delay points. Network queueing spikes (jitter) pollute the mean, causing massive over-estimation of clock offset. You MUST filter for minimum delays.
- - **Breaking Monotonicity**: Applying raw linear corrections without checking $T_i \ge T_{i-1}$, causing time to jump backward and throwing off backtesting engines or order-sequence matching.
- - **In-Sample Overfitting**: Estimating clock skew on the entire day's dataset and applying it backwards (lookahead leakage). Calibration should be done on a rolling expanding window.
+ - **Regressing on all delay points.** Network queueing is positive-only. Ordinary least
+ squares over every sample is biased upward by the mean queueing delay and reads a
+ congestion episode as clock drift. Filter to window minima first.
+ - **Re-basing the correction on each batch.** If `transform` measures elapsed time from
+ its *own* first tick rather than the epoch used at `fit` time, the intercept is applied
+ at the wrong origin. At 100 ppm, a 300 s gap between calibration and application is a
+ silent **30 ms** error — larger than most of the effects being studied.
+ - **Calling the intercept a clock offset.** It is `minimum_transit + clock_offset`, and
+ the two are not separable from one-way data. Reporting it as an offset overstates the
+ clock error by the whole propagation delay.
+ - **Trusting a "strict monotonicity" guarantee in float64 seconds.** Adding a 1 ns
+ epsilon to a POSIX-epoch float64 is a no-op — the ULP there is ~238 ns — so a naive
+ implementation emits equal timestamps while claiming they increase. Use int64
+ nanoseconds.
+ - **Fitting across a clock step.** A 128 ms NTP step or a leap second is a discontinuity,
+ not drift; one line across it is wrong everywhere. Detect and segment.
+ - **Treating a sustained latency regime change as clock skew.** If *every* sample in a
+ run of windows is delayed — a route change, a switch upgrade, a saturated link — the
+ lower envelope itself moves, and one-way data cannot distinguish that from the clock.
+ Minimum filtering only rejects congestion that leaves quiet samples in each window.
+ Corroborate large fitted drifts against `clock-drift-monitoring-alerting-thresholds`
+ telemetry before believing them.
+ - **Extrapolating a stale fit.** Crystal drift changes with temperature. Applying a
+ morning calibration to an afternoon session accumulates error; the implementation
+ warns when asked to extrapolate beyond its calibration span.
+ - **In-sample correction leaking into a backtest.** `fit_transform` over a whole day uses
+ future ticks to stamp earlier ones. In any research pipeline that feeds a strategy,
+ calibrate on a prior window only. See `lookahead-bias-elimination`.
+ - **Silently accepting unsorted input.** Shuffled exchange timestamps used to collapse
+ the windowing and return zero drift with no error at all, which looks exactly like a
+ healthy clock.
## Verification
- - Simulate 1,000 tick messages with a constant clock drift rate (e.g. 50 microseconds per second) and random positive network delays. Run the `ClockSkewCorrector`. Verify that the estimated drift rate matches the true drift within 5% tolerance and that all output timestamps are strictly monotonic.
- - Run `python scripts/test_clock_skew_corrector.py`.
+ - `python -m unittest discover -s skills/clock-skew-correction-for-tick-timestamps/scripts`
+ (30 tests). Coverage includes: recovery of a known injected drift rate (positive,
+ negative and zero) to within 5 ppm; rejection of intermittent congestion that would
+ push naive OLS past 100 ppm; the reference-epoch regression (chunked `transform` must
+ equal a single call); strict monotonicity with an epsilon below the float64 ULP;
+ int64-nanosecond mode preserving 100 ns separation; and rejection of unsorted, NaN,
+ empty and mismatched input.
+ - Generate a synthetic feed with a known drift (e.g. 100 µs/s) plus strictly positive
+ jitter, then confirm `drift_ppm` matches the injected rate within a few ppm and
+ `np.all(np.diff(out) > 0)` holds.
+ - Confirm scale: a 2M-tick session fits and transforms in well under a second. If it
+ takes tens of seconds, the windowing has degraded to a per-window scan of the whole
+ array.
+ - Cross-check any material fitted drift against independent clock telemetry
+ (`chronyc tracking`, `pmc`/`ptp4l` offsets). A regression on market data alone cannot
+ tell a drifting clock from a drifting network path.
## Related Skills
+ - `clock-synchronization-ptp-for-trading-hosts`
- `clock-drift-monitoring-alerting-thresholds`
+ - `cross-vendor-timestamp-precision-reconciliation`
+ - `hardware-timestamping-vs-software-timestamping-accuracy`
- `historical-order-book-reconstruction-from-message-logs`
+ - `lookahead-bias-elimination`