v2.0.0 to v2.0.0

---

name: clock-skew-correction-for-tick-timestamps

license: Apache-2.0

---

## When to Use

Use this skill when you hold a capture of market data in which each event carries both

a **venue-stamped send time** and a **locally stamped receive time**, and the two clocks

run at measurably different rates. Uncorrected skew shows up as a delay series that

trends across the session: the same event appears to take longer and longer to arrive,

when in fact the local clock is losing or gaining time. Typical consumers:

- Merging captures from several hosts into one ordered event stream for order-book

reconstruction or backtesting.

- Latency analysis where a drifting recorder would otherwise produce a fake trend in

tick-to-trade or feed-arrival measurements.

- Re-stamping an archive taken while a host was running on plain NTP, or before PTP was

deployed.

`ClockSkewCorrector` fits `delay ≈ alpha + beta·t` to the **minimum** delay in each time

window, then subtracts the fitted line from the local timestamps and forces the output

to increase strictly.

## When NOT to Use

- **To satisfy a clock-synchronisation obligation.** Under MiFID II RTS 25 the

timestamps a firm records for reportable events must come from a clock synchronised

and traceable to UTC — 100 µs maximum divergence and 1 µs granularity for

high-frequency algorithmic trading. That is achieved by disciplining the clock

(`clock-synchronization-ptp-for-trading-hosts`), not by regressing timestamps after

the fact. Statistically corrected timestamps are a research artifact; never file them

or present them as an RTS 25 record. See `references/standards.md`.

- **To measure one-way latency after removing the constant term.** From one-way data the

minimum transit delay and the constant clock offset are *not separable* — they enter

the measurement only as a sum. The default correction removes both, so

`corrected − exchange_ts` is excess delay above the session minimum, not latency. Use

`remove_constant_offset=False` for latency work, which removes drift only.

- **When the host is already PTP-disciplined to sub-microsecond accuracy.** Fitting

skew to a hardware-disciplined clock mostly fits your own measurement noise. Monitor

it instead (`clock-drift-monitoring-alerting-thresholds`).

- **On ticks merged from multiple venues or multiple capture hosts.** Skew is a property

of one clock *pair*. One line fitted across a mixed series describes no real pair of

clocks. Split by (venue, host) and fit each separately; the implementation rejects

input that is not ordered by exchange time, which catches the common form of this

mistake.

- **Across a clock step.** NTP *slews* small offsets but *steps* the clock once the

offset exceeds 128 ms by default, and a leap second is a 1 s step. A single straight

line fitted across a step is wrong on both sides of it — split the series at the step.

- **When the venue timestamp is coarser than the drift you are chasing.** A venue that

stamps to the millisecond cannot evidence tens of microseconds of drift over a short

window; you need a much longer span or a better source timestamp.

## Prerequisites

- Paired `exchange_ts` and `local_ts` per event, **both** from the same venue/host pair,

sorted by exchange time, in a single unit (`s`, `ms`, `us`, `ns`).

- Timestamps as **UTC epoch counts**, never local wall-clock — a DST transition in a

wall-clock series is a one-hour "skew" that this model will happily fit. See

`daylight-saving-time-transition-handling`.

- For tick-resolution work, **int64 nanoseconds** with `time_unit="ns"`. Float64 POSIX

seconds have a ULP of ~238 ns near 1.7e9 and physically cannot hold two events 100 ns

apart, whatever the correction does.

- Enough events per window to make a minimum meaningful — the default

`min_points_per_window=10` drops sparser windows rather than trusting them.

- NumPy. No other runtime dependency.

## Workflow

1. **Split the series by clock pair.** One fit per (venue, capture host). Sort by

exchange timestamp; `fit` raises rather than silently returning zero drift if you do

not.

2. **Compute one-way delays** `local_ts − exchange_ts`, differencing in the input's

integer unit before scaling to seconds so the difference is not swallowed by float

rounding at epoch magnitudes.

3. **Take the minimum per window, not the mean or median.** Queueing delay is one-sided

and strictly positive, so any central statistic is biased upward by congestion and

moves with it. Anchor each retained point at the exchange time of the sample that

*achieved* the minimum, not at the window's start edge.

4. **Drop windows with too few samples.** The minimum of *k* draws falls as *k* grows, so

a window holding one tick contributes a raw jitter draw as though it were a lower

bound. Under a U-shaped intraday volume profile that converts tick density into fake

skew.

5. **Fit the line and then interrogate it before believing it.** Check `diagnostics`:

`reliable` False means the fit fell back to zero drift, not that the clock is good.

A low `r_squared`, a large `residual_std_sec`, or a step warning in the log means

the lower envelope is not a straight line. (A step mid-series inflates the residual

spread too, so `fit` detects steps from jumps between consecutive window minima

rather than from residual outliers.) A drift beyond

`max_drift_ppm` (default 1000 ppm) raises — the Unix kernel discipline NTP relies on

slews at most 500 ppm, so a larger *sustained* rate is a data problem: mixed units,

mixed hosts, or a step inside the window.

6. **Decide what to subtract.** Default (`remove_constant_offset=True`) puts the output

on the venue's timescale and origin. For latency analysis pass `False` to remove

drift only and keep delays positive.

7. **Apply, then enforce monotonicity.** `T_final,i = max(T_corr,i, T_final,i−1 + ε)`.

ε is clamped up to one output tick in integer mode; in float mode the guarantee is

still enforced, falling back to one-ULP separation with a warning when the epsilon is

below what the magnitude can represent.

8. **Keep the model causal in production.** `fit` records its first exchange timestamp

as the reference epoch and `transform` measures against *that* epoch, so calibrating

on a warm-up window and applying to later ticks is correct. `fit_transform` is

in-sample by construction — offline research only.

> Full procedure: see `references/workflows.md`.

> Standards and evidence: see `references/standards.md`.

> Printable pre-flight checklist: see `assets/checklist.md`.

## Common Pitfalls

- **Regressing on all delay points.** Network queueing is positive-only. Ordinary least

squares over every sample is biased upward by the mean queueing delay and reads a

congestion episode as clock drift. Filter to window minima first.

- **Re-basing the correction on each batch.** If `transform` measures elapsed time from

its *own* first tick rather than the epoch used at `fit` time, the intercept is applied

at the wrong origin. At 100 ppm, a 300 s gap between calibration and application is a

silent **30 ms** error — larger than most of the effects being studied.

- **Calling the intercept a clock offset.** It is `minimum_transit + clock_offset`, and

the two are not separable from one-way data. Reporting it as an offset overstates the

clock error by the whole propagation delay.

- **Trusting a "strict monotonicity" guarantee in float64 seconds.** Adding a 1 ns

epsilon to a POSIX-epoch float64 is a no-op — the ULP there is ~238 ns — so a naive

implementation emits equal timestamps while claiming they increase. Use int64

nanoseconds.

- **Fitting across a clock step.** A 128 ms NTP step or a leap second is a discontinuity,

not drift; one line across it is wrong everywhere. Detect and segment.

- **Treating a sustained latency regime change as clock skew.** If *every* sample in a

run of windows is delayed — a route change, a switch upgrade, a saturated link — the

lower envelope itself moves, and one-way data cannot distinguish that from the clock.

Minimum filtering only rejects congestion that leaves quiet samples in each window.

Corroborate large fitted drifts against `clock-drift-monitoring-alerting-thresholds`

telemetry before believing them.

- **Extrapolating a stale fit.** Crystal drift changes with temperature. Applying a

morning calibration to an afternoon session accumulates error; the implementation

warns when asked to extrapolate beyond its calibration span.

- **In-sample correction leaking into a backtest.** `fit_transform` over a whole day uses

future ticks to stamp earlier ones. In any research pipeline that feeds a strategy,

calibrate on a prior window only. See `lookahead-bias-elimination`.

- **Silently accepting unsorted input.** Shuffled exchange timestamps used to collapse

the windowing and return zero drift with no error at all, which looks exactly like a

healthy clock.

## Verification

- `python -m unittest discover -s skills/clock-skew-correction-for-tick-timestamps/scripts`

(30 tests). Coverage includes: recovery of a known injected drift rate (positive,

negative and zero) to within 5 ppm; rejection of intermittent congestion that would

push naive OLS past 100 ppm; the reference-epoch regression (chunked `transform` must

equal a single call); strict monotonicity with an epsilon below the float64 ULP;

int64-nanosecond mode preserving 100 ns separation; and rejection of unsorted, NaN,

empty and mismatched input.

- Generate a synthetic feed with a known drift (e.g. 100 µs/s) plus strictly positive

jitter, then confirm `drift_ppm` matches the injected rate within a few ppm and

`np.all(np.diff(out) > 0)` holds.

- Confirm scale: a 2M-tick session fits and transforms in well under a second. If it

takes tens of seconds, the windowing has degraded to a per-window scan of the whole

array.

- Cross-check any material fitted drift against independent clock telemetry

(`chronyc tracking`, `pmc`/`ptp4l` offsets). A regression on market data alone cannot

tell a drifting clock from a drifting network path.

## Related Skills

- `clock-synchronization-ptp-for-trading-hosts`

- `clock-drift-monitoring-alerting-thresholds`

- `cross-vendor-timestamp-precision-reconciliation`

- `hardware-timestamping-vs-software-timestamping-accuracy`

- `historical-order-book-reconstruction-from-message-logs`

- `lookahead-bias-elimination`