47 added, 28 removed. Audit A to A.
---
name: backtest-determinism-and-reproducibility
description: Use when designing backtest execution frameworks to enforce deterministic
- random seeding, sort input data streams strictly, eliminate async race conditions,
- and verify bit-identical P&L results via cryptographic audit checksums.
+ random seeding, sort input data streams strictly, replace wall-clock reads with a
+ simulated clock, and detect run-to-run divergence via exact-bit trade log checksums.
domain: algorithmic-trading
subdomain: backtesting-methodology
tags:
- backtesting-methodology
- reproducibility
- determinism
- random-seed
- bit-identical
- audit-checksum
- regression-testing
brokers_frameworks:
- Determinism Reproducibility Engine
- - Python NumPy
- - Random
- version: "1.0.0"
+ - Python random / hashlib
+ - NumPy
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Invoke this skill when building backtest simulation engines or automated CI/CD regression test suites. If backtest runs produce varying P&L curves or Sharpe ratios across identical code and data inputs, strategy parameter optimization becomes impossible to trust. Non-determinism stems from unseeded pseudo-random number generators, unordered dictionary iteration, async thread race conditions, or unanchored system clocks. This skill guarantees 100% bit-identical backtest results.
+ Invoke this skill when building backtest simulation engines or CI regression suites. If backtest runs produce varying P&L curves or Sharpe ratios across identical code and data, parameter optimization becomes impossible to trust — you cannot tell an improvement from noise. Non-determinism comes from unseeded RNGs, set iteration order, unordered event streams, wall-clock reads, and floating-point accumulation order.
- ## Prerequisites
+ ## What This Can and Cannot Guarantee
- - Master integer random seed $S_{\text{master}}$ (e.g. `seed=42`).
- - Fixed sorting specification for input bar/tick event streams.
+ **It cannot guarantee bit-identical results in general, and no tool can.** PyTorch's reproducibility guidance states plainly that "completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms," and that results may differ between CPU and GPU "even when using identical seeds." The same holds for BLAS thread counts, SIMD code paths, and library versions.
- ## Workflow
+ Determinism is a property of a **pinned environment** — same platform, same library versions, same thread configuration. This skill does two things within that envelope:
- 1. **Inject Global Master Random Seeds**:
- - Enforce fixed seeding across all RNG libraries (`random.seed(S)`, `numpy.random.seed(S)`, `torch.manual_seed(S)`).
- - Set environment variable `PYTHONHASHSEED=0`.
+ 1. **Eliminates the controllable sources**: seeding, event ordering, clock reads.
+ 2. **Detects divergence when it happens**, by comparing exact float bits.
- 2. **Enforce Deterministic Data Stream Sorting**:
- - Sort input event streams strictly by `(timestamp, symbol, sequence_id)`. Resolve identical-timestamp ties deterministically by symbol name.
+ Point 2 is the part that must not be compromised. Earlier versions of this skill rounded every float to 6 decimals before hashing, which meant the canonical non-determinism signature — `sum([0.1]*10)` (`0.9999999999999999`) versus `0.1*10` (`1.0`) — was reported as bit-identical. A detector that cannot see the thing it was built to detect is worse than no detector, because it produces confident false assurance.
- 3. **Replace System Clocks with Simulated Time**:
- - Rebind `time.time()` calls to simulated bar/tick event timestamps ($T_{\text{sim}}$).
+ ## When NOT to Use
- 4. **Generate Cryptographic Audit Checksum**:
- - Compute SHA256/MD5 hash of resulting trade execution logs and equity curve array:
- $$\text{Hash}_{\text{audit}} = \text{MD5}(\text{trade\_list\_json})$$
- - Assert $\text{Hash}_{\text{run1}} \equiv \text{Hash}_{\text{run2}}$.
+ - **As a tamper-evident audit record.** The checksum is an unkeyed SHA256: anyone can edit a trade log and recompute it. For an authenticated record see `backtest-audit-trail-for-regulatory-review`.
+ - **To compare runs across different environments.** Differing Python/NumPy/BLAS versions or thread counts will legitimately diverge; that is an environment-pinning problem, not something a checksum can fix.
+ - **To validate strategy correctness.** A perfectly reproducible backtest can be reproducibly wrong — see `lookahead-bias-elimination`.
+ ## Prerequisites
+
+ - A master integer seed (e.g. `42`).
+ - `PYTHONHASHSEED` set **in the environment before the interpreter starts**.
+ - Pinned library versions and a fixed thread configuration for any run you intend to compare.
+
+ ## Workflow
+
+ 1. **Set `PYTHONHASHSEED` before launching Python.** This is the trap that catches people: hash randomization is fixed at interpreter startup, so `os.environ["PYTHONHASHSEED"] = "0"` inside running code does **nothing** while looking like it works. Launch as `PYTHONHASHSEED=0 python backtest.py`. `check_hash_seed()` reports whether the current process is safe; it cannot fix it.
+ 2. **Seed the RNGs — and know what seeding misses.** `apply_master_seeds()` seeds `random`, NumPy's legacy global `RandomState`, and torch when present. NumPy documents `np.random.seed` as a legacy function and recommends a dedicated `Generator`; code calling `np.random.default_rng()` draws fresh OS entropy and stays non-deterministic no matter how often you seed the global. Use `make_numpy_generator()` and thread it through explicitly.
+ 3. **Sort the event stream by `(timestamp, symbol, sequence_id)`.** All three keys are required — a missing timestamp is rejected rather than defaulted to `0.0`, which would silently move that event to the front. Duplicate sort keys are also rejected: Python's sort is stable, so tied events would keep their input order and the result would depend on how the file happened to be read.
+ 4. **Replace wall-clock reads with `SimulatedClock`.** Inject it rather than monkey-patching `time.time()` — patching a global affects every library in the process, including ones whose correctness depends on real time. The clock refuses to move backwards, which surfaces an unsorted stream immediately.
+ 5. **Checksum with exact float bits and compare runs.** `audit_reproducibility()` reports `is_bit_identical` and, on failure, `first_divergence_index` so you can go straight to the offending trade instead of staring at two mismatched hex strings.
+
> Full procedure: see `references/workflows.md`.
- > Standards reference: see `references/standards.md`.
+ > Standards and sources: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Unseeded Monte Carlo Simulations**: Running slippage or execution timing perturbations without fixing random seed states across runs.
- - **Unordered Set/Dict Iteration**: Iterating over Python sets or dict keys when iterating strategy universe symbols, introducing non-deterministic execution order.
- - **Calling System Clock `time.time()`**: Using live wall-clock timestamps inside strategy signal generation logic instead of simulated event time.
+ - **Rounding before hashing.** Any tolerance applied before comparison erases exactly the sub-ulp divergences that signal non-determinism. `float_precision` exists for callers who explicitly want tolerant comparison, and it logs a warning saying what it costs.
+ - **Setting `PYTHONHASHSEED` from inside Python.** It has no effect. The assignment succeeds, the variable reads back correctly, and set iteration order stays random.
+ - **Assuming dicts and sets behave alike.** Dicts have preserved insertion order since Python 3.7 and are deterministic. **Sets are not** — iterating a symbol universe as a set gives a different order per process. With `PYTHONHASHSEED` at 0, 1, and 2, `{'AAPL','MSFT','GOOG','TSLA'}` iterates in three different orders. Iterate a sorted list.
+ - **Seeding only the NumPy global.** `np.random.seed()` does not affect any `Generator` created by `default_rng()`.
+ - **NaN in a trade log.** `json.dumps` renders every NaN as the same token, so two runs that both corrupted to NaN hash identically and are declared reproducible. Non-finite values are now rejected outright.
+ - **Unsigned notional mistaken for P&L.** Summing `quantity * price` across a round trip gives gross turnover, not equity: BUY 100@150 then SELL 100@155 sums to 30,500 while the realized P&L is 500. Cash flow must carry the side's sign.
+ - **Monte Carlo without a fixed seed.** Slippage or execution-timing perturbations reseeded per run make every comparison meaningless.
+ - **Treating a matching checksum as proof of correctness.** It proves two runs agreed, nothing more.
## Verification
- - Execute backtest twice with identical inputs, verifying bit-identical trade log checksums ($\text{MD5}_1 == \text{MD5}_2$).
- - Change master random seed and verify deterministic checksum shift.
- - Run `python scripts/test_reproducibility_engine.py` and confirm 100% pass rate.
+ - Run `python scripts/test_reproducibility_engine.py` — 40 tests, 100% pass rate.
+ - **The test that matters**: feed `sum([0.1]*10)` and `0.1*10` as prices for otherwise identical runs and confirm `is_bit_identical` is `False`. Pre-2.0 this returned `True`.
+ - Run a backtest twice with identical inputs and confirm matching checksums; change the master seed and confirm the checksum shifts.
+ - Confirm a missing `timestamp` or a duplicate `(timestamp, symbol, sequence_id)` raises `DeterminismError` rather than sorting silently.
+ - Confirm `SimulatedClock.advance_to()` rejects a backwards jump.
## Related Skills
- `walk-forward-validation-setup`
- `monte-carlo-strategy-robustness-testing`
- `vectorized-vs-event-driven-backtest-tradeoffs`
+ - `backtest-audit-trail-for-regulatory-review` — for an authenticated audit record, which this skill's unkeyed checksum is not
+ - `reproducible-ml-training-pipelines` — the same problem for model training
+ - `dependency-pinning-and-reproducible-builds` — pinning the environment determinism depends on
---