57 added, 21 removed. Audit A to A.
---
name: backtest-audit-trail-for-regulatory-review
- description: Use when conducting production backtests to record and sign a immutable
- audit trail capturing code git commit SHA, data version checksum, hyperparameter
- manifest, and execution environment metadata for regulatory review (e.g. SEC/FINRA/MiFID
- II).
+ description: Use when conducting production backtests to record an HMAC-authenticated
+ audit trail capturing code git commit SHA, per-file data checksums (SHA256),
+ hyperparameter manifest, and execution environment metadata for internal governance
+ and regulatory review (SEC Rule 17a-4 recordkeeping context).
domain: algorithmic-trading
subdomain: backtesting-methodology
tags:
- backtesting-methodology
- regulatory-compliance
- audit-trail
- reproducibility
- sec-compliance
- data-lineage
+ - hmac-integrity
brokers_frameworks:
- Regulatory Audit Trail Engine
- - Python Cryptography
- version: "1.0.0"
+ - Python hashlib/hmac (standard library)
+ version: "2.0.0"
author: algo-trading-skills-contributors
license: Apache-2.0
---
## When to Use
- Invoke this skill when performing strategy validation prior to live deployment or institutional client presentation. Regulatory authorities (e.g. SEC, FINRA, ESMA/MiFID II) and institutional risk committees mandate that backtested performance claims be backed by an immutable audit trail. This skill generates cryptographic audit manifests linking code version (git commit SHA), input data checksums, hyperparameter configurations, and execution logs.
+ Invoke this skill when performing strategy validation prior to live deployment or institutional client presentation, and you need a durable record of exactly what code, data, and parameters produced a set of backtested numbers. It captures git commit SHA, per-file input data checksums, hyperparameters, performance metrics, and execution environment, then authenticates the record with HMAC-SHA256 so later modification by anyone without the signing key is detectable.
+ ## When NOT to Use
+
+ - **As proof to a regulator that your firm did not alter its own results.** HMAC is symmetric: whoever can sign can also forge. This gives integrity against outsiders, not non-repudiation against the issuer. See the trust-boundary note below.
+ - **As a substitute for compliant recordkeeping storage.** Under 17 CFR 240.17a-4(f) the durability obligation is met by the storage system (WORM or the audit-trail alternative), not by a hash embedded in a file.
+ - **For live order/trade event capture.** Order and execution event reporting is a different obligation with different systems — see the CAT and RTS 25 notes in `references/standards.md`.
+ - **When you have not yet eliminated lookahead bias.** An impeccably signed manifest for a biased backtest is a precisely documented wrong answer.
+
+ ## Trust Boundary — read before relying on this
+
+ An unkeyed hash over a manifest is **not** tamper-evidence. Anyone can edit the file, recompute the hash, and every check passes. Earlier versions of this skill made exactly that mistake. This version emits two separate values:
+
+ | Value | What it proves | What it does not prove |
+ |---|---|---|
+ | `content_digest_sha256` | The bytes were not corrupted in transit or at rest. Anyone can recompute it. | Nothing about authenticity. |
+ | `manifest_hmac_sha256` | The record was produced by a holder of the signing key and has not been altered since. | Nothing against the key holder themselves. |
+
+ For assurance against the issuing firm, anchor trust outside it: asymmetric signatures with the private key held independently, an RFC 3161 timestamp authority, or third-party-administered write-once storage.
+
## Prerequisites
- - Git repository commit SHA.
- - SHA256 checksums of input price/volume datasets.
- - Strategy parameter configuration dictionary.
+ - Git repository with clean working tree — a commit SHA taken against uncommitted changes documents code that never existed.
+ - Input data files accessible for checksumming.
+ - Strategy parameter configuration (all tunable hyperparameters).
+ - A signing key of ≥32 random bytes from a secrets manager or environment variable — **never** committed to the repository and never stored alongside the manifests it authenticates.
+ - Standard library only: `hashlib`, `hmac`, `json`, `platform`.
## Workflow
- 1. **Capture Lineage Metadata**: Extract git commit SHA, user ID, system timestamp, and OS platform metadata.
- 2. **Compute Input Data Checksums**: Generate SHA256 hashes of all input CSV/Parquet files.
- 3. **Assemble Audit Manifest**: Combine code SHA, data hashes, parameters, and resulting Sharpe/drawdown performance metrics.
- 4. **Generate Immutable Audit Cryptographic Hash**:
- $$\text{AuditHash} = \text{SHA256}(\text{JSON}(\text{Manifest}))$$
+ 1. **Capture Lineage Metadata**: Record git commit SHA (`git rev-parse HEAD`), UTC timestamp as integer nanoseconds, OS/architecture, and Python version. Confirm the working tree is clean first — if `git status --porcelain` is non-empty, stop; the SHA does not describe the code that ran.
+ 2. **Compute Per-File Data Checksums**: Stream each input file to compute SHA256 individually. Never hash a concatenation of files: a single aggregate digest tells you something changed but not which file, which is exactly what an auditor asks.
+ 3. **Assemble Manifest**: Organization, strategy ID, git SHA, per-file checksum map, all hyperparameters, performance metrics, execution metadata, data sources, notes. Any field left out of the signed payload is a field an attacker can change undetected — including the key ID.
+ 4. **Digest and Authenticate**: Serialize as canonical JSON (sorted keys, no whitespace, `allow_nan=False`), then compute both the unkeyed digest and the keyed HMAC. Reject non-finite metrics rather than serializing them — Python writes bare `NaN`, which is invalid JSON and unparseable by strict verifiers in other languages.
+ 5. **Verify on Load**: Check the HMAC, not just the digest. A manifest whose digest matches but whose HMAC does not has been re-digested by someone without the key — treat it as forged, not as corrupted.
+ 6. **Store & Archive**: Write to storage whose retention controls match your regulatory status. If you are a registered broker-dealer, 17 CFR 240.17a-4(f) requires either WORM or an audit-trail system that can recreate a modified or deleted record. Determine the applicable retention tier from the record's classification under 17a-3 — do not assume six years applies (see `references/standards.md`).
> Full procedure: see `references/workflows.md`.
> Standards reference: see `references/standards.md`.
> Printable pre-flight checklist: see `assets/checklist.md`.
## Common Pitfalls
- - **Untracked Code Modifications**: Running backtest on uncommitted local git working copy, rendering SHA meaningless.
- - **Missing Data Versioning**: Failing to record exact dataset version or vendor download date.
+ - **Calling an Unkeyed Hash a Signature**: A plain SHA256 embedded in the record it describes is a checksum. Recomputing it after editing takes three lines. Use a keyed MAC or an asymmetric signature.
+ - **Testing Tamper Detection Without Re-Signing**: Mutating a field and asserting verification fails only proves the digest covers that field. The real adversary recomputes the digest. A tamper test that never re-signs tests nothing about forgery.
+ - **Storing the Signing Key With the Manifests**: A key in the same bucket, repo, or backup set as the records it authenticates provides no protection against anyone who can reach the records.
+ - **Untracked Code Modifications**: Running a backtest against a dirty working tree — the SHA is meaningless for audit.
+ - **Single Aggregate Checksum**: Hashing concatenated data instead of per-file checksums masks which input changed.
+ - **Non-Canonical JSON Serialization**: Unsorted keys, incidental whitespace, or `NaN`/`Infinity` values break verification across implementations.
+ - **Aliasing Caller State**: Holding a reference to the caller's parameter dict rather than a copy means the "immutable" manifest silently changes after signing.
+ - **Float Formatting Across Languages**: Metrics stored as JSON floats may re-serialize differently in another runtime and break byte-level canonical comparison. Where cross-toolchain verification matters, encode metrics as decimal strings.
+ - **Assuming a Retention Period**: 17a-4 sets different tiers for different record classes and applies to registered broker-dealers. Asserting "six years" for a research artifact is unsupported.
## Verification
- - Generate audit manifest for strategy backtest, verify valid SHA256 signature and metadata schema.
- - Run `python scripts/test_regulatory_audit_trail.py` and confirm 100% pass rate.
+ - Run `python scripts/test_regulatory_audit_trail.py` — 100% pass rate (32 tests).
+ - **Forgery test (the one that matters)**: modify a metric, recompute `content_digest_sha256`, then verify. The digest will match and verification must still fail on the HMAC.
+ - Confirm a manifest signed with a different key, or a different `signing_key_id`, fails verification.
+ - Confirm mutating the caller's parameter dict after `build_manifest` does not change the manifest.
+ - Confirm non-finite metrics are rejected at build time rather than serialized.
## Related Skills
- - `backtest-determinism-and-reproducibility`
- - `paper-to-live-promotion-checklist`
+ - `backtest-determinism-and-reproducibility` — ensures bit-identical runs; this skill records what was run
+ - `paper-to-live-promotion-checklist` — final gate before live capital; requires a valid audit trail
+ - `lookahead-bias-elimination` — must pass before an audit trail is meaningful
+ - `walk-forward-validation-setup` — validation methodology whose results belong in the manifest
+ - `centralized-secrets-management-vault-integration` — where the signing key should live
+ - `record-retention-periods-by-jurisdiction` — determining the applicable retention tier
---