72 added, 22 removed. Audit A to A.
---
name: backtest-infrastructure-cost-budgeting
- description: Budgeting compute/storage cost for large-scale backtesting before it
- becomes a surprise cloud bill.
+ description: Budgeting compute and storage cost for large-scale backtest sweeps before
+ it becomes a surprise cloud bill, with explicit modelling of spot interruption
+ overhead, per-task billing minimums, and non-linear parallel scaling.
domain: Backtesting
subdomain: Infrastructure
tags:
- backtesting
- cost
- budgeting
- cloud
+ - capacity-planning
+ - spot-instances
brokers_frameworks:
- - AWS
- - GCP
- version: "1.0.0"
- author: System
- license: MIT
+ - AWS Fargate / EC2
+ - Google Cloud Run / Compute Engine
+ version: "2.0.0"
+ author: algo-trading-skills-contributors
+ license: Apache-2.0
---
- # Backtest Infrastructure Cost Budgeting
-
## When to Use
- Use this skill before kicking off massive grid searches or tick-level backtests over thousands of instruments. Large-scale backtesting can quietly consume thousands of dollars in cloud computing resources if not carefully budgeted in advance.
+ Use this skill before kicking off massive grid searches or tick-level backtests over thousands of instruments. Large-scale backtesting can quietly consume thousands of dollars of cloud spend if it is not budgeted in advance, and the bill arrives long after the decision to run was made.
+
+ ## When NOT to Use
+
+ - **As a billing reconciliation tool.** This estimates ahead of a run. For tracking actual spend against forecast, see `cost-monitoring-for-cloud-trading-infrastructure`.
+ - **As a live cost cut-off.** It produces a number; it does not stop a running job. Wire `is_over_budget` into your launcher yourself.
+ - **For a single small backtest.** Below a few dollars the estimation effort exceeds the spend.
+ - **Without a measured baseline.** Every input is a per-unit measurement. Guessed inputs produce a confidently wrong total — profile a real sample first.
+
+ ## Which Billing Model This Fits
+
+ Costs are modelled as **separately metered vCPU-hours and GB-hours**. That is how serverless container platforms bill: AWS Fargate charges "based on the vCPU, memory, Operating Systems, CPU Architecture, and storage resources used", with CPU and memory as separate line items.
+
+ **EC2 does not bill this way.** AWS states "pricing is per instance-hour consumed for each instance, from the time an instance is launched until it is terminated or stopped" — one rate covering the instance's fixed vCPU and memory together, whether or not you use it. To budget an instance fleet with this tool:
+
+ ```python
+ pricing = CloudPricing(
+ cpu_hourly_rate=0.192, # full instance hourly rate
+ ram_hourly_rate=0.0, # memory is already inside the instance price
+ storage_monthly_rate_per_gb=0.10,
+ )
+ ```
+
+ and express `cpu_hours_per_unit` as *instance-hours* per unit. Leaving `ram_hourly_rate` at a non-zero value under bundled billing double-counts memory you have already paid for.
+
+ No prices are hard-coded in this skill. Published rates change and vary by region, CPU architecture, OS, and commitment level — pull current numbers from the provider's pricing page for your region.
+
## Prerequisites
- - Basic knowledge of cloud instance pricing (e.g., AWS EC2, GCP Compute).
- - Rough estimates of single-backtest resource consumption (memory, time, storage).
- - Total number of parameter combinations and instruments.
+ - A measured baseline from a small representative run: CPU time, peak memory, storage produced, and egress per unit.
+ - Total sweep dimensions (instruments × parameter combinations).
+ - Current rates for your region and billing model.
+
## Workflow
- 1. Run a small representative sample of the backtest.
- 2. Measure the CPU time, peak memory usage, and storage generated.
- 3. Feed these metrics into the `BacktestCostBudgeter` alongside the total sweep space.
- 4. Review the estimated cloud cost before scaling up.
+ 1. **Run a representative sample** — one instrument, one parameter set — and measure CPU time, peak RAM, storage produced, and any egress.
+ 2. **Choose the billing model** and populate `CloudPricing` accordingly (separable vCPU/GB, or bundled instance rate with `ram_hourly_rate=0.0`).
+ 3. **Set the billing quantum.** Fargate bills "per second with a 1-minute minimum". A sweep of thousands of few-second units pays that minimum thousands of times: 5,000 units of 3.6 s each bill as 83 vCPU-hours, not 5 — a 16.7× difference. Set `minimum_billable_seconds=60.0` on Fargate, or `0.0` where billing is genuinely continuous.
+ 4. **Budget the overheads you know you have.** `parallel_overhead_factor` above 1.0 covers scheduler waste, contention, and coordination. Perfect linear scaling is the assumption to justify, not the default to assume.
+ 5. **If using spot, budget the interruptions too.** AWS states "it is always possible that your Spot Instance might be interrupted", and interrupted work has to be redone. `spot_interruption_overhead=0.15` budgets 15% rework on top of the discounted rate. Leaving it at 0.0 assumes perfect checkpointing.
+ 6. **Review `is_over_budget` before scaling up**, and act on it: cut the parameter space, shorten retention, or optimise `cpu_hours_per_unit`.
+
+ > Full procedure: see `references/workflows.md`.
+ > Rate sources and modelling assumptions: see `references/standards.md`.
+ > Printable pre-flight checklist: see `assets/checklist.md`.
+
## Common Pitfalls
- - **Ignoring Storage Costs:** Huge tick databases or massive output logs can dominate costs over CPU hours.
- - **Assuming Linear Scaling:** Sometimes parallel overhead or database locks cause costs to scale non-linearly.
- - **Forgetting Spot Instances:** Not budgeting for the cost savings of spot/preemptible instances.
+ - **Treating the spot discount as free money.** A flat multiplier with no interruption allowance biases the budget *down* — the dangerous direction for a guard. Spot instances are reclaimed when the provider needs capacity back, and partially complete units must be rerun.
+ - **Ignoring the per-task billing minimum.** The single largest error source for wide sweeps of short units, and it always understates.
+ - **Assuming linear scaling.** Parallel overhead, database locks, and scheduler queuing mean 10,000 units rarely cost exactly 100× what 100 units cost. `parallel_overhead_factor` exists to make that assumption explicit.
+ - **Ignoring storage and egress.** Huge tick databases and verbose output logs can dominate CPU cost, and pulling datasets across cloud boundaries is billed per GB.
+ - **Double-counting RAM under bundled instance billing.** See the billing-model section above.
+ - **Feeding in guessed per-unit numbers.** The estimate inherits the accuracy of its inputs and reports the result with the same confidence either way.
+ - **Trusting a NaN.** `float('nan') > budget` is False, so before v2.0 a single NaN input silently reported the sweep as within budget. Non-finite and negative inputs are now rejected outright.
+ - **Budgeting only the successful path.** Failed and retried units are billed too.
+
## Verification
- - Cross-check the budgeter's estimate with actual bills after running a 10% scale job.
+ - Run `python scripts/test_cost_budgeter.py` — 30 tests, 100% pass rate.
+ - Cross-check the estimate against the actual bill after running a 10% scale job, then correct the per-unit inputs rather than the total.
+ - Confirm `is_over_budget` is True for a deliberately oversized sweep, and that a NaN or negative input raises `CostBudgetError` rather than passing the guard.
+ - Confirm the exact-budget boundary behaves as intended: a total equal to the budget is treated as **within** budget.
+
## Related Skills
- - `backtest-parameter-sensitivity-analysis`
+
+ - `backtest-parameter-sensitivity-analysis` — shrinking the sweep space is the cheapest cost lever
+ - `cost-monitoring-for-cloud-trading-infrastructure` — tracking actual spend against this forecast
+ - `capacity-planning-for-symbol-universe-growth` — how the sweep dimensions grow over time
+ - `walk-forward-hyperparameter-search-budget` — bounding the search before it is priced
+ - `load-testing-before-scaling-to-new-instrument-universe` — validating the per-unit baseline at scale