ai-cost-audit ยท diff
git:20260807.975fe85 to git:20260807.fac9f85
59 added, 16 removed. Audit A to A.
---
name: ai-cost-audit
- description: Audit AI model usage and calculate unit economics, margins, routing, caching, and batch savings. Use when evaluating AI costs, pricing, or model changes.
+ description: Audit AI usage, billing, unit economics, routing, caching, batching, and optimization. Use when AI cost, margin, pricing, model choice, or savings must be measured or verified.
license: MIT
---
# AI cost audit
- Turn model calls into an evidence-backed cost model and a prioritized optimization plan. Audit the code and current provider pricing; never estimate from model names alone.
+ Build a cost model from observed usage and current primary-source prices. Treat repository intent, measured usage, invoices, and projections as different evidence classes.
+ ## Choose a mode
+
+ - **Inventory:** map model/media calls, routing, retries, and ownership.
+ - **Economics:** calculate cost per action, user, plan, and month.
+ - **Simulation:** compare pricing, volume, model, cache, batch, or abuse scenarios.
+ - **Optimization:** rank changes after measuring quality and operational risk.
+ - **Reconciliation:** explain the gap between bottom-up estimates and provider invoices.
+ - **Verification:** confirm that a completed change reduced spend without unacceptable quality or latency regressions.
+
+ Use the narrowest mode that answers the request. Combine modes only when the user asks for a full audit or the dependency is necessary.
+
## Workflow
- 1. Inventory every model call, embedding job, image/audio operation, reranker, fallback, retry, and background batch. Record file, provider, model, purpose, input source, output limit, frequency, and user-facing latency requirement.
- 2. Trace routing and retries. Identify calls hidden behind SDK wrappers, agents, queues, cron jobs, evaluations, and provider fallbacks.
- 3. Gather measured usage from logs or billing exports where available. Separate p50, p95, and worst-case tokens or media units. Mark assumptions explicitly.
- 4. Fetch current primary-source pricing for every provider and model, including cached input, batch, reasoning tokens, tool calls, media units, and minimum charges.
- 5. Calculate cost per action, active user, free user, paid plan, and month. Include retry rates, cache-hit rate, provider overhead, payment fees when relevant, and the cost of free allowances.
- 6. Compare revenue and cost to compute gross margin and break-even usage. Run normal, high-usage, and abuse scenarios.
- 7. Evaluate optimizations in order: remove unnecessary calls, reduce context, cap outputs, cache stable prefixes/results, batch asynchronous work, route by task difficulty, and change models only after quality evaluation.
- 8. Rank recommendations by expected monthly savings, quality risk, engineering effort, latency effect, and reversibility.
- 9. Add or propose measurement where the repository cannot support a defensible calculation.
+ 1. Inspect the repository, existing telemetry, billing exports, pricing configuration, and prior analyses before asking questions.
+ 2. State the audit boundary: environments, date range, providers, features, currencies, taxes, credits, and whether non-model infrastructure is included.
+ 3. Inventory direct and indirect calls: generation, reasoning, embeddings, reranking, tools, image/audio/video, moderation, retries, fallbacks, agents, queues, evaluations, and batches.
+ 4. Prefer provider-metered tokens or media units. Keep measured values, code-derived estimates, generic estimates, and assumptions visibly separate. Use p50, p95, and worst-case where available.
+ 5. Fetch current prices only from official provider sources when pricing affects the answer. Record URL, retrieval date, region/tier/currency, and special terms such as cached input, reasoning tokens, batch, storage, or minimum charges. Do not rely on bundled price tables or memory.
+ 6. Model each cost path, including failed calls, retry amplification, tool loops, cache writes/reads, storage, egress, gateway fees, payment fees, free allowances, and shared fixed costs when relevant.
+ 7. Reconcile the modeled total against invoices or billing dashboards. Quantify unexplained variance instead of forcing agreement.
+ 8. Run normal, high-usage, abuse, and sensitivity scenarios. Do not apply universal margin or traffic thresholds without the product's business constraints.
+ 9. Rank recommendations by expected savings range, evidence confidence, quality risk, latency effect, engineering effort, reversibility, and measurement plan.
+ 10. Require an evaluation and canary before changing models, prompts, routing, or output limits. Verify spend, quality, latency, error rate, and user outcomes afterward.
- ## Load deeper guidance
+ Ask only for inputs that cannot be recovered from the scoped artifacts and materially change the result.
- - Read [model inventory](references/model-inventory.md) for codebase discovery, routing analysis, and audit reporting.
- - Read [unit economics](references/unit-economics.md) for formulas, margin scenarios, caching, and batch calculations.
+ ## Safety and evidence rules
- ## Output
+ - Never invent usage, conversion, revenue, invoice, cache-hit, or quality inputs.
+ - Never expose secrets found in environment files or billing exports.
+ - Do not recommend a cheaper model solely from task labels; require representative evaluation data.
+ - Label projections as projections and show formulas with units.
+ - Treat prices and model availability as time-sensitive.
- Return the inventory, assumptions, formulas, scenario table, highest-cost paths, and prioritized changes. Cite pricing sources with retrieval dates. Never invent usage, conversion, or margin inputs.
+ ## Load conditional references
+
+ - Read [model inventory](references/model-inventory.md) for discovery patterns, call-path fields, and routing analysis.
+ - Read [unit economics](references/unit-economics.md) for formulas, reconciliation, scenario design, and recommendation ranking.
+
+ These references provide specialist detail only. This file controls mode, interaction, safety, output, and verification.
+
+ ## Output contract
+
+ Return only sections supported by the selected mode:
+
+ - scope and evidence ledger;
+ - model/media call inventory with file paths and ownership;
+ - assumptions and data gaps;
+ - formulas with units and a scenario table;
+ - highest-cost paths and invoice reconciliation;
+ - prioritized changes with savings ranges and quality gates;
+ - verification results or an explicit measurement plan.
+
+ Cite every time-sensitive price with an official source and retrieval date.
+
+ ## Verify
+
+ - Inventory covers wrappers, retries, fallbacks, background jobs, and non-text operations.
+ - Totals preserve units, currencies, date ranges, and environment boundaries.
+ - Measured and estimated inputs are distinguishable.
+ - Bottom-up totals are reconciled to billed totals when billing is available.
+ - Recommendations include a quality gate, rollback path, and owner or next action.
+ - Completed optimizations are verified with before/after spend, quality, latency, and error data.