run-pipeline git:20260902.98fd632A

Immutable. This exact content is served forever at /api/v1/blob/661bc702e5ca4144.

---
name: run-pipeline
description: |
  Run the complete end-to-end analysis pipeline: from data and a business question to a validated Marp deck with charts, via the DAG of agents in agents/registry.yaml. Use for `/run-pipeline`, "run the full pipeline", "build me a deck", "do the full analytical treatment", or any L4-L5 request that needs hypothesis generation, root cause, sizing, storyboarding, charting and deck creation orchestrated together.
---

# Skill: Run Pipeline

## Purpose
Single entry point for end-to-end analysis — from raw data to finished slide deck. Uses a DAG-based execution engine that reads agent dependencies from `agents/registry.yaml`, resolves execution order automatically, and supports parallel agent execution, resume from failure, and execution plan pruning.

## When to Use
Invoke with: `/run-pipeline`, "run the full pipeline", "analyze end-to-end", or "take this data through the full workflow".

## Completion

The pipeline's deliverable is a presentation-ready deck; validation, checkpoints and metrics serve that. You are operating autonomously: the user is not watching in real time, so for reversible steps that follow from the request, proceed without asking. Before ending your turn, check your last paragraph — if it is a plan, a question, or a promise about work not yet done, do that work now. End only when the deck exists or you are blocked on input only the user can provide.

**If you cannot complete the full DAG**, produce the best deck the reached phase supports (data profile → 5-slide profile deck; analysis → findings deck with charts; storyboard → draft deck from beats), mark `pipeline_state.json` as `degraded` with the skipped phases, save every completed artifact to `outputs/`, and tell the user what `/resume-pipeline` would add.

## Fast Path Guidance

The full DAG with all 11 rules is designed for complex L4-L5 analyses. For simpler cases, you can streamline:

**When to use the simplified path:**
- User provides ALL required arguments inline (`/run-pipeline data_path=... question=... theme=...`)
- The question is focused and doesn't require hypothesis generation (e.g., "compare conversion rates")
- Time or context constraints make full execution challenging

**Fast path workflow:**
1. Skip or abbreviate framing (use question as-is)
2. Run data exploration + basic analysis inline (no separate agents)
3. Generate 2-3 key charts directly
4. Create a short deck (5-10 slides) with findings
5. Export to PDF/HTML if Marp CLI available

**Still required on fast path:**
- R1 (theme handling), R2 (chart ≠ headline), R3 (chart background), R7 (figsize)
- At least one chart
- A Marp deck (even if short)
- HTML components (can be minimal - just `.so-what` + `.chart-container`)

The fast path prioritizes **delivery over perfection** - a 7-slide deck with 2 charts beats stopping partway through a 20-slide masterpiece.

## Accepted Arguments

| Argument | Required | Default | Description |
|----------|----------|---------|-------------|
| `data_path` | Yes | — | Path to CSV, parquet, or directory of data files |
| `question` | Yes | — | The business question to answer |
| `context` | No | `"stakeholder readout"` | Presentation context: "stakeholder readout", "workshop", "talk", "team standup" |
| `theme` | No | `analytics` (light) | Theme override: "analytics" (light) or "analytics-dark" (dark) |
| `audience` | No | `"senior stakeholders"` | Who will see the deck — controls content density |
| `dataset_name` | No | Derived from data_path | Short name for file naming (e.g., "hawaii", "my_dataset") |
| `plan` | No | `full_presentation` | Execution plan: `full_presentation`, `deep_dive`, `quick_chart`, `refresh_deck`, `validate_only`, or inline agent list |
| `dry-run` | No | `false` | If `true`, print execution plan without running agents |
| `agents` | No | — | Inline agent allow-list (e.g., `agents=question-framing,hypothesis,data-explorer`) |

Arguments can be passed inline or prompted interactively:
```
/run-pipeline data_path=data/your_dataset/ question="What's driving the decline in revenue?" plan=deep_dive
/run-pipeline dry-run=true
/run-pipeline plan=refresh_deck
```

If required arguments are missing, prompt the user before proceeding.

---

## Pipeline rules (checked at checkpoints)

### R1: Theme Default is Light
Standard analysis → `analytics` (light theme). Dark theme (`analytics-dark`) only when `context` is "workshop" or "talk", OR when the user explicitly passes `theme=analytics-dark`. When in doubt, use light.

### R2: Chart Title ≠ Slide Headline
The chart's baked-in `title` (the SWD action title) MUST differ from the slide headline. The chart title is a specific data claim with numbers. The slide headline is narrative framing.

| Slide Headline | Chart Title | Verdict |
|---------------|-------------|---------|
| "Payment issues drove the June spike" | "Payment issues drove the June spike" | **BAD** — identical |
| "Payment issues drove the June spike" | "Payment tickets jumped 147% while other categories grew <20%" | **GOOD** |

### R3: Chart Background is #F7F6F2
All charts use warm off-white background (#F7F6F2), never pure white (#FFFFFF). This is set by `swd_style()` — verify it was called before every chart.

### R4: Recommendations Ordered by Confidence
Recommendations are always ordered High → Medium → Low confidence. Never alphabetical, never by topic.

### R5: Voice — Banned Words
Headlines, transitions, and breathing slides must never use: **surgical, devastating, exploded, ticking time bomb, smoking gun, alarm/fire metaphors, unprecedented** (unless literally true), **unleash, supercharge, game-changing, skyrocketed**.

### R6: Breathing Slides Every 3-4 Insight Slides
Never more than 4 consecutive chart/insight slides without a pacing break. Pacing classes: `impact`, `dark-impact`, `section-opener`, `takeaway`.

### R7: Charts at Standard Figsize
Every chart is generated at (10, 6) figsize / 150 DPI (~1500x900px) and used directly on slides. CSS `object-fit: contain` handles all containment. No slide variants needed.

### R8: Agent Files Must Be Read from Disk
At each phase, read the agent file from its path on disk. Do NOT rely on cached knowledge or memory.

### R9: Cross-Verification After Analysis
After analysis agents complete and before storytelling, run the Cross-Verification agent (step 6.5) to verify analytical findings via same-source, different-calculation-path checks. The data-explorer sanity gate (step 2.5, always-on) handles pre-analysis boundary checks. HALT if cross-verification confidence score is 0-7.

### R10: All Marp Decks Must Use HTML Components
Every Marp deck must use at least 3 different HTML component types from the theme (e.g., `.kpi-row`, `.so-what`, `.finding`, `.rec-row`, `.chart-container`). Valid slide classes include `chart-full`, `kpi`, `takeaway`, `recommendation`, `appendix` (new one-job-per-slide classes) and all existing classes (`insight`, `impact`, `chart-left`, `chart-right`, `two-col`, `diagram`, `section-opener`, `title`). Plain-markdown-only insight slides are a pipeline failure. The deck-creator agent must read `templates/marp_components.md` for the snippet library and `templates/deck_skeleton.marp.md` for the skeleton template. Frontmatter must include all 6 required keys: `marp`, `theme`, `size`, `paginate`, `html`, `footer`. Run `helpers/export/marp_linter.py` to validate.

### R11: Pipeline Exports Both PDF and HTML
After deck creation and Checkpoint 4, the pipeline must export the deck to both PDF and HTML using `helpers/export/marp_export.py`. Export paths are recorded in `pipeline_state.json`. If Marp CLI is not available, log a warning and skip export (do not HALT). The exported files go alongside the deck: `outputs/deck_{{DATASET_NAME}}_{{DATE}}.pdf` and `outputs/deck_{{DATASET_NAME}}_{{DATE}}.html`.

### R12: Query Log — Every SQL Query Gets Logged
Every data-touching agent must log every SQL query using `append_entry()` from `helpers/provenance/query_log.py`. The log file is `working/query_log_{dataset}_{date}.jsonl`. Required fields: agent name, pipeline step, purpose (analytical reason, not SQL content), SQL text, tables accessed, result summary, row count. Failed queries are logged with `status="error"`. The query log feeds the validation agent (claim matching) and the receipt generator (audit trail). Agents that do not run SQL (question-framing, hypothesis, story-architect, deck-creator) are exempt.

### R13: Tier 1 Validation is Always-On
All analysis agents must run Tier 1 checks before writing findings. Tier 1a (hard boundaries: negative revenue, % > 100, future dates, zero denominators) → HALT on failure. Tier 1b (soft: SQL sanity, arithmetic consistency, source citation) → FLAG only. Zero additional queries. Results logged to `working/tier1_checks_{{DATASET_NAME}}.md`.

### R14: Data Stamps on Every Finding
Every finding presented to the user — in terminal, slides, Google Docs, Notion, or Slack — must include a data stamp. The storytelling agent calls `build_provenance_blocks()` from `helpers/provenance/provenance_assembler.py` and embeds `data_stamp.one_liner` per finding. Format: `[{row_count} rows | {date_range} | {primary_table} | Confidence: {grade} ({score}/100)]`. Data stamps are assembled from provenance blocks if available, or from analysis metadata if not. Data stamps are never gated on validation tier — they appear at all tiers, all export formats.

---

## DAG EXECUTION ENGINE

The pipeline runs on a DAG (directed acyclic graph) derived from `agents/registry.yaml`. Instead of hardcoded steps, the engine resolves execution order from agent dependencies.

### Step 0: Pre-execution Cleanup (Crash Recovery)

Before validation, detect and clean up artifacts from a previous crashed run.

1. **Detect stale runs:** Check if `working/pipeline_state.json` (or `working/latest/pipeline_state.json`) exists with `status: running`.
   - Parse `updated_at` and compute elapsed time. If > 30 minutes ago, treat as stale.
   - Print: `"Found stale pipeline state from {updated_at}. Previous run may have crashed."`
   - Ask: `"Archive stale state and start fresh? (Y/n)"`
   - **If yes:** Rename `working/pipeline_state.json` to `working/crashed_{run_id}_state.json`. Continue to Phase 0.
   - **If no:** Redirect to `/resume-pipeline` to attempt resuming the previous run. Stop here.
   - If `updated_at` is within 30 minutes, assume another run is active. HALT with: `"Pipeline state shows an active run from {updated_at}. Use /resume-pipeline or wait for it to finish."`

2. **Clean temp files:** Delete any `working/*.tmp.json` files (partial atomic writes from a crashed run).

3. **Validate per-run directory:** If prior run left an orphaned `working/latest` symlink:
   - Remove the stale symlink (the new run will create its own in Phase 1).
   - Create `working/runs/{run_id}/` directory structure with `working/`, `outputs/` subdirectories.

4. **Initialize fresh state:** The actual `pipeline_state.json` creation happens in Phase 1 with `schema_version: 2` and all agents set to `pending`. Step 0 only ensures the workspace is clean.

After cleanup completes (or is skipped if no stale state found), proceed to Phase 0.

---

### Phase 0: Pre-flight Validation

The registry parse, file-existence and dangling-dependency checks, cycle detection, tier computation, and plan filtering are all in `helpers/pipeline/dag.py`. Run them once:

```python
from helpers.pipeline.dag import load_registry, load_plans, validate_registry, resolve_plan, plan_warnings

registry = load_registry()                      # agents/registry.yaml, keyed by name
plans = load_plans()                            # .claude/skills/run-pipeline/plans.md
plan = pipeline_args.get("plan", "full_presentation")   # a plan name, or the inline agents= list
plan_agents = plans[plan]["agents"] if isinstance(plan, str) else plan

errors = validate_registry(registry)            # missing agent files, unknown dependencies
if errors:
    raise SystemExit("Pre-flight failed:\n" + "\n".join(errors))   # HALT

tiers = resolve_plan(registry, plan_agents)     # raises DagError on a cycle or unknown agent
for w in plan_warnings(registry, plan_agents):  # plan agents whose dependencies are skipped
    print("WARNING:", w)
```

`tiers` is the execution order: `tiers[0]` has no in-plan dependencies, each later tier depends only on earlier ones. Agents not in `plan_agents` are skipped.

### Phase 1: Initialize Run Directory & Pipeline State

Every run gets an isolated directory under `working/runs/`. `init_run` creates it, writes the initial `pipeline_state.json` (schema in `agents/pipeline_state_schema.md`, every plan agent `pending`, other pipeline agents `skipped`, `tiers` recorded), creates the empty query log, and points `working/latest` at the run:

```python
from helpers.pipeline.dag import init_run

RUN_DIR = init_run(DATASET_NAME, question, plan_agents, registry=registry)
# RUN_DIR = working/runs/{YYYY-MM-DD}_{DATASET_NAME}_{question-slug}/
#   working/   outputs/   pipeline_state.json   working/query_log_{DATASET_NAME}_{DATE}.jsonl
QUERY_LOG = RUN_DIR / "working" / f"query_log_{DATASET_NAME}_{DATE}.jsonl"
```

Agents keep writing to the top-level `working/` and `outputs/` as before; also keep a copy of the query log at `working/query_log_{DATASET_NAME}_{DATE}.jsonl` for the agents that expect it there, and set `{{QUERY_LOG}}` in every agent's context. At pipeline end the final artifacts are copied into `{RUN_DIR}/working/` and `{RUN_DIR}/outputs/` so the run directory is self-contained.

If **resuming** (`working/latest/pipeline_state.json`, else `working/pipeline_state.json`, exists with `status: paused` or `failed`): leave `completed` agents alone, reset `failed` agents to `pending`, then compute the ready set and skip to Phase 2:

```python
from helpers.pipeline.dag import ready_set
READY = ready_set(registry, state["plan_agents"], state)
print(f"Resuming from {completed_count} completed agents. Next: {READY}")
```

### Phase 2: Walk the DAG

Execute agents tier by tier:

```
FOR each tier in execution_tiers:
  1. READY_SET = ready_set(registry, plan_agents, state) ∩ this tier
     (AND-gate on `depends_on`, OR-gate on `depends_on_any`; only completed
     agents satisfy a gate, skipped ones do not — the helper applies both rules)

  2. If is_deadlocked(registry, plan_agents, state) → HALT

  3. FOR each agent in READY_SET:
     a. Mark agent status: running in pipeline_state.json
     b. Record started_at timestamp
     c. Assemble dynamic context (see Context Assembly below)
     d. Read agent file from disk (R8)

  4. LAUNCH agents:
     - If a sub-agent tool is available and READY_SET has 2+ agents: launch them
       in parallel, each with agent file + context, and keep working while they run
     - Else: Execute sequentially inline

  5. WAIT for completion (see Agent Failure Handling)

  6. FOR each completed agent:
     a. Record completed_at, output_files in pipeline_state.json
     b. Record timing in pipeline_metrics
     c. If FAILED and agent.critical is true (default): increment failure counter
     d. If FAILED and agent.critical is false (warn_on_failure):
        - Log warning: "⚠ Non-critical agent {name} failed: {error}. Continuing."
        - Write stub output to agent's first output path:
          `# {name} — SKIPPED (failure)\nReason: {error}\nTimestamp: {iso_now}`
        - Mark status as `degraded` in pipeline_state.json
        - Queue warning for display at next checkpoint
        - Do NOT increment tier failure counter

  7. CIRCUIT BREAKER: If 3+ critical agents failed in this tier → HALT pipeline
     Report: "Circuit breaker tripped: {N} failures in tier {T}. Failed: {names}"

  8. CHECKPOINT: If a checkpoint fires after this tier, run it (see Checkpoints)

  9. Update working/pipeline_summary.md with phase results

  10. ADVANCE to next tier
```

### Dynamic Context Assembly

Before launching each agent, resolve its runtime context:

1. **System variables:**
   - `{{DATE}}` → current date YYYY-MM-DD
   - `{{DATASET_NAME}}` → from `dataset_name` argument or derived from data_path
   - `{{ACTIVE_DATASET}}` → from `.knowledge/active.yaml`
   - `{{BUSINESS_CONTEXT_TITLE}}` → derived from question

2. **Knowledge context:** For each path in the agent's `knowledge_context` from registry:
   - Replace `{active}` with the active dataset name
   - Read the file and include its content as context for the agent

3. **Dependency outputs:** For each completed dependency agent, gather its `output_files` from pipeline_state.json. These become available inputs for the current agent.

4. **Pipeline arguments:** Pass through `context`, `theme`, `audience`, `data_path` as relevant to the agent's `inputs` list.

### Dry-Run Mode

When `dry-run=true`:

1. Run Phase 0 (pre-flight validation) — detect any issues
2. Print `tiers` from `resolve_plan` with the checkpoint that fires after each tier:
   ```
   Execution Plan (dry-run):
   Plan: {plan_name}
   Agents: {count} active, {count} skipped

   Tier 0: [agent-a, agent-b]           (parallel)
   Tier 1: [agent-c]                    (sequential)
     Checkpoint 1: Frame Verification
   Tier 2: [agent-d, agent-e]           (parallel)
     Checkpoint 2: Analysis Verification
   ...

   Estimated steps: {count}
   Checkpoints: {list}
   ```
3. Do NOT execute any agents. Return after printing.

---

## CHECKPOINTS

Checkpoints are gates between pipeline phases. They verify quality before advancing. Checkpoints fire based on which agents just completed, not on hardcoded step numbers.

### Checkpoint 1 — Frame Verification (after hypothesis completes)

**Type:** B (user-facing). **Plans:** full_presentation, deep_dive.

Self-checks:
- [ ] Business question is specific and decision-oriented
- [ ] Analysis design spec names specific tables/columns
- [ ] At least 3 hypotheses span multiple cause categories
- [ ] Agent files were read from disk

Present summary:
> "Questions framed. Design spec ready.
> - Business question: [summary]
> - Tables: [list]
> - Hypotheses: [count] across [N] categories
>
> Proceed to analysis?"

**Skip if:** User said "just do it" or provided all params.

### Checkpoint 2 — Analysis Verification (after cross-verification completes)

**Type:** A (automated). **Plans:** full_presentation, deep_dive.

Verify:
- [ ] Cross-verification confidence score >= 8 (from `working/cross_verification_{{DATASET_NAME}}_{{DATE}}.md`)
- [ ] No Tier 1a boundary check failures (HALT if any)
- [ ] Root cause is specific and actionable
- [ ] Findings are validated via Type B-D checks
- [ ] Query log coverage >= 80%
- [ ] Data quality issues documented
- [ ] Opportunity sizing includes sensitivity analysis

If cross-verification score is 0-7, HALT. If root cause is vague, re-run root-cause-investigator.

### Checkpoint 2.1 — Validation Menu (after validation completes)

**Type:** B (user-facing). **Plans:** full_presentation, deep_dive.

Offer the user a validation menu based on the question level:

**L3 (Guided Analysis) — Compact offer:**
```
Validation passed (score: {score}/100, grade {grade}).
Run deeper verification? (y/N)
```
Default: skip. If yes → run Tier 2 standard (Types A+B always, C+D when schema affords, 3-run reproducibility).

**L4/L5 (Investigation/Presentation) — Full menu:**
```
Validation passed (score: {score}/100, grade {grade}).
Choose validation depth:
  (a) Standard — Types A+B, 3-run reproducibility (default)
  (b) Deep — All types + 5-run reproducibility + full receipt
  (c) Skip — Proceed without additional validation
```
Default: (a) Standard. Option (b) sets `validation_tier=3` and triggers receipt generation at step 18.5.

**Skip conditions (CP 2.1 does not fire):**
- User said "just do it" or passed `validation_tier` explicitly
- Question level is L1/L2 (Tier 1 always-on is sufficient)
- Preference learning auto-apply is active (see below)

**Preference learning:**
Track the user's validation choices per question level across pipeline runs. Store in `working/validation_preferences.yaml`:
```yaml
preferences:
  L3: {choice: skip, run_count: 3}
  L4: {choice: standard, run_count: 5}
```
After 3 consecutive same-choice runs at a given level, auto-apply on the 4th without asking. Print: `"Auto-applying your usual choice ({choice}) for L{level}. Use /validate reset-preferences to see the menu again."`

The user can reset with `/validate reset-preferences` to clear saved preferences and restore the menu.

### Checkpoint 2.5 — Storyboard Review (after narrative-coherence-reviewer completes)

**Type:** B (user-facing). **Plans:** full_presentation only (L5).

Present storyboard summary with beat headlines and arc structure.

**Skip if:** User said "just do it" or reviewer flagged issues (go to revision).

### Checkpoint 3 — Story & Charts (after visual-design-critic chart-level completes)

**Type:** A (automated). **Plans:** full_presentation, quick_chart.

Verify: R2 (title collision scan), R3 (backgrounds), R5 (banned words), R7 (chart figsize), story arc, chart fan-out results. Print title collision table.

**Fix Loop (chart-maker-fixes):**
After the visual-design-critic completes, read `working/design_review_{{DATASET}}.md` and extract the verdict:

1. **APPROVED** → Mark `chart-maker-fixes` as `skipped` in pipeline_state.json. Proceed to storytelling tier.

2. **APPROVED WITH FIXES** → Extract the fix report section from the design review. Set `chart-maker-fixes` to `ready`. Pass the fix report as `FIX_REPORT` input. The chart-maker-fixes agent (same file as chart-maker, with `FIX_REPORT` provided) re-generates only the charts listed in the fix report. After completion, re-run visual-design-critic as a quick re-check. If still `APPROVED WITH FIXES` after the re-check, proceed anyway (one fix loop iteration max).

3. **NEEDS REVISION** → HALT the pipeline with message: `"Design critic returned NEEDS REVISION. Manual intervention required. Review: working/design_review_{{DATASET}}.md"`. Do NOT proceed to storytelling.

### Checkpoint 4 — Final Deck (after deck-creator and visual-design-critic slide-level complete)

**Type:** A (automated). **Plans:** full_presentation, refresh_deck.

Verify: R1 (theme), R2 (titles), R3 (backgrounds), R4 (recommendation order), R5 (banned words), R6 (breathing slides), R7 (chart figsize), R10 (HTML components), R11 (export), deck size 8-22 slides, speaker notes present, receipt generated (if Tier 3).

**Receipt check (Tier 3 only):**
If `validation_tier == "tier_3"`, verify that `outputs/analysis_receipt_*.md` exists and is non-empty. If missing, trigger the receipt-generator agent at step 18.5 before completing the checkpoint. If the receipt-generator fails, WARN but do not FAIL the checkpoint.

**Marp Lint Gate (R10):**
Run `helpers/export/marp_linter.py` against the deck output. Print the lint report.

```python
from helpers.export.marp_linter import lint_deck, format_report

result = lint_deck("outputs/deck_{{DATASET_NAME}}_{{DATE}}.marp.md")
print(format_report(result))

if not result["summary"]["pass"]:
    # FAIL checkpoint — report errors
    print(f"CHECKPOINT 4 FAIL: {result['summary']['errors']} lint errors")
    for issue in result["issues"]:
        if issue["severity"] == "ERROR":
            print(f"  - {issue['code']}: {issue['message']}")
```

Lint errors that FAIL Checkpoint 4:
- `FM-*`: Missing or wrong frontmatter keys
- `COMP-MIN`: Fewer than 3 HTML component types
- `CLASS-INVALID`: Invalid slide class (e.g., `breathing`)
- `R2-COLLISION`: Chart title identical to slide headline

Lint warnings that are reported but do NOT fail the checkpoint:
- `COMP-PLAIN`: Plain-markdown content slides
- `SLIDES-LOW` / `SLIDES-HIGH`: Slide count outside 8-22
- `R6-PACING`: Consecutive content slides without pacing break
- `IMG-BARE-MD`: Bare markdown image (`![](...)`) not wrapped in `.chart-container`

---

## Chart Fan-Out Protocol

When chart-maker becomes READY (after narrative-coherence-reviewer):

1. **Parse storyboard:** Read `working/storyboard_{{DATASET}}.md`. For each beat, traverse the `slides` array and collect slides with `type: chart-full`, `chart-left`, or `chart-right`. Each chart-type slide references its parent beat's chart spec.
2. **Build chart_specs list:** `[{beat_number, slide_index, headline, chart_spec, output_name}, ...]`
3. **Sequential execution:** Invoke Chart Maker once per chart spec, one at a time (no parallelism). For each invocation:
   - Pass the specific `chart_spec`, `output_name`, and shared pipeline context
   - Charts are generated at standard (10, 6) figsize (R7)
   - Track: `chart_results[beat] = {status, files, error}`
   - On failure: log error, mark chart as `failed`, continue to next chart
4. **Batch review:** After ALL charts are generated, invoke Visual Design Critic once with the full set of chart files for batch review. Pass all `chart_results` output paths.
5. **Verify:** Check all output files exist (base PNG + SVG per chart). Report missing/failed charts at Checkpoint 3 for retry.

---

## AGENT FAILURE HANDLING

If an agent errors or returns without its declared outputs, retry it once with the same context. On a second failure, mark it `failed` and apply the degradation policy: non-critical agents (visual-design-critic, narrative-coherence-reviewer, opportunity-sizer) degrade with a warning; critical agents (cross-verification, validation, data-explorer) HALT.

---

## CIRCUIT BREAKER

Prevents runaway failures from consuming resources:

- Track failure count per execution tier
- **Threshold: 3 failures in a single tier** → HALT the pipeline
- On HALT, report:
  ```
  Circuit breaker tripped in tier {N}.
  Failed agents: {list with error messages}
  Completed agents: {list}
  Suggestion: Fix the underlying issue and /resume-pipeline
  ```
- The circuit breaker does NOT fire for skipped agents, only for failed agents

---

## EXECUTION METRICS

Timing is derived from the `started_at` / `completed_at` stamps in `pipeline_state.json`; nothing is computed by hand. After each tier completes, and again at pipeline end:

```python
from helpers.pipeline.dag import write_metrics
metrics = write_metrics(state, RUN_DIR / "pipeline_metrics.json")   # tiers taken from state["tiers"]
```

The file carries per-agent durations, per-tier wall-clock vs. sequential duration, `parallel_efficiency` (sequential / wall-clock; 2.0 means a 2x speedup from parallelism), and a summary (completed / degraded / failed / skipped, `avg_parallel_efficiency`). Report `total_duration_seconds` and `avg_parallel_efficiency` at Pipeline Complete.

---

## Progress Reporting

At the start and end of each tier (mapped to phases), emit progress:

**Phase mapping** (tiers to phases for user-facing messages):

| Phase | Agents | Name |
|-------|--------|------|
| 1 | question-framing, hypothesis | Framing |
| 2 | data-explorer, descriptive-analytics, root-cause-investigator, cross-verification, validation, opportunity-sizer | Exploration & Analysis |
| 3 | story-architect, narrative-coherence-reviewer, chart-maker, visual-design-critic | Storytelling & Charts |
| 4 | storytelling, deck-creator, visual-design-critic-slides, close-the-loop | Deck & Delivery |

**Start format:** `[Phase N/4: {Name}] Starting... ({agent_count} agents)`
**End format:** `[Phase N/4: {Name}] Complete. ({summary}) | Overall: {completed}/{total} agents done`

---

## COMMON FAILURE MODES

| Failure | Root Cause | Prevention Rule | When Caught |
|---------|-----------|----------------|-------------|
| Dark theme on standard analysis | Deck Creator defaulted to dark | R1 | Checkpoint 4 |
| Chart title = slide headline | Story Architect wrote same text | R2 | Checkpoint 3, 4 |
| Chart on pure white background | `swd_style()` not called | R3 | Checkpoint 3 |
| Recommendations in random order | Listed by topic not confidence | R4 | Checkpoint 4 |
| Sensational language | Dramatic words in headlines | R5 | Checkpoint 3, 4 |
| Wall of charts, no pacing | No breathing slides | R6 | Checkpoint 4 |
| Tiny chart text on slides | Chart rendered at small figsize | R7 | Checkpoint 3 |
| Agent guidance not followed | Didn't read agent file from disk | R8 | All checkpoints |
| Analysis on corrupted data | Data loading error | R9 | Checkpoint 2 |
| Cycle in registry | New agent added with circular dep | Cycle detection | Pre-flight |
| Deadlock in DAG | Tier has no READY agents | Deadlock detection | Phase 2 loop |
| Runaway failures | Multiple agents failing | Circuit breaker | Phase 2 loop |
| No HTML components | Deck uses only plain markdown | R10 | Checkpoint 4 (lint) |
| Missing html:true | Components render as raw HTML text | R10 | Checkpoint 4 (lint) |
| Missing size:16:9 | Slides render at 4:3 with broken layouts | R10 | Checkpoint 4 (lint) |
| Export fails | Marp CLI not installed or crashes | R11 | Post-Checkpoint 4 |
| Stale pipeline state | Previous run crashed mid-execution | Step 0 cleanup | Pre-flight |
| Chart text overlap | Labels collide at rendered size | R7 | Checkpoint 3 + chart-maker HALT |
| Chart overflows slide | Bare `![](...)` image not in `.chart-container` | R10 | Checkpoint 4 (lint: IMG-BARE-MD) |

---

## Post-Checkpoint 4: Deck Export (R11)

After Checkpoint 4 passes, export the deck to PDF and HTML:

```python
from helpers.export.marp_export import export_both, check_ready

deck_path = "outputs/deck_{{DATASET_NAME}}_{{DATE}}.marp.md"
theme = pipeline_args.get("theme", "analytics")

# Check if Marp CLI is available
status = check_ready()
if not status["marp_cli"]:
    print("WARNING: Marp CLI not available. Skipping PDF/HTML export.")
    print("  Install: npm install -g @marp-team/marp-cli")
    # Record skip in pipeline_state.json
    pipeline_state["export"] = {"status": "skipped", "reason": "marp_cli_unavailable"}
else:
    try:
        exports = export_both(deck_path, theme)
        print(f"PDF:  {exports['pdf']}")
        print(f"HTML: {exports['html']}")
        # Record in pipeline_state.json
        pipeline_state["export"] = {
            "status": "completed",
            "pdf": str(exports["pdf"]),
            "html": str(exports["html"]),
        }
    except Exception as e:
        print(f"WARNING: Export failed: {e}")
        pipeline_state["export"] = {"status": "failed", "error": str(e)}
```

Export is non-blocking — failures are logged as warnings, not pipeline halts. The Marp
markdown deck is always the primary deliverable; PDF/HTML are convenience outputs.

---

## Post-Pipeline: Finalize Run Directory

After export and before metric capture, consolidate the run directory:

1. **Copy artifacts** from `working/` and `outputs/` into `{RUN_DIR}/working/` and `{RUN_DIR}/outputs/`
2. **Update pipeline_state.json** in `{RUN_DIR}/`: set `status: completed`, record `completed_at`
3. **Verify symlink:** Confirm `working/latest` points to this run directory

The run directory is now a self-contained snapshot of the entire analysis.

---

## Post-Pipeline: Metric Capture & Archive

After all checkpoints pass, before reporting completion:

**Metric capture hook:**
1. Scan analysis report for metric references
2. Check `.knowledge/datasets/{active}/metrics/index.yaml` for each metric
3. Note new metrics: "New metric detected: {name}. Use `/metrics` to define it."
4. Update `last_used` on existing entries

**Archive hook:**
1. Apply archive-analysis skill (`.claude/skills/archive-analysis/SKILL.md`)
2. Capture: title, question, level, key findings, metrics used, agents invoked, output files
3. Write to `.knowledge/analyses/index.yaml`

## Pipeline Complete

When all checkpoints pass, report:
1. Output files (deck, charts, narrative paths, PDF/HTML export paths)
2. Checkpoint results summary (including Marp lint report)
3. Execution metrics summary (duration, parallel efficiency)
4. Metrics status (new/updated)
5. Archive confirmation (analysis ID)
6. Export status (PDF/HTML generated, or skipped with reason)
6. Any manual follow-ups needed