--- name: insights description: Use when turning the reports/manifests other workbench tools emit into a queryable lineage graph + common metrics store, or when querying/comparing metrics, tracing artifact lineage, or building a run dashboard. --- # Insights (Lineage & Metrics backbone) Insights is the connective tissue that makes workflow runs legible. It turns the structured reports/manifests other tools already emit into a queryable **lineage graph + common metrics store** — the foundation for dashboards and an orchestrating agent. It does not replace any tool; it aggregates them. ## Three-access pattern Source of truth is the FastAPI service (`npa/src/npa/workbench/insights/service.py`). The CLI (`npa/src/npa/cli/workbench/insights.py`) and SDK (`npa/src/npa/sdk/workbench/insights.py`) are thin clients. Do not duplicate logic across layers. ## Store layout The store is an **append-only index on S3** under a configurable prefix (`--output-path`), with a JSONL fallback so it works without any database: - `records.jsonl` — metric records (`npa.insights.metric_record.v1`). - `edges.jsonl` — lineage edges (`npa.insights.lineage_edge.v1`). - `records.d/` and `edges.d/` — immutable append shards, one object per write. Every append writes a **new shard object**; readers concatenate the base object (legacy stores) plus all shards. Never rewrite a whole JSONL object to append: object storage has no native append, so read-modify-write silently drops rows when two writers overlap (both read N, both write N + their own). Readers expose a logically idempotent view: metric rows are deduplicated by run, source artifact URI, metric, stage/tool, canonical labels (including curve step), and lineage; lineage edges use their endpoint/version/relation/run identity. This also repairs legacy stores that already contain duplicate shards. Explicit emissions without an artifact URI retain their timestamp/value identity so distinct observations with the same metric name are not collapsed. Do NOT introduce a database service or hardcode a metadata backend. Reuse the **LanceDB** tool as the optional query index (HTTP seam in `integrations.py`), exactly as `dataset` does; absence degrades to the JSONL scan. ## Interfaces CLI: ```bash npa workbench insights record --input-path --output-path npa workbench insights ingest-run --input-path --output-path npa workbench insights query --input-path --tool --metric-name npa workbench insights lineage --input-path --uri npa workbench insights compare --input-path --base-run --candidate-run npa workbench insights dashboard --input-path --output-path npa workbench insights status --input-path npa workbench insights system-info npa workbench insights list ``` Endpoints: `/health`, `/status`, `/system-info`, `/list`, `POST /record`, `POST /ingest-run`, `GET /lineage`, `GET /query`, `GET /compare`, `GET /dashboard`. ## API contract - `POST /record`: append one or more metric emissions (+ lineage edges) keyed by run id + lineage refs. Rows validate against `npa.insights.metric_record.v1`. - `POST /ingest-run`: **non-invasive ingestion** — scan an S3 run prefix for known schemas (`npa.dataset.manifest.v1`, `npa.dataset.validation_report.v1`, `npa.scenario_gen.adversarial_set.v1`, and gate/threshold decision JSONs), extract their metrics + provenance, and write them into the store. This does NOT require modifying the emitting tools. - `GET /lineage`: traverse the provenance graph (ancestors + descendants) for an artifact/version, reconstructed from recorded `lineage_edge` records. - `GET /query`: query metric records by facet (workflow, run id, tool, stage, dataset/model version, metric name, metric kind, cost basis, time range, threshold predicate). Cost records always label `cost_basis` as `estimated` or `billed`; only billing artifacts are authoritative for billed dollars. - `GET /compare`: cross-run/cross-stack comparison; emits `npa.insights.comparison.v1` (per-metric delta + regressed/improved flags). Metrics whose name looks failure-like (corruption/latency/loss/…) are treated as lower-is-better; override with `--lower-is-better`. - `GET /dashboard`: return `npa.insights.dashboard.v1` (grouped metrics + latest-run rollup) and optionally write a self-contained static HTML report to `--output-path`. Keep viz thin — JSON + a single-file HTML, no web UI. Known evaluation reports use an explicit numeric taxonomy: score/quality fields become `eval_score`, step/sample/epoch values become `counter`, and latency or `*_ms` values become `duration`. Arbitrary numeric metadata is not promoted to a score. When root placeholders conflict with nested `metrics` or `success_summary`, the authoritative nested value wins deterministically. ## Lineage Thread and preserve lineage that already exists in upstream manifests (input URIs, dataset/checkpoint versions, parent versions, produced_from/derived_from/ evaluated_on relations). The whole point is cross-tool traceability — do not drop it. ## GPU routing CPU-only. Aggregation, query, comparison, lineage, and dashboard need no GPU and no rendering path (headless). ## Workflows SkyPilot stays the execution engine, but the shown catalog is npa.workflow-only (no raw SkyPilot task YAMLs). All insights pipelines are declarative `npa.workflow/v0.0.1` specs, CPU-only and `cloud: kubernetes`: - Aggregate a run (ingest-run -> dashboard): `npa/workflows/workbench/npa-workflows/insights-aggregate.yaml` - Hardening with insights (hardening stages -> ingest-run -> dashboard): `npa/workflows/workbench/npa-workflows/hardening-with-insights.yaml` - CPU-only smoke (ingest fixture -> compare -> dashboard): `npa/workflows/workbench/npa-workflows/insights-smoke.yaml` toolRefs: `workbench.insights.record`, `workbench.insights.ingest_run`, `workbench.insights.compare`, `workbench.insights.dashboard`. ## Known issues - Object storage has no native append, so each write lands in its own immutable shard under `records.d/` / `edges.d/` and reads concatenate base + shards. This is what makes concurrent ingests safe; a store is never rewritten in place. - Re-ingesting a source prefix may append nothing (`recorded_count: 0`) when all logical observations already exist. Concurrent writers may still create duplicate immutable shards, but readers deduplicate them without losing distinct source observations or skewing dashboard means. - **Reader version skew:** a reader older than sharding sees only the base object and silently reports a truncated store (e.g. an agent VM answering "no runs found" for runs that did ingest). Re-bootstrap deployed agents (`npa agent bootstrap --project --name agent`) after upgrading the store writers. - `compare` needs both run ids present in the store; comparing a run to itself reports every metric as unchanged (useful as a smoke self-check). A `compare` that fails with `no metrics recorded for base run` right after a successful ingest means the base run's rows are missing from the store, not that the run never ingested — check for rows dropped by a writer that rewrote the object. - A `gpus` metric only exists when an ingested `npa.workflow.run.v1` manifest has a step whose `resources_profile.accelerators` parses to >= 1, and manifests with status `planned` are skipped. CPU-only and never-executed runs therefore report no GPU count at all rather than a fabricated zero.