Immutable. This exact content is served forever at /api/v1/blob/6bda3db2d86d64a8.
--- name: dataset description: Use when ingesting, validating, curating, or querying production sensor data as a versioned dataset-of-record, or wiring the dataset-ingest-curate workflow. --- # Dataset (Dataset-of-Record) A unified ingestion / validation / curation layer that turns raw production sensor data into a queryable, versioned **dataset-of-record**, filterable by event, location, and quality. It composes existing primitives (FiftyOne for curation/visualization, LanceDB for the vector/metadata query index, S3 as the bus) behind one tool instead of leaving them disconnected. ## Three-access pattern Source of truth is the FastAPI service (`npa/src/npa/workbench/dataset/service.py`). The CLI (`npa/src/npa/cli/workbench/dataset.py`) and SDK (`npa/src/npa/sdk/workbench/dataset.py`) are thin clients. Do not duplicate logic across layers. ## Interfaces CLI: ```bash npa workbench dataset ingest --input-path <s3> --output-path <s3> --dataset-id <id> npa workbench dataset validate --input-path <s3-manifest> --output-path <s3> npa workbench dataset curate --input-path <s3-manifest> --output-path <s3> --event <e> --location <l> npa workbench dataset query --input-path <s3-manifest> --event <e> --location <l> npa workbench dataset status --dataset-id <id> --version <v> npa workbench dataset system-info npa workbench dataset list ``` Endpoints: `/health`, `/status`, `/system-info`, `/list`, `POST /ingest`, `POST /validate`, `POST /curate`, `GET /query`. ## API contract - `POST /ingest`: pull raw sensor data from `--input-path`, validate against the declared sensor schema, normalize to canonical records, and register a versioned manifest at `--output-path` (schema `npa.dataset.manifest.v1`: dataset id + version, record count, sensor modalities, source lineage, per-record S3 pointers, quality stats). - `POST /validate`: schema + quality-metric validation (completeness, corruption, per-sensor sanity); emits `npa.dataset.validation_report.v1`. - `POST /curate`: filter/slice by **event of interest, location, and quality metric**; writes a derived version whose manifest records lineage back to the parent (parent dataset id/version + filter predicate). - `GET /query`: query records by event/location/quality facets. Backed by the LanceDB index when `--lancedb-endpoint` is set; falls back to the manifest so the tool works without a running LanceDB. Reuse the FiftyOne tool for curation/visualization handoff and the LanceDB tool for the query index (see `integrations.py`) rather than re-implementing either — these are HTTP seams mocked in tests. ## Lineage Every manifest threads provenance (workflow run, input URIs, dataset version, parent dataset id/version, filter predicate) so a later lineage/metadata service can consume it. Do not hardcode a metadata backend; keep lineage in the S3 manifests. ## GPU routing Ingest / validate / curate / query are CPU-only. The optional embedding backfill that populates the LanceDB query index runs on H100 (general training class). ## SkyPilot + workflow - Declarative pipeline (ingest -> validate quality gate -> curate -> register queryable version): `npa/workflows/workbench/npa-workflows/dataset-ingest-curate.yaml`. The raw SkyPilot twin is retired. - The `register` stage needs the LanceDB service reachable from a pod: `npa workbench lancedb deploy --runtime kubernetes --namespace workbench --storage-path s3://<bucket>/lancedb/` toolRefs: `workbench.dataset.ingest`, `workbench.dataset.validate`, `workbench.dataset.curate`, `workbench.dataset.query`, `workbench.dataset.write_quality_decision`, `workbench.dataset.report_rejection`. ## Known issues - The quality gate rejects the version when mean completeness is below `config.completeness_min` or the corruption rate exceeds `config.max_corruption_rate`. - Curated child versions are content-addressed (`<parent>.curated-<hash>`); a workflow that queries a curated version wires the concrete manifest URI at runtime. - The service defaults to token authentication and an empty request storage scope. Set `DATASET_TOKEN`, plus the narrow `DATASET_ALLOWED_S3_ROOTS` and/or `DATASET_ALLOWED_LOCAL_ROOTS` boundary. These allowlists apply to deployed FastAPI requests, not default embedded CLI, SDK, or workflow toolRef execution. `DATASET_AUTH_MODE=none` is an explicit local/test service opt-in only; see `docs/security/workbench-service-boundaries.md`.