i4h-workflow-dataset-convert · diff
v0.6.0 to v0.8.0
61 added, 71 removed. Audit A to A.
---
name: i4h-workflow-dataset-convert
- version: "0.6.0"
- description: Convert an agentic HDF5 recording into a LeRobot dataset (parquet, meta, videos). Use when asked to convert HDF5, prepare for training, or export to LeRobot; not for viewing — use [[i4h-lerobot-viz]].
+ description: Convert workflow HDF5 recordings to LeRobot datasets for training or browser inspection. Use for conversion; do not use for replay, augmentation, or raw-data repair.
license: Apache-2.0
metadata:
author: "Isaac for Healthcare Team <isaac-for-healthcare-support@nvidia.com>"
+ version: "0.8.0"
tags:
- isaac-for-healthcare
- i4h
- dataset
+ - hdf5
- lerobot
- - conversion
---
- # i4h Workflow — Convert Dataset
+ # Convert Workflow HDF5 to LeRobot
## Purpose
- Convert an agentic HDF5 recording into a LeRobot dataset (parquet + meta + videos). Use when the user asks to convert HDF5, prepare for training, or export to LeRobot.
+ Preserve recorded actions, state, cameras, task text, and embodiment labels in a local LeRobot dataset.
- ## Base Code
+ ## Instructions
- These steps drive the i4h-workflows base code (the `workflows/agentic/` tree). To reuse an existing checkout, set `I4H_WORKFLOWS` to its path (no clone happens). Otherwise this resolves the current repo, or clones to `~/i4h-workflows` — pick that default without prompting. Run every command below from the resolved root:
+ 1. Run the checkout resolver and select the source HDF5.
+ 2. Read the workflow, Scene, embodiment, and instruction.
+ 3. Run conversion for the selected successful episodes.
+ 4. Inspect metadata, parquet, videos, and feature widths.
+ ## Resolve and inspect
+
```bash
- # Resolve the i4h-workflows base code (provides workflows/agentic/).
+ export I4H_WORKFLOWS_REPO_URL="${I4H_WORKFLOWS_REPO_URL:-https://github.com/isaac-for-healthcare/i4h-workflows}"
+ I4H_REPO_DIR_NAME="${I4H_WORKFLOWS_REPO_URL%/}"
+ I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*/}"
+ I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME##*:}"
+ I4H_REPO_DIR_NAME="${I4H_REPO_DIR_NAME%.git}"
+ [ -n "$I4H_REPO_DIR_NAME" ] || { echo "Cannot derive a checkout name from I4H_WORKFLOWS_REPO_URL" >&2; exit 2; }
ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"
- if [ ! -d "$ROOT/workflows/agentic" ]; then
- ROOT="${I4H_WORKFLOWS:-$HOME/i4h-workflows}"
- [ -d "$ROOT/workflows/agentic" ] || git clone https://github.com/isaac-for-healthcare/i4h-workflows "$ROOT"
+ if [ ! -d "$ROOT/workflows/i4h_workflows" ]; then
+ ROOT="${I4H_WORKFLOWS:-$HOME/$I4H_REPO_DIR_NAME}"
+ [ -d "$ROOT/workflows/i4h_workflows" ] || git clone "$I4H_WORKFLOWS_REPO_URL" "$ROOT"
fi
- export I4H_WORKFLOWS="$ROOT"; cd "$ROOT"
+ export I4H_WORKFLOWS="$ROOT"
+ cd "$ROOT"
+ HDF5_PATH=/absolute/path/to/recording.hdf5
+ uv run --project tools/dataset i4h-dataset inspect "$HDF5_PATH" --segments
```
- ## Basics
-
- - Use the same `--env` that produced the HDF5.
- - **Env config (source of truth):** `workflows/agentic/config/environments/<env>.yaml` supplies the robot, task, cameras, and `dataset.*` (action/state names, splits, modality) converter defaults.
- - Output goes to `HF_LEROBOT_HOME/<repo-id>`.
-
- ## Run
+ Treat the resolver above as part of the skill contract: a hosted copy may run outside the base repository, so never assume the current checkout contains `workflows/i4h_workflows`. `I4H_WORKFLOWS_REPO_URL` selects the clone source. When `I4H_WORKFLOWS` is unset, derive the fallback directory from that URL; set `I4H_WORKFLOWS` only to reuse or choose a specific destination. Never replace an existing checkout.
- Run the steps below in order. Each step is a separate bash call; variables persist in the local agent's tmux session.
+ Use the explicit/current-chain recording. Resolve its workflow and Scene from recording metadata/context, then read the Scene manifest for the embodiment and instruction. Use the embodiment manifest for labels. Do not assume state width equals action width; the converter derives both from the recording.
- ### Step 1 — setup and resolve HDF5
+ ## Convert
```bash
- REPO_ROOT="${I4H_WORKFLOWS:-$(git rev-parse --show-toplevel 2>/dev/null)}"; [ -d "$REPO_ROOT/workflows/agentic" ] || REPO_ROOT="$HOME/i4h-workflows"
- ENV_ID=scissor_pick_and_place
- RUNS_ROOT="${REPO_ROOT}/workflows/agentic/runs"
+ RUN_DIR="$(pwd)/runs/<workflow>/$(date +%Y%m%d_%H%M%S)"
+ DATASET_DIR="$RUN_DIR/lerobot/local/<name>"
+ mkdir -p "$(dirname "$DATASET_DIR")"
+ [ ! -e "$DATASET_DIR" ] || { echo "Destination already exists: $DATASET_DIR" >&2; exit 2; }
+ uv run --project tools/dataset i4h-dataset convert \
+ "$HDF5_PATH" "$DATASET_DIR" \
+ --robot <embodiment> \
+ --repo-id "local/<name>" \
+ --successful-only \
+ --task "<instruction>"
+ ```
- # Point HDF5_PATH at a real recording (absolute path). Recordings come from teleop, mimic, or
- # validate (which writes data/verify.hdf5 under each runs/eval_* dir). List candidates newest-first:
- # find "${RUNS_ROOT}" -name '*.hdf5' -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort -r | head
- HDF5_PATH="${HDF5_PATH:-}"
- if [ ! -f "${HDF5_PATH}" ]; then
- echo "convert: set HDF5_PATH to an existing .hdf5 (got '${HDF5_PATH:-<unset>}'). Candidates:" >&2
- find "${RUNS_ROOT}" -name '*.hdf5' -printf '%TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null | sort -r | head
- exit 1
- fi
+ Use `--fps` or `--skip-frames` only when the user requests it or source metadata justifies it. Keep the default H.264 video codec for compatibility with GR00T's fast decord loader; select another `--video-codec` only when the target consumer requires it.
- RUN_DIR="${RUNS_ROOT}/convert_${ENV_ID}_$(date +%Y%m%d_%H%M%S)"
- mkdir -p "${RUN_DIR}/logs"
- ln -sfn "${RUN_DIR}" "${RUNS_ROOT}/.latest"
- export HF_LEROBOT_HOME="${RUN_DIR}/lerobot"
- ```
+ Conversion writes aggregate `meta/stats.json` for downstream policy loaders. Native G1 rule-based WBC recordings already contain 43-D state and 50-D action; the converter recognizes that contract and writes GR00T's required semantic `meta/modality.json` automatically. For a G1 recording made through the legacy 23-D Pink/keyboard contract and destined for a 50-D G1 WBC policy Task, add `--g1-wbc-policy-actions`. That explicit mapping combines the measured 43-joint state with the recorded navigation, base-height, and torso commands; require source action width 23 and state width 43.
- ### Step 2 — convert
+ ## Verify
- ```bash
- "${REPO_ROOT}/workflows/agentic/dataset/run.sh" \
- --env "${ENV_ID}" \
- --hdf5-path "${HDF5_PATH}" \
- --repo-id "local/${ENV_ID}" \
- --video-codec h264 \
- --overwrite \
- 2>&1 | tee "${RUN_DIR}/logs/convert.log"
- ```
+ Require:
- ## Notes
+ - `meta/info.json`
+ - `meta/stats.json`
+ - `meta/modality.json` when the target trainer requires semantic modality slices
+ - episode parquet data
+ - video files for every recorded camera
+ - converted episode count matching selected successful sources
+ - action/state feature widths and names matching the recording plus embodiment descriptor
- - `--video-codec h264` is required. The converter's default AV1 codec breaks GR00T's `decord` video reader at finetune time.
- - Scissor SO-ARM generates `meta/modality.json` from YAML splits and does not need `dataset.modality_template_path`.
- - G1 locomanip and assemble-trocar use `dataset.modality_template_path` from the env YAML.
- - All camera streams are resized to the env YAML `policy.image_size` (override with `--image-size H W`), normalizing mixed-resolution cameras (e.g. head cam + overview cam) to the one size the modality config expects.
+ For G1, require modality metadata for both supported paths: native `state=43/action=50`, or explicitly mapped `state=43/source-action=23/output-action=50`. Treat a native 50-D dataset without `meta/modality.json` as incomplete.
- ## Verify
+ Treat missing inputs or zero converted episodes as failure. If conversion leaves a partial destination, quarantine or remove that exact incomplete directory before retrying; never report it as usable.
- - `${HF_LEROBOT_HOME}/local/${ENV_ID}/meta/info.json` exists.
- - Log reports the saved episode count.
- - Per-episode video files are present under `${HF_LEROBOT_HOME}/local/${ENV_ID}/`.
+ ## Troubleshooting
+ On dimension errors, resolve the source workflow and embodiment again. On missing videos, confirm frames existed before conversion.
+
## Prerequisites
- - Workflow set up via [[i4h-workflow-setup]] (the `.venv` must exist).
- - An existing HDF5 recording to convert (set `HDF5_PATH` to an absolute path; the Run block lists candidates if it's unset or wrong).
- - The same `--env` that produced the HDF5 (its YAML supplies robot, task, camera, modality, and converter defaults).
- - `HF_LEROBOT_HOME` set to the output location for `<repo-id>`.
+ Require a readable HDF5 recording and its matching Scene plus embodiment manifests.
## Limitations
- - `--video-codec h264` is required; the converter's default AV1 codec breaks GR00T's `decord` reader at finetune time.
- - All camera streams are resized to the env YAML `policy.image_size` (override with `--image-size H W`).
- - G1 locomanip and assemble-trocar require `dataset.modality_template_path` from the env YAML; scissor SO-ARM generates `meta/modality.json` from YAML splits.
+ Conversion cannot reconstruct missing cameras, actions, state, task text, or successful episodes.
- ## Troubleshooting
+ ## Examples
- - **Error:** `.venv` not found / module import fails - Cause: workflow not set up. Fix: run [[i4h-workflow-setup]] first.
- - **Error:** input HDF5 not found - Cause: wrong or missing `--hdf5-path`. Fix: point `HDF5_PATH` at an existing recording.
- - **Error:** `decord` fails to read video at finetune time - Cause: dataset written with the default AV1 codec. Fix: re-convert with `--video-codec h264`.
- - **Error:** missing/incorrect modality config - Cause: wrong `--env`, so robot/task/camera/modality defaults do not match the HDF5. Fix: use the same `--env` that produced the recording.
+ - `Convert my scissor pick-and-place recording into a LeRobot dataset.` → resolve `so101`, convert successful episodes, and verify metadata, parquet, and both camera videos.
- ## Final Response
+ ## Completion gate
- Report source HDF5, dataset path, repo id, episode count, skipped or failed episodes.
+ Report source HDF5/workflow, embodiment, task text, source/converted/skipped counts, action/state widths, output directory/repo id, aggregate-stats/modality/parquet/video checks, and any missing modality.