--- name: neural-reconstruction description: Use when reconstructing real sensor captures into renderable 3D scenes with NVIDIA Omniverse NuRec / the Neural Reconstruction Engine (NRE) on Nebius — COLMAP source conversion to NCore V4, existing NCore V4 input, 3DGUT Gaussian training, renderable USDZ, novel-view rendering, and the Rerun recording the NPA agent displays. Also use when an NCore sequence will not load in NRE, when picking the GPU for a reconstruction, or when changing the nurec workbench tool, CLI, or SkyPilot workflow. --- # Neural Reconstruction (NuRec / NRE) ## Source And Attribution Adapted from the NVIDIA Omniverse NuRec agent skills at (`skills/nre`, `skills/physical-ai-datasets`, `skills/ncore`) and the NVIDIA NCore data library at . The capability routing table, the easy mix-ups, the safe secret-verification pattern, and several troubleshooting rows below are adapted from the NVIDIA router skill (Apache-2.0), pinned at commit `0122ea0` (2026-08-01). That skill is a *router*: it never runs anything, it decides which upstream sibling skill answers a question. This skill is the opposite — it is the workbench implementation — so the router's picker table is re-pointed at real `npa workbench nurec` verbs, and each row upstream owns is marked as such rather than reproduced. Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. Upstream licenses are Apache-2.0 and CC-BY-4.0. Trademarks (NVIDIA, Omniverse, NuRec, NRE, Isaac Sim, Cosmos) belong to NVIDIA. See `skills/NOTICE-NVIDIA-SKILLS`. The NCore conversion image packages pinned Apache-2.0 NVIDIA source and its MIT COLMAP reader; it contains no dataset, weights or proprietary NRE payload. NRE remains a separately licensed NGC runtime. The new COLMAP ingestion path is **not yet live validated** and has no accepted public image release. See `skills/NOTICE-NVIDIA-NCORE-COLMAP` and `docs/workbench/guides/nurec-colmap-reconstruct.md`. Existing preconverted-NCore results below do not validate the new conversion path. ## When To Use Load this skill when the user wants to: - turn a real sensor capture (photographs, multi-camera clips) into a renderable 3D Gaussian scene and render **novel views** from it; - run, modify, or debug `npa workbench nurec` or `npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml`; - work out why an NCore sequence fails to load in NRE; - choose the GPU for a reconstruction or rendering job; - make a reconstruction run show up in the NPA agent's Rerun panel. **Do NOT use this skill for:** Cosmos video augmentation (use `skills/workflows/physical-ai-data-factory/SKILL.md`), per-object asset extraction from sparse views (upstream `asset-harvester`, not implemented here), or generic Rerun visualization of a Sim2Real run (`skills/workbench/sim2real-engine/SKILL.md`). ## GPU Routing (hard constraint) Gaussian reconstruction and rasterization are **RT-core** work. Route these jobs at: - **RTX PRO 6000 Blackwell** — `RTXPRO-6000-BLACKWELL-SERVER-EDITION:1` on the RT-core Kubernetes context (what the shipped workflow uses), or - **L40S** — `L40S:1`. Never route the reconstruct/render path at **H100 / H200 / A100 / B200**: they have no RT cores. `npa workbench nurec check` fails with an explicit error when the visible GPU is one of those (`has_rt_cores`). NRE additionally requires driver **R580+** on Blackwell and >= 24 GB VRAM (48 GB+ recommended). See `skills/atomic/gpu-selection/SKILL.md`. ## Container Access (read this before debugging a pull failure) NRE ships only as a closed-source NGC container, so the container *is* the runtime — there is no source/pip path. | Repository | With a standard `NGC_API_KEY` | | --- | --- | | `nvcr.io/nvidia/nre/nre` | **402 Payment Required** — needs an extra entitlement | | `nvcr.io/nvidia/nre/nre-ga` | **pullable** — the General Availability channel | | `nvcr.io/nvidia/nre/nre-tools-ga` | pullable (auxiliary seg/depth data only) | Use the **`-ga`** repositories. `npa workbench nurec check` reports `ngc_image: entitlement-required` rather than failing opaquely when a non-GA reference is configured. Image startup preflight overrides the vendor entrypoint with `/bin/bash`, as SkyPilot does, and installs missing SSH, rsync, and service helpers before checking worker capabilities. NRE's `/app/run` entrypoint cannot forward an arbitrary shell probe. Prepared results are never cached as image attestations; they depend on package access from the selected cluster. First-party image attestations and strict runtime byte probes keep their existing requirements. The live regression is `test_vendor_image_runtime_bootstrap_live` in `npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py`. Select the exact preflighted image/digest, `NPA_E2E_KUBECONTEXT`, `KUBECONFIG`, and any existing pull Secret names through `NPA_E2E_IMAGE_PULL_SECRETS` (comma separated). It requires compatible worker capabilities and verified deletion of its probe pod. ```bash NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_image_bootstrap_terminal_probe_live.py \ -k vendor_image_runtime_bootstrap -q ``` ## Real Entrypoints Every stage is a real command; nothing here is a manifest stub. ```bash npa workbench nurec check # NGC pullability + HF download rights + RT-core GPU npa workbench nurec probe-storage # S3 conditional write/read/list/delete preflight npa workbench nurec stage-source # conditional source staging + exact read-back npa workbench nurec control-source # pre-extraction wrong-hash / zero-output control npa workbench nurec observe-runtime # exact-job/stage-bound pod image/GPU observation npa workbench nurec bundle-runtime # bind reconstruct + render pod receipts npa workbench nurec convert-colmap # CPU: official Apache-2.0 COLMAP -> NCore V4 npa workbench nurec audit-colmap # CPU: independent post-S3 object/V4 read-back npa workbench nurec audit-qualification # bind native receipts, media, USDZ, metrics, RRD npa workbench nurec cleanup-qualification # cancel, terminality, down, local/orphan audit npa workbench nurec fetch # real NCore V4 shards + derived rig pose edge npa workbench nurec reconstruct # NRE 3DGUT training -> renderable USDZ + metrics npa workbench nurec render # `nre render` novel views (rig offset, not training views) npa workbench nurec visualize # reports/sim2real.rrd for the agent's Rerun panel npa workbench nurec finalize # reports/final.json aggregate npa workbench nurec status # what a run prefix holds, stage by stage ``` | Concern | Implementation | | --- | --- | | Pure logic + argv builders | `npa/src/npa/workbench/nurec/nurec.py` | | COLMAP ingestion + complete V4 verification | `npa/src/npa/workbench/nurec/colmap.py` | | NCore rig-pose derivation | `npa/src/npa/workbench/nurec/ncore_rig.py` | | CLI | `npa/src/npa/cli/nurec/__init__.py` | | SDK | `npa.sdk.workbench.nurec` (`check`, `convert_colmap`, `fetch`, `reconstruct`, `render`, `visualize`, `finalize`, `status`); the framework-free API is re-exported from `npa.workbench.nurec` | | SkyPilot workflow | `npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml` | | Main declarative workflow | `workflows/main/nurec-reconstruct.yaml` | | COLMAP source workflow (not yet live validated) | `workflows/testing/nurec-colmap-reconstruct.yaml` | | Pre-publication downstream-only route | `workflows/testing/nurec-reconstruct-render.yaml` | | Rerun recording | `npa.workflows.data_factory_viz.build_run_rrd` | ## Input Data Default: **`nvidia/PhysicalAI-NuRec-PPISP`** — ungated, CC-BY-4.0, real photographic captures of four outdoor object-centric scenes shipped **already in NCore V4**, which is what NRE consumes. The main demo, single-pod example and standalone fetch select scene `struktur28`, variant `auto`: all 59 images across two cameras. The `standard` variant has 518 HDR exposure-bracketed photographs across three cameras. It is an opt-in photometric calibration input, not simply a denser version of the default. Both variants share the ~1.1 GB archive. See [NVIDIA's dataset description](https://arxiv.org/html/2601.18336v1). The `nvidia/PhysicalAI-Autonomous-Vehicles*` family (raw clips, `-NCore`, and the pre-built `-NuRec` USDZ scenes) is **gated**: the account that owns `HF_TOKEN` must accept the NVIDIA AV Dataset License on each dataset page before any token can pull a byte. `npa workbench nurec check` reports `hf_dataset: gated` for those until it is accepted — it probes real download authorization, not just metadata visibility, because a gated repo still answers 200 for `/api/datasets/`. ### COLMAP source ingestion (not yet live validated) Use `workflows/testing/nurec-colmap-reconstruct.yaml` for original COLMAP images, camera calibration/poses and sparse points. Its states are conversion -> audit -> existing NRE reconstruction -> render -> visualize -> finalize; the workflow owns the graph, with no separate Python orchestrator. The exact CLI/toolRef contract is documented in `docs/workbench/guides/nurec-colmap-reconstruct.md`. - Select the full `struktur28` directory in the PPISP ZIP pinned at `2521064a3af6ab1c1caa2ba1b01ddde7eecded69`: 518 images, 3 cameras and 163,453 sparse source points. The separate `struktur28_auto` directory is not the selected capture. Never replace the full source with that smaller variant. - `--input-path` takes an S3 ZIP or prefix; `--output-path` is the exact portable sequence prefix. Feed the same trailing-slash prefix to NRE's `--ncore-uri`. Use `--dataset-root struktur28 --rig-mode derive` and `npa_rig` poses. - Choose a fresh destination prefix for every publication attempt. The permanent provider-conditional claim prevents overlapping writers and replacement; after interruption, retry into a new prefix rather than deleting or expiring a claim. - The CPU image runs NVIDIA/ncore `59c698d206da92b406a4f72619fce3b3a2c64bfd`, with its pinned MIT trueprice/pycolmap reader. Scope the immutable development image override to `workbench.nurec.convert_colmap`; no accepted/public availability is implied. - Use the same immutable development image for both `workbench.nurec.convert_colmap` and `workbench.nurec.audit_colmap`. The audit re-lists the fresh prefix, downloads every object plus the original ZIP, rejects changed or extra members, and independently reopens every V4 image, calibration, pose and point before NRE starts. - For qualification before registry publication, run conversion locally by the checked OCI archive's immutable loaded ID with `--pull=never`, then submit `nurec-reconstruct-render.yaml`. Observe and bind the reconstruct and render pod image IDs separately; a requested digest is never observed identity. - Preserve **1 FPS** ordering within each camera, assigning disjoint virtual intervals to independent cameras and the same interval to their downsample aliases. Bind original image hashes and frame indices to the new timestamps. These are photographic indices, not synchronized capture time or a physical rig trajectory. Sparse SfM points are not physical LiDAR. Record upstream's near-origin point filtering and any derived rig changes. - Reopen every image/calibration/pose/point, compare full source counts and finite geometry, and retain hashes, source notices and CC-BY-4.0 attribution with runtime artifacts. No datasets or weights may be baked. - Compare the complete camera model and distortion coefficients, including each downsampled camera, against source calibration. Matching counts and intrinsics alone miss equal-intrinsics cameras with different distortion. Keep the drift-checked upstream compatibility patches and post-patch source inventory. - Verify converted member hashes and the publication claim before the NuRec reader opens a sequence. Return only the fresh download generation, never an overlay into an old cache: even two valid inventories can select the wrong capture after merging. Reject an explicit source without sequence metadata instead of falling back to a previously fetched dataset. - NRE remains proprietary and separately gated. Keep its full native recipe (`max_epochs: "0"`) on the explicit RTX PRO 6000 profile; never remap to B200. - Live matrix: real executing `gpu` case, with complete hash-verified source seeding and S3 conversion-member verification. Supply the development digest through `NPA_E2E_IMAGE_OVERRIDE_NCORE`. Report actual numeric GPU results before changing the pending validation status; older NuRec results do not qualify. ## Static Navigation Scene Handoff For actual calibrated metric RGB-D rather than an existing Gaussian artifact, use `workflows/testing/rgbd-scan-to-isaac.yaml` and the `docs/workbench/guides/rgbd-scan-to-isaac.md` contract. Real Open3D TSDF integration derives colored visual and exact collision surfaces from measured depth/poses; held-out depth frames gate geometry before the existing USDZ/PhysX handoff. The manifest uses column-vector optical-camera poses in metric Z-up world, hashes every pair, and requires disjoint integration/validation membership. Unknown space stays unknown. CPU reconstruction uses the existing SONIC Open3D runtime without Isaac bootstrap, followed by Content Agents OpenUSD assembly and RTX Isaac physics. It needs no new weights or image. Runtime/data provenance and private capture attribution stay with the sealed artifacts. GPU, industrial scene, robot clearance, and policy learning qualification remain separate. Use `workflows/testing/scan-to-isaac-navigation.yaml` to combine an existing NuRec visual USD/USDZ with independently supplied, triangulated collision USD. The workflow reuses reconstruction outputs; it does not duplicate NCore or COLMAP ingress. See `docs/workbench/guides/scan-to-isaac-navigation.md` for the input manifest, module adapters, calibration convention, and live test. - Require explicit source units, coordinate registration, capture-provenance digest, collision geometry, and expected physics ray intersections. Gaussian splats alone are visual assets, never collision surfaces. - Audit raw Sdf layers before composition, recursively including nested USDZ, inactive prims and unselected variants. Reject external dependencies, OmniScripting/OmniGraph declarations, time samples and value clips; preserve ordinary shaders and contained neural volume payloads. - Export portable USDZ with static triangle colliders and geometry/input hashes. Publish only to fresh prefixes using permanent conditional claims, immutable member writes, and a verified completion seal. Retry failures in a new prefix. - Run the native PhysX probe through the Isaac interpreter on RTX infrastructure. Supply an immutable Isaac image declaration; reports bind both scene and exact assembly-provenance hashes, actual Isaac/PhysX versions, and the declaration's explicitly unverified image-attestation scope. - CPU tests prove assembly and containment. Native GPU acceptance, NuRec rendering, metric RGB-D GPU execution, robot clearance, and navigation-policy quality remain unverified. The live test requires operator assets and must run before claiming physics acceptance. ## The rig -> world Pose Edge (the thing that breaks first) NRE's NCore data source requires a `("rig", "world")` pose-graph edge: ```text # nre/datasets/ncore.py, nre-ga 26.04 # TODO: frame-pose only data might fail here as there are no rig poses ... rig_world_edge = unpack_optional( sequence_loader.pose_graph.get_edge("rig", "world"), msg="Rig-to-world poses are currently required to determine scene extend") ``` Object-centric captures have no vehicle rig, and NVIDIA's own COLMAP -> NCore converter stores per-camera ` -> world` poses with **no rig node** (`tools/data_converter/colmap/converter.py`: `reference_frame = "world"`). So every COLMAP-derived NCore sequence — including PPISP's own export — fails to load with: ```text ValueError Rig-to-world poses are currently required to determine scene extend ``` Some captures carry camera-to-world transforms only in the per-frame `T_sensor_worlds` generic-data field. NPA reads that representation when no dynamic camera pose edges exist; dynamic edges remain preferred. NPA's COLMAP conversion records each real frame world pose, including downsample-parent composition. Declared virtual-time-v2 conversions assign disjoint photographic intervals to independently moving cameras, then merge their original world poses into one exact-knot virtual rig with identity calibrations. Every global-shutter frame must match its unique rig knot. The native training consumer defaults to shared-rig interpolation even when the dataset stores per-camera frame poses; the old overlapping virtual-time-v1 representation is therefore rejected, not accepted through a dataset flag. Original camera trajectories remain in the default group and are independently compared against every frame pose. Conflicting overrides, malformed frame data, missing or mismatched knots fail before reconstruction. This repair does not establish a successful NRE run; exact image, canonical conversion, effective native poses/rays and downstream RTX evidence remain required. `npa workbench nurec fetch` fixes this by default. For a single-camera capture the rig **is** the camera, so `rig -> world` is exactly that camera's pose trajectory. `ncore_rig.derive_rig_poses` writes one small extra component store plus a new sequence meta-file that symlinks the original shards, so the source data is never modified: - the derived poses live in their own component instance **`npa_rig`**, not `default` — `open_component_readers` asserts instance names are unique across a sequence's stores, and re-using `default` raises `Component instance default encountered multiple times`; - selecting a poses group **replaces** the pose set rather than merging, so the legacy derived component carries a complete copy of the original edges plus the rig edge; declared COLMAP frame-pose conversion instead uses the virtual calibration contract above; - the frame-pose fallback validates every camera frame before writing: timestamps must be strictly increasing and each `T_sensor_worlds` value must be a finite 4×4 transform. Missing, malformed, duplicate, inconsistent, or non-finite data fails closed without publishing a partial derived sequence; - the sidecar records the selected camera and pose source. An explicit reference camera wins; otherwise the longest trajectory wins with camera ID as a stable tie-breaker; - `reconstruct` then passes `dataset.poses_component_group=npa_rig`. The fallback makes exactly one geometric claim: `rig` is the selected camera. It does not infer multi-camera extrinsics, and compatibility tests do not prove reconstruction quality. Local regression coverage establishes conversion compatibility only; recording live workflow readiness still requires a decoded NRE reconstruction on applicable RT-core hardware and inspected retained artifacts. Pass `--no-derive-rig` for AV-style sequences that already ship a rig edge (the derivation short-circuits with `already_present: true` anyway), and `--reference-camera ` to pin which camera becomes the rig. For derived multi-camera photographic captures, reconstruction prepares a separate training input with one unique virtual timestamp per photograph. The virtual rig visits each photograph's original camera pose, and camera-to-rig transforms are static identities. This matters during training as well as rendering: NRE 26.04 otherwise interpolates every camera against one reference camera's rig trajectory. Original camera images remain byte-identical, intrinsics and sparse-point stores are retained, and the fetched or converted source generation is unchanged. These timestamps describe photographic ordering, not synchronized capture time. The published `reconstruction/photographic-timeline.json` records all source and training timestamps, camera poses, image hashes, and retained-store hashes. The preparation currently requires instantaneous photographic frames in separate camera stores; it rejects incompatible layouts instead of losing source data. Native physical rigs and custom recipes retain their own input representation. ## Recipe Selection The container resolves `--config-name` against its own `configs/` tree. Pick by capture shape: | Capture | Recipe | | --- | --- | | Object-centric / static / camera-only (PPISP, COLMAP-style) | `configs/experimental/3dgut/3dgut_colmap.yaml` (**the default**) | | Waymo Open Dataset | `configs/apps/AV/Waymo/3dgut_dynamic*.yaml` | | PhysicalAI Autonomous Vehicles (Hyperion-8.1) | `configs/apps/prod/Hyperion-8.1/car2sim_6cam.yaml` | | PandaSet / NV / Tesla / Alpasim | `configs/apps/AV/{PandaSet,NV,Tesla}/...`, `configs/apps/Alpasim/...` | The default recipe already composes `options/artifact: default` (which is what sets `checkpoint.artifact.enabled`, i.e. the renderable USDZ), MCMC densification, SfM-point-cloud initialization, and disables difix/mesh/ground. For a derived-rig capture with multiple selected cameras, NPA replaces only the background initializer through the native `model/gaussians/initialization@model.layers.background.initialization=accumulated_point_cloud` config group. The native SfM initializer asserts one camera on the entire data source; restricting an initialization-only camera list cannot fix it. Keep all discovered cameras, export all verified NCore world XYZ and uint8 RGB points to the reconstruction output's `initialization/ncore-sfm.ply`, and let NRE apply its world-to-NRE transform. Match the complete point count and disable optional random near/far seed points to preserve source-point-only initialization. Never mutate the conversion generation or change the native training budget. Single-camera selections retain native SfM; custom recipes and explicit initialization overrides remain the caller's responsibility. The dry run shows the planned native arguments without writing initialization artifacts. Retain `initialization/ncore-sfm.json` with the source/conversion and PLY hashes, selected cameras, count, and recipe/image identity. Native `parsed.yaml` records effective configuration, while USDZ `data_info.json` is input sequence metadata. Retain `reconstruction/reconstruction.json` and `novel_views/nre-render.json`; they bind the exact input member inventory, requested digest-pinned NRE invocation, observed RT-core GPU, resolved recipe, metrics, USDZ and independently decoded render bytes. Acceptance also needs a separate control-plane image-ID attestation matching that digest; the requested image string is not an observation of runtime bytes. Available frames and exported ground truth do not establish split membership or actual sampled training frames. Require separate native split/sampler evidence before claiming that every source image participated in training. The COLMAP multi-camera path still requires a real full RTX run after integration; offline PLY tests do not validate native reconstruction quality. The default object-centric recipe uses full-resolution training (`dataset.n_train_sequential_image_subsample=1`) and native PPISP camera-response correction (`model/post_processing@model.post_processing.b=ppisp`). Explicit Hydra overrides take precedence. Custom recipes keep their own defaults. The native 30,000-step budget remains unchanged. Enumerate what a given release actually ships with: ```bash find /app/run.runfiles/_main/configs -name '*.yaml' | sort /app/run --help # sub-command inventory /app/run render --help # authoritative flag surface ``` ## Novel Views vs Training Views `nre render` defaults to `--replicate-training-views`, which re-renders views the model was trained on. That is **not** a novel view. The tool therefore emits `--no-replicate-training-views` plus a rig offset by default: - `--rig-translation-offset` and `--rig-rotation-offset` are `FLOAT...` (three values) upstream; the CLI accepts one `"x,y,z"` string and expands it; - a zero offset with no custom trajectory is rejected rather than silently producing training views; - `--renderer default` (the artifact's own trained renderer) is the default. `nrend` is faster but needs the nrend model dictionary embedded in the USDZ, which the object-centric recipe disables. Independent photographic cameras do not form a rigid sensor rig. NPA embeds `npa-capture-trajectory.json` in their USDZ after training, preserving every camera's exact source pose and the original native archive entries. Rendering selects native `training-rig-poses-per-frame` calibration with that trajectory, identity camera-to-rig transforms, and the requested offset in each camera's local frame. Intrinsics, frame times and world coordinates must match the native artifact; missing exact source poses fail before publishing the reconstruction. Explicit custom trajectories, training-view replication and native rigid-rig captures retain their existing paths. ## Artifact Layout One S3 run prefix per run, so the agent's artifact browser picks it up: ```text s3:////neural-reconstruction// ncore/manifest.json # dataset, scene, sensors, rig derivation input/camera_images//*.jpg # real capture frames (export-ncore-benchmark-gt) reconstruction/last.usdz # renderable Gaussian scene reconstruction/parsed.yaml reconstruction/metrics.yaml # test/psnr, test/ssim, test/lpips reconstruction/val/... # NRE validation renders + videos novel_views//*.png # rig-offset novel views novel_views/.mp4 reports/final.json reports/sim2real.rrd # preferred artifact for the Rerun panel ``` `` must be a single safe segment embedding the submit timestamp, e.g. `neural-reconstruction-struktur28-20260731t050118z`, so the run picker dates the run by when it **started** (`npa.workflows.artifacts._run_started_at`). `.usdz` classifies as `download`, which is correct: no browser renders USDZ and the agent has no USDZ viewer. Viewability comes from the `.rrd`, `.png`, `.mp4` and `.json`. For a COLMAP-derived run, visualization fails closed unless the source attribution, conversion report, and rig-derivation sidecar are all readable. Detection uses both the pre-download S3 inventory and local files, so a partial subtree download cannot produce an RRD backed only by novel-view media. This strict trio applies only after a COLMAP lineage marker is present; optional data-factory subtrees and legacy preconverted-NCore runs remain best effort. ## Which Capability Answers This? Adapted from the NVIDIA router skill's picker table, re-pointed at what this repo actually implements. "Upstream" means the workbench has no verb for it: read the named sibling skill at and run it yourself; do not invent a workbench command for it. | I want to... | Where | | --- | --- | | Check NGC/HF access and that the GPU has RT cores, before pulling 14 GB | `npa workbench nurec check` | | Download a published NVIDIA NuRec/PhysicalAI capture in NCore V4 | `npa workbench nurec fetch` | | Train a reconstruction from an NCore clip and get a USDZ | `npa workbench nurec reconstruct` | | Render novel views along a shifted rig trajectory | `npa workbench nurec render` | | Get a Rerun recording the NPA agent will display | `npa workbench nurec visualize` | | Run all of the above on a GPU as one pipeline | `workflows/main/nurec-reconstruct.yaml` | | Measure PSNR / SSIM / LPIPS | Already emitted -- `reconstruction/metrics.yaml`, and `gaussians/summary` in the `.rrd` | | Convert a COLMAP reconstruction to NCore V4 | `npa workbench nurec convert-colmap` on CPU; not yet live validated | | Convert a non-COLMAP recording (drone, RGB-D, ROS 2 bag, ScanNet++) | Upstream `ncore`; these input-specific converters are not wired here | | Serve frames to CARLA / Isaac Sim / a custom simulator | Upstream `nre` (`serve-grpc`) -- not wired, see Limitations | | Render LiDAR sweeps from a USDZ | Upstream `nre` (`render-grpc --lidar`) -- not wired | | Extract individual 3D objects (cars, pedestrians) from a clip | Upstream `asset-harvester` -- not wired | | Clean up ghosting / floaters / flicker in rendered frames | Upstream `nurec-fixer` (DiffusionHarmonizer), or NRE's inline `--enable-difix` -- neither wired | | Generate segmentation / depth / ego-mask auxiliary inputs | Upstream `nre` via the `nre-tools` image -- not wired, see Limitations | | Package CAD or source meshes for simulation | Not NuRec at all -- that is SimReady, a different pipeline | ## Easy Mix-Ups Adapted from the router skill's `references/mix-ups.md`; the last row is workbench-specific. - **NuRec vs NRE.** NuRec is the product, NRE ("Neural Reconstruction Engine") is the engine that trains and renders. Used interchangeably in most docs. - **`ncore` then `nre`, never instead of.** NCore V4 is the input format; NRE reads it. They run in order. If NRE says a clip "is not valid NCore V4", the conversion step is missing, not a training bug. - **3DGUT vs 3DGRT.** Two Gaussian-splatting flavours inside NRE. The Hydra recipe picks one; you should not normally set it by hand. - **`PhysicalAI-Autonomous-Vehicles-NuRec` vs `Cosmos-Drive-Dreams`.** Both AV datasets on Hugging Face and easy to confuse. The former is *real* driving footage under the gated AV license; the latter is *synthetic* weather-augmented video under CC-BY-4.0. - **NRE's inline `--enable-difix` vs the standalone `nurec-fixer`.** Upstream documents `--enable-difix` as a built-in cleanup pass during rendering, while `nurec-fixer` wraps the public DiffusionHarmonizer release for frames already rendered. Neither is wired into a workbench verb, and the flag has not been exercised against `nre-ga 26.04` here -- treat it as upstream-documented, not as a verified workbench feature. - **A missing `rig -> world` edge is not a corrupt download.** The most common first failure is a pose-graph gap this workflow derives for you. See [The rig -> world Pose Edge](#the-rig---world-pose-edge-the-thing-that-breaks-first). ## Troubleshooting Rows marked (upstream) are adapted from the router skill's troubleshooting table; the rest were hit for real while landing this capability. | Symptom | Cause | Fix | | --- | --- | --- | | `402 Payment Required` pulling `nvcr.io/nvidia/nre/nre` | That repo needs an extra entitlement (upstream: `denied: requested access ...`) | Use the `-ga` channel, `nvcr.io/nvidia/nre/nre-ga:26.04`. `nurec check` reports `entitlement-required` | | `401`/`403` on a gated `nvidia/PhysicalAI-Autonomous-Vehicles*` override (upstream) | Gated dataset access is absent, or `HF_TOKEN` lacks `read` | Accept/request access on Hugging Face as the token owner, then re-run `nurec check`; the default `PhysicalAI-NuRec-PPISP` remains anonymous | | NRE will not load a clip: "not valid NCore V4" (upstream) | The recording was never converted | For COLMAP, use `nurec convert-colmap`; other source formats still need upstream conversion | | `KeyError: ('rig', 'world')` / no scene extent | The clip has camera poses but no rig edge -- NVIDIA's own COLMAP converter omits it | Automatic: `reconstruct` derives it. See the pose-edge section | | `Requested lidars not present in the data: dummy_lidar` | The recipe ships placeholder sensor ids | Automatic: the sequence's real ids are adopted | | `Requested cameras not present: camera_front_wide_120fov` | Same, for AV camera names | Automatic: same adoption path | | `Only one camera sensor is currently supported` | SfM point-cloud init is single-camera | Automatic: trains on the recorded reference camera and warns which cameras were dropped | | Cluster never finishes provisioning, `sudo: command not found` | The image ships no `sudo`; SkyPilot's K8s bootstrap calls it unconditionally | Automatic: `pod_config` initContainer installs a shim | | OOM / bus error early in training | `/dev/shm` defaults to 64 MB | Automatic: 64 Gi `emptyDir{medium: Memory}` | | USDZ looks like an early preview | `/artifacts/.usdz`, first-alphabetical picks step 1000 | Automatic: `latest_usdz()` picks the newest by step | | Renders look identical to the input frames | `nre render` defaults to `--replicate-training-views` | Automatic: the negation plus a non-zero rig offset is always emitted; a zero offset is rejected | | Output files owned by `root` after a local `docker run` (upstream) | `-u $(id -u):$(id -g)` was omitted | `sudo chown -R "$(id -u):$(id -g)" `, and pass `-u` next time. Not an issue in-pod, which runs as root by design | | Ghosting / floaters / flicker in rendered frames (upstream) | No cleanup pass | Upstream `nurec-fixer`, or NRE's inline `--enable-difix`. Neither is wired here | | A stage runs but publishes nothing to S3 | The stage ran in its own pod and wrote only to `/tmp` | Pass the handoff URIs. Every declarative stage is a separate pod | ## Verifying Secrets Safely From the router skill's `references/secrets-handling.md`. Never interpolate a token into an ad-hoc shell check. In particular this common line **prints the token**: ```bash echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}" # WRONG: emits yes ``` `${VAR:-no}` only falls back when the variable is *empty*, so a set token is echoed straight into the log. If you suspect one was printed, rotate it at or . Safe checks: ```bash hf auth whoami [ -n "${HF_TOKEN:-}" ] && echo "HF_TOKEN length=${#HF_TOKEN}" || echo "HF_TOKEN unset" [ -n "${NGC_API_KEY:-}" ] && echo "NGC_API_KEY length=${#NGC_API_KEY}" || echo "NGC_API_KEY unset" ``` Better, because it probes real *download authorization* rather than mere visibility (a gated HF repo still answers 200 on `/api/datasets/`): ```bash npa workbench nurec check --json ``` Every `nurec` failure payload is redacted before it is rendered, so a token cannot reach a log or a `--json` body. ## Running The Workflow ```bash # 1. Confirm access and GPU suitability first (seconds, no image pull). npa workbench nurec check --require-gpu --output json # 2. Submit the real GPU workflow. The wrapper substitutes ${...}; SkyPilot 0.12.2 # does not interpolate them itself. RUN_ID="neural-reconstruction-struktur28-$(date -u +%Y%m%dt%H%M%S)z" npa workbench workflow submit npa/src/npa/workbench/nurec/examples/nurec-reconstruct.yaml \ --run-id "$RUN_ID" \ --infra k8s/ \ --var NPA_NUREC_IMAGE=nvcr.io/nvidia/nre/nre-ga:26.04 \ --var NPA_NUREC_RUN_ID="$RUN_ID" \ --var NPA_NUREC_RUN_URI="s3:////neural-reconstruction/$RUN_ID" \ --var NPA_SRC_S3_URI="s3:///npa-src/" \ --var AWS_ENDPOINT_URL="" \ --var AWS_ACCESS_KEY_ID="" --var AWS_SECRET_ACCESS_KEY="" \ --var NGC_API_KEY="" # 3. Watch it. npa workbench workflow status "$RUN_ID" --watch npa workbench workflow logs "$RUN_ID" --follow # 4. Inspect the run tree. npa workbench nurec status --run-uri "s3:////neural-reconstruction/$RUN_ID/" --output json ``` Budget roughly: image pull (~14 GB compressed) on a cold node, a few minutes of setup, ~1 min fetch, ~20 min for 30k 3DGUT steps on one RTX PRO 6000, then rendering, the `.rrd`, and the upload. ## Two Container Quirks The Workflow Already Handles 1. **No `sudo`.** SkyPilot's runtime setup calls bare `sudo` when it writes the nofile limits and `/etc/fuse.conf` (`sky/templates/kubernetes-ray.yml.j2`). The NRE image has no `sudo`, so that step exits 127 and the cluster never becomes usable even though provisioning otherwise succeeds. The workflow's `pod_config` runs an initContainer that drops a shim which execs its arguments into `/usr/local/sbin` (on the image's PATH, and empty) — equivalent to SkyPilot's own `alias sudo=""` root path. 2. **64 MB `/dev/shm`.** NRE wants tens of GB; the workflow mounts a 64 Gi `emptyDir{medium: Memory}` at `/dev/shm`. Also: the image ships no `unzip`, `git`, or `ffmpeg`. Extraction uses stdlib `zipfile`; `setup` installs `ffmpeg` for `render --export-video`. ## Artifact Layout Surprise Upstream docs describe `/usd-out/last.usdz`. nre-ga 26.04 actually writes one artifact per checkpoint as `/artifacts/.usdz` plus `/artifacts/last.usdz`. `latest_usdz()` handles both and picks the newest, because taking the first alphabetical match would ship the 1000-step preview instead of the trained scene. ## Verify ### Real-GPU (live) verification The committed live e2e provisions a real RT-core GPU and asserts the whole capability against S3. It skips unless the environment is supplied: ```bash NPA_INTEGRATION_E2E=1 \ NPA_NUREC_E2E_BUCKET= \ NPA_NUREC_E2E_NPA_SRC_S3_URI=s3:///npa-src/ \ NPA_NUREC_E2E_INFRA=k8s/ \ NPA_NUREC_E2E_PREFIX=checkpoints \ npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_nurec_reconstruct_live_e2e.py -v -s ``` Also needs `AWS_ENDPOINT_URL`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `NGC_API_KEY` exported. `HF_TOKEN` is optional for the public PPISP default and required only for a gated/private dataset override. The current two-camera, six-stage default completed in about 70 minutes including worker setup; the [qualification receipt](../../../docs/workbench/evidence/nurec-default-quality-20261008.json) records PSNR 30.272694, SSIM 0.825870 and LPIPS 0.255115, plus decoded artifacts and visual review of actual nonzero-offset views. Raise `NPA_NUREC_E2E_MAX_WAIT_SECONDS` (default 7200) when a slower target requires it. The earlier single-pod result (**1 passed in 28m47s**, PSNR 31.19 / SSIM 0.833 / LPIPS 0.267) belongs to its historical input and settings, not these defaults. The default object recipe allows two million Gaussians (`model.strategy.add.max_n_gaussians=2000000`) to retain detail across both full-resolution photographic cameras. The native 30,000-step training recipe still applies. An explicit Hydra override selects a different model capacity. ### Default quality readback After the main demo completes, independently read its metrics and effective settings from S3: ```bash NPA_INTEGRATION_E2E=1 \ NPA_NUREC_QUALITY_RUN_URI=s3:////neural-reconstruction/ \ npa/.venv/bin/python -m pytest \ npa/tests/e2e/test_nurec_reconstruct_live_e2e.py \ -k default_quality_readback -q ``` Supply the same AWS credentials and endpoint as other live readback tests. This requires the default auto-exposure capture, native 30,000-step recipe, full-resolution training and rendering, two-million-Gaussian capacity, PPISP, photographic poses and the nonzero offset. It rejects metrics below PSNR 28 / SSIM 0.8 or above LPIPS 0.3. These checks supplement visual inspection of multiple cameras; they do not prove unseen surface quality. ### Offline verification ```bash npa/.venv/bin/python -m pytest \ npa/tests/workbench/test_nurec.py \ npa/tests/workbench/test_nurec_access.py \ npa/tests/workflows/test_nurec_viz.py \ npa/tests/workflows/test_nurec_artifacts.py \ npa/tests/orchestration/npa_workflow/test_real_components.py -q npa/.venv/bin/python -m ruff check npa/src/npa/workbench/nurec npa/src/npa/cli/nurec npa workbench nurec reconstruct --help ``` GREEN when: the workflow name equals its file stem, the accelerator is an RT-core GPU with no H100/H200 reference, every stage in the YAML is a real `npa workbench nurec` call, the `.rrd` carries the `novel_view` / `reconstruction` / `gaussians` entities, and a synthetic run prefix is listed by `list_runs` with `has_viewable=True` and `preferred == reports/sim2real.rrd`. ## Limitations For a dedicated RTX PRO 6000 project and offline digital-twin handoff, use `npa/examples/fleet/digital-twin.yaml` and `docs/workbench/guides/digital-twin.md`. Native renders emit `novel_views/render-evidence.json`, binding the source USDZ and rendered image hashes to observed allocated-device GPU telemetry. The HTML verifies these bindings and labels missing hardware activity unverified. Do not describe device-wide telemetry as process attestation or recorded-view playback as live free-camera/XR rendering. Keep exact infrastructure identities in private runtime configuration, never in the HTML or reusable spec. - **Linux x86_64 + NVIDIA RT-core GPU only.** aarch64 is unsupported upstream. - **NGC entitlement.** Only the `-ga` repositories are pullable with a standard key. - **Multi-GPU with `dataset.aux_data=false` is a known upstream crash.** The camera-only default therefore stays single-GPU; raise `--world-size` only with aux data enabled. - **No gRPC sensor-sim path yet.** `serve-grpc` / `render-grpc` (LiDAR sweeps, actor editing, simulator loops) exist in the container but are not wired into a workbench verb. - **`nre-tools` auxiliary data (segmentation, depth, DINOv2) is not wired in.** It is a second ~22 GB image and the object-centric default does not need it. - **No object harvesting or actor editing.** Upstream `asset-harvester` extracts per-object `.ply` splats and `nre export-external-assets` packages them; the workbench does neither. Asset Harvester always runs *before* USDZ packaging. - **No frame-cleanup pass.** Upstream `nurec-fixer` (DiffusionHarmonizer) and NRE's inline `--enable-difix` are both unwired and unverified here. - **COLMAP ingestion is not yet live validated.** Other source-format converters and novel sensor-rig integrations remain upstream `ncore` work.