guardrail-failures git:20261105.63f0ad7A

Immutable. This exact content never changes and is served at /api/v1/blob/b049053bca7793bd.

---
name: guardrail-failures
description: Use when a guardrail test or CI gate in npa fails and you need to map the failure to its cause and fix rather than reverse-engineering the assertion.
---

# Decoding A Guardrail Failure

The guardrail suite checks thousands of repository contracts. It reads the
repo and checks that surfaces which must
agree still agree. A failure almost always means one of two things — you changed
one side of a contract and not the other, or you added something the contract
requires you to register.

Run the whole suite; it is cheap enough that narrowing is rarely worth it:

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails -q
```

Narrow once you know the file:

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_tool_catalog_argv.py -q
```

Most of these tests carry a module docstring explaining the incident that caused
them to exist. When the fix is not obvious, read it — it usually names the
production failure the check prevents.

## Triage Order

1. Read the assertion message. These tests are written to name the fix, not just
   report a mismatch.
2. Identify which side of the contract you changed. The guard is almost never
   wrong; it is reporting that a second file needs the same edit.
3. Only weaken a pinned exemption list if the exemption is genuinely obsolete.
   Several lists are explicitly designed to shrink and never grow.

## CLI, Catalog, Spec, And SDK Coherence

These fire when a tool's surfaces drift apart. This is the group a workbench
change hits most.

| Guardrail | Fix when it fails |
|---|---|
| `test_tool_catalog_argv` | A `toolRef` argv names a CLI option that does not exist, or passes a value the option cannot mean. Correct the argv in `catalog.py` — see `skills/atomic/toolref-argv-contract/SKILL.md`. |
| `test_module_toolref_argv` | A `python -m` toolRef no longer parses against its module's own argparse. Sync the module parser and the catalog argv. |
| `test_three_tier_contract` | A new CLI group has neither a `CapabilityContract` nor a seam entry, or a CLI parameter is unreachable from a spec without a stated reason. Add the contract or record the `spec_gap` category. |
| `test_spec_declared_outputs` | A spec's declared `outputs:` path is not what the tool writes. Fix the spec, or the tool's `result_uri_for()` helper. |
| `test_spec_paths_are_not_repo_relative` | A stage argv points at a path that exists in this checkout. Stages run in a pod; use an `s3://` URI or an in-image absolute path. |
| `test_shown_workflow_catalog` | A catalog entry no shipped spec reaches, or a spec outside the npa.workflow catalog. Add a reference spec or list the entry in `PUBLIC_REUSABLE_TOOLREFS`. |
| `test_workflow_schema_chaining` | A state depends on another but reads a schema that one does not produce. Align the producing and consuming stages. |
| `test_typer_command_calls` | A Typer command is called as a plain function without `@resolve_typer_defaults`. Add the decorator to the callee. |
| `test_internal_cli_entrypoint` | An internal argv no longer matches `python -m npa`. Build it with `internal_cli_argv()`. |
| `test_help_to_markdown` | The generated-docs converter mishandles wrapped Rich help. Fix `scripts/_help_to_markdown.py`. |

## Skills And Documentation

| Guardrail | Fix when it fails |
|---|---|
| `test_skills_index` | A `SKILL.md` exists without an `skills/index.yaml` entry (or the reverse), frontmatter `name` disagrees with the index, or a smoke does not run. Register the skill with a passing smoke. |
| `test_develop_skills` | A development skill names a repo path or guardrail that does not exist, or a new guardrail file is undocumented. Update the skill. |
| `test_no_dangling_workflow_references` | A doc, skill, or script names a workflow YAML that does not exist. Repoint or remove the reference. |
| `test_cli_docs_drift` | An `npa ...` command mentioned in `docs/workbench/`, `workflows/guides/`, or `skills/` does not resolve in the Typer tree. Update the doc/skill to the current command, or add a precise allowlist entry in `npa/tests/guardrails/test_cli_docs_drift.py` with the reason. |
| `test_docs_green_path` | The documented path from README to a real submit is no longer runnable end to end. Update the quickstart with the new step. |
| `test_documentation_examples` | Fix the reported Markdown link/heading, shell/Python syntax, CLI command/option, or undeclared workflow variable. Quote shell placeholders; label HTTP routes and abbreviated grammar as `text`. Keep the package README's planning example executable offline. Generated CLI references and historical command records retain separate validation scope. |
| `test_agent_workflow_operations` | The agent operations guide omitted a lifecycle stage, exposed a non-NPA subprocess, or coupled workflow operation to a model/provider flag. Restore the fixed, provider-neutral NPA command journey. |
| `test_audit_container_docs_skill` | The public container catalog disagrees with the publish inventory. Update `docs/workbench/container-image-catalog.md`. |
| `test_access_approval_skill` | Operator-owned gated access stopped being exact-artifact scoped, `Ready` became a legal or redistribution claim, or child solutions began repeating the same run-scoped use question. Restore the narrow access and inheritance contract in `skills/atomic/access-approval/SKILL.md`. |
| `test_solution_licensing_skill` | The licensing skill no longer covers an artifact boundary. Update the skill, not the test. |
| `test_third_party_eula_preflight_skill` | The EULA preflight skill is not discoverable from the operational skills that need it. Add the link. |
| `test_runtime_fetch_onboard_skill` | Runtime-fetch onboarding lost a legal boundary, safe delivery shape, byte/secret exclusion, real-workload proof, or discovery link. Restore the contract in `skills/workflows/runtime-fetch-onboard/SKILL.md` and its copyable reference; do not treat runtime fetch as permission. |
| `test_nebius_cli_compatibility` | The `nebius-cli` version drifted between packaging, `images.py`, and docs. Bump all of them together. |
| `test_skypilot_absence_contract` | The required SkyPilot version, reviewed naming-source hashes, or installed wheel no longer agree. Follow the upgrade procedure in `docs/architecture/skypilot-absence-recovery.md`; review all nine source files and refresh the contract together. |
| `test_paidf_image_tags_match_code` | Keep explicit private build tags aligned with `supported_tool_version()`, describe governed public candidates separately, and retain the fresh-cluster GPU dispatch smoke that initializes SkyPilot pull authority. |
| `test_public_runtime_registry_defaults` | A supported workload inherited a generic build/private registry, emitted an unqualified first-party image, or attached credentials to the official public release. Keep repository-owned defaults on `ghcr.io/nebius/nebius-physical-ai`; require an explicit scoped override for custom bytes. |
| `test_default_cluster_fits_quickstart` | The default cluster can no longer schedule the documented quickstart. Raise the preset or lower the spec's requests. |
| `test_workflow_readiness_records` | A readiness record is malformed, orphaned, or no longer matches its adjacent workflow SHA-256. Review the workflow-byte changes and reassess every affected claim before updating the digest; never mechanically preserve or upgrade evidence statuses. |

## Secrets, Confidentiality, And Consent

Treat every failure here as blocking. Do not add an exemption to make one pass.

| Guardrail | Fix when it fails |
|---|---|
| `test_confidentiality_scan` | The scanner, its built-in Nebius patterns, or the gitleaks wiring changed. Keep `.gitleaks.toml` and `npa/src/npa/guardrails/confidentiality.py` in sync. |
| `test_gitleaks_policy` | A broad exception can hide real secrets, or bootstrap code is being activated before it becomes trusted base code. Keep default rules enabled, forbid global path exceptions, bind fixture exceptions to exact anchored lines, scope test-tree paths only on the custom infrastructure rules, and leave scanner-helper activation to a later PR based on this bootstrap. |
| `test_gitleaks_tree_materialization` | The complete-tree Gitleaks scan transformed, omitted, or binary-classified committed content. Require the trusted policy revision to equal or precede the excluded history base. Stream bounded authenticated history, tree, blob, and selected commit-metadata bytes from Git objects; charge both native-history scanner invocations against a bounded preflight of the pinned Git patch stream; preserve each payload behind neutral and logical-path scanner-safe wrappers; reproduce pinned scanner fragmentation and scan prefixed overlap windows under both synthetic and exact logical paths with the real policy; materialize every bounded derived output in a source-start exact-path view even when no custom path selector exists, and compact complete raw and transformed exact-path projections into independent roots in deterministic candidate-control-resistant forced-text Git history batches, each bounded to 100,000 files and 2 GiB with at most twelve batches per population; bind every exact-path regular file to its ordered path, size, and content digest, reject links/non-regular entries or later inventory/content changes, and prepare all candidate, historical, and transformed histories before any Gitleaks process, so pathless and path-constrained absolute content anchors retain byte-zero semantics without one scanner process per projection; retain `outer!member` archive paths and isolate encoded candidates, bounded repository-determined RFC 1950 output decoded from them or embedded directly in ordinary files or complete archive-member views, every recursively emitted RFC 1950 child, and every required path association; recognize all 66 bounded-window RFC 1950 headers, enforce CINFO, skip FDICT's DICTID before retaining dictionary-independent bytes, and report arbitrary embedded output that requires an absent external dictionary as unresolved rather than covered or invalid at every raw, text-decoded, encoded-derived, or recursive-derived layer; carry the original source-offset canonicality through every text-decoding layer, recursive-output node, root capability, and cache identity, reject unresolved FDICT only when both the original chain and current node begin at canonical byte zero, and require authenticated repository-determined output before treating another strict-source offset as FDICT; grant archive-member RFC 1950 capability only to a real writer that retains a complete ZIP member or single decompressed stream and every derived byte under its exact logical path, while keeping parser-visible TAR structure strict before member attribution; reuse one physical exact-path file only when the raw and transformed bytes are identical, count that identity reuse explicitly, and advance repeated exact-path allocation monotonically; shard historical file/directory prefix transitions; require distinct findings for generated binary, archive, encoded, and derived-output controls; materialize symlink blobs as regular files; reject gitlinks and unsupported modes; bound logical archive members, parser-visible structure, policy, reports, generated files, decompressed bytes, exact-path roots, direct and recursive compressed candidates, decode depth, derived output, and exact-path history batches before deduplication; fail closed on malformed/encrypted/nested archives, non-RFC1950 or encoded derived containers, recursive cycles, over-depth or incompletely accounted repository-determined output, authenticated direct RFC 1950 output when the caller cannot retain a complete scan view, archive directories whose exact paths cannot be represented, scanner diagnostics, any generated-root commit allowlist collision, and a matching custom rule whose exact root `.gitleaksignore` path cannot be represented without activating candidate scanner control; leave trusted path-regex evaluation to pinned Gitleaks rather than a different regex engine; bind the executable to the official release digest; sanitize the child environment; revalidate mutable Git metadata after scanning; and keep private temporary roots fail-closed. Supplemental paths can invalidate original-path fingerprints and should fail safe; use only reviewed content-scoped exceptions. |
| `test_ncore_attribution` / `test_ncore_source_attribution` | NCore's exact CPython notice, immutable dual-archive proof, Git path/mode boundary, diff line mapping, or source disposition drifted. Restore the exact provenance-bound contract; never add a general exemption or hide raw findings. |
| `test_security_source` | Source scanner output, inventory, ignore handling or stable finding identity is incomplete. Restore fail-closed scanning; run the real scanner workload in `docs/security/merge-security-gate.md`. |
| `test_security_gate` | A regression comparison, snapshot boundary or dependency report check failed. Preserve duplicate detection and reject incomplete reports or source paths outside the snapshot. |
| `test_dockerfile_security_inventory` | Preserve literal exact Python pins from shell and JSON RUN instructions, separate conflicting versions across stages, reject malformed dependency-bearing instructions, and require every declared package in Trivy output. Run the real Dockerfile regression fixtures in `scripts/security_regression_checks.py`; do not replace missing coverage with an empty inventory. |
| `test_robocasa_security_inventory` | Restore `imageio-ffmpeg==0.6.0` with only sdist SHA-256 `e2556bed8e005564a9f925bb7afa4002d82770d6b08825078b7697ab88ba1755` in RoboCasa's input and lock; retain `--no-binary=imageio-ffmpeg` at install and lock generation, normalize with `pin_imageio_source.py`, and preserve the `fsspec>=2026.6.0` and `multidict>=6.9.1` floors. Reject wheel hashes and unknown direct URLs rather than widening inventory trust. |
| `test_security_install` | Scanner bootstrap no longer retries transient download failures or rejects permanent HTTP/checksum failures before extraction. Exercise real curl against a local HTTP fixture and preserve the pinned checksum. A hosted 504 during scanner installation is a transport failure; rerun its failed jobs after recovery, without retrying or suppressing security findings. |
| `test_image_security_gate` | The required security check stopped waiting for image scans, lost PR/merge-queue coverage, duplicated automatic runs, or changed existing alert identity. Restore full PR image validation or trusted identical-tree queue evidence, fail-closed result checks, distinct concurrency groups, and reporting configuration. |
| `test_security_scan_input` | The manual image-scan input became shell syntax. Pass it through a step environment variable and quote that variable in the command. |
| `test_agent_secret_guard` | A secret path became tracked, `.gitignore` stopped covering agent/cursor secrets, or a literal secret or live IP landed in agent files. Remove it and rotate. |
| `test_agent_credential_provenance_isolation` | Non-live tests inherited agent credential provenance and could reach a provider. Keep `NPA_NEBIUS_CREDENTIAL_SOURCE` in the shared environment scrub; set explicit fixture provenance only after isolation runs. |
| `test_agent_no_hardcoded_data` | Agent or insights source embeds run names, answers, or infra endpoints. Resolve them from live tool observations instead. |
| `test_access_key_list_safety` | Docs or code request secret-bearing access-key list JSON. Use a `--format jsonpath=...` projection. |
| `test_isaac_eula_plumbing` | Isaac consent uses something other than the single run-scoped mechanism. Use the scoped `ACCEPT_EULA` plumbing. |
| `test_live_tests_never_declare_a_licence` | A live test accepts vendor terms for the operator. Gate on an operator-provided variable instead. |
| `test_operator_env_covers_gated_solutions` | A solution whose smoke hits a vendor gate has no operator-env entry. Add it. |
| `test_hygiene_guards` | Several rules: no `sky launch --down/--autodown`, GPU tests skip on explicit env flags only, examples use a registry placeholder rather than a first-party ID, monolith modules must not grow, and no silent `except Exception: pass`. The message names which one. |

## Images And Packaging

| Guardrail | Fix when it fails |
|---|---|
| `test_workbench_image_k8s_prereqs` | An image lacks what SkyPilot needs on Kubernetes (python3, rsync, sudo with NOPASSWD, or the Isaac group/PATH rules). Update the Dockerfile and the shared install script together. |
| `test_isaac_skypilot_apt_closure` | An Isaac image omits an early SkyPilot transfer package or accepts a missing or removed package. Install `curl`, `fuse`, `netcat-openbsd`, `rsync`, and `wget` in both Isaac build paths, and require installed status for every package. |
| `test_unbuilt_image_records_agree` | The four files that record whether an image is built disagree. Make them agree; do not mark an image built that is not. |
| `test_seedvr2_public_release_workflows` | SeedVR2 pre-push or post-pull byte verification lost its exact image identity, local tarball, or clean-root source SHA binding. Restore both scan invocations and keep the pre-publication gate before the push. |
| `test_trivy_policy` | `trivy.yaml` no longer matches the current nested schema. Update the config. |
| `test_public_release_workflows` | The single public GHCR channel drifted: development tags must be immutable full-SHA references, all pre-publication gates must precede the push, promotion must remain exact-digest-bound, failed-build cleanup must refuse shared digests, and release health must stay anonymous/read-only. Restore those contracts in the public publication workflows; never recreate a private candidate channel. |
| `test_public_workflow_candidate_governance` | Fetch full main history, rebuild and repin a candidate from a main commit, and retain its exact digest, build evidence and qualification scope in `public_release_manifest.json`. Keep PAIDF's submitted-adapter source overlay enabled. Update the documented blocked-workflow inventory after a qualified resolution change; never conceal it with wildcard overrides or remove quarantine to make planning pass. |
| `test_secure_image_build_skill` | The secure image procedure no longer names every irreversible publication gate or its governing licensing, build, validation, and confidentiality procedures. Restore the missing gate or delegation in `skills/atomic/secure-image-build/SKILL.md`; do not weaken the test or substitute labels for artifact evidence. |
| `test_paidf_starter_asset` | The PAIDF starter asset was bundled instead of runtime-fetched, or its contract drifted. Keep the runtime-fetch model. |

For RFC 1950 FDICT specifically, deterministic bytes emitted before a malformed
trailer or unavailable dictionary are still scanner input. A caller must retain
and scan those bytes, or reject the candidate when no derived-output sink exists.
For selected commit metadata, retain deterministic output from structural header
bytes and apply the strict no-sink boundary to the complete commit message.

## Retired Surfaces And Example Boundaries

These exist because a retired thing keeps coming back. The fix is always to use
the current surface, never to recreate the old one.

| Guardrail | Fix when it fails |
|---|---|
| `test_skypilot_catalog_retirement` | `npa/src/npa/workflows/skypilot/` reappeared, or a spec carries retired `skypilotTwin` metadata. Author an npa.workflow spec instead. |
| `test_skypilot_readme` | The retired catalog README came back. Delete it. |
| `test_burst_examples` | A burst example became multi-stage, lost its `${VAR}` placeholders, or unpinned its image. Multi-stage work belongs in an npa.workflow spec. |
| `test_byof_profiles` | A BYOF profile drifted from being a single-task resource profile. Same boundary as burst. |
| `test_nurec_examples` | The NuRec single-pod example drifted toward being a catalog entry. Keep it an example. |
| `test_workflow_image_check` | Image extraction, placeholder resolution, or the retired-catalog check broke. Use the fixtures for raw SkyPilot image tests. |

## Infrastructure, CI, And Meta

| Guardrail | Fix when it fails |
|---|---|
| `test_ci_concurrency` | A validation workflow introduces a shared job lock, a matrix parallelism cap, or a workflow group that can serialize unrelated candidates or collide with its caller. Restore independent job scheduling and per-candidate workflow groups as documented in `CONTRIBUTING.md`. |
| `test_ci_workflows` | A workflow lacks per-candidate supersession, duplicates feature-branch runs, drops a PR Python compatibility version or its early regression check, or makes mypy blocking. Restore those contracts. |
| `test_ci_media_setup` | A coverage shard lost mandatory media setup, setup downloads packages despite working tools, uses the slow Azure mirror, changes Ubuntu signature policy, or accepts a failed installation. Restore the isolated signed-source installer and executable probes. |
| `test_daily_vm_network` | Daily SSH access loses host-key verification or isolated staging cleanup, requests unnecessary identity permissions, or adds a PR trigger. Restore pinned host keys, direct SSH, per-run private directories, and cleanup on failure. |
| `test_e2e_gate_reachability` | A new `NPA_*` e2e gate has no runner mapping. Wire it into `scripts/dev-vm-daily-tests.sh` or record a reviewed manual reason. |
| `test_terraform_provisioner_shell` | Bash embedded in the agent Terraform is not syntactically valid, or an SSH wait is unbounded. Check the heredocs. |
| `test_no_syntax_warnings` | The package emits a `SyntaxWarning`, usually an invalid escape in a regex. Use a raw string. |
| `test_collection_guard` | The zero-collection guard itself broke. Something made pytest collect nothing. |

## When The Gate Is Not A Guardrail Test

`Lint / docs-drift` is not in this suite. It fails when a CLI option changed and
`docs/cli/` was not regenerated; fix it with `bash scripts/build_docs.sh`. The
full mapping from CI job to local command is in
`skills/atomic/pre-pr-validation/SKILL.md`.