42 added, 0 removed. Audit A to A.
---
name: testing-conventions
description: Use before running or interpreting NPA tests, lint checks, or validation reports.
---
# Testing Conventions
Use the repository virtualenv. Never use bare `python`; use `npa/.venv/bin/python`.
Correct command from repo root:
```bash
npa/.venv/bin/python -m pytest npa/tests/ --ignore=npa/tests/e2e --timeout=120 -q
```
Correct command from inside `npa/`:
```bash
cd npa
.venv/bin/python -m pytest tests/ --ignore=tests/e2e --timeout=120 -q
```
`ruff` is available in the venv:
```bash
npa/.venv/bin/python -m ruff check <files>
```
## Existing Failures And Gates
- Known pre-existing failure: `tests/smoke/test_cosmos_serverless_smoke.py` has 5 tests gated by `NPA_COSMOS_SERVERLESS_SMOKE=1`; they fail with `Unable to list Nebius VPC networks for project project-smoke: unsupported`.
- E2E tests are gated by `NPA_INTEGRATION_E2E=1` and excluded from standard runs with `--ignore=npa/tests/e2e`.
- Pipeline E2E tests use the `e2e_pipeline` pytest marker.
- Live Nebius Token Factory tests use the `token_factory_e2e` marker (in `npa/tests/e2e/test_token_factory_e2e.py`). They self-skip without a real `NEBIUS_TOKEN_FACTORY_KEY`; the marker is in conftest `_LIVE_MARKERS` so the key is not scrubbed. Run with `NEBIUS_TOKEN_FACTORY_KEY=... pytest npa/tests/e2e/test_token_factory_e2e.py`.
Expected baseline: 1242+ passed, 21 skipped, 1 xpassed, 0 failures.
Use evidence-based convergence: report numeric pass counts and exact failure messages, not subjective assessment.
## Unit Test Rules
Tests must not hit real infrastructure. Mock SSH, S3, Nebius APIs, GPUs, and network calls at the call site. CLI tests use `typer.testing.CliRunner` against `npa.cli.main:app`.
+ ## Live-Infra Testing Is A Priority (not optional)
+
+ Smoke + mocked-unit tests are necessary but **not sufficient**. Any change to an
+ `npa.workflow` spec, a workbench tool / `toolRef`, or the agent/deploy
+ (Terraform / `provisioner`) path MUST also land committed **live-infra** coverage
+ — do not stop at smoke. Definition of done: the change is wired into a live path
+ and you report numeric results from running it.
+
+ - **New/changed npa.workflow spec:** register it in `SUBMIT_LIVE_MATRIX`
+ (`npa/src/npa/orchestration/npa_workflow/submit_matrix.py`) with the right tier
+ (`cpu` | `gpu` | `multi`). If it has a dynamic gate/loop, also add it to
+ `DYNAMIC_SPECS` in `npa/tests/e2e/npa_workflow_live_helpers.py` so the runner
+ supplies `--assume-decision`; if it actually executes (non-plan-only), seed
+ inputs in `seed_live_workflow_inputs`. Use `plan_only=True` **only** when stages
+ are stubs or a real run would burn a GPU on a stub (the repo convention:
+ "do not burn GPUs on stubs"). Keep `test_submit_live_matrix.py` green.
+ Run it:
+ ```bash
+ NPA_E2E_NPA_WORKFLOW_SUBMIT_TIERS=<cpu|gpu|multi> \
+ NPA_E2E_NPA_WORKFLOW_SUBMIT_SPECS=<spec>.yaml \
+ ./scripts/npa-workflow-submit-live-e2e.sh
+ # plan-only preflight (no job launch):
+ NPA_E2E_NPA_WORKFLOW_SUBMIT_PLAN_ONLY=1 ./scripts/npa-workflow-submit-live-e2e.sh
+ ```
+ - **Live workflow render/infra:**
+ ```bash
+ NPA_INTEGRATION_E2E=1 npa/.venv/bin/python -m pytest \
+ npa/tests/e2e/test_npa_workflow_live_e2e.py \
+ npa/tests/e2e/test_npa_workflow_live_infra.py -q
+ ```
+ - **Token Factory-backed stages:** the `token_factory_e2e` live tests must pass
+ with a real `NEBIUS_TOKEN_FACTORY_KEY`.
+ - **Agent/deploy changes:** validate a real `npa agent destroy` + `deploy` (and,
+ for credential-path changes, that it is reproducible). Known trap: a stale
+ ambient `NEBIUS_IAM_TOKEN` shadows the fresh `var.iam_token` in the Nebius
+ Terraform provider (`PermissionDenied`/`Unauthenticated` even though the CLI
+ works); `provisioner._run` scrubs it, but when reproducing by hand
+ `unset NEBIUS_IAM_TOKEN NPA_NEBIUS_IAM_TOKEN` first.
+ - If a full live run is genuinely infeasible in the environment, say so
+ explicitly and still commit the `plan_only` live-matrix entry — never silently
+ ship smoke-only.
+