oxo-flow-scientific-analysis · diff
git:20261115.7a5f43f to git:20261117.fc6780d
1 added, 1 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: oxo-flow-scientific-analysis
description: >-
Run reproducible bioinformatics analyses with the oxo-flow pipeline engine
from natural-language tasks: plan workflows, author .oxoflow TOML projects,
pass static gates (validate/dry-run/lint), execute on local or cluster
backends, monitor and correct failures via checkpoint resume, QC outputs
independently, and deliver reports plus a reproducibility receipt.
Treats input data as read-only, confines agent scratch work and journaling
to the project working directory, and requires human authorization before
any destructive operation.
Use when the user asks to analyze sequencing data with oxo-flow, design or
fix an oxo-flow workflow, run/validate/debug oxo-flow, or produce a
reproducible analysis report or receipt.
---
# oxo-flow Scientific Analysis
oxo-flow is a Rust-native bioinformatics workflow engine: DAG execution, 8
environment backends, sample wildcards, checkpoint/resume, provenance.
This skill is a compressed operating manual for agents — read it in full once
per task; follow the links for details on demand.
| Canonical entry points | URL |
|---|---|
| Documentation | https://traitome.github.io/oxo-flow/ |
| **This skill (site)** | https://traitome.github.io/oxo-flow/latest/SKILL/ (newest tagged release; `/dev/SKILL/` tracks main) |
| This skill (raw) | https://raw.githubusercontent.com/Traitome/oxo-flow/main/docs/guide/src/SKILL.md |
| Repository (dev context: AGENTS.md) | https://github.com/Traitome/oxo-flow |
| Workflow JSON Schema | https://raw.githubusercontent.com/Traitome/oxo-flow/main/docs/schema/oxoflow-v1.schema.json |
| Community template library | https://oxo-flow-community.github.io/ |
| Reference workflows | https://github.com/Traitome/oxo-flow/tree/main/examples/gallery |
| VS Code extension | https://traitome.github.io/oxo-flow/latest/reference/vscode-extension/ (`.oxoflow` editing: completion, diagnostics, one-click run/validate/graph; `code --install-extension traitome.oxo-flow` — published to Open VSX and the VS Code Marketplace; offline VSIX on every GitHub release) |
**GitHub unreachable?** Prefix any GitHub URL with a mirror proxy
(`https://ghfast.top/<url>` or `https://gh-proxy.com/<url>`), or web-search for
a current proxy; ecosystem mirrors: see `how-to/china-mirrors`.
## The Analysis Loop
Seven stages, four approval gates (stages 0, 1, 2, and 6). Every stage
ends with a **Done when** condition — do not proceed without it.
### 0. Data & Task Contract
- Restate the natural-language task as: scientific question, inputs,
expected outputs, success criteria, compute budget.
- **Verify every input path on the filesystem** (`ls`/`stat`); never invent
or assume paths.
- Done when: all inputs exist, formats identified, PI role approves.
### 1. Plan the Workflow Shape
- Prefer an existing base: the docs gallery (validated example workflows),
the community library, `examples/gallery`. Write new only when no base
matches.
- Choose tools via the knowledge bases (see Link Index); containers must be
fully qualified (`quay.io/...`) — there is no implicit registry fallback.
- Decide sample dimensions (`sample_pattern`, `[[pairs]]`, `[[sample_groups]]`),
one of the 8 environment backends, and explicit `depends_on` between rules.
- Done when: rule sketch + tool list + chosen base approved by PI.
### 2. Author the .oxoflow Project
- `oxo-flow init` scaffolds the project; write the workflow TOML (+ scripts);
keep it in git — versioning is reproducibility.
- **Mandatory static gates, in order, before any compute:**
1. `oxo-flow validate --json <workflow.oxoflow>` — structure, missing inputs, capacity
2. `oxo-flow dry-run --json <workflow.oxoflow>` — expansion preview, resource audit, `-j` suggestion
3. `oxo-flow lint <workflow.oxoflow>` — best practices
- Fix and re-run until clean; `oxo-flow graph` to eyeball the DAG.
Edge inference is best-effort but broad: exact strings, globs, and
directory inputs all form edges when a producer's output matches;
keep `depends_on` for ordering files can't express (side-effect-only
steps).
- Optional AI assistance: `[ai] enabled = true` in the workflow activates
AI validate/lint/debug/recover.
- Done when: all gates pass; engine-QC role sign-off.
### 3. Submit & Execute
- Local: `oxo-flow run -j <N> --provenance` (always `--provenance`).
- Detached + scripted supervision: `oxo-flow run … --detach` then
`oxo-flow wait --workdir <dir> --timeout <secs>` (exit 0 success,
1 failed/incomplete, 3 timeout, 4 no run) instead of a poll-for-files loop.
- Cluster: `oxo-flow run --profile <NAME>` with a `[cluster]` block;
manual escape hatch `oxo-flow cluster submit|status|cancel|logs`.
- Web: `POST /api/runs` or the web UI.
- Done when: run id + checkpoint path recorded.
### 4. Monitor & Correct
- `oxo-flow status` (or `cluster status`) to completion.
- On failure: read structured diagnostics, diagnose, fix, re-run the static
- gates, then resume — `oxo-flow resume` / `run --resume-failed`
+ gates, then resume — `oxo-flow resume <checkpoint>` (or `run --resume-failed`)
(checkpoint avoids recompute). Optional AI recovery:
`run --ai-recover --ai-max-retries 3`.
- Retry budget: max 3 correction loops, then escalate to the human.
- Done when: all rules complete, evidenced by checkpoint state.
### 5. QC Review (independent)
- Outputs exist, non-empty, plausible; `oxo-flow provenance verify
.oxo-flow/checkpoint.json` passes; spot-check logs for silent errors.
- Judge against the Stage 0 success criteria. Performed by an evaluator
role — **never the author of the workflow**.
- Fail → back to Stage 2 with written findings.
### 6. Reproducible Delivery (Receipt)
- `oxo-flow report -f md -o REPORT.md --ci` (execution truth: real exit
codes, checksums, expanded commands) + `-o report.json` copy.
- **Receipt bundle:** `REPORT.md` + checkpoint JSON + provenance-verify
output + workflow git SHA + `--versions-yml versions.yml` (newer releases;
skip if the flag is rejected) + input manifest.
- Optional archival: `oxo-flow export -f toml|docker|singularity`.
- Done when: PI signs off against the original task.
## Data Safety & Workspace Discipline (BINDING)
You are the actual operator of the user's data. Input data is **read-only**.
The engine protects rule shells at run time (the shell-command guard blocks
deletions outside the run workdir — lint code E011 flags the pattern early;
`[engine] protected_data_trees` detects drift), but those
gates do not cover your own terminal. These rules bind every command you run:
1. **Never delete or overwrite inputs without explicit human authorization.**
This includes `rm`, `mv`, `truncate`, `dd of=`, `sed -i`, `git checkout
--`, `git clean`, and any `>` redirect into a path you did not create
this task. Ask first: quote the exact command, name the target, say why —
then wait for a human "yes". Destructive + no human reachable = stop.
2. **Snapshot before mutating.** Copy what you are about to change into your
scratch tree (`cp -a`) and record `sha256sum`. Write to NEW paths and
rename on success; never edit in place. Append with `>>`; never `>` an
existing file you did not create.
3. **Do all probing in one scratch tree inside the project** —
`<project>/.oxo-flow/agent/<date>-<task>/` (gitignored), not `$HOME`,
not a global `/tmp` scatter, not outside the working directory. Point
tools there with `OXO_FLOW_TMPDIR=<scratch>/tmp` (or `[engine] tmp_dir`).
Delete only your own scratch, and only when the task is done.
4. **Journal every mutating operation** — one JSON line per command appended
to `.oxo-flow/agent/operations.log`:
`{"ts":"<ISO8601>","cwd":"<dir>","cmd":"<argv>","exit":<code>,"note":"<why>"}`.
Read-only inspections need not be journaled.
5. **Protect input trees at plan time.** For every workflow consuming vendor
data, recommend `[engine] protected_data_trees = ["raw/", "reference/"]`
— drift is then detected before the next run. Detection, not prevention:
the deletion itself must still never happen.
6. **Inspect without mutating.** Prefer `oxo-flow status`, `oxo-flow report`,
`oxo-flow provenance verify` over ad-hoc shell. When unsure, dry-run.
Recipes (snapshots, journal helper, scratch layout, protected-tree config):
`how-to/agent-safety`.
## Guardrails
- **Evidence first** — never claim success without checkpoint/report evidence.
- **Real paths only** — verify inputs exist before referencing them.
- **Static gates before compute** — every workflow edit re-runs
validate → dry-run → lint. An unbound placeholder (`{sample}`,
`{meta.<col>}`, `{config.<key>}`) is a hard run-time error by default;
lint W041 names the rule whose sample domain has no binding, and
`singleton = true` declares a deliberate one-instance rule. Shell
`${...}` is never a placeholder.
- **Correct, don't restart** — prefer checkpoint resume unless inputs were
invalidated.
- **Explicit edges** — edge inference covers exact strings, globs, and
directory inputs; `depends_on` remains the escape hatch for ordering
that file matching can't express.
- **Version everything** — workflow in git; its SHA lands in checkpoint and
report.
- **Budget & stop-loss** — retry caps, human escalation, resource limits from
dry-run.
- **Don't guess tools** — search the knowledge bases; prefer live-tested
gallery/community workflows.
## Scientist Team Mode
One skill covers 3–4 roles. Roles are stage owners in the loop above; the
handoff artifact is each stage's output. Roles can be played sequentially by
one agent or by parallel subagents; the independence rule is binding either way.
| Role | Stages | Responsibility |
|---|---|---|
| **PI** (science lead) | 0, 1 gate; 6 sign-off | Task contract, scope veto, final acceptance |
| **Computational biologist** | 1–4 | Workflow design, tools, TOML, fixes |
| **oxo-flow engine & QC expert** | 2 gates, 4, 5 | Static gates, failure diagnosis, verification |
| **Independent evaluator** (optional 4th; a community expert doubles as tool-choice advisor in Stage 1) | 5–6 | Adversarial review vs. the scientific question |
Minimal configuration (3 roles): PI doubles as evaluator — but must be a
**separate agent instance** (independence rule: the author never verifies
their own work).
## Link Index
Progressive disclosure — open only what the current stage needs.
| Topic | Links |
|---|---|
| Start | `tutorials/quickstart`, `tutorials/first-workflow`, `how-to/create-workflow` |
| Execute | `how-to/run-on-cluster`, `commands/run`, `commands/cluster`, `commands/resume`, `commands/status` |
| QC | `commands/validate`, `commands/dry-run`, `commands/lint`, `commands/debug`, `commands/graph`, `how-to/troubleshooting` |
| Delivery | `how-to/generate-reports`, `commands/report`, `commands/provenance`, `commands/export`, `reference/versioning`, `reference/reporting-system` |
| Reference | `reference/architecture`, `gallery/`, `tutorials/custom-scripts`, `tutorials/environment-management` |
| Data safety (read early) | `how-to/agent-safety`, `reference/security`, `reference/workflow-format` (`[engine] protected_data_trees`, `tmp_dir`) |
| Knowledge bases (raw) | `crates/oxo-flow-ai/src/knowledge/bioconda_tools.jsonl`, `skills_index.jsonl`, `nfcore_modules.jsonl`, `pipeline_graph.jsonl` |
Docs links resolve under `https://traitome.github.io/oxo-flow/latest/<path>`
(mike serves each version under `/latest/`, `/dev/`, or `/<tag>/`); raw file
links under
`https://raw.githubusercontent.com/Traitome/oxo-flow/main/<path>`.