bio-prefect-dask-nextflow · diff
git:20260202.b63f50a to git:20260206.ca21f07
38 added, 67 removed. Audit A to A.
---
name: bio-prefect-dask-nextflow
- description: Designs and scaffolds bioinformatics pipelines using Prefect (Python) with Dask for local/distributed task execution and Nextflow for HPC scheduler-native execution. Use when an agent must choose between Prefect+Dask vs Nextflow, generate runnable project skeletons, or adapt workflows for laptops, workstations, and HPC clusters (e.g., Slurm/PBS) with reproducibility, caching/resume, and resource-aware configuration.
+ description: Design and scaffold bioinformatics pipelines using Prefect+Dask for local/distributed execution or Nextflow for HPC schedulers.
---
# Bio Prefect + Dask + Nextflow
- This skill helps an agent design, scaffold, and harden bioinformatics pipelines across:
- - **Local workstation/laptop** (Prefect + Dask LocalCluster)
- - **HPC clusters** (Nextflow executors; or Prefect → Slurm worker patterns)
- - **Hybrid** patterns (Prefect orchestrates metadata/approvals/notifications; Nextflow runs heavy compute)
+ Choose and scaffold the right workflow engine for local, distributed, or HPC bioinformatics pipelines.
- ## Use this skill when
- - The user mentions **Prefect**, **Dask**, **Nextflow**, **HPC**, **Slurm**, **PBS**, **nf-core**, or “bioinformatics pipeline”.
- - The user needs **parallelism**, **distributed execution**, **retries**, **scheduling**, **reproducible runs**, or “local prototype then scale”.
+ ## Instructions
- ## Outputs this skill should produce
- When activated, the agent should return (and/or generate in a repo):
- 1. **Engine choice**: `prefect+dask`, `nextflow`, or `hybrid`, with rationale.
- 2. **Runnable scaffold** (files + commands) for the chosen engine.
- 3. **Resource plan** per step (cpus/mem/time) + I/O layout (scratch vs shared).
- 4. **Validation plan**: tiny test run + failure/retry + resume test.
- 5. **Pitfalls & mitigations**: what will likely break on HPC and why.
+ 1. Collect requirements (scheduler, container policy, data location, scale).
+ 2. Choose engine: Prefect+Dask, Nextflow, or Hybrid.
+ 3. Generate a runnable scaffold with clear data layout and resources.
+ 4. Validate with a small test and resume/retry checks.
- ## 2-minute decision
- Use the decision matrix for nuance: [decision-matrix.md](decision-matrix.md)
+ ## Quick Reference
- Default heuristics:
- - Choose **Nextflow** if the pipeline is mainly **CLI tools over files**, must run on **HPC schedulers**, and reproducibility/caching are top priorities.
- - Choose **Prefect + Dask** if the pipeline is mainly **Python functions**, needs **dynamic branching**, API/DB integration, or interactive development.
- - Choose **Hybrid** if Prefect should own the “outer loop” (metadata, batching, approvals, notifications) while Nextflow owns “inner loop” compute.
+ | Task | Action |
+ |------|--------|
+ | Engine choice | See `decision-matrix.md` |
+ | Prefect+Dask scaffold | See `prefect-dask.md` |
+ | Prefect on Slurm | See `prefect-hpc-slurm.md` |
+ | Nextflow on HPC | See `nextflow-hpc.md` |
+ | Examples | See `examples.md` |
- ## Standard workflow (agent playbook)
- 1. **Requirements intake**
- - Scheduler type (Slurm/PBS/LSF/etc), queue/partition rules, walltime limits, node topology.
- - Container policy (Docker vs Singularity/Apptainer vs no containers) and module/conda availability.
- - Data location and throughput constraints (shared FS vs scratch, object storage).
- - Parallelism shape (many independent samples? big distributed arrays? long single jobs?).
- 2. **Choose engine** using [decision-matrix.md](decision-matrix.md); state assumptions.
- 3. **Scaffold the project**
- - Prefect path → [prefect-dask.md](prefect-dask.md) and (if needed) [prefect-hpc-slurm.md](prefect-hpc-slurm.md)
- - Nextflow path → [nextflow-hpc.md](nextflow-hpc.md)
- 4. **Implement steps with replayable boundaries**
- - Each step idempotent; deterministic output paths.
- - Pass around *paths/URIs*, not giant in-memory objects.
- 5. **Add operational glue**
- - Logging, retries/timeouts, resource hints, output manifest.
- 6. **Validate locally**
- - Tiny dataset run + forced failure + resume/retry test.
- 7. **Scale to HPC**
- - Confirm filesystem layout, job submission permissions, and environment bootstrap.
+ ## Input Requirements
- ## Response template
- Use this template in your final answer to the user:
+ - Workflow requirements and steps
+ - Target environment (local, cluster, cloud)
+ - Scheduler and container constraints
+ - Data locations and expected volumes
- ```markdown
- # Pipeline plan: [name]
+ ## Output
- ## Recommended engine
- - Choice: [prefect+dask | nextflow | hybrid]
- - Why: [3–6 bullet rationale]
+ - Engine recommendation with rationale
+ - Runnable scaffold (files + commands)
+ - Resource plan per step
+ - Validation plan and checkpoints
- ## Project scaffold
- - Files to create:
- - ...
- - Commands to run:
- - ...
+ ## Quality Gates
- ## Execution model
- - Parallelism strategy:
- - Resource plan (per step):
- - Data layout (work/results/cache):
+ - [ ] Tiny test run completes end-to-end
+ - [ ] Resume/retry behavior verified
+ - [ ] Resource plan matches cluster limits
- ## Pitfalls & mitigations
- - ...
+ ## Examples
- ## Validation checklist
- - ...
+ ### Example 1: Engine recommendation
+
+ ```text
+ Choice: Nextflow
+ Why: CLI-heavy pipeline, HPC scheduler required, reproducible cache/resume needed.
```
- ## Deep references (read as needed)
- - Engine comparison and “when to use what”: [decision-matrix.md](decision-matrix.md)
- - Prefect + Dask local patterns: [prefect-dask.md](prefect-dask.md)
- - Prefect on Slurm + Dask-on-HPC options: [prefect-hpc-slurm.md](prefect-hpc-slurm.md)
- - Nextflow on HPC (executors, modules, resume/cache): [nextflow-hpc.md](nextflow-hpc.md)
- - Examples (Prefect-only, Nextflow-only, Hybrid): [examples.md](examples.md)
- - Validation loop + common failure modes: [validation-checklist.md](validation-checklist.md)
+ ## Troubleshooting
+ **Issue**: Workflow fails on HPC due to environment mismatch
+ **Solution**: Pin container/conda versions and validate with a minimal test dataset.