safactory-workflows · diff
git:20260803.3ac199f to git:20260921.9e21b6e
42 added, 46 removed. Audit A to A.
---
name: safactory-workflows
- description: Use this skill when helping users onboard a benchmark or environment into SAfactory, run SAfactory Docker-mode evaluation, or start SAfactory GRPO/RL training. It guides Codex to inspect the SAfactory repository, read the relevant docs, follow the established Geo3K baseline and my_env placeholder workflows, configure Gateway/model routes safely, and run or prepare launcher/RL commands without leaking private credentials.
+ description: Integrate a benchmark or custom environment into SAfactory using fixed adapter templates and local contract tests, optionally run Docker/RJob evaluation, or prepare GRPO/RL training. Use for SAfactory onboarding and runtime workflows.
---
# SAfactory Workflows
- This skill helps agents operate the SAfactory repository for three user intents:
+ For environment onboarding, read [references/environment-integration.md](references/environment-integration.md) and `docs/guides/custom-environment.md` (or its Chinese translation). The guide defines the runtime contract; the [assets/environment/](assets/environment/) templates implement it. `env/geo3k/` is only an example of multi-turn agent interaction, not a layout to copy — the standard reference is `env/prmeval/`.
- 1. Add a benchmark or custom environment to SAfactory.
- 2. Run Docker-mode evaluation for an environment.
- 3. Start GRPO/RL training for an environment.
+ For evaluation requests read [references/docker-evaluation.md](references/docker-evaluation.md). For GRPO/RL requests read [references/grpo-training.md](references/grpo-training.md). Read `docs/internal/rjob-mode.md` or its Chinese translation only when working on RJob deployment.
- Keep the root README and `docs/` as the source of truth. Use this skill to choose the right workflow, files, and checks; do not duplicate or rewrite full documentation.
+ ## Scope and intake
- ## First Steps
+ Infer available values from the request and benchmark source before asking:
- From the repository root:
+ - environment name, source/check-out path, dataset row schema, and 1–2 representative cases;
+ - native single-case command and native output location, when wrapping a benchmark;
+ - target deployment mode (`docker` or `rjob`) and image, when known;
+ - whether evaluation is requested; **default to integration only**.
- 1. Inspect the current layout before assuming paths: `rg --files README* docs env rl gateway`.
- 2. Identify the user intent:
- - Benchmark/environment onboarding: read [references/environment-integration.md](references/environment-integration.md).
- - Docker evaluation: read [references/docker-evaluation.md](references/docker-evaluation.md).
- - GRPO/RL training: read [references/grpo-training.md](references/grpo-training.md).
- 3. Load only the matching reference file and the linked SAfactory docs needed for the task.
- 4. Prefer existing patterns in `env/geo3k/`, current docs, and root README commands.
+ Only evaluation or training requires the native score field, scale, and pass condition. Do not block integration on missing rewards, evaluator code, a running Gateway, or internal cluster access. If mode is unknown, implement and test the shared adapter first; clarify mode before filling deployment values. Ask about an unknown native command or output format before implementing the dependent hook, while continuing the protocol scaffold and tests.
- ## Repository Sources
+ The adapter runs one dataset row through the existing native harness. It does not reimplement the benchmark's case-solving/scoring logic or repair image internals unless that work is requested. Preserve native scores in metrics when available; their presence does not enable evaluation.
- Use these documents as canonical sources:
+ ## Template-based implementation
- - Root quick start: `README.md` or `README_CN.md`.
- - Custom environment guide: `docs/guides/custom-environment.md` or `docs/guides/custom-environment_CN.md`.
- - Evaluation guide: `docs/guides/evaluation.md` or `docs/guides/evaluation_CN.md`.
- - RL guide: `docs/guides/rl-training.md` or `docs/guides/rl-training_CN.md`.
- - Gateway reference: `docs/reference/gateway.md` or `docs/reference/gateway_CN.md`.
- - Environment reference: `docs/reference/environments.md` or `docs/reference/environments_CN.md`.
- - CLI/config reference: `docs/reference/configuration.md` or `docs/reference/configuration_CN.md`.
+ 1. Scaffold the fixed file set with `scripts/scaffold_environment.py <name>` (both Docker and RJob config pairs are always emitted; append `--enable-evaluation` only when scoring is requested). For an existing directory, adapt the relevant templates without overwriting user files.
+ 2. Keep the protocol shell in `runner.py` fixed — it is a standard part shared with `env/prmeval`; benchmark logic goes only in `adapter.py:run_case`. The generated greeting example only verifies the scaffold; replace it for a real integration.
+ 3. Fill the task/start YAML templates with the image, dataset, workdir, mounts, and dependencies. `agent_name` must equal `env_name`; one row is one episode; `env_params.results_root` must equal the results mount target; absolute paths in dataset rows must point under a mount target.
+ 4. Create `rule_evaluator.py` only when evaluation is requested. Fill its `score_metrics` hook with the agreed normalization; keep the evaluator interface and failure handling fixed. Do not rerun cases in the evaluator or register its path in YAML. **With `--enable-evaluation` set, a missing evaluator marks every episode as failed** — omit the flag for integration-only runs.
+ 5. Python is the default template language. If the native runtime needs Node/shell, retain a thin Python wrapper when possible. If another runner language is required, port the same protocol shell and document why; keep native logic separate and cover the same contract tests.
- Use Chinese docs when the user writes Chinese; otherwise use English docs.
+ Change shared shell behavior only for an actual runtime-contract requirement, and update the template and its tests together. New files should come from templates, not a fresh implementation of the protocol.
- ## Operational Rules
+ ## Standard reference: `env/prmeval`
- - Treat `env/geo3k/` as the standard reference implementation, not as a hardcoded target.
- - Use generic placeholders such as `my_env` or the user's requested environment name for new workflows.
- - Do not write private model endpoints, API keys, or internal route names into committed docs or shared configs.
- - Do not overwrite `gateway/config.local.yaml` without checking its existing contents and preserving user edits.
- - Ensure `--llm-model` and `RL_MODEL` match a Gateway `llm_routes` key.
- - Keep Launcher, Gateway, and Buffer Server storage pointing at the same backend or SQLite URI.
- - For new environments, run a minimal Docker evaluation before recommending RL training.
- - Read `docs/internal/` only when the user explicitly asks about RJob, Sandbox, or internal deployment modes.
+ When an existing benchmark is supplied, compare its integration against `env/prmeval/` before inventing a new layout. The reference keeps the fixed `runner.py` protocol shell separate from `adapter.py`, uses one dataset row per episode, and keeps the optional score mapping in `rule_evaluator.py`.
- ## When Information Is Missing
+ The Docker pair (`*_config.yaml`, `*_start.yaml`) is the base contract. The RJob pair (`*_config.rjob.yaml`, `*_start.rjob.yaml`) repeats task parameters but adds only cluster image, resources, embedded files, and cluster-accessible mounts. `env_params` is delivered in the request JSON; static `container.env` or `rjob.env` values are not a replacement for it.
- Ask the smallest set of concrete questions required to proceed. Common blockers:
+ ## Validation and completion
- - New benchmark source path, dataset path, or native run command is unknown.
- - Desired SAfactory environment name is unknown.
- - Model route key or Gateway endpoint is unknown for an actual run.
- - Docker image, runtime dependencies, or scoring contract is unknown.
+ Verify with one command and iterate until green, then add stages as access permits:
- If the user asks only for commands and prerequisites are missing, provide a fill-in command with placeholders and state exactly what must be replaced.
+ ```bash
+ # Static + contract stages (default; no Gateway, Docker, model credentials, or cluster):
+ python skills/safactory-workflows/scripts/check_environment.py --env env/<name>
+ ```
- ## Verification Expectations
+ 1. **Static checks:** file set, naming, and config ↔ start ↔ dataset invariants. Every finding is labeled `[config]`, `[env]`, or `[safactory]` — fix by that ownership, starting with `[config]` errors.
+ 2. **Local contract check:** the real `adapter.py` runs against an owned mock model endpoint. Native dependencies must be available locally; otherwise pass `--fixture-adapter` to verify the protocol shell only and state that native behavior was not validated.
+ 3. **Live deployment check (opt-in via `--live`):** starts Gateway, waits for readiness, runs Launcher, cleans up. Do not require a user to keep a separate terminal open. If an existing Gateway is used, verify it and invoke Launcher directly.
+ 4. **Evaluation check (opt-in):** validate native score conversion and final reward only when evaluation/training is in scope. Pass `--enable-evaluation` explicitly; omit it for integration-only runs. `total_reward: 0.0` satisfies the ungraded runtime result contract and is not an evaluation outcome.
- - For file edits: check created paths with `rg --files` and inspect changed files.
- - For evaluation: run the smallest feasible `launcher.py --mode docker --enable-evaluation` smoke test when credentials, image, and data are available.
- - For RL: first verify or ask for evidence that Docker evaluation passes; then prepare `env.sh`; start long-running training processes only when the user requested execution.
- - If a command cannot be run because credentials, Docker, data, or dependencies are unavailable, report the blocker precisely and leave the exact next command.
+ When something fails, classify it first with the triage table in the integration reference (symptom → owning side → first action) instead of debugging blind; a geo3k live baseline helps isolate shared-infrastructure (safactory-side) problems.
+
+ Report the adapter/config changes, the cases and fixtures used, which validation level passed, and remaining deployment prerequisites. Local tests can complete adapter validation without an internal cluster; do not claim they validate image pulls, mounts, real Gateway persistence, or RJob scheduling. If a requested live check is blocked, complete independent local checks and report the specific blocker and next command.
+
+ ## Repository rules
+
+ - Preserve the user's selected deployment mode. Local contract testing does not substitute Docker deployment for RJob deployment.
+ - RJob needs the same runner, plus `<name>_config.rjob.yaml` and `<name>_start.rjob.yaml`. The evaluator remains optional. Use cluster-accessible images/storage and `rjob.mount_config`/`rjob.mount`; include all runner dependencies in `rjob.embedded_files`; replace `CLUSTER_STORAGE` placeholders before live runs. Do not use loopback Gateway addresses for live cluster runs.
+ - Keep Launcher, Gateway, and Buffer Server on the same storage backend/SQLite URI.
+ - Preserve local Gateway configuration and never commit private endpoints or credentials.
+ - Establish a working reward path before starting RL; this requirement does not apply to integration-only work.