huawei-cloud-skill-tester · diff
git:20260807.e6eafec to git:20260818.ab85690
30 added, 705 removed. Audit A to A.
---
name: huawei-cloud-skill-tester
description: |
End-to-end functional testing framework for Huawei Cloud skills — three-tier pipeline covering
single-skill unit testing, multi-skill orchestration, and end-to-end full flow testing.
Each phase produces structured JSON output with chain verification.
Supports skill installation validation, functional analysis, CLI→SDK→API feasibility research,
test case generation, real-environment execution with resource lifecycle, resource cleanup,
multi-skill scenario orchestration, trigger-conflict detection, and consolidated reporting.
Triggers include: 测试技能, 执行技能测试, 跑测试流程, 技能回归测试,
skill test, run skill tests, test huawei cloud skill, verify skill,
测试华为云skill, 全流程测试, 编排测试, 技能完整性检查,
skill-tester, 跑测试, 回归测试, 组合测试, 多skill编排, verification, e2e.
tags: [huawei-cloud, testing, e2e, orchestration, qa]
---
# Huawei Cloud Skill Tester — Three-Track Eight-Phase E2E Testing Pipeline
> Independent, repeatable Huawei Cloud Skill testing framework. Does not depend on skill-creator; can test any existing Huawei Cloud Skill.
> Focuses on **real-environment functional testing** and **multi-skill orchestration scenarios**.
---
## Overview
This Skill provides a **three-track, eight-phase** standardized testing pipeline
(three-tier layout × 8 phase total: Phase 0~6 执行 + Phase 7 终报告):
| Tier | Phases | Goal |
|------|--------|------|
| **Tier 1: Single-Skill Unit Testing** | Phase 0~4 | Verify each skill item by item: installation, feature extraction, technical research, test case generation, execution |
| **Tier 2: Integration Testing** | Phase 5~6 | Multi-skill orchestration scenario derivation + end-to-end real-environment flow verification |
| **Tier 3: Final Report** | Phase 7 | Consolidated report merging all phase outputs |
### Core Design Principles
1. **Chain Verification** — Before each Phase, check that the previous phase's JSON exists; if missing, refuse to execute
2. **Agent-proof** — Write operations require user confirmation for each item; automatic gate bypassing is not allowed
3. **Three-Track Layering** — Clear gates between Tiers; Tier 1 must be completed before entering Tier 2
4. **Batch Repeatable** — Supports `--skills "skill-a,skill-b"` or `--all-installed`
5. **Fallback Strategy** — When only 1 skill, Phase 5/6 automatically downgrade to single-skill lifecycle testing. **Sibling auto-scan is ON by default** (Phase 5/6 自动从被测 skill 同目录找其他 huawei-cloud-* skill 做编排组合测试). Use `--no-siblings` to opt-out.
6. **Standardized JSON Output** — All phases output in a unified schema; Phase 7 merges into a single report
7. **Real-Environment First** — All Tier 2 orchestrations execute against real Huawei Cloud; no mocks or simulations
### Data Flow Diagram
```
User Input (--skills or --all-installed)
│
├── Tier 1 ──── Iterate over each skill ────
│ Phase 0 → 1 → 2 → 3 → 4
│ (phase-N-summary.json chain validation)
│
├── Tier 2 ──── Integration ────
│ Phase 5 (orchestration scenario derivation + real-environment execution)
│ Phase 6 (e2e full-flow: create→query→update→delete lifecycle)
│ Only 1 skill → Downgrade single-skill closed loop
│
└── Tier 3 ──── Final ────
Phase 7 (merge phase-0~6 JSON into consolidated report)
```
---
## Prerequisites
1. **hcloud CLI** installed and authenticated (for Tier 2 CLI mode testing) — Reference: https://support.huaweicloud.com/qs-hcli/hcli_02_003.html
2. **Python 3.8+** + `huaweicloudsdk` packages (for SDK mode testing) — SDK Reference: https://console.huaweicloud.com/apiexplorer/#/sdkcenter
- 3. **Huawei Cloud AK/SK** — 自动扫描所有以 `HUAWEI` / `HW` / `HWC` 开头的环境变量，匹配其中含 `ACCESS_KEY` / `_AK` / `SECRET_KEY` / `_SK` 的键值对。**If missing, the framework emits the env-var setup template to stderr and exits 77. NEVER ask the user to type AK/SK in chat; user must set env vars in their shell profile out-of-band and re-run.**
+ 3. **Huawei Cloud AK/SK** — 自动扫描所有以 `HUAWEI` / `HW` / `HWC` 开头的环境变量（键名含 AK/SK 标记的键值对）。**If missing, the framework emits the env-var setup template to stderr and exits 77. NEVER ask the user to type AK/SK in chat; user must set env vars in their shell profile out-of-band and re-run.**（完整协议见 `references/agent-protocol.md`）
4. **Target Skill** must be under `$SKILL_INSTALL_DIR/` (auto-detected: `~/.agents/skills/` → `~/.hermes/skills/` → default) or a user-specified path
5. **jq** command (all JSON processing depends on it)
6. **API Reference**: https://console.huaweicloud.com/apiexplorer/#/openapi
---
## Workflow — Three-Track Eight-Phase
```
Tier 1: Single-Skill Unit Testing
Phase 0: Installation Verification (install/uninstall/reinstall)
- Phase 1: Feature Extraction (metadata + commands + resource types)
+ Phase 1: Feature Extraction (metadata + commands + resource types + doc checks)
Phase 2: Technical Research (CLI→SDK→API three-level availability)
- Phase 3: Test Case Generation (functional cases TC-F + API cases TC-A)
- Phase 4: Real-Environment Execution (read-only automatic + write operations require confirmation)
+ Phase 3: Test Case Generation (functional TC-F + boundary + negative + API TC-A)
+ Phase 4: Real-Environment Execution (read-only automatic + write ops require confirmation)
Tier 2: Integration Testing — Real-Environment Orchestration
- Phase 5: Multi-Skill Orchestration (scenario derivation → step execution → state verification)
+ Phase 5: Multi-Skill Orchestration (conflict scan / scenario derivation / self-check)
Phase 6: End-to-End Flow (resource lifecycle: create→query→update→delete)
Tier 3: Final Report
- Phase 7: Consolidated Report (merge phase-0~6 JSON into single report)
- ```
-
- ---
-
- ### Phase 0: Installation Verification
-
- **Goal:** Verify whether the skill can be installed/uninstalled/reinstalled normally, and confirm the directory structure meets the contract.
-
- **Steps:**
-
- 1. Check skill directory structure — **all four must exist** (this is a hard contract; missing any one fails the phase):
- - `SKILL.md`
- - `scripts/`
- - `references/`
- - `references/iam-policies.md`
- 2. Confirm whether the runtime has the skill installed (check if a directory with the same name exists under `$SKILL_INSTALL_DIR/`)
- 3. Perform installation verification (simulated or actual installation)
- 4. Perform uninstallation verification (simulated or actual uninstallation)
- 5. Perform reinstallation verification (install → uninstall → install)
- 6. Record installation duration, directory integrity, and installation status
-
- ```bash
- # Directory integrity check (all four are REQUIRED)
- [ -f "${skill_path}/SKILL.md" ] && echo "SKILL.md exists"
- [ -d "${skill_path}/scripts" ] && echo "scripts/ exists"
- [ -d "${skill_path}/references" ] && echo "references/ exists"
- [ -f "${skill_path}/references/iam-policies.md" ] && echo "iam-policies.md exists"
-
- # Installation status
- [ -d "${SKILL_INSTALL_DIR:-$HOME/.agents/skills}/${skill_name}" ] && echo "installed" || echo "not installed"
- ```
-
- **Output:** `phase-0-summary.json`
-
- ```json
- {
- "install": {"status": "pass", "existing": true, "duration_s": 1.2},
- "uninstall": {"status": "skipped", "reason": "not installed"},
- "reinstall": {"status": "skipped", "reason": "not installed"},
- "directory_integrity": {"pass": true, "checks": {...}}
- }
- ```
-
- ---
-
- ### Phase 1: Feature Extraction
-
- **Goal:** Extract structured feature information from SKILL.md as input for all subsequent phases.
-
- **Steps:**
-
- 1. Read YAML frontmatter from SKILL.md → name, description, tags, triggers
- 2. Extract core commands table (Core Commands section)
- 3. Extract parameter confirmation table
- 4. Identify feature types (query/create/modify/delete)
- 5. Extract resource types involved (ECS instance, VPC, voucher, etc.)
- 6. Note whether there are write operations (Create/Update/Delete)
- 7. Read the list of test scripts under scripts/
- 8. Read the list of reference files under references/
-
- **Output:** `phase-1-summary.json`
-
- ```json
- {
- "metadata": {
- "name": "huawei-cloud-bss-voucher-manage",
- "triggers": ["查代金券", "删除代金券", "list vouchers", ...],
- "tags": ["huawei-cloud", "bss", "voucher"]
- },
- "capabilities": {
- "list": ["查询代金券", "统计代金券"],
- "create": [],
- "update": [],
- "delete": ["删除代金券"]
- },
- "has_write_operations": true,
- "resource_types": ["bss_voucher"],
- "commands": [
- {"id": "CMD-01", "source": "SKILL.md", "description": "查询代金券列表", "executor": "sdk"},
- {"id": "CMD-02", "source": "SKILL.md", "description": "统计代金券", "executor": "sdk"},
- {"id": "CMD-03", "source": "SKILL.md", "description": "删除代金券", "executor": "sdk"}
- ],
- "scripts": ["scripts/test-cli-commands.sh"],
- "references": ["references/iam-policies.md", "references/api-paths.md"]
- }
- ```
-
- ---
-
- ### Phase 2: Technical Research
-
- **Goal:** Perform CLI→SDK→API three-level fallback verification for each command extracted in Phase 1, determining the actual executable method.
-
- **Dependency:** Phase 1 completed (phase-1-summary.json exists)
-
- **Research order (per command):**
-
- | Priority | Method | Verification |
- |----------|--------|-------------|
- | 1st | **CLI** | `hcloud <Service> <Operation> --cli-region=cn-north-4 --help` |
- | 2nd | **SDK** | `python3 -c "from huaweicloudsdk{service}.v2 import ..."` |
- | 3rd | **API** | Only from SDK source `_http_info` or Huawei Cloud API Explorer |
-
- **Rule:** API endpoints are **strictly prohibited from being inferred**; only allowed from SDK `_http_info.resource_path` or user confirmation from API Explorer.
-
- **Output:** `phase-2-summary.json`
-
- ```json
- {
- "research": [
- {
- "cmd_id": "CMD-01",
- "description": "查询代金券列表",
- "cli": {"available": false, "reason": "BSS not in hcloud service list"},
- "sdk": {
- "available": true,
- "package": "huaweicloudsdkbss.v2",
- "method": "list_sub_customer_coupons",
- "api_path": "/v2/promotions/benefits/coupons"
- },
- "api": {"available": true, "endpoint": "/v2/promotions/benefits/coupons"},
- "recommended_executor": "sdk",
- "risk_level": "low"
- }
- ]
- }
- ```
-
- ---
-
- ### Phase 3: Test Case Generation
-
- **Goal:** Generate two types of test cases based on Phase 1+2, present them to the user for confirmation.
-
- **Dependency:** Phase 1+2 completed (phase-1-summary.json + phase-2-summary.json exist)
-
- **Functional case division rules:**
-
- | Operation Type | Case Requirements | Risk Level |
- |---------------|-------------------|------------|
- | Query (List/Show/Get) | 1 positive + 1 boundary (limit=0 or empty filter) | low |
- | Create | 1 standard + 1 parameter variant | high |
- | Update | 1 verification that modification took effect | medium |
- | Delete | 1 pre-deletion confirmation + 1 post-deletion verification | high |
- | Statistics (count/aggregate) | 1 positive + 1 time range boundary | low |
-
- **Case IDs:** `TC-F-01` ~ `TC-F-NN` (functional), `TC-A-01` ~ `TC-A-NN` (API/SDK)
-
- **Output:** `phase-3-summary.json`
-
- ```json
- {
- "functional_cases": [
- {
- "id": "TC-F-01",
- "name": "List vouchers - positive",
- "command": "list_sub_customer_coupons(limit=10)",
- "expected": "Return voucher list, no more than 10 items",
- "is_write": false,
- "risk_level": "low",
- "executor": "sdk",
- "prerequisites": [],
- "verification": "resp.count >= 0"
- },
- {
- "id": "TC-F-03",
- "name": "Delete voucher - positive",
- "command": "reclaim_partner_coupons(coupon_id=...)",
- "expected": "Voucher status changed to reclaimed",
- "is_write": true,
- "risk_level": "high",
- "executor": "sdk",
- "prerequisites": ["TC-F-01 (provide valid coupon_id)"],
- "verification": "Query after delete to confirm status change"
- }
- ],
- "api_cases": [...]
- }
- ```
-
- ---
-
- ### Phase 4: Execution
-
- **Goal:** Execute test cases from Phase 3 one by one; read-only runs automatically, write operations are gated.
-
- **Dependency:** Phase 3 completed (phase-3-summary.json exists)
-
- **AK/SK resolution (two-tier, NO interactive prompt):**
-
- ```
- 1. Environment variables (any HUAWEI*/HW*/HWC* prefixed *_AK / *_SK / *_ACCESS_KEY / *_SECRET_KEY)
- 2. hcloud CLI config (~/.hcloud/config.json profile, mode = $HCLOUD_PROFILE_MODE)
-
- If neither source yields credentials, the framework emits the env-var setup
- template to stderr and exits 77 — see "Agent Protocol" below.
- ```
-
- **Execution rules:**
-
- ```
- Pre-check: Is AK/SK set?
- if AK/SK obtained from env or hcloud config:
- Continue
- else (no interactive prompt — applies even in TTY):
- Emit env-var setup template to stderr, exit 77 — see "Agent Protocol" below
-
- Per-case gate:
- if is_write == false:
- Auto-execute → Record pass/fail
- if is_write == true:
- if env ALLOW_WRITES == "1":
- Auto-execute under AK/SK credentials → Record pass/fail + resource_changes
- entry.execution_meta.user_confirmed = true
- else (default):
- Skip → status=skip, reason="ALLOW_WRITES=0"
- ```
-
- **Note on "Agent-proof" in non-interactive mode:** Phase 4 runs as a non-interactive
- shell script, so the per-case `y/N` prompt described in early design is replaced by
- the `ALLOW_WRITES=0|1` environment gate. The "user confirmation" that is recorded on
- each write case (`user_confirmed=true` when `ALLOW_WRITES=1`) is the audit trail, not
- an interactive consent flow. To get the original interactive confirm, run the
- pipeline from a TTY-attached agent that intercepts each write case before invocation.
-
- **Resource change record (key fields):**
-
- ```json
- "resource_changes": [
- {
- "tc_id": "TC-F-03",
- "resource_type": "bss_voucher",
- "resource_id": "VCH-abc123",
- "change_type": "deleted",
- "cleanup_method": {"type": "sdk", "command": "already deleted, no cleanup needed"},
- "cleanup_required": false
- }
- ]
- ```
-
- **Output:** `phase-4-summary.json`
-
- ```json
- {
- "execution_results": [
- {"tc_id": "TC-F-01", "status": "pass", "duration_s": 2.1, "output_snippet": "..."},
- {"tc_id": "TC-F-03", "status": "pass", "duration_s": 1.5, "output_snippet": "...",
- "resource_changes": [{"resource_type": "bss_voucher", "change_type": "deleted", ...}]}
- ],
- "statistics": {"total": 10, "pass": 9, "fail": 0, "skip": 1},
- "all_resources_changed": [...]
- }
- ```
-
- ---
-
- ### Phase 5: Multi-Skill Orchestration — Trigger Conflict & Data-Flow Scan
-
- **Goal:** Detect cross-skill routing conflicts (overlapping triggers that would confuse an Agent router) and identify potential data-flow paths between skills, so integration issues are caught **before** any real resources are touched.
-
- **Dependency:** Phase 4 completed for all tested skills
-
- > **What Phase 5 does NOT do:** It does NOT execute real multi-skill business scenarios
- > (create→query→delete chains) against the live environment. Scenario execution and
- > rollback live in Phase 6. Phase 5 is a static, offline analysis over Phase 1 metadata.
-
- **Sibling discovery (default ON):**
-
- ```
- Phase 5/6 默认会自动从被测 skill 的同级目录找其他 huawei-cloud-* skill 做编排组合测试。
- Opt-out: --no-siblings 或 WITHOUT_SIBLINGS=1 或 SIBLING_LIMIT=0
- Opt-tune: --sibling-limit N (默认 5 个兄弟)
-
- 排除规则:
- - 自己 (target skill)
- - *-test-files (测试 artifacts 目录)
- - huawei-cloud-skill-tester, huawei-cloud-skill-creator, huawei-cloud-new-tester (meta skills)
- - 缺 SKILL.md 的目录
- ```
-
- **SKILL.md fallback (兄弟 skill 没 phase-1 时):**
-
- ```
- 当兄弟 skill 没有 phase-1-summary.json (因为没单独跑过 phase 1), Phase 5 实时解析
- 它的 SKILL.md frontmatter 提取 triggers / resource_types, 支持三种格式:
- 1. `triggers: [a, b, c]` (inline list)
- 2. `triggers:\n - a\n - b` (block list)
- 3. `description: | ... Triggers include: "x","y","z"` (embedded in description)
- ```
-
- **Branch logic:**
-
- ```
- if skills_count == 1 (含 --no-siblings):
- Downgrade to single-skill self-check ("downgraded_self_check"):
- - Scan the skill's own triggers for substring overlaps (internal_ambiguities)
- - Scan the skill's own write commands for ordering hints (cycle_warnings)
- - Skip data_flow_tests and parallel_load_test
-
- if skills_count >= 2:
- 5a: Trigger conflict scan
- - Pairwise compare all triggers across all skills
- - "high" severity = exact match (Agent will route ambiguously)
- - "medium" severity = substring containment (one is contained in the other)
- 5b: Data-flow identification
- - For each skill A's resource_types, check if any skill B's command
- description references that type → emit DF-NN candidate tests
- - These are *identified* (status="identified"), not *executed*
- 5c: Parallel load test
- - Re-parse each SKILL.md's YAML frontmatter (validates the skill can be
- loaded side-by-side with siblings in the same Agent)
- 5d: No rollback
- - Phase 5 makes zero API calls. No resources to roll back.
- ```
-
- **Auto-derivation example (only the conflict scan is real, the rest is metadata):**
-
- ```
- Input skills: [ECS-manage, VPC-manage, EIP-manage]
- Output:
- conflict_scan.conflicts: [
- {"severity":"medium","skill_a":"ecs-manage","skill_b":"vpc-manage",
- "trigger":"vpc ↔ vpc-list", "recommendation":"改其中一个触发词"}
- ]
- data_flow_tests: [
- {"test_id":"DF-01","from_skill":"vpc-manage","to_skill":"ecs-manage",
- "data_item":"vpc","status":"identified",
- "detail":"vpc-manage 的输出 vpc_id 可能作为 ecs-manage 的输入"}
- ]
- parallel_load_test: {"verdict":"pass","detail":"3 个 skill 均可解析"}
- ```
-
- **Output:** `phase-5-summary.json`
-
- ```json
- {
- "mode": "full" | "downgraded_self_check",
- "conflict_scan": {
- "pairs_checked": <int>,
- "conflicts": [
- {"severity": "high|medium|low", "skill_a": "<>", "skill_b": "<>", "trigger": "<>", "recommendation": "<>"}
- ],
- "no_conflict_pairs": <int>
- },
- "data_flow_tests": [
- {"test_id": "DF-01", "from_skill": "<>", "to_skill": "<>", "data_item": "<>", "status": "identified", "detail": "<>"}
- ],
- "parallel_load_test": {
- "skills_loaded": ["<skill>", ...],
- "verdict": "pass | fail | skipped",
- "detail": "<>"
- },
- "cleanup": {"resources_cleaned": 0, "resources_failed": 0}
- }
- ```
-
- ---
-
- ### Phase 6: End-to-End Flow Testing — Real-Environment Lifecycle
-
- **Goal:** End-to-end verification of complete resource lifecycles against real Huawei Cloud, automatically deriving scenario chains from Phase 1 feature lists and executing them.
-
- **Dependency:** Phase 5 completed (phase-5-summary.json exists)
-
- > **Sibling discovery:** Phase 6 与 Phase 5 共享 `discover_siblings()` 函数（见 § Phase 5）。
- > 默认自动扫同级兄弟 skill，opt-out 用 `--no-siblings`。
-
- **Branch logic:**
-
- ```
- if skills_count == 1 (含 --no-siblings):
- Single-skill closed loop ("downgraded_single_skill_flow"):
- - Sort all feature points by list → create → update → delete
- - Chain into a single-skill resource lifecycle
- - Execute each step via real SDK/CLI against the live environment
- - Read-only steps auto-execute; write steps gated by ALLOW_WRITES=0|1
- (default 0, skipped with status=skip in the report)
- - Verify resource state after each mutation
- - AK/SK credentials required (3-tier resolution, see § Phase 4)
-
- if skills_count >= 2:
- 6a: Scenario derivation (from Phase 1 capabilities + SKILL.md fallback)
- - Group by create→query→delete across skills
- - Steps default to status=pass once derived (see *Implementation status* below)
- 6b: Lifecycle execution — single-skill branch only
- - The multi-skill branch currently derives the step chain but does NOT
- invoke live APIs step-by-step; it records the derived scenario for
- review. The per-step `status: pass` on derivation is a plan marker,
- not an execution result.
- 6c: State consistency verification
- - Single-skill branch: real post-step checks via SDK output
- - Multi-skill branch: derived-only, no real verification
- 6d: Cleanup verification
- - Single-skill branch: per-step resource_changes tracked
- - Multi-skill branch: cleanup.verdict defaults to "pass" (no real cleanup)
- ```
-
- **Implementation status (read this before relying on Phase 6 multi-skill):**
-
- | Branch | Derives scenario | Runs against live API | Tracks resource_changes | State consistency check |
- |--------|------------------|-----------------------|-------------------------|--------------------------|
- | Single-skill | ✅ | ✅ (under `ALLOW_WRITES=1`) | ✅ | ✅ |
- | Multi-skill | ✅ | ❌ (currently a plan record; steps marked `pass` on derivation) | ❌ | ❌ (defaults to `pass`) |
-
- If you need real multi-skill E2E execution, treat Phase 6 multi-skill as a
- **derived plan** and run each step's command manually (or via Phase 4 with
- `ALLOW_WRITES=1`), then re-run the report.
-
- **Single-skill closed loop example (RDS query skill):**
-
- ```
- Steps:
- Step 1: ListInstances (read-only, auto) → get instance count
- Step 2: ShowInstanceDetail (if instance exists) → get instance config
- Step 3: ListConfigurations (auto) → get parameter templates
- Step 4: ShowBackupPolicy (if instance exists) → get backup config
- Step 5: Analyze slow SQL (read-only, auto) → ListSlowLogs + ListTopSqls
- Step 6: Parameter tuning recommendation (analysis, no API call)
- Step 7: Backup strategy assessment (analysis, no API call)
- ```
-
- **Multi-skill derived scenario example (currently NOT auto-executed):**
-
- ```
- Derived steps (status=pass on derivation only):
- Step 1: Create EVS volume (from evs-manage)
- Step 2: Attach volume to ECS instance (from ecs-manage)
- Step 3: Query volume metrics (from evs-manage)
- Step 4: Detach volume (from evs-manage)
- Step 5: Delete volume (from evs-manage)
+ Phase 7: Consolidated Report (merge phase-0~6 JSON, verdict + issues + reasons)
```
- **Output:** `phase-6-summary.json`
+ > 各 Phase 的完整实现规范（步骤、判定标准、JSON 字段）见 `references/phase-details.md`。
- ```json
- {
- "mode": "full | downgraded_single_skill_flow",
- "scenario": {
- "name": "<derived>",
- "skills_involved": ["<skill>", ...],
- "description": "<auto-derived>",
- "derived_automatically": true,
- "user_confirmed": false,
- "steps": [
- {"seq": 1, "tc_id": "FF-01", "skill": "<>", "action": "<>",
- "status": "pass | fail | skip", "resource_changes": []}
- ]
- },
- "state_consistency": {
- "pass": true | false,
- "detail": "<>",
- "final_state_summary": "<>"
- },
- "cleanup": {
- "verdict": "pass | partial | fail",
- "resources_cleaned": <int>,
- "resources_failed": <int>,
- "manual_required": [<manual_instruction>]
- }
- }
- ```
---
- ### Phase 7: Consolidated Report
-
- **Goal:** Merge Phase 0~6 JSON outputs into a single, comprehensive test report
- for the user. The report follows a strict **Summary → Per-Phase Detail →
- Attachments** structure so the reader can quickly decide whether to dive in.
-
- **Dependency:** Phase 0~6 all exist (`phase-0-summary.json` through
- `phase-6-summary.json`)
-
- **Output:**
- - `<skill-name>-test-files/reports/report-<YYYYMMDD-HHMMSS>/test-report.json`
- - `<skill-name>-test-files/reports/report-<YYYYMMDD-HHMMSS>/test-report.md`
-
- > **Two files, identical content.** JSON is machine-readable; Markdown is the
- > human-facing version. Both must list every generated test case and every
- > execution result — no hiding behind summary numbers.
-
- #### Report Structure (mandatory)
-
- ```text
- ┌────────────────────────────────────────────────────────────────┐
- │ Header — test_id / generated_at / skills / report_dir │
- ├────────────────────────────────────────────────────────────────┤
- │ §1 Summary (TL;DR) │
- │ - Overall Verdict (✅/⚠️/❌) │
- │ - 指标表 (phases / test cases / pass rate / 资源变更) │
- │ - Key Findings (3-5 条) │
- ├────────────────────────────────────────────────────────────────┤
- │ §2 Per-Phase Detail │
- │ 对每个 skill: │
- │ ### Skill: <name> │
- │ Path: <skill_path> │
- │ 对每个 phase (0..6): │
- │ #### Phase N — <name> │
- │ 阶段小结 + 该阶段产生的用例 / 实际执行结果 │
- ├────────────────────────────────────────────────────────────────┤
- │ §3 Attachments — phase JSONs / archive / reports 路径 │
- └────────────────────────────────────────────────────────────────┘
- ```
-
- #### §1 Summary 规则
-
- `summary.verdict` 由所有 phase 的 verdict 联合判定：
-
- | 条件 | verdict | label |
- |------|---------|-------|
- | 全部 phase pass + test cases pass_rate == 100 | `pass` | `✅ PASS` |
- | 至少 1 个 phase partial/fail，但有 phase pass | `partial` | `⚠️ PARTIAL` |
- | 全部 phase fail / 全部 skipped / 全部 missing | `fail` | `❌ FAIL` |
-
- 指标表必含列：
-
- | Metric | Value |
- |--------|-------|
- | Phases | X pass / Y partial / Z fail / W skipped (of 7) |
- | Test Cases | total N \| pass P \| fail F \| warn W \| skip S \| error E |
- | Pass Rate | P% |
- | Manual Items | N (need real business data) |
- | Cloud Resources Changed | N (Phase 4 only) |
-
- 指标表之后必须紧跟 **Skills Tested 列表**（`# | Skill | Path` 三列），
- 列出本次测试涉及的所有 skill。详情见下文 § "报告硬性要求" 第 2 条。
-
- `key_findings[]` 由脚本自动生成 3-5 条结论，每条形如：
-
- - `huawei-cloud-rds-query: 4 项目录硬要求全部通过 (4/4)`
- - `huawei-cloud-rds-query Phase 4: 12 个用例失败 (查看下方失败归类)`
-
- #### §2 Per-Phase Detail 规则
-
- 每个 phase 小节必须包含「阶段小结」+「该阶段产物（用例 / 结果）」：
-
- | Phase | 必含产物 |
- |-------|----------|
- | 0 | 目录硬要求表（4 行）+ install/uninstall/reinstall 表 |
- | 1 | metadata + 提取的 commands 表 |
- | 2 | CLI/SDK/API 可用性矩阵 + per-command 表 |
- | 3 | **生成的测试用例完整列表**（functional + api）+ statistics |
- | 4 | **用例执行结果完整列表** + Failures 归类 + Warns + Manual items |
- | 5 | 冲突扫描 + 数据流候选 + 并行加载 |
- | 6 | E2E 场景步骤表 + state consistency + cleanup |
-
- > **用户硬性要求**：Phase 3 产生的每一条用例（`TC-F-01` ~ `TC-F-NN`）和
- > Phase 4 的每一条执行结果，必须在 Markdown 报告里以表格形式完整列出。
- > 不允许只给统计数、不给明细。
-
- #### §3 JSON 顶层 schema
-
- ```json
- {
- "test_id": "test-20260716-100300",
- "generated_at": "2026-07-16T10:03:00.000Z",
- "skills": [
- {
- "name": "huawei-cloud-rds-query",
- "skill_path": "C:/.../skills/huawei-cloud-rds-query",
- "phases_summary": [
- {"phase": 0, "name": "install-check", "verdict": "pass", "duration_s": 1.0, "summary": "4/4 目录硬要求通过"},
- {"phase": 1, "name": "skill-analysis", "verdict": "pass", "duration_s": 0.0, "summary": "提取 15 条命令"},
- {"phase": 2, "name": "tech-research", "verdict": "pass", "duration_s": 1.0, "summary": "CLI 15/15 可用"},
- {"phase": 3, "name": "test-case-generation","verdict": "pass", "duration_s": 0.0, "summary": "生成 28 条用例"},
- {"phase": 4, "name": "test-execution", "verdict": "partial", "duration_s": 6.0, "summary": "9 pass / 12 fail / 7 warn"},
- {"phase": 5, "name": "orchestration", "verdict": "pass", "duration_s": 1.0, "summary": "单 skill self-check"},
- {"phase": 6, "name": "full-flow", "verdict": "pass", "duration_s": 9.0, "summary": "15 步闭环"}
- ],
- "phases_detail": {
- "0": { "directory_integrity": {...}, "install": {...}, "uninstall": {...}, "reinstall": {...} },
- "1": { "metadata": {...}, "commands": [ ... ] },
- "2": { "research": [ ... ], "summary": {...} },
- "3": { "functional_cases": [ ... ], "api_cases": [ ... ], "statistics": {...} },
- "4": { "execution_results": [ ... ], "statistics": {...},
- "all_resources_changed": [ ... ], "manual_test_items": [ ... ] },
- "5": { "mode": "...", "conflict_scan": {...}, "data_flow_tests": [ ... ],
- "parallel_load_test": {...} },
- "6": { "mode": "...", "scenario": {...}, "state_consistency": {...}, "cleanup": {...} }
- }
- }
- ],
- "summary": {
- "verdict": "partial",
- "verdict_label": "⚠️ PARTIAL",
- "phases_total": 7,
- "phases_pass": 6,
- "phases_partial": 1,
- "phases_fail": 0,
- "phases_skipped": 0,
- "phases_missing": 0,
- "test_cases_total": 28,
- "test_cases_pass": 9,
- "test_cases_fail": 12,
- "test_cases_warn": 0,
- "test_cases_skip": 7,
- "test_cases_error": 0,
- "pass_rate": 32.1,
- "manual_items_count": 7,
- "cloud_resources_changed": 0
- },
- "key_findings": [
- "huawei-cloud-rds-query: 4 项目录硬要求全部通过 (4/4)",
- "huawei-cloud-rds-query: 纯只读, 提取 15 条命令 / 20 个触发词",
- "huawei-cloud-rds-query Phase 4: 12 个用例失败 (查看下方失败归类)"
- ],
- "environment": {
- "python_version": "3.11.15"
- },
- "report_dir": "C:/.../skills/huawei-cloud-rds-query-test-files/reports/report-20260716-100300"
- }
- ```
-
- #### 报告硬性要求（违反视为 bug）
+ ## Phases at a Glance
- 1. **必须有 Summary** — 哪怕只跑了一个 phase，Summary 也得填上对应统计
- 2. **必须有 Skills Tested 列表** — Summary 段必备，列出本次测试涉及的所有
- skill（`# | Skill | Path` 三列表格）。单 skill 也必须有这一行。
- 3. **必须有组合测试涉及 skill 列表** — Phase 5 / Phase 6 小节开头必备：
- - Phase 5 标题：`Skills involved in this orchestration (N):`
- - Phase 6 标题：`Skills involved in this E2E flow (N):`
- - 数据来源：`phase-{5,6}-summary.json` 的 `target.skills`
- - 降级单 skill 模式也必须列出 `["<该 skill>"]`
- 4. **每个阶段必须有：阶段小结 + 阶段产物（用例 / 结果）** — 不允许只写
- "Phase 4 pass" 然后跳过 28 条用例执行结果
- 5. **Markdown 和 JSON 必须字段对齐** — JSON 的 `phases_detail.<N>` 中出现的
- 字段，Markdown 报告里对应的 phase 小节必须有等价内容
- 6. **关键数字必须一致** — Summary 的 `test_cases_total` 必须等于
- `phases_detail.3.statistics.total`；`test_cases_pass` 必须等于
- `phases_detail.4.statistics.pass`，依此类推
- 7. **不得包含敏感信息** — AK/SK、token、内部 hostname 必须在日志和报告里
- `sed` 掉（参考 `lib/utils.sh` 的 redaction 规则）
- 8. **写资源变更必须列出** — `phases_detail.4.all_resources_changed` 不能省略
- 9. **Manual items 必须显式列出** — 不允许在 Markdown 里折叠 / 隐藏
- `manual_test_items`，要让用户能在 Summary 段就看到 `N (need real business data)`
- 10. **报告路径必须是绝对路径** — `report_dir` 必须是绝对路径，不接受相对路径
+ | Phase | 名称 | 核心要点 | 判定 |
+ |-------|------|---------|------|
+ | 0 | 安装验证 | 目录完整性(SKILL.md + references/ + iam-policies.md 硬性, scripts/ 软性兼容纯 CLI skill); install/uninstall/reinstall(同路径/符号链接防护) | 目录四件套 + 生命周期 |
+ | 1 | 功能提取 | metadata/triggers/commands/capabilities/resource_types + doc_checks(引用一致性、禁用文件 .bak/.template) | 命令与触发词非空 |
+ | 2 | 技术调研 | CLI→SDK→API 三级可用性; 纯本地工具自动标记 not_applicable | 按可用性计数 |
+ | 3 | 用例生成 | 正向 + 边界(limit=1) + 负向(未知参数)用例; 占位符替换(/path/to、./xxx、{region}); 模板/交互命令过滤 | 用例数 > 0 |
+ | 4 | 执行 | 自动执行 + 输出质量判定(rc=0 空输出→warn); 负向用例报错质量; 文档缺口分析(doc_gap_issues); 依赖缺失标注 | pass/fail/warn/skip |
+ | 5 | 编排 | 多 skill: 触发词冲突扫描 + 数据流候选; 单 skill: 降级自检(内部歧义 + 写操作排序) | conflict scan |
+ | 6 | 全流程 | 多 skill: 场景派生; 单 skill: 降级完整功能闭环(create→query→update→delete) | scenario steps |
+ | 7 | 报告 | 合并 phase-0~6; verdict + pass_rate + issues_found + 各 phase reason; test-report.json/md | 汇总判定 |
- 完整 JSON 字段规范见 `references/output-schema-spec.md` § Phase 7。
+ > 详细步骤、链式验证规则、JSON schema 见 `references/phase-details.md` 与 `references/output-schema-spec.md`。
---
## KooCLI Command Format Standard
This testing framework uses `bash` scripts as the primary execution mode, not direct `hcloud` CLI commands. However, when executing test cases, the framework constructs `hcloud` CLI commands in the following format:
```bash
hcloud <Service> <Operation> --cli-region={region} [--param1=value1 ...]
```
**Format Rules:**
| Rule | Description |
|------|-------------|
| Service name | Follows KooCLI Services (uppercase: ECS, VPC, OBS; title case: CloudPond, IAMAccessAnalyzer) |
| Operation name | PascalCase (e.g., ListServersDetails, ListBuckets) |
| Region | Always include `--cli-region={region}` parameter |
| Parameters | Use `--param=value` syntax |
| Read-only limit | Always append `--limit=1` for exploratory queries |
For OBS service, the framework uses `hcloud obs` (obsutil) subsystem:
```bash
hcloud obs <command> [args...] [options...]
```
---
## Core Commands
### Full Pipeline Run
```bash
# Specify skills
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage"
# Specify multiple skills (comma-separated)
bash scripts/run-test-pipeline.sh --skills "huawei-cloud-bss-voucher-manage, huawei-cloud-ecs-manage"
# Scan all installed
bash scripts/run-test-pipeline.sh --all-installed
# Start from a specific phase (recovery scenarios only)
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --phase 4
# Fresh mode
bash scripts/run-test-pipeline.sh --skills "bss-voucher" --fresh
```
### Single Phase Run (Debug)
```bash
bash scripts/tier1/phase-0-install-check.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier1/phase-1-skill-analysis.sh --skill "huawei-cloud-bss-voucher-manage"
bash scripts/tier2/phase-5-orchestration.sh --skills "skill-a, skill-b"
bash scripts/tier3/phase-7-final-report.sh --skills "skill-a, skill-b"
```
### Run Multi-Skill Orchestration
```bash
# Derive and execute orchestration scenarios for 3 skills
bash scripts/tier2/phase-5-orchestration.sh --skills "ecs-manage, vpc-manage, eip-manage"
# Run E2E lifecycle test for a single skill
bash scripts/tier2/phase-6-full-flow.sh --skill "huawei-cloud-rds-intelligent-service"
```
---
## Parameters
| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--skills` | Mutually exclusive | Comma-separated skill names or directory names | `"bss-voucher-manage, ecs-manage"` |
| `--all-installed` | Mutually exclusive | Scan all `huawei-cloud-*` under `$SKILL_INSTALL_DIR/huawei-cloud/` | — |
| `--phase` | No | Start from a specific Phase (defaults to resume from missing phase) | `--phase 0` |
| `--fresh` | No | **Archive** (move, not delete) existing `phase-*.json` to `phases/archive/<timestamp>/` and start from scratch. History is preserved; nothing is deleted. | — |
| `--output` | No | Report output directory (default: reports/) | `--output ./test-reports` |
| `--skill-path` | No | Skill directory path. When set, `find_skill_path` searches **only here** (no install-dir fallback) | `--skill-path ./skills` |
| `--no-siblings` | No | Phase 5/6 **不**自动扫描同目录其他 huawei-cloud-* skill（默认: 开启扫描） | `--no-siblings` |
| `--sibling-limit <N>` | No | 兄弟 skill 数量上限（默认 5；`0` = 等同 `--no-siblings`） | `--sibling-limit 3` |
### Environment Variables (Advanced)
| Variable | Default | Purpose |
|----------|---------|---------|
| `SKILL_INSTALL_DIR` | auto-detect: `~/.agents/skills` → `~/.hermes/skills` → `~/.agents/skills` | Where skills are installed by the agent runtime |
| `SKILL_PATH_HERMES` | alias of `SKILL_INSTALL_DIR` | Legacy name, kept for back-compat |
| `SKILL_INSTALL_CMD` | `hermes skills` | Command for remote skill install/uninstall. Set to `""` to skip real install. |
| `ALLOW_WRITES` | `0` | When `1`, Phase 4/6 write cases actually execute against the live API (default is skip) |
| `HUAWEI_REGION` | `cn-north-4` | Huawei Cloud region |
| `HUAWEI_ACCESS_KEY` / `HUAWEI_SECRET_KEY` | — | Required for Phase 4/6 SDK/CLI execution; any `HUAWEI*` / `HW*` / `HWC*` prefixed AK/SK env var is also accepted |
---
## References
### 核心文档（必读）
- `references/architecture.md` — Three-track eight-phase architecture diagram (Mermaid)
- `references/output-schema-spec.md` — Complete JSON field specification for each phase (including Phase 7 final report schema)
- `references/phase-transition-rules.md` — Phase transition / fallback / skip rules
- `references/acceptance-criteria.md` — Quality gates + 17-item report acceptance checklist
- `references/verification-method.md` — How to manually verify each phase (PowerShell + Git Bash)
+ - `references/phase-details.md` — 各 Phase 完整实现规范（步骤、判定标准、JSON 字段）
+ - `references/agent-protocol.md` — 凭证请求协议（AK/SK 缺失时的完整处理流程）
### 配套参考
- `references/cli-installation-guide.md` — How to install and configure hcloud CLI (prerequisite for Phase 2/4)
- `references/iam-policies.md` — Minimum IAM permissions required to run the tester
### 模板（JSON Schema）
- `templates/phase-report-schema.json` — JSON Schema for `phase-N-summary.json` (N=0..6)
- `templates/test-case-schema.json` — JSON Schema for individual test cases (`TC-F-*` / `TC-A-*` / `OF-*` / `FF-*`)
- `templates/scenario-template.json` — Reference example for Phase 6 multi-skill scenario derivation
---
## Output Format
All test artifacts go to a **sibling directory of the tested skill**, named `<skill-name>-test-files/`. This keeps the skill source dir clean and preserves run history for diff/regression.
```
skills/
├── huawei-cloud-rds-query/ ← skill source (untouched)
└── huawei-cloud-rds-query-test-files/ ← test artifacts (created on first run, kept across runs)
├── phases/
│ ├── phase-0-summary.json
│ ├── phase-1-summary.json
│ ├── ...
│ └── phase-7-summary.json
└── reports/
├── report-20260724-152309/ ← one subdir per run (timestamped)
│ ├── test-report.json
│ └── test-report.md
├── report-20260725-090000/
│ ├── test-report.json
│ └── test-report.md
└── ...
```
- **Phase 0~6** output `phase-N-summary.json` to `<skill-name>-test-files/phases/`
- **Phase 7** merges them into `<skill-name>-test-files/reports/report-<timestamp>/test-report.{json,md}`
- **Test artifacts are preserved** across runs (no auto-cleanup, ever). `--fresh` does NOT delete — it **archives** old `phases/*.json` to `phases/archive/<timestamp>/` so the chain check resets while history stays intact. `reports/` is always kept.
- **The test-installed copy** of the skill in `$SKILL_INSTALL_DIR/` is still uninstalled on exit, so the next run sees a clean install state.
See `references/output-schema-spec.md` for the JSON schema. Phase 5 and 6 additionally output scenario execution logs with real CLI/SDK responses for auditability.
## Best Practices
- Complete Tier 1 before entering Tier 2 to ensure skills are individually functional before orchestration
- Confirm write operations one by one in Phase 4 and Phase 5/6; do not batch-confirm to avoid misoperations
- With only 1 skill, Phase 5/6 automatically downgrade to single-skill closed loop; no need to manually skip
- When using `--fresh` to reset and rerun, confirm there are no uncleaned test resources
- Review orchestration scenarios before execution to ensure resource dependency order is correct
## Notes
- Three-track eight-phase strictly follows sequential order; chain verification prevents skipping
- API endpoints are strictly prohibited from being inferred; only obtain from SDK `_http_info` or API Explorer
- Credentials are read from environment variables; hardcoding is prohibited
- - **If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user verbatim and instruct them to set env vars out-of-band (in their shell / PowerShell $PROFILE). The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.**
+ - **If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user verbatim and instruct them to set env vars out-of-band (in their shell / PowerShell $PROFILE). The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.**（完整协议见 `references/agent-protocol.md`）
- Resources created during testing must be tracked; if any are left behind, output manual cleanup instructions
- Orchestration scenarios are auto-derived; user should review and confirm before execution
- Write operations in orchestration scenarios require per-step user confirmation
## Edge Cases
| Scenario | Handling |
|----------|----------|
| Skill directory does not exist | Report error and terminate, output available skill list |
- | AK/SK environment variables not set | Framework emits the env-var setup template (with `export HUAWEI_ACCESS_KEY=...` / `$env:HUAWEI_ACCESS_KEY=...` snippets) to stderr and exits 77. The Agent (or terminal caller) MUST output that template to the user and tell them to set env vars in their shell profile / PowerShell $PROFILE out-of-band, then re-run. **Never** ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping. |
+ | AK/SK environment variables not set | Framework emits the env-var setup template (with `export HUAWEI_ACCESS_KEY=<your-access-key>` / `$env:HUAWEI_ACCESS_KEY=<your-access-key>` placeholder snippets) to stderr and exits 77. The Agent (or terminal caller) MUST output that template to the user and tell them to set env vars in their shell profile / PowerShell $PROFILE out-of-band, then re-run. **Never** ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping. |
| User specifies skill name but not installed in Hermes | `--fresh` performs directory-level detection; if not found, report error with guidance |
| Some Phase JSON files deleted | Chain detection → Restart from the deleted Phase |
| Network interruption during Phase 4 execution | Already executed case results are not lost; on rerun, skip passed cases (via `--phase` flag) |
| User hits Ctrl+C mid-execution | Already output phase JSON is valid; next time `--resume` will recover from the current phase |
| Only 1 skill under test | Phase 5 → `downgraded_self_check` (single-skill trigger ambiguity scan); Phase 6 → `downgraded_single_skill_flow` (real execution) |
| User unsatisfied with derived orchestration scenarios | Manually edit the derived scenario or skip it; Phase 5 derivation is metadata-only, not executed |
| Phase 4 write op with no `ALLOW_WRITES=1` | Skipped with `status=skip`, no resource_changes recorded |
| Phase 4 hits missing business params (e.g. coupon_id) | Marked `status=warn`, surfaces as `manual_test_items` in the report; user must supply real data and retry |
| Cross-skill data flow mismatch | Logged in Phase 5 as `data_flow_tests` candidate; not auto-executed |
| Orphaned resources detected after E2E flow | Listed in `phase-6-summary.json` under `cleanup.manual_required` with concrete cleanup commands |
## Agent Protocol — Credential Request
- When Phase 4 or Phase 6 needs to call live Huawei Cloud APIs but cannot find
- credentials in the environment, the framework does **not** silently skip. It
- emits a structured request and exits with a sentinel code so the calling
- agent is forced to surface the need to the user — **but the agent must not
- ask the user to type or paste AK/SK in chat**. The user must set the
- variables in their shell profile out-of-band.
-
- **Sentinel string** (emitted to stderr, one line):
- ```
- __HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__
- ```
-
- **Exit code**: `77`
-
- **Agent response protocol** (MUST follow):
-
- 1. **Detect**: When running the test framework (directly or via
- `run-test-pipeline.sh`), catch exit code `77` OR the sentinel line in stderr.
- 2. **Pause**: Stop further pipeline execution. Do not skip ahead to Phase 5/7 —
- that would silently produce a "passing" report without live API coverage.
- 3. **Output the template**: Read the env-var setup template that the framework
- already emitted to stderr (the block beginning with
- `===== copy from here =====`) and output it to the user **verbatim**. Tell
- the user to fill in `<your-access-key-id>` and `<your-secret-access-key>`
- in their shell profile / PowerShell `$PROFILE` (out-of-band, NOT in chat).
- 4. **Never ask for AK/SK in chat**: Forbidden actions include
- `ask_user`, `read -p`, any web form, any clipboard paste back to the agent,
- any inline `read` loop, or any path through `~/.hcloud/config.json` /
- `~/.aliyun/config.json` / `~/.aws/credentials` that the user might
- silently trust.
- 5. **Re-run**: Once the user confirms they have set env vars out-of-band, the
- agent simply re-runs the failing phase:
- `HUAWEI_ACCESS_KEY=... bash run-test-pipeline.sh --skills <name> --phase 4`
- (the user is expected to have `export`-ed the vars in the shell where the
- agent executes the command).
- 6. **If the user declines**: Surface the decline to the human, do NOT mark
- Phase 4/6 as `pass`. You may abort the whole test, or report a partial run
- explicitly tagged "live phases skipped — no credentials".
-
- **Direct-terminal mode (no agent)**: If a human runs the script directly from
- a real terminal (`[ -t 0 ]` is true), the framework still emits the template
- to stderr and exits 77 — the user runs the same `export HUAWEI_ACCESS_KEY=...`
- in their shell and re-invokes the script. **There is no inline `read`
- prompt path any more.** The TTY prompt used to exist for human convenience
- but was removed because (a) it can leak values through terminal scrollback
- and clipboard, (b) it is inconsistent with the agent protocol, and (c) it
- violates the "no in-session secret entry" rule used elsewhere in the
- Huawei Cloud skill ecosystem.
-
- **Example agent behavior**:
+ When Phase 4 or Phase 6 needs to call live Huawei Cloud APIs but cannot find credentials in the environment, the framework does **not** silently skip. It emits a structured request (sentinel line `__HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__` to stderr) and exits with code `77` so the calling agent is forced to surface the need to the user.
- ```text
- > bash run-test-pipeline.sh --skills rds-query
- ... Phase 0-3 pass ...
- [Phase 4] __HUAWEI_SKILL_TESTER_CRED_REQUEST_v1__
- [Phase 4] HUAWEI_CREDENTIALS_REQUIRED
- [Phase 4] exit code: 77
- <agent detects 77, pauses run>
- <agent outputs the env-var template verbatim to the user, with a one-line instruction>
- <user opens their shell, pastes the export HUAWEI_ACCESS_KEY=... lines, re-runs>
- <agent re-runs: bash run-test-pipeline.sh --skills rds-query --phase 4>
- ```
+ **核心规则（MUST follow）:**
+ 1. 检测到 exit 77 或 sentinel → 暂停流水线,不得跳过 Phase 5/7
+ 2. 将 stderr 中的 env-var 设置模板**原样**输出给用户,引导用户在 shell profile / PowerShell $PROFILE 中带外配置(占位符 `<your-access-key-id>` / `<your-secret-access-key>`)
+ 3. **禁止**在对话中索要 AK/SK 明文(ask_user / read -p / 剪贴板回传 / 读取 ~/.hcloud/config.json 等路径)
+ 4. 用户带外设置后,重跑失败 phase;用户拒绝则明确标记"live phases skipped — no credentials",不得标记 pass
- **Rationale**: The test framework's job is to actually verify the skill
- against the real cloud, not to produce green checkmarks from offline analysis
- alone. Silently skipping live tests would let a broken skill pass. The
- sentinel + exit code is a hard "stop and ask" signal so the human is always
- in the loop for live-credential decisions — but the only safe way to handle
- credentials is to keep them out of the in-session channel entirely.
+ > 完整协议(6 步响应流程、直接终端模式、示例行为)见 `references/agent-protocol.md`。
## Design Principles
- **Chain Verification** — Each Phase checks the previous phase's JSON to prevent skipping
- **Agent-proof** — Write operations must be confirmed by the user; fake confirmations are not allowed
- **Data-Driven** — All phases output in JSON format; Phase 7 merges
- **Batch Repeatable** — The same set of skills can be tested repeatedly; --fresh resets
- **Real-Environment First** — All orchestrations and E2E flows execute against real Huawei Cloud; no mocks
- **Degrade Without Losing Value** — Single skill does not run empty orchestration phases; degrades to meaningful single-skill lifecycle tests
- **Resource Safety** — Resources created during testing must be tracked; if any remain, output clear manual cleanup instructions
- **Credentials Mandatory** — If AK/SK is missing, the framework emits the env-var setup template to stderr and exits 77. The Agent MUST output that template to the user and instruct them to set env vars out-of-band. The Agent MUST NEVER ask the user to type or paste AK/SK in chat. Strictly prohibited from silently skipping any step that requires credentials.