AGENTS.md@tests git:20260518.8456ac4A

Immutable. This exact content is served forever at /api/v1/blob/e8156187c31813f2.

# Skills Testing Guide

> **For AI Agents**: This document provides patterns and conventions for creating and maintaining tests for Azure Copilot skills. When asked to "scaffold tests" for a skill, follow the instructions below.

## Scaffolding Tests for a Skill

When a user asks to scaffold, create, or add tests for a skill, follow these steps:

### Step 1: Copy the template
```bash
cp -r tests/_template tests/{skill-name}
```

> **For AI Agents**: Do not add or change files outside of the new test directory.

### Step 2: Read the skill's SKILL.md
Load the file at `plugin/skills/{skill-name}/SKILL.md` to understand:
- The skill's name and description (from frontmatter)
- What the skill does (from content)
- What Azure services/tools it references

**Also check for references:** If the skill has a `references/` folder, note the structure:
- `references/recipes/` - Multiple implementation approaches (azd, bicep, terraform)
- `references/services/` - Multiple Azure services the skill supports
- References load only when explicitly linked, so understand what paths SKILL.md links to

> **Frontmatter Format Rule:** Descriptions over 200 characters MUST use folded YAML format (`>-`) for maintainability. The `>-` format keeps descriptions readable in source while parsing to a flat string compatible with skills.sh and other registries. Do NOT use `|` (literal block) as it preserves newlines.

### Step 3: Update test files
In each test file `triggers.test.ts`, change:
```typescript
const SKILL_NAME = '{skill-name}';  // Must match the folder name exactly
```

### Step 4: Generate trigger prompts
Based on the skill's description and content, add to `triggers.test.ts`:

**shouldTriggerPrompts** (at least 5) - prompts that mention:
- The skill's primary Azure service (e.g., "Redis", "Cosmos DB", "Key Vault")
- Common tasks the skill helps with
- Keywords from the skill's description

**shouldNotTriggerPrompts** (at least 5) - prompts about:
- Unrelated topics ("weather", "poetry")
- Different cloud providers ("AWS", "GCP")
- Different Azure services not covered by this skill

### Step 5: Configure integration tests (optional)
In `integration.test.ts`, customize the prompts to test real agent behavior.

Follow existing `integration.test.ts` files for how to implement such tests. 

### Step 6: Run and verify
```bash
cd tests

# Run all tests (integration runs if SDK available)
npm test -- --testPathPatterns={skill-name}

# Skip integration tests explicitly
SKIP_INTEGRATION_TESTS=true npm test -- --testPathPatterns={skill-name}
```

### Step 8: Update coverage grid
```bash
npm run coverage:grid
```

---

## Overview

This testing framework uses **Jest** to validate skill behavior across these test categories:
- **Trigger Tests** - Skill activation validation  
- **Integration Tests** - MCP tool interaction testing

## Quick Reference: Test File Conventions

---

## Test File Conventions

### File Naming

| File | Purpose |
|------|---------|
| `triggers.test.ts` | Tests skill activation on prompts |
| `integration.test.ts` | Tests real Copilot agent behavior (optional) |
| `fixtures/*.json` | Test data and mock responses |

### Directory Structure

```
tests/{skill-name}/
├── unit.test.js
├── triggers.test.js
├── integration.test.js   # Optional - requires Copilot CLI auth
├── __snapshots__/        # Auto-generated by Jest
│   └── triggers.test.js.snap
└── fixtures/
    └── prompts.json      # Trigger test prompts
```

---

## Writing Trigger Tests

Trigger tests verify that prompts correctly activate (or don't activate) your skill.

### Parameterized Tests

Use `test.each` for testing multiple prompts:

```typescript
const shouldTriggerPrompts = [
  'How do I deploy to Azure App Service?',
  'Configure my Azure storage account',
  'Help with Azure CLI commands',
];

test.each(shouldTriggerPrompts)(
  'triggers on: "%s"',
  (prompt) => {
    const result = triggerMatcher.shouldTrigger(prompt);
    expect(result.triggered).toBe(true);
  }
);
```

### Snapshot Tests

Snapshots catch unintended changes to trigger behavior:

```typescript
test('skill keywords match snapshot', () => {
  expect(triggerMatcher.getKeywords()).toMatchSnapshot();
});
```

### Updating Snapshots

When trigger behavior intentionally changes:

```bash
npm run update:snapshots -- --testPathPatterns=your-skill-name
```

**Always review snapshot changes before committing!**

---

## Using Fixtures

### Loading Fixtures

```typescript
import { loadFixtures, loadFixture } from '../utils/fixtures';

// Load all fixtures for a skill
const fixtures = loadFixtures('azure-validation');

// Load a specific fixture
const prompts = loadFixture('azure-validation', 'prompts');
```

### Fixture File Format

`fixtures/prompts.json`:
```json
{
  "shouldTrigger": [
    "Deploy to Azure",
    "Configure storage account"
  ],
  "shouldNotTrigger": [
    "Help with AWS",
    "Write a poem"
  ]
}
```

---

## Writing Integration Tests

Integration tests run a real Copilot agent session to verify skill behavior.

### Prerequisites

1. Install Copilot CLI: `npm install -g @github/copilot-cli`
2. Authenticate: Run `copilot` and follow prompts

### Basic Integration Test

```typescript
import { 
  run, 
  isSkillInvoked, 
  doesAssistantMessageIncludeKeyword,
  shouldSkipIntegrationTests,
  getIntegrationSkipReason
} from '../utils/agent-runner';

const SKILL_NAME = 'azure-rbac';

// Integration tests auto-skip if SDK unavailable or in CI
const skipTests = shouldSkipIntegrationTests();
const skipReason = getIntegrationSkipReason();
if (skipTests && skipReason) {
  console.log(`⏭️  Skipping integration tests: ${skipReason}`);
}

const describeIntegration = skipTests ? describe.skip : describe;

describeIntegration(`${SKILL_NAME}_ - Integration Tests`, () => {
  test('invokes skill for relevant prompt', async () => {
    const agentMetadata = await run({
      prompt: 'What role should I assign for blob storage access?'
    });

    expect(isSkillInvoked(agentMetadata, SKILL_NAME)).toBe(true);
    expect(doesAssistantMessageIncludeKeyword(agentMetadata, 'Storage Blob')).toBe(true);
  });
});
```

### Agent Runner Helpers

| Helper | Purpose |
|--------|---------|
| `agentRunner.run(config)` | Execute agent session with prompt |
| `agentRunner.isSkillInvoked(metadata, skillName)` | Check if skill was invoked |
| `agentRunner.areToolCallsSuccess(metadata, toolName)` | Check if tool calls succeeded |
| `agentRunner.doesAssistantMessageIncludeKeyword(metadata, keyword)` | Search response for keyword |

### Test with Workspace Setup

```javascript
test('works with project files', async () => {
  if (shouldSkip()) return;
  
  const agentMetadata = await agentRunner.run({
    setup: async (workspace) => {
      const fs = require('fs');
      const path = require('path');
      fs.writeFileSync(path.join(workspace, 'main.bicep'), 'resource ...');
    },
    prompt: 'Validate my Bicep file'
  });

  expect(agentRunner.isSkillInvoked(agentMetadata, 'azure-validation')).toBe(true);
});
```

---

## Running Tests

### Local Development

```bash
# Run all tests (integration runs if SDK available, skips if not)
npm test

# Unit and trigger tests only (always skips integration)
npm run test:unit

# Skip integration tests explicitly
SKIP_INTEGRATION_TESTS=true npm test

# Specific skill
npm test -- --testPathPatterns=azure-validation

# Watch mode
npm run test:watch -- --testPathPatterns=azure-validation

# With coverage
npm run test:coverage -- --testPathPatterns=azure-validation

# Verbose output
npm run test:verbose
```

### CI Environment

Tests automatically run on:
- Push to `main` affecting skill or test files
- Pull requests affecting skill or test files
- Manual workflow dispatch

Output formats:
- **Console**: Human-readable Jest output
- **CI**: JUnit XML in `tests/reports/junit.xml`
- **PR**: Annotations via GitHub Actions

---

## Coverage Requirements

### Minimum Thresholds

| Metric | Target |
|--------|--------|
| Statements | 60% |
| Branches | 50% |
| Functions | 60% |
| Lines | 60% |

### Checking Coverage

```bash
npm run test:coverage
```

Coverage reports are generated in `tests/coverage/`.

---

## GitHub Actions Integration

### Running Tests in CI

Use the **workflow_dispatch** trigger on `test-all-skills.yml`:

1. Go to **Actions** → **Test All Skills**
2. Click **Run workflow**
3. Enter a skill name (e.g., `azure-validation`) or leave empty for all skills

### Full Test Suite

`.github/workflows/test-all-skills.yml` runs all skill tests and updates the README coverage grid.

---

## Troubleshooting

### Test Not Running

1. Check skill name matches folder name exactly
2. Verify test file ends with `.test.js`
3. Check `testPathIgnorePatterns` in `jest.config.js`

### Snapshot Mismatch

1. Review the diff carefully
2. If change is intentional: `npm run update:snapshots`
3. If change is unintentional: investigate and fix

---

## Best Practices

1. **One assertion per test** when possible for clear failure messages
2. **Descriptive test names**: `test('rejects storage names over 24 characters')`
3. **Test edge cases**: Empty input, very long input, special characters
4. **Keep fixtures small**: Only include data needed for the test
5. **Review snapshots**: Don't blindly update—understand the change
6. **Clean up mocks**: Reset between tests to prevent interference

---

## Adding Tests Checklist

When adding tests for a new skill:

- [ ] Copy `_template/` to `tests/{skill-name}/`
- [ ] Update `SKILL_NAME` in all test files
- [ ] Add 5+ prompts that should trigger
- [ ] Add 5+ prompts that should NOT trigger  
- [ ] Run tests locally and verify passing
- [ ] Update coverage grid if significant changes

---

*Last updated: Auto-generated by skill testing framework*