tool-evaluator · diff
git:20260415.1275cf5 to git:20260417.4f9c495
188 added, 186 removed. Audit C to C.
# Tool Evaluator Skill
- Sub-agent skill for evaluating tools during `/design-ops:setup`. Checks MCP availability, API capabilities, and builds capability matrices.
+ Sub-agent skill for evaluating tools during `/design-ops:setup`. Coordinates MCP availability checks and API capability evaluation.
## Purpose
- When a user selects tools in the setup flow, this skill evaluates each tool to determine:
- 1. Is there an MCP available and connected?
- 2. If no MCP, is there an API suitable for reporting?
- 3. What data can this tool provide (daily, weekly, monthly)?
- 4. What's the connection type needed?
+ When a user selects tools in the setup flow, this skill:
+ 1. Checks if the tool is already installed and connected
+ 2. Invokes the `mcp-discovery` skill for unknown tools
+ 3. Returns a unified result with connection type and status
## Trigger
Invoked by `/design-ops:setup` during tool evaluation phase. Should run as a background sub-agent.
---
## Input
```yaml
tools:
- name: "notion"
pillar: "operations"
- name: "figma"
pillar: "design"
- name: "substack"
pillar: "analytics"
```
## Output
- Returns a capability matrix for each tool:
+ Returns a capability matrix for each tool, including discovery metadata:
```yaml
results:
- tool: notion
connection_type: mcp
mcp_name: "notion"
- status: connected # connected | needs_setup | unavailable
+ status: connected # connected | available | api_only | unavailable | skipped
+
+ # Discovery metadata (from mcp-discovery skill)
+ mcp_source: official # official | vendor | community | none
+ mcp_package: "@notionhq/notion-mcp-server"
+ mcp_confidence: high # high | medium | low
+
+ # Capabilities
capabilities:
data_types: [pages, databases, tasks, comments]
reporting:
daily: [recent_pages, task_counts, recent_comments]
weekly: [page_activity, task_completion]
monthly: [content_growth]
+
setup_required: false
+ warning: null
- tool: figma
connection_type: api
- api_endpoint: "https://api.figma.com/v1"
- status: needs_token
+ status: api_only
+
+ # Discovery metadata
+ mcp_source: official
+ mcp_package: null # MCP exists but is code-focused
+ mcp_confidence: low
+ api_docs_url: "https://www.figma.com/developers/api"
+
capabilities:
data_types: [files, versions, comments, users]
reporting:
daily: [files_edited, active_users]
weekly: [design_versions, comment_activity]
monthly: [project_progress]
+
setup_required: true
setup_instructions: "Generate API token at figma.com/developers"
+ note: "Official MCP is code-focused. Use API for reporting."
- - tool: substack
- connection_type: custom_wrapper
- api_docs: "https://substack.com/api"
- status: needs_wrapper
+ - tool: gitlab
+ connection_type: mcp
+ status: available
+
+ # Discovery metadata
+ mcp_source: community
+ mcp_package: "mcp-gitlab"
+ mcp_confidence: medium
+
capabilities:
- data_types: [subscribers, posts, email_stats]
+ data_types: [projects, commits, merge_requests, issues, pipelines]
reporting:
- daily: [new_subscribers]
- weekly: [subscriber_growth, post_views]
- monthly: [audience_trends]
+ daily: [recent_commits, open_mrs]
+ weekly: [contributions, mr_activity]
+
setup_required: true
- setup_instructions: "Create MCP wrapper using /mcp-builder"
+ setup_instructions: "claude mcp add gitlab -- npx -y mcp-gitlab"
+ warning: "Community package - not officially supported"
+ warning_code: community_package
- tool: instagram
connection_type: unavailable
status: unavailable
+
+ mcp_source: none
+ mcp_confidence: none
+
reason: "API limited to business accounts with complex approval process"
alternatives: ["Buffer", "Later", "Sprout Social"]
```
---
## Evaluation Process
- ### Step 1: Check for MCP
+ ### Step 1: Check if Already Installed
Read `~/.claude/settings.json` and check for MCP servers:
```bash
cat ~/.claude/settings.json | jq '.mcpServers'
```
For each tool, check:
1. Is there an MCP with matching name?
- 2. Is it connected and responding?
- 3. Does it support reporting-relevant data?
+ 2. Is it responding to basic queries?
- **MCP naming patterns:**
- - Notion: `notion`, `@anthropic/mcp-notion`
- - GitHub: `github`, `mcp-github`
- - Google Workspace: `google-workspace`, `google-calendar`, `gmail`
- - Linear: `linear`, `mcp-linear`
- - Slack: `slack`, `mcp-slack`
+ If installed and responding → status: `connected`
- ### Step 2: Evaluate MCP Capabilities
+ ### Step 2: Invoke mcp-discovery Skill
- If MCP found, determine its reporting capabilities:
+ If not installed, delegate to the `mcp-discovery` skill:
- **For Notion MCP:**
```yaml
- data_types: [pages, databases, tasks, comments]
- reporting:
- daily:
- - recent_pages: "search(filter: last_edited_time > 24h)"
- - task_counts: "query_database(filter: status)"
- weekly:
- - page_activity: "Aggregate page creation/modification"
- - task_completion: "Tasks completed this week"
+ skill: mcp-discovery
+ input:
+ tool: "gitlab"
+ pillar: "design"
```
- **For GitHub MCP:**
- ```yaml
- data_types: [repos, commits, prs, issues]
- reporting:
- daily:
- - recent_commits: "list_commits(since: 24h)"
- - open_prs: "list_pull_requests(state: open)"
- weekly:
- - team_contributions: "Commits grouped by author"
- - pr_activity: "PRs opened, merged, closed"
- ```
+ The discovery skill returns:
+ - `mcp_source`: official | vendor | community | none
+ - `mcp_package`: npm package name (or null)
+ - `mcp_confidence`: high | medium | low
+ - `api_docs_url`: API documentation URL
+ - `recommendation`: mcp | api | both | unavailable
+ - `warning`: any warnings about community packages
- ### Step 3: Check API If No MCP
+ ### Step 3: Determine Status from Discovery
- If no suitable MCP, consult `references/tool-registry.md` for API information:
+ Based on `discovery_result.recommendation`:
- 1. Does the tool have a public API?
- 2. What authentication is required?
- 3. What reporting endpoints exist?
- 4. Can it provide daily/weekly/monthly data?
+ | Recommendation | Confidence | Status | Connection Type |
+ |----------------|------------|--------|-----------------|
+ | mcp | high | available | mcp |
+ | mcp | medium | available (with warning) | mcp |
+ | mcp | low | api_only suggested | api |
+ | api | — | api_only | api |
+ | unavailable | — | unavailable | unavailable |
- ### Step 4: Determine Connection Type
+ ### Step 4: Build Capability Matrix
- Based on evaluation:
+ For connected/available tools, populate capabilities from known data:
- | Scenario | Connection Type | Status |
- |----------|-----------------|--------|
- | MCP found and connected | `mcp` | `connected` |
- | MCP exists but not added | `mcp` | `needs_setup` |
- | No MCP, API supports reporting | `api` | `needs_token` |
- | No MCP, API needs wrapper | `custom_wrapper` | `needs_wrapper` |
- | No API or restricted | `unavailable` | `unavailable` |
+ **Operations pillar:**
+ - Notion, Google Workspace, Linear, Slack: See known capabilities below
+ **Design pillar:**
+ - GitHub, GitLab, Figma: See known capabilities below
+
+ **Analytics pillar:**
+ - GA4, Dub.co, Plausible, Substack: See known capabilities below
+
---
- ## Known Tool Evaluations
+ ## Known Tool Capabilities
### Operations Pillar
**Notion**
- - MCP: Official Notion MCP available
- - Capabilities: Pages, databases, blocks, comments, users
- - Daily: Recent pages, task counts, comments
- - Weekly: Activity summaries, task completion
+ - Data types: pages, databases, blocks, comments, users
+ - Daily: recent_pages, task_counts, recent_comments
+ - Weekly: page_activity, task_completion
**Google Workspace**
- - MCP: Multiple MCPs (Calendar, Gmail, Drive)
- - Capabilities: Events, emails, documents
- - Daily: Today's events, unread emails
- - Weekly: Meeting count, email volume
+ - Data types: calendar_events, emails, documents
+ - Daily: todays_events, unread_emails
+ - Weekly: event_count, email_volume
**Linear**
- - MCP: Community MCP available
- - Capabilities: Issues, projects, cycles, teams
- - Daily: Issues due, assigned work
- - Weekly: Cycle progress, velocity
-
- **Asana**
- - MCP: Limited community support
- - API: Good reporting API
- - Daily: Tasks due, section counts
- - Weekly: Project progress
+ - Data types: issues, projects, cycles, teams, labels
+ - Daily: issues_due, assigned_issues
+ - Weekly: issues_completed, cycle_progress
**Slack**
- - MCP: Community MCPs available
- - Capabilities: Messages, channels, reactions
- - Daily: Unread counts, mentions
- - Weekly: Channel activity
+ - Data types: messages, channels, reactions
+ - Daily: unread_counts, mentions
+ - Weekly: channel_activity
### Design Pillar
**GitHub**
- - MCP: Official GitHub MCP
- - Capabilities: Repos, commits, PRs, issues, actions
- - Daily: Recent commits, open PRs
- - Weekly: Contributions by person, merge activity
+ - Data types: repos, commits, prs, issues, actions
+ - Daily: recent_commits, open_prs
+ - Weekly: team_contributions, pr_activity
**GitLab**
- - MCP: Community MCP
- - Capabilities: Similar to GitHub
- - Daily: Recent commits, merge requests
- - Weekly: Pipeline stats, contributions
-
- **Figma**
- - MCP: Official MCP (code-focused, NOT for reporting)
- - API: Direct API better for reporting
- - Capabilities: Files, versions, comments, users
- - Daily: Files edited, active users
- - Weekly: Version history, comment activity
+ - Data types: projects, commits, merge_requests, issues, pipelines
+ - Daily: recent_commits, open_mrs
+ - Weekly: contributions, ci_stats
- **Sketch**
- - MCP: None
- - API: Limited (Sketch Cloud API)
- - Status: Often unavailable for reporting
+ **Figma** (API only for reporting)
+ - Data types: files, versions, comments, users
+ - Daily: files_edited, active_users
+ - Weekly: design_versions, comment_activity
### Analytics Pillar
**Google Analytics (GA4)**
- - MCP: Official GA4 MCP
- - Capabilities: Pageviews, sessions, events, goals
- - Daily: Session count, top pages
- - Weekly: Traffic trends, source breakdown
+ - Data types: pageviews, sessions, events, goals
+ - Daily: session_count, top_pages
+ - Weekly: traffic_trends, source_breakdown
**Dub.co**
- - MCP: Unofficial but functional
- - Capabilities: Links, clicks, referrers, geo
- - Daily: Click counts per link
- - Weekly: Top performing links
-
- **Plausible**
- - MCP: None
- - API: Good reporting API
- - Status: Needs wrapper or direct API
-
- **Substack**
- - MCP: None
- - API: Basic API available
- - Status: Needs custom wrapper
- - Capabilities: Subscribers, posts, email stats
-
- **Instagram**
- - MCP: None
- - API: Very limited (business only)
- - Status: Often unavailable
- - Note: Consider alternatives like Buffer
+ - Data types: links, clicks, referrers, geo
+ - Daily: click_counts
+ - Weekly: top_links
- **Twitter/X**
- - MCP: None
- - API: Paid tiers only
- - Status: Often unavailable
+ **Plausible** (API only)
+ - Data types: visitors, pageviews, sources
+ - Daily: visitor_count, top_pages
+ - Weekly: traffic_trends
- **LinkedIn**
- - MCP: None
- - API: Limited to company pages
- - Status: Restricted
+ **Substack** (API only, needs wrapper)
+ - Data types: subscribers, posts, email_stats
+ - Daily: new_subscribers
+ - Weekly: subscriber_growth, post_views
---
## Integration with Setup Flow
### During Chapter 1 (Operations)
```
User selects: [Notion, Google Workspace, Slack]
↓
Spawn tool-evaluator sub-agent:
- - Check Notion MCP → Found, connected
- - Check Google Workspace MCP → Found, connected
- - Check Slack MCP → Found, NOT connected
+ - Check Notion MCP → Found, connected → status: connected
+ - Check Google Workspace MCP → Found, connected → status: connected
+ - Check Slack MCP → Not found → invoke mcp-discovery
+ → mcp-discovery returns: community, medium confidence, warning
+ → Slack status: available (with warning)
↓
Return capability matrix
↓
- Main agent presents findings with Slack showing "needs_setup"
+ Main agent presents findings with status indicators
```
### During Chapter 2 (Design)
```
User selects: [GitHub, Figma]
↓
Spawn tool-evaluator sub-agent:
- - Check GitHub MCP → Found, connected
- - Check Figma MCP → Found, but code-focused (not for reporting)
- - Check Figma API → Available, needs token
+ - Check GitHub MCP → Found, connected → status: connected
+ - Check Figma MCP → Found, but code-focused → invoke mcp-discovery
+ → mcp-discovery returns: official but code-focused, api recommended
+ → Figma status: api_only (with note)
↓
- Return capability matrix with Figma showing "api" type, "needs_token"
+ Return capability matrix
↓
- Main agent guides Figma token setup
+ Main agent guides Figma API token setup
```
### During Chapter 3 (Analytics)
```
User selects: [GA4, Substack, Instagram]
↓
Spawn tool-evaluator sub-agent:
- - Check GA4 MCP → Found, connected
- - Check Substack → No MCP, API available → needs_wrapper
- - Check Instagram → No MCP, API limited → unavailable
+ - Check GA4 MCP → Found, connected → status: connected
+ - Check Substack → invoke mcp-discovery
+ → Returns: no MCP, API available, needs wrapper
+ → Substack status: api_only (with wrapper option)
+ - Check Instagram → invoke mcp-discovery
+ → Returns: unavailable
+ → Instagram status: unavailable
↓
Return capability matrix
↓
- Main agent offers to create Substack wrapper, skips Instagram
+ Main agent offers Substack wrapper creation, skips Instagram
```
---
## Error Handling
**MCP not responding:**
```yaml
tool: notion
connection_type: mcp
status: error
error: "MCP timeout - server not responding"
suggestion: "Restart Claude or check MCP server"
```
- **API evaluation failed:**
+ **Discovery failed:**
```yaml
- tool: substack
+ tool: unknown_tool
connection_type: unknown
status: error
- error: "Could not evaluate API capabilities"
- suggestion: "Check references/tool-registry.md for manual setup"
- ```
-
- **Unknown tool:**
- ```yaml
- tool: "custom-crm"
- connection_type: unknown
- status: needs_evaluation
+ error: "Discovery could not determine connection method"
suggestion: "Check if tool has public API documentation"
```
---
## Usage Example
When invoked by setup flow:
```python
# Pseudo-code for sub-agent task
- def evaluate_tools(tools: list[str], pillar: str) -> dict:
+ def evaluate_tools(tools: list[dict]) -> dict:
results = []
- for tool in tools:
- # Step 1: Check MCP
- mcp_info = check_mcp(tool)
+ for tool_info in tools:
+ tool = tool_info['name']
+ # Step 1: Check if already installed
+ mcp_info = check_installed_mcp(tool)
+
if mcp_info.found and mcp_info.connected:
results.append({
'tool': tool,
'connection_type': 'mcp',
'status': 'connected',
- 'capabilities': get_mcp_capabilities(mcp_info)
- })
- elif mcp_info.found:
- results.append({
- 'tool': tool,
- 'connection_type': 'mcp',
- 'status': 'needs_setup',
- 'setup_instructions': mcp_info.install_command
+ 'mcp_source': 'installed',
+ 'mcp_confidence': 'high',
+ 'capabilities': get_capabilities(tool)
})
else:
- # Step 2: Check API
- api_info = check_api(tool)
+ # Step 2: Invoke mcp-discovery skill
+ discovery = invoke_skill('mcp-discovery', tool=tool)
- if api_info.suitable_for_reporting:
+ # Step 3: Determine status from discovery
+ if discovery.recommendation == 'mcp':
+ if discovery.mcp_confidence == 'high':
+ status = 'available'
+ warning = None
+ else:
+ status = 'available'
+ warning = discovery.warning
+
results.append({
'tool': tool,
- 'connection_type': 'api' if api_info.direct_use else 'custom_wrapper',
- 'status': 'needs_token' if api_info.direct_use else 'needs_wrapper',
- 'capabilities': api_info.capabilities
+ 'connection_type': 'mcp',
+ 'status': status,
+ 'mcp_source': discovery.mcp_source,
+ 'mcp_package': discovery.mcp_package,
+ 'mcp_confidence': discovery.mcp_confidence,
+ 'capabilities': get_capabilities(tool),
+ 'setup_instructions': f"claude mcp add {tool} -- npx -y {discovery.mcp_package}",
+ 'warning': warning
})
+
+ elif discovery.recommendation == 'api':
+ results.append({
+ 'tool': tool,
+ 'connection_type': 'api',
+ 'status': 'api_only',
+ 'mcp_source': discovery.mcp_source,
+ 'api_docs_url': discovery.api_docs_url,
+ 'capabilities': get_capabilities(tool),
+ 'setup_required': True,
+ 'note': discovery.note
+ })
+
else:
results.append({
'tool': tool,
'connection_type': 'unavailable',
'status': 'unavailable',
- 'reason': api_info.limitation_reason
+ 'mcp_source': 'none',
+ 'reason': discovery.reason,
+ 'alternatives': discovery.alternatives
})
return {'results': results}
```
---
## References
- - `references/tool-registry.md` — Known tools database with MCP/API status
- - `references/mcp-setup/` — MCP installation guides
+ - `skills/mcp-discovery/SKILL.md` — Dynamic MCP discovery
+ - `references/tool-registry.md` — Reference documentation (not source of truth)
- `references/config-schema.md` — Config structure for storing results
---
- *Version: 1.0*
+ *Version: 2.0*