evaluation-plan · diff
git:20261111.028da5f to git:20261111.c3fdee5
2 added, 1 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
---
name: evaluation-plan
description: |
Evidence-based evaluation plan for a course or a training program. Starts from the
decisions that the results must support, then plans measures of reaction, learning,
behavior on the job and results (the Kirkpatrick levels), a data collection schedule,
and the support that learners need to use the skills after the course. Works
standalone or reads the idstack project manifest. Use when asked to plan a course or
training evaluation, measure training effectiveness or impact, write a learner survey,
- show ROI, or apply Kirkpatrick. (idstack)
+ show ROI, or apply Kirkpatrick.
+ Framework phase: ADDIE Evaluate (summative). (idstack)
allowed-tools:
- Bash
- Read
- Write
- Edit
- Glob
- Grep
- AskUserQuestion
---
<!-- AUTO-GENERATED from SKILL.md.tmpl -- do not edit directly -->
<!-- Edit the .tmpl file instead. Regenerate: scripts/idstack-gen-skills -->
## Preamble: Interaction Conventions
idstack runs in Claude Code and in Cowork. Skill bodies name these tools:
- **AskUserQuestion** — when a skill says "ask via AskUserQuestion" or "using AskUserQuestion",
present a single multiple-choice question (e.g., "Which of these best describes X?")
and stop, waiting for the user's answer before proceeding. Ask **one** question at a time,
never batch. This maps to the `AskUserQuestion` tool. The tool takes **2 to 4 options**
per question and adds an "Other" option itself, so do not add "Other". When there are
more than 4 choices, write them as a numbered list in chat and ask the user to type a
number. When the answer is free text (a name, a URL, a file path, a description), ask
in plain chat and stop for the answer.
- **Agent (sub-agents)** — four skills start sub-agents with the `Agent` tool.
accessibility-review and course-quality-review run their reviews in parallel. red-team
starts one orchestrator, which runs its five dimensions. course-builder
starts one agent that does the alignment review.
The written-out steps after each dispatch define the work. If a sub-agent fails or
returns nothing, accessibility-review and course-quality-review run its steps
themselves, in sequence. course-builder skips the alignment review and records that in
the Limitations. red-team tells the user and stops if the orchestrator returns no valid
findings file. A sub-agent cannot start its own
sub-agents, so the red-team orchestrator runs its five dimensions in sequence when it
cannot dispatch them.
- **Skill (cross-skill invocation)** — used by `/idstack:pipeline`, which invokes each
child skill in-process through the `Skill` tool. `/idstack:start` also uses it to start
the first skill for the user.
- **Skill invocation syntax in user-facing text** — every skill is invoked as
`/idstack:<name>`. Always write the namespaced form: a bare `/<name>` is not a valid
command. This applies in reports, manifests, and AskUserQuestion options as much as in
chat output.
## Preamble: Writing Standard (ASD-STE100)
All text that idstack writes for a person must obey the rules in this section. This
includes chat text, AskUserQuestion questions and options, reports, the course
dashboard, course content for learners, and export packages. `scripts/idstack-ste-check`
reads the word list between the two `ste-core` markers. `scripts/idstack-gen-skills` writes
`extension/shared/ste-rules.js` from the text between the markers.
The rules also apply to the text that idstack writes in `.idstack/project.json` and in
learnings, for example finding text and recommendations. Other skills use this text in
their output. The checker does not read these files. Do not change text that comes from
the course. Do not change JSON keys or the values that a script or a skill reads.
<!-- ste-core:begin -->
Writing standard: ASD-STE100 Simplified Technical English (STE)
This summary uses the ASD-STE100 writing rules. ASD does not endorse idstack. This summary does not replace the ASD-STE100 standard.
Use these rules only for text in English. If the text is not in English, do not use these rules.
Words:
- Use one word for one meaning. Use the same term for the same item in all of the text.
- Use short words that are easy to understand. Do not use the words in the first column of the word list.
- Use the technical nouns and technical verbs of instructional design when no simple word has the same meaning. Examples of technical nouns are learning objective, ILO, rubric, criterion, formative assessment, scaffold, cognitive load, meta-analysis, evidence tier, finding and severity.
- The Bloom's taxonomy verbs are technical verbs. Examples are remember, understand, apply, analyze, evaluate, create, assess, critique, demonstrate, design and implement. Use them in learning objectives and in text about Bloom's levels.
- Software verbs are technical verbs. Examples are run, click, open, save, install and update.
- Do not use a technical noun as a verb.
- Do not use contractions. Write "do not", not "don't".
- Do not use Latin abbreviations. Write "for example", not "e.g.".
Verbs:
- Use the active voice. In a description, use the passive voice only when you do not know who or what does the work.
- Use only the simple tenses. Write "writes", "wrote" or "will write". Do not write "has written", "had written" or "is writing".
- Do not use a verb that ends in "-ing". You can use an "-ing" word only in a technical noun, for example "learning objective".
Sentences:
- Write one topic in each sentence.
- An instruction has a maximum of 20 words. A description has a maximum of 25 words.
- Do not remove words, for example "the", "a" and "is", to make a sentence shorter.
- A noun cluster has a maximum of three nouns.
- Do not use semicolons. Write two sentences.
- Use a vertical list for text that has many parts.
Instructions and descriptions:
- Write an instruction as a command. Write one instruction in each sentence. Give a number to each step.
- If the reader must know a condition first, write the condition first. Example: "If the course has no rubric, add a rubric."
- Do not write commands in a description. A paragraph has one topic and a maximum of six sentences.
Recommendations:
- First show the problem that idstack found. Then show the evidence. Then give the recommendation.
- In a description, write a recommendation as "idstack recommends that you ..." or as "You can ...".
- In a numbered list of steps, write each step as a command.
- Do not use "consider", "may", "should" or "suggest".
Text that you do not change:
- Do not change a quotation from a course, a person, a standard or a different software tool. Put each quotation in quotation marks. A new version of a text is not a quotation. Write it with these rules.
- Do not change code, file names, commands, URLs, citations, for example [Domain-N] [T1], or the names of products and standards.
Word list. Do not use the word in the first column. Use the word in the second column.
| Do not use | Use | Note |
|---|---|---|
| "utilize" | "use" | |
| "ensure" | "make sure" | |
| "verify" | "make sure" | |
| "confirm" | "make sure" | |
| "commence" | "start" | |
| "begin" | "start" | |
| "initiate" | "start" | |
| "terminate" | "stop" | |
| "prior to" | "before" | |
| "in order to" | "to" | |
| "assist" | "help" | |
| "facilitate" | "help" | |
| "obtain" | "get" | |
| "achieve" | "get" | |
| "require" | "necessary" | |
| "indicate" | "show" | |
| "appear" | "show" | |
| "consider" | "idstack recommends that you" or "think about" | |
| "suggest" | "recommend" | |
| "suggestion" | "recommendation" | |
| "may" | "can" | lowercase only |
| "might" | "can" | |
| "should" | "must" or "idstack recommends that you" | |
| "would" | "can" | |
| "shall" | "must" | |
| "perform" | "do" | |
| "accomplish" | "do" | |
| "additional" | "more" | |
| "numerous" | "many" | |
| "enough" | "sufficient" | |
| "provide" | "give" | |
| "allow" | "let" | |
| "choose" | "select" | |
| "determine" | "find" | |
| "locate" | "find" | |
| "modify" | "change" | |
| "attempt" | "try" | |
| "therefore" | "thus" | |
| "however" | "but" | |
| "whether" | "if" | |
| "via" | "through" | |
| "upon" | "on" | |
| "such as" | "for example" | |
| "e.g." | "for example" | |
| "i.e." | "that is" | |
| "etc." | (write the full list) | |
| "please" | (remove the word) | |
<!-- ste-core:end -->
How to use the standard:
1. If the conversation or the course is not in English, do not run the checker.
2. When you write a file for a person, run `scripts/idstack-ste-check` on it at the step that the skill shows.
3. If the checker shows problems, write each sentence that it shows again. Then write the file again and run the checker again.
4. Run the checker a maximum of three times. If problems stay after the third time, tell the user which lines have problems.
5. If the output is `STE_CHECK_UNAVAILABLE`, python3 is not available. Continue the skill. Tell the user one time that the check did not run.
6. If the output starts with `STE_CHECK_MISSING:`, the idstack install at the path after the token has no checker. Continue the skill. Tell the user one time that the check did not run. Give the user the path and tell the user to update idstack. If the path is empty, tell the user that idstack did not find its install.
7. If the checker shows an error, continue the skill. Tell the user one time which file the checker did not examine.
The checker finds only some problems: long sentences, long paragraphs, semicolons,
contractions, "has been" verbs and the words in the word list. Apply all of the rules
when you write. The verbosity and experience settings change how much you write. They do
not change these rules.
Identify text from a different source for the checker:
- In HTML, put a short quotation in `<q>` and a long quotation in `<blockquote>`. If a full list item or table cell is a quotation, add `data-ste="quoted"` to that element.
- In Markdown, put a long quotation on lines that start with `> `. Put a short quotation in quotation marks.
- The name of a standard or a principle can have a word from the word list. Identify that name as a short quotation. Example: `<q>3.3.3 Error Suggestion</q>`.
- The checker counts each quotation, citation, code span and placeholder as one word.
A <placeholder> that is not in backticks counts as no word.
When a skill sends work to a sub-agent, put a copy of this section in the sub-agent
prompt. The sub-agent cannot see this preamble.
Example sentences in a skill show the content of a message. Write them in STE when you
use them.
## Preamble: Project Manifest
Before starting, check for an existing project manifest.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ -f ".idstack/project.json" ]; then
echo "MANIFEST_EXISTS"
"$_IDSTACK/scripts/idstack-migrate" .idstack/project.json 2>/dev/null || cat .idstack/project.json
else
echo "NO_MANIFEST"
fi
```
**If MANIFEST_EXISTS:**
- Read the manifest. If the JSON is malformed, report the specific parse error to the
user, offer to fix it, and STOP until it is valid. Never silently overwrite corrupt JSON.
- Preserve all existing sections when writing back.
**If NO_MANIFEST:**
- This skill will create or update the manifest during its workflow.
## Preamble: Preferences
```bash
if [ -f ".idstack/project.json" ] && command -v python3 &>/dev/null; then
python3 -c "
import json, sys
try:
data = json.load(open('.idstack/project.json'))
prefs = data.get('preferences', {})
v = prefs.get('verbosity', 'normal')
if v != 'normal':
print(f'VERBOSITY:{v}')
except: pass
" 2>/dev/null || true
fi
```
**If VERBOSITY:concise:** Keep explanations brief. Do not cite evidence in chat prose.
Always cite evidence in Step Review tables and in reports.
**If VERBOSITY:detailed:** Include full evidence citations, alternative approaches
considered, and rationale for each recommendation.
**If VERBOSITY:normal or not shown:** Default behavior — cite evidence tiers inline,
explain key decisions, skip exhaustive alternatives.
## Preamble: Designer Profile
```bash
_PROFILE="$HOME/.idstack/profile.yaml"
if [ -f "$_PROFILE" ]; then
# Simple YAML parsing for experience_level (no dependency needed)
_EXP=$(grep -E '^experience_level:' "$_PROFILE" 2>/dev/null | sed 's/experience_level:[[:space:]]*//' | tr -d '"' | tr -d "'")
[ -n "$_EXP" ] && echo "EXPERIENCE:$_EXP"
else
echo "NO_PROFILE"
fi
```
**If EXPERIENCE:novice:** Provide more context for recommendations. Explain WHY each
step matters, not just what to do. Define jargon on first use. Offer examples.
**If EXPERIENCE:intermediate:** Standard explanations. Assume familiarity with
instructional design concepts but explain idstack-specific patterns.
**If EXPERIENCE:expert:** Be concise. Skip basic explanations. Focus on evidence
tiers, edge cases, and advanced considerations. Trust the user's domain knowledge.
**If NO_PROFILE:** On first run, after the main workflow is underway (not before),
mention: "Note: To change how much information idstack gives, type `/idstack:start` and
select your experience level."
## Preamble: Report Links
A skill that writes a report runs a block that prints a `Report path:` line.
The line gives the path of the Markdown report, relative to the course folder.
`/idstack:pipeline` and `/idstack:start` print a `Dashboard path:` line for the
course dashboard.
When you tell the user about a report or the dashboard, put the path from that
line in a Markdown link. Use the same path as the link text. For example:
[.idstack/exports/bio-101/course-import.md](.idstack/exports/bio-101/course-import.md)
Do not change the path to a full path, and do not put angle brackets around it.
The course slug has no spaces, so the relative path always makes a link that
works. In the Claude desktop app, the user clicks the link to open the report.
Give only a path from a `Report path:` or `Dashboard path:` line, or a path that
a sub-agent returns. Do not make up a path. If the skill stopped before it wrote
the report, do not give the path.
## Preamble: Step Review
Only the skills that write a report do a Step Review. After each step that makes
an output, idstack adds the output to the report as a draft section with a table.
The user reads the section in the report file, not in the chat. The user approves
the section, and idstack removes the draft mark. If the session stops, the report
keeps each section, and a section that the user did not approve keeps its draft
mark.
The skill tells you where to do a Step Review with a `**Step review:**` line.
That line gives the section heading and the table columns.
**The shape of a section**
- Start the section with a `## <Heading>` line. Then write the table.
- The first column is `ID`. Give each row an ID that does not change, for
example `T-1` or `ILO-2`.
- The last two columns are `Justification` and `Evidence`.
- In the Justification cell, write one sentence. Tell why the row is correct for
this course. Use the evidence from the research.
- In the Evidence cell, write only the citation, for example
`[Assessment-8] [T1]`. Use a code from the "Evidence list:" file. Use the tier
that the file gives for that code.
- If you cannot cite a row, do not put it in the table. Put the text in the
Limitations section.
- These fact sections have no Justification or Evidence columns: Project context,
Course information, Imported structure, Readiness, Export contents and
Generated artifacts.
- A findings table has these columns:
`ID | Severity | What we saw | Recommendation | Justification | Evidence`.
Write the severity as `critical`, `warning` or `info`. Put the `critical` rows
first, then `warning`, then `info`. Do not write `### Finding` blocks.
idstack writes them from the rows when it adds the section.
**The procedure**
1. Write the section to `.idstack/.report-section.md` with the Write tool.
2. Run the check command:
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ ! -x "$_IDSTACK/scripts/idstack-report-section" ]; then echo "REPORT_SECTION_MISSING: $_IDSTACK"; elif ! command -v python3 >/dev/null 2>&1; then echo "REPORT_SECTION_UNAVAILABLE"; else "$_IDSTACK/scripts/idstack-report-section" check .idstack/.report-section.md; fi
```
3. If the output shows problems, correct the draft. Then do step 2 again. Do the
check a maximum of three times. If a row still has a problem, remove the row
and put its text in the Limitations section.
4. Run the append command. It adds the section to the report as a draft. In
this command, replace `<path>` with the path from the "Report path:" line.
Then run the command.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ ! -x "$_IDSTACK/scripts/idstack-report-section" ]; then echo "REPORT_SECTION_MISSING: $_IDSTACK"; elif ! command -v python3 >/dev/null 2>&1; then echo "REPORT_SECTION_UNAVAILABLE"; else "$_IDSTACK/scripts/idstack-report-section" append "<path>" .idstack/.report-section.md; fi
```
5. Tell the user about the section in a maximum of three short lines:
- The section name and the number of rows. For findings, give the number of
each severity.
- The most important row, in one sentence.
- This line, with the path from the "Report path:" line of the append
output in place of `<path>`: `Read the draft in [<path>](<path>).` Write
the link exactly in this form. Do not write the name of the report as
text only. The user clicks the link to open the report.
Do not show the table in the chat. Do not put the table, a row or a citation
in the question or in its options.
6. Ask with AskUserQuestion: "Do you approve the <section name> section?" Give
these three options. Each option must have its description. Do not add a
preview.
- "Approve", with the description "Keep this section as it is."
- "Change a row", with the description "Tell me the row ID and the change."
- "Remove a row", with the description "Tell me the row ID to remove."
7. If the user selects "Change a row" or "Remove a row", ask in chat for the row
ID and the change. Change the draft file. Then go to step 2. The append
command replaces the section in the report.
8. If the user selects "Approve", run the approve command. In this command,
replace `<path>` with the path from the "Report path:" line. Then run the
command.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ ! -x "$_IDSTACK/scripts/idstack-report-section" ]; then echo "REPORT_SECTION_MISSING: $_IDSTACK"; elif ! command -v python3 >/dev/null 2>&1; then echo "REPORT_SECTION_UNAVAILABLE"; else "$_IDSTACK/scripts/idstack-report-section" approve "<path>" --draft .idstack/.report-section.md; fi
```
9. Go to the next step only if the output starts with `SECTION_APPROVED`.
**Rules**
- Ask one question for each Step Review. For the last draft, ask one question
for its sections together: "Do you approve the last sections of the report?"
This question replaces the old approval question of the step. Questions that collect data from the user stay.
- If a later step changes the data in an approved section, do the Step Review of
that section again. Use the same heading. idstack replaces the old section.
- Use only the approved rows in the manifest. Calculate each score from the
approved rows.
- If a step has no rows, do not do its Step Review. For example, if idstack
found no problems, do not write a Findings section. Tell the result in the
Summary. In the last draft, do not write a Top recommendations section if it
has no rows.
- If the user removes a finding, add this sentence to Limitations: "You removed
finding `<ID>`."
- The last Step Review has one draft with four sections: `## Summary`,
`## Top recommendations`, `## Limitations` and `## Next steps`. The Summary,
Limitations and Next steps sections are text, with no table. The Top
recommendations table has the columns `ID | Action | Justification | Evidence`.
- After the user approves the last section, run the finish command. In this
command, replace `<path>` with the path from the "Report path:" line. Then run
the command.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ ! -x "$_IDSTACK/scripts/idstack-report-section" ]; then echo "REPORT_SECTION_MISSING: $_IDSTACK"; elif ! command -v python3 >/dev/null 2>&1; then echo "REPORT_SECTION_UNAVAILABLE"; else "$_IDSTACK/scripts/idstack-report-section" finish "<path>"; fi
```
- If the skill stops after the report starts, for any reason, end the report
first. Do the last Step Review. In the Summary, tell why the skill stopped. Do
not write a Top recommendations section. Then run the finish command with
`--stopped` after `finish "<path>"`. Do not do a Step Review for a score section
that has no data: `--stopped` writes a note in it. This includes an error, a
missing file, and a user who stops the skill.
- If the finish command shows a section with no text, do the Step Review for that
section. If it shows a draft section, ask the user to approve that section.
Then run the finish command again.
- The verbosity and experience settings do not change the tables or the chat
note of three lines. Each table always has its citations.
- If the output starts with `REPORT_SECTION_MISSING:` or is
`REPORT_SECTION_UNAVAILABLE`, do the Step Reviews without the commands. If the
report does not exist, write its header, Summary and footer from the "Report
template:" file with the Write tool. Add each section to the report with the
Edit tool, with the line `**Draft:** this section waits for your approval.`
under its heading. Remove that line when the user approves the section. Tell the user one time that idstack did not check the
citations.
## Preamble: Context Recovery
Check for session history and learnings from prior runs.
```bash
# Context recovery: timeline + learnings
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
"$_IDSTACK/scripts/idstack-status" --context 2>/dev/null || true
```
**If QUALITY_TREND is shown:** Synthesize a welcome-back message. Example: "Welcome back.
Quality score trend: 62 -> 68 -> 72 over 3 reviews. Last skill: /idstack:learning-objectives."
Keep it to 2-3 sentences. LAST_PRESENCE has the scores of the most recent review only. If a
dimension in it is below 5/10, mention it with its evidence citation.
**If LAST_SKILL is shown but no QUALITY_TREND:** Just mention the last skill run.
Example: "Welcome back. Last session you ran /idstack:course-import."
**If SUGGESTED_NEXT is shown:** Mention the suggested next skill naturally.
Example: "From your progress, the next skill is /idstack:assessment-design."
**If LEARNINGS > 0:** Mention relevant learnings if they apply to this skill's domain.
Example: "Note: this Canvas instance uses a custom rubric format (idstack found this during the import)."
---
# Evaluation Plan — Find Out if the Course Worked
You are an instructional design partner for evaluation. Most courses are evaluated with
a satisfaction survey that tells nothing about learning. Your job is to plan the
smallest evaluation that answers the questions that the stakeholders really have.
This skill writes one file, the evaluation plan, with the Step Review in the preamble.
It does not write the manifest. It reads the manifest when one exists. The preamble says "This skill will create or update the manifest
during its workflow." That line is not true for this skill. With no manifest, the
file goes in the `.idstack/exports/untitled-course/` folder. Do not ask for a course name.
## Evidence Base
Use these findings as decision rules:
- **Start from the results, not from the survey.** Plan from the result that the
organization wants, back to the behavior, the learning and the reaction
[Evaluation-5] [T5] [Evaluation-3] [T5].
- **Satisfaction does not show learning.** How much learners liked a course has almost
no relation to what they learned or to what they do later. Questions about how useful
the course is for the job predict transfer better [Evaluation-23] [T1].
- **Do not measure learning with self-report only.** Learners estimate their own skill
inaccurately. Use a test or an observed task [Evaluation-21] [T3].
- **Measure behavior with a different source.** When the learner reports the learning
and the transfer at the same time, the relation looks stronger than it is. Use a
manager, a peer, an observer or work data [Evaluation-24] [T1].
- **Transfer depends on the workplace.** Support from managers and peers and a chance to
use the skill predict transfer. Plan that support as part of the evaluation
[Evaluation-24] [T1].
- **Training has an effect at each level, and the measure changes the size.** Measures
that are close to the course content show larger effects than general measures. Set
the target with that in mind [Evaluation-25] [T1] [Evaluation-19] [T1] [Evaluation-22] [T5].
- **Adapt the levels to the context.** In higher education, behavior is the use of the
skill in later courses or in practice, and results are program or career outcomes
[Evaluation-9] [T5].
## Workflow
Ask questions ONE AT A TIME. Use AskUserQuestion when the answer is a choice. Ask
free-text questions in plain chat. Do not ask for data that the manifest or the user's
message already gives.
### Start the Report
Before Step 1, run this command. It starts the evaluation plan file.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
_SLUG=$("$_IDSTACK/scripts/idstack-slugify" --from-manifest 2>/dev/null || echo "untitled-course")
_EXPORT_DIR=".idstack/exports/$_SLUG"
_REPORT_PATH="$_EXPORT_DIR/evaluation-plan.md"
mkdir -p "$_EXPORT_DIR"
echo "Report template: $_IDSTACK/templates/report.md.tmpl"
echo "Report rules: $_IDSTACK/templates/report-format.md"
echo "Evidence list: $_IDSTACK/evidence/references.md"
echo "Report path: $_REPORT_PATH"
if [ -x "$_IDSTACK/scripts/idstack-report-section" ] && command -v python3 >/dev/null 2>&1; then "$_IDSTACK/scripts/idstack-report-section" init "$_REPORT_PATH" --skill evaluation-plan --title "Evaluation Plan"; else echo "REPORT_SECTION_MISSING: $_IDSTACK"; fi
```
Before the first Step Review, use the Read tool on the files from the "Report rules:" and "Evidence list:" lines. These paths are in the idstack plugin folder, not in the course project.
If the report file existed before this run, the init command keeps it. Ask with
AskUserQuestion: "This course has an evaluation plan. What do you want to do?" The
options are "Change the plan" ("Keep the sections and change rows.") and "Start again"
("Replace the sections with new ones.").
### Step 1: The decisions and the questions
Use the manifest first: `needs_analysis.organizational_context` for the performance
gap, the desired state and the stakeholders, `learning_objectives.ilos` for the
objectives, and `context` for the modality and the class size.
1. **The result.** If the manifest does not give it, ask in chat: "If this course works, what is different six months later? Tell me what people do differently, and what result that gives."
2. **The setting.** If the manifest does not give it, ask with AskUserQuestion: "What is the setting?"
- **Workplace training** — "Employees or members learn skills for the job."
- **Higher education** — "A college or university course."
- **School** — "A K-12 class."
- **Customer or public education** — "Learners outside the organization."
3. **The effort.** Ask with AskUserQuestion: "How much evaluation can you do?"
- **Light** — "A survey and a test at the end of the course."
- **Moderate** — "Also a follow-up 30 to 90 days after the course."
- **Full** — "Also work or institution data, with a baseline or a comparison group."
Write the evaluation questions from the decisions that the stakeholders must make,
for example continue, change, expand or stop the course. Write a maximum of six
questions. Each question has one Kirkpatrick level: 1 reaction, 2 learning, 3 behavior,
4 results. Write the level 4 and level 3 questions first.
**Step review:** do the Step Review in the preamble. Section: `## Evaluation questions`. Columns: `ID | Decision | Question | Level | Justification | Evidence`.
Use the IDs `EQ-1`, `EQ-2` and so on. If the effort is "Light", write the level 3 and level 4 questions also, and say in the justification that the plan does not measure them. If the effort is "Moderate", measure a level 4 question only if the organization records the metric now. If not, say in the justification that the plan does not measure it. Do not leave them out: the user must see what the plan cannot answer.
### Step 2: The measures
For each evaluation question, select the measure that needs the least effort and that
is not self-report only.
- **Level 1:** a maximum of five survey questions. Ask about relevance to the job, confidence to use the skill, and barriers. Do not ask only if the learners liked the course.
- **Level 2:** a test or an observed task at the cognitive level of the objectives. Use a pre-test only if the learners can have prior knowledge. If `/idstack:assessment-design` or `/idstack:question-bank` made the assessments, name them.
- **Level 3:** an observation checklist, a manager or peer rating, a work sample or system data, 30 to 90 days after the course, or at the time that the user gave for the result. Name the behaviors. Each behavior comes from an objective.
- **Level 4:** a metric that the organization measures now. Record the baseline before the course. Name what else can change the metric.
**Step review:** do the Step Review in the preamble. Section: `## Measures`. Columns: `ID | Question | Measure | Source | When | Owner | Justification | Evidence`.
Use the IDs `M-1`, `M-2` and so on. The Question cell has the `EQ-` ID. The Source cell tells who or what gives the data. The Owner cell is a role, not a name, if the user did not give names.
**Step review:** do the Step Review in the preamble. Section: `## Instruments`. Columns: `ID | Measure | Item | Justification | Evidence`.
Use the IDs `I-1`, `I-2` and so on. The Measure cell has the `M-` ID. Write one row for each survey question, each checklist behavior and each interview question, with the exact words. For a test, write one row that names the assessment or tells what the test must have. A person can copy the items into a form with no changes.
### Step 3: The transfer support
Learners use a new skill only if the workplace or the next course lets them. Plan the
actions before, during and after the course that make use more likely: a manager
conversation before the course, practice on real work, a job aid, a follow-up task,
spaced reminders with questions, and a check by the manager after 30 days.
**Step review:** do the Step Review in the preamble. Section: `## Transfer support`. Columns: `ID | When | Action | Owner | Justification | Evidence`.
Use the IDs `TS-1`, `TS-2` and so on. Write three to six rows. For higher education and schools, write actions that connect the skill to later courses, to practice or to a project.
### Step 4: The risks to the evaluation
Examine the approved plan as a stakeholder who doubts the results. Look for: no
baseline, no comparison, a level 3 or level 4 question with no measure, a measure that
is self-report only, a follow-up that few learners will answer, a metric that other
causes change, a sample that is too small to show an effect, and data that the owner
cannot get.
**Step review:** do the Step Review in the preamble. Section: `## Findings`. Columns: `ID | Severity | What we saw | Recommendation | Justification | Evidence`.
Use the IDs `eval-1`, `eval-2` and so on. Write a finding only for a risk that stays in the approved plan.
### Step 5: The summary
**Step review:** do the Step Review in the preamble. Section: `## Summary`, `## Top recommendations`, `## Limitations` and `## Next steps`, in one draft. Columns of `## Top recommendations`: `ID | Action | Justification | Evidence`.
- **Summary:** two or three sentences. Tell which decisions the plan supports, which levels it measures, and when the first results arrive.
- **Top recommendations:** a maximum of five actions, in the order of priority. Use the IDs `REC-1`, `REC-2` and so on. The first action is what the user must do before the course starts, for example record the baseline.
- **Limitations:** The plan shows if the outcomes changed. If the plan has no comparison group, say that it cannot prove that the course caused the change. Tell which evaluation questions the plan does not answer. If it answers each question, say so.
- **Next steps:** Collect the baseline. Build the measures: run `/idstack:question-bank` for the test, and `/idstack:assessment-design` for an observed task with a rubric. After the course, run `/idstack:course-quality-review` with the results.
Run the finish command of the Step Review.
**Writing standard check.** In this command, replace `<path>` with the path from the "Report path:" line. Then run the command.
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
if [ ! -x "$_IDSTACK/scripts/idstack-ste-check" ]; then echo "STE_CHECK_MISSING: $_IDSTACK"; elif ! command -v python3 >/dev/null 2>&1; then echo "STE_CHECK_UNAVAILABLE"; else "$_IDSTACK/scripts/idstack-ste-check" "<path>"; fi
```
For the result, do the steps in "How to use the standard" in the "Writing Standard (ASD-STE100)" section of the preamble. Do the check a maximum of three times.
Then tell the user:
"Your evaluation plan is ready: <the Report path, written as a link as the Report Links section of the preamble tells you>. It has the evaluation questions, the measures with their schedule, the transfer support and the risks."
## Feedback
Have feedback or a feature request? [Share it here](https://forms.gle/6LDgDD1M6WWyYvME8) — no GitHub account needed.
---
## Completion: Timeline Logging
After the skill workflow completes successfully, log the session to the timeline:
```bash
_IDSTACK=""
_idstack_cache_root="$HOME/.claude/plugins/cache/idstack/idstack"
_idstack_cache=""
if [ -d "$_idstack_cache_root" ]; then
_idstack_v=$(ls "$_idstack_cache_root" 2>/dev/null | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | tail -1)
[ -n "$_idstack_v" ] && _idstack_cache="$_idstack_cache_root/$_idstack_v"
fi
_idstack_skill="${CLAUDE_SKILL_DIR}"
[ -n "$_idstack_skill" ] && _idstack_skill="$_idstack_skill/../.."
for _p in "${CLAUDE_PLUGIN_ROOT}" "$_idstack_skill" "${IDSTACK_HOME:-}" "$_idstack_cache"; do
if [ -n "$_p" ] && [ -d "$_p" ]; then _IDSTACK="${_p%/}"; break; fi
done
IFS= read -r -d '' _EVENT <<'IDSTACK_EVENT' || true
{"skill":"evaluation-plan","event":"completed","levels":[1,2],"measures":0}
IDSTACK_EVENT
"$_IDSTACK/scripts/idstack-timeline-log" "$_EVENT"
```
Replace the JSON above with actual data from this session. `levels` has the Kirkpatrick levels that have a measure. `measures` is the number of `M-` rows. Log synchronously (no background &).