--- name: feedback-synthesizer version: 1.2.0 author: Personify Labs description: > Turns raw survey results (scores + open-text comments) into a scored, statistically-tested, thematically-coded synthesis — and connects the two. Computes instrument-correct scores (SUS formula, NPS calc, SUPR-Q percentile), real confidence intervals, and significance testing across waves or segments. Codes qualitative comments into themes (emergent per study, converging toward a fixed taxonomy per product over time) and cross-references low scorers against recurring themes. Owns severity/priority scoring. Can run standalone on any pile of feedback (reviews, support tickets, open-ended comments) — doesn't require a survey-architect-built survey as input. Trigger when someone pastes raw survey data, a CSV of responses, or a pile of qualitative feedback and wants it turned into findings. Entry point 2 of the research loop (see research-loop). Bundles scripts/scoring.py for deterministic instrument math (run it, don't calculate by hand) and evals/ for regression testing after any edit — run evals/test_scoring.py after touching scoring.py, and check evals/qualitative_cases.md after touching this file's prose. tags: - user-research - ux-research - qualitative-analysis - statistics - thematic-coding license: CC-BY 4.0 --- # Feedback Synthesizer Takes raw numbers and raw comments and produces one connected picture, not two separate reports that happen to sit in the same doc. Most teams either report a score or summarize comments. This skill's whole job is to make the two inform each other — a score drop with no thematic explanation isn't a finding yet. ----- ## Step 0: What are you looking at? Identify the input type before doing anything: - **From survey-architect** (a `/research//01-survey-spec.json` exists) → read it for instrument, scoring formula, and expected n. Use it, don't re-derive. - **Standalone raw data** (pasted CSV, app store reviews, support ticket export, no spec file) → identify the instrument from the data itself if it's a known format (SUS items, an 0–10 NPS column, etc.), or treat it as unstructured qualitative feedback if there's no scale attached. Say explicitly which case you're in — don't silently assume a scale that isn't there. ----- ## Step 1: Quantitative scoring **Run `scripts/scoring.py` — do not compute these by hand.** Every instrument formula, the confidence interval, and the significance tests are exact, deterministic calculations with no judgment involved, which is exactly the kind of task that should never depend on an LLM doing arithmetic in its head. Call the matching function for the instrument in use: | Instrument | Function | |---|---| | SUS | `sus_score(responses)` | | UMUX | `umux_score(responses)` | | UMUX-Lite | `umux_lite_score(item1, item2)` | | NPS | `nps_score(responses)` | | CSAT | `csat_top_two_box(responses)` or `csat_mean(responses)` — state which | | SEQ / CES | `seq_score(responses)` / `ces_score(responses)` | | SUPR-Q | `supr_q_score(five_point_items, nps_item_0_to_10)` | If the instrument doesn't map to one of these (a custom item set), say so explicitly rather than forcing it through the nearest function. ### Confidence intervals and significance testing Also not optional, also run through the script — call `confidence_interval(scores)` and, when comparing against a benchmark or prior wave, `significance_vs_benchmark(scores, benchmark)` or `compare_waves(scores_a, scores_b)`. Never report a headline score without its CI, and never call a difference "significant" without having actually run the test. State the result in plain language alongside the numbers ("the 6-point drop is outside the margin of error — this is a real change" vs. "the 3-point drop is within the CI — could be noise"). If `scripts/scoring.py` isn't available in the execution environment for some reason, say so explicitly before falling back to manual calculation — don't silently compute by hand and present it as if the script ran. ### Small-sample handling If n falls below the sample-size floor survey-architect specified (or below the general rule-of-thumb minimums if there's no spec file): - **Compute the score anyway.** Never refuse. - **Widen the CI accordingly** and say so explicitly. - **Flag it loudly** at the top of the synthesis output, not buried in a footnote: `⚠️ LOW CONFIDENCE — n=, below recommended minimum of . Treat this score as directional, not conclusive.` - If the gap is large enough that the score is essentially unusable for the stated decision (see survey-architect's learning goal), **kick back to survey-architect** with a note to extend collection — this is the loop-back condition, not a silent proceed. ----- ## Step 2: Qualitative coding - **New product or first study:** code comments with an emergent/open approach — let themes surface from the data, don't force them into a predefined bucket. - **Product with prior studies:** check `/research/_benchmarks/.md` for the existing theme taxonomy. Reuse those theme labels where the data fits; only add a new theme label if genuinely nothing existing captures it. This is how the taxonomy converges over time instead of resetting every study. - For each theme, report: frequency (n and %), representative paraphrased examples (never verbatim quotes — see copyright note below), and whether it skews toward low or high scorers. **Never quote respondents verbatim in synthesis output** — paraphrase every comment. This matters for two reasons: respondent privacy in smaller samples, and because verbatim quotes get copy-pasted into decks and lose the aggregation context. ----- ## Step 3: Cross-reference qual and quant This is the step most synthesis work skips. For every notable score change (a wave-over-wave drop, a below-benchmark score, a segment that scores lower than others): 1. Pull the subset of open-text comments from respondents in that score band. 2. Check whether a theme is disproportionately represented in that subset vs. the overall theme distribution. 3. If yes, state the connection explicitly: "SUS dropped 8 points this wave (95% CI: −11 to −5, statistically significant vs. wave 2). 60% of respondents scoring below 60 mentioned the new checkout step, vs. 12% of respondents scoring above 80." 4. If no clear theme explains a score change, say that too — an unexplained score movement is itself a finding, not a gap to paper over. ----- ## Step 4: Severity and priority scoring This skill owns severity/priority — not the reporter. **Once you have the raw counts (how many respondents in each score band mentioned the theme), run `scoring.severity_tier(...)` rather than eyeballing the percentages** — this is the exact calculation that was wrong in v1.0.0 (diluted by whole-sample frequency instead of within-band frequency), and it's now a tested function specifically so that bug can't quietly reappear in prose form. Getting the raw counts right — which respondents go in which band, whether a comment belongs to the theme — is still your judgment call; the arithmetic on top of those counts is not. Score each theme on two axes: - **Frequency:** what % of respondents *in the affected score band* raised it — not what % of the whole sample raised it. A theme mentioned by 3 of 4 low scorers is high-frequency even if it's only 3 of 14 respondents overall. Measuring frequency against the full sample dilutes a concentrated signal and can demote a real driver of low scores to a lower tier than it deserves. Also report the plain overall % for context, but don't let it drive the tier. - **Impact:** does it correlate with low scores / high effort / low completion — i.e., is the theme's presence skewed toward one score band vs. another, per the cross-reference in Step 3 Combine into a simple priority tier (not a false-precision numeric score): - **Critical** — high frequency *within the affected band* (roughly ≥50% of respondents in that band) **and** a clear skew vs. the unaffected band (that band's rate is meaningfully higher than the rest of the sample's rate for the same theme) - **Notable** — meets one of the two Critical conditions, not both (e.g., mentioned by a majority of low scorers but also shows up somewhat among high scorers, or a clear skew but only a small minority of the affected band mentions it) - **Minor** — low frequency within the affected band and no meaningful skew vs. the rest of the sample State the reasoning for each tier assignment explicitly, including both the within-band and overall frequency numbers — don't just output the label, and don't let a low overall frequency read as an argument against a tier the within-band numbers support. **Coding themes from judgment, not keyword matching.** A comment can belong to a theme without using any of that theme's obvious keywords — e.g., "had to re-enter my card info twice" belongs in a payment/checkout friction theme even though it never says "payment." Read each comment for what it's actually describing, not for whether it contains a specific word. A literal string search will systematically undercount themes and produce frequency numbers that look lower than reality. ----- ## Step 5: Output Write `/research//02-synthesis.md` (or `.json` for the machine-readable version consumed by research-reporter) containing: scores + CIs + significance results, theme list with frequency/severity, qual-quant cross-references, and the low-confidence flag if triggered. Append the wave's headline score(s) to `/research/_benchmarks/.md` so the next study's comparison has something to compare against. Include date, instrument, score, CI, and n. ----- ## What this skill never does - Reports a score without its confidence interval - Calls a numeric difference "significant" without running the actual test - Refuses to score a small sample — flags it loudly and proceeds instead - Silently proceeds when the sample is so small the stated learning goal can't be answered — kicks back to survey-architect instead - Quotes respondents verbatim - Forces new data into an existing theme taxonomy when it genuinely doesn't fit - Reports a score movement and a theme in the same study without checking whether they're actually connected - Assigns severity/priority without stating the frequency + impact reasoning behind the tier - Codes themes by keyword/string matching instead of reading each comment for what it actually describes - Measures a theme's frequency only against the whole sample when deciding severity — always check frequency within the affected score band first, since that's what a diluted overall % can hide ----- ## Quick reference | Situation | Action | |---|---| | n below recommended minimum | Score it, widen CI, flag loudly | | n so small the learning goal is unanswerable | Kick back to survey-architect | | Score drop, no theme correlation found | Report the drop as unexplained, don't force a story | | Score drop + strong theme correlation | State both together as one finding | | New product, first study | Emergent coding | | Existing product with prior studies | Reuse taxonomy from benchmark file, extend only if needed | | Standalone data, no spec file | Say so explicitly, identify instrument from data structure |