topic-modeling · diff
git:20260909.08bf795 to git:20261008.697c89f
81 added, 69 removed. Audit A to A.
---
disable-model-invocation: true
name: topic-modeling
- description: Specifies and diagnoses structural topic models for survey and experimental text. Covers STM, LDA, and BERTopic choice, preprocessing consequences, prevalence and content formulas with spectral initialization and recorded seed, topic-count selection across semantic coherence, exclusivity and FREX, held-out likelihood, and residuals, interpretation and validation against representative documents, robustness checks, and DA-RT-compliant reporting. Use when the user has open-ended responses or another corpus and asks what topics are present, how many topics to use, about STM, searchK, coherence, or FREX, or wants topic prevalence compared across treatment arms or countries. Fixed-codebook coding goes to text-classification.
+ description: Advises on setting up and validating topic models for survey and experimental text. Covers the discovery-versus-measurement question, choosing STM, LDA, or BERTopic, preprocessing consequences, prevalence and content formulas, topic-count selection across coherence, exclusivity and FREX, held-out likelihood, and residuals, validation of topics against human judgment, robustness, and DA-RT-compliant reporting. Use when the user has open-ended responses or another corpus and asks what topics are present, how many topics to use, about STM, searchK, coherence, or FREX, or wants topic prevalence compared across treatment arms or countries. Fixed-codebook coding goes to text-classification.
argument-hint: '[describe your corpus and research question]'
---
# Topic Modeling for Survey and Experimental Text Data
## Instructions
- ### 1. Model Selection
+ Help the user design the analysis before fitting anything. Work through the decisions below in order, explain what each choice changes and which standard governs it, and record the choice with its rationale. Every one of them — model, preprocessing, covariates, K, labels — shapes which topics appear, so each belongs in the methods section.
- - Default to Structural Topic Models (STM) when analyzing text from surveys or experiments. STM incorporates document-level metadata — treatment conditions, respondent demographics, country — directly into estimation, allowing prevalence and content to vary with covariates (Roberts et al. 2014).
- - Use standard LDA only when no document-level covariates are needed and the corpus is large enough for unsupervised discovery (Blei, Ng & Jordan 2003).
- - Consider BERTopic when working with short texts where word co-occurrence is sparse, or when multilingual embedding-based similarity is required (Grootendorst 2022). BERTopic clusters document embeddings via HDBSCAN, which yields hard single-topic-per-document assignments (unlike STM/LDA mixed membership), c-TF-IDF topic words that can be unstable on small corpora, and no native covariate framework. Use it when embedding-based semantic similarity matters more than mixed-membership structure or covariate estimation.
- - Recognize the distinction between topic categorization and attitude inference: topic models reveal what respondents *discussed*, not what they *believe*. If the goal is inferring latent attitudes rather than categorizing surface content, complement topic modeling with an embeddings-based scaling of contextually common words or a separate supervised classifier (Hobbs & Green 2025; see also the companion `text-classification` skill).
+ ### 1. Discovery or Measurement?
- ### 2. Preprocessing
+ Settle this first; it decides the rest.
- - Make preprocessing decisions explicit and justify each choice. Preprocessing is not neutral — stemming, stopword removal, and term-frequency thresholds all affect which topics emerge (Denny & Spirling 2018).
- - Lowercase all text. Remove punctuation and numbers unless they carry substantive meaning in the domain.
- - Remove stopwords using a standard list, but inspect the list for domain-relevant terms that should be retained (e.g., "foreign" in immigration research).
- - Apply stemming only after checking that it does not collapse substantively distinct terms. Compare results with and without stemming as a robustness check (Denny & Spirling 2018).
- - Set a lower-frequency threshold to prune very rare terms, and express it as a fraction of documents rather than an absolute count so it scales with corpus size. For small open-ended corpora a fixed count of 2–5 documents is often appropriate; for larger corpora, pruning terms that appear in fewer than roughly 0.5%–1% of documents is a common starting point (Grimmer & Stewart 2013). In `stm`, use `plotRemoved()` to visualize how many documents and terms are dropped across candidate thresholds before committing to one, then pass it to `prepDocuments()`. Report the threshold chosen and the counts of terms and documents retained.
- - For translated text, preprocess after translation. Note that translation artifacts (e.g., inconsistent phrasing across translators) may affect topic coherence — document any translation pipeline.
+ - Topic models are discovery tools: they show what respondents *discussed*. If the user already has categories and wants each document coded against them, that is classification — use `text-classification`.
+ - If the goal is to infer latent attitudes rather than categorize surface content, topics alone will not do it; complement them with an embeddings-based scaling of contextually common words or a supervised classifier (Hobbs & Green 2025).
+ - A common sound design is two-stage: discover categories with a topic model, then write those categories into a codebook and measure them with validated supervised classification for confirmatory tests (Grimmer & Stewart 2013).
+ - If topic-prevalence effects will be tested as hypotheses, pre-register them (`pre-registration-writing`) so `estimateEffect()` output is confirmatory rather than exploratory by default (Nosek et al. 2018).
- ### 3. Model Specification
+ ### 2. Model Choice
- - Specify the prevalence formula to include theoretically relevant covariates. For survey experiments, include treatment conditions; for cross-national data, include country. Example: `prevalence = ~ treatment + country` (Roberts et al. 2014). For experimental applications, pre-register the prevalence-covariate hypotheses (see the companion `pre-registration-writing` skill) so that `estimateEffect()` outputs are confirmatory rather than exploratory by default (Nosek et al. 2018).
- - Optionally specify a content formula when you expect the *words* associated with a topic (not just its prevalence) to vary by covariate — e.g., different framings of the same topic across countries or treatment arms. Content covariates parameterize word distributions via SAGE-style deviations from a baseline, which complicates direct comparison of β across groups; inspect group-specific vocabulary with `sageLabels()` and interpret with care (Roberts, Stewart & Tingley 2014; 2019).
- - Use spectral initialization (`init.type = "Spectral"`) for reproducibility. Spectral initialization is deterministic on a given machine/BLAS configuration, unlike random initialization which requires multiple runs; cross-machine numerical precision can still produce minor differences, so record the hardware/OS when sharing replication materials (Roberts, Stewart & Tingley 2019).
- - Set a random seed and record it regardless of initialization method. Also record the `stm` package version and R session info for DA-RT-compliant replication.
+ - **STM** is the default for survey and experimental text. It brings document metadata — treatment arm, demographics, country — into estimation so prevalence and content can vary with covariates (Roberts et al. 2014).
+ - **LDA** fits when no document-level covariates are needed and the corpus is large enough for unsupervised discovery (Blei, Ng & Jordan 2003).
+ - **BERTopic** fits short texts where word co-occurrence is sparse, or multilingual corpora where embedding similarity is needed (Grootendorst 2022). It clusters embeddings with HDBSCAN, so each document gets one hard topic (no mixed membership), c-TF-IDF topic words can be unstable on small corpora, and there is no native covariate framework.
- ### 4. Selecting the Number of Topics
+ ### 3. Preprocessing
- - Do not rely on a single metric. Evaluate candidate models across a range of K values (e.g., K = 5 to 30) using multiple diagnostics: semantic coherence, exclusivity, held-out likelihood, and residuals (Roberts, Stewart & Tingley 2019). In `stm`, `searchK()` sweeps over K and returns all four diagnostics at once; `selectModel()` fits multiple initializations at a fixed K and returns the coherence-exclusivity frontier (useful when spectral initialization is not being used).
- - Semantic coherence measures whether high-probability words within a topic co-occur in the corpus. Higher is better, but coherence alone favors very common topics (Mimno et al. 2011). When a topic scores poorly, inspect the failure mode: Mimno et al. describe *chained* (two concepts linked by a shared word), *intruded* (unrelated words mixed in), *random* (no coherent connections), and *unbalanced* (mixes very general and very specific terms) — the diagnosis shapes whether the fix is K, preprocessing, or covariate specification.
- - Exclusivity measures whether high-probability words are distinctive to one topic. The `stm` package reports FREX, a weighted harmonic mean of frequency and exclusivity ranks (default weight ω = 0.7). The coherence-FREX frontier identifies models that balance both (Roberts, Stewart & Tingley 2019).
- - Treat held-out likelihood as one input among several, not the deciding metric. Chang et al. (2009) show that predictive likelihood can be *negatively* correlated with human interpretability — topic models that fit held-out words better are not necessarily more semantically coherent. When coherence, FREX, and held-out likelihood disagree, prefer interpretability (Chang et al. 2009).
- - After narrowing candidates statistically, read the top words and representative documents for each topic. The final K should be interpretable — topics should be substantively meaningful and distinguishable from one another (Chang et al. 2009).
- - Report the range of K values considered, the diagnostics used, and the rationale for the final choice.
+ Preprocessing is not neutral: stemming, stopword removal, and frequency thresholds all change which topics emerge (Denny & Spirling 2018). Justify each choice.
- ### 5. Interpretation and Validation
+ - Lowercase; drop punctuation and numbers unless they carry meaning in the domain.
+ - Use a standard stopword list, but inspect it for domain terms that should stay (e.g., "foreign" in immigration research).
+ - Stem only after checking that it does not merge substantively distinct terms, and compare results with and without stemming (Denny & Spirling 2018).
+ - Prune rare terms with a threshold that scales with the corpus: a fixed 2-5 documents for small open-ended corpora; roughly 0.5-1% of documents for larger ones (Grimmer & Stewart 2013). In `stm`, `plotRemoved()` shows terms and documents dropped across candidate thresholds before you pass one to `prepDocuments()`. Report the threshold and the retained counts.
+ - For translated text, preprocess after translation and document the translation pipeline; inconsistent translation can degrade coherence.
- - For each topic, report the top 10-15 words by probability and by FREX (frequency-exclusivity weighted). FREX words are often more interpretable because they are both common within the topic and rare outside it (Roberts, Stewart & Tingley 2019).
- - Extract and read 3-5 representative documents per topic using `findThoughts()` or equivalent. Do not interpret topics from word lists alone — the documents provide essential context.
- - Assign short substantive labels to each topic based on both word lists and representative documents. Labels should be descriptive (e.g., "economic contribution concerns") not technical (e.g., "Topic 7").
- - Estimate covariate effects on topic prevalence using `estimateEffect()`. For experimental data, test whether treatment conditions significantly shift which topics respondents discuss. For cross-national data, test whether topic prevalence differs by country. Plot these effects with confidence intervals (Roberts et al. 2014). When testing many topics against a treatment, report corrections for multiple comparisons (e.g., FDR) alongside uncorrected estimates.
- - To guard against spurious treatment-on-topic effects, use `permutationTest()` (Roberts, Stewart & Tingley 2014; 2019). The function permutes the treatment label and refits the STM, returning a null distribution of topic-level treatment effects against which the observed effect is compared — essentially a randomization inference check for `estimateEffect()`.
- - For construct validation in survey experiments: check whether topics related to the intended construct (e.g., civic engagement) show higher prevalence among respondents exposed to relevant treatment conditions. A null prevalence effect is *suggestive* of a construct-validity problem but does not establish one — rule out alternative explanations first, including insufficient statistical power, a suboptimal K that splits or merges the construct topic, a content-covariate effect (the manipulation shifted *how* the topic was framed rather than *whether* it was discussed), and preprocessing choices that dropped construct-relevant vocabulary.
+ ### 4. Model Specification
- ### 6. Robustness and Sensitivity
+ - Put theoretically relevant covariates in the prevalence formula — treatment arm for experiments, country for cross-national data: `prevalence = ~ treatment + country` (Roberts et al. 2014).
+ - Add a content formula only when you expect the *words* of a topic, not just its prevalence, to differ by covariate (different framings across countries or arms). Content covariates parameterize SAGE-style deviations from a baseline, which complicates comparing β across groups; inspect group vocabulary with `sageLabels()` (Roberts, Stewart & Tingley 2014; 2019).
+ - Use spectral initialization (`init.type = "Spectral"`). It is deterministic on a given machine and BLAS configuration; cross-machine precision can still differ slightly, so record hardware and OS (Roberts, Stewart & Tingley 2019).
+ - Set and record a seed regardless of initialization, plus the `stm` version and R session info.
- - Run the model at neighboring values of K (e.g., K-2, K+2) to check whether substantive conclusions are sensitive to topic count.
- - Compare results with and without stemming as a preprocessing robustness check (Denny & Spirling 2018).
- - If using STM, compare results with and without covariates to assess how much the covariate structure shapes the topic solution.
- - For experimental data, verify that topic assignments are not driven by a small number of high-frequency terms. Inspect whether removing the single most common term in a topic changes its interpretation.
+ ### 5. Selecting the Number of Topics
- ### 7. Reporting
+ - Use several diagnostics across a range of K (e.g., 5-30): semantic coherence, exclusivity, held-out likelihood, and residuals (Roberts, Stewart & Tingley 2019). `searchK()` returns all four; `selectModel()` fits several initializations at a fixed K and returns the coherence-exclusivity frontier (useful only with non-spectral initialization).
+ - Coherence rewards topics whose top words co-occur but favors very common topics (Mimno et al. 2011). For a poor topic, diagnose the failure — *chained* (two concepts joined by a shared word), *intruded* (unrelated words mixed in), *random*, or *unbalanced* (general and specific terms mixed) — because the diagnosis says whether to change K, preprocessing, or covariates.
+ - Exclusivity: `stm` reports FREX, a weighted harmonic mean of frequency and exclusivity ranks (default ω = 0.7). The coherence-FREX frontier marks models that balance both (Roberts, Stewart & Tingley 2019).
+ - Held-out likelihood is one input, not the decider: it can correlate *negatively* with human interpretability (Chang et al. 2009). When the metrics disagree, prefer interpretability.
+ - After narrowing statistically, read top words and representative documents for each candidate. The final K must give substantively meaningful, distinguishable topics.
+ - Report the K range, the diagnostics, and the rationale for the final choice.
- - Report: preprocessing pipeline (stopwords, stemming, frequency thresholds, counts of documents and terms retained after `prepDocuments`), number of topics and selection rationale, initialization method, convergence diagnostics (max EM iterations and final variational lower bound), random seed, `stm` package version, and R session info. For the broader reporting checklist see the companion `methods-reporting` skill.
- - Present a topic summary table with: topic number, label, top FREX words, prevalence (overall and by covariate group), and 1-2 example documents.
- - When reporting covariate effects on prevalence, present coefficient estimates with confidence intervals. Visualize cross-group differences (e.g., side-by-side prevalence plots by country or treatment).
- - Distinguish discovery from confirmation: topic modeling is exploratory. State explicitly which findings were anticipated and which emerged from the data. Treating every topic-level comparison as a hypothesis test without that distinction re-creates the researcher-degrees-of-freedom problem documented in the false-positive literature (Simmons, Nelson & Simonsohn 2011; Nosek et al. 2018). If using topics to validate constructs, frame findings in terms of whether the intended constructs appeared, not whether the model "confirmed" a hypothesis (Grimmer & Stewart 2013).
- - Make replication materials available: analysis code, preprocessing decisions, and random seed. If the corpus cannot be shared, provide the topic-term matrix and document-topic proportions (Grimmer & Stewart 2013).
+ ### 6. Interpretation and Validation Against Human Judgment
+ Statistical fit does not show that a topic measures what its label claims. Validate the topics as measures (Grimmer & Stewart 2013).
+
+ - For each topic, report the top 10-15 words by probability and by FREX (Roberts, Stewart & Tingley 2019).
+ - Read 3-5 representative documents per topic (`findThoughts()`). Word lists alone are not enough to interpret a topic.
+ - Label topics descriptively ("economic contribution concerns"), from both word lists and documents. Write each label with a one-sentence definition so the labels work as a codebook others can apply.
+ - Test the labels against human coders. Have coders blind to the model assign a random sample of documents to the labeled topics (or to none) and report agreement with the model's dominant topic, using Krippendorff's α or κ (Krippendorff 2019; Landis & Koch 1977). Human word-intrusion and topic-intrusion tasks test whether topics are interpretable at all (Chang et al. 2009). Low agreement means the labels or K need revision before any substantive claim.
+ - Estimate covariate effects on prevalence with `estimateEffect()` and plot them with confidence intervals (Roberts et al. 2014). When testing many topics against a treatment, report FDR or similar corrections alongside uncorrected estimates.
+ - Guard against spurious treatment effects with `permutationTest()`, which permutes the treatment label and refits to give a randomization-inference null for each topic's effect (Roberts, Stewart & Tingley 2014; 2019).
+ - For construct validation in survey experiments, check whether topics tied to the intended construct rise among respondents exposed to the relevant treatment. A null is *suggestive* of a construct-validity problem, not proof: first rule out low power, a K that splits or merges the construct topic, a content-covariate effect (the treatment changed *how* the topic was framed, not *whether* it was discussed), and preprocessing that dropped construct vocabulary.
+
+ ### 7. Robustness
+
+ - Refit at neighboring K (K−2, K+2) and check whether substantive conclusions hold.
+ - Compare with and without stemming (Denny & Spirling 2018).
+ - For STM, compare with and without covariates to see how much they shape the solution.
+ - Check that topics are not driven by one high-frequency term: remove a topic's most common term and see whether its interpretation changes.
+ - Inspect topic correlations with `topicCorr()`.
+
+ ### 8. Reporting
+
+ - Preprocessing pipeline (stopwords, stemming, thresholds, documents and terms retained after `prepDocuments`), K and its rationale, initialization, convergence (max EM iterations, final variational bound), seed, `stm` version, R session info. For the broader checklist see `methods-reporting`.
+ - A topic summary table: number, label and definition, top FREX words, prevalence overall and by covariate group, and 1-2 example documents.
+ - Human validation results: coder procedure, sample size, and agreement with the model's topic assignments.
+ - Covariate effects as estimates with confidence intervals, with side-by-side prevalence plots by group.
+ - Discovery versus confirmation: say which findings were anticipated and which emerged. Treating every topic comparison as a hypothesis test recreates the researcher-degrees-of-freedom problem (Simmons, Nelson & Simonsohn 2011; Nosek et al. 2018). When topics validate constructs, report whether the intended constructs appeared, not that the model "confirmed" a hypothesis (Grimmer & Stewart 2013).
+ - Replication materials: code, preprocessing decisions, seed. If the corpus cannot be shared, provide the topic-term matrix and document-topic proportions (Grimmer & Stewart 2013).
+
## Quality Checks
- - [ ] Model type justified (STM vs. LDA vs. BERTopic) given data structure and research question
- - [ ] Preprocessing decisions documented and justified (stemming, stopwords, thresholds)
- - [ ] Number of topics selected using multiple diagnostics (`searchK` / `selectModel` over coherence, FREX/exclusivity, held-out likelihood, residuals) plus substantive interpretability
- - [ ] Held-out likelihood vs. coherence disagreement resolved in favor of interpretability (Chang et al. 2009)
- - [ ] Covariates included in prevalence (and, where justified, content) formula match the experimental or observational design; `sageLabels()` inspected when a content covariate is used
- - [ ] Each topic interpreted using both word lists (probability + FREX) and representative documents via `findThoughts()`
- - [ ] Covariate effects on topic prevalence estimated with `estimateEffect()` and reported with confidence intervals; multiple-comparison adjustment reported when testing many topics
- - [ ] For experimental applications, `permutationTest()` run against the treatment covariate
- - [ ] Robustness checks performed (alternative K, preprocessing variants, topic correlations via `topicCorr()`)
- - [ ] Topics labeled descriptively based on content, not just numbered
- - [ ] Discovery vs. confirmation framing made explicit; confirmatory topic-prevalence hypotheses pre-registered where feasible
- - [ ] Random seed, initialization method, convergence diagnostics, and `plotRemoved()` / `prepDocuments` retention counts reported
- - [ ] Replication materials (code, preprocessing pipeline, seed, `stm` version, R session info) available or planned
+ - [ ] Discovery vs. measurement settled; categories already known routed to `text-classification`
+ - [ ] Model type (STM, LDA, BERTopic) justified by data structure and question
+ - [ ] Preprocessing choices documented and justified; retained counts reported
+ - [ ] K chosen from multiple diagnostics (`searchK`/`selectModel`) plus interpretability; held-out vs. coherence conflicts resolved toward interpretability (Chang et al. 2009)
+ - [ ] Prevalence (and, if justified, content) covariates match the design; `sageLabels()` inspected for content covariates
+ - [ ] Topics interpreted from probability and FREX words plus `findThoughts()` documents, and labeled with definitions
+ - [ ] Topic labels validated against blind human coding with agreement reported (Grimmer & Stewart 2013)
+ - [ ] Prevalence effects from `estimateEffect()` with confidence intervals and multiple-comparison adjustment
+ - [ ] `permutationTest()` run for experimental treatment effects
+ - [ ] Robustness: neighboring K, preprocessing variants, covariate structure, `topicCorr()`
+ - [ ] Discovery vs. confirmation stated; confirmatory prevalence hypotheses pre-registered where feasible
+ - [ ] Seed, initialization, convergence, `stm` version, session info, and replication materials archived
## Example
- A minimal STM workflow for a survey experiment in which `df` holds one row per respondent with columns `open_ended_responses` (text) and `treatment_arm` (factor). The four stages — (a) preprocess, (b) sweep K, (c) refit with multiple initializations, (d) estimate treatment effects — map directly onto §§2–5. The four diagnostic functions (`searchK`, `selectModel`, `permutationTest`, `topicCorr`) should all appear in the final analysis script.
+ A minimal STM pattern for a survey experiment where `df` has one row per respondent with `open_ended_responses` (text) and `treatment_arm` (factor). Adapt it; stages (a)-(d) map onto Sections 3-6.
```r
- # Minimal STM workflow: survey experiment with open-ended responses.
library(stm)
set.seed(20260418)
- # (a) Preprocess and prune rare terms (see §2).
+ # (a) Preprocess and choose the rare-term threshold (Section 3).
processed <- textProcessor(df$open_ended_responses, metadata = df)
plotRemoved(processed$documents, lower.thresh = seq(1, 20, by = 2))
prepped <- prepDocuments(processed$documents, processed$vocab,
processed$meta, lower.thresh = 5)
- # (b) Sweep K over candidate values (coherence, exclusivity, held-out, residuals).
- k_search <- searchK(prepped$documents, prepped$vocab,
- K = c(5, 10, 15, 20),
- prevalence = ~ treatment_arm,
- data = prepped$meta, init.type = "Spectral")
+ # (b) Sweep K: coherence, exclusivity, held-out likelihood, residuals (Section 5).
+ k_search <- searchK(prepped$documents, prepped$vocab, K = c(5, 10, 15, 20),
+ prevalence = ~ treatment_arm, data = prepped$meta,
+ init.type = "Spectral")
plot(k_search)
- # (c) At the chosen K, refit with multiple RANDOM initializations; pick off the
- # frontier. Do not pass init.type = "Spectral" here: spectral init is
- # deterministic, so all runs would be identical and the frontier degenerate.
- k_chosen <- 10
+ # (c) At the chosen K, compare random initializations on the frontier.
+ # Spectral init is deterministic, so selectModel needs a random init here.
+ k_chosen <- 10
candidates <- selectModel(prepped$documents, prepped$vocab, K = k_chosen,
prevalence = ~ treatment_arm, data = prepped$meta,
runs = 20, init.type = "LDA")
plotModels(candidates)
fit <- candidates$runout[[1]]
- # (d) Estimate treatment effects on topic prevalence (§5); then permutationTest / topicCorr.
+ # (d) Treatment effects on prevalence (Section 6); follow with permutationTest and topicCorr.
effects <- estimateEffect(1:k_chosen ~ treatment_arm, fit,
metadata = prepped$meta, uncertainty = "Global")
summary(effects)
```