AGENTS.md · diff
git:20261116.35ddfe8 to git:20261117.5743bff
1 added, 1 removed. Audit A to A.
What this update changes in the checks
No check changed: every fixed check gives the same answer before and after this update.
A tripped check is a label, not a verdict: read the matched text and the lines below. This week's updates that newly trip a check.
# last30days Skill
Agent Skills package for researching any topic across Reddit, X, YouTube, and web. Installable across Claude Code (most common host), Codex, Cursor, GitHub Copilot, Gemini CLI, Grok (xAI), and 50+ other [Agent Skills](https://agentskills.io) hosts. Python scripts with multi-source search aggregation.
## Structure
- `skills/last30days/SKILL.md` — canonical skill definition / runtime spec the model reads when the slash command fires; its reference-routing table gates the stage and mode instructions in `skills/last30days/references/`
- `skills/last30days/scripts/last30days.py` — main research engine
- `skills/last30days/scripts/lib/` — search, enrichment, rendering modules
- `skills/last30days/scripts/lib/vendor/bird-search/` — vendored X search client
- `docs/solutions/` — documented solutions to past problems (bugs, best practices, workflow patterns), organized by category with YAML frontmatter (`module`, `tags`, `problem_type`)
- `CONCEPTS.md` — shared domain vocabulary (Skill, Engine, Harness, Beta channel) — relevant when orienting to the codebase or discussing project terminology
- `CONFIGURATION.md` — user-facing knobs (env vars, flags, per-host install patterns); keep in sync per the rules below
- `CHANGELOG.md` — structured release history built by towncrier at release time (launch copy lives in GitHub Releases)
- `changelog.d/` — per-PR news fragments; feature PRs write here, never edit `CHANGELOG.md` directly
- `CONTRIBUTING.md` — setup, fragments, and release notes for humans and agents (towncrier is release-only)
- `.github/scripts/prepare_release.py` — lockstep version bump + towncrier build (release PRs only)
- `HERMES_SETUP.md` — install instructions for the Hermes harness specifically
- `mcp/` — Go MCP server (`.mcpb` bundle for Claude Desktop) exposing `research` and `preflight` (what a research run would read, write, and contact) tools. It embeds a copy of `skills/last30days/scripts/` (gitignored `mcp/internal/engine/vendored/`, regenerated by `mcp/scripts/sync-engine.sh`) and shells out to `python3`. See `mcp/README.md`.
## Orientation
- This is an Agent Skills package, not a CLI tool. The product is the slash-command-invoked skill (`/last30days <topic>` in most harnesses); `scripts/last30days.py` is implementation. Claude Code is the most common host but not the only one — features must work across every harness the skill installs into.
- Feature design starts from the slash-command UX. A new engine flag with no SKILL.md integration (the root or a routed reference) is incomplete — the model invoking the skill won't know the flag exists.
- README and PR examples show `/last30days <topic>` first. Direct CLI invocation (`python3 scripts/last30days.py ...`) is a fallback for scripting, cron, and dev-time engine testing; label it as such, never as the primary path.
- Slash commands don't pass shell mechanics through. `/last30days OpenClaw --emit=html | pbcopy` is invalid in any harness — either use the slash form (no flags or pipes; let the model translate user intent into engine flags) or use the direct CLI form (full `python3 ...` with explicit flags and a real shell).
## Commands
```bash
# Dev/fallback: direct engine invocation (scripting, cron, or engine testing only).
# Saves to $LAST30DAYS_MEMORY_DIR when set in shell or ~/.config/last30days/.env;
# add --save-dir <path> for a one-off override. Mirrors LAST30DAYS_STORE convention.
python3 skills/last30days/scripts/last30days.py "test query" --emit=compact
npx skills add . -g -y # copies skill into ~/.agents/skills/<name>/ (frozen at install time); re-run to sync working-tree edits — see Rules below
# Tests (pytest; tests/ discovery is configured in pyproject.toml)
uv run pytest # full suite
uv run pytest tests/test_dedupe_v3.py # single file
uv run pytest tests/test_dedupe_v3.py -k some_case # single case
uv run pytest --cov # with coverage (skips lib/vendor/)
PYTHONPATH=skills/last30days/scripts uv run python -m unittest tests.test_subproc -v # subprocess tests (Windows CI job; unittest skips conftest.py, so set the path)
uv run pytest tests/eval -x -s # research-quality eval (separate CI job)
# MCP server (Go, under mcp/)
bash mcp/scripts/sync-engine.sh # re-mirror the Python engine into the Go embed dir
(cd mcp && go test -race ./...)
# Release prep (maintainers / release automation — not feature PRs):
# Prefer GitHub Actions → "Prepare release". Local equivalent:
uv run python .github/scripts/prepare_release.py --bump patch # or --version X.Y.Z
```
Python 3.12+ required. Use `uv` for the env; the venv lives at `.venv/`. CI (`.github/workflows/validate.yml`) runs the full suite in parallel with coverage (`uv run pytest -n auto --dist worksteal --cov`, failing below the `fail_under` floor), plus the subprocess tests, the eval suite, and the Go MCP tests after `sync-engine.sh`.
## Changelog and releases (agents)
Agents open most PRs. Follow this so `CHANGELOG.md` stops conflicting and versions stay lockstep:
1. **Feature/fix PRs:** add `changelog.d/<pr-or-issue>.<type>.md` (`added` / `changed` / `fixed` / `removed` / `deprecated` / `security`) when the change belongs in the next release notes. See `changelog.d/README.md` and `CONTRIBUTING.md`. Fill the PR template’s Summary, Agent disclosure, and Relationship sections.
2. **Never** edit `CHANGELOG.md` in a feature PR. **Never** bump version strings in `pyproject.toml`, `SKILL.md`, plugin/marketplace JSON, or `uv.lock` outside a release PR. CI (`changelog-guard.yml`) enforces this.
3. **Nothing for release notes:** omit the fragment, check Skip changelog in the template, and add the `skip-changelog` label. Dependabot PRs (author `dependabot[bot]`) skip that requirement automatically.
4. **Cutting a release:** run Actions → **Prepare release** (patch/minor/major). That opens a `chore(release): bump version to X.Y.Z` PR which runs towncrier and bumps every lockstep surface. Merging to `main` triggers **Tag release**, which pushes `vX.Y.Z` and existing `release.yml` publishes `.skill` / `.mcpb` artifacts. Do not hand-edit ten version files. Contributors do not need a global towncrier install — `uv sync --group dev` (or the Action) provides it for release prep only.
5. Lockstep gate remains `tests/test_plugin_contract.py::test_versions_match_across_manifests`. Workflow contract: `tests/test_changelog_workflow.py`.
## Rules
- `lib/__init__.py` must be bare package marker (comment only, NO eager imports)
- One-time setup: `npx skills add . -g -y` copies the skill into `~/.agents/skills/<name>/` (real directory) and, for harnesses that support symlinked skill dirs, drops a per-host symlink pointing at that copy. **Working-tree edits do NOT propagate automatically** — the `~/.agents/skills/<name>/` copy is frozen at install time. To sync after edits, re-run `npx skills add . -g -y`. For live-edit on a dev machine, move that real directory to a backup outside `~/.agents/skills/` first, then create the working-tree symlink: `ln -sfn "$PWD/skills/last30days" ~/.agents/skills/last30days` (run from the repo root). `ln -sfn` replaces an existing symlink but does not replace a real directory; with the install copy still present, it creates a link inside that directory and leaves the stale skill active.
- Git remote: origin = public (`mvanhorn/last30days-skill`)
- Do not reduce `fail_under` in `pyproject.toml` (`[tool.coverage.report]`) without documenting why in the PR. The coverage gate is a floor meant to rise over time, not to be relaxed when new code is under-tested.
- Every `lib/*.py` call to `log.source_log(...)` must pass `tty_only=False`. The default is `True`, which silently drops every line when stderr isn't a TTY (Claude Code, Codex, CI, captured output) — turning source observability into invisible failure. Enforced by `tests/test_source_log_visibility.py`.
- **CLI-gated optional sources** (Digg via `digg-pp-cli`, YouTube via `yt-dlp`) activate only when `shutil.which` resolves the binary on the **agent subprocess PATH** — not merely when the file exists on disk. First-run setup installs Digg through `@mvanhorn/printing-press-library` (default `$HOME/.local/bin`); Hermes/OpenClaw gateways often need that directory on PATH. Setup must distinguish PATH-visible installs from off-PATH binaries and must not claim "now active" unless the engine gate would pass. See `docs/solutions/integration-issues/digg-cli-agent-path-setup-wizard.md`.
- **First-run onboarding is consent-driven, model-led, and host-split.** Spec lives in the root `SKILL.md` gate and Step 0 of `references/setup-wizard.md`; contract locked by `tests/test_onboarding_contract.py` (and `tests/test_grok_bot_host_contract.py`).
- The setup subprocess does only mechanical work (cookie reads, tool installs, GitHub device-auth, emitting the engine-owned welcome via `--welcome`). It cannot prompt, so consent lives in the skill instructions.
- Step 0 has THREE branches:
- **Claude Code Modal Flow** for hosts with modals — the restored v3.0.0 `AskUserQuestion`-driven NUX (welcome, Auto/Manual/Skip, cookie consent, ScrapeCreators offer, `INCLUDE_SOURCES` opt-in, first-topic picker). The welcome pitch is embedded in the modal's question, because the modal is the only always-fully-visible surface — a separate welcome message or `--welcome` Bash run gets buried behind "ctrl+o to expand".
- **Non-Modal Prose Flow** for hosts without modals (OpenClaw, Codex, Cursor, Gemini CLI, Grok CLI) — relays `last30days.py --welcome` verbatim.
- **Grok Bot Prose Flow** for `LAST30DAYS_HOST=grok-bot` — leads with the bot's own X lanes (its built-in X tools first, then the X for Grok Bot connector; `LAST30DAYS_X_HOST_LANE=1` plus a `--x-posts` envelope the model fetches itself, exported only when the fetch returned posts), persists any backup key (`X_BEARER_TOKEN` / `XAI_API_KEY`) only through the engine's `setup --store-key` path, and never reads a browser session.
- The logged-in Reddit session has its own consent question (`references/reddit-session.md`, routed from the root): asked after the X cookie answer (yes or skip) in the Modal and Non-Modal flows (a Reddit yes is its own consent and outranks a saved `BROWSER_CONSENT=false`; only `FROM_BROWSER=off` blocks it), once for returning users through the root gate's `REDDIT_OFFER` token (`REDDIT_SESSION_OFFERED` records it), and never on a Grok Bot host. X consent never implies it.
- Do NOT collapse the modal flow back into a bare silent `setup` call or flatten it to prose-only — the guided modals are the feature (they eroded once and were restored).
- The first two flows ask before reading cookies and surface the macOS Full Disk Access fix on permission-denied; all three offer the ScrapeCreators GitHub signup (10,000 free calls) on every first run.
- GitHub device auth is a two-command split: `setup --github-start` returns the code fast (foreground, copies to clipboard), `setup --github-poll` waits for authorization (`setup --github` chains both for back-compat). Success persists `SCRAPECREATORS_API_KEY` via `setup_wizard.write_api_key` (0o600) and masks the key in stdout.
- Step 5 source opt-in has two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts and top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything** (also Threads + Pinterest). Comments are on by default (posts on → comments on for all three platforms). **Threads and Pinterest are the only opt-in extras** — they appear only in the Step 5 Everything option, never in the welcome or the Step 4 offer.
- **Comment ranking.** Instagram comments come from ScrapeCreators (`/v2/instagram/post/comments`, ranked by `comment_like_count`) with vote-weighting parity to YouTube/TikTok (a dedicated `_instagram_engagement` carve-out plus `_VOTE_LOG_REFERENCE`/label/threshold entries). The "Top Community Comments" list (`render._render_top_comments`) is the report's quote pool, chosen and ordered **by position, never by like count across items**: each item (thread, video, reel, story) gives its top comment and its #2 by its own stored order (`metadata["top_comments"]`: score for Reddit, like count for TikTok and Instagram, YouTube's "top comments" order, Hacker News thread order), nothing below. All top comments come first, in the order of the stories (shown clusters as numbered, then other kept items as ranked), each story's standalone X/Threads/Bluesky/LinkedIn posts (10+ likes, labelled `post`, a LinkedIn post only with a real author) and what its YouTube, TikTok or Instagram video said (one line per video with 10+ likes, labelled `said`: YouTube's `transcript_highlights`, or the best highlight `youtube_yt.extract_transcript_highlights` draws from a reel's or TikTok's spoken caption) after its top comments, then all #2 comments in the same order; within a story and position, entries on a candidate that cleared `_BEST_TAKE_FUNNY_FLOOR` lead and end `funny` (`fun_score` belongs to the candidate). The per-platform display floors (YouTube 50, TikTok 500...) do not apply; on a platform that counts likes, upvotes or reactions (Instagram, TikTok, YouTube, X, Reddit, Threads, Bluesky, LinkedIn, GitHub) an item joins only when its top community comment has at least 5 (a Reddit thread's floor scales with its own score, `min(5, max(2, score // 20))`, so a small on-topic thread is not shut out), and its #2 needs at least 1, while Hacker News comments are exempt. A bot's comment (`[bot]` and bot-named authors, "automated review" text, HTML-comment markup), a comment by the item's own author (channel, creator, OP, submitter; read from the stored comment's `is_uploader` flag where the source provides one, names compared only when it is absent) and one with nothing quotable left are never community comments: the next comment takes the position. Quote text also drops lines quoting the parent (`>`), and markdown emphasis markers. Every eligibility test runs before duplicates are removed. Up to 40 one-line entries, shown clusters covered first and taking turns for slots (one top comment each, then round by round, so a busy cluster cannot starve a later one), each in the quote line's own shape (dot, author, `on r/sub` or platform, `· count`), then metadata: URL, `cluster N` or `re: "title"`, and `top comment` / `#2 comment` / `post`. Quote text is one line with raw URLs dropped, inner double quotes made single, a sentence-end cut when over 240 characters, and leading reply handles stripped from X posts. Compact output also prints `## Posts by @handle` and `## Polymarket Markets` (each on-topic market as a ready-to-copy odds line, `📊 Polymarket · {subject}: {outcome} {price} · ...`, at most three outcomes, movement in points after the outcome it belongs to, built from the `odds` summary `polymarket._odds_summary` stores on the item; then URL and metadata), and the command line (not the renderer) appends the `## Before you write the report` checklist to `--emit=compact` stdout only (`lib/report_checklist.py`).
## Security hygiene
- Never commit real API keys, browser cookies, auth tokens, app passwords, access tokens, or `.env` contents.
- Use the env-based auth patterns in `skills/last30days/scripts/lib/env.py`; tests and fixtures must use obvious dummy values only.
- Keep examples safe by redacting secrets and avoiding copy/pasteable live credentials in docs, fixtures, and test data.
- Do not weaken or disable the advisory security workflow (`.github/workflows/security.yml`) without explaining why in the PR description or review thread.
## Maintaining README translations
- `README.md` is the canonical English README. When changing it, reflect the same substantive updates in `README.fr.md`, `README.de.md`, `README.es.md`, `README.pt-BR.md`, `README.ja.md`, and `README.zh-CN.md`, preserving commands, links, tables, and reciprocal language navigation.
+ `README.md` is the canonical English README. When changing it, reflect the same substantive updates in `README.fr.md`, `README.de.md`, `README.es.md`, `README.pt-BR.md`, `README.ja.md`, `README.zh-CN.md`, `README.ar.md`, `README.fa.md`, and `README.tr.md`, preserving commands, links, tables, and reciprocal language navigation.
## Maintaining CONFIGURATION.md
`CONFIGURATION.md` is the user-facing configuration reference — save paths, per-source API keys, web-search backend priority, trend-monitoring stack, per-client install patterns. Distinct from `SKILL.md` (the canonical runtime spec).
Update `CONFIGURATION.md` when:
- adding a new env var (e.g. `LAST30DAYS_*`, `BSKY_*`, `*_API_KEY`)
- adding a new CLI flag that affects configuration (e.g. `--store`, `--web-backend`)
- adding a new per-client install pattern (Claude Code, Gemini, Codex, Cursor, Grok, Hermes…)
- adding a new optional source that requires its own credential
- changing the priority order of config layers (per-run flag > env > `.env` file > defaults)
Keep the existing structure organized by how often each layer is touched: per-run flags → env vars / `.env` → optional trend-monitoring stack → per-client patterns. Add new content into the right section rather than appending at the end.
When a new config concept lands in `SKILL.md` or `AGENTS.md`, mirror the user-facing knob in `CONFIGURATION.md` so non-agent readers can configure the skill without reverse-engineering it from the runtime spec.
## Plugin manifests (Grok)
The repo doubles as a native Grok Build plugin via `.grok-plugin/plugin.json` + `.grok-plugin/marketplace.json`. Grok also reads `.claude-plugin/*` for compatibility; the native pair is the first-class lane and what an official xAI marketplace listing points at. The self-hosted catalog uses a bare Git URL source (`{"source":"url","url":"https://github.com/mvanhorn/last30days-skill.git"}`) so `grok plugin marketplace add mvanhorn/last30days-skill` tracks HEAD — not a self-referential local `path: "."` (Grok does not enumerate those). Version lockstep with Claude/Codex/Gemini manifests is enforced by `tests/test_plugin_contract.py`. Validate with `grok plugin validate .`.
## Submitting to the xAI plugin marketplace
Listing in xAI's official catalog (`xai-org/plugin-marketplace`) is a post-merge outbound PR to *their* repo that pins a full 40-char commit SHA — unlike our own `.grok-plugin/marketplace.json`, which uses a bare URL source and tracks HEAD. Step-by-step procedure: `docs/reference/xai-marketplace-submission.md`.
## Beta channel
Experimental changes get tested on `mvanhorn/last30days-skill-private`, which installs as a parallel `/last30days-beta` slash command. Beta-only changes never ship to public without a review PR here. Workflow guide (and the plan that established this setup) lives at `BETA.md` in the private repo.