git:20260930.acbf14d to git:20261103.4dbcc13

# tg-scraper / slop-writer

A **skill plus an MCP server** that analyze a Telegram channel, can queue a

post to it, and can ban the bot subscribers an audit flags. The thirteen MCP

tools are the whole agent-facing surface; the skill under `skills/slop-writer/`

says which tool answers which question and what the numbers mean. Published to

PyPI as `slop-writer`; two install channels (`slop-writer install`,

`npx skills@latest add …`) serve that same skill directory, and one version

covers package and skill. `install` wires it into a **client** — Claude Code or

Codex, each independently (adr/0009).

Rationale for a decision lives in `docs/adr/`; vocabulary in `CONTEXT.md`;

structure (what calls what, where a symbol is) in codegraph.

## Commands

- `uv run pytest` — the suite, from the repo root. `uv` installs the project

editable, so it exercises the working tree.

- `uv run tools/check_schema_doc.py` — dev-only drift guard; run after editing

`SCHEMA` or `references/schema.md`. CI runs it as its own step.

- `uv run --with-editable . tools/tg_*.py` — the dev CLIs against the tree.

- `slop-writer` (the shipped console script): `install` wires this project's

client config (`--client claude|codex`, repeatable; bare = claude),

`uninstall` removes exactly what it wrote, `serve --mcp` runs the server over

stdio, `init` writes Telegram credentials and logs in.

**`init` needs a TTY for the SMS code — the user runs it, not the agent.**

## Library rules (`src/slop_writer/`)

- **The domain raises, the entrypoint reports.** Nothing under

`src/slop_writer/` prints an error or exits: it raises `SlopWriterError`

(`UsageError` for exit 2) and `cli.py` or `server.py` decides how to report.

That is what lets the server call the same function and build JSON instead.

- **`errors.ERROR_CODES` is a closed vocabulary**, enforced at construction.

Reuse a code; adding one is a deliberate edit with a raise site.

- **A message or a hint names an argument or an operation, never a surface** —

the same string reaches a human at stderr and a model at a tool result, and

only one of them has a flag or a script. `tests/test_errors.py` scans every

literal statically over the modules `server.py` can reach. When the surface

genuinely differs, the caller supplies the noun (`prepare_schedule(

body_source=…)`) or the boundary swaps the hint (`server._SETUP_HINT`).

- **The domain uses a client, the entrypoint owns it.** Scrape and scan paths

come in twins: `X_with_client(client, …)` does the work, `X(…, session_file)`

wraps it in one `channel_session`. A **new command adds both forms.**

- **`resolve_peer` runs before any `open_db`** — `open_db` lives inside the

`channel_session` body (for the group scan, inside `scan_group_with_client`

after `resolve_group_target`). A typo then fails on the handle instead of

leaving a stray empty `.tg-analytic/<typo>.db`, and zero posts always means

an empty window.

- **Renderers return a string.** `render.py` takes plain dicts and returns

Markdown — no Telethon or SQLite types, and no `print`: on a stdio server

stdout is the JSON-RPC transport. The domain returns the result shape

(`ScrapeResult`, `GroupScanResult`, …); the entrypoint calls `summarize_*`.

- `db.py`, `errors.py` and `query.py` stay **stdlib-only** — a query stays

answerable without a Telegram client.

- `publish.py` is the **write surface** (adr/0003) — posting *and* banning

(adr/0008); nothing on a read path imports it, which is why `scheduled.py`

and `audit.py` are separate. `publish` reads them, never the reverse.

- **`MESSAGE_TOO_LONG` comes from the network, by design** — the cap depends on

the account, so `_too_long` translates Telethon's error rather than measuring

the body.

- **A refusal is not an absence.** `CANNOT_RESOLVE` means the handle names

nothing, `NOT_A_MEMBER` means Telegram confirmed the peer and refused this

account — one is fixed by correcting the handle, the other by joining. Every

new Telethon call that can refuse classifies the two.

- `cli.py` is **argparse**, so an installed server carries no CLI framework;

typer stays a `tools/` script dependency. `install.py` reads TOML with

`tomllib` and writes it with a local emitter for the same reason — no runtime

dependency for one config format (adr/0009).

- **A client is selected, never detected**, and each is wired independently:

`install_project`/`uninstall_project` take the selection, return one result

per client, and read first-install from *that client's* own config key. A new

client adds an installer, an uninstaller and a renderer section — plus its

gate emitter in `server.py`. The cross-agent pair (`AGENTS.md` block,

`.agents/skills/`) belongs to no client: written on every install, removed

only by an unnarrowed uninstall.

- `.tg-analytic/` is runtime state at the project root (gitignored): `.env`,

`session.session`, one `<channel>.db` per channel, `media/`. **Entrypoints**

anchor it on the cwd (`--project` overrides); the library never reads the cwd.

## MCP server (`server.py`)

- **The `publish_` prefix is the read/write split**, not a naming style: Claude

Code matches permission rules by tool *name*. Roster, `permission_rules()`

and `codex_approval_rules()` live in one module because a renamed tool

without its rule is silently ungated; `tests/test_server.py` compares all

three. **A new client adds an emitter here, never a translation in

`install.py`** (adr/0009) — the module that decides which tools write is the

one that says who approves them.

- **Text only, never `structuredContent`** — Claude Code drops the content

blocks when structure is present, so structure travels *inside* text. Register

through the local `_tool` wrapper (`structured_output=False`);

`assert_text_only` refuses to start a server whose tools carry an

`outputSchema`.

- **Errors are `{code, message, hint}` JSON in the text block** with

`isError: true`. `_guarded` wraps every tool, names `FLOOD_WAIT` (with

`seconds`), and labels anything unanticipated `INTERNAL`.

- **`isError` is the *call* refusing, never one request inside it** (adr/0007).

`run_query` takes a list, and a statement SQLite rejects comes back as that

question's section — bad SQL beside three good answers must not discard them.

Only a condition with no per-question answer to give (`NO_DATA`: no database

at all) raises. A tool that takes N requests classifies the two; counting

failures would make the flag mean two unrelated things.

- **Two hints belong to the entrypoint**, which is why `_guarded` takes the

project's `output_dir`: the setup pair answers with `slop-writer init`, and

`CANNOT_RESOLVE` answers with the channels this project holds

(`db.scraped_channels()`) — an agent that never had a handle cannot fix a typo.

- **Write tools validate before they require a session** — a malformed `at`

answers with the argument to fix, not with "run `slop-writer init`".

- `select` is a **nested** tagged union (`extra="forbid"` on both arms); a

root-level combinator gets flattened by the client. Pydantic rejects a bad arm

*before* `_guarded`, so those alone escape the JSON contract.

- `run_query` alone carries `_meta["anthropic/alwaysLoad"]`; the rest are

deferred behind `ToolSearch`.

- `_meta["anthropic/requiresUserInteraction"]` stays **rejected** and server

`instructions` stay **empty** (adr/0003): the first forecloses headless

autoposting by the channel's owner, the second is context spent every turn on

what belongs in the skill.

## The skill (`skills/slop-writer/`)

Five files, no `scripts/`. The seam is *tool description = how to call, skill =

what it means*, checkable both ways: **no metric fact in a tool description, no

argument name in the skill.** A new tool therefore edits `server.py` and usually

nothing here; a new *invariant* edits here and not `server.py`. `SKILL.md` is a

router (adr/0005) to `references/{analysis,schema,publishing,markup}.md`.

## Data invariants

- `post_metrics` is **append-only** — `MAX(id)` for "latest snapshot", never

`MAX(scrape_date)`. Canonical CTE in `references/schema.md`.

- **A selection counts posts, never messages.** Telethon's `limit` counts

messages and an album is many messages, so `take_posts` walks an unbounded

stream and stops on the first message past the count instead. That is what

makes a short run mean exactly one thing — the history ended — which

`ScrapeResult.history_exhausted` carries and `summarize_scrape` states.

**Absent is not `False`** there either: a refresh walks no window.

- **One album = one `posts` row.** `complete_albums` re-fetches missing members

before `process_post` picks the head, so a captionless member never becomes a

post of its own (a phantom row doubles the album in every `SUM`/`AVG`).

`heal_album_phantoms` cleans up rows from pre-fix runs.

- `group_messages` is the **only** comment store (adr/0002): engagement queries

filter `is_thread_root = 0`. `group_events` PK is `(id, user_id)` — one

service message can carry several users.

- **A join count carries its source, or it is unreadable.** Without the admin

log the counts are a *floor*, so `scan_group` records whether it read the log

in `overview["admin_log"]` and `summarize_group` states it on the line

carrying the numbers. **Absent is not `False`** there: a hand-built overview

does not know.

- FTS5 external-content indexes (adr/0004) sync via `FTS_SCHEMA` triggers;

`open_db` applies them best-effort (no FTS5 → search lost, scraping fine).

`MATCH 'stem*'` is the canonical text search.

## Tests

`slop_writer.*` functions and their return shapes — never a script's stdout or

exit code. Assert on `SlopWriterError.code`, not on the message. Messages are

real Telethon TL objects, the client is hand-faked (`factories.FakeClient` is

the whole network boundary). Nothing in the suite touches a session,

`.tg-analytic/`, or the network; live runs stay the acceptance step. pytest

lives in a PEP 735 `dev` group, never an extra.

## Releases

**One version string in four places**: `pyproject.toml`'s `version` (the

source), `SKILL.md`'s `metadata.version`, the git tag (`v` + it), and the

GitHub release title (the tag verbatim, `v0.4.1`) — three components

throughout, with the release's prose kept to its notes.

`tests/test_packaging.py` pins the skill's copy to the source, which it needs

because the `npx skills add` channel never sees `pyproject.toml`. Tag and title

rest on this convention alone: `publish.yaml` compares them with the `v`

stripped. PyPI versions are immutable, so check the tag before publishing.

## Dev CLIs (`tools/tg_*.py`)

`tg_scrape.py`, `tg_publish.py`, `tg_query.py` are PEP-723 typer scripts kept as

**dev tools only** — `login` needs a TTY, and they are the only way to drive

Telethon without an MCP client. They stay **undocumented**: nothing under

`skills/` names a script or a flag, and a `grep` is the acceptance check. Flags

are `--help`; keep it that way rather than restoring a reference file.

`--at` is ISO-8601 **with offset** (naive rejected); the now+1h floor is

`MIN_LEAD` in `publish.py` with no override, so the agent cannot schedule too

soon (adr/0003). `reschedule`/`edit` are `editMessage` with `schedule_date`, and

Telethon returns `None` for scheduled edits — both callers report from known

inputs. `--caption-above` rides an `invert_media` monkey patch.

## Gotchas

- Telethon TL types are generated dynamically, so Pyright flags `.sender`,

`.chats`, `.full_chat`, `.forwards` as unknown attributes throughout —

expected noise, not real errors.

- **mistune**, not Python-Markdown, for a post body: it keeps `#hashtag` lines

literal instead of parsing them as `<h1>`. `markdown.py` walks its AST straight

to Telethon `MessageEntity` objects (no HTML hop) and owns UTF-16 offsets.

- Wheel data ships `skills/` *beside* the package (`data = { purelib = "skills" }`)

so `install` can copy it out; it cannot live under `src/slop_writer/`, since

`npx skills add` serves the same directory from the repo root.

`install.skill_source()` tries the source checkout **first** — an editable

install materialises the data directory once and never refreshes it. **One

source, several destinations**: a project holds the skill under

`.claude/skills/` *and* `.agents/skills/`, so "one directory" is now true of

the source only (adr/0009).

bot API).

<!-- slop-writer:start -->

Telegram channel analytics goes through the `mcp__slop-writer__*` tools.

Read `.claude/skills/slop-writer/SKILL.md` before calling one — it routes to the metric

invariants and the DB schema, without which the numbers come back wrong.

<!-- slop-writer:end -->