crawl-and-queue · git:20260722.45cbafa · 2026-07-22 · sha256 3a6bbcc1857d9c5b

crawl-and-queue git:20260722.45cbafaA

Immutable. This exact content never changes and is served at /api/v1/blob/3a6bbcc1857d9c5b.

No tool to install first
claude plugin marketplace add https://markdownregistry.com/mp/a/art_ln2gkm7c5yzgmemr/3a6bbcc1857d9c5b982f516c32eea802bb959e5030291a99e26393fec981073f.json

Then /plugin install crawl-and-queue@markdownregistry-ln2gkm7c5yzgmemr in Claude Code. It installs this exact version, the one graded A here, pinned to the commit it was graded at, and leaves it there.

---
name: crawl-and-queue
description: Run the crawl4ai discovery pipeline to fill the idea queue for an agent built with content-agent-kit. Use when the user says "crawl for ideas / news", "refresh the idea queue", or when setting up scheduled discovery. Crawls sources.yaml, dedups against the queue API, pushes fresh excerpts.
---

# Crawl → idea queue

1. **Check `sources.yaml`** exists and each source has a `list_url` + a tuned
   `link_pattern`. Loose patterns catch navigation links — verify with a dry run:
   `python scripts/crawl/crawl.py --dry-run` (prints found links only, pushes nothing).
2. **Ensure deps** (once): `pip install -r scripts/crawl/requirements.txt && crawl4ai-setup`
   (installs Chromium — run in CI or on a real machine, not serverless).
3. **Ensure env**: `SITE_URL`, `INGEST_API_TOKEN` (+ `QUEUE_PATH` if not `/api/queue`).
4. **Run**: `python scripts/crawl/crawl.py [--limit N]`. It:
   - GETs `?known=1` to skip URLs the server already has (server = dedup memory),
   - extracts new article links, crawls them, keeps **excerpt ≤1500 chars + link only**,
   - POSTs `{ items }` to the queue (upsert, unique on `source_url`).
5. **Copyright**: never store or republish full source text. The publishing agent writes
   ORIGINAL content from the idea. Prefer public-domain / permitted sources.
6. For automation, install the cron workflow from `templates/workflows/crawl.yml.template`
   and add `SITE_URL` + `INGEST_API_TOKEN` to repo secrets.

> Antigravity: same steps, run the commands directly.