cp-skill-snapshot-web git:20261030.efff270A

Immutable. This exact content never changes and is served at /api/v1/blob/d4f29809b365131f.

---
name: cp-skill-snapshot-web
description: Snapshot a URL into the local kb/sources/.snapshots/ cache, routing GitHub, X/Twitter, PDF, and ordinary web sources to the appropriate capture path.
type: types/instruction.md
user-invocable: true
allowed-tools: Read, Write, Grep, Glob, Bash
context: fork
model: sonnet
argument-hint: "[url] [reobserve] — URL to snapshot (web page, PDF, GitHub issue/PR, or X/Twitter post); add reobserve to capture a changed source again"
---

## Intent and execution authority

Preserve a faithful local reading copy of the requested source so later ingest
can analyse it against an exact-file checksum.

The tools and commands below are supported routes, not an exhaustive list of
permitted methods. Choose another available tool or write local extraction code
when needed to capture the requested content faithfully. A failed tool is not
by itself a reason to abandon the capture. Requirements imposed by a caller,
such as a version-pinned PDF capture, still bind.

**Done means:** the requested content, not merely its page shell, is saved with
accurate source identity, capture method and scope, and its exact Markdown-file
SHA-256 is reported. Preserve relevant links and footnotes. If only part of the
requested content is available, disclose the missing part and label the scope;
do not silently substitute an abstract, surrounding page, or commentary.

**Binding boundaries:** keep captures local and ignored, never overwrite an
existing observation, do not fabricate or analyse source content, and copy the
captured body locally rather than re-emitting it through model output. Do not
install software, bypass access controls, or execute code supplied by the page.
Return to the user when completion needs new permission or when available
evidence cannot establish source identity or capture fidelity.

## EXECUTE NOW

**Target: $ARGUMENTS**

If no URL provided, ask the user for one.

If URL provided, start Step 1 immediately.

**START NOW.**

---

## Step 1: Verify Local Storage and Check for Duplicates

Keep the provided URL as `source_url`. The request is a **re-observation** when
it says `reobserve` or asks to capture a source again because it changed since
its existing capture. Verify that
`kb/sources/.snapshots/` is ignored by the project. The shipped scaffold does
this through `kb/sources/.gitignore`. If the directory is not ignored, stop
before writing and report the missing rule.

For PDF and ordinary web captures, keep temporary downloads, extracted bodies,
and assembly files in a unique directory under `kb/reports/cache/snapshot-web/`. Verify
that payloads there are ignored before writing; stop and report a missing
ignore rule. The final snapshot still belongs in `kb/sources/.snapshots/`.

Use Grep to search for an exact frontmatter `source: {source_url}` in existing
Markdown files in `kb/sources/.snapshots/`. For a re-observation, an existing
capture is expected: continue, and write the new capture under a distinct
basename as described under Step 2 and **slug** below. Never overwrite or edit
the existing capture; its ingest pins its bytes. Otherwise, if found, compute
the SHA-256 of the exact file bytes, tell the user, and stop:

> Already snapshotted: kb/sources/.snapshots/{filename}
> SHA-256: {64-character lowercase checksum}

## Step 2: Route by URL Type

Detect the `source_url` type and branch:

- **GitHub issue/PR** (`github.com/.../issues/N` or `github.com/.../pull/N`) → **Step 2a**
- **X/Twitter** (`x.com/.../status/...` or `twitter.com/.../status/...`) → **Step 2b**
- **arXiv abstract page** (`arxiv.org/abs/...`) → **Step 2c**
- **PDF** (URL ends in `.pdf`, or `arxiv.org/pdf/`) → **Step 2c**
- **Everything else** → **Step 2d**

### Step 2a: GitHub Issue/PR

Run:

```bash
commonplace-github-snapshot "{source_url}"
```

For a re-observation, add `--reobserve`: the command skips the duplicate check
and names the new capture with the capture date.

Parse either the `Snapshot saved:` or `Already snapshotted:` line from the
output to get the file path. Tell the user and stop — the script handles
metadata, formatting, and saving. If it fails, use **Step 3**.

### Step 2b: X/Twitter Post

Run:

```bash
commonplace-x-snapshot "{source_url}"
```

For a re-observation, add `--reobserve`: the command skips the duplicate check
and names the new capture with the capture date.

Parse either the `Snapshot saved:` or `Already snapshotted:` line from the
output to get the file path. Tell the user and stop — the script handles
metadata, formatting, and saving. If it fails, use **Step 3**.

### Step 2c: Resolve and Fetch PDF

Verify that the PDF capture prerequisites are available:

```bash
command -v curl
command -v pdfinfo
command -v pdftotext
```

If a command is missing, use another available tool that preserves the paper's
content, or go to **Step 3**. Do not install prerequisites.

Set `pdf_url`:

- For an arXiv abstract URL, replace `/abs/` with `/pdf/` and discard any query string or fragment. Preserve an explicit terminal version such as `v1`. If the abstract URL has no terminal version, leave the PDF URL unversioned so arXiv serves the latest paper version. For example, `https://arxiv.org/abs/2606.03979` becomes `https://arxiv.org/pdf/2606.03979`. Do not route the abstract page through ordinary HTML capture.
- For an existing PDF URL, use `source_url` unchanged.

Create the temporary directory and retain the printed path as `{snapshot_tmp}`:

```bash
set -e
mkdir -p kb/reports/cache/snapshot-web
snapshot_tmp=$(mktemp -d kb/reports/cache/snapshot-web/capture.XXXXXX)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"
```

Download in a separate invocation with the literal temporary path and URL:

```bash
curl -fsSL -o '{snapshot_tmp}/source.pdf' '{pdf_url}'
```

Then extract locally:

```bash
pdfinfo -isodates "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfinfo.txt"
pdfinfo -meta "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfmeta.xml" || true
pdftotext -enc UTF-8 -eol unix -nopgbrk \
  "$snapshot_tmp/source.pdf" "$snapshot_tmp/extracted.txt"
```

Use Read to inspect `pdfinfo.txt`, non-empty `pdfmeta.xml`, and a bounded
beginning of `extracted.txt`. DOI metadata inspection is best effort and its
failure does not make an otherwise successful capture fail. Treat PDF metadata
fields as leads, not as authority: confirm the title and authors against the
document text when available. Use Grep plus bounded Read ranges to locate an
abstract, executive summary, introduction, or the source document's own DOI
when the beginning does not supply enough metadata. Do not treat a DOI found
only in the references as the paper's DOI. Do not read the whole extracted
file merely to copy it. If `extracted.txt` is empty or contains no substantive
text, go to **Step 3**.

Set `capture_method` to `pdftotext`, set `body_file` to
`{snapshot_tmp}/extracted.txt`, and go to **Step 4**.

### Step 2d: Fetch Web Page

Verify that the HTML capture prerequisites are available:

```bash
command -v curl
command -v trafilatura
```

If either command is missing, use another available fetch or extraction tool,
or go to **Step 3**. Do not install prerequisites.

Create a temporary directory and retain the printed path as `{snapshot_tmp}`:

```bash
set -e
mkdir -p kb/reports/cache/snapshot-web
snapshot_tmp=$(mktemp -d kb/reports/cache/snapshot-web/capture.XXXXXX)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"
```

Download with curl in a separate Bash invocation. Substitute the literal
temporary path and source URL before running:

```bash
curl -fsSL -o '{snapshot_tmp}/source.html' -A 'Mozilla/5.0' '{source_url}'
```

If the download fails, go to **Step 3**. Otherwise extract from the saved HTML
through standard input; do not use Trafilatura's `-u` download option or its
`-i` URL-list option:

```bash
trafilatura \
  --markdown --with-metadata --links --no-comments --recall \
  < "{snapshot_tmp}/source.html" > "{snapshot_tmp}/extracted.md"
```

If extraction fails, go to **Step 3**.

Use Read to inspect only the leading metadata and a bounded beginning of
`extracted.md`. Its leading YAML block, when present, is Trafilatura metadata:
retain it as input to Step 4 but do not copy that block into the snapshot body.
Use Grep with bounded output on `{snapshot_tmp}/source.html` to inspect article-level DOI
metadata such as `citation_doi`, `dc.identifier`, `prism.doi`, or a JSON-LD
`doi` property. DOI inspection is best effort: inability to inspect the raw
HTML does not make an otherwise successful capture fail.
Strip that block locally without re-emitting the document:

```bash
awk '
NR == 1 && $0 == "---" { in_metadata = 1; next }
in_metadata && $0 == "---" { in_metadata = 0; next }
!in_metadata { print }
' "{snapshot_tmp}/extracted.md" > "{snapshot_tmp}/body.md"
```

If `body.md` is empty, contains no substantive main content, or contains only
an access-denied page or browser challenge instead of the article, go to
**Step 3**.

Set `capture_method` to `trafilatura`, set `body_file` to
`{snapshot_tmp}/body.md`, and go to **Step 4**.

## Step 3: Adapt to Capture Failures

Before declaring capture blocked, inspect the failure and the material already
retrieved. Choose a bounded alternative supported by that evidence. Do not
repeat an unchanged failing method or search indefinitely for more tools.

- If an extractor returns boilerplate, an empty body, or the wrong page section,
  inspect bounded portions of the saved HTML. The requested content may be in
  the DOM or embedded JSON even when the default extractor misses it.
- For a linked comment or other subdocument, verify its identifier and author
  against the URL and page records. Extract that record's content, not the
  surrounding page or replies. Include its footnotes and relevant links.
- You may parse embedded data locally without executing page scripts. Write the
  recovered body directly to a local file. Raw HTML embedded in Markdown is
  acceptable when it preserves content that conversion would lose.
- If fetching fails or a tool is unavailable, use another available route to
  the same source when permitted. For an image-only PDF, use available OCR only
  if you can check its fidelity; otherwise report that text capture is blocked.

After recovery, set `body_file` to the local extraction and `capture_method` to
an accurate name for the method actually used, then continue at **Step 4**.
Keep the raw response until the snapshot has been assembled and verified.

Stop and report the blocker when no evidence-backed route remains, fidelity or
source identity cannot be established, or another route requires installation,
new access, or material cost outside the request. State what was tried and what
is missing. If a missing standard tool would resolve it, name its installation:
`uv tool install "trafilatura>=2.2"`, Poppler (`poppler-utils` on Debian/Ubuntu,
`poppler` through Homebrew, or `oschwartz10612.Poppler` through WinGet), or curl.
Offer pasted text as a fallback rather than the first response to tool failure.
Remove the unique temporary directory before returning.

## Step 4: Determine Metadata

**(Only for PDF and web page paths — GitHub and X scripts handle their own metadata.)**

This workflow supplies `types/snapshot.md` as the type, a global type specified in [snapshot.md](../../types/snapshot.md). Open that spec and verify from its own frontmatter that it is a type spec before determining metadata. Stop if it is missing or invalid.

From the bounded excerpts, extractor metadata, and `source_url`, determine:

- **title**: The article/post title. Use the first H1 if present, otherwise derive from content.
- **author**: If identifiable from the content or URL (e.g. simonwillison.net → Simon Willison)
- **doi**: For a scholarly article or paper, try to identify the DOI from the
  `source_url`, extractor or document metadata, and the document's own title or
  citation block. Store the bare identifier beginning with `10.`; remove a
  leading `https://doi.org/` or `doi:` label and surrounding whitespace. Accept
  a candidate only when the source identifies it as the DOI of the captured
  work. A DOI found only in references is not sufficient. If candidates
  conflict or none is attributable to the captured work, omit `doi`; never
  guess or manufacture one.
- **genre**: the source's genre per the snapshot type spec's vocabulary. This is a surface judgment of what kind of document the source is as evidence — ingestion may correct it later. Prefer a value from the type spec's list; a value outside it validates with a warning, so extend only for a genuinely new evidential kind, not a container.
- **capture_scope**: `full-source`, `partial-source`, `abstract`, or `excerpt`
  under the snapshot type contract. Judge the retained body, not the success of
  the extraction command. In particular, label a publisher page that exposes
  only an abstract as `abstract`, even when that abstract is substantive.
- **description**: One sentence describing what makes this source worth retrieving. Not a summary — a retrieval filter (e.g. "Anthropic CEO's capability-timeline predictions — verifiable domains get confident timelines, unverifiable ones get hedged"). Focus on what distinguishes this source from others on the same topic.
- **slug**: Lowercase, hyphenated, max 63 chars. The paired ingest adds
  `.ingest` to the validated stem, so the snapshot basename must reserve those
  seven characters within the 70-character authored-artifact limit. Derive it
  from the title; shorten it to identifying words rather than copying the full
  title. Check the length before writing. An overlong proposed slug is a
  naming choice to fix automatically, not a reason to stop or ask the user.
  Example: `simon-willison-karpathy-claws`. For a re-observation, append the
  capture date as `-YYYYMMDD` within the same 63-character limit, and stop if a
  file with that name already exists.

For academic papers: prefer the title and complete author list printed in the
paper over `pdfinfo` or Trafilatura metadata.

## Step 5: Materialize the Snapshot

The extracted body must move from `body_file` to the snapshot through local
byte copying. Never place the whole source body in a Write or Edit call.

Use Write to create `{snapshot_tmp}/header.md` with this content and no source
body. End the file with the blank line after `Date`:

```markdown
---
source: {source_url}
description: {description}
captured: "{YYYY-MM-DD}"
capture: {capture_method}
capture_scope: {capture_scope}
genre: {genre}
doi: "{bare DOI; omit this line when no DOI was verified}"
type: types/snapshot.md
---

# {title}

Author: {author}
Source: {source_url}
DOI: {bare DOI; omit this line when no DOI was verified}
Date: {publication date if known}

```

For a recovered capture, use the locally extracted body and record the capture
boundary in the snapshot body when it needs clarification. Do not copy embedded
transport wrappers, unrelated records, or site boilerplate as source content.

Trafilatura has already produced the default web body as Markdown. A PDF body remains
the complete plain text emitted by `pdftotext`; plain text is valid Markdown.
Do not make model-mediated PDF cleanup a condition of capture. If the user
explicitly requested cleanup, transform bounded chunks into a candidate body,
never send the whole document through one Write, and retain the raw
`extracted.txt` as fallback. Set `body_file` to the candidate only after every
source chunk is present and in order; otherwise keep the raw body.

Assemble the snapshot without sending the extracted bytes through model output:

```bash
set -e
snapshot_path="kb/sources/.snapshots/{slug}.md"
cp "{snapshot_tmp}/header.md" "$snapshot_path"
cat "{body_file}" >> "$snapshot_path"
header_bytes=$(wc -c < "{snapshot_tmp}/header.md")
body_bytes=$(wc -c < "{body_file}")
snapshot_bytes=$(wc -c < "$snapshot_path")
test "$snapshot_bytes" -eq "$((header_bytes + body_bytes))"
```

Compute SHA-256 after the file is complete. Hash the exact `.md` bytes,
including frontmatter, line endings, and the presence or absence of a final
newline. Do not include a PDF, JSON, image, or other capture companion. Verify
that the saved body matches `body_file` and that the retained content matches
the requested source and declared scope. Report the path using the exact line
`Snapshot saved: kb/sources/.snapshots/{slug}.md`, followed by its lowercase
checksum and a one- or two-line preview. Disclose any missing content and any
alternative capture method used.

## Critical Constraints

**Never:**
- Fabricate or hallucinate content not on the page
- Add analysis or commentary — this is capture, not ingestion
- Re-emit a complete extracted body through Write or Edit
- Make model-mediated cleanup a prerequisite for saving a snapshot
- Save final snapshots outside `kb/sources/.snapshots/` or temporary
  download/extraction files outside `kb/reports/cache/snapshot-web/`
- Install software or bypass access controls — use available permitted means or report the blocker

**Always:**
- Copy every `body_file` byte in order, including on alternative capture paths
- Include the source URL in frontmatter
- Use today's date for `captured`
- Check for duplicates before fetching
- Keep the snapshot and every capture companion local and ignored
- Remove the unique temporary download/extraction directory after the snapshot
  is written and hashed