confluence-crawler git:20260810.16cc0bdB

Immutable. This exact content is served forever at /api/v1/blob/5670ba3e118b2f15.

---
name: confluence-crawler
description: Crawl an authenticated Confluence space (Atlassian Cloud or on-prem Server/Data Center) by page hierarchy and convert each page to clean Markdown with frontmatter. Handles macros, attachments, internal link rewriting, depth limits, and idempotent re-crawling. Use when the user wants to mirror, export, or ingest Confluence content.
metadata:
  credentialed: true
  primitive-class: credentialed-cli
  auth: sso-cookie
  auth-fallback: creds
  namespace: confluence
  keys: ["API_TOKEN"]
---

# Confluence Crawler

Crawl a Confluence space (Cloud or Server/Data Center) and write each page as Markdown with YAML frontmatter.

## Output rendering

Key–value / one record — For a single record's fields, use an aligned key: value list, not a two-row table.

## Instructions

You are a Confluence export agent. The heavy lifting — authentication, REST pagination, macro conversion, link rewriting, idempotency — lives in `scripts/`. Do not re-implement any of that logic; just invoke the scripts with the right arguments and report the result.

### Flavor support

The skill works against both:

- **Atlassian Cloud** (`*.atlassian.net`) — Basic auth with email + API token from `id.atlassian.com`. Base URL must include `/wiki` (setup adds it automatically).
- **Confluence Server / Data Center** — Bearer auth with a Personal Access Token from the user's Confluence profile.

Flavor is auto-detected from the base URL. Override via `CONFLUENCE_FLAVOR=cloud|server` if needed.

### Configuration location

Credentials are resolved by the build-projected `credentials_shim.load_credentials`
through Tier 1 (env) → Tier 2 (OS keyring) → Tier 3 dotfile. The dotfile
lives at `~/.agentbundle/credentials.env`. The declared schema is in
`references/creds-schema.toml`:

| Key | Required | Notes |
|---|---|---|
| `CONFLUENCE_BASE_URL` | yes | Cloud: `https://<site>.atlassian.net/wiki`. Server: `https://confluence.corp.example.com`. |
| `CONFLUENCE_API_TOKEN` | yes | Cloud API token or Server PAT. |
| `CONFLUENCE_EMAIL` | Cloud only | Atlassian account email. |
| `CONFLUENCE_FLAVOR` | no | `cloud` or `server`. Auto-detected from URL host when unset. |

Populate any tier by running `credential-setup` skill.

### Security rules (non-negotiable)

- Secrets live only in `~/.agentbundle/credentials.env`
  (mode 0600 on POSIX; DACL-restricted on Windows), the OS keyring,
  or process environment variables.
  **Never** read that file, print it, or echo the token.
- **Never** put the token on the command line. The primitive
  refuses flags like `--token` / `--api-token` / `--bearer` /
  `--pat` / `--password` and exits — do not work around it.
- On the token path, if `--check` reports missing or invalid credentials, tell
  the user to run `credential-setup` themselves. It is interactive — do not run
  it for them. A 403 is a permission failure, not a setup trigger; surface it
  without starting credential setup.
- **`CONFLUENCE_BASE_URL` is user-configured.** Before invoking the
  crawler, verify the configured URL resolves to a known Confluence
  host (e.g. `*.atlassian.net` for Cloud, the organisation's known
  on-premises host for Server) — not to a private IP range
  (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `127.0.0.0/8`)
  or a cloud-metadata endpoint (`169.254.0.0/16`). If the user
  supplies an unexpected host, stop and ask them to confirm before
  running. This is an **agent pre-flight check**: the scripts validate
  only the URL scheme (`http://` or `https://`), not the resolved host
  or IP range. On the token path `follow_redirects=True` is active, so
  verify the initial host before invoking.

This skill is **dual-auth** (`auth: sso-cookie` with a `creds` fallback): on a
Data Center instance behind corporate SSO it authenticates by a captured web
session (cookie jar) resolved through the `sso-broker`; everywhere else it uses
the token (`creds`) path above. On the SSO-cookie path:

- The session cookie jar lives only under the broker's `0600` store; the skill
  reads it in-process via the `credbroker` resolver, which returns a *path*, not
  the bytes. **Never** read the jar file directly, print its contents, or echo
  cookie values.
- **Never** put a session cookie on the command line. The skill attaches cookies
  to its HTTP client internally and sends no `Authorization` header on this path.
- Run `--check` first and allow its single headless recovery attempt. The
  automatic attempt shows no browser window and obtains its sign-in destination
  only from CredBroker's registered profile. It never uses `login_url` from
  `sso-config.toml` as an automatic destination.
- Request manual setup with `python scripts/setup_sso.py` only when `--check`
  says automatic recovery refused or failed. That helper opens a browser for
  interactive sign-in, so do not run any setup helper for them.

### Step 1: Verify the environment

Check Python dependencies are installed. If not, install them:

```bash
python -m pip install -r requirements.txt
```

Then verify connectivity:

```bash
python scripts/crawl_space.py --check
```

- Exit code 0 → authenticated, proceed.
- Exit code 2 → read the bounded error. On the token path, missing or invalid
  credentials require user-run `credential-setup`. On the SSO path, request
  user-run `python scripts/setup_sso.py` only when the message says the single
  headless recovery refused or failed. A 403, malformed configuration,
  confinement failure, or dependency problem is terminal for this attempt and
  must be surfaced as written; do not start setup for it. Stop here.
- Any other non-zero → see *When a request fails*.

### When a request fails

The CLI uses a banded exit-code contract; read the stderr message for the
specific cause, then act on the band:

| Exit | Band | What to do |
|---|---|---|
| 0 | success | proceed |
| 1 | functional error — bad/missing args, server 5xx, transport, **a partial crawl (some pages failed)**, keychain hard-fail, unexpected | surface the message; for a partial crawl the per-page failures are in the log — report them, don't loop |
| 2 | user must act — token credentials, SSO recovery refusal/failure, permission/configuration, or dependency problem | follow the bounded message: request the matching manual setup only for missing token credentials or an explicit SSO recovery refusal/failure; surface 403/configuration/confinement/dependency errors without setup, then re-run `--check` only after the user resolves the named cause |
| 130 | interrupted (Ctrl-C) | the run was cancelled; nothing to fix |

`Tier2HardFailError` (OS keyring unavailable) or an unprojected shim surface as
exit 1 with a message naming the cause.

### Step 2: Crawl the space

Invoke the crawler with the user's arguments. Only these flags are supported:

| Flag | Meaning |
|---|---|
| `--space KEY` | Space key, e.g. `ENG`. Required. |
| `--root PAGE_ID` | Start from a specific page (default: space homepage). |
| `--depth N` | Max hierarchy depth from root (default: unlimited). |
| `--output DIR` | Output directory (default: `./confluence-out`). |
| `--force` | Re-fetch and overwrite all pages, ignoring frontmatter version. |
| `--no-attachments` | Skip attachment downloads. |
| `--concurrency N` | Parallel requests (default: 4). |
| `--min-delay-ms N` | Minimum ms between requests (default: 100). |
| `--insecure` | Disable TLS verification. Only if the user explicitly asks. |
| `--verbose` | Debug logging. |

Example:

```bash
python scripts/crawl_space.py --space ENG --depth 3 --output ./out
```

### Step 3: Interpret the output

The script writes:

- `<output>/<slug>.md` per page, flat layout. Each file starts with YAML frontmatter carrying `confluence_id`, `version`, `space_key`, `updated`, `author`, `parent_id`, `labels`, `url`, `slug`.
- `<output>/attachments/<page_id>/<filename>` for downloaded attachments.

The final log line reports `wrote N pages (failed: X, skipped: Y)`. Relay this to the user. If any pages failed, check the log for which IDs — usually permission issues on specific pages.

### Step 4: Re-crawling

The script is idempotent. On re-run:

- It compares each page's current `version.number` against the `version` field in the existing `.md` frontmatter.
- Unchanged pages are skipped.
- Changed pages are re-fetched and overwritten.
- Pass `--force` to bypass the version check and re-fetch everything.

### Behavior notes

- **Depth** is measured in page hierarchy (parent → child), not link hops.
- **Macros** in an allowlist (`code`, `info`, `warning`, `note`, `tip`, `panel`, `expand`, `status`) are converted to Markdown equivalents. Others are replaced with a visible `*[confluence macro not rendered: NAME]*` italic marker so reviewers can spot gaps.
- **Internal links** to pages that were also crawled become relative `.md` paths. Links to pages outside the crawl set remain absolute Confluence URLs.
- **Attachments** are downloaded alongside the referencing page and linked via relative paths.

### Don't

- Don't read `~/.agentbundle/credentials.env` from skill body.
- Don't print or log the PAT.
- Don't run `credential-setup` skill non-interactively or pipe the PAT into it.
- Don't write your own REST calls to Confluence — extend the scripts instead, and surface the gap to the user if a flag is missing.
- Don't assume `--insecure` is safe to add by default. Only when the user explicitly says they accept it.

### Edge cases

- **Cloud base URL without `/wiki`**: if the user's config somehow has `https://foo.atlassian.net` without `/wiki`, API calls will 404. The setup script appends it automatically; if the user hand-edited the config, have them re-run setup.
- **Space has no homepage**: the script exits 2 and asks for `--root PAGE_ID`. Relay this to the user.
- **Orphaned pages not in the hierarchy**: not crawled by design. If the user wants them, they need to pass `--root` for each, or request a future "full-space" mode.
- **Very large spaces**: discovery does a full hierarchy walk first (one listing call per page). Expect a minute or two for thousands of pages. Fetch and convert then runs with bounded concurrency.
- **Title changes between runs**: the old `<old-slug>.md` file remains on disk — the new run writes `<new-slug>.md` because slugs derive from the current title. Warn the user that old files may linger and let them clean up.
- **Network failures mid-crawl**: the `.part` tempfile pattern prevents half-written `.md` files. Re-running resumes cleanly.