firecrawl ยท diff
git:20260728.3aaa44b to git:20260905.b7d83c3
1 added, 1 removed. Audit A to A.
---
name: firecrawl
- description: "Firecrawl CLI for web data. Use to search the web, scrape or crawl a site, map URLs, extract structured data, interact with a page, monitor changes, or parse local PDF, DOCX, XLSX and HTML files."
+ description: "Firecrawl CLI for search, scrape/crawl, map, extract, interact, monitor, and parse local PDF/DOCX/XLSX/HTML."
license: ISC
allowed-tools: Bash(firecrawl *) Bash(bunx --bun firecrawl-cli@latest *) Bash(node *firecrawl-*.mjs *)
---
# Firecrawl CLI
Use the Firecrawl CLI for live web search, URL extraction, site discovery,
bulk crawls, browser-backed interaction, recurring monitors, offline site
download, and local document parsing.
Use the installed CLI as command truth. Run `firecrawl --help` or
`firecrawl <command> --help` before relying on version-sensitive flags. This
skill is written for released `firecrawl-cli` 1.19.x behavior; do not teach or
use unreleased GitHub-main flags unless local help confirms them.
Do not run `firecrawl init`, `firecrawl setup skills`, `firecrawl setup mcp`,
`firecrawl launch`, or `firecrawl make default` from this skill unless the user
explicitly asks for Firecrawl workstation maintenance. Those commands can
modify installed skills, MCP config, or native web-provider defaults.
## First Checks
1. Check setup with `firecrawl --status`.
2. If a command or flag matters, confirm it with `firecrawl <command> --help`.
3. Before paid Firecrawl commands, check `.firecrawl/` for reusable artifacts
with `scripts/firecrawl-cache-index.mjs`; see
[references/cache-reuse.md](references/cache-reuse.md).
4. Write large outputs to `.firecrawl/` with `-o`; do not stream large page
content into the agent context.
5. Quote URLs and paths. Shells treat `?`, `&`, spaces, and brackets specially.
6. Use deterministic artifact names so follow-up commands can find evidence.
7. Do not send private, confidential, repo-proprietary, or secret-bearing
material to Firecrawl unless the user explicitly permits external
processing.
For setup/auth troubleshooting, read [references/install-auth.md](references/install-auth.md).
For output safety, read [references/output-security.md](references/output-security.md).
For local drift checks, run [scripts/firecrawl-doctor.mjs](scripts/firecrawl-doctor.mjs).
## Command Selection
Read [references/command-selection.md](references/command-selection.md) for the
full decision tree. Default order:
1. `search` when no exact URL is known.
2. `scrape` when a URL is known.
3. `map` when a site is known but the exact page is not.
4. `crawl` when many pages from a site/section are needed.
5. `monitor` when the user needs ongoing change tracking.
6. `agent` when the user wants structured data from complex sites and provides
a schema, or schema-like target fields.
7. `interact` only after `scrape` when content requires clicks, forms, login,
pagination, session state, or browser actions.
8. `parse` for local documents, not URLs.
9. `x download` when the user wants a local offline site copy.
10. `research` only for Firecrawl-native public arXiv or GitHub-history
research, and verify important claims against the underlying source.
11. `doctor` and `feedback` for diagnostics and concise upstream quality
feedback.
## Scope And Cost Defaults
- Use `--limit` on `search`, `map`, `crawl`, `agent`, and `x download`.
- Reuse fresh local `.firecrawl` artifacts before spending credits.
- Prefer `map --search` plus targeted `scrape` before broad crawls.
- Keep `crawl` scoped with `--include-paths`, `--exclude-paths`, `--max-depth`,
and `--wait`.
- Use `agent --max-credits` and a schema for complex structured extraction.
- Use `--redact-pii` for contact pages, PDFs, user-generated pages, lead
research, or anything likely to enter logs, shared artifacts, or vector
stores.
- Do not crawl whole domains, allow external links, or allow subdomains unless
the user explicitly needs that breadth.
## Default Recipes
Prefer these short chains before opening a detailed reference. Read
[references/recipes.md](references/recipes.md) for schema, monitor JSON, jq,
output-shape probes, profile, feedback, and download variants.
Search local cache before fetching a known URL:
```bash
FIRECRAWL_SKILL_DIR="${FIRECRAWL_SKILL_DIR:-$HOME/.agents/skills/firecrawl}"
node "$FIRECRAWL_SKILL_DIR/scripts/firecrawl-cache-index.mjs" find \
--url "https://example.com/page" \
--intent docs \
--json
```
Search with page content, inspect, then send feedback:
```bash
firecrawl search "query" --scrape --json -o .firecrawl/search-query.json
jq -r '.data.web[] | "\(.title): \(.url)"' .firecrawl/search-query.json
firecrawl search-feedback "$(jq -r '.id' .firecrawl/search-query.json)" --rating good --valuable-sources '[{"url":"https://example.com","reason":"Useful result"}]' --silent &
```
Find a page on a known site, then scrape it:
```bash
firecrawl map "https://docs.example.com" --search "authentication" --json -o .firecrawl/map-auth.json
firecrawl scrape "https://docs.example.com/auth-page" -o .firecrawl/auth-page.md
```
Scoped docs crawl:
```bash
firecrawl crawl "https://docs.example.com" --include-paths /docs --limit 50 --wait --pretty -o .firecrawl/crawl-docs.json
```
Scrape, interact, then stop:
```bash
firecrawl scrape "https://example.com" --profile example-site
firecrawl interact "Click the pricing tab and extract the visible plans"
firecrawl interact stop
```
Parse a local document:
```bash
firecrawl parse "./report.pdf" -o .firecrawl/report.md
```
Offline docs copy:
```bash
firecrawl x download "https://docs.example.com" --include-paths /docs --format markdown,links --limit 50 -y
```
Basic recurring monitor:
```bash
firecrawl monitor create --name "Changelog" --schedule "every 30 minutes" --scrape-urls "https://example.com/changelog"
```
## Artifact Names
Create `.firecrawl/` first and use names that encode command plus subject:
```text
.firecrawl/search-<slug>.json
.firecrawl/search-<slug>-scraped.json
.firecrawl/map-<site>-<topic>.json
.firecrawl/scrape-<site>-<page>.md
.firecrawl/scrape-<site>-<page>.json
.firecrawl/crawl-<site>-<scope>.json
.firecrawl/agent-<task>.json
.firecrawl/monitor-<name>.json
.firecrawl/parse-<document>.md
.firecrawl/schema-<purpose>.json
.firecrawl/index.jsonl
```
## Evidence Closeout
Before finalizing web-data work, preserve enough evidence to audit the answer:
```bash
printf 'Command: %s\nArtifact: %s\n' \
'firecrawl scrape "https://example.com/page" -o .firecrawl/scrape-example-page.md' \
'.firecrawl/scrape-example-page.md' \
> .firecrawl/scrape-example-page.evidence.txt
rg -n "pricing|limit|changed|released" .firecrawl/scrape-example-page.md \
>> .firecrawl/scrape-example-page.evidence.txt
```
Final answers should cite source URLs and local artifact paths when Firecrawl
evidence materially supports the claim.
## Failure Recovery
- Auth/401: run `firecrawl --status`; see `references/install-auth.md`.
- Credits/402 or rate limit: reduce `--limit`, narrow scope, or stop and report.
- Failed run/job: use `firecrawl doctor --json` or
`firecrawl doctor <job-id> --query "why did this run fail?"`.
- Timeout: add `--timeout`, reduce scope, or use `--wait-for` for rendering.
- Blocked or JS-heavy page: retry `scrape` with `--wait-for`; escalate to
`interact` only after a successful scrape.
- Monitor unavailable: report the account/retention limitation and use one-off
scrape plus local diff instead.
- Missing interact session: run `scrape` first and pass `--profile` or
`--scrape-id`.
- Malformed JSON: save raw output, validate with `jq`, and rerun with `--json`
or `--pretty`.
- CLI help differs from this skill: trust local `firecrawl <command> --help`
and update the skill later.
## Reference Loading
- Rich command chains: [references/recipes.md](references/recipes.md)
- Local cache/reuse: [references/cache-reuse.md](references/cache-reuse.md)
- Reusable schemas: [references/schemas.md](references/schemas.md)
- Search/discovery first: [references/search.md](references/search.md)
- Known URL extraction: [references/scrape.md](references/scrape.md)
- Site URL discovery: [references/map.md](references/map.md)
- Bulk site extraction: [references/crawl.md](references/crawl.md)
- AI structured extraction: [references/agent.md](references/agent.md)
- Browser interaction: [references/interact.md](references/interact.md)
- Local document parsing: [references/parse.md](references/parse.md)
- Recurring change tracking: [references/monitor.md](references/monitor.md)
- Offline site copy: [references/download.md](references/download.md)
- Firecrawl-native public paper/GitHub research:
[references/research.md](references/research.md)
- Local maintenance/drift: [references/maintenance.md](references/maintenance.md)
For Firecrawl SDK/API integration into an application, adding
`FIRECRAWL_API_KEY` to a project, or choosing product endpoints, do not use
this CLI skill as the implementation authority. Use the Firecrawl build skills
if installed. For outcome deliverables such as research briefs, SEO audits,
lead lists, QA reports, or design extraction, use the dedicated Firecrawl
workflow skills if installed.
## Output Defaults
Use `.firecrawl/` for fetched or parsed output unless the user explicitly wants
inline content:
```bash
mkdir -p .firecrawl
firecrawl search "query" --json -o .firecrawl/search-query.json
firecrawl scrape "https://example.com/page" -o .firecrawl/example-page.md
```
Inspect output incrementally:
```bash
wc -l .firecrawl/example-page.md
head -80 .firecrawl/example-page.md
rg -n "pricing|authentication" .firecrawl/example-page.md
```
Single-format scrape/parse output is raw content. Multiple formats usually
return JSON. When using `search --scrape`, do not re-scrape those result URLs
unless the scraped payload is missing what the task needs.
For command-specific output shapes and resilient `jq` probes, open the matching
reference file rather than a separate shape reference.
## Validation
When maintaining this skill:
```bash
node scripts/firecrawl-doctor.mjs --json
node scripts/firecrawl-help-snapshot.mjs --output /tmp/firecrawl-help.json
```
The scripts are diagnostics only. They do not wrap Firecrawl operations and
they do not install Firecrawl skills.