Mass crawl/scrape sites incl. anti-bot (Cloudflare, Akamai, DataDome, PerimeterX). Crawl4AI = scale (arun_many, dispatchers, deep crawl, LLM-ready markdown); Camoufox = fingerprint evasion. For: scrape N pages, crawl a site, extract structured data, 403/challenge/bot-detection, captcha.
Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.
What the file says
# Mass Scraping — Crawl4AI + Camoufox
Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested.
## Install (if `crawl4ai-python` is not on PATH)
- **crawl4ai** — `github.com/unclecode/crawl4ai` · PyPI `crawl4ai`
- **camoufox** — `github.com/daijro/camoufox` · PyPI `camoufox[geoip]`
- **curl_cffi** — `github.com/lexiforest/curl_cffi` · PyPI `curl_cffi`
Dedicated venv, then `crawl4ai-setup` and `python -m camoufox fetch`. Four traps, each verified here:
1. **Python 3.10–3.13 only.** 3.14 is not supported — if `python3` is 3.14, install 3.13 explicitly.
2. **`crawl4ai`, not `crawl4ai[all]`.** The extra pulls torch + transformers (GBs) that scraping never uses.
3. **`crawl4ai-setup`, not `playwright install chromium`.** Setup also installs **Patchright**, which
rung 3 needs; without it the undetected adapter dies.
4. **Shim the venv python with a wrapper, never a symlink.** A symlinked venv python resolves
`sys.prefix` to the system interpreter and cannot import crawl4ai:
```sh
printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python
```
## Entry points
```bash
…
The line at the top of this page pins the label git:20260929.c1f17bd, which is what most people want. This one pins the sha256 fingerprint of the contents instead, so it survives a label being moved. Either way the exact version is written to mdr.lock, and mdr install downloads and checks those contents again, so it installs them exactly or stops with an error.
GET https://markdownregistry.com/api/v1/artifacts/art_uvt2izbj2zijziwi
GET https://markdownregistry.com/api/v1/resolve?ref=sergeyizmailov/knowledge-delta-skills/web-scraping
GET https://markdownregistry.com/api/v1/blob/085b47a0120b4fb2d8feb549046400fe12c4aa591d1adec7abc8d2b30c5ec824
Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.
sergeyizmailov/knowledge-delta-skills · skills/engineering/deep-research/SKILL.md · Use when asked to research deeply, compare options or vendors, fact-check a claim, investigate an unfamiliar topic…
sergeyizmailov/knowledge-delta-skills · skills/engineering/knowledge-delta-skill-architect/SKILL.md · Triggers: write a skill for X, improve/shrink this skill, is this skill worth its tokens, my skill never triggers, port…
sergeyizmailov/knowledge-delta-skills · skills/frontend/design-stack-picker/SKILL.md · Use when building or restyling frontend UI and choosing building blocks — icon sets, fonts, component/block libraries…
sergeyizmailov/knowledge-delta-skills · skills/frontend/normcore-web/SKILL.md · Use when building or restyling a website that should look like ordinary, long-running commercial web rather than a…
sergeyizmailov/knowledge-delta-skills · skills/frontend/responsive-adapter/SKILL.md · Make an existing landing page, admin panel, dashboard, or any non-responsive page work at every viewport from 320px to…
sergeyizmailov/knowledge-delta-skills · skills/meta/measurement-experimentation-ops/SKILL.md · Gatekeeper for 'is this result real before scaling' on Meta/Facebook buys: SRM, peeking, contamination, lag and…
sergeyizmailov/knowledge-delta-skills · skills/meta/meta-ads/SKILL.md · Meta (Facebook/Instagram) Ads theory and diagnostics for the policy-compliant (clean) lane: 'why is CPA up', 'CBO vs…
sergeyizmailov/knowledge-delta-skills · skills/meta/meta-grey-ops/SKILL.md · Launches, monitors and operates Meta (Facebook/Instagram) ads for any vertical and offer through the metaops CLI…
sergeyizmailov/knowledge-delta-skills · skills/meta/senior-buyer-ops/SKILL.md · Senior buyer operations for Meta (FB/IG): portfolio-level decisions on what to kill, watch or scale across accounts…
sergeyizmailov/knowledge-delta-skills · skills/security/js-obfuscation/SKILL.md · Use when obfuscating JavaScript, building anti-detection layers, evading Google Safe Browsing, hiding payloads from…
maxmiksa/auto-company · .claude/skills/web-scraping/SKILL.md · Web scraping with anti-bot bypass, content extraction, undocumented APIs and poison pill detection. Use when extracting…
ginlix-ai/langalpha · plugins/alternative_data/skills/web-scraping/SKILL.md · Web scraping: scrape_page / scrape_pages MCP tools for fetching pages as markdown, HTML, or text (fast HTTP, browser…
platonai/browser4 · browser4-agentic/src/main/resources/skills/web-scraping/SKILL.md · Extracts data from web pages using browser automation and CSS/JavaScript selectors. Use when you need to scrape…
jamditis/claude-skills-journalism · dev-toolkit/skills/web-scraping/SKILL.md · Authorized web scraping with fallback cascades and access-failure handling. Use for social media, yt-dlp, CAPTCHA or…
billy-enrizky/openbrowser-ai · plugin/skills/web-scraping/SKILL.md · Extract structured data from websites, scrape page content, and collect information across multiple pages.
Trigger when…
