web-scraping SKILL.md skillA

This file is byte-identical to the first copy the registry indexed. Same content hash, so the same checks give both the same grade.

web-scraping from slilbudget/knowledge-delta-skills is an agent skill: a SKILL.md folder an AI agent loads when a task calls for it.

mdr add slilbudget/knowledge-delta-skills/web-scraping@git:20260929.c1f17bd

Pins git:20260929.c1f17bd in mdr.lock. Pin the fingerprint instead.

Audit grade

A, 17 of 17 checks passed

Kind

SKILL.md skill

Repository

slilbudget/knowledge-delta-skills

Path

skills/engineering/web-scraping/SKILL.md

Versions recorded

1

Latest version

git:20260929.c1f17bd, committed Sep 29, 2026

Size

4,140 bytes

Repository stars

3

License

MIT

Mass crawl/scrape sites incl. anti-bot (Cloudflare, Akamai, DataDome, PerimeterX). Crawl4AI = scale (arun_many, dispatchers, deep crawl, LLM-ready markdown); Camoufox = fingerprint evasion. For: scrape N pages, crawl a site, extract structured data, 403/challenge/bot-detection, captcha.

Collected from public GitHub. Every version is saved exactly as it was and never changes. Install it at an exact version with the mdr tool, and every file is checked before it reaches your agent. The main file is checked against the SHA-256 fingerprint recorded here, and the others against the fingerprints GitHub keeps for its source commit. The automatic checks below grade the latest version, and the same checks always give the same file the same grade.

What the file says

# Mass Scraping — Crawl4AI + Camoufox

Verified 2026-08-27 (macOS arm64). Both engines and all three templates smoke-tested.

## Install (if `crawl4ai-python` is not on PATH)

- **crawl4ai** — `github.com/unclecode/crawl4ai` · PyPI `crawl4ai`
- **camoufox** — `github.com/daijro/camoufox` · PyPI `camoufox[geoip]`
- **curl_cffi** — `github.com/lexiforest/curl_cffi` · PyPI `curl_cffi`

Dedicated venv, then `crawl4ai-setup` and `python -m camoufox fetch`. Four traps, each verified here:

1. **Python 3.10–3.13 only.** 3.14 is not supported — if `python3` is 3.14, install 3.13 explicitly.
2. **`crawl4ai`, not `crawl4ai[all]`.** The extra pulls torch + transformers (GBs) that scraping never uses.
3. **`crawl4ai-setup`, not `playwright install chromium`.** Setup also installs **Patchright**, which
   rung 3 needs; without it the undetected adapter dies.
4. **Shim the venv python with a wrapper, never a symlink.** A symlinked venv python resolves
   `sys.prefix` to the system interpreter and cannot import crawl4ai:
   ```sh
   printf '#!/bin/sh\nexec "$VENV/bin/python" "$@"\n' > ~/.local/bin/crawl4ai-python
   ```

## Entry points

```bash
…

Read the whole file at its exact version.

How to install

mdr add slilbudget/knowledge-delta-skills/web-scraping@sha256:085b47a0120b4fb2

[![mdr](https://markdownregistry.com/badge/art_fokqbgaqviwufhih.svg)](https://markdownregistry.com/a/art_fokqbgaqviwufhih)

Versions

The registry has recorded 1 version of this file, committed on Sep 29, 2026.

Audit of the latest version

Source

API

GET https://markdownregistry.com/api/v1/artifacts/art_fokqbgaqviwufhih
GET https://markdownregistry.com/api/v1/resolve?ref=slilbudget/knowledge-delta-skills/web-scraping
GET https://markdownregistry.com/api/v1/blob/085b47a0120b4fb2d8feb549046400fe12c4aa591d1adec7abc8d2b30c5ec824

Your AI assistant does the legwork. You hear about the deals worth your word. Hand yours the standing instructions at modelranch.com and it joins the network that reads files like this one.

Guides: the SKILL.md format and its required fields · what to check before you install a skill · how to pin a skill to an exact version · where to find agent skills · keeping a team on the same skills · which agent file does what.

More from slilbudget/knowledge-delta-skills

Every file in slilbudget/knowledge-delta-skills

Other files named web-scraping

Browse by kind, by grade A, or by owner.