web-fetch ยท diff
git:20260113.222b7de to git:20260705.e9e4712
65 added, 36 removed. Audit A to A.
---
name: web-fetch
- description: Fetches web content with intelligent content extraction, converting HTML to clean markdown. Use for documentation, articles, and reference pages http/https URLs.
+ description: Fetches web content as clean markdown by preferring markdown-native responses and falling back to selector-based HTML extraction. Use for documentation, articles, and reference pages at http/https URLs.
---
# Web Content Fetching
- Fetch web content using `curl | html2markdown` with CSS selectors for clean, complete markdown output.
+ Fetch web content in this order:
+ 1. Prefer markdown-native endpoints (`content-type: text/markdown`)
+ 2. Use selector-based HTML extraction for known sites
+ 3. Use the bundled Bun fallback script when selectors fail
- ## Quick Usage (Known Sites)
+ ## Prerequisites
- Use site-specific selectors for best results:
+ Verify required tools before extracting:
```bash
- # Anthropic docs
- curl -s "<url>" | html2markdown --include-selector "#content-container"
+ command -v curl >/dev/null || echo "curl is required"
+ command -v html2markdown >/dev/null || echo "html2markdown is required for HTML extraction"
+ command -v bun >/dev/null || echo "bun is required for fetch.ts fallback"
+ ```
- # MDN Web Docs
- curl -s "<url>" | html2markdown --include-selector "article"
+ Install Bun dependencies for the bundled script:
- # GitHub docs
- curl -s "<url>" | html2markdown --include-selector "article" --exclude-selector "nav,.sidebar"
+ ```bash
+ cd ~/.claude/skills/web-fetch && bun install
+ ```
- # Generic article pages
- curl -s "<url>" | html2markdown --include-selector "article,main,[role=main]" --exclude-selector "nav,header,footer"
+ ## Default Workflow
+
+ Use this as the default flow for any URL:
+
+ ```bash
+ URL="<url>"
+ CONTENT_TYPE="$(curl -sIL "$URL" | awk -F': ' 'tolower($1)=="content-type"{print tolower($2)}' | tr -d '\r' | tail -1)"
+
+ if echo "$CONTENT_TYPE" | grep -q "markdown"; then
+ curl -sL "$URL"
+ else
+ curl -sL "$URL" \
+ | html2markdown \
+ --include-selector "article,main,[role=main]" \
+ --exclude-selector "nav,header,footer,script,style"
+ fi
```
- ## Site Patterns
+ ## Known Site Selectors
| Site | Include Selector | Exclude Selector |
|------|------------------|------------------|
| platform.claude.com | `#content-container` | - |
| docs.anthropic.com | `#content-container` | - |
| developer.mozilla.org | `article` | - |
| github.com (docs) | `article` | `nav,.sidebar` |
- | Generic | `article,main` | `nav,header,footer,script,style` |
-
- ## Universal Fallback (Unknown Sites)
-
- For sites without known patterns, use the Bun script which auto-detects content:
-
- ```bash
- bun ~/.claude/skills/web-fetch/fetch.ts "<url>"
- ```
+ | Generic | `article,main,[role=main]` | `nav,header,footer,script,style` |
- ### Setup (one-time)
+ Example:
```bash
- cd ~/.claude/skills/web-fetch && bun install
+ curl -sL "<url>" \
+ | html2markdown \
+ --include-selector "#content-container" \
+ --exclude-selector "nav,header,footer"
```
## Finding the Right Selector
When a site isn't in the patterns list:
```bash
# Check what content containers exist
curl -s "<url>" | grep -o '<article[^>]*>\|<main[^>]*>\|id="[^"]*content[^"]*"' | head -10
# Test a selector
- curl -s "<url>" | html2markdown --include-selector "<selector>" | head -30
+ curl -sL "<url>" | html2markdown --include-selector "<selector>" | head -30
# Check line count
- curl -s "<url>" | html2markdown --include-selector "<selector>" | wc -l
+ curl -sL "<url>" | html2markdown --include-selector "<selector>" | wc -l
```
+ ## Universal Fallback Script
+
+ When selectors produce poor output, run the bundled parser:
+
+ ```bash
+ bun ~/.claude/skills/web-fetch/fetch.ts "<url>"
+ ```
+
+ If already in the skill directory:
+
+ ```bash
+ bun fetch.ts "<url>"
+ ```
+
## Options Reference
```bash
- --include-selector "CSS" # Only include matching elements
+ --include-selector "CSS" # Keep only matching elements
--exclude-selector "CSS" # Remove matching elements
--domain "https://..." # Convert relative links to absolute
```
- ## Comparison
+ ## Troubleshooting
- | Method | Anthropic Docs | Code Blocks | Complexity |
- |--------|----------------|-------------|------------|
- | Full page | 602 lines | Yes | Noisy |
- | `--include-selector "#content-container"` | 385 lines | Yes | Clean |
- | Bun script (universal) | 383 lines | Yes | Clean |
+ **Empty output with selectors**: The page might be markdown-native. Check headers first:
- ## Troubleshooting
+ ```bash
+ curl -sIL "<url>" | grep -i '^content-type:'
+ ```
- **Wrong content selected**: The site may have multiple articles. Inspect the HTML:
+ **Wrong content selected**: The site may have multiple article/main regions:
+
```bash
curl -s "<url>" | grep -o '<article[^>]*>'
```
- **Empty output**: The selector doesn't match. Try broader selectors like `main` or `body`.
+ **`html2markdown` not found**: Install it, then retry selector-based extraction.
+
+ **`bun` or script deps missing**: Run `cd ~/.claude/skills/web-fetch && bun install`.
**Missing code blocks**: Check if the site uses non-standard code formatting.
**Client-rendered content**: If HTML only has "Loading..." placeholders, the content is JS-rendered. Neither curl nor the Bun script can extract it; use browser-based tools.