web-fetch · v1.1.0 · 2026-03-27 · sha256 eb8f5f2e8a9bc678
web-fetch v1.1.0B
Immutable. This exact content never changes and is served at /api/v1/blob/eb8f5f2e8a9bc678.
---
name: web-fetch
description:
'Web content fetching and URL retrieval via curl and WebFetch — replaces
the Fetch MCP server (fetch_html, fetch_json, fetch_markdown, fetch_txt). Use this
skill when a specific URL is provided and the user wants its content. Covers HTTP
GET/POST, JSON API consumption with jq, HTML retrieval, markdown conversion, plain
text extraction, authenticated requests, redirects, cookies, and timeouts. Trigger
phrases: "fetch this URL", "get the page content", "download HTML", "call this API",
"curl this endpoint", "grab the JSON", "fetch markdown from", "retrieve web content",
"hit this endpoint", "scrape this page", "read this URL", "pull data from API",
"make an HTTP request", "extract page content", "get article text". NOT for web
searches without a URL — use tavily for that.
'
metadata:
version: 1.1.0
category: content
tags: [http, curl, api, web-content]
difficulty: beginner
---
# Web Fetch
All web content retrieval uses `curl` (Bash) or the built-in `WebFetch` tool. No MCP
server needed — Claude Code's native tools cover every Fetch MCP operation with more
control.
## Quick Reference
| Fetch MCP Tool | Replacement | When to Use |
| ---------------- | --------------------------- | -------------------------------------------------- |
| `fetch_html` | `curl -s URL` | Raw HTML needed for parsing |
| `fetch_json` | `curl -s URL \| jq '.'` | API responses, structured data |
| `fetch_markdown` | `WebFetch` | Readable page content (default output is markdown) |
| `fetch_txt` | `curl -s URL` or `WebFetch` | Plain text extraction |
**Default choice:** Use `WebFetch` for general page content. Use `curl` when you need
headers, authentication, POST bodies, or raw format control.
---
## WebFetch (Built-in Tool)
The `WebFetch` tool fetches a URL and returns clean markdown content. It handles
JavaScript-rendered pages, strips navigation and boilerplate, and returns readable text.
Best for: documentation pages, articles, blog posts, README files — any content where
you want readable text rather than raw HTML.
Limitations: no custom headers, no POST bodies, no cookie management. Use `curl` for
those.
---
## curl Patterns
### Fetch HTML
```bash
curl -sL "https://example.com/page"
```
| Flag | Purpose |
| -------------- | ------------------------------------- |
| `-s` | Silent mode — suppress progress meter |
| `-L` | Follow redirects (3xx) |
| `-o file.html` | Save to file instead of stdout |
| `-I` | Headers only (HEAD request) |
| `-i` | Include response headers in output |
Fetch and extract specific elements with `xmllint` or `python3`:
```bash
curl -sL "https://example.com" | python3 -c "
from html.parser import HTMLParser
import sys
class TitleParser(HTMLParser):
def __init__(self):
super().__init__()
self.in_title = False
self.title = ''
def handle_starttag(self, tag, attrs):
self.in_title = tag == 'title'
def handle_data(self, data):
if self.in_title:
self.title += data
def handle_endtag(self, tag):
if tag == 'title':
self.in_title = False
p = TitleParser()
p.feed(sys.stdin.read())
print(p.title)
"
```
### Fetch JSON
```bash
curl -s "https://api.example.com/v1/data" \
-H "Accept: application/json" | jq '.'
```
Filter and reshape JSON responses:
```bash
# Extract specific fields
curl -s "https://api.example.com/users" | jq '.[] | {name, email}'
# Filter by condition
curl -s "https://api.example.com/items" | jq '[.[] | select(.status == "active")]'
# Count results
curl -s "https://api.example.com/items" | jq 'length'
# Get nested value
curl -s "https://api.example.com/config" | jq '.database.host'
```
### Fetch Plain Text
```bash
# Strip HTML tags for plain text
curl -sL "https://example.com/page" | python3 -c "
import html.parser, sys
class Stripper(html.parser.HTMLParser):
def __init__(self):
super().__init__()
self.text = []
def handle_data(self, d):
self.text.append(d)
def get_text(self):
return ''.join(self.text)
s = Stripper()
s.feed(sys.stdin.read())
print(s.get_text())
"
```
Or use `WebFetch` which returns clean markdown — close enough to plain text for most
purposes.
---
## Authenticated Requests
### Bearer Token
```bash
curl -s "https://api.example.com/data" \
-H "Authorization: Bearer $API_TOKEN" \
-H "Content-Type: application/json"
```
### API Key in Header
```bash
curl -s "https://api.example.com/data" \
-H "X-API-Key: $API_KEY"
```
### API Key in Query Parameter
```bash
curl -s "https://api.example.com/data?api_key=$API_KEY"
```
### Basic Auth
```bash
curl -s -u "username:$PASSWORD" "https://api.example.com/data"
```
Store credentials in environment variables. Never hardcode tokens or passwords in
commands.
---
## POST, PUT, PATCH, DELETE
### POST with JSON Body
```bash
curl -s -X POST "https://api.example.com/items" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_TOKEN" \
-d '{
"name": "item-name",
"value": 42
}' | jq '.'
```
### POST with Form Data
```bash
curl -s -X POST "https://api.example.com/upload" \
-F "file=@./document.pdf" \
-F "description=Uploaded via curl"
```
### PUT (Full Update)
```bash
curl -s -X PUT "https://api.example.com/items/123" \
-H "Content-Type: application/json" \
-d '{"name": "updated-name", "value": 99}' | jq '.'
```
### PATCH (Partial Update)
```bash
curl -s -X PATCH "https://api.example.com/items/123" \
-H "Content-Type: application/json" \
-d '{"value": 100}' | jq '.'
```
### DELETE
```bash
curl -s -X DELETE "https://api.example.com/items/123" \
-H "Authorization: Bearer $API_TOKEN"
```
---
## Advanced Patterns
### Pagination
```bash
PAGE=1
while true; do
RESPONSE=$(curl -s "https://api.example.com/items?page=$PAGE&per_page=50" \
-H "Authorization: Bearer $API_TOKEN")
COUNT=$(echo "$RESPONSE" | jq 'length')
echo "$RESPONSE" | jq '.[]'
[ "$COUNT" -lt 50 ] && break
PAGE=$((PAGE + 1))
done
```
### Timeout and Retry
```bash
curl -s --connect-timeout 10 --max-time 30 \
--retry 3 --retry-delay 2 \
"https://api.example.com/data"
```
### Response Headers Inspection
```bash
curl -sI "https://example.com" | grep -i "content-type"
```
### Save Response with Status Code
```bash
HTTP_CODE=$(curl -s -o /tmp/response.json -w "%{http_code}" "https://api.example.com/data")
echo "Status: $HTTP_CODE"
cat /tmp/response.json | jq '.'
```
### Cookie Handling
```bash
# Save cookies
curl -s -c /tmp/cookies.txt "https://example.com/login" \
-d "user=admin&pass=$PASSWORD"
# Reuse cookies
curl -s -b /tmp/cookies.txt "https://example.com/dashboard"
```
---
## Error Handling
| HTTP Status | Meaning | Resolution |
| ----------- | ------------------- | -------------------------------------------------------- |
| 301/302 | Redirect | Add `-L` flag to follow |
| 401 | Unauthorized | Check token/credentials; verify env var is set |
| 403 | Forbidden | Insufficient permissions or IP restriction |
| 404 | Not Found | Verify URL path; resource may be deleted |
| 429 | Rate Limited | Respect `Retry-After` header; add delay between requests |
| 500 | Server Error | Retry once; if persistent, report upstream |
| SSL error | Certificate issue | Do not use `-k` (insecure) — fix the root cause |
| Timeout | Network/server slow | Increase `--max-time`; check connectivity |
Verify a URL is reachable before complex operations:
```bash
curl -sI -o /dev/null -w "%{http_code}" "https://example.com"
```
---
## Limitations
- **WebFetch** does not support custom headers, POST bodies, or cookies. Use `curl` for
authenticated or stateful requests.
- **curl** does not render JavaScript. For JS-heavy SPAs, prefer `WebFetch` which handles
rendered content.
- **Large responses** may exceed context limits. Pipe through `jq`, `head`, or `python3`
to extract only needed data before loading into context.
- **Binary content** (images, PDFs, archives) should be saved to disk with `-o`, not
piped to stdout.
---
## Calibration Rules
1. **Default to WebFetch for reading web pages.** It returns clean markdown, handles JS
rendering, and requires no flags. Switch to curl only when you need headers, auth,
POST, or raw format control.
2. **Always pipe JSON through jq.** Raw JSON in context wastes tokens. Filter to only the
fields needed.
3. **Never hardcode credentials.** Use `$ENV_VAR` references. If the variable is not set,
surface the error immediately.
4. **Follow redirects by default.** Always use `-L` with curl unless you specifically need
to inspect the redirect chain.
5. **Prefer `-s` (silent) on every curl call.** Progress meters add noise to output.