wikimedia-api-access git:20260623.70ec56aA

Immutable. This exact content never changes and is served at /api/v1/blob/ce729e1ec1351f9c.

---
name: wikimedia-api-access
description: Access Wikipedia and Wikimedia APIs (REST, Action API, SPARQL) with correct User-Agent headers, rate limiting, and 429/403 error handling
license: MIT
compatibility: opencode
skill_discovery_hints:
  - keywords: ["Wikipedia API", "Action API", "REST API", "API endpoint", "api.php", "rest_v1", "User-Agent", "rate limit", "API call"]
  - keywords: ["Site Matrix", "sitematrix", "domain mapping", "language code", "language domain", "yue wikipedia", "zh-yue", "interlanguage"]
  - keywords: ["page summary", "page extract", "extintro", "exintro", "page content", "fetch article", "get page"]
last_verified: 2026-06-10
---

All requests to Wikimedia APIs **must** include a descriptive `User-Agent` header or they will be blocked (HTTP 403 or 429). This is enforced by the [Wikimedia Foundation User-Agent Policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy).

> **💡 Pi users:** If you use the [pi coding agent](https://github.com/earendil-works/pi), install the companion extension from this repo to **automatically inject the User-Agent header** on every `curl`, `wget`, `python`, and `node` command that targets a Wikimedia server — no manual intervention needed. See the [README](../../../README.md#pi-agent-setup) for setup instructions.

> 💡 **Not sure which tool to use?** See **[wikimedia-api-strategy](../wikimedia-api-strategy/SKILL.md)** for a decision framework comparing REST API, Action API, SPARQL, SQL replicas, EventStreams, and Pywikibot — with latency/complexity/authentication trade-offs and common anti-patterns.

## **User-Agent Format**

The generic format is:

```
<client name>/<version> (<contact information>) <project identifier>/<version>
```

Parts that are not applicable can be omitted. The contact information should be an email address, website URL, or wiki user page.

### **Required Pattern**

```python
headers = {
    'User-Agent': 'YourBot/1.0 (https://your-site.com; your@email.com) ProjectName'
}
```

The `ProjectName` (e.g. `ContentGapResearch`) helps Wikimedia distinguish traffic from different tools.

### **Concrete Example**

```python
import requests

headers = {
    'User-Agent': 'my-wiki-bot/1.0 (https://example.com; user@example.com) ContentGapResearch'
}

response = requests.get('https://en.wikipedia.org/w/api.php', headers=headers, timeout=30)
```

## **Why This Matters**

- Requests without a `User-Agent` or with generic agents (`python-requests/x`, `curl`, `Python-urllib`) are blocked with HTTP **403** or **429** errors.
- A descriptive UA allows Wikimedia to contact you if your bot misbehaves.
- Always include "bot" in the UA string (case-insensitive) for automated agents — this helps Wikimedia classify traffic correctly and provide accurate statistics.
- **Never** copy a browser's UA string for bot requests — bot-like behavior with a browser UA will be treated as malicious.

## **Key API Endpoints**

| API | Base URL | Common Use |
|-----|----------|------------|
| Action API | `https://en.wikipedia.org/w/api.php` | Search, page content, edits, categories |
| REST API | `https://en.wikipedia.org/api/rest_v1/` | Page summaries, mobile content, transforms |
| Pageviews | `https://wikimedia.org/api/rest_v1/metrics/pageviews/` | Traffic statistics |
| Commons Analytics | `https://wikimedia.org/api/rest_v1/metrics/commons-analytics/` | GLAM category/file usage stats (monthly, pre-compiled) |
| Lift Wing ML | `https://api.wikimedia.org/service/lw/inference/v1/models/` | ML predictions (revert risk, article quality, topics) |
| Site Matrix | `https://en.wikipedia.org/w/api.php?action=sitematrix` | Get correct domains for all Wikimedia wikis (handles code/domain mismatches like `yue`→`zh-yue`) |
| Wikidata Query (SPARQL) | `https://query.wikidata.org/sparql` | Structured data queries |
| Toolforge | `https://*.toolforge.org/` | Community tool hosting |

> 💡 **These APIs are project-agnostic.** The Action API and REST API shown above with `en.wikipedia.org` are MediaWiki APIs that work on **any** Wikimedia project — just swap the domain. For example:
> - **Commons:** `https://commons.wikimedia.org/w/api.php` — search media files, fetch EXIF metadata, inspect categories
> - **Wikidata:** `https://www.wikidata.org/w/api.php` — query entities, manage labels and descriptions, fetch claims
> - **Wiktionary:** `https://en.wiktionary.org/w/api.php` — dictionary data, word definitions, etymology
> - **Meta-Wiki:** `https://meta.wikimedia.org/w/api.php` — global user info, cross-wiki settings, Wikimedia Foundation policies
>
> The query parameters and response structures are the same; only the domain and the wiki's content differ.

## **General Implementation Pattern**

```python
import requests
import time

SESSION = requests.Session()
SESSION.headers.update({
    'User-Agent': 'my-wiki-bot/1.0 (https://example.com; user@example.com) ContentGapResearch'
})

def wikimedia_request(url, params=None, max_retries=3):
    for attempt in range(max_retries):
        resp = SESSION.get(url, params=params, timeout=30)
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code == 429:
            retry_after = int(resp.headers.get('Retry-After', 10))
            time.sleep(retry_after)
            continue
        if resp.status_code == 403:
            raise PermissionError(
                "403 Forbidden — likely a missing or invalid User-Agent. "
                "See: https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy"
            )
        resp.raise_for_status()
    raise Exception(f"Request failed after {max_retries} retries")

# Typical response structure (Action API query):
# {
#   "batchcomplete": "",
#   "query": {
#     "pages": {
#       "12345": {                                # ← page ID as string key
#         "pageid": 12345,
#         "ns": 0,
#         "title": "Albert Einstein",
#         ...other requested properties...
#       }
#     }
#   },
#   "continue": {"continue": "-||", ...}        # ← pagination cursor (if more results)
# }
# Access: data["query"]["pages"]["12345"]["title"]
# Iterate: for page_id, page_data in data["query"]["pages"].items():
```

### **Browser-Based JavaScript**

Browser JavaScript cannot control the `User-Agent` header (the browser sets it). Instead, use the `Api-User-Agent` header:

```javascript
fetch('https://en.wikipedia.org/w/api.php?action=query&format=json', {
    headers: new Headers({
        'Api-User-Agent': 'MyScript/1.0'
    })
})
```

## **Rate Limiting & Error Handling Guardrails**

1. **Connection reuse** — Always use a `requests.Session()` (or equivalent) to reuse connections. Do not create a new connection per request.
2. **Retry-After** — On 429, respect the `Retry-After` header value. Never retry immediately.
3. **Pacing & Batching** — For batch operations, add a small delay (at least 0.5s) between requests. For the Action API, respect the `maxlag` parameter. **Always use the largest batch size the API supports** (e.g., `rvlimit=500`, `uclimit=500`) rather than fetching items one at a time. See the **SOP: Batching and Pagination for Efficiency** in the [`wikipedia-edit-history`](../wikipedia-edit-history/SKILL.md) skill for detailed patterns.
4. **403 handling** — A 403 almost always means a bad/missing UA. Check the UA string before debugging anything else.
5. **User-Agent per project** — Parameterize the contact info so users can swap in their own details. Never hardcode someone else's email.
6. **SPARQL queries** — For Wikidata Query Service, always set the UA and use `&format=json`. Consider using `SPARQLWrapper` with the `agent` parameter.

### 429 Retry-After Handling

When a 429 (Too Many Requests) response is received, Wikimedia includes a
`Retry-After` header specifying the number of seconds to wait. **Do not retry
immediately with a fixed backoff** (e.g., always waiting exactly 5 seconds)
— this is counterproductive and may lead to a temporary ban. Always use the
server-supplied value:

```python
if resp.status_code == 429:
    retry_after = int(resp.headers.get('Retry-After', 10))
    time.sleep(retry_after)
    continue  # retry the request in a loop
```

### Caching Strategy (Prevents Redundant Calls)

When fetching the same set of pages across multiple runs (e.g., re-running
a tool with different parameters), a simple disk cache avoids redundant
HTTP requests and reduces rate-limit pressure.

**Recommended pattern — JSON key-value cache:**

```python
import json, os

CACHE_FILE = ".api_cache.json"
_cache = {}

if os.path.exists(CACHE_FILE):
    with open(CACHE_FILE) as f:
        _cache = json.load(f)

def fetch_with_cache(lang, title):
    key = f"{lang}:{title}"
    if key in _cache:
        return _cache[key]
    url = f"https://{lang}.wikipedia.org/api/rest_v1/page/summary/{title}"
    resp = requests.get(url, headers=HEADERS, timeout=15)
    if resp.status_code == 200:
        data = resp.json()
        result = data.get("extract", "")
        _cache[key] = result
        with open(CACHE_FILE, "w") as f:
            json.dump(_cache, f)
        return result
    return None
```

**Key points:**
- Use `{lang}:{title}` as the cache key to avoid collisions
- Write back to disk after each fetch (appending is fine for ~300 entries)
- Keep cache files gitignored (they're session artifacts)
- For large caches (>1000 entries), consider SQLite instead of JSON

### Structured Error Returns from Fetch Functions

When a fetch fails, return a **dict with `lead` and `error` fields** rather
than a bare `None`. This lets callers distinguish between different failure
modes without making additional API calls:

```python
def fetch_lead(lang, title) -> dict:
    """Returns {"lead": str|None, "error": str|None}."""
    url = f"https://{lang}.wikipedia.org/api/rest_v1/page/summary/{title}"
    resp = requests.get(url, timeout=15)
    if resp.status_code == 200:
        data = resp.json()
        extract = data.get("extract", "")
        if extract and extract.strip():
            return {"lead": extract, "error": None}
        # HTTP 200 but empty extract — investigate why
        page_type = data.get("type", "")
        if page_type == "disambiguation":
            return {"lead": None, "error": "Disambiguation page (no lead text)"}
        if data.get("description"):
            return {"lead": None, "error": f"Has description but no extract; type={page_type}"}
        return {"lead": None, "error": f"No content; type={page_type}"}
    return {"lead": None, "error": f"HTTP {resp.status_code}"}
```

This pattern lets consumers show actionable error messages:
```
❌  scn    Has description but no extract; type=standard
❌  mhr    No content; type=standard
❌  nb     HTTP 500
```

### Common Causes of 429 Responses

- **Too many requests in a short window** even with a proper User-Agent.
  Stay under ~2 requests/second for batch operations, and add deliberate
  delays (0.3–0.5s) between requests.
- **Too many titles in a single `titles` parameter.** Keep batches under 50
  titles per call for `prop=pageprops` and 50 IDs per call for `wbgetentities`.
- **Fetching very large pages via `action=parse` without a section limit.**
  Add `rvsection=0` or `section=0` if you only need the lead section, or
  use `exintro` with `prop=extracts` for a concise summary.

### ⚠️ Login Username Quirk: Spaces → Underscores

The MediaWiki `action=login` API requires the **internal database form** of
usernames, where spaces are stored as underscores (`_`). The `lgname` parameter
will fail silently (returning `"Unknown error"`) if you pass a username with
literal spaces, even though that's how the username displays on wiki pages and in
the `Special:BotPasswords` confirmation message.

**Always normalize** usernames before passing them to `lgname`:

```python
# ✗ This will FAIL with "Unknown error"
lgname = "AL Wiki MIT"

# ✓ This works
lgname = "AL_Wiki_MIT"
```

If you're using bot passwords, the same rule applies:

```python
# ✗ Fails
lgname = "AL Wiki MIT@mybot"

# ✓ Works
lgname = "AL_Wiki_MIT@mybot"
```

**Note:** `action=login` also requires **POST**, not GET. See the
[MediaWiki API documentation](https://www.mediawiki.org/wiki/API:Login) for
the full login flow (including the two-step `NeedToken` case).

> 🛡️ **Critical: always verify login succeeded.** A login failure (wrong password,
> expired token, network issue) returns `NeedToken` rather than raising an error.
> If you proceed without checking, you'll get an anonymous CSRF token (`+\`) and
> your edits will be attributed to a temporary account (`~2026-XXXXX-XX`) instead
> of your bot. Always:
> 1. Assert `login_resp["login"]["result"] == "Success"`
> 2. Verify `userinfo` matches the expected username
> 3. Use `assert="user"` on every write request
>
> See the **[wikimedia-auth-oauth](../wikimedia-auth-oauth/SKILL.md)** skill for
> the full canonical login function with all guardrails built in, and the
> **[pywikibot](../pywikibot/SKILL.md)** skill's Troubleshooting section for
> bot password authentication.

## **Example Use Cases**

- **Fetch article content:** "Get the wikitext for 'Python (programming language)' using the Action API."
- **Search:** "Search English Wikipedia for articles about 'machine learning' and return the top 10 titles."
- **Get page metadata:** "Using the REST API, get the description and thumbnail for 'Albert Einstein'."
- **SPARQL query:** "Query Wikidata for all museums in Paris with their coordinates."
- **Batch lookup:** "For each page in this list of 50 titles, fetch the page ID and word count from the API."
- **Troubleshoot 403:** "Check the User-Agent header and confirm it follows the Wikimedia format."

### REST API `/page/summary` Response Details

The `/{lang}.wikipedia.org/api/rest_v1/page/summary/{title}` endpoint returns a JSON response with several useful fields beyond `extract` and `thumbnail`:

```json
{
  "type": "standard",                // ← page type: "standard" or "disambiguation"
  "title": "Albert Einstein",
  "displaytitle": "Albert Einstein",
  "extract": "Albert Einstein was a German-born...",  // ← lead paragraph (may be empty for DAB pages)
  "description": "German-born theoretical physicist",
  "pageid": 736,
  "content_urls": {
    "desktop": {"page": "https://en.wikipedia.org/wiki/Albert_Einstein"},
    "mobile": {"page": "https://en.m.wikipedia.org/wiki/Albert_Einstein"}
  }
}
```

**Key notes:**

- **Mobile domains deprecated (Oct 2025):** Wikimedia unified mobile and desktop domains. `en.m.wikipedia.org` no longer serves content — it redirects to `en.wikipedia.org`. Do not construct `https://{lang}.m.wikipedia.org/` URLs in new code. The `content_urls.mobile.page` field returned by the REST API still exists but redirects to the desktop domain. See [Unifying mobile and desktop domains](https://diff.wikimedia.org/2025/11/21/unifying-mobile-and-desktop-domains/).

  **To request mobile-formatted content, use one of:**
  - **URL parameter:** `https://en.wikipedia.org/wiki/Chess?useformat=mobile` (recommended — works on all page views)
  - **index.php:** `https://en.wikipedia.org/w/index.php?title=Chess&mobileformat=1` (legacy, being phased out)
  - **Action API parse:** `action=parse&page=Chess&mobileformat=1` or the newer `action=parse&page=Chess&prop=text&useformat=mobile`
  - **REST API:** `/page/mobile-html/{title}` returns mobile-optimized HTML (stable, not deprecated)
- **`type` field:** Use `type == "disambiguation"` to detect disambiguation pages without parsing templates or categories (more reliable than checking for `{{disambiguation}}` templates).
- **`extract` may be empty** even when the page exists. Disambiguation pages and pages consisting only of an infobox return an empty string for `extract` but still return HTTP 200.
- **Unicode control characters:** Some language editions include bare Unicode formatting characters (e.g., U+200E LEFT-TO-RIGHT MARK) in the `extract` text. These are not removed by `.strip()` — filter them with `''.join(c for c in text if c.isprintable())` before processing.
- **The `type` field is only available in the REST API**, not in the Action API's `prop=extracts`. If you use the Action API, you'd need to check categories or templates instead.

---

## **Tooling**

This skill includes helper scripts, reference docs, and templates:

### 🔧 Connectivity Test (`scripts/test-api.sh`)

Tests 14 endpoints across 6 API families with your User-Agent and reports which ones work.

```bash
# Test with the default User-Agent
./scripts/test-api.sh

# Test with YOUR User-Agent
./scripts/test-api.sh "MyBot/1.0 (https://example.com; me@example.com) MyProject"
```

Tests 14 endpoints across 6 API families:
- 📡 Core APIs (Action API: siteinfo, search, page content) — 3 tests
- 🌐 REST APIs (page summary, mobile-html) — 3 tests
- 🔗 SPARQL / Wikidata — 2 tests
- 📊 Pageviews API — 1 test
- 🖼️  Commons Analytics API — 3 tests (category metrics, top wikis, top edited)
- 🧠 Lift Wing ML API — 2 tests (POST-based, revert risk + article quality)

**Note on Commons Analytics:** Data is only available for categories on the
[allow list](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/main/dags/commons/commons_category_allow_list.tsv)
and their subcategories. The `category-metrics-snapshot` and `top-wikis-per-category`
tests use `Smithsonian_American_Art_Museum` (an allow-listed category) and should
return data. See `references/endpoints.md` for details on the allow list.

### 📚 API Endpoint Reference (`references/endpoints.md`)

Full catalog of Wikimedia endpoints organized by API family:
- Action API parameters and pagination pattern
- REST API and RESTBase endpoints
- Pageviews API parameters and date formats
- SPARQL query patterns
- Quick selection guide (task → best endpoint)

Read it when you need to find the right endpoint:

```bash
# Load by asking the agent, or reference directly
cat references/endpoints.md
```

### 🐍 Python Client Template (`assets/user-agent-template.py`)

A ready-to-use Python client with:
- Proper User-Agent configuration
- `requests.Session` with connection reuse
- Rate limiting (configurable delay between requests)
- Automatic retry with exponential backoff on 429/timeouts
- Helpful error messages for 403 (bad UA) and 404
- Convenience methods for all major API types
- Works as a standalone demo script

```bash
# Copy and customize
cp assets/user-agent-template.py my_bot.py
# Edit my_bot.py with your User-Agent, then run
python3 my_bot.py
```

Supports:
```python
from user_agent_template import WikimediaClient

with WikimediaClient("MyBot/1.0 (user@example.com) MyProject") as client:
    # Action API
    results = client.search("Albert Einstein", limit=5)
    extract = client.get_page_extract("Python (programming language)")
    
    # REST API
    summary = client.page_summary("Albert Einstein")
    
    # SPARQL
    entities = client.sparql_query("SELECT ?item WHERE { ... }")
    entity = client.get_entity("Q937")
    
    # Pageviews
    top = client.top_pageviews("en.wikipedia")
```

---

## CORS: Browser Applications and Cross-Origin Requests

When building **browser-based applications** (JavaScript, not server-side Python), you may encounter CORS (Cross-Origin Resource Sharing) restrictions. This section explains when this happens and how to handle it.

### When CORS Becomes an Issue

Most Wikimedia APIs (Action API, REST API, SPARQL) **do send CORS headers** and work fine from browser JavaScript. However, **media files served from `upload.wikimedia.org` do not** — and this is where you'll encounter CORS errors.

| Resource | CORS Headers? | Browser Access |
|----------|:-------------:|----------------|
| Action API (`/w/api.php`) | Yes | Works from any origin |
| REST API (`/api/rest_v1/`) | Yes | Works from any origin |
| SPARQL (`query.wikidata.org/sparql`) | Yes | Works from any origin |
| **Media files** (`upload.wikimedia.org`) | **No** | **Blocked for Canvas/WebGL/`background-image`** |
| File page redirects (`Special:FilePath/`) | Yes | Works for `<img>` display |

### The Problem: `upload.wikimedia.org` Has No CORS Headers

```
Access to image at 'https://upload.wikimedia.org/wikipedia/commons/thumb/...'
from origin 'http://localhost:3000' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
```

Chrome DevTools shows: `net::ERR_BLOCKED_BY_ORB` (Opaque Response Blocking).

This affects:
- **Canvas pixel access** (`canvas.getContext('2d').getImageData()`)
- **WebGL textures** (`texImage2D`)
- **CSS `background-image`** in some browsers
- **Audio/video elements** for certain operations

### Solution: Proxy Through Your Server

For browser apps that need full access to media content, **proxy the request through your own server**:

```
Browser -> Your Server -> Commons API -> Your Server -> Browser
         (same-origin)   (no CORS)     (same-origin)
```

The browser sees the response as **same-origin**, so CORS restrictions don't apply.

**Server-side pattern (Node.js):**

```javascript
// API endpoint: /api/media?url=...
app.get('/api/media', async (req, res) => {
  const mediaUrl = req.query.url;
  const resp = await fetch(mediaUrl);
  const buffer = await resp.arrayBuffer();
  
  // Cache with SHA-256 hash filename
  const hash = createHash('sha256').update(buffer).digest('hex');
  const cachedPath = `cache/${hash}.jpg`;
  await writeFile(cachedPath, buffer);
  
  res.json({ url: `/cache/${hash}.jpg` });
});
```

**Client-side usage:**

```javascript
// Direct Commons URL - CORS blocked
const directUrl = 'https://upload.wikimedia.org/wikipedia/commons/thumb/...';

// Proxied URL - same-origin, no CORS issues
const proxiedUrl = '/cache/abc123.jpg';
document.getElementById('img').src = proxiedUrl;  // Works!
```

### When You Don't Need a Proxy

- **Server-side code** (Python, Node.js) — no CORS restrictions, fetch directly
- **Simple `<img>` display** — use `Special:FilePath/` redirect URLs
- **Metadata-only APIs** — Action API, REST API, SPARQL all work from browsers

### Related Documentation

For detailed thumbnail/media CORS handling, see **[wikimedia-commons-thumbnails](../wikimedia-commons-thumbnails/SKILL.md)** Section 10.

---

## Cross-References

| Related Skill | Why |
|--------------|-----|
| **[wikimedia-api-strategy](../wikimedia-api-strategy/SKILL.md)** | Decision framework for choosing the right API |
| **[wikimedia-auth-oauth](../wikimedia-auth-oauth/SKILL.md)** | OAuth 1.0a/2.0 and bot passwords for authenticated requests |
| **[wikipedia-error-handling](../wikipedia-error-handling/SKILL.md)** | Retry strategies and backoff patterns for 403/429/5xx responses |
| **[wikimedia-commons-thumbnails](../wikimedia-commons-thumbnails/SKILL.md)** | CORS proxy patterns for serving Commons media in browser apps |
| **[pywikibot](../pywikibot/SKILL.md)** | Python bot framework with built-in User-Agent and rate limiting |

---