wikimedia-wikitext git:20261009.dd43296A

Immutable. This exact content never changes and is served at /api/v1/blob/8e02240b3cc781af.

---
name: wikimedia-wikitext
description: Parse, extract, and manipulate Wikipedia and MediaWiki wikitext (wiki markup, templates, infoboxes, citations, links) using proper AST-based tooling instead of fragile regex patterns
license: MIT
compatibility: opencode
depends_on: [wikimedia-api-access]
skill_discovery_hints:
  - keywords: ["wikitext", "wiki markup", "parse", "mwparserfromhell", "wikitext parsing", "template parsing"]
  - keywords: ["wikitext AST", "syntax tree", "section parsing", "link extraction", "template expansion"]
last_verified: 2026-10-03
---

> ⚠️ **User-Agent required:** The API examples below use the Action API and REST API. All requests must include a descriptive `User-Agent` header or they will be blocked. See the **[wikimedia-api-access](../wikimedia-api-access/SKILL.md)** skill for the correct format.

## SOP: The Two Strategies

There are **two fundamentally different approaches** to reading and extracting data from MediaWiki wikitext. Choose based on whether you need to **write back** wikitext changes.

| Strategy | When to use | Tool |
|----------|-------------|------|
| **AST Parsing** | You need to modify wikitext or extract structured data from raw markup | `mwparserfromhell` |
| **Parsoid HTML** | You only need to *read* page content as structured text/data | MediaWiki REST API (`/html` endpoint) + BeautifulSoup/lxml |

**Never** mix the two — do not parse Parsoid HTML to reconstruct wikitext, and do not use regex on raw wikitext to extract data.

---

## SOP: AST Parsing with `mwparserfromhell`

Wikitext is non-regular, context-dependent, and recursive (templates inside templates inside image captions). **Never use regex** to parse wikitext structures like templates, tables, or links. Always use `mwparserfromhell`, an LL(1)-based parser that builds a proper Abstract Syntax Tree (AST).

### Installation

```bash
pip install mwparserfromhell
```

### Safe Template Extraction

```python
import mwparserfromhell

wikitext = "[[File:Example.jpg|thumb|{{Location|40|-74}}]] Text with {{Template|param=value|nested={{Value}}}}."
code = mwparserfromhell.parse(wikitext)

for template in code.filter_templates():
    if template.name.matches("Template"):
        param = template.get("param").value
        print(f"Found param: {param}")
```

### Mutating Wikitext

```python
import mwparserfromhell

code = mwparserfromhell.parse(wikitext)
for template in code.filter_templates():
    if template.name.matches("Infobox"):
        template.add("image", "new_image.jpg")

updated_wikitext = str(code)
```

### Stripping Markup to Plain Text

```python
code = mwparserfromhell.parse(wikitext)
plain_text = code.strip_code()  # Removes templates, comments, wikilinks, etc.
```

**⚠️ `strip_code()` removes:**
- HTML comments (`<!-- ... -->`)
- Templates (replaced with their rendered output if available, else stripped)
- Wiki markup formatting

**`strip_code()` preserves:**
- Plain text content
- External URLs

**⚠️ Template *payloads* are the trap (verified 2026-10-09).** Text wrapped for language or formatting —
`{{en|1=Text}}`, `{{Langswitch|en=…}}`, `{{nowrap|…}}` — is not "plain content" to a stripper: a pass that drops
everything between `{{` and `}}` blanks the text entirely and leaves an empty string, which then reads like a page
with nothing to say rather than a parser failure. Resolve the payload of language/formatting templates
(`{{en|1=X}}` → `X`), and fall back to the labels of any wikilinks when a payload cannot be read.

Measured on a 30,032-note corpus: fixing this moved an agreement rate from 4.1% → 4.2% and resolved links from
2,879 → 2,990. Small corrections to an aggregate — but the same bug emptied *individual* notes, and an empty note
counts as "no evidence" in every downstream number, so the error is silent by construction.

Use `str(code)` when you need the raw wikitext with all markup intact.

### Filtering Specific Node Types

```python
# All templates
code.filter_templates()

# All wikilinks ([[...]])
code.filter_wikilinks()

# All external links ([...])
code.filter_external_links()

# All tags (<ref>, <nowiki>, etc.)
code.filter_tags()

# All headings
code.filter_headings()

# Filter by name pattern
code.filter_templates(matches=lambda name: name.startswith("Infobox"))
```

### Parameter Access Patterns

```python
template = code.filter_templates(matches="Infobox person")[0]

# Safe access with default
name = template.get("name").value
# raises ValueError if param doesn't exist

# Check existence first
if template.has("birth_date"):
    bd = template.get("birth_date").value

# Get all parameters
for param in template.params:
    print(f"{param.name}: {param.value}")

# Add or update a parameter
template.add("new_param", "value")
template.add("existing_param", "new_value")  # Replaces if exists
```

---

## SOP: Parsoid HTML Strategy (Read-Only)

If you do not need to write back wikitext changes, **do not parse wikitext at all**. Use the MediaWiki Core REST API's `/html` endpoint, which returns clean, semantic HTML5/RDFa generated by Parsoid (natively embedded in MediaWiki — the standalone JS version is obsolete).

### Fetching Page HTML

```python
import requests

url = "https://en.wikipedia.org/w/rest.php/v1/page/Python_(programming_language)/html"
headers = {"User-Agent": "MyTool/1.0 (user@example.com)"}

response = requests.get(url, headers=headers, timeout=30)
if response.status_code == 200:
    html = response.text
    # Parse with BeautifulSoup or lxml
```

### Extracting Tables as DataFrames

Wikitext tables (`{| ... |}`) contain deep edge cases (implicit row spans, embedded CSS). **Never** parse them manually.

```python
import pandas as pd
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
tables = pd.read_html(str(soup))  # pandas handles HTML tables natively

for i, df in enumerate(tables):
    print(f"Table {i}: {df.shape[0]} rows × {df.shape[1]} columns")
```

### Extracting Section Content

```python
soup = BeautifulSoup(html, "html.parser")
for section in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6"]):
    heading = section.get_text(strip=True)
    # Next siblings until the next heading are this section's content
```

---

## SOP: Anti-Patterns to Avoid

| ❌ Anti-Pattern | Why It Breaks | ✅ Correct Approach |
|---|---|---|
| `r'\{\{Infobox(.*?)\}\}'` for template params | Breaks on nested templates, multi-line params, pipes in wikilinks | `mwparserfromhell.filter_templates()` |
| Splitting on `|` to get table cells | Pipes inside wikilinks `[[a|b]]`, templates, or image captions will fragment | Parsoid HTML + `pandas.read_html()` |
| `r'\[\[(.*?)\]\]'` for wikilinks | Misses piped links, interwiki links, category links, file links | `mwparserfromhell.filter_wikilinks()` |
| `r'<ref>(.*?)</ref>'` for references | Misses self-closing `<ref name="x" />`, multi-line refs, nested refs | `mwparserfromhell.filter_tags()` |
| Manually stripping HTML from Parsoid output | Fragile — Parsoid's HTML structure may change across MediaWiki versions | BeautifulSoup/lxml with semantic selectors |
| Dropping every `{{…}}` to "remove templates" | Blanks `{{en|1=Text}}`, `{{Langswitch|en=…}}`, `{{nowrap|…}}` — the text *is* the payload | Resolve language/formatting template payloads; strip only structural templates |

---

## SOP: Troubleshooting Edge Cases

### Complex Tables
Fetch via `/html` endpoint and pass to `pandas.read_html()`. Do not manually chunk `{|` and `|}` tokens.

### Invisible/Stripped Content
`mwparserfromhell.strip_code()` removes HTML comments and template markup. Use `code.get_sections()` if you need to preserve section structure, or `str(code)` for raw wikitext.

### Nested Templates
`mwparserfromhell` handles arbitrary nesting depth — let the parser do the work:

```python
code = mwparserfromhell.parse("{{Outer|{{Inner|param={{Deepest|value}}}}}}")
for t in code.filter_templates():
    print(t.name)  # Correctly identifies all three templates
```

### Unicode and Special Characters
Wikitext may contain non-ASCII characters, HTML entities (`&amp;`, `&lt;`), and magic words. `mwparserfromhell` handles these correctly; regex approaches will not.

### ⚠️ Gallery images are NOT wikilinks (verified 2026-08-18)
Inside `{{gallery|File:X.jpg|caption|File:Y.jpg|caption}}` templates, file names
are **plain-text template parameters**, not `[[File:...]]` wikilinks.
`mwparserfromhell` correctly reports them as `Text` nodes — so
`code.filter_wikilinks()` will NOT find gallery images. If you need the set of
images *actually rendered* in an article, use the Parsoid REST endpoint
(`GET /w/rest.php/v1/page/<title>/html`) and select semantic elements:
`<figure>` + `<figcaption>` for standard thumbnails, `<li class="gallerybox">` +
`<div class="gallerytext">` for `<gallery>` items. This also excludes navbox
icons and other template chrome that `prop=images` would include. Chapters can
be assigned in the same walk: each figure/gallerybox belongs to the nearest
preceding `<h2>` (Parsoid emits plain `<h2>` elements in document order).

### ⚠️ Parsoid hrefs are relative (verified 2026-08-18)
The Parsoid REST `/html` endpoint emits **relative** file links:
`<a href="./File:Name.jpg">` (not `/wiki/File:Name.jpg`), and the `<img>`
`resource` attribute also uses `./File:`. Handle all three forms:
`re.search(r"(?:\./|/wiki/)?(File|Image):(.*)$", href)`.

---

## Tooling

This skill includes helper scripts, reference docs, and templates:

### 🔧 Wikitext Inspector (`scripts/test-mwparser.sh`)

Inspect any wikitext file and get a structured overview of what's in it — templates,
wikilinks, tags, headings, sections, and potential issues. Also doubles as an
installation smoke test when called with no arguments.

```bash
# Smoke test (no args) — verify installation
./scripts/test-mwparser.sh

# Inspect a wikitext file
./scripts/test-mwparser.sh page.wikitext

# Pipe mode — inspect wikitext from stdin
curl -s "https://en.wikipedia.org/w/index.php?title=Python_(programming_language)&action=raw" \
  | ./test-mwparser.sh -

# Usage: ./test-mwparser.sh [file.wikitext | -]
#   (no args)  — run installation smoke test
#   file.wikitext — parse file and report structure
#   -          — read wikitext from stdin (pipe mode)
```

Reports: total AST nodes, templates with parameter previews, wikilinks with display
text, tags, headings (section outline), plain text length, and potential issues
(e.g., empty file, very large pages).

### 🔧 Parsoid HTML Fetcher (`scripts/fetch-parsoid-html.sh`)

Fetch a Wikipedia page as clean Parsoid HTML and save to a file.

```bash
# Fetch page HTML
./scripts/fetch-parsoid-html.sh "Python (programming language)"

# Fetch into a specific output file
./scripts/fetch-parsoid-html.sh "Albert Einstein" albert.html
```

### 📚 `mwparserfromhell` Reference (`references/mwparserfromhell-guide.md`)

Complete reference covering:
- All node types and filter methods
- Template parameter manipulation
- Section and heading handling
- Tag and comment handling
- Common patterns for infoboxes, citations, lists
- Performance considerations for large pages

### 📚 Parsoid HTML API Reference (`references/parsoid-html-api.md`)

Complete guide to the MediaWiki REST API `/html` endpoint:
- Endpoint structure and parameters
- Content negotiation (JSON vs HTML)
- If-None-Match / ETag caching
- Section-only retrieval
- Handling revisions vs latest
- Error response guide

### 📚 Wikitext Pitfalls (`references/wikitext-pitfalls.md`)

Deep reference of documented edge cases:
- Table syntax gotchas (implicit row/colspan, multi-line cells)
- Template argument separation (pipes in different contexts)
- Comment placement affecting template parsing
- Tag extension boundaries (<nowiki>, <pre>, <syntaxhighlight>)
- Unicode normalization issues
- Parser function intricacies
- Behavior of strip_code() with various markup types

### 🐍 Wikitext Parser Template (`assets/parse-wikitext.py`)

Ready-to-use Python script with:
- Template extraction and parameter manipulation
- Wikilink and external link extraction
- Section splitting and plain-text conversion
- Infobox data extraction
- Citation list extraction
- Multi-file batch processing

```bash
# Extract all templates from a wikitext file
python3 assets/parse-wikitext.py page.wikitext --templates

# Extract all internal links
python3 assets/parse-wikitext.py page.wikitext --links

# Convert to plain text
python3 assets/parse-wikitext.py page.wikitext --plaintext
```

### 🐍 Parsoid Extractor Template (`assets/parsoid-extractor.py`)

Ready-to-use Python script for the HTML DOM strategy:

```bash
# Fetch a page and extract tables as CSV
python3 assets/parsoid-extractor.py "Python (programming language)" --tables

# Extract section text
python3 assets/parsoid-extractor.py "Albert Einstein" --sections

# Extract infobox data from rendered HTML
python3 assets/parsoid-extractor.py "Berlin" --infobox
```

### 🧩 Table to DataFrame Template (`assets/table-extractor.py`)

Example script for converting wikitext tables to pandas DataFrames via the Parsoid HTML pipeline, with CSV/JSON export.

---

## Cross-References

| Related Skill | Why |
|--------------|-----|
| **[wikimedia-api-access](../wikimedia-api-access/SKILL.md)** | User-Agent and API patterns — the foundation all API calls build on |
| **[wikimedia-api-strategy](../wikimedia-api-strategy/SKILL.md)** | Choosing how to *read* a page: REST `/page/summary` vs TextExtracts vs Parsoid HTML — decision matrix + fidelity-test pattern in `references/lead-text-extraction.md` |
| **[wikipedia-templates](../wikipedia-templates/SKILL.md)** | Template syntax — `mwparserfromhell` is the primary parsing tool |
| **[wikipedia-wikitables](../wikipedia-wikitables/SKILL.md)** | Table parsing — always use Parsoid HTML + `pandas.read_html()`, never regex |
| **[wikipedia-citations](../wikipedia-citations/SKILL.md)** | Citation/reference extraction from wikitext |
| **[wikipedia-page-anatomy](../wikipedia-page-anatomy/SKILL.md)** | Page structure — sections, infoboxes, navboxes parsed from wikitext |
| **[pywikibot](../pywikibot/SKILL.md)** | Bot framework — uses `mwparserfromhell` for template manipulation |
| **[wiktionary](../wiktionary/SKILL.md)** | Dictionary entry structure and AST-based parsing of translation tables |
| **[wikisource](../wikisource/SKILL.md)** | Proofread page text extraction, `Page:`/`Index:` namespaces, transclusion |
| **[wikivoyage](../wikivoyage/SKILL.md)** | Wikivoyage listing templates (`{{see}}`, `{{eat}}`, etc.) are structured wikitext templates parsed with `mwparserfromhell` |