web-manual-tier-quick-refs · diff

v1.0.0 to v1.1.0

19 added, 178 removed. Audit A to A.

---

name: web-manual-tier-quick-refs

description: >

- Nested web-manual reference for tier quick-reference commands: Tier 0 PDF,

- Tier 1 APIs, Tier 1.5 Trafilatura, Tier 2 BeautifulSoup, Tier 3 Playwright

- stealth, Tier 4 Jina/Firecrawl, and Tier 5 AI-native search.

- version: 1.0.0

- last_changed_at: "2026-07-26T20:55:00-07:00"

+ Nested web-manual reference: compact tier index (Tier 0 PDF, Tier 1 APIs,

+ Tier 1.5 Trafilatura, Tier 2 BeautifulSoup, Tier 3 Playwright stealth, Tier 4

+ Jina/Firecrawl, Tier 5 AI-native search) routing to each tier's deep-dive

+ file, which is the single source of truth for its commands.

+ version: 1.1.0

+ last_changed_at: "2026-09-04T00:00:00Z"

related_files:

- src/lingtai/tools/web_search/manual/SKILL.md

maintenance: "If you find stale or incorrect information here, use the lingtai-issue-report skill to assemble evidence and obtain per-issue human consent before filing an issue. Never include secrets, credentials, tokens, or private paths."

---

# Web Browsing Tier Quick References

- Nested web-manual reference. Open this after the top-level router when you need

- manual commands for a specific extraction tier instead of the auto-tier script.

- ## Tier 0 — PDF Direct Download (Quick Reference)

- ```bash

- # Direct PDF link

- curl -L "https://arxiv.org/pdf/1706.03762.pdf" -o paper.pdf

- # arXiv ID → derive the PDF path

- curl -L "https://arxiv.org/pdf/2401.12345.pdf" -o paper.pdf

- ```

- ```python

- import fitz # pip install pymupdf

- doc = fitz.open("paper.pdf")

- text = doc[0].get_text() # first page

- ```

- → Full details: [reference/tier-0-pdf.md](../tier-0-pdf.md)

- ---

- ## Tier 1 — API Metadata Queries (Quick Reference)

- ### Academic APIs at a Glance

- | API | Best for | Free? |

- |-----|----------|-------|

- | **OpenAlex** | Any DOI → metadata + citations | ✅ |

- | **CrossRef** | DOI → title, authors, journal | ✅ |

- | **Semantic Scholar** | AI/ML papers, citation graphs | ✅* |

- | **arXiv** | CS/Physics/Math papers | ✅ |

- | **Unpaywall** | Find free PDF for any DOI | ✅ |

- | **PubMed E-utilities** | Biomedical literature | ✅ |

- | **CORE** | Open access full text (30M+) | ✅† |

- | **Europe PMC** | Biomedical + PMC full text | ✅ |

- | **DBLP** | CS conference papers | ✅ |

- | **Papers With Code** | ML papers with code + benchmarks | ✅ |

- | **DOAJ** | Open access journal articles | ✅ |

- | **Zenodo** | Research data, datasets | ✅ |

- | **NASA ADS** | Astrophysics/astronomy | ✅ |

- ```python

- import requests

- # OpenAlex — most powerful, completely free

- r = requests.get("https://api.openalex.org/works/https://doi.org/10.1038/s41586-023-05995-9")

- data = r.json()

- # Unpaywall — find free PDF

- r = requests.get(f"https://api.unpaywall.org/v2/{doi}?email=lingtai@users.noreply.github.com")

- if r.json().get("best_oa_location"):

- pdf_url = r.json()["best_oa_location"].get("url_for_pdf")

- ```

- → Full details: [reference/tier-1-apis.md](../tier-1-apis.md)

- → Academic search pipeline: [reference/academic-pipeline.md](../academic-pipeline.md)

- ---

- ## Tier 1.5 — Trafilatura (Quick Reference)

- ```python

- import trafilatura

- # Fetch + extract in one call

- downloaded = trafilatura.fetch_url("https://example.com/article")

- text = trafilatura.extract(downloaded)

- # With metadata

- metadata = trafilatura.extract(downloaded, output_format="json", include_metadata=True)

- # Batch extraction

- results = [trafilatura.extract(trafilatura.fetch_url(url)) for url in urls]

- ```

- → Full details: [reference/tier-1-5-trafilatura.md](../tier-1-5-trafilatura.md)

- ---

- ## Tier 2 — BeautifulSoup (Quick Reference)

- ```python

- import requests

- from bs4 import BeautifulSoup

- resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0..."})

- soup = BeautifulSoup(resp.text, "html.parser")

- # Common patterns

- title = soup.find("h1").get_text(strip=True)

- links = [a["href"] for a in soup.select("a[href]") if a["href"].startswith("http")]

- article = "\n".join(p.get_text() for p in soup.select("article p"))

- meta_desc = soup.find("meta", attrs={"name": "description"})["content"]

- ```

- → Full details: [reference/tier-2-beautifulsoup.md](../tier-2-beautifulsoup.md)

- ---

- ## Tier 3 — Playwright Stealth (Quick Reference)

- ```python

- from playwright.sync_api import sync_playwright

- try:

- from playwright_stealth import Stealth

- _apply_stealth = lambda page: Stealth().use_sync(page)

- except ImportError:

- from playwright_stealth import stealth_sync

- _apply_stealth = lambda page: stealth_sync(page)

- with sync_playwright() as p:

- browser = p.chromium.launch(headless=True)

- ctx = browser.new_context(

- user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",

- viewport={"width": 1920, "height": 1080},

- )

- page = ctx.new_page()

- _apply_stealth(page) # ← 关键：绕过反爬虫检测

- # Block heavy resources

- page.route("**/*.{png,jpg,jpeg,gif,svg,woff,woff2}", lambda r: r.abort())

- page.goto(url, wait_until="domcontentloaded") # NOT networkidle!

- page.wait_for_timeout(2000)

- content = page.content()

- browser.close()

- ```

- → Full details: [reference/tier-3-playwright.md](../tier-3-playwright.md)

- → Anti-detection deep-dive: [reference/stealth.md](../stealth.md)

- ---

- ## Tier 4 — Jina Reader / Firecrawl (Quick Reference)

- ```python

- import requests

- # Jina Reader — FREE, no key needed

- resp = requests.get(f"https://r.jina.ai/{url}",

- headers={"Accept": "text/markdown"})

- markdown = resp.text

- # Firecrawl — production-grade (needs API key)

- resp = requests.post("https://api.firecrawl.dev/v1/scrape",

- json={"url": url},

- headers={"Authorization": "Bearer YOUR_KEY"})

- ```

- → Full details: [reference/tier-4-jina-firecrawl.md](../tier-4-jina-firecrawl.md)

- ---

- ## Tier 5 — AI-Native Search (Quick Reference)

- ```python

- # DuckDuckGo — free, no key

- from ddgs import DDGS

- with DDGS() as ddgs:

- results = list(ddgs.text("attention is all you need", max_results=5))

- # Tavily — search + extract + answer (needs key)

- import requests

- r = requests.post("https://api.tavily.com/search",

- json={"api_key": "tvly-xxx", "query": "transformer paper", "max_results": 5})

- # Exa — neural/semantic search (needs key)

- r = requests.post("https://api.exa.ai/search",

- json={"query": "transformer architecture", "numResults": 5},

- headers={"x-api-key": "xxx"})

- ```

+ Nested web-manual reference. Open this after the top-level router to find

+ which tier's deep-dive file to load next — the runnable commands live in

+ those files (single source of truth; a copy here would drift from them, see

+ `maintenance-bundles/SKILL.md` Rule 3), not in this index.

- → Full details: [reference/tier-5-ai-search.md](../tier-5-ai-search.md)

- → Search strategy guide: [reference/search-strategies.md](../search-strategies.md)

+ | Tier | What it's for | Load |

+ |---|---|---|

+ | 0 | PDF direct download + `fitz` text extraction | [reference/tier-0-pdf.md](../tier-0-pdf.md) |

+ | 1 | Academic/metadata API queries (OpenAlex, CrossRef, Semantic Scholar, arXiv, Unpaywall, PubMed, CORE, Europe PMC, DBLP, Papers With Code, DOAJ, Zenodo, NASA ADS) | [reference/tier-1-apis.md](../tier-1-apis.md); full pipeline: [reference/academic-pipeline.md](../academic-pipeline.md) |

+ | 1.5 | Trafilatura fetch+extract, metadata, batch mode | [reference/tier-1-5-trafilatura.md](../tier-1-5-trafilatura.md) |

+ | 2 | BeautifulSoup CSS-selector extraction | [reference/tier-2-beautifulsoup.md](../tier-2-beautifulsoup.md) |

+ | 3 | Playwright stealth (JS-rendered/protected pages) | [reference/tier-3-playwright.md](../tier-3-playwright.md); anti-detection deep-dive: [reference/stealth.md](../stealth.md) |

+ | 4 | Jina Reader / Firecrawl — verify current vendor access and quota policy | [reference/tier-4-jina-firecrawl.md](../tier-4-jina-firecrawl.md) |

+ | 5 | AI-native search (DuckDuckGo, Tavily, Exa) | [reference/tier-5-ai-search.md](../tier-5-ai-search.md); strategy guide: [reference/search-strategies.md](../search-strategies.md) |

---