reading-papers git:20260727.37a3f77A

Immutable. This exact content is served forever at /api/v1/blob/a5a875aae4fdd49c.

---
name: reading-papers
description: Look up, resolve, and read academic papers when the user sends a link, DOI, title, author name, or vague description, across marketing (Marketing Science, JMR, JM, JCR), economics (QJE, AER, ECMA, JPE, ReStud, AEJs, AER:Insights), psychology (Psych Science), and CS (ICLR/ICML/NeurIPS, JMLR, ACL). Preserves LaTeX equations; integrates the user's Zotero library and an optional ~/ocr-examples HPC OCR pipeline. TRIGGER whenever the user asks to read, summarize, check, find, or cite a paper, asks "what does X argue", pastes a paper URL, or asks what an author has written. For a whole literature on a topic, use litreview instead.
---

# Reading papers

Get from a fuzzy reference to clean full text with equations intact, cheaply, without wasting
context on dead ends. The intelligence is yours; the script is a tool. Don't blindly trust its
first answer on hard cases (common author names, paywalled venues). Verify, as flagged below.

## The command

```bash
~/.claude/skills/reading-papers/scripts/paper.py <cmd> "<query>" [flags]
```

| cmd | does |
|---|---|
| `resolve` | fuzzy ref → canonical record (DOI, arXiv id, venue, year, OA status, best free URL, free AEA appendices) |
| `get` | resolve, then fetch the cleanest full text available |
| `search` | topic/keyword search across Semantic Scholar + OpenAlex + arXiv, deduped and merged |
| `author` | works by a person, newest first, with disambiguation |
| `cites` | citation graph: cited-by and references; `--contexts` adds the citing sentences |

Flags: `--venue "…"` · `--since/--until YEAR` · `-n N` · `--json` · `--save` · `--raw` ·
`--list-sections` · `--section "<substr>"` · `--affiliation "…"` · `--orcid "…"` · `--contexts`

Keys (optional, all free) live in `~/.claude/secrets/scholar.env`; the script auto-loads them.

## How resolution works, and why order matters

`resolve` is cheap and prints whether a free copy exists and where. Always resolve before
web-searching or guessing URLs. The routing is cost- and Cloudflare-aware:

- An arXiv id or arxiv link → arXiv metadata (free) + OpenAlex if a DOI exists.
- A DOI → OpenAlex (1 credit) + Unpaywall + NBER direct-PDF + AEA free appendices + a Semantic
  Scholar OA-PDF fallback (catches NBER copies the others miss).
- An NBER or SSRN URL → mapped to a DOI *from the URL itself*, skipping the Cloudflare wall.
- A bare title → Crossref first (free); on a confident title match it pulls the record by
  DOI (1 credit). Only an unconfident title falls through to OpenAlex `search` (10 credits).
  So hand it a DOI or arXiv id when you have one; a title costs up to 10× more.

Version-of-record vs preprint: a title search can surface the NBER or SSRN copy. The matcher
ranks the journal version above preprint containers, but econ papers genuinely exist as several
records with different DOIs and different citation counts, so say which one you mean.

`search` is the topic-level entry point (`resolve` is for a known item). One query per source,
merged on DOI → arXiv id → normalized title, re-ranked by reciprocal-rank fusion, one OpenAlex
`search` call per invocation. `--json` carries abstracts and a `sources` block saying which
sources answered; each record names the source of its citation count. A source that 429s or errors
is reported and skipped, never fatal. For a whole literature, use the `litreview` skill, which
wraps this with scoring, Zotero cross-check, and parallel readers.

## Read long papers one section at a time

For any paper with an arXiv copy, map first, then pull only what's needed:

```bash
paper.py get "10.1257/aer.20181169" --list-sections          # cheap map
paper.py get "10.1257/aer.20181169" --section "decomposition" # that section, raw LaTeX
```

The single biggest context saver: a 70K-char paper becomes a 10K-char section with every
equation, `\label`, and `\ref` intact. `--section` carries the preamble's `\newcommand` /
`\newtheorem` definitions along, so macros stay resolvable.

## Math fidelity ladder

`get` tries these in order and prints the `source:` rung it used:

1. arXiv raw LaTeX (`--raw`, or auto when HTML is absent): the authors' own source. Best for
   reproducing/checking an equation. Quote the LaTeX verbatim; don't retype it.
2. arXiv HTML (LaTeXML): `<math alttext="…">` holds the TeX, macros already expanded, clean
   prose around it. Best default for *discussing* a paper.
3. ar5iv: same engine, for papers before arXiv's native HTML (pre-Dec 2023).
4. Born-digital PDF (NBER, JMLR, PMLR, repositories): saved to `~/.claude/cache/papers/`;
   read with `~/.claude/assets/bin/pdfread.py text <pdf>` (`png` for figure pages).
   Text layer survives; math is fine.
5. OCR: only when `get` reports `(SCANNED)`. See the OCR rung below.

Rule of thumb: discuss → HTML; reproduce/check an equation → `--raw`.

### The OCR rung: `~/ocr-examples` (scanned PDFs only)

`get` runs a text-layer check and tags a PDF `(SCANNED)` when the first pages have almost no
extractable text. Only then is OCR worth it. If you have such a setup, route it through the
`~/ocr-examples` repo
(an OCR pipeline on an HPC cluster); do not hand-roll OCR.

- Read first: `~/ocr-examples/README.md`, `docs/ocr-engines.md`, and the relevant
  `scripts/*.py --help`. Load any HPC plugin skills you use before launching a cluster job.
- Two modes. `disk` is for public/non-sensitive PDFs (bytes may live on HPC disk).
  `tunnel` is for sensitive documents: the Mac reads the PDF and streams pages to a compute-node
  service over SSH, so bytes never land on HPC storage and the output is written locally. Use
  tunnel when in doubt about sensitivity.
- Engines (best math first): `olmocr2` (olmOCR-2, Apache-2.0, the default), `deepseek_ocr`,
  `glm_ocr`, `docling` (structure-strong, math-weaker), `pypdf` (text-native only, no OCR).
- Smallest safe start: one document, one worker, one engine, RTX 8000 not A100.
  `just smoke olmocr2 tunnel` proves the path; `just documents-process` / `just engine-olmocr2`
  run real work. Layout is `<documents-root>/<guid>/document.pdf`; feed a GUID list via
  `--from-file` or direct paths via `--pdf-list`. Run `just sync-hpc` after editing that repo.
- Apple-Silicon note: `olmocr2`/`deepseek_ocr`/`glm_ocr` also have a local MLX backend, so a
  single non-sensitive page can be OCR'd on the Mac without the cluster. A large batch belongs on
  a GPU node via Slurm, never the login node.

## Zotero: the user's citation library

A Zotero MCP server is connected; its tools are underscore-named under `mcp__zotero__`, e.g.
`zotero_search_items`. This is the user's own reference manager.

- Reads work only while the Zotero desktop app is open with the local API enabled
  (Settings → Advanced → "Allow other applications on this computer to communicate with Zotero").
  If a `zotero_*` call fails, that's almost always the cause, so ask the user to open Zotero.
- Use it to: search what they already have, read PDF full text and their own annotations/notes,
  pull BibTeX for citing, and find items by tag. Prefer *their* copy of a paper over re-fetching;
  it's faster and it's the version they annotated.
- Writes (adding a found paper to the library) need a web API key in `scholar.env` and are
  hybrid-mode. Don't add items unless asked.
- Natural pairing: `paper.py` finds and reads anything on the open web; Zotero is the private
  library. When the user says "the paper I saved / my notes on X," reach for Zotero first.

## When there is no free copy

Verified July 2026, hard-blocked to plain HTTP: INFORMS (Marketing/Management Science), SSRN,
AEA direct PDF, Elsevier/Wiley/OUP/Chicago, OpenReview anonymous `api2`. Escalate in order:

1. `resolve` already checked for a green-OA copy (repository, arXiv, RePEc, NBER) and iterates
   *all* OA locations, including the ones it doesn't rank first. Trust it; it finds most paywalled
   AER / Marketing Science papers as a free copy elsewhere.
2. Working-paper version: NBER (`nber.org/papers/wXXXXX`), CEPR, author's site. For econ this
   is usually near-identical to the published version.
3. Free AEA appendix/data: for AER/AEJ the article PDF is walled but the online appendix
   (where the proofs live) and replication package are free; `resolve` surfaces them.
4. Playwright MCP, a real browser, the only thing that clears Cloudflare. An institutional
   proxy has the shape
   `https://<your-library>.idm.oclc.org/login?URL=<target>`, but login is typically SSO + MFA, so the
   user must be in the loop. Ask first; one paper at a time. Systematic proxy downloading can
   get the whole university cut off, so never loop it.
5. Scholar Gateway (`semanticSearch`): Wiley-leaning licensed corpus, returns *passages*, not
   full text; can't fetch by DOI. Good for corroborating a claim, not reading a paper.
6. Say plainly that only the abstract is reachable. Never paraphrase an abstract as if you read the
   paper.

## Cost model (OpenAlex, metered since 2026)

| call | credits |
|---|---|
| DOI / ID lookup, plain `filter=` | 1 |
| anything with `search`, incl. `filter=title.search:` | 10 |

Anonymous = 1,000 credits/day. A free key (`openalex.org/settings/api`) → 10×; put it in
`scholar.env` as `OPENALEX_API_KEY`. The script disk-caches every response 30 days, so re-reads
are free. Prefer DOIs/arXiv ids over titles.

## Author queries: where you must stay in the loop

`author` resolves a name via OpenAlex (most-works profile) and prints runner-up matches. Common
names are genuinely hard; do not trust the top pick blindly:

- OpenAlex fragments one person across profiles: the same author appears as a 300-work record
  and a 2-work stub, and even an ORCID can resolve to a stub. Always sanity-check that the
  returned *works match the person's field* (a marketing scholar shouldn't be returning optics
  papers). If they don't, you have the wrong fragment.
- `--affiliation "Columbia"` filters to the current institution, the reliable fix for a common
  name. If nothing matches, the script lists candidates with their ORCIDs instead of guessing.
- `--orcid` is the most deterministic single input, but see the fragment caveat above.
- In CS, DBLP is the gold standard for name disambiguation (explicit homonym suffixes) and its
  author XML also yields the person's Google Scholar / ORCID / ACM ids.
- Diacritics matter: the canonical `display_name` may be `Acemoğlu`, not `Acemoglu`.

`cites --contexts` shows the actual sentences citing a paper (Semantic Scholar; needs `S2_API_KEY`).
Use it to see what a paper is actually being used for.

## Gray areas / known false positives

- Abstract-vs-fulltext: `get` classifies an HTML page structurally (has a References section?
  heading count? body word count?), not by raw length, so a short genuine paper with references
  (an AER P&P note, a Comment) reads as full text and is not flagged. When a page looks
  abstract-only, `get` first tries any PDF the page advertises (the `citation_pdf_url` meta tag, or
  a linked author manuscript) before falling back, which recovers the full text on many repository
  landing pages. If it still returns only the abstract, it says so (label ends `abstract-only`).
- Scanned-PDF flag: heuristic (little text in the first 3 pages). A born-digital paper with a
  figure-only opening could false-flag; conversely a mostly-image PDF with a text cover could slip
  through. Glance at the PDF before committing to an OCR job.
- "Open access" that isn't fetchable: for hybrid-OA Oxford/SAGE, OpenAlex reports a PDF URL on
  `academic.oup.com` / `journals.sagepub.com` (legally open, technically Cloudflare-dead). The
  green-OA (repository/PMC/OSF/arXiv) locations are the ones that resolve; `resolve` prefers them.
- Missing abstracts: Crossref lacks abstracts for JPE (Chicago) and JPSP (APA), though OpenAlex has
  them. Semantic Scholar *elides AEA abstracts* by publisher request.

## Venue cheat sheet

- CS: almost all on arXiv → `--raw` gives exact math. Bulk-friendly mirrors are PMLR (per-volume
  `bibliography.bib`), ACL Anthology (whole corpus is a git repo), JMLR, NeurIPS proceedings.
  Reviews/scores need an authenticated OpenReview client.
- Econ: arXiv `econ.EM`, then NBER (direct PDF). AER/AEJ article PDF is members-only but
  appendices are free. IDEAS pages give the working-paper ↔ published crosswalk.
- Marketing: INFORMS is walled but green OA, so the readable copy is normally an
  institutional repository, which `resolve` finds. 2023+ INFORMS DOIs changed shape; listings
  contain non-article DOIs (`…ack…`, `…eb…`) worth ignoring.
- Psychology: Psych Science is SAGE (PMC holds front matter only). Try PsyArXiv/OSF.

## Honesty rules

- State which version you read (published, arXiv preprint, or NBER working paper). The numbers
  can differ.
- Citation counts are source-specific and disagree (same Econometrica paper: 235 OpenAlex / 177 S2
  / 210 Crossref / 157 OpenCitations). Name the source; never mix counts in one table.
- Never invent a DOI, page number, or quotation. If `resolve` returns nothing, say so.

## Working with many papers

Per the user's standing preference, for several papers at once spawn parallel subagents, one
paper each, returning structured summaries. Every API call in `paper.py` now retries with
exponential backoff on 429/503 (arXiv, Semantic Scholar, OpenAlex, Crossref), and the OpenAlex key
gives a 10,000-credit daily budget, so a fan-out of a dozen readers is safe. Two things still
apply: results are disk-cached (so re-reads across agents are free), and for a very large batch
(many dozens) keep arXiv-heavy concurrency modest, since arXiv politeness is ~1 request / 3s.

## Setup state

- This setup assumes Zotero MCP (`zotero-mcp-launch.sh` reads `scholar.env`), Playwright MCP,
  and Scholar Gateway are installed and connected; adjust to your machine. `paper.py` needs no keys.
- `~/.claude/secrets/scholar.env`: put `OPENALEX_API_KEY` and `S2_API_KEY` here, plus the
  optional Zotero web-API creds; REFERENCE.md §7 says how to get all of them.
- Background on the whole landscape (what's blocked, what's open, why a script beat a fleet of MCP
  servers): `REFERENCE.md` in this directory.