paper-search · diff

git:20260930.33fc1c2 to git:20261002.427d761

53 added, 14 removed. Audit A to A.

---
name: paper-search
- description: Search research papers across alphaXiv, OpenAlex, bioRxiv, PubMed, Hugging Face Papers, and Papers with Code in one command that honours the human's per-source on/off switches. Use when finding papers, related work, prior art, or code for a method, or when the human asks to switch a literature source on or off.
+ description: Search and read research papers across alphaXiv, OpenAlex, bioRxiv, PubMed, Hugging Face Papers, Papers with Code, and arXiv, honouring the human's per-source on/off switches and full-text read order. Use when finding papers, related work, prior art, or code for a method; when reading a paper's full text or record from its id; or when the human asks to switch a source on or off or reorder where papers are read from.
---
# Paper search
`scripts/paper_search.py` in this skill's directory (`$PS` below) searches every
literature source the human has switched on, in parallel, and returns one JSON
- shape. The switches live in the repository's
+ shape; `fetch` reads one paper through the human's read order. The switches and
+ the read order live in the repository's
[`memory/paper-sources.json`](../../../memory/paper-sources.json); the human
- flips them from the **Papers** button in os-ui's agent windows.
+ flips the switches from the **Papers** button in os-ui's agent windows.
A source that is off is off on every route. Reach literature through `$PS`, and
leave a switched-off source alone through other routes as well: MCP connectors,
other skills, and web fetches. When the evidence needs a source that is off,
say so and let the human decide.
## Search
Run from anywhere inside the repository; the script finds the switch file by
walking up from the working directory.
```bash
uv run $PS sources # which sources are on
uv run $PS search "sparse autoencoder feature splitting"
uv run $PS search "KV cache compression" --source alphaxiv --source huggingface --published-after 2025-01-01
uv run $PS search "ImageNet top-1" --mode keyword --limit 5
```
Without `--source`, every source that is on is searched. Pick the sources that
fit the field: machine learning reads best through `alphaxiv`, `huggingface`,
and `pwc`; journal and cross-disciplinary work through `openalex`; biology
- through `biorxiv` and `pubmed`.
+ through `biorxiv` and `pubmed`; an exact title, author, or arXiv category
+ through `arxiv` with a fielded query such as `ti:"context language models"`.
| Source | Covers | Extra fields |
|---|---|---|
| `alphaxiv` | arXiv full text: CS, math, physics, stats | `votes`, full-text `snippets` |
| `openalex` | all disciplines, journals and conferences | `citations` |
| `biorxiv` | biology preprints, via OpenAlex's bioRxiv index | `citations` |
| `pubmed` | biomedical and life-science journals | `doi`, `venue` |
| `huggingface` | arXiv ML papers on Hugging Face Papers | `votes` (upvotes), `code` (GitHub) |
| `pwc` | Papers with Code catalog, through its MCP server | `citations`, `code_repos`, `official_code`; no abstract |
+ | `arxiv` | all of arXiv, through its own API; `ti:` `au:` `abs:` `cat:` queries pass through | `doi`, `venue` (journal reference) |
Controls, applied to every source that supports them:
- `--mode semantic` (default) or `keyword`: alphaXiv embedding vs full-text
- keyword search, PwC hybrid vs keyword. Use `keyword` for exact method names,
- acronyms, benchmarks, and title phrases. Other sources ignore it.
+ keyword search, PwC hybrid vs keyword, arXiv all-words vs exact phrase. Use
+ `keyword` for exact method names, acronyms, benchmarks, and title phrases.
+ Other sources ignore it.
- `--published-after` / `--published-before YYYY-MM-DD`: inclusive window.
Supply one only when the question asks for a period.
- `--limit N`: results per source (default 10; PwC caps at 25).
## Read the result
The script prints `{query, searched, disabled, errors, results}`. Each result
has `source`, `id`, `title`, `url`, and, when its source provides them,
`abstract`, `publication_date`, `authors`, and the extra fields above. `id` is
an arXiv id, a DOI, an OpenAlex `W…` id, a PMID, or a PwC catalog id.
- Results interleave the sources rank by rank. A paper several sources found
appears once, under the first source, with the others in `also_in`; it keeps
that source's `votes` and takes missing fields such as `code` from the rest.
- `errors` names each source that failed. Report a failed source as unsearched
rather than as evidence that no such literature exists; the same holds for
an empty result. Exit 0 means at least one source answered, 1 means every
searched source failed, 2 means invalid use, including asking for a source
that is off.
- Cite only papers the results or a follow-up read actually returned.
Known source behaviour:
- PwC allows 10 hybrid searches per minute per IP; bursts need `--mode keyword`.
`PWC_MCP_URL` points at another server, such as a local `pwc-mcp`.
- Hugging Face has no date filter, so a window is applied to a wider pool
afterwards and can return fewer results than `--limit`.
- A PubMed window matches print or electronic dates, while `publication_date`
shows the electronic one, so a hit can predate the window.
+ - arXiv's terms allow one request every three seconds; the script waits out
+ the gap, so back-to-back arXiv calls are slow rather than refused.
## Keys
Every user brings their own optional keys: `OPENALEX_API_KEY` (OpenAlex and
bioRxiv), `NCBI_API_KEY` (PubMed), and `HF_TOKEN` (Hugging Face). The script
reads them from the environment, else from the repository's gitignored `.env`
(template: [`.env.example`](../../../.env.example)); `uv run $PS keys` shows
which are set. alphaXiv and PwC need none.
A rate-limited source (`HTTP 429`) comes back with the key that lifts the
limit. Relay that next step to the human: they paste the key under **Keys** in
os-ui's Papers panel, which runs `keys --set NAME` with the value on stdin, or
edit `.env` themselves. Keep key values out of commits, prompts, and logs.
- ## Reading an alphaXiv paper
+ ## Read a paper
+ `fetch` takes an arXiv id or URL, a DOI, an OpenAlex `W…` id, or a PubMed id
+ and routes it by shape:
+
+ ```bash
+ uv run $PS fetch 2609.37725 # full text through the read order
+ uv run $PS fetch 2609.37725 --via arxiv # one source only
+ uv run $PS fetch 2609.37725 --report # alphaXiv's ~10 KB overview, for skimming
+ uv run $PS fetch 10.1038/nature14539 # a DOI or PMID: its record, no full text
+ ```
+
+ An arXiv paper is read from each source in the read order (default
+ `huggingface`, `alphaxiv`, `arxiv`), skipping switched-off sources, until one
+ returns a complete text: at least 5,000 characters with a heading such as
+ Introduction or Method. A cut-short text, such as the lone figure caption
+ Hugging Face serves for some fresh submissions, falls through to the next
+ source. The text goes to a file (`--out`, default under the system temp
+ directory); stdout is a JSON report with `tried` (each source's outcome),
+ `source`, `url`, `file`, `chars`, and `metadata` from arXiv, else Hugging Face.
+ Read the file in parts and cite the `source` and `url` the report names.
+
+ Exit 1 means no source returned a complete text: report the `tried` outcomes
+ and the `pdf` link, and let the human decide; `--via` or `--report` through a
+ switched-off source exits 2. A DOI, OpenAlex id, or PMID returns its record
+ from OpenAlex, bioRxiv, or PubMed under that source's switch; follow its
+ `url` or `doi` for the text.
+
When the harness has a logged-in alphaXiv connector (Claude Code's, for
- instance), use it to read an alphaXiv hit's full text or put questions to its
- PDF. It belongs to the `alphaxiv` source, so it follows that switch. Search
- itself stays with `$PS`, which needs no login and works in every harness.
+ instance), use it to put questions to a paper's PDF. It belongs to the
+ `alphaxiv` source, so it follows that switch.
## Switch a source
Only on the human's word, since the switches are the human's choice:
```bash
uv run $PS sources --disable pubmed
uv run $PS sources --enable pubmed --json
```
- This rewrites `memory/paper-sources.json`, a tracked file; commit it with the
+ The read order is the human's choice too: `sources --read-order
+ alphaxiv,huggingface,arxiv` sets it, and leaving a source out of it keeps
+ `fetch` from reading through it while search still uses it.
+
+ Both rewrite `memory/paper-sources.json`, a tracked file; commit it with the
change that motivated it.
_Adapted from [alphaXiv OpenResearch](https://github.com/alphaXiv/OpenResearch)
- (MIT): its `orx discover` source switches and shared result shape are
- reimplemented here, and Hugging Face Papers and the
+ (MIT): its `orx discover` source switches and shared result shape, and its
+ `orx paper` id routing and alphaXiv reads, are reimplemented here, and Hugging
+ Face Papers and the
[Papers with Code MCP server](https://github.com/huggingface/pwc-cli#mcp-server)
- are added as sources._
+ are added as sources. The arXiv source reimplements the query syntax and
+ rate limit of
+ [science-skills' `literature_search_arxiv`](https://github.com/google-deepmind/science-skills)
+ (Apache-2.0)._