google-scholar-search · diff
git:20260409.17cffef to git:20260409.1ce409c
74 added, 74 removed. Audit A to A.
---
name: google-scholar-search
- description: "Search Google Scholar for academic papers and author profiles. Use when discovering papers to triage — by keyword, author, or year range. Returns titles, authors, abstracts, and links. Triggers: search papers, find papers, google scholar, search scholar, find author, author profile, literature search, paper discovery."
+ description: "Search Google Scholar for academic papers and author profiles. Use when discovering papers to triage — by keyword, author, or year range. Returns titles, authors, abstracts, and links. Triggers: search papers, find papers, google scholar, search scholar, find author, author profile, literature search, paper discovery, search academic literature."
allowed-tools: Read, Write, Bash
---
# Google Scholar Search
- Search Google Scholar for academic papers and author profiles to support literature discovery before triaging.
+ Search Google Scholar for academic papers and author profiles to build a candidate list before triaging.
## When to Use
- Discovering papers on a topic before running `triage-paper`
- - Finding all papers by a specific author
- - Scoping a research area with year-range filtering
- - Exporting a candidate list to JSON for batch triage
+ - Finding papers by a specific author, optionally filtered by year range
+ - Scoping a research area and exporting a candidate list to JSON for batch review
## When Not to Use
- The paper is already known (arxiv ID, DOI, URL) — go straight to `triage-paper`
- - You need stable, structured metadata — prefer the `semantic-scholar` MCP if configured (no scraping, no rate-limit risk)
- - The search is for biomedical literature — use PubMed instead
+ - The `semantic-scholar` MCP is configured — optionally prefer it; it returns structured data with no rate-limit risk
+ - The search is for biomedical literature — use PubMed or a domain-specific source
+ - A candidate JSON already exists at `/tmp/<topic>-candidates.json` — reuse it rather than re-running
## Mindset
Search is discovery, not triage. The goal is a candidate list, not a finished reference.
- 1. **Return candidates, not conclusions**: hand the list to the user and let them choose what to triage.
- 2. **Scraping is fragile**: Google Scholar blocks scrapers aggressively. A blocked response is not a bug — set expectations and offer the MCP fallback.
- 3. **Prefer MCP when available**: if `semantic-scholar` or `google-scholar` MCP servers are configured, use them instead of this script — they are more reliable and return richer structured data.
-
- ## Setup
+ 1. **Return candidates, not conclusions**: hand the list to the user; let them choose what to triage.
+ 2. **Scraping is fragile**: a gotcha is that a blank result may be a silent CAPTCHA redirect, not an empty query. Always verify by checking the raw response length before reporting "no results".
+ 3. **MCP first**: a common pitfall is invoking the script without checking whether a `semantic-scholar` or `google-scholar` MCP is available. MCPs are faster, structured, and avoid rate-limit risk.
- Install Python dependencies once before first use:
+ ## Workflow
- ```bash
- # Recommended: uv (fastest)
- cd skills/documentation/research/google-scholar-search
- uv venv && source .venv/bin/activate
- uv pip install -r requirements.txt
+ ### 1. Check MCP availability
- # Alternative: standard venv
- python3 -m venv .venv && source .venv/bin/activate
- pip install -r requirements.txt
- ```
+ Consider the `semantic-scholar` or `google-scholar` MCP server first. If either is configured and reachable, prefer it over this script.
- ## Workflow
+ ### 2. Set up the environment (first run only)
- ### 1. Confirm MCP availability
+ See [setup-and-troubleshooting.md](references/setup-and-troubleshooting.md) for venv creation and dependency installation.
- Check whether a `semantic-scholar` or `google-scholar` MCP server is configured. If yes, prefer the MCP — stop here and use it directly.
+ ### 3. Run the search
- ### 2. Run the search
+ Activate the venv, then choose the appropriate subcommand:
```bash
- # Activate the venv if not already active
- source skills/documentation/research/google-scholar-search/.venv/bin/activate
-
# Basic keyword search
- python3 scripts/google-scholar-search.py search --query "<topic>" --results 10
+ ./scripts/google-scholar-search.py search --query "retrieval-augmented generation" --results 10
+ ```
+ ```bash
# Advanced: filter by author and year range
- python3 scripts/google-scholar-search.py advanced \
- --query "<topic>" \
- --author "<author name>" \
- --year-start <YYYY> --year-end <YYYY> \
- --results 10
+ ./scripts/google-scholar-search.py advanced \
+ --query "LLM reasoning" --author "Yann LeCun" \
+ --year-start 2020 --year-end 2024 --results 10
+ ```
+ ```bash
# Author profile
- python3 scripts/google-scholar-search.py author --name "<author name>"
+ ./scripts/google-scholar-search.py author --name "Geoffrey Hinton"
+ ```
- # Export to JSON for review
- python3 scripts/google-scholar-search.py search --query "<topic>" \
- --format json --output /tmp/candidates.json
+ ```bash
+ # Export candidate list to JSON
+ ./scripts/google-scholar-search.py search \
+ --query "prompt compression" --format json --output /tmp/candidates.json
```
- ### 3. Handle blocked responses
+ ### 4. Parse JSON output (optional)
- If the script returns an HTTP 429 or CAPTCHA response:
+ ```bash
+ # Preview titles from a saved candidate list
+ python3 -c "import json; [print(p['title']) for p in json.load(open('/tmp/candidates.json'))]"
+ ```
- - Wait 60 seconds and retry once.
- - If still blocked, report to the user and offer the `semantic-scholar` MCP as fallback.
- - Do not retry in a loop.
+ ### 5. Handle blocked responses
- ### 4. Present candidates
+ If the script returns HTTP 429 or an unusually short HTML body, wait 60 seconds and retry once:
- Show the result list to the user. For each promising paper, offer:
+ ```bash
+ sleep 60 && ./scripts/google-scholar-search.py search --query "<topic>" --results 10
+ ```
- > Found N results for your query. Would you like to triage any of these with `triage-paper`?
- > Provide the paper URL or title and I will run the full triage workflow.
+ NEVER retry in a tight loop. If still blocked, offer the `semantic-scholar` MCP as fallback and document the blocker.
- Do not triage automatically — always confirm with the user first.
+ ### 6. Present candidates and hand off
- ## Quick Commands
+ Show the result list. For each promising paper, offer:
- ```bash
- # Install dependencies
- cd skills/documentation/research/google-scholar-search && uv venv && source .venv/bin/activate && uv pip install -r requirements.txt
+ > Found N results. Would you like to triage any of these with `triage-paper`?
- # Basic search
- python3 scripts/google-scholar-search.py search --query "attention mechanism" --results 10
+ NEVER triage automatically — always confirm with the user first.
- # Advanced search
- python3 scripts/google-scholar-search.py advanced --query "LLM reasoning" --year-start 2022 --year-end 2024 --results 10
+ ## Anti-Patterns
- # Author profile
- python3 scripts/google-scholar-search.py author --name "Ilya Sutskever"
+ ### NEVER auto-triage search results
- # JSON export
- python3 scripts/google-scholar-search.py search --query "RAG retrieval" --format json --output /tmp/candidates.json
- ```
+ **WHY:** Discovery and triage are separate quality gates. Auto-triaging bypasses user review.
- ## Anti-Patterns
+ **BAD** Run `triage-paper` on every result. → **GOOD** Present the list; wait for the user to choose.
- ### NEVER triage without user confirmation
+ ### NEVER loop on blocked responses
- **WHY:** Discovery and triage are separate steps. Auto-triaging a search result bypasses the quality gate.
+ **WHY:** Repeated retries worsen the block and may trigger a longer ban.
- **BAD** Run `triage-paper` on every search result automatically. → **GOOD** Present the candidate list and ask which papers to triage.
+ **BAD** Retry in a tight loop. → **GOOD** Retry once after 60 s; switch to MCP fallback on second failure.
- ### NEVER loop on blocked responses
+ ### NEVER treat scraped abstracts as canonical
- **WHY:** Retrying aggressively after a 429 or CAPTCHA worsens the block and can cause a longer ban.
+ **WHY:** Scraped text is truncated and misformatted. Passing it to `triage-paper` corrupts the research record.
- **BAD** Retry in a tight loop until results appear. → **GOOD** Retry once after 60 seconds; if still blocked, switch to the `semantic-scholar` MCP fallback.
+ **BAD** Use the scraped abstract as the paper summary. → **GOOD** Use results for discovery only; re-fetch from arxiv or DOI during triage.
- ### NEVER treat search results as verified metadata
+ ### NEVER skip the MCP availability check
- **WHY:** Web-scraped titles, authors, and abstracts can be truncated or misformatted. Always re-resolve from the source (arxiv, DOI) during triage.
+ **WHY:** The script is the fragile fallback. Skipping the check needlessly risks rate-limiting.
- **BAD** Use the scraped abstract as the triage summary. → **GOOD** Use search results as discovery only; re-fetch the full paper during `triage-paper`.
+ **BAD** Invoke the script without checking for an MCP first. → **GOOD** Check for `semantic-scholar`/`google-scholar` MCP; only fall back to the script if neither is available.
+ ### NEVER treat year-range results as exhaustive
+
+ **WHY:** Google Scholar's year filter is a ranking hint, not a strict predicate. Papers outside the range may appear; papers inside may be missing.
+
+ **BAD** Claim "all 2020–2024 papers on X". → **GOOD** Qualify results as "a sample in the requested range, as returned by Google Scholar."
+
+ ### NEVER report "no results" without checking for a silent block
+
+ **WHY:** A 200 response with a near-empty body is often a CAPTCHA redirect, not a genuine empty result set.
+
+ **BAD** Report "no papers found" when the HTML body is unusually short. → **GOOD** Check response length; if under ~1 KB, treat it as a potential block and retry once.
+
## References
- - **Script**: [google-scholar-search.py](scripts/google-scholar-search.py)
+ - **Setup & troubleshooting**: [setup-and-troubleshooting.md](references/setup-and-troubleshooting.md)
+ - **Script source**: [google-scholar-search.py](scripts/google-scholar-search.py)
- **Dependencies**: [requirements.txt](requirements.txt)
- **Upstream**: [JackKuo666/google-scholar-search-skill](https://github.com/JackKuo666/google-scholar-search-skill)
- - **Preferred alternative**: `semantic-scholar` MCP — see `triage-paper` for MCP config
+ - **Preferred alternative**: see `triage-paper` for `semantic-scholar` MCP config