Immutable. This exact content is served forever at /api/v1/blob/d2d55794cb494367.
# AGENTS ## Project Purpose `hermes-arxiv-agent` is a Hermes-based paper monitoring project for arXiv. It is designed to: - search arXiv daily with configurable keywords - download newly discovered PDFs - use Hermes/LLM to extract author affiliations and generate `summary_cn` - write results back to `papers_record.xlsx` - send a daily Feishu report - provide a paper viewer website, either locally or via GitHub Pages The default research topic is LLM quantization, configured in `search_keywords.txt`. ## Core Data Flow Main runtime entry: - [monitor.py](/home/wsg/hermes_path/hermes-arxiv-agent/monitor.py:1) Persistent data: - `papers_record.xlsx`: source of truth for paper metadata, affiliations, summaries, and crawl dates - `papers/`: downloaded PDFs - `new_papers.json`: intermediate file for Hermes/agent processing - `pending_llm_ids.txt`: papers still missing `affiliations` or `summary_cn` - `viewer/papers_data.json`: static website data built from the Excel file Daily flow: 1. `python3 monitor.py` 2. arXiv search returns the newest matching papers 3. new PDFs are downloaded and appended/upserted into Excel 4. Hermes/agent reads `new_papers.json` 5. Hermes/agent extracts `affiliations` from PDFs and writes `summary_cn` 6. Hermes/agent updates `papers_record.xlsx` 7. `python3 viewer/build_data.py` rebuilds `viewer/papers_data.json` 8. `python3 monitor.py --sync-pending-state` refreshes `pending_llm_ids.txt` 9. optional: `bash scripts/publish_viewer.sh` pushes viewer changes and triggers GitHub Pages Important: - affiliation extraction is now done by Hermes/agent during the cron workflow - old standalone extractor scripts were removed from the main repo flow - `crawled_date` means the latest processing/write-back date, not a permanent first-seen date ## Deployment Modes This repo supports two deployment modes. ### 1. Local Mode Use this when the user wants: - daily Feishu delivery - local Excel/PDF storage - local web viewer via `python3 viewer/run_viewer.py` Characteristics: - uses the upstream repository directly - does not require a fork - does not push to GitHub Pages - generated cron prompt comes from `cronjob_prompt.txt` ### 2. GitHub Pages Mode This is an enhanced version of local mode. It includes everything in local mode, plus: - automatic static-site publishing to the user's own GitHub fork - GitHub Actions deployment for Pages Characteristics: - requires the user to fork the repository first - `origin` should point to the user's own fork - SSH is preferred for Git pushes - generated cron prompt comes from `cronjob_prompt.pages.txt` - cron includes `bash scripts/publish_viewer.sh` ## Deployment Files Key deployment and ops files: - [README.md](/home/wsg/hermes_path/hermes-arxiv-agent/README.md:1): user-facing install and usage guide - [AGENT_SKILL.md](/home/wsg/hermes_path/hermes-arxiv-agent/AGENT_SKILL.md:1): Hermes deployment skill - [UPDATE_CRON_SKILL.md](/home/wsg/hermes_path/hermes-arxiv-agent/UPDATE_CRON_SKILL.md:1): Hermes skill for refreshing cron only - [prepare_deploy.sh](/home/wsg/hermes_path/hermes-arxiv-agent/prepare_deploy.sh:1): generates `cronjob_prompt.generated.txt` - [cronjob_prompt.txt](/home/wsg/hermes_path/hermes-arxiv-agent/cronjob_prompt.txt:1): local mode cron template - [cronjob_prompt.pages.txt](/home/wsg/hermes_path/hermes-arxiv-agent/cronjob_prompt.pages.txt:1): GitHub Pages mode cron template Mode persistence: - `.deploy_mode` stores `local` or `pages` - `prepare_deploy.sh` reads it unless `DEPLOY_MODE` is explicitly provided ## Viewer Viewer files: - [viewer/index.html](/home/wsg/hermes_path/hermes-arxiv-agent/viewer/index.html:1) - [viewer/app.js](/home/wsg/hermes_path/hermes-arxiv-agent/viewer/app.js:1) - [viewer/styles.css](/home/wsg/hermes_path/hermes-arxiv-agent/viewer/styles.css:1) - [viewer/build_data.py](/home/wsg/hermes_path/hermes-arxiv-agent/viewer/build_data.py:1) Viewer behavior: - reads `viewer/papers_data.json` - supports filtering by `crawled_date` or `published_date` - favorites are stored in browser `localStorage` - GitHub Pages mode does not use server-side favorites ## Publish Safety Rules GitHub Pages publishing is intentionally guarded. - `scripts/publish_viewer.sh` refuses to publish if `pending_llm_ids.txt` is non-empty - it only stages site-related files - it retries `git push` with backoff - it pushes to the current configured remote; it should never be hardcoded to the upstream repo If a user is in GitHub Pages mode, publishing must target their own fork, not `genggng/hermes-arxiv-agent` unless they explicitly own and use that repository. ## Common Commands Local viewer: ```bash cd viewer python3 run_viewer.py ``` Rebuild viewer data: ```bash python3 viewer/build_data.py ``` Refresh pending state: ```bash python3 monitor.py --sync-pending-state ``` Regenerate deploy prompt: ```bash bash prepare_deploy.sh ``` Regenerate deploy prompt for Pages mode: ```bash DEPLOY_MODE=pages bash prepare_deploy.sh ``` Manual Pages publish: ```bash bash scripts/publish_viewer.sh ``` ## What To Check First In A New Session When starting a new work session on this repo, check these first: 1. `git status --short` 2. current deployment mode from `.deploy_mode` if it exists 3. whether the task is about local mode or GitHub Pages mode 4. whether `pending_llm_ids.txt` is empty 5. whether the user wants code changes, cron updates, or only data publishing Useful questions to answer early: - Is this a deployment problem, a data-processing problem, or a viewer/UI problem? - Is the user working in local mode or GitHub Pages mode? - If publishing is involved, does `origin` point to the user's own fork? ## Current Conventions - prefer `rg` for search - use `apply_patch` for file edits - avoid touching unrelated untracked user files - do not revert user changes unless explicitly asked - for Pages-related changes, preserve the distinction between local mode and Pages mode ## Recent Changes And Known Pitfalls Recent important changes: - GitHub Pages support was added as an optional deployment mode on top of local mode - viewer favorites were migrated from server-side file storage to browser `localStorage` - mobile viewer layout was improved, including narrow-screen filter behavior and date input sizing - a favicon was added for the viewer - arXiv fetch size was increased from 20 to 50 - SSH is now the recommended Git remote mode for Pages publishing - update-cron instructions were split into explicit local-mode and GitHub-Pages-mode phrases - legacy standalone affiliation extractor scripts were removed from the repo flow Known historical pitfalls: ### 1. Incomplete viewer data was previously published too early Old behavior: - `monitor.py` used to export `viewer/papers_data.json` before Hermes finished writing `affiliations` and `summary_cn` - this caused GitHub Pages to publish incomplete summaries or affiliations Current fix: - `monitor.py` no longer exports viewer data at that premature point - `scripts/publish_viewer.sh` refuses to publish when `pending_llm_ids.txt` is non-empty ### 2. Git push could fail after local commit Old behavior: - `publish_viewer.sh` could commit successfully and then fail on `git push` - there was no retry logic Current fix: - push now retries with backoff - if push still fails, the local commit remains and can be pushed manually later ### 3. HTTPS remotes caused auth confusion for Pages publishing Old behavior: - users could add SSH keys to GitHub, but the repo remote was still `https://...` - pushes failed because the repo was not actually using SSH auth Current fix: - Pages-mode docs and deployment logic now prefer SSH remotes - publishing should target the current configured remote, ideally the user's fork over SSH ### 4. Upstream repo must not be the default publish target for open-source users Old risk: - automation could accidentally be interpreted as publishing to the upstream public repo Current fix: - local mode does not publish at all - GitHub Pages mode requires the user's own fork - docs and skills now distinguish the two modes explicitly ### 5. `crawled_date` is intentionally the latest processed date This is not a bug. - `published_date` is the arXiv-side date - `crawled_date` is the latest date the record was processed and written back locally - reprocessed papers may therefore show today's `crawled_date` ### 6. Affiliation extraction now depends on the Hermes/agent workflow This is intentional. - there is no standalone extractor in the main production path anymore - if affiliation quality becomes a problem, the first place to inspect is the cron prompt and Hermes/LLM workflow, not a removed local script