fetch-llm-apps · diff
git:20260803.ce45f46 to git:20260929.e51f396
16 added, 129 removed. Audit A to A.
---
name: fetch-llm-apps
- description: "Workflow for updating the popular LLM applications pool (section/x_llm_apps.md) using fetch_llm_apps.py. Covers full refresh, alternate exports, topic tuning, and common pitfalls. USE FOR: Refreshing the ranked GitHub applications list linked from applications.md. DO NOT USE FOR: Hand-curating application entries inside applications.md or adding GitHub star badges to the generated file."
- ---
-
- ## Overview
-
- The pool file `section/x_llm_apps.md` is a generated ranked list of GitHub repositories related to LLM apps, agents, chat UIs, workflow builders, and similar application-layer projects.
-
- It is generated by `code/fetch_llm_apps.py` using GitHub topic search, deduplicated across multiple topics, and sorted by GitHub star count descending.
-
- The section `#### Popular LLM Applications (GitHub Stars >= 1000)` in `section/applications.md` links to this generated file with a one-line description only. Do not paste generated entries directly into `applications.md`.
-
- ---
-
- ## Script Reference
-
- **Script:** `code/fetch_llm_apps.py`
- **Python env:** `.venv\Scripts\python.exe`
-
- ### Key CLI Arguments
-
- | Argument | Default | Purpose |
- |----------|---------|---------|
- | `--output` | `section/x_llm_apps.md` | Output file path. Extension controls format: `.md`, `.json`, `.csv` |
- | `--min-stars` | `1000` | Minimum GitHub star threshold |
- | `--topics` | curated list | GitHub topics to query and merge |
- | `--token` | `GITHUB_TOKEN` env var | GitHub PAT for higher rate limits |
- | `--show` | `30` | Number of repos printed to console |
- | `--timeout` | `20` | Per-request timeout in seconds |
- | `--max-retries` | `4` | Max retries per request |
- | `--backoff` | `1.0` | Initial retry backoff |
- | `--sleep` | `1.0` | Delay between successful page requests |
- | `--include-archived` | off | Include archived repositories |
- | `--append` | off | Merge new results with an existing output file, then re-sort by stars |
-
+ description: "Regenerate the ranked GitHub LLM applications pool (section/x_llm_apps.md) with fetch_llm_apps.py. USE FOR: refreshing the pool linked from applications.md. DO NOT USE FOR: hand-curating entries in applications.md or adding star badges to the generated file."
---
- ## Workflow
-
- ### 1. Full refresh of the markdown pool
-
- Use this for the normal update path.
-
- ```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py
- ```
-
- - Rewrites `section/x_llm_apps.md`.
- - Uses `--min-stars 1000` by default to match the section title.
- - Excludes archived repos unless `--include-archived` is passed.
- - Writes compact numbered entries instead of badge-heavy markdown.
-
- ### 2. Authenticated refresh to avoid GitHub API limits
-
- ```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py --token $env:GITHUB_TOKEN
- ```
-
- Use a GitHub PAT when doing a full refresh across many topics. Unauthenticated search is heavily rate-limited.
-
- ### 3. Generate a stricter ranking
-
- ```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py --min-stars 2000
- ```
-
- Use this when you want a tighter list. If you change the threshold materially, update the descriptive text in `section/applications.md` so the label stays truthful.
-
- ### 4. Export raw data for review
-
- ```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py --output files/x_llm_apps.json
- .venv\Scripts\python.exe code/fetch_llm_apps.py --output files/x_llm_apps.csv
- ```
-
- Use JSON or CSV when you want to inspect or post-process the ranked repo pool before regenerating markdown.
-
- ### 5. Tune the topic set
-
- ```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py --topics llm agent rag chatbot ai-workflow
- ```
-
- - Topics are GitHub repository topics, not free-text queries.
- - Results are deduplicated by `full_name` after all topic passes complete.
- - If the topic set changes substantially, regenerate the markdown pool rather than editing it by hand.
-
- ### 6. Topic-specific doc update
+ `section/x_llm_apps.md` is generated by `code/fetch_llm_apps.py` from GitHub topic and phrase searches, deduplicated and sorted by stars. Phrase facets broaden discovery across AI engineering, applications, infrastructure, and learning resources. External catalogs may inform search vocabulary only; never copy their repository records into the generated file. `applications.md` only links to it. Never edit the generated file by hand or paste its entries elsewhere.
- If the output is intentionally narrowed to a subset such as `gemini claude azure-openai copilot assistant`, keep the ranked entries as generated, then update the document metadata and the linking description in `section/applications.md` to reflect the narrowed scope.
+ Search hits are candidates, not proof of relevance. The fetcher keeps repositories only when their name, description, or GitHub topics explicitly identify an AI/LLM, model, agent, or related application/tooling focus; generic lists and incidental mentions are filtered out.
- ### 7. Add topics without discarding the current pool
+ ## Commands
```powershell
- .venv\Scripts\python.exe code/fetch_llm_apps.py `
- --append `
- --topics llm agent rag chatbot ai-workflow
- ```
-
- `--append` parses the existing compact entries, merges newly fetched repositories by `full_name`, and rewrites the file sorted by star count. Use it when expanding coverage; use the normal full refresh when the default topic set changes substantially.
-
- ---
-
- ## Output Format
-
- Each entry in `section/x_llm_apps.md` follows this compact format:
-
- ```markdown
- 1. [owner/repo](https://github.com/owner/repo): Short GitHub description. [Mon YYYY] (⭐ 12,345)
+ .venv\Scripts\python.exe code/fetch_llm_apps.py # full refresh (min 1000 stars)
+ .venv\Scripts\python.exe code/fetch_llm_apps.py --token $env:GITHUB_TOKEN # avoid rate limits
+ .venv\Scripts\python.exe code/fetch_llm_apps.py --append --topics llm agent # add topics, keep current pool
+ .venv\Scripts\python.exe code/fetch_llm_apps.py --search-terms "agent memory" "AI coding agent"
+ .venv\Scripts\python.exe code/fetch_llm_apps.py --output files/x_llm_apps.json # raw export (.json/.csv)
```
- - Entries are sorted by GitHub stars descending.
- - The date is the repository creation month, formatted as `[Mon YYYY]`.
- - The star count is static text in parentheses using the star symbol.
- - Do **not** append realtime shields or badges in this generated file.
-
- The file header includes:
-
- - UTC generated timestamp
- - GitHub Search API source note
- - searched topic list
- - total repository count
-
- ---
-
- ## Common Pitfalls
-
- 1. **Using the wrong output target:** The generated ranking belongs in `section/x_llm_apps.md`, not inline inside `section/applications.md`.
-
- 2. **Adding GitHub star badges:** This file intentionally uses static text like `(⭐ 12,345)`. Do not run `add_github_stars.py` on it.
-
- 3. **Forgetting the star threshold contract:** The linked section title says `≥1000`. If you generate with a different threshold, either restore `1000` or update the section label and description.
-
- 4. **Unauthenticated rate limits:** Full runs over many topics can stall or fail without a token. Prefer `GITHUB_TOKEN` for routine refreshes.
-
- 5. **Assuming GitHub topics are comprehensive:** Some strong repos do not declare useful topics and may be missed. Expand `--topics` or curate separately if coverage is insufficient.
+ Other options: `--min-stars`, `--topics`, `--include-archived`, `--show`. Run with `--help` for the rest.
- 6. **Archived repos polluting the ranking:** Archived repos are excluded by default. Only include them deliberately.
+ ## Notes
- 7. **Hand-editing generated entries:** Manual changes will be lost on the next run. Adjust the script inputs or post-process separately instead.
+ - Entry format: `1. [owner/repo](url): Description. [Mon YYYY] (⭐ 12,345)`. Stars are static text; do not run `add_github_stars.py` on it.
+ - The section title in `applications.md` says `>= 1000`. If you change `--min-stars` or narrow the topics, update that label and description.
+ - Unauthenticated runs hit rate limits; use `GITHUB_TOKEN`.
+ - Phrase searches across repository name, description, and README help find projects without useful topic tags; add `--search-terms` facets when coverage gaps appear.
+ - Do not source or copy repository records from third-party catalogs; use their category vocabulary only to improve GitHub query facets.