building-github-index · diff
v0.0.1 to v2.0.0
58 added, 38 removed. Audit A to A.
---
name: building-github-index
description: Generate progressive disclosure indexes for GitHub repositories to use as Claude project knowledge. Use when setting up projects referencing external documentation, creating searchable indexes of technical blogs or knowledge bases, combining multiple repos into one index, or when user mentions "index", "github repo", "project knowledge", or "documentation reference".
metadata:
- version: 0.0.1
+ version: 2.0.0
---
# Building GitHub Index
- Create markdown indexes of GitHub repositories optimized for Claude project knowledge. Indexes enable retrieval of source content (.md, .qmd, .ipynb) via GitHub API without parsing rendered HTML/CSS/JS noise.
+ Create markdown indexes of GitHub repositories optimized for Claude project knowledge. Indexes enable retrieval via GitHub API with semantic descriptions for effective matching.
## Quick Start
```bash
+ # Documentation repos (markdown/notebooks)
python scripts/github_index.py owner/repo -o index.md
+
+ # Code repos (extract symbols via tree-sitter)
+ python scripts/github_index.py owner/repo --code-symbols -o index.md
+
+ # Multiple repos combined
python scripts/github_index.py owner/repo1 owner/repo2 -o combined.md
- python scripts/github_index.py owner/repo --include-patterns "docs/**" "blog/**"
```
## Script Options
| Flag | Description |
|------|-------------|
| `-o, --output` | Output file (default: `github_index.md`) |
- | `--token` | GitHub PAT; also reads `GITHUB_TOKEN` env var |
- | `--include-patterns` | Only index matching globs: `"docs/**" "blog/**"` |
- | `--exclude-patterns` | Skip matching globs: `"test/**" "drafts/**"` |
+ | `--token` | GitHub PAT; also reads `GITHUB_TOKEN` env |
+ | `--include-patterns` | Only index matching globs: `"docs/**" "src/**"` |
+ | `--exclude-patterns` | Skip matching globs: `"test/**"` |
| `--max-files` | Cap files per repo (default: 200) |
- | `--skip-fetch` | Tree only, no content fetch; faster but filename-derived descriptions |
+ | `--skip-fetch` | Tree only, no content fetch (fast, filename-only descriptions) |
+ | `--code-symbols` | Include code files, extract function/class names via tree-sitter |
- **Default exclusions:** `node_modules`, `.git`, `__pycache__`, `venv`, `dist`, `build`, `_site`, `.next`, plus underscore-prefixed files.
+ ## Description Extraction Priority
- **Rate limits:** 60/hr unauthenticated, 5000/hr with token.
+ 1. **YAML frontmatter** - `title:` and `description:` fields
+ 2. **Markdown headings** - First h1/h2 as title, subsequent as topics
+ 3. **Notebook cells** - First markdown cell heading
+ 4. **Code symbols** - Public function/class names (with `--code-symbols`)
+ 5. **Path-derived** - Convert filename to words (fallback)
- ## Output Structure
+ ## When Descriptions Fail
+ Some repos have stub files (links to external docs, empty readmes). In these cases:
+
+ **Manual curation recommended.** Use the tree output and domain knowledge:
+
+ ```bash
+ # Get tree structure only (fast)
+ python scripts/github_index.py owner/repo --skip-fetch -o skeleton.md
+ # Then manually enhance descriptions based on domain knowledge
+ ```
+
+ For code-heavy repos with embedded apps:
+ - Directory names encode purpose: `acc_wav_gen` → "ACC waveform generation"
+ - Peripheral acronyms map to functions: AFEC=ADC, MCAN=CAN, TWIHS=I2C
+ - Operation modes: blocking, interrupt, dma, polled
+
+ ## Output Format
+
```markdown
- # {Repo Name} - Content Index
+ # {Repo} - Content Index
**Repository:** {url}
**Branch:** `{branch}`
## Retrieval Method
{API curl commands}
---
## {Category}
| Description | Path |
|-------------|------|
- | {What this covers} | `{path/to/file.md}` |
-
- ## Key Themes
- {Concepts that reduce need to fetch}
+ | {What this covers} | `{path/file.md}` |
```
- Lead rows with **description** (relevance matching), path second (retrieval key).
+ Description column leads (relevance matching), path follows (retrieval key).
## API Access
- Fetch tree (enumerate files):
+ Enumerate files:
```bash
curl -sL "https://api.github.com/repos/OWNER/REPO/git/trees/BRANCH?recursive=1"
```
- Fetch file content:
+ Fetch content:
```bash
curl -s "https://api.github.com/repos/OWNER/REPO/contents/PATH?ref=BRANCH" \
-H "Accept: application/vnd.github+json" | \
python3 -c "import sys,json,base64; print(base64.b64decode(json.load(sys.stdin)['content']).decode())"
```
- Include retrieval instructions in generated indexes.
-
- ## Manual Index Creation
-
- When script doesn't fit or for curated indexes:
-
- 1. Get tree, filter to content files (.md, .qmd, .ipynb)
- 2. Fetch files, extract frontmatter (title/description from YAML between `---` markers)
- 3. Group by directory structure or theme
- 4. Write tables: Description | Path
- 5. Add Key Themes section for frequent queries
+ ## Network
- ## Quality Standards
+ Both scripts download a repo tarball (single HTTP request, no per-file rate limits) then process files locally. Allowlist: `api.github.com` (tarball redirects via this endpoint)
- **Descriptions**: Lead with problem/question addressed. Include searchable terms. Under 100 chars.
+ ## Related Skills
- **Categories**: Semantic groupings, not flat lists. Subcategories help navigation.
+ - `accessing-github-repos` - Private repos, PAT setup, tarball download
+ - `mapping-codebases` - Detailed code structure (methods, imports, line numbers)
- **Themes**: Extract after reviewing content. Summarize frequently-asked concepts.
+ ## Condensed Format (pk_index.py)
- ## Network Requirements
+ For token-constrained project knowledge, use the condensed script:
- Project allowlist: `api.github.com` (required), `raw.githubusercontent.com` (optional).
+ ```bash
+ python scripts/pk_index.py owner/repo -o repo_pk.md
+ ```
- ## Related Skills
+ Produces ~80% smaller output:
+ - Single line per file: `path` — description
+ - Symbols only (no signatures)
+ - 15 files max per category
+ - No retrieval instructions section
- - `accessing-github-repos` - Private repos, PAT setup
- - `mapping-codebases` - Code structure (functions/classes) vs documentation
+ Ideal when adding multiple repo indexes to project knowledge.