obsidian-wiki-builder · diff
git:20260415.3dad459 to git:20260417.0d3eb8c
36 added, 1 removed. Audit A to A.
---
name: obsidian-wiki-builder
description: "Transforms raw source files registered in wiki_sources.json into Karpathy-style LLM wiki nodes inside the wiki-root. Generates concept pages, cluster pages, index, and table of contents. Use when building or rebuilding the wiki from raw content."
allowed-tools: Bash, Read, Write
---
## Dependencies
Requires `pyyaml` and Python 3.8+. Also requires `rlm-factory` plugin installed.
```bash
pip install -r requirements.txt
```
---
# Obsidian Wiki Builder
**Status:** Active
**Author:** Richard Fremmerlid
**Domain:** Obsidian Wiki Engine
## Purpose
Transforms raw source documents (registered via `wiki_sources.json`) into
**Karpathy-style wiki nodes** inside the designated `wiki_root`. Each concept
gets its own `{concept}.md` node with wikilinks, cross-references, and an
RLM summary layer.
## Output Layout (Rigid)
```
{wiki_root}/
wiki/
_index.md ← master concept index
_toc.md ← table of contents
_{cluster}.md ← per-topic cluster page
{concept}.md ← individual wiki node
rlm/
{concept}/
summary.md ← 1-5 sentence distilled summary
bullets.md ← key idea bullets
deep.md ← full multi-pass distillation
meta/
wiki_sources.json ← raw source registry (created by wiki-init)
config.yaml ← wiki settings
agent-memory.json ← stale file tracking
```
+ ## Build Pipeline
+
+ `wiki_builder.py` runs a 3-step pipeline internally:
+
+ ```
+ ingest.py → concept_extractor.py → wiki node formatting
+ ```
+
+ 1. **ingest.py**: Reads raw source files, normalizes content, hashes for staleness detection
+ 2. **concept_extractor.py**: Groups records by concept slug, **merges multi-source records**
+ into one authoritative node (the Karpathy "compile" step), improves cluster assignment
+ via keyword extraction
+ 3. **wiki node formatting**: Renders merged records into Karpathy-format `.md` files
+ with wikilinks, cluster pages, `_index.md`, and `_toc.md`
+
+ ## Multi-Source Concept Merging
+
+ When two source files (e.g. `arch-docs/auth.md` and `research/oauth.md`) produce
+ the same concept slug, they are merged into **one wiki node** that:
+ - Combines content from all sources with attribution headers
+ - Lists all source files in `source_files` frontmatter
+ - Re-derives the cluster from combined content keywords
+ - Sets `multi_source: true` for identification
+
+ This implements the core Karpathy "compile" metaphor: N raw files → M concept nodes (M ≤ N).
+
## Usage
### Build wiki nodes from all registered sources
```bash
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root
```
### Build from one named source only
```bash
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root --source arch-docs
```
+ ### Use shared .agent/learning/ RLM cache for existing summaries
+ ```bash
+ python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root \
+ --rlm-cache-dir /path/to/project/.agent/learning/rlm_wiki_cache
+ ```
+
### Dry run (plan without writing)
```bash
python ./scripts/wiki_builder.py --wiki-root /path/to/wiki-root --dry-run
```
## Karpathy Wiki Node Format
Each concept node uses this structure:
```markdown
---
concept: {concept_name}
source: {source_label}
source_file: {relative_path}
wiki_root: {wiki_root}
generated_at: {timestamp}
cluster: {cluster_name}
---
# {Concept Name}
{1-sentence RLM summary}
## Key Ideas
- {bullet}
- {bullet}
## Details
{full content, truncated to 800 tokens if needed}
## See Also
- [[{related_concept_1}]]
- [[{related_concept_2}]]
## Raw Source
- `{source_label}` → `{source_file}`
```
- ## Source Registry Format (`wiki_sources.json`)
+ ## Source Registry Format (`rlm_wiki_raw_sources_manifest.json`)
+
+ Canonical location: `.agent/learning/rlm_wiki_raw_sources_manifest.json`
+ (mirrors rlm-factory's `.agent/learning/rlm_profiles.json` convention).
Managed by `raw_manifest.py` via the Guided Discovery agent during `/wiki-init`:
```json
{
"namespace": "my-project",
"wiki_root": "/Users/me/vault/wiki-root",
"sources": {
"daily-notes": {
"path": "/Users/me/vault/notes",
"label": "daily-notes",
"extensions": [".md"],
"excludes": ["_archive", "*.tmp"],
"description": "Daily journal and quick capture notes"
},
"arch-docs": {
"path": "/Users/me/docs/architecture",
"label": "arch-docs",
"extensions": [".md", ".txt"],
"excludes": [],
"description": "Architecture decision records and design docs"
}
},
"global_excludes": ["_archive", "*.tmp", "__pycache__"]
}
```
## When to Use
- After `/wiki-init` completes source registration
- After adding new raw source documents
- After running `/wiki-distill` to inject fresh RLM summaries
- When the wiki index is stale (`agent-memory.json` shows diffs)
## Related Scripts
- `raw_manifest.py` — `WikiSourceConfig` loader (mirrors `RLMConfig`)
- `ingest.py` — raw file parsing and normalization
- `wiki_builder.py` — Karpathy node formatter and linker
- `distill_wiki.py` — cheap-model distillation wrapper