--- name: obsidian-ingest description: > Universal import pipeline for an Obsidian vault. Handles ANY source: social-media posts, Telegram HTML exports, ChatGPT JSON, voice-note transcripts, WhatsApp exports, Apple Notes, Notion dumps, pasted text, articles β€” anything that needs to become well-linked atomic notes with proper structure, frontmatter, provenance, dedup against existing notes, and a RAG reindex at the end. Trigger on "/obsidian-ingest", "import this into the vault", "file this". license: MIT --- # Obsidian Ingest > πŸ§’ **Always, in every reply and progress log to the operator:** end with a child-simple "In plain words" recap in their language (plain words, no jargon) β€” their standing request. Scope: reports TO the operator only, never inside the vault notes you generate. See memory `eli5-always` / global `CLAUDE.md`. This skill captures how the operator wants raw text turned into a clean, linked knowledge base. He is a perfectionist and values **thorough extraction over speed** β€” the goal is not to dump files but to transform raw material into atomic, well-linked notes that compound in value over time. Read `references/vault-conventions.md` for the exact folder layout, naming, tag canon, and frontmatter templates. Read `references/pipeline.md` for the big-batch script pipeline. Read `references/source-adapters.md` for per-source parse specs (Facebook, Telegram, ChatGPT, WhatsApp, etc.). Read `references/second-brain-layer.md` for the distillation & retrieval layer (concept-synthesis "what I think about X" notes, prediction ledger, semantic search) β€” build/refresh after a big import so the vault stays a thinking partner, not just an archive. This SKILL.md is the decision layer; the references are the mechanics. ## Step zero: establish ownership Before processing ANY material, establish provenance. For **known source types** (see table below) the provenance is already confirmed β€” skip the question and proceed. For **unknown material** (a file the operator just dropped, a pasted note, an unfamiliar export), always ask: *"Whose material is this β€” your own thinking, a conversation with others, or external content?"* ### Known-provenance sources (no need to ask) | Source | origin | authored_by | Extra tags | |---|---|---|---| | Facebook posts export (`posts.md`) | `anton` | `human` | `facebook-diary, anton-original` | | Telegram voice archive (content team) | `anton` | `human` (voice) / `hybrid` (summaries) | `anton-original` | | the operator's own voice notes (m4a β†’ transcribed) | `anton` | `human` | `anton-original` | | ChatGPT conversations (the operator's chats) | `mixed` | `hybrid` | β€” | | Nexus distilled (02-Decisions/03-Insights/05-Resources) | `anton` | `claude-cowork` | β€” | | External podcasts / YouTube transcripts | `external` | `ai` | β€” | | Dialogue subfolders (dialogues_work_acct_b_*) | `mixed` | `hybrid` | β€” | ### Unknown material β€” always ask Record the answer as `origin` in frontmatter: - `origin: anton` β€” the operator's own ideas β†’ also tag `#anton-original` - `origin: mixed` β€” conversations where the operator and others speak - `origin: external` β€” others' content collected for reference `origin` (whose *ideas*) is a different axis from `authored_by` (who *wrote* the text β€” human vs AI). Both are mandatory. **Transcription β‰  authorship** β€” if the operator spoke the words, it's `authored_by: human` + `origin: anton` even when a tool did voiceβ†’text. > **Source NOT the operator's? Name the real author β€” never hang it on them (standing demand, 2026-06-08).** If evidence shows a source was authored by anyone other than the operator β€” a blogger, channel owner, journalist, podcast guest, a forwarded/saved article β€” **even when the operator collected or forwarded it** β€” set `origin: external` and record the real creator in an `author:` field; **never** `origin: anton` / `#anton-original`. Tells: self-identification, cross-post handles, third-person refs to the channel's own figures, and the **absence of the subject's own projects across the years they were active**. When unsure, ASK β€” the owner's words: "do not pin on me what I did NOT write". Canonical: memory [[provenance-attribute-real-author]]. > > **Author β‰  processing tool (2026-06-09).** `authored_by` = WHO created the text (human / a specific AI). The import/enrich/transcribe/salvage TOOL goes in a SEPARATE field β€” `processed_by:` (or `transcribed_by:`/`summarized_by:`). NEVER let a pipeline name (`claude-cowork-*`) overwrite `authored_by` β€” mark the real author *and* keep the tool visible for traceability ("who is on the hook when something breaks"). > > **Name the SPECIFIC AI (2026-06-09).** If the author is an AI, always name it concretely in `ai_author:` β€” **ChatGPT / Claude / Gemini + version** β€” never bare `ai`. ChatGPT-distilled research β†’ `authored_by: ai` + `ai_author: ChatGPT` + `origin: mixed` (+ `processed_by:` the distiller), not `origin: anton`. ## First decision: small batch or big batch? - **Small (≀ ~20 items / a few pasted notes):** handle inline, by hand, full manual control over each note. Go to "Inline processing". - **Big (50+ items, a chat export, hundreds–thousands of messages):** use the checkpointed script pipeline. Hand-writing does not scale and drifts. Go to "Batch pipeline". When unsure, count the input. A WhatsApp/Telegram export or a folder of transcripts is always "big". ## Storage decision β€” which layer? (set 2026-06-07 β€” decide BEFORE generating, EVERY import) Don't reflexively turn everything into markdown files. At scale that lags Obsidian (it slows past ~50–100k files) and makes corpus-wide questions token-expensive. Route each kind of data to its STORE β€” the 3-layer model (full version: memory `vault-data-architecture` β†’ **SQLite facts + RAG meaning + markdown thinking**): - **High-volume structured records** (contacts, leads, transactions, rows, raw message text) β†’ **SQLite**: entities in `leads.db`; raw **dialogue/message text β†’ a separate `*_messages.db` + FTS5** (full-text search, 0 tokens, ms). One DB file beats 10⁴ note files. - **Curated, human-read atomic notes** (the substantial relationships / concepts / insights worth *thinking* with) β†’ **markdown** in the vault (the graph + reading layer) β€” generated FROM the data, for the top slice only. - **Semantic recall** ("what do I think about X") β†’ **RAG** (`brain_ask.py`) over the curated markdown, NOT the raw bulk; reindex after (`brain_embed_update.py`). **Rule of thumb:** a source yielding >~2–3k items β†’ the BULK lands in SQLite, only the curated top-slice becomes markdown. *Personal-DMs precedent (2026):* 80,791 contacts + 1.47M messages + 781k Airtable rows β†’ `leads.db` (contacts/leads/airtable, joined by tg_id/handle) + `dm_messages.db` (message FTS); only **β‰₯50-msg** relationships became `person-*` notes; **<5-msg** pings stayed index-only (a CSV registry), never tens of thousands of junk files. The **token-economy law** governs every LLM step (cheapest tool first β€” `CLAUDE.md` / `operating-agreement`). ## The non-negotiable rules (apply to both paths) These are the operator's standing instructions. Don't re-ask them each time. 00. **RE-IMPORT GATE β€” sha256 the source BEFORE anything else (Rule 0β€², set 2026-06-13 after a NO-OP catch saved a multi-hour pipeline run).** The very first action of *any* "import this corpus / add this dump / process these notes" task β€” **before Rule 0, before reading the file, before spawning a sub-agent** β€” is to ask *"did I already import this exact byte-stream?"* Run the gate: `python "$USERPROFILE/.claude/skills/obsidian-ingest/scripts/precheck_corpus.py" ""` and read **line 1** (tab-separated `DECISION\thash\tbytes\tmatch-path`): - **`NO_OP`** β†’ the source is byte-identical to an archived original under `_originals\`. **STOP. Do NOT re-run the pipeline.** Report to the operator where the existing import lives + that it's unchanged. Re-running would clear+regenerate identical files and burn tokens on already-done work (violates the token-economy law β€” cheap tool answered first). - **`CHANGED`** β†’ same basename in `_originals\` but different bytes (the operator re-exported a grown source). Proceed as an **incremental** re-import: Rule 0 archives THIS version as a new snapshot, diff vs the prior snapshot, run the idempotent pipeline on the delta only. (This is the `telegram-reimport` path.) - **`NEW`** β†’ no prior archive. Proceed with a normal first-time ingest (Rule 0 next). - **`MISSING` / `DIR_MODE_NOT_IMPLEMENTED`** β†’ fix the path, or for a directory hash each top-level item separately. Why a script and not just "remember to check": a rule that lives only in memory gets skipped under load β€” the gate is a deterministic Step 0 that *can't* be forgotten because it sits at the top of this list. Provenance: memory [[crypto-essays-reimport-idempotent]]. Verified live across all 4 decision paths on 2026-06-13. 0. **Preserve the original β€” do this FIRST, every import (Rule 0).** Before parsing or transforming ANYTHING, archive the raw source **verbatim** to the permanent originals store and never delete it: `python $IMPORTS_ROOT/archive_original.py "" --source --label ""`. It copies the file/folder to `$OBSIDIAN_ROOT/_originals/\__\` with a sha256 manifest, copy-only (the source is never moved or edited), idempotent (re-archiving identical content is skipped), and integrity-checked. This holds for **any** source β€” a Telegram/WhatsApp export, a CSV/Takeout dump, pasted text (write it to a `.txt` first, then archive that), a voice `.m4a`. The point: if we later delete or restructure derived notes, the untouched upstream source is still there to lean on. This is **distinct** from `vault_backup.py` (which git-commits the *vault*, i.e. the derived layer) and from rule 5 (one raw copy *inside a note*) β€” those protect the processed layer; Rule 0 protects the source artifact itself. Originals under `_originals\` are **never** auto-deleted. 1. **Decompose, don't just file.** One raw conversation usually becomes: a master/source note (raw text, one copy) **plus** extracted `concept-*` and `insight-*` notes for anything genuinely reusable. Filing without extraction wastes the material. 2. **One transcript = one primary concept.** Every transcript/long note should belong to at least one concept. Link it to an **existing** concept when one fits; create a new concept only when nothing fits. **Never proliferate near-duplicate concepts** β€” prefer linking over creating. **Weight the create-vs-link decision by `origin`:** for `#anton-original` material (the operator's own thinking β€” voice/diary/reflections, their side of dialogues, their AI prompts) actively look for an *emergent new concept* before defaulting to the nearest existing one β€” their authentic ideas are exactly where new concepts are born, so don't bury a genuinely new one inside a too-broad existing concept. For `origin: external` (podcasts, articles, others' lectures) and operational ledgers (purchases, CRM, tasks) stay strictly link-first β€” a new concept only for a genuinely new topic they care about. 3. **Provenance is mandatory β€” two independent axes.** Every note records both: - `authored_by: human | ai | hybrid` β€” who *wrote* the text. Voice transcripts & the operator's own writing = `human`; GPT/LLM summaries = `ai`; assistant-drafted/translated = `hybrid`; pre-2023 = `human`. - `origin: anton | mixed | external` β€” *whose ideas* they are (established in Step zero by **asking**, never guessing). `origin: anton` also gets the `#anton-original` tag. **Transcription is not authorship.** If the operator spoke or wrote the content, it stays `authored_by: human` + `origin: anton` **even when an assistant or a tool did the voiceβ†’text conversion**. The transcriber/translator goes in `transcribed_by`, never in `authored_by`/`origin`. Someone counts as author/origin only if the *ideas and words* are theirs β€” not if they merely typed up the operator's voice. These protect the operator's authentic voice from being blurred by AI text **and** from being confused with other people's thinking β€” keeping their own thoughts separable is their explicit, top priority. 4. **Auto-add new tags β€” never ask.** Reuse canonical tags from the vault first (see references). When the material needs a tag that doesn't exist, just add it. Don't pause to request approval. 5. **Raw text lives in exactly one copy.** Never store the same raw text twice. Dedup by **body content-hash** (exact dups only β€” near-dups with the same opening but different length are *different content*, never auto-delete them). If a curated note needs raw that lives elsewhere, link to it β€” don't re-embed. **Cross-source duplicates** are common: the same voice note may arrive both as a pasted note and inside a later chat export. When that happens, keep the systematic archive as the single raw copy and convert the earlier note into an **overlay** (frontmatter + summary + concept links that point to the raw via `[[links]]`), and record it in the MOC. 6. **Always cross-link, always maintain a Cross-MOC.** The vault must become a graph, not a pile. Wire links in every direction: noteβ†’concept (`concept:` field), conceptβ†’note (backlink section), **conceptβ†’concept** (a `## Related concepts` section built from a relatedness pass β€” see pipeline.md "Concept interlinking"), and noteβ†’source / sourceβ†’derived. Each import gets/updates a `_-MOC.md` index that connects to the rest of the vault. Short `[[wikilinks]]` only (basename), never long `[[../../path.md]]` β€” short links survive reorganization (and Obsidian won't even resolve `..` relative wikilinks). 7. **Never edit the raw text itself.** Preserve wording, profanity, repetition, style. Strip only transcription-tool footers (e.g. "Transcribed by whisper AI") and move that metadata into frontmatter (`transcribed_by`, `summarized_by`). 8. **Finish with a report + link-integrity check.** End every run by validating that all `[[links]]` resolve (0 broken) and giving the operator a diff-style summary: files created, tags added, concepts mapped, duplicates resolved, anything flagged. **When the import contains `#anton-original` material, add a `candidate new concepts` line** β€” which emergent concepts you considered, which you created (nothing fit), and which you folded into an existing `[[concept-…]]` and why β€” so new-concept calls on the operator's own thinking are visible for them to confirm, never silent. 9. **Capture rules into the Bible β€” proactively.** Whenever the operator explicitly asks to record a rule ("add a rule", "this goes into the Bible"), OR a durable rule/policy is clearly needed, add it to the Bible as a `reglament-*` note β€” don't just acknowledge it. Append the rule to `$IMPORTS_ROOT/manual_rules.json` and rebuild with `build_rules2.py` (it merges manual rules, regenerates `_Operations-Bible-MOC.md`, and is guarded so it won't clobber the fleshed `concept-bible-*` sub-concepts). Ground the rule in a real message when possible; set provenance (the owner's directive β†’ `origin: anton` + `#anton-original`; team SOP β†’ `mixed`), assign a `theme` so it links the right `concept-bible-`, and update the MOC. Mechanics: the "Assistants-Ops" adapter in `references/source-adapters.md`. 10. **Prioritize by value Γ— speed β€” quick wins first; never let slow-but-valuable block fast-and-valuable.** When a source β€” or a multi-source / multi-table job β€” offers more than can be done at once, order the work by the operator's triage (their explicit standing rule, 2026-06-06): 1. **High value + fast β†’ ALWAYS first.** Take the most important/valuable material that is *also* quick to extract. **If something is valuable but slow, do NOT do it in this pass** β€” it must never delay the quick high-value wins. 2. **High value + slow β†’ next (dig deeper).** Only after the quick wins are in, come back for the valuable material that takes longer to scrape / parse / collect / build. 3. **Low value + very slow β†’ last, or skip.** Not-very-valuable material that is expensive to gather comes dead last; do it only if it's genuinely worth the cost. (Fourth quadrant β€” *low value + fast* β€” do opportunistically only if nearly free, else skip.) This also fixes **ordering inside a big import**: front-load the small, high-signal tables/sources; leave giant low-signal dumps for the tail (e.g. pull a CRM's Investors/Contacts before a 250k-row scraped-members table). Always **state out loud what you are deferring or skipping** so nothing is silently dropped β€” deferred β‰  forgotten. 11. **Long-running task β†’ periodic liveness check (~every 20 min).** On any task that runs for many minutes β€” a background export/import, a big batched GPU job, a long browser / computer-use session β€” do NOT fire-and-forget. At a regular cadence (the operator's rule: ~once every 20 minutes, or tied to natural checkpoints) verify everything is **still actually working**, not just nominally "running": - **Still making PROGRESS?** "Running" β‰  "advancing". Read the job's output / manifest / checkpoint and confirm counts are climbing, files are being written, the phase is moving β€” not stalled on a hang or an endless 429/retry loop. - **Tools & connections still ALIVE?** MCP servers (Telegram, Chrome, computer-use) and browser renderers can silently disconnect or freeze mid-run (both happened this session). Re-check whatever the task depends on. - **Any silent error?** backoff that isn't recovering, disk/auth failure, a crashed worker. If something broke: tell the operator, diagnose, and **recover from the last checkpoint** (restart the job, reconnect / re-auth the tool) β€” never assume it's fine just because it was an hour ago. Self-bounded jobs (hard per-request timeouts + the harness auto-notifying on completion/crash) need less babysitting; jobs that *can* hang silently need the active 20-min check. Mechanism when idle: a long (~1200 s) scheduled re-check, not tight polling. Complements the pre-flight MCP health-check rule. 12. **Finish the job + proactive status + a final report (the owner's standing demand, 2026-06-07 β€” set after I went silent during a multi-hour reindex and leaned on them to ping me).** Liveness (rule 11) is not enough; three more non-negotiables: - **Drive it to DONE β€” never park the work on the operator.** Never end a turn with "ping me when it's ready" except when genuinely blocked on (a) HIS decision or (b) a hard session/usage limit. A background job is NOT a stop: on completion the harness fires a `` and re-invokes you β†’ continue the downstream steps (validate β†’ move β†’ report) THEN. On a hard limit: write the exact resume command and resume yourself the moment it clears. - **Never go silent.** On any job longer than a couple minutes: give an ETA up front and a one-line update at each phase boundary / each return from background. Silence between "launched" and "done" is the failure. - **A final report ALWAYS** (generalizes rule 8 from imports to *every* task): what changed (numbers) Β· self-checks run (0 broken links, parser counts, `pid in == pid out` β€” **measure, don't trust subagents' "done N" or your own assumption**) Β· what's deferred Β· next step Β· πŸ§’-recap. A task isn't done until the report is written. 13. **No-orphan rule (the operator's STANDING demand, 2026-06-13 β€” set after he found a high-value claude-ai artifact sitting with 0 backlinks).** Every note added to the vault MUST end up with β‰₯1 **incoming** wikilink from an existing layer (concept / MOC / decision / parent note) β€” forward-links alone are not enough, the note must be discoverable by `grep` from the OTHER side. Last mandatory step of EVERY ingest, inline or batch: - **Orphan-check** for each new basename: `grep -lE '\[\[(\||\])' $OBSIDIAN_VAULT` β†’ must return β‰₯1 file *besides* the note itself. Batch: one pass over the basename list (deterministic, 0 LLM tokens β€” see [[vault-data-architecture]]). - **If 0 incoming** β†’ wire it in: - forward-linked to concepts β†’ add a one-line back-mention in the relevant `06-Concepts/concept-*.md` (under "See also" / "Related notes"). - no covering concept β†’ hang it under the nearest `_*-MOC.md`. - architectural / decision-shaped β†’ entry in `02-Decisions/`. - **Report line is mandatory:** `orphan-check: N created Β· N back-linked Β· 0 orphaned`. `>0 orphaned` means ⚠️ unfinished, not "done". - **Exemptions (narrow):** `_originals/` (raw archive, [[preserve-originals-rule]]), `_drafts/`, `_imports/staging/`. Live vault notes never exempt. - Smell test before claiming done: "a year from now the owner greps by person / concept / topic β€” will this note surface?" No β†’ finish the linking. ## Inline processing (small batch) 1. **Fingerprint first.** Skim the vault's current tag canon and existing `concept-*`/`person-*` names so new notes match (don't invent parallel vocabulary). See `references/vault-conventions.md`. 2. **Read each note fully** β€” not the first lines. Extract: type (concept/person/project/insight/conversation), entities mentioned, language, date, one-sentence thesis. 3. **Classify β†’ folder** using the decision tree in references. 4. **Write the note**: canonical filename, full frontmatter (incl. `authored_by`), raw body untouched, `## See Also` with cross-links. 5. **Extract** concepts/insights worth their own note; link them both ways. For `#anton-original` notes, actively check for an *emergent new concept* (rule 2), not just the nearest existing fit. 6. **Map to a concept** (rule 2) and **update/!create the relevant MOC** (rule 6). 7. **Report** (rule 8). ## Batch pipeline (big batch) Run the checkpointed pipeline so a mid-run failure never half-corrupts the vault. Each phase writes a checkpoint to `$IMPORTS_ROOT/` so any later phase can re-run without redoing earlier work. Full details and the bundled scripts are in `references/pipeline.md`. Phases: **parse β†’ JSONL** Β· **triage + sessionize** Β· **generate β†’ STAGING** Β· **validate (0 broken links) in staging** Β· **move staging β†’ vault** Β· **dedup (content-hash)** Β· **provenance backfill** Β· **build/refresh Cross-MOC** Β· **concept mapping (1 note = 1 concept)** Β· **report**. Critical: generate into `$IMPORTS_ROOT/staging/` (mirroring the vault tree), validate there, and only then `cp -r` into the live vault. Never generate hundreds of files directly into the vault unverified. ## Compute: ALWAYS prefer GPU, ALWAYS detect first The operator's standing rule: for any heavy/accelerable compute (embeddings, model inference, large tensor work), **use the GPU, not the CPU** β€” and because **he runs me on different machines**, never assume or hardcode a specific GPU. **Detect first, every time:** run `python $IMPORTS_ROOT/gpu_check.py` (or `nvidia-smi` + `torch.cuda.is_available()`). If a GPU exists but torch is the `+cpu` build, install the CUDA wheel `gpu_check.py` prints, then proceed on GPU. Falls back to CPU only when there's genuinely no accelerator (then prefer incremental/batched work). I/O-bound vault scripts (file walks, regex) gain nothing from GPU β€” this rule is for ML/tensor work. **Never launch a long GPU job twice.** A GPU-OOM can leave the python process ALIVE as a zombie holding ~1.4 GB VRAM; relaunching "because it looks dead" stacks zombies until nothing loads (this actually happened β€” cost hours). Verify death via `nvidia-smi --query-compute-apps=pid`, not `wmic`/`tasklist` name-match (unreliable). The brain_* scripts now self-protect (single-instance lock, auto-kill stale, fp16, `--wait-gpu`, checkpoint-resume) β€” full **REINDEX PROTOCOL** + tooling in `references/second-brain-layer.md`. `gpu_check.py --kill` clears zombies. ## Windows / Cyrillic gotchas (learned the hard way) - **Python `print()` of Cyrillic crashes** on Windows (cp1252 stdout). Write results to UTF-8 files and Read them; keep stdout ASCII-only (counts, slugs). - **Don't `rm -rf` inside `%VAULT_ROOT%\`** β€” Obsidian and the Windows indexer hold file handles ("Device or resource busy"). Delete with `find ... -delete`, or generate into a fresh directory. - **Filenames: transliterate Cyrillic β†’ latin kebab**, date-prefix (`YYYY-MM-DD-slug.md`) so they sort chronologically; de-dup collisions with a numeric suffix. - **Escaped pipes in tables:** inside a markdown table, write `[[target\|alias]]` (escaped pipe) so the table doesn't break; Obsidian still resolves it. ## What success looks like A reader (or future the operator) lands on the Cross-MOC, navigates by month/theme/person, clicks into an atomic note, sees its provenance and its concept, and follows `[[links]]` to related ideas β€” with zero broken links and zero duplicated raw text. The vault gets *more* navigable with every import, not just bigger. --- ## About & contact Built and battle-tested at **Palo Alto AI Research Lab** β€” a fleet of Claude Code machines running 24/7 as a second brain and synthetic cofounder. Every skill here survived real production use before publication. - πŸ“¦ All 101 skills: https://github.com/tonydzi/second-brain-starter-kit - πŸ‘€ Author: **Anton Dziatkovskii** β€” Telegram [@tonydzi](https://t.me/tonydzi) Β· WhatsApp [+1 341 222 9178](https://wa.me/13412229178) Β· X [@Tony_Stef_](https://x.com/Tony_Stef_) - πŸ§ͺ **Engineers: want to test-drive this setup?** Message me β€” I hand out free starter seeds to engineers who test and report back. Custom skill requests welcome.