myco:myco-sh-docs-pipeline git:20260613.ad8665dA

Immutable. This exact content is served forever at /api/v1/blob/3088fadbf924e3f2.

---
name: myco:myco-sh-docs-pipeline
description: |
  Apply this skill when working on the myco.sh documentation site — including
  adding or updating guides, modifying the build pipeline, maintaining the
  GitHub Actions Pages deploy workflow, auditing SEO/AI-crawler discovery, or
  fixing rendering bugs in the static HTML output — even if the user doesn't
  explicitly ask about build infrastructure or deployment. Covers four
  interlocking procedures: (1) dual-surface architecture — keeping raw .md
  files for AI crawlers while generating parallel .html in docs/_site via
  docs/build.mjs; (2) build pipeline configuration using markdown-it + Shiki
  with the linkify fuzzyLink guard in docs/lib/render.mjs; (3) discovery layer
  maintenance — robots.txt, sitemap.xml, llms.txt, JSON-LD, and canonical
  tags; and (4) GitHub Pages workflow — matching upload-pages-artifact and
  deploy-pages version pairs to prevent silent deploy failures.
managed_by: myco
user-invocable: true
allowed-tools: Read, Edit, Write, Bash, Grep, Glob
---

# myco.sh Docs Site — Dual-Surface Build Pipeline, Pages Deployment, and Content Safety

The myco.sh docs site serves **two distinct audiences from one content base**: raw `.md` files for AI crawlers and LLM contexts (indexed via `llms.txt`, `sitemap.xml`, and `robots.txt`), and statically-rendered HTML pages for human readers. The build step (`npm run build` inside `docs/`) compiles everything into `docs/_site/` — both surfaces live there: copied raw `.md` files and generated `.html` files. That `_site/` directory is what gets deployed to GitHub Pages.

Changes to any of these areas are coupled — a new guide needs an entry in `docs/lib/nav.mjs` (the sidebar manifest), and a link in `llms.txt`. The `sitemap.xml` is auto-generated from the nav manifest. Understand the full picture before editing any one part.

---

## Prerequisites

- Node.js available; build runs as `npm run build` from inside the `docs/` directory (executes `docs/build.mjs`).
- Familiarity with `docs/` directory structure:
  - `docs/*.md` — source markdown guides (do not move or rename without updating `llms.txt` and the nav manifest)
  - `docs/lib/nav.mjs` — **sidebar nav manifest and single source of truth for guide ordering**; add new guides here
  - `docs/build.mjs` — build orchestrator; reads nav manifest, renders HTML, copies static files, generates sitemap
  - `docs/lib/render.mjs` — markdown-it renderer (Shiki theme + linkify guard)
  - `docs/lib/template.mjs` — HTML template with left-sidebar nav
  - `docs/lib/links.mjs` — link rewriter (`.md` → clean URL; out-of-tree → GitHub blob)
  - `docs/_site/` — build output directory (deployed to Pages); not committed to git
  - `.github/workflows/docs.yml` — Pages deploy on push to `main`
- Read `docs/lib/nav.mjs` before adding guides — guides not listed in the `NAV` array get no inbound navigation links and their source files are not picked up by the build.

---

## Procedure 1: Dual-Surface Architecture — Adding or Restructuring Guides

The fundamental constraint: **raw `.md` source files must stay at their paths under `docs/`**. The build copies them verbatim into `docs/_site/`, preserving the AI-crawler surface. Deleting or moving `.md` source files breaks `llms.txt` links and the LLM-facing surface even if the human-facing HTML is intact.

**How the dual surface works:**

```
docs/quickstart.md            ← Source (not deployed directly)
                                 ↓ build step
docs/_site/quickstart.md      ← AI crawlers, llms.txt links, sitemap (raw Markdown copy)
docs/_site/quickstart.html    ← Human readers (generated by build step)
/quickstart                   ← Clean URL served by GitHub Pages (maps to .html)
```

**Adding a new guide:**

1. Write the guide as `docs/<name>.md`. Follow the `write-myco-user-documentation` skill for content style.
2. Open `docs/lib/nav.mjs` and add a `{ slug: '<name>', title: '<Display Title>' }` entry to the appropriate group in the `NAV` array. The `NAV` array is an ordered list of `{ group, items }` objects; each item uses `slug` (filename without `.md`) and `title` (sidebar label). Omitting this step means the build ignores the guide entirely — no HTML generated, no sitemap entry, no inbound nav links.
3. Add a link to `llms.txt` under the appropriate section so LLM crawlers discover it. Use the `.md` URL (e.g., `https://myco.sh/<name>.md`), not the `.html` URL.
4. Run `npm run build` from inside `docs/` and verify `docs/_site/<name>.html` is generated without errors.
5. Smoke-test with a Pages-faithful local server (see Procedure 4) — confirm internal links resolve and no bare filenames were linkified (see Procedure 2).

**Note — sitemap is auto-generated.** The build script generates `docs/_site/sitemap.xml` from the `NAV` manifest on every build using today's date (or `DOCS_BUILD_DATE` env var). Do not manually edit a static `sitemap.xml` — it will be overwritten on the next build.

**Link rewriting rules (enforced by `docs/lib/links.mjs` via `docs/build.mjs`):**

| Link type in .md source | Rewritten to in .html output |
|---|---|
| `[text](other-guide.md)` | `/other-guide` (clean URL) |
| `[text](../../AGENTS.md)` | GitHub blob URL (out-of-tree) |
| `https://example.com` | Unchanged |

Do not write `.html` links in markdown source — the renderer handles the conversion.

---

## Procedure 2: Build Pipeline Configuration (`docs/lib/render.mjs`)

The renderer uses `markdown-it` with Shiki syntax highlighting. There are two non-obvious configuration decisions that must be preserved when modifying `render.mjs`.

**Shiki theme:** The dark theme is applied at build time (no client-side JS required). If you change the theme, rebuild and visually verify code blocks render correctly — Shiki theme names are not validated at import time.

**The `linkify` fuzzyLink guard (critical — do not remove):**

`markdown-it` with `linkify: true` auto-linkifies bare strings that look like domains. Because `.md`, `.sh`, `.ts`, `.io` are real ccTLDs, filenames like `CLAUDE.md`, `AGENTS.md`, and `install.sh` get converted to broken external links (`http://CLAUDE.md`) in rendered prose. Technical docs mention filenames constantly, so this produces broken links throughout the site.

The fix is already in place in `docs/lib/render.mjs` at lines 9–13 — **do not remove or change these lines**:

```js
const md = MarkdownIt({ html: true, linkify: true, typographer: true });
// Required: disables fuzzy (schemeless) linkification while keeping
// explicit-scheme URLs (https://...) auto-linked.
md.linkify.set({ fuzzyLink: false, fuzzyEmail: false, fuzzyIP: false });
```

This preserves `https://example.com` auto-linking while suppressing `CLAUDE.md` → `http://CLAUDE.md` conversion. Removing `fuzzyLink: false` re-introduces the bug across all rendered guides.

This bug is **not caught by unit tests on the render function** — it only surfaces when real prose with filenames is rendered and crawled. Always smoke-test with the link crawler (see Procedure 4) after render changes.

---

## Procedure 3: Discovery Layer Maintenance

The discovery layer (robots.txt / sitemap.xml / llms.txt / JSON-LD) is independently important from the HTML rendering. It must be kept current whenever guides are added, renamed, or removed.

**Component inventory and update triggers:**

| File | Purpose | Update when |
|---|---|---|
| `docs/robots.txt` | Explicit allow rules for crawlers + sitemap pointer | Rarely — only if adding new crawler allow rules |
| `docs/_site/sitemap.xml` | Auto-generated from NAV manifest on every build | Automatic — just keep `docs/lib/nav.mjs` current |
| `docs/llms.txt` | Curated LLM-readable entry point (Markdown) | Adding or significantly revising a guide |
| `docs/index.html` (head) | Canonical URL, robots meta, OG/Twitter tags, JSON-LD | Changing site name, URL, or description |

**`docs/robots.txt` — currently allows:**
- `GPTBot` (OpenAI training)
- `ClaudeBot` (Anthropic)
- `PerplexityBot`
- `OAI-SearchBot` (ChatGPT search)

Add new crawlers here as they emerge with stable user-agent strings. The `sitemap` directive must point to the live `https://myco.sh/sitemap.xml`.

**`docs/llms.txt` — keep concise.** It is not a full index — it is a curated entry point. Include the guide name, a one-line description, and the `.md` URL (not the `.html` URL — LLMs benefit from the raw Markdown). Do not link to GitHub URLs for docs that live on `myco.sh`; local links preserve topical authority. The build script verifies that all `myco.sh/*.md` links in `llms.txt` resolve to actual files in `_site/` — broken links will abort the build.

**`docs/index.html` JSON-LD schemas** — three schemas are present: `Organization`, `WebSite`, `SoftwareApplication`. Update the `SoftwareApplication` version string and `datePublished`/`dateModified` when releasing a new myco version. These fields feed structured-data cards in search results.

**SEO baseline for guides:** The build injects `<link rel="canonical">`, `<title>`, and `<meta name="description">` into each generated HTML page using the guide's slug, first H1, and opening paragraph. Verify these are populated correctly in the rendered output after adding a new guide.

---

## Procedure 4: GitHub Actions Pages Deployment Workflow

The deploy workflow (`.github/workflows/docs.yml`) triggers on push to `main` (when `docs/**` changes) and runs `npm run build` from inside the `docs/` directory before uploading `docs/_site` as the artifact and deploying to Pages.

**Version pairing rule — do not mix major versions:**

GitHub hard-deprecated `upload-pages-artifact@v3` + `deploy-pages@v4` on **2025-01-30**. The v3 artifact can no longer hand off to the deploy action — workflows using this pairing fail silently or with an opaque error at the deploy step (the upload step reports success, making the failure hard to diagnose).

Always use a matched pair. The current workflow uses v4:

```yaml
# Current (both @v4)
- uses: actions/upload-pages-artifact@v4
  with:
    path: docs/_site
- uses: actions/deploy-pages@v4

# If bumping to v5 (latest as of 2025) — bump BOTH at the same time:
- uses: actions/upload-pages-artifact@v5
  with:
    path: docs/_site
- uses: actions/deploy-pages@v5
```

**When bumping versions:** bump both actions in the same commit. A partial bump restores the mismatch problem. Check the [GitHub Actions changelog](https://github.blog/changelog/) for deprecation notices before merging workflow PRs.

**Workflow validation checklist before merging any `.github/workflows/docs.yml` change:**

1. `upload-pages-artifact` and `deploy-pages` are the same major version.
2. The `path:` passed to `upload-pages-artifact` is `docs/_site` (the build output directory — contains both `.md` and `.html` files for both surfaces).
3. The build step (`npm run build` from `working-directory: docs`) runs before the upload step.
4. Permissions block includes `pages: write` and `id-token: write`.

**Local smoke-test before pushing (Pages-faithful routing):**

GitHub Pages serves clean URLs by trying `path` → `path.html` → `path/index.html` → `404.html`. A plain `python3 -m http.server` does NOT replicate this — `.html` extension stripping won't work and internal clean-URL links will 404, giving false negatives.

Use a Pages-faithful preview server pointed at `docs/_site`:

```js
// /tmp/myco-docs-preview.mjs  (Node, run with: node /tmp/myco-docs-preview.mjs)
import http from 'http';
import fs from 'fs';
import path from 'path';

const ROOT = './docs/_site';
http.createServer((req, res) => {
  const tries = [
    path.join(ROOT, req.url),
    path.join(ROOT, req.url + '.html'),
    path.join(ROOT, req.url, 'index.html'),
    path.join(ROOT, '404.html'),
  ];
  for (const f of tries) {
    if (fs.existsSync(f) && fs.statSync(f).isFile()) {
      res.writeHead(200);
      return fs.createReadStream(f).pipe(res);
    }
  }
  res.writeHead(404); res.end('Not found');
}).listen(8080, () => console.log('http://localhost:8080'));
```

Then run a BFS link crawler against `http://localhost:8080` and assert all internal links return 200. Flag any external links that look like bare filenames (e.g., `http://CLAUDE.md`) — those indicate the `linkify` fuzzyLink guard was removed.

---

## Cross-Cutting Gotchas

**Adding a guide without updating `docs/lib/nav.mjs` means the build ignores it.** The guide won't be rendered to HTML, won't get a sitemap entry, and won't have inbound navigation links. The build throws an error if a NAV slug has no corresponding `.md` source file — but the reverse is silent: a `.md` file with no NAV entry is silently skipped.

**Do not serve docs with `python3 -m http.server` for testing.** It does not strip `.html` extensions, so clean-URL internal links (`/quickstart`, `/install`) will 404. Always test against `docs/_site` using the Pages-faithful preview server in Procedure 4.

**`sitemap.xml` is generated, not committed.** It lives in `docs/_site/sitemap.xml` after a build — do not create or maintain a static `docs/sitemap.xml`. Keeping guide URLs current in `sitemap.xml` is automatic once the NAV manifest is accurate.

**Link rewriting is path-relative.** If you restructure the `docs/` directory hierarchy (e.g., add subdirectories like `architecture/`), re-verify the out-of-tree link detection logic in `docs/lib/links.mjs`. Links currently classified as out-of-tree (rewritten to GitHub blob URLs) may become in-tree or vice versa.

**`llms.txt` links must point to `.md` URLs, not `.html` URLs.** LLMs benefit from the raw Markdown. The rendered HTML is for humans. Mixing this up undermines the dual-surface strategy. The build verifies `llms.txt` `.md` links resolve — broken ones abort the build.

**Both surfaces must be present in `docs/_site`.** The build copies raw `.md` files alongside generated `.html` files into `_site/` and runs a post-build guard to verify the copy succeeded. If the artifact path in the workflow is narrowed to only `.html` files, the raw `.md` files disappear from `myco.sh` and break the AI-crawler surface.