--- name: sk-vision description: "Local vision for text-only models: OCR, inspect, detect and pixel analysis on screenshots via a private Moondream runtime." allowed-tools: [Read, Bash] version: 0.3.6.0 --- > Canonical source: `.skilled/skills/sk-vision/`. This is a generated markdown-only copy so Hermes can load the > skill without scanning the canonical directory. Read `references/`, `assets/` and `scripts/` > from the canonical path above. # sk-vision Local vision skill. Text-only coding models get grounded OCR, layout, detect, and inspect evidence from a private Moondream runtime. The default posture is opt-in and idle. OpenCode does not register the tools by default. Pi registers them hidden. Devin gets evidence injected by a prompt-time hook. Cursor runs the CLI. No API keys, no cloud upload, no per-image cost. **Setup.** The runtime is built with [Bun](https://bun.sh) 1.0 or newer, which nothing else in this repository needs. A fresh clone carries no build, so both plugin links, `.skilled/plugins/sk-vision.js` and `.opencode/plugins/sk-vision.js`, point at a missing `vision-runtime/dist/plugin.js`, the Cursor CLI at `vision-runtime/dist/vision-cli.js` is missing too, and `/vision` fails with ENOENT. Run `bun install && bun run build` in `vision-runtime/` once to create them. --- ## 1. WHEN TO USE ### Activation Triggers Use this skill when the user invokes `/vision` or explicitly requests vision evidence for a local image. Route on any of: - screenshot OCR - attached image - mockup - error.png - local vision - moondream - grounded evidence - `/vision` - any request for the 13 `sk_vision_*` tools (`inspect`, `detect`, `point`, `ocr`, `status`, `segment`, `metadata`, `crop`, `zoom`, `colors`, `diff`, `annotate`, `reverse`) The default posture is idle. The skill does not inspect attached images or advertise its tools unless the user invokes `/vision`, calls a tool on a host that exposes it, or sets `SK_VISION_AUTOINSPECT=1` to restore the legacy OpenCode and Pi behavior. ### Use Cases **Reading a screen the model cannot see** - Extract exact text from a screenshot or error dialog (`sk_vision_ocr`). - Read a UI's structure: layout, elements and state (`sk_vision_inspect`). - Answer a specific question about an image (`sk_vision_inspect` with `question`). **Locating and measuring** - Get labeled bounding boxes for objects or UI elements (`sk_vision_detect`). - Get normalized center coordinates for a target (`sk_vision_point`). - Cut out, upscale or mask a region (`sk_vision_crop`, `sk_vision_zoom`, `sk_vision_segment`). **Pixel analysis (no model required)** - Read image dimensions, format and EXIF (`sk_vision_metadata`). - Read the dominant palette and luminance (`sk_vision_colors`). - Compare two images pixel-by-pixel (`sk_vision_diff`). - Annotate a copy for visual validation (`sk_vision_annotate`). - Reverse-search an image locally and on Yandex (`sk_vision_reverse`). ### When NOT to Use - The primary model is already multimodal and can see the image itself. - The ask is audio, video or documents. Those pipelines are not built. - Publishing under the upstream npm name `opencode-senses` (the fork's `publishConfig` is neutralized). - Inventing a tool named `sk_vision_query`. `sk_vision_inspect` without a `question` already runs caption + scene + OCR together. --- ## 2. SMART ROUTING Standalone **Class S** skill. One workflow mode: `sk-vision`. Single routed leaf root: `feature-catalog/` (the per-feature deep corpus). No `mode-registry.json`, no `hub-router.json`, no `description.json`. ### Resource Loading Levels | Level | When to load | Resources | |-------|--------------|-----------| | ALWAYS | Every invocation | This `SKILL.md` (self-contained: protocol, tools, env, troubleshooting) | | ON_DEMAND | A specific tool's deep behavior, edge cases or measured results | `feature-catalog//` for the tool, `manual-testing-playbook/` for scenarios | `SKILL.md` is self-contained for routing and the runtime contract. There is no separate `references/` reference doc. Deep per-tool material lives in `feature-catalog/`. ### Smart Router Pseudocode ```python from pathlib import Path SKILL_ROOT = Path(__file__).resolve().parent INTENT_SIGNALS = { "VISION": { "weight": 4, "keywords": [ "screenshot OCR", "attached image", "mockup", "error.png", "local vision", "moondream", "grounded evidence", "sk_vision_ocr", "sk_vision_inspect", "sk_vision_detect", ], }, } # The self-contained SKILL.md answers most invocations. Route to the catalog only # when a specific tool's deep behavior or a measured scenario is needed. RESOURCE_MAP = { "VISION": ["feature-catalog/", "manual-testing-playbook/"], } UNKNOWN_FALLBACK = { "load_level": "UNKNOWN_FALLBACK", "needs_disambiguation": True, "checklist": [ "Confirm the input is a local image path, data URL, or http(s) image URL", "Confirm the primary model is text-only", "Do not route audio, video, or document work here", ], } def discover_markdown_resources() -> set[str]: """Inventory the routable corpus so routing never names a missing file.""" docs = [] for base in (SKILL_ROOT / "feature-catalog", SKILL_ROOT / "manual-testing-playbook"): if base.exists(): docs.extend(path for path in base.rglob("*.md") if path.is_file()) return {doc.relative_to(SKILL_ROOT).as_posix() for doc in docs} def _guard_in_skill(relative_path: str) -> str: """Refuse any path that escapes the skill root or is not markdown.""" resolved = (SKILL_ROOT / relative_path).resolve() resolved.relative_to(SKILL_ROOT) if resolved.suffix.lower() != ".md": raise ValueError("Only skill-local markdown resources are routable") return resolved.relative_to(SKILL_ROOT).as_posix() def route_resources(request): """SKILL.md answers most invocations; load the corpus only when needed.""" inventory = discover_markdown_resources() if not inventory: return {**UNKNOWN_FALLBACK, "resources": []} loaded = [] for prefix in RESOURCE_MAP["VISION"]: for candidate in sorted(p for p in inventory if p.startswith(prefix)): guarded = _guard_in_skill(candidate) if guarded not in loaded: load(guarded) loaded.append(guarded) return {"resources": loaded} ``` --- ## 3. HOW IT WORKS The skill owns a host-agnostic JSON-RPC runtime under `vision-runtime/` and thin per-host adapters. All 13 tools share one `RuntimeClient` (NDJSON over stdio) talking to `vision-runtime/python/runtime.py`, which wraps the Moondream vision model. The runtime stays idle until a command or explicit tool call starts it. ```text [`/vision` command or sk_vision_* tool called] | v [host adapter: OpenCode plugin / Pi extension] | v [RuntimeClient: NDJSON JSON-RPC over stdio] | v [python/runtime.py + Moondream] --> [ evidence block] ``` ### The 13 tools Locked tool names. Do not add or rename them: | Tool | Provider method | What it returns | |------|-----------------|-----------------| | `sk_vision_inspect` | `query` (with `question`) or `caption`+`scene`+`ocr` | An answer, or a structured scene read + caption + exact OCR | | `sk_vision_detect` | `detect` | Labeled bounding boxes with confidence | | `sk_vision_point` | `point` | Normalized (0-1) center coordinates for a target | | `sk_vision_ocr` | `ocr` | Exact text (`all` / `code` / `error` kinds) | | `sk_vision_status` | `health` | Model load state, device, VRAM, request count | | `sk_vision_segment` | `segment` | Mask/PNG cutout of a target plus its bbox | | `sk_vision_metadata` | `metadata` | Dimensions, format, mode, bytes, DPI, EXIF (no model) | | `sk_vision_crop` | `crop` | A normalized bbox region saved as a PNG (no model) | | `sk_vision_zoom` | `zoom` | LANCZOS upscale of a region, optionally re-analyzed (no model unless re-analyzed) | | `sk_vision_colors` | `colors` | Dominant palette, luminance buckets, average RGB (no model) | | `sk_vision_diff` | `diff` | Pixel-change map between two images, optionally described | | `sk_vision_annotate` | `annotate` | Boxes/points drawn onto a copy (no model) | | `sk_vision_reverse` | `reverse` | Local perceptual-hash matches + Yandex URLs (no API key) | Host tool parameter contracts (bbox strings, `kind`, `analyze`, `providers`) live in `hooks/pi/sk-vision.ts` and `vision-runtime/src/opencode/tools.ts`. ### JSON-RPC protocol The runtime is a line-delimited JSON (NDJSON) JSON-RPC service over stdio. `RuntimeClient` spawns `python/runtime.py` and exchanges one JSON message per line: ```text Request : {"id": int, "method": str, "params": {...}} Response: {"id": int, "result": {...}} {"id": int, "error": {"code": str, "message": str}} ``` `source` is `{"type": "path", "path": str}` or `{"type": "data", "data": dataUrl}`. A `path` may be an http(s) URL. The TypeScript side downloads it into the cache before it reaches the runtime. Runtime methods (the 13 tools map onto these): `ping`, `status`, `load`, `unload`, `query`, `caption`, `scene`, `detect`, `point`, `segment`, `ocr`, `metadata`, `crop`, `zoom`, `colors`, `diff`, `annotate`, `hash_search`, `reverse`, `shutdown`. Error codes include `INVALID_INPUT`, `EMPTY_RESULT`, `FETCH_FAILED` and `NOT_AN_IMAGE`. The host surfaces them as `SK_VISION_ERROR (): ` and never crashes the session. ### Model lifecycle The runtime lazy-loads the model on the first inference request (default `moondream2`). Each `/vision` call follows `SK_VISION_TEARDOWN`: `close` shuts down the runtime by default, `unload` frees the model and keeps the process, and `keep` leaves it running. `load` warms it explicitly, `unload` frees it, and `status` reports load state, device, VRAM and request count. `segment`, `reason` and `ocr` require a Moondream 3.x checkpoint. `moondream2` lacks them. See `MOONDREAM3_ONLY_TASKS` in `runtime.py`. `sk_vision_ocr` requires a Moondream 3.x checkpoint. The default `moondream2` cannot transcribe text and fails loudly because the runtime enforces the `ocr` task. Set `SK_VISION_MODEL=moondream3-preview` for OCR. Treat its output as approximately correct because a preview checkpoint can repeat tokens. Verify against ground truth when exactness matters. ### Host adapters OpenCode and Pi expose in-process plugin APIs, so the skill owns their adapter source under `hooks/` and each host loads it through a symlink or re-export. Cursor and Devin have neither, and they differ from each other: Devin delivers a prompt-time lifecycle hook, Cursor delivers no prompt-time event at all. Each therefore gets the strongest mechanism its runtime actually supports. - **OpenCode**: the built plugin `vision-runtime/dist/plugin.js` registers only the `command.execute.before` hook by default. The hook runs for `/vision`, fetches the latest session image, injects a `` evidence block and tears the runtime down afterward. `SK_VISION_AUTOINSPECT=1` restores the legacy visible tools and always-on inspection. The plugin loads through `.skilled/plugins/sk-vision.js`, which points to the built entry inside the runtime package so it resolves `python/runtime.py`. - **Pi**: `hooks/pi/sk-vision.ts` registers the 13 tools hidden by default. The native `/vision` command calls the hidden `sk_vision_inspect` tool and opens a fresh runtime for each call. It tears the runtime down after each call. `SK_VISION_AUTOINSPECT=1` restores the legacy visible tools and always-on inspection. The extension loads through `.pi/extensions/sk-vision.ts`. - **Devin**: `hooks/devin/sk-vision.mjs` runs on `UserPromptSubmit`, registered in `.devin/hooks.v1.json`. When a prompt names an image path that resolves on disk, it analyzes the image and injects a `` block into the same turn, then tears the runtime down. The model never has to ask. Every path fails open. - **Cursor**: runs the CLI at `vision-runtime/dist/vision-cli.js`, driven by its `/vision` command and its always-apply rule. Cursor delivers no prompt-time hook event, so injection is impossible there and the call has to be made rather than forced. Claude has no sk-vision integration. ### `/vision` command `/vision ` answers the question against the most-recent image in the session. Bare `/vision` behaves by host: - OpenCode reads the latest image and returns scene, caption and OCR evidence. - Pi asks for a question in the conversation or returns a full read. A prompt file cannot open a UI input box. - Cursor runs the CLI against the path the user names, since the CLI cannot see the conversation. OpenCode handles the command in a `command.execute.before` plugin hook. The hook fetches the latest session image, runs the analysis, injects a `` evidence block and tears the runtime down. Pi's command prompt drives its hidden tool, opening a fresh runtime per call. Cursor's command runs the CLI. Devin has no command surface and needs none, because its hook injects the evidence without being asked. ### Environment variables | Variable | Default | Purpose | |----------|---------|---------| | `SK_VISION_MODEL` | `moondream2` | Model checkpoint (Moondream 3.x required for `segment`, `reason`, `ocr`) | | `SK_VISION_PYTHON` | auto | Python interpreter. It wins over the provisioned venv | | `SK_VISION_UV` | `uv` | `uv` binary used to provision the venv | | `SK_VISION_DEBUG` | unset | `1` enables debug logging and disables HF noise suppression | | `SK_VISION_KV_CACHE_PAGES` | unset | KV-cache page count for Moondream 3.x | | `SK_VISION_DISABLE_AUTO_PROVISION` | unset | Stops automatic venv provisioning | | `SK_VISION_AUTOINSPECT` | unset | `1` restores the legacy always-on auto-inspect and visible tools in OpenCode and Pi | | `SK_VISION_TEARDOWN` | `close` | Runtime teardown after each `/vision` call: `close` shuts down, `unload` frees the model and `keep` leaves the process running | | `SK_VISION_CACHE_DIR` | `~/.cache/sk-vision` | Cache root: `fetched/` downloads, `converted/` analysis copies | | `SK_VISION_VENV_DIR` | `~/.cache/sk-vision/venv` | Auto-provisioned runtime venv | ### GPU and first load Requires an NVIDIA Ampere-or-newer GPU or Apple Silicon (M-series). About 6 GB VRAM is enough for the default `moondream2`. The first `load` downloads ~3.9 GB of weights from Hugging Face and may provision a venv under `~/.cache/sk-vision/venv` via `uv`. Do not hide this cost from the user. ### Troubleshooting | Symptom | Likely cause | Fix | |---------|--------------|-----| | `SK_VISION_ERROR (FETCH_FAILED)` | URL download failed or timed out | Check the URL. Raise `fetchTimeoutMs` if needed (default 60s in `photon.ts`) | | `SK_VISION_ERROR (NOT_AN_IMAGE)` | URL returned a non-image content type | Confirm the URL serves an image | | Model fails to load on first inference | Missing weights or unsatisfied dependencies | Run once with `SK_VISION_DEBUG=1`. Check `~/.cache/sk-vision/venv` | | Venv provisioning fails | `uv` missing or Python version mismatch | Set `SK_VISION_PYTHON` to a working interpreter or `SK_VISION_UV` to a working `uv` | | Out-of-memory on load | VRAM below ~6 GB or model too large | Use the default `moondream2`. Close other GPU consumers | | `segment`/`ocr` errors on default model | `moondream2` lacks Moondream 3.x tasks | Set `SK_VISION_MODEL` to a Moondream 3.x checkpoint | | Evidence block missing after `/vision` | The session has no recent image or the command did not run | Confirm the command host and call `sk_vision_inspect` directly when the host exposes it | --- ## 4. RULES ### ALWAYS 1. Author `graph-metadata.json` and `leaf-manifest.config.json`. Generate `leaf-manifest.json` and `leaf-aliases.json` with `ci-skill-root-metadata.cjs --fix`. 2. Keep the skill source of truth in the runtime package. Each host loads it through a symlink, re-export or config, never a hand-copied duplicate. 3. Keep `bun run build` and `bun test` in `vision-runtime/` green before any completion claim. 4. Surface the first-load download and VRAM cost to the user. ### NEVER 1. Invent `sk_vision_*` tool names beyond the locked 13. 2. Publish or refer to the skill as `opencode-senses`. 3. Add `description.json`, `mode-registry.json`, `hub-router.json`, or `command-metadata.json` at this Class S root. 4. Treat `context/` (the upstream dump under the spec packet) as this skill's corpus. It is read-only reference material. 5. Hand-edit the generated `leaf-manifest.json` or `leaf-aliases.json`. ### ESCALATE IF - A host needs a tool the locked 13 do not cover. Decide with the operator before adding a tool. - The runtime contract in `runtime.py` diverges from this SKILL.md. The code wins. Update this doc. --- ## 5. SUCCESS CRITERIA | Check | How to verify | |-------|---------------| | Skill root metadata is valid | `ci-skill-root-metadata.cjs` reports `OK [S] sk-vision` | | Skill package validates | `validate_skill_package.py .skilled/skills/sk-vision` PASS | | Runtime builds and tests | `bun run build && bun test` in `vision-runtime/` exit 0 | | 13 tools are available on both in-process hosts | `rg -c 'name: "sk_vision_' hooks/pi/sk-vision.ts` = 13 (Pi registers them through one wrapper). `SK_VISION_AUTOINSPECT=1` makes the OpenCode plugin and Pi tools visible | | OpenCode command adapter loads | `.opencode/plugins/sk-vision.js` resolves to `vision-runtime/dist/plugin.js` and `/vision` runs its command hook | | Pi command adapter loads | `.pi/extensions/sk-vision.ts` resolves to `hooks/pi/sk-vision.ts`. Pi tools are hidden by default and `/vision` starts a fresh call | | No accidental publishing | no `publishConfig` / `publish:npm` in `vision-runtime/package.json` | | Docs describe shipped behavior | no scaffold-stub language in `SKILL.md` / `README.md` | --- ## 6. REFERENCES The authoritative behavior lives in the code. When this doc and the code disagree, the code wins and this doc is updated. | Detail | Authoritative source | |--------|----------------------| | RPC handlers, defaults, model lifecycle | `vision-runtime/python/runtime.py` | | Provider method contracts, cache dirs | `vision-runtime/src/providers/photon.ts`, `src/providers/types.ts` | | OpenCode command hook and legacy auto-inspect path | `vision-runtime/src/plugin.ts`, `vision-runtime/src/opencode/command.ts`, `vision-runtime/src/opencode/attachments.ts` | | Pi tool registrations and parameter shapes | `hooks/pi/sk-vision.ts` | | Evidence rendering | `vision-runtime/src/core/context-builder.ts` | | Per-tool deep behavior and measured results | `feature-catalog/`, `manual-testing-playbook/`, `benchmark/` | | Class S root-metadata contract | `.skilled/skills/sk-doc/sk-create-skill/references/shared/skill-root-metadata-contract.md` | --- ## 7. INTEGRATION POINTS - **Host load paths**: host-adapter sources live under `hooks/` and are mirrored into the shared hook hub at `.skilled/hooks/sk-vision/{pi,opencode,devin}`. Pi loads `hooks/pi/` through `.pi/extensions/sk-vision.ts`. OpenCode loads the built package plugin through `.opencode/plugins/sk-vision.js`. The entry sits inside the runtime package so it resolves `python/runtime.py`. Devin loads `hooks/devin/sk-vision.mjs` through `.devin/hooks.v1.json`. Cursor holds no adapter source: it runs the built CLI. - **Related skills**: `sk-code` builds and verifies the runtime package. `sk-doc` and `sk-create-skill` own this SKILL.md, README shape and validation gate. - **Tool usage**: the 13 `sk_vision_*` tools are the public surface. The JSON-RPC methods above are internal and reached only through the adapters, never called directly by the host model. --- ## 8. RELATED RESOURCES - [README.md](../../../.skilled/skills/sk-vision/README.md): installation, host setup and the environment reference. - [feature-catalog/feature-catalog.md](../../../.skilled/skills/sk-vision/feature-catalog/feature-catalog.md): the per-tool deep behavior index for all 13 tools. - [manual-testing-playbook/manual-testing-playbook.md](../../../.skilled/skills/sk-vision/manual-testing-playbook/manual-testing-playbook.md): manual verification scenarios per host and tool family. - [changelog/](../../../.skilled/skills/sk-vision/changelog): release notes, one file per version. - [benchmark/README.md](../../../.skilled/skills/sk-vision/benchmark/README.md): measured runtime results and how to reproduce them.