health · diff
git:20260927.7c8bf40 to git:20260929.6fd9ebd
9 added, 10 removed. Audit A to A.
---
name: health
description: >
Monitor solstone uptime, troubleshoot capture/processing failures, review
agent runs and errors, pipeline health. CLIs: journal health (service),
journal talent (agent runs), solstone call health pipeline (per-day summary).
TRIGGER: health, status, is it running, service down, errors, agent runs,
logs, pipeline, journal health, journal talent logs.
---
# Health CLI Skill
Monitor solstone service uptime, troubleshoot failures, and inspect agent runs. Invoke via Bash: `journal health ...`, `journal talent ...`, or `solstone call health <command>`.
**Scope note**: Three CLI surfaces live here: `journal health*` (supervisor/service level), `journal talent*` (agent run level), and `solstone call health <command>` (app-level pipeline health). They're grouped together because health troubleshooting routinely crosses the three levels.
**Typical workflow**: `journal health` → `journal health logs` → `journal talent logs` → `journal talent log <ID>` for agent-run detail → `solstone call health pipeline` for a day-level pipeline summary.
## status
```bash
journal health
```
Show current supervisor status: running services (names, PIDs, uptimes), crashed services, active tasks, queue depths, heartbeat health, and callosum client count.
Connects to `journal/health/callosum.sock` with a 10-second timeout.
Example:
```bash
journal health
```
## logs
```bash
journal health logs [-c N] [-f] [--since TIME] [--service NAME] [--grep PATTERN]
```
- View service health logs from today's log files.
+ View operational logs from today, or from `--since` through today.
- - `-c N`: lines per service (default `5`).
- - `-f`: follow mode — tail all logs continuously.
+ - `-c N`: number of lines to show, newest last (default `5`).
+ - `-f`: follow mode — tail all logs continuously. It ignores `-c`, `--since`, `--service` and `--grep`.
- `--since TIME`: filter by time. Accepts relative (`30m`, `2h`, `1d`) or absolute (`4pm`, `16:00`).
- `--service NAME`: filter to one service.
- `--grep PATTERN`: filter lines matching a Python regex.
Behavior notes:
- - Reads symlinked logs from `journal/YYYYMMDD/health/*.log`.
- - Includes `journal/health/supervisor.log` when no filters are active.
- - Log line format: `ISO8601 [service:stream] LEVEL:logger:message`.
- - `-f` mode handles symlink target rotation at midnight.
+ - Reads the logs under `journal/chronicle/YYYYMMDD/health/`, one file per process run per day.
+ - Process log line format: `ISO8601 [service:stream] message`.
+ - The supervisor's own output is the `service` source. Its lines are shown as written, without the `[service:stream]` prefix; select them with `--service service`.
Examples:
```bash
journal health logs
journal health logs -c 20 --service cortex
journal health logs --since 30m --grep "ERROR"
journal health logs -f
```
## agent runs
```bash
journal talent logs [AGENT] [-c COUNT] [--day YYYYMMDD] [--daily] [--errors] [--summary]
```
List recent agent runs.
- `AGENT`: optional agent name filter.
- `-c, --count`: max runs shown (default `20`; `50` when `--daily`).
- `--day YYYYMMDD`: show only runs from a specific day.
- `--daily`: show only daily-scheduled runs.
- `--errors`: show only error runs.
- `--summary`: show grouped aggregation instead of individual lines.
Flags compose with AND logic. For example, `--daily --errors` shows only daily runs that errored.
Output columns: use_id, time, name, status, runtime, events, tools, output_size, model, facet.
Examples:
```bash
journal talent logs
journal talent logs activity -c 10
journal talent logs --daily
journal talent logs --daily --summary
journal talent logs --day 20260228
journal talent logs --daily --errors
```
## agent run detail
```bash
journal talent log <ID> [--json] [--full]
```
Show events for a single agent run.
- `ID`: agent run ID (from `journal talent logs` output).
- `--json`: raw JSONL events.
- `--full`: expanded event detail (no truncation).
Without flags, shows a one-line-per-event timeline: timestamp, event type, detail.
Examples:
```bash
journal talent log 1700000000001
journal talent log 1700000000001 --json
journal talent log 1700000000001 --full
```
## pipeline summary
```bash
solstone call health pipeline [--day YYYYMMDD | --yesterday]
```
Summarize think-pipeline health for one day — anomalies, performance metrics, and per-stage outcomes across the day's processing runs. Emits JSON.
- `--day YYYYMMDD`: target day. Defaults to today.
- `--yesterday`: shortcut for yesterday. Mutually exclusive with `--day`.
Use this when you want a day-level view after daily processing completes, rather than a per-run drilldown via `journal talent log`.
Examples:
```bash
solstone call health pipeline
solstone call health pipeline --yesterday
solstone call health pipeline --day 20260115
```
## journal layout
Reference map of key paths. `journal/` is the journal root.
### journal level
| Path | Purpose |
|------|---------|
- | `health/` | Service logs: `<service>.log` symlinks, `callosum.sock`, `supervisor.log` |
+ | `health/` | `callosum.sock` |
| `agents/` | Agent run logs: `<name>/<id>.jsonl`, `<name>/<id>_active.jsonl`, `<name>.log` symlink, `<day>.jsonl` day index |
| `config/` | `journal.json`, `convey.json`, `schedules.json`, `actions/YYYYMMDD.jsonl` |
| `facets/<facet>/` | Per-facet data: `facet.json`, `entities/`, `events/`, `news/`, `logs/` |
| `entities/<id>/` | Canonical entity records: `entity.json` |
| `tokens/` | Token usage: `YYYYMMDD.jsonl` per day |
| `indexer/` | Search index: `journal.sqlite` (FTS5) |
| `streams/` | Stream state: `<name>.json` |
| `imports/` | Imported audio and processing artifacts |
### day level (`YYYYMMDD/`)
| Path | Purpose |
|------|---------|
| `<stream>/HHMMSS_LEN/` | Segment folders (captures, extracts, agent outputs) |
| `agents/` | Daily agent outputs: `<name>.md`, `<name>.json` |
- | `health/` | Service logs for that day: `<ref>_<service>.log` (symlinked from journal-level `health/`) |
+ | `health/` | That day's operational logs: `oplog--*.log`, one file per process run |
| `stats.json` | Day statistics |
### segment level (`YYYYMMDD/<stream>/HHMMSS_LEN/`)
| Path | Purpose |
|------|---------|
| `audio.*` | Audio captures (`.flac`, `.m4a`, `.ogg`, `.opus`) |
| `<pos>_<connector>_screen.*` | Screen captures (`.webm`, `.mov`, `.mp4`) |
| `audio.jsonl` | Audio transcript extract |
| `<pos>_<connector>_screen.jsonl` | Screen analysis extract |
| `stream.json` | Segment metadata and stream linkage |
| `*.md` | Segment-level agent outputs |
## services
Which services write where:
| Service | Writes to |
|---------|-----------|
| Observer | Audio/video captures in segment folders |
| Sense | Transcripts + screen analysis (JSONL) in segment folders |
| Cortex | Agent JSONL in `agents/<name>/`, outputs in segment/day dirs |
| Indexer | `indexer/journal.sqlite` |
- | Supervisor | `health/supervisor.log`, service logs in `YYYYMMDD/health/` |
+ | Supervisor | its own output and every process log in `chronicle/YYYYMMDD/health/` |
## Troubleshooting
### `journal health` returns "Connection refused" or times out
The supervisor is not running. Check if `journal supervisor` is active. The owner may need to start the service with `journal up` (`make dev` in a dev checkout). ⛔ Do not tell an owner to run `journal start` — it runs the supervisor in the foreground, tied to their terminal, and does not touch the installed service.
### Agent run shows "error" status in `journal talent logs`
Run `journal talent log <ID> --full` to see the complete event timeline including the error. Common causes:
- API key issues (rate limits, expired keys)
- Prompt too large (context overflow)
- Network connectivity
### Missing segments or capture gaps
1. Run `journal health` to check observer service status
2. Run `journal health logs --service sense --since 2h` to check for transcription errors
3. Check if the stream is active: `journal streams`
### Slow or failing agents
Run `journal talent logs --summary` for each agent's completed and failed runs and its runtime range. Filter by agent: `journal talent logs <agent-name> --summary`.
## Gotchas
- **`journal health` times out at 10 seconds.** If the supervisor is slow or hung, you'll hit the timeout before seeing results. Confirm the supervisor process is alive (`ps` / `journal supervisor` status) before assuming the service is down.
- **Talent log IDs are millisecond timestamps.** `journal talent log 1700000000001` expects the full ID from `journal talent logs`, not a seconds-precision value.
- **`solstone call health pipeline` needs today's processing to have run.** Running it at 6am before the daily pipeline has executed will return sparse results for today; use `--yesterday` instead.