jev-style-serve ยท diff
git:20260926.e885ac8 to git:20261104.4c49bf7
42 added, 16 removed. Audit C to B.
---
name: jev-style-serve
description: Install and start the Jev-Style local decision server (systemone-compatible API + Playground at http://127.0.0.1:8765) on macOS, Linux or Windows/WSL, pick the right backend (MLX on Apple silicon, PyTorch CUDA/CPU elsewhere, llama.cpp GGUF optional), verify it answers, and optionally keep it running at login. Use when the user wants to run Jev-Style, set up a local decision model / systemone endpoint, or when another jev-style skill finds no server on :8765.
---
# Run Jev-Style locally
Goal: a working `http://127.0.0.1:8765/v1/systemone` with the real model, verified by a request.
Work through the steps in order and report what you did. Ask before anything that changes the user's
login items or installs system packages.
## 1. Check what is already there
```sh
curl -s http://127.0.0.1:8765/healthz || echo "no server"
command -v jev-style && jev-style --version
command -v uv || echo "no uv"
uname -sm; python3 --version
```
- If `healthz` already answers with `"model": "jev-style-0.8b-decision-v3"`, stop here: it is running.
+ If `healthz` already answers with a `"model"` that starts with `jev-style-` (`jev-style-0.8b-decision-v3` is the
+ default release, `jev-style-2b-decision-v4` the 2B), stop here: it is running. If it says `jev-style-fake`, the
+ server runs the fake engine (meaningless answers): restart it without `--fake`.
## 2. Install the CLI
Prefer `uv` (installs into its own environment, puts `jev-style` on PATH):
```sh
- # Apple silicon (MLX backend, fastest on a Mac) + PyTorch fallback + MCP server:
- uv tool install "jev-style[all]"
- # Linux / Windows / Intel Mac (PyTorch on CUDA or CPU) + MCP server:
- uv tool install "jev-style[torch,mcp]"
+ # Apple silicon (MLX backend, fastest on a Mac; text only):
+ uv tool install "jev-style[mlx]"
+ # Linux / Windows / Intel Mac (PyTorch on CUDA or CPU), and for images on any machine:
+ uv tool install "jev-style[torch]"
```
- No `uv`? Install it (`curl -LsSf https://astral.sh/uv/install.sh | sh`, ask first) or use
- `pip install "jev-style[all]"` inside a venv.
- Needs Python 3.10+.
+ The MCP server is part of every install. The bare `pip install jev-style` is only the client. No `uv`? Ask first,
+ then install it with a package manager (`brew install uv` or `pipx install uv`; other options:
+ https://docs.astral.sh/uv/getting-started/installation/), or use `pip install "jev-style[mlx]"` /
+ `"jev-style[torch]"` inside a venv of its own (`[mlx]` pins mlx-lm to 0.31.3, which the 2B MLX runtimes require).
+ Needs Python 3.10+. Native Windows runs the PyTorch backend; the Claude Code hook commands are POSIX shell, so
+ the guard needs Git Bash or WSL there.
## 3. Pick the backend
| Machine | Backend | Flag |
|---|---|---|
| Apple silicon with `mlx` installed | MLX, bf16 weights (default `auto`) | `--backend mlx` (`--precision 8bit` for less memory) |
- | NVIDIA GPU | PyTorch CUDA | `--backend torch` |
+ | NVIDIA GPU | PyTorch CUDA | `--backend torch` (`--dtype bfloat16` for the 2B v4, as it was measured) |
| anything else | PyTorch CPU (slower, still works) | `--backend torch --device cpu` |
- | llama.cpp users | GGUF via the `jev-score` scorer | `--backend gguf --scorer /path/to/jev-score` |
+ | llama.cpp users | GGUF via the release's scorer | `--backend gguf --scorer /path/to/<scorer>` |
- `auto` picks MLX on Apple silicon when installed, else PyTorch. The GGUF backend needs the `jev-score`
- binary built once against llama.cpp (`sh build_jev_score.sh /path/to/llama.cpp`, shipped in the
- `chaoliangUNSW/Jev-Style-0.8B-Decision-v3-GGUF` repo); only set it up if the user asks for llama.cpp.
+ `auto` picks MLX on Apple silicon when installed, else PyTorch. The model is chosen with `--release`
+ (`JEV_STYLE_RELEASE` sets it for every command): the default `0.8b` (= `0.8b-v3`, 1.5 GB), or `2b`
+ (= `2b-v4`, 2.2B parameters, 32,768 tokens; reads images with `--backend torch` only, 4.4 GB; MLX 3.8 GB, 2 GB
+ at `--precision 8bit`), or `2b-v3`. Users who ask for "the 2B", "the newest", or "the one that reads
+ images" want `--release 2b` (images: `--release 2b --backend torch`, also on a Mac).
+ The GGUF backend needs a scorer binary built once against llama.cpp: `jev-score` for the 0.8B (sources in
+ `chaoliangUNSW/Jev-Style-0.8B-Decision-v3-GGUF`), `jev-score-v2` for both 2Bs (sources in the
+ `Jev-Style-2B-Decision-v3-GGUF` / `-v4-GGUF` repos; one binary serves both). `jev-style download --backend gguf
+ [--release 2b]` prints the model folder; run `sh build_jev_score.sh /path/to/llama.cpp` there and pass the
+ path it prints as `--scorer` (or set `JEV_SCORE_BIN` / `JEV_SCORE_V2_BIN`). `serve` refuses to start the download
+ while no scorer can be found. Only set this up if the user asks for llama.cpp.
+
## 4. Download and start
```sh
jev-style download # ~1.5 GB once, into the Hugging Face cache (optional; serve does it too)
jev-style serve # http://127.0.0.1:8765 (--port N, --backend ..., --api-key-env NAME)
+ jev-style download --release 2b # the 2B v4 instead (then: jev-style serve --release 2b [--backend torch])
```
+ `download` refuses to fetch anything the installed backend cannot run (`error: ... pip install
+ "jev-style[torch]"`), checks every file against the repo's manifest, and prints a one-line `error:` when
+ the download fails (network, `HF_HUB_OFFLINE`); there is no traceback to parse.
+
Run `serve` in the background or a separate terminal, since it keeps running. Startup takes a few
seconds after the first download. Bind to `127.0.0.1` (the default); only use `--host 0.0.0.0` together
- with `--api-key-env` if the user explicitly wants other machines to reach it.
+ with `--api-key-env` if the user explicitly wants other machines to reach it. The server answers only
+ requests addressed to loopback names (or its `--host`) and refuses browser requests from other sites, so a
+ web page elsewhere cannot use it; `--allow-any-host` lifts that if the user has a reason.
## 5. Verify with a real request
```sh
curl -s http://127.0.0.1:8765/healthz
curl -s http://127.0.0.1:8765/v1/systemone -H 'content-type: application/json' -d '{
"state": "I was charged twice for my subscription this month.",
"questions": {"billing": {"type": "noul", "instructions": "This ticket is about billing."}}}'
```
- Expect `"model": "jev-style-0.8b-decision-v3"` and a `noul` close to 1. Then point the user at the
+ Expect a `"model"` starting with `jev-style-` (`jev-style-0.8b-decision-v3` by default,
+ `jev-style-2b-decision-v4` with `--release 2b`) and a `noul` close to 1. Then point the user at the
Playground: http://127.0.0.1:8765/ (also demos: agent action approval, Snake, 51 languages).
## 6. Optional: start at login
Only if the user asks. macOS: write `~/Library/LaunchAgents/com.jev-style.serve.plist` running
`$(command -v jev-style) serve` with `RunAtLoad` and `KeepAlive`, then
`launchctl load ~/Library/LaunchAgents/com.jev-style.serve.plist`. Linux: write `~/.config/systemd/user/jev-style.service`
(`[Service] ExecStart=%h/.local/bin/jev-style serve`, `Restart=on-failure`; `[Install] WantedBy=default.target`),
then `systemctl --user enable --now jev-style`.
Settings can also come from the environment: `JEV_STYLE_BACKEND`, `JEV_STYLE_PORT`, `JEV_STYLE_DEVICE`,
`JEV_STYLE_MODEL_DIR`.
## Troubleshooting
- `There is no Stream(gpu, ...)` or MLX errors: update (`uv tool upgrade jev-style`), or use `--backend torch`.
+ - `needs mlx-lm 0.31.3 exactly`: another tool changed mlx-lm in the same environment; give jev-style its own
+ (`uv tool install`) rather than downgrading the shared one.
+ - `the mlx backend ... only runs on Apple silicon`: Intel Mac, Linux or Windows: use `--backend torch`.
- `transformers.models.qwen3_5` missing: the torch backend needs `transformers>=5.0`.
- Port in use: `--port 8766` (or `JEV_STYLE_PORT`), then point the other tools at it: `JEV_STYLE_URL=http://127.0.0.1:8766`
for `jev-style decide/eval/mcp` and the guard (or `server_url` in `~/.config/jev-style/guard_config.json`),
`--url` for `jev-style guard-replay`, and `JevStyle(base_url=...)` in Python.
- - No network for the download: copy a model folder and use `--model-dir /path/to/Jev-Style-0.8B-Decision-v3`.
+ - No network for the download: copy a model folder and use `--model-dir /path/to/Jev-Style-0.8B-Decision-v3`
+ (`--verify` checks it against the folder's manifest at load time).
- Just testing plumbing without the 1.5 GB download: `jev-style serve --fake` (answers are meaningless).