v0.3.0 to v0.3.0

2 added, 2 removed. Audit A to A.

---

name: spark-model-scheduler

version: "0.3.0"

description: Plan and check unified-memory use on a single DGX Spark (GB10, one memory pool shared by CPU and GPU) before starting or switching inference services - LLM, vision model, ComfyUI image or video generation. Use before bringing the stack up, before a video generation stage, when a model load fails or the machine starts swapping, or when the user asks whether a model "fits" next to the others.

license: BUSL-1.1

metadata:

author: 贺家豪 (Jiahao He)

category: production

tags:

- dgx-spark

- unified-memory

- vllm

- scheduling

---

# Spark model scheduler

DGX Spark has one ~121 GB pool for the OS, every LLM's weights and KV cache, and every

diffusion model. A service that loads fine alone can kill another one later. Decide the

budget first, then start things.

## Workflow

1. Measure before acting:

```bash

cinefoundry mem status

```

It reports total and available unified memory, per-container memory, and ComfyUI's

loaded-model state. `nvidia-smi` shows no total on GB10; trust `/proc/meminfo`.

Per-container numbers cover host allocations only: CUDA memory is not charged to the

container's cgroup, so a vLLM or SGLang container holding 100 GB can show under 5 GB.

Use the drop in available memory, not `docker stats`, to size a GPU service.

**Available is not free.** `mem status` reports `MemAvailable`, which counts the page cache as reclaimable. On DGX Spark CUDA can only allocate memory that is truly free (`MemFree`): it does not make the kernel drop cached pages for it. Every model file read from NVMe - a 22B video transformer, its text encoder, the image models, a vLLM reload after sleep - stays in the page cache, so a machine that shows 60 GB available can have far less free, and a GPU load fails or a sampler runs short. Before every GPU phase, and always before the video phase, compare and evict:

```bash

grep -E '^(MemFree|MemAvailable|Cached):' /proc/meminfo

# The engine does this itself at every phase boundary (enterPhase -> releaseFileCache, no root needed).

# By hand, for a process outside the engine:

python3 -c "import os,sys; [os.posix_fadvise(os.open(os.path.join(d,f),os.O_RDONLY),0,0,os.POSIX_FADV_DONTNEED) for d,_,fs in os.walk(sys.argv[1]) for f in fs]" ~/models

```

Plan against `MemFree` after the eviction, not against `MemAvailable` before it.

2. Check the plan against the budget:

```bash

cinefoundry mem plan --need nemotron=30 --need stepvl=20 --need comfyui=45

```

Exit code 1 means the plan does not fit with the reserve (default 10 GB). Reduce

`--gpu-memory-utilization` / `--max-model-len` on the vLLM services, or stage the work.

3. Keep one diffusion family resident at a time. With the image, edit and video models all left in ComfyUI next to the two LLMs, the GPU driver started failing allocations (`NV_ERR_NO_MEMORY` in the kernel log) and a 25-minute job died on a request timeout. Release at every phase boundary: hero still -> frames -> clips -> interpolation -> voice -> music. The harness frees ComfyUI and records available memory at each boundary (`enterPhase`); and releases the model files' page cache at the same boundary (*checked*: `releaseFileCache` drops every file over 1 MiB under the model root with `posix_fadvise`, no root needed; `CINEFOUNDRY_RELEASE_CACHE=0` turns it off).

4. Stage work by phase when it does not fit at once. The Cinefoundry pipeline is naturally

phased: text planning -> images -> video -> composition. Diffusion memory is only

needed in the middle two phases, and the vision model only for breakdown and QA.

The video phase is the longest: the default 30-second master is six 5-second clips, each

generated by LTX-2.5 Fast NVFP4 at native 1080p and then interpolated by RIFE, one at a time

with the model kept loaded between clips and released after the last.

5. Release diffusion memory at each boundary:

```bash

cinefoundry mem free

```

6. Do not run a second GPU workload (an eval, another film) next to a film job; they share one pool and one GPU.

7. Record `cinefoundry mem status` before and after each phase into `eval/` - these numbers are

the platform evidence for the submission. Report measured values only.

## Reference budget (measure and replace)

| Service | Planned | Notes |

|---|---:|---|

| Nemotron 3.5 Lightning 30B-A3B NVFP4 | 35 GB | measured on GB10 at 0.25 of the pool, 64K context |

| Step3-VL-10B FP8 | 24 GB | measured: 14.25 GiB weights; 0.20 of the pool gives a 47,616-token KV cache |

| ComfyUI, Qwen-Image fp8 | 32 GB | 20.4 GB model + 9.4 GB text encoder |

| ComfyUI, Qwen-Image-Edit-2509 | 32 GB | same size as Qwen-Image; a different family, so release one before loading the other |

| ComfyUI, Wan2.2 I2V A14B fp8 | 30 GB | two 14.3 GB experts loaded in turn + 6.7 GB text encoder + activations |

| ComfyUI, Wan2.2 TI2V-5B | 20 GB | 10 GB model + 6.7 GB text encoder + activations |

| ComfyUI, LTX-2.5 Fast 22B NVFP4, native 1080p | about 45 GB | measured 2026-10-17: lowest free memory 8.4 GB with both LLMs resident; 1088x1920, 121 frames, about 4 min per 5 s clip |

| ComfyUI, RIFE 4.26 interpolation (x5 to 120 fps) | about 20 GB | 601 float frames at 1088x1920 (about 15 GB) plus the encoder; run after the video model is freed |

| Qwen3-TTS voice container | about 8 GB | runs in its own container; free ComfyUI first (a film died here with CUDA out of memory before this was enforced) |

| ComfyUI, ACE-Step music | 9 GB | 7.7 GB checkpoint |

- | Nemotron 3.5 Super VL 120B-A12B NVFP4 (writers' room) | 85 GB awake (65.7 GB weights and buffers, 15.4 GB KV cache) | measured 2026-10-17: cold start 12 min 32 s, restart 537 s, sleep 3.7 s. A level-2 wake with weight reload took 232 s and left it answering nonsense, so it is never reloaded: it works while every other model sleeps, is released for the rest of the run before the first picture, and the next run restarts it |

+ | Nemotron 3.5 Super VL 120B-A12B NVFP4 (writers' room) | 85 GB awake (65.7 GB weights and buffers, 15.4 GB KV cache) | measured 2026-10-17: cold start 12 min 32 s, restart 537 s, sleep 3.7 s. A level-2 wake with weight reload took 232 s and left it answering nonsense, so it is never reloaded: the writers' room starts its container, the other models make way (slept with sleep mode, their containers stopped without it), and the room stops it again before the first picture and starts the others |

| OS, Cinefoundry, reserve | 15 GB | |

vLLM's `--gpu-memory-utilization` is a fraction of the whole pool and is claimed at

startup, so start LLM services first and let ComfyUI allocate on demand.

Phase plan the harness follows (sleep level 2 discards a service's weights and reloads them from NVMe on wake):

| Phase | Awake | Asleep |

|---|---|---|

- | writers' room, when `CINEFOUNDRY_WRITER_URL` is set (development, screenplay, script notes, compliance, shot list, casting briefs) | writer (Super) | planner, vision gate; ComfyUI freed |

+ | writers' room, when `CINEFOUNDRY_WRITER_URL` is set (development, screenplay, script notes, compliance, shot list, casting briefs) | writer (Super) | planner, vision gate (slept, or their containers stopped); ComfyUI and the model page cache freed |

| text (brief, script, compliance, storyboard) | planner, reviewer | - |

| hero still and frames | vision gate | planner |

| clips and interpolation | - | planner, vision gate |

| cut (voice, music, assembly) | woken in the background | - |

Step-3.7-Flash (198B) needs ~116 GB even at IQ4_XS: it cannot share the machine with

diffusion. Use it through the StepFun API, or run it alone in a dedicated planning phase.