v0.3.0 to v0.3.0

2 added, 1 removed. Audit A to A.

---

name: spark-model-scheduler

version: "0.3.0"

description: Plan and check unified-memory use on a single DGX Spark (GB10, one memory pool shared by CPU and GPU) before starting or switching inference services - LLM, vision model, ComfyUI image or video generation. Use before bringing the stack up, before a video generation stage, when a model load fails or the machine starts swapping, or when the user asks whether a model "fits" next to the others.

license: BUSL-1.1

metadata:

author: 贺家豪 (Jiahao He)

category: production

tags:

- dgx-spark

- unified-memory

- vllm

- scheduling

---

# Spark model scheduler

DGX Spark has one ~121 GB pool for the OS, every LLM's weights and KV cache, and every

diffusion model. A service that loads fine alone can kill another one later. Decide the

budget first, then start things.

## Workflow

1. Measure before acting:

```bash

cinefoundry mem status

```

It reports total and available unified memory, per-container memory, and ComfyUI's

loaded-model state. `nvidia-smi` shows no total on GB10; trust `/proc/meminfo`.

Per-container numbers cover host allocations only: CUDA memory is not charged to the

container's cgroup, so a vLLM or SGLang container holding 100 GB can show under 5 GB.

Use the drop in available memory, not `docker stats`, to size a GPU service.

**Available is not free.** `mem status` reports `MemAvailable`, which counts the page cache as reclaimable. On DGX Spark CUDA can only allocate memory that is truly free (`MemFree`): it does not make the kernel drop cached pages for it. Every model file read from NVMe - a 22B video transformer, its text encoder, the image models, a vLLM reload after sleep - stays in the page cache, so a machine that shows 60 GB available can have far less free, and a GPU load fails or a sampler runs short. Before every GPU phase, and always before the video phase, compare and evict:

```bash

grep -E '^(MemFree|MemAvailable|Cached):' /proc/meminfo

# The engine does this itself at every phase boundary (enterPhase -> releaseFileCache, no root needed).

# By hand, for a process outside the engine:

python3 -c "import os,sys; [os.posix_fadvise(os.open(os.path.join(d,f),os.O_RDONLY),0,0,os.POSIX_FADV_DONTNEED) for d,_,fs in os.walk(sys.argv[1]) for f in fs]" ~/models

```

Plan against `MemFree` after the eviction, not against `MemAvailable` before it.

2. Check the plan against the budget:

```bash

cinefoundry mem plan --need nemotron=30 --need stepvl=20 --need comfyui=45

```

Exit code 1 means the plan does not fit with the reserve (default 10 GB). Reduce

`--gpu-memory-utilization` / `--max-model-len` on the vLLM services, or stage the work.

3. Keep one diffusion family resident at a time. With the image, edit and video models all left in ComfyUI next to the two LLMs, the GPU driver started failing allocations (`NV_ERR_NO_MEMORY` in the kernel log) and a 25-minute job died on a request timeout. Release at every phase boundary: hero still -> frames -> clips -> interpolation -> voice -> music. The harness frees ComfyUI and records available memory at each boundary (`enterPhase`); and releases the model files' page cache at the same boundary (*checked*: `releaseFileCache` drops every file over 1 MiB under the model root with `posix_fadvise`, no root needed; `CINEFOUNDRY_RELEASE_CACHE=0` turns it off).

4. Stage work by phase when it does not fit at once. The Cinefoundry pipeline is naturally

phased: text planning -> images -> video -> composition. Diffusion memory is only

needed in the middle two phases, and the vision model only for breakdown and QA.

The video phase is the longest: the default 30-second master is six 5-second clips, each

generated by LTX-2.5 Fast NVFP4 at native 1080p and then interpolated by RIFE, one at a time

with the model kept loaded between clips and released after the last.

5. Release diffusion memory at each boundary:

```bash

cinefoundry mem free

```

6. Do not run a second GPU workload (an eval, another film) next to a film job; they share one pool and one GPU.

7. Record `cinefoundry mem status` before and after each phase into `eval/` - these numbers are

the platform evidence for the submission. Report measured values only.

## Reference budget (measure and replace)

| Service | Planned | Notes |

|---|---:|---|

| Nemotron 3.5 Lightning 30B-A3B NVFP4 | 35 GB | measured on GB10 at 0.25 of the pool, 64K context |

| Step3-VL-10B FP8 | 24 GB | measured: 14.25 GiB weights; 0.20 of the pool gives a 47,616-token KV cache |

| ComfyUI, Qwen-Image fp8 | 32 GB | 20.4 GB model + 9.4 GB text encoder |

| ComfyUI, Qwen-Image-Edit-2509 | 32 GB | same size as Qwen-Image; a different family, so release one before loading the other |

| ComfyUI, Wan2.2 I2V A14B fp8 | 30 GB | two 14.3 GB experts loaded in turn + 6.7 GB text encoder + activations |

| ComfyUI, Wan2.2 TI2V-5B | 20 GB | 10 GB model + 6.7 GB text encoder + activations |

| ComfyUI, LTX-2.5 Fast 22B NVFP4, native 1080p | about 45 GB | measured 2026-10-17: lowest free memory 8.4 GB with both LLMs resident; 1088x1920, 121 frames, about 4 min per 5 s clip |

| ComfyUI, RIFE 4.26 interpolation (x5 to 120 fps) | about 20 GB | 601 float frames at 1088x1920 (about 15 GB) plus the encoder; run after the video model is freed |

| Qwen3-TTS voice container | about 8 GB | runs in its own container; free ComfyUI first (a film died here with CUDA out of memory before this was enforced) |

| ComfyUI, ACE-Step music | 9 GB | 7.7 GB checkpoint |

- | Nemotron 3.5 Super VL 120B-A12B NVFP4 (planned) | 75 GB weights, about 91 GB with KV cache | vLLM recipe minimum for one Spark; cannot stay resident next to Step3-VL and diffusion - must sleep (level 2) outside the text stages. Measure before relying on it |

+ | Nemotron 3.5 Super VL 120B-A12B NVFP4 (writers' room) | 85 GB awake (65.7 GB weights and buffers, 15.4 GB KV cache) | measured 2026-10-17: cold start 12 min 32 s, restart 537 s, sleep 3.7 s. A level-2 wake with weight reload took 232 s and left it answering nonsense, so it is never reloaded: it works while every other model sleeps, is released for the rest of the run before the first picture, and the next run restarts it |

| OS, Cinefoundry, reserve | 15 GB | |

vLLM's `--gpu-memory-utilization` is a fraction of the whole pool and is claimed at

startup, so start LLM services first and let ComfyUI allocate on demand.

Phase plan the harness follows (sleep level 2 discards a service's weights and reloads them from NVMe on wake):

| Phase | Awake | Asleep |

|---|---|---|

+ | writers' room, when `CINEFOUNDRY_WRITER_URL` is set (development, screenplay, script notes, compliance, shot list, casting briefs) | writer (Super) | planner, vision gate; ComfyUI freed |

| text (brief, script, compliance, storyboard) | planner, reviewer | - |

| hero still and frames | vision gate | planner |

| clips and interpolation | - | planner, vision gate |

| cut (voice, music, assembly) | woken in the background | - |

Step-3.7-Flash (198B) needs ~116 GB even at IQ4_XS: it cannot share the machine with

diffusion. Use it through the StepFun API, or run it alone in a dedicated planning phase.