git:20261006.8798449 to git:20261016.432b170

---

name: branch-and-checkpoint

description: Saves a running smolvm machine as checkpoints on a schedule, keeps a bounded history, restores any generation in it, pauses and resumes machines without losing running processes, and branches a warm machine into children with a checkpoint of the exact point they started from. Use when a machine's state has to survive the machine; when an agent or a job needs to roll back; when checkpoints must be taken periodically, since smolvm has no scheduler or retention of its own; when fanning a prepared machine out into workers; or when a checkpoint, a pause or a branch fails with an error that names none of its preconditions. Do not use it to ship an environment to another host as a file, which is the pack packet, or for state that only has to survive a stop and start, which is the dev-env packet.

---

# Checkpoints, restores, pauses and branches

brings back both the disk and the memory you left, the schedule keeps exactly the checkpoints you

asked for and no fewer, and every child of a branch starts from a state you still hold as a

checkpoint after the children are gone.

The Linux runs used the scripts of their date; this version's preflight and cleanup scripts ran

on Linux aarch64 on v1.22.2 on 2026-10-03.

`README.md` says what each of these operations is. This file is how to run them, and the traps.

**Three facts shape everything here.**

- **smolvm has no scheduler and no retention.** `scripts/checkpoint.sh` is the one capture a timer

runs, with the rules the engine asks a scheduler to follow built in; `references/scheduling.md`

puts it under cron, systemd or launchd.

- **A branch's captured state stays on this host and cannot be exported.** So a branch point is

paired with a checkpoint taken at the same moment, which is the copy you keep.

- **`--branchable` at `start` decides what a machine can do later.** A branch source needs it on

## Procedure

```

- [ ] 1. preflight.sh read result= and store_space_ok=

- [ ] 2. create-source.sh a machine with state in RAM and on disk

- [ ] 3. checkpoint.sh / schedule.sh captures, with retention

- [ ] 4. restore.sh any generation, into a new machine

- [ ] 5. pause and resume the restored machine, or any branchable one

- [ ] 6. branch-point.sh children plus a checkpoint of where they started

- [ ] 7. cleanup.sh machines, checkpoints, and smolvm's restore cache

```

**1. Preflight.** Read-only.

```bash

scripts/preflight.sh --store ./store

```

It checks that this binary has `--store`, `--history`, `checkpoint-log`, `pause` and restore

`--at`, says whether this host needs `--branchable`, and reports `scheduler=builtin_none` and

`retention=builtin_none` so nobody goes looking for them. `store_space_ok=no` blocks: a capture

into a full disk publishes nothing, and a machine restored while the disk was full was left with

guest I/O errors that stopped it being deleted normally.

**2. A source with state you can check.**

```bash

scripts/create-source.sh smolskill-src # for checkpoints and restores

scripts/create-source.sh smolskill-bp --branch-ready # for a batch branch

```

The workload writes a counter into `/tmp` every second, which is tmpfs and so RAM, and a marker to

`/root/setup.txt` on disk. A restore that brings the counter back proves memory came back, not only

the disk. `--branch-ready` makes the workload park in `smolvm-branch-ready` after its setup, which a

batch branch waits for.

**Or the user's own machine.** Name it with `--name` in the steps below. On macOS, if it was

started without `--branchable`, the first capture fails with `deferred durable save requires

file-backed guest RAM` on v1.22.2 or `guest RAM has no file-backed regions` before v1.20.0, and the

only fix is a restart with the flag:

```bash

smolvm machine stop --name worker

smolvm machine start --name worker --branchable

```

That restart is a normal stop and start: disk state survives, and anything in RAM or `/tmp` is

lost, once. Say so to the user before doing it. Every start of that machine needs the flag from

then on for it to stay checkpointable.

A capture also refuses a machine with resources bound to this host: a host mount (`-v`), a

published socket, a remote volume, a host secret reference, custom DNS, a named network, a GPU,

CUDA, Rosetta, `--ssh-agent` or a Docker socket. The error, `the initial portable checkpoint

**3. Capture, once or on a schedule.**

```bash

scripts/checkpoint.sh --name smolskill-src --store ./store --keep 6 --history 2

scripts/schedule.sh --name smolskill-src --store ./store --every 10 --times 4 --keep 2 --history 2

```

Each capture goes to a new directory named `<machine>-<label>-<UTC time>.smolcheckpoint` beside the

store, or in the directory `--out <dir>` names, which has to be on the same filesystem as the

store. It is judged published only when `checkpoint-log` lists it as `(this checkpoint)`, and only

then are the oldest beyond `--keep` deleted and the store pruned. A capture started while another

of the same machine runs exits 3 with `result=skipped`. `schedule.sh` is for a session you are

watching; a schedule that outlives it is `checkpoint.sh` under a system timer,

`references/scheduling.md`.

```

run=1 rc=0 took_s=2 ... kept=1 store_size= 59M result=captured

run=2 rc=0 took_s=5 ... kept=2 store_size= 76M result=captured

run=3 rc=0 took_s=2 ... kept=2 store_size= 95M result=captured

run=4 rc=0 took_s=1 ... kept=2 store_size=114M result=captured

captures_ok=4

captures_failed=0

longest_capture_s=5

result=schedule_ok

```

That is the v1.18.2 run. Ten seconds is for watching it work; a schedule uses minutes,

`references/scheduling.md`.

**Set the interval from `longest_capture_s`, not from the pause.** The source pauses for a few

hundredths of a second; the command takes seconds, because retained RAM is hashed and compressed

after the source resumes. **`--keep` alone does not bound the disk**: every kept checkpoint also

retains `--history` earlier generations, 32 by default, so the store above kept growing with two

directories kept. Bound both.

**4. Restore, any generation.**

```bash

smolvm machine checkpoint-log ./smolskill-src-ckpt-<time>.smolcheckpoint

scripts/restore.sh --from <checkpoint> --name smolskill-old --at '~2'

scripts/restore.sh --from <checkpoint> --name smolskill-new

```

`~0` is the checkpoint itself and `~N` is N generations back along its history. `restore.sh`

takes only a name starting with `smolskill-`, so cleanup can find it, and refuses any other with

`name must start with smolskill- so cleanup.sh will delete it`; to restore under a name the user

chooses, run `smolvm machine create --name <name> --from <checkpoint> --at '~N'` and then

`smolvm machine start --name <name> --branchable` yourself. The restore path is

`machine create --from`; **there is no `machine restore`**. A restored machine takes its own name as

its hostname, and its disks are copy-on-write over what the checkpoint holds, so a restore costs

about 29 MB of real disk on macOS for a 243 MiB checkpoint while `du` reports five times that.

**5. Pause and resume.**

```bash

smolvm machine pause --name smolskill-new

smolvm machine resume --name smolskill-new

```

Pause saves RAM, disks and the running execution and stops the machine; resume brings the same

execution back under the same name. Check a value the workload holds in memory: on both hosts the

counter read 31 before the pause and 34 and 33 after the resume, so it continued where it stopped

and did not run while paused. A paused machine refuses `stop` and `start`; resume it or delete it.

A resume that fails keeps the saved state, and a later resume of the same machine succeeded.

**6. Branch, and keep the branch point.**

```bash

scripts/branch-point.sh --from smolskill-bp --store ./store --count 2 --name-prefix smolskill-w

scripts/branch-point.sh --from smolskill-src --store ./store --name smolskill-one

```

the source is parked there, then branches. Every child and that checkpoint then hold the same

state: a random value the source wrote to RAM before parking came back identical in both children

and in a restore of the checkpoint.

```

source_parked_after_s=1

checkpoint=.../smolskill-bp-branchpoint-<time>.smolcheckpoint

child=smolskill-w-0

child=smolskill-w-1

source_state=running

result=branched

```

For one child with `--name` the checkpoint is taken first and the branch a second or two later, so

the two points differ by whatever the source did in between.

**A restored branch point is not a parked source.** On start the helper releases and the workload

runs the child program with an empty `SMOLVM_BRANCH_NAME`, so a batch branch from it waits for a

branch point that never comes. Branch it with `--name`, one child at a time, or restore it once per

worker; either way the child gets the state the original children started from.

**7. Clean up.**

```bash

scripts/cleanup.sh --purge --restore-base --checkpoints ./store ./*.smolcheckpoint

```

It deletes the machines the scripts recorded, children first by `--cascade`, removes the named

checkpoint directories and stores, and with `--restore-base` removes smolvm's copies of restored

checkpoints, which otherwise stay after every machine and checkpoint is gone, printing a

`removed=` or `absent` line for each. It waits up to 20 seconds, polling the machine list, and

says so, because an ephemeral entry retires after its run returns.

## Traps

Full detail in `references/traps.md`. The ones that cost the most:

- **`--output` must end in `.checkpoint` or `.smolcheckpoint`**, with `--store` too, where it names

a directory. `.checkpoint` is the name from v1.19.1, and the scripts use `.smolcheckpoint`, which

every release accepts.

- **`pgrep -f smolvm-branch-ready` inside the guest always matches**, because the `sh -c` running it

- **`smolvm machine list | grep -q` under `pipefail` can report a machine missing that exists**:

`grep -q` closes the pipe early and the failed write fails the pipeline. Read the list into a

variable first.

- **`machine stop --name` on a name that does not exist leaves an empty directory** under the VM

cache on v1.18.2, and a second stop of it on v1.22.2 leaves the name inside, which reads as a

leak. `cleanup.sh` stops only names still listed.

## Security defaults, and why they are the defaults

- **A checkpoint is the machine's memory and disks.** Anything the workload held, secrets included,

is in it; keep stores on a disk only you can read, and remove `_restore-base` and

`_restore-checkpoints` when you remove the checkpoints.

- **Retention deletes only after the new checkpoint is published**, so a failed capture never

leaves you with fewer good checkpoints than before.

- **A store is local.** It protects against a bad change, not a lost host; export a checkpoint

with `machine checkpoint --export-from` and copy it elsewhere for that.

- **Cleanup deletes only machines the scripts recorded under the `smolskill-` prefix, and through

`--cascade` any machine branched from one of them, whatever its name**, and removes only the

checkpoint paths you name.

- **Nothing here escalates privilege** or edits smolvm configuration.

## Platform arms

and **Linux aarch64** on v1.18.2 and once on v1.22.2, with the one difference that Linux needs no

`--branchable`. On v1.16.1 Linux aarch64 refused `--store` and froze a branch source; from v1.18.2

it does neither. **Linux x86_64**

was not re-run. **Windows** on v1.22.2 checkpoints to a file, pauses and branches with

`--freeze-source` on every branch, and refuses `--store`, so `checkpoint.sh` cannot run there.

From v1.23.0 a checkpoint from an arm64 Mac restores on arm64 Linux, run once from this Mac to Lima.

## Eval prompts, and what they produced

Run 2026-09-24 PT against v1.18.2 from the published release, under an isolated `HOME`, on macOS

26.6.2 arm64 and Lima `linux-kvm` (Ubuntu 24.04 aarch64), every machine at 1024 MiB. Output is

verbatim unless marked.

**1. "Checkpoint this machine every few minutes, keep only the last few, and show me I can go back

to any of them."**

`schedule.sh` with `--keep 2 --history 2`, four captures ten seconds apart, both hosts:

`captures_ok=4`, `kept=2`, and `checkpoint-log` on the newest listed `~0`, `~1` and `~2`. A marker

written between captures came back as `GEN1` from `--at '~2'` and `GEN3` from the newest, with the

RAM counter restored alongside. A second capture started while one ran:

```

result=skipped

note=a capture of smolskill-src is still running (pid 13935). Run one capture at a time per source: ...

```

**2. "Fan this prepared machine out into workers, and keep a copy of exactly the state they started

from."**

`branch-point.sh --count 2` on a `--branch-ready` source, macOS: the source wrote `8445` to RAM

before parking; both children read `SETUP_DONE`, their own `CHILD=` name and `8445`, and a restore of

the branch-point checkpoint read `8445`. Linux the same with `22642`. `source_state=running` on both.

**3. "Pause the machine I restored and bring it back later without losing what is running."**

The restored machine's counter: `ram before=31 after=34` on macOS and `ram before=31 after=33` on

Linux, across a 10 s pause. Separately, on macOS a restored machine paused while the host disk was

full failed its first resume with `extract paused checkpoint: failed to unpack ...`, and the same

`machine resume` succeeded once space was freed: counter 8 before the pause, 14 after, still

counting.

## Re-verified on v1.23.0

Run 2026-10-04 PT against v1.23.0 from the published release, checksum checked, under a fresh

isolated `HOME` on macOS 27.0.1 arm64, once. On Lima `linux-kvm` (Ubuntu 24.04 aarch64) the

checks named below ran once, so the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (`captures_ok=3`, `longest_capture_s=1`),

`checkpoint-log` listing three generations, restores of `~2` and the newest, the counter at 20

before a pause and 24 after the resume, both children of a batch branch holding the source's token,

and `cleanup.sh --restore-base` removing 258 MB of `_restore-checkpoints`. A capture now reports

when the source resumed, `Source resumed after 0.024s; packaging continued in the background`

([#1519](https://github.com/smol-machines/smolvm/pull/1519)).

**A checkpoint taken on this Mac restored on Linux aarch64**

([#1531](https://github.com/smol-machines/smolvm/pull/1531)). `create-source.sh`, then

`machine checkpoint --name smolskill-mac --output mac.checkpoint` on the Mac with the RAM counter

at 6, which printed `40 MiB written` and `0.066s source pause`. Copied to Lima unchanged,

`machine create --from` and `machine start` there took a second each; the counter read 7 and five

seconds later 12, the RAM token and the disk marker matched the Mac's, and the guest said

`Linux 6.12.95`.

## Re-verified on v1.22.2

Run 2026-10-03 PT against v1.22.2 from the published release, checksum checked, under an isolated

`HOME` on macOS 27.0.1 arm64, twice, the second time from a fresh `HOME`. On Lima `linux-kvm`

(Ubuntu 24.04 aarch64) on 2026-10-03 guests above 2048 MiB timed out, so the Linux lines below are

a single run and the Linux stamp stays on its earlier release.

macOS: every script. Three scheduled captures (`captures_ok=3`, `longest_capture_s=1`),

`checkpoint-log` listing three generations, restores of `~2` and the newest, the counter at 21

before a pause and 25 after the resume, both children of a batch branch holding the source's token,

and `cleanup.sh --restore-base` removing 275 MB of `_restore-checkpoints`. The `--branchable`

facts in "Branchability is decided at start" were measured on releases v1.18.2 to v1.22.2 on

2026-10-03.

Linux aarch64: every script, with `checkpoint_needs_branchable=no`, the counter 63 then 66 and both

children holding the same token.

## What was not run

- **Linux x86_64**, and the scripts on Windows, which are bash.

- **A schedule over hours.** Four captures ten seconds apart on each host, and single captures

within each run; the growth of a store under `--history 32` over a day was not measured.

- **Restores on another host, beyond Mac to Linux.** Through v1.22.2 a checkpoint restores only on

a host of the same platform, and another is refused with `checkpoint host platform '...' does not

match '...'`. From v1.23.0 one taken on an arm64 Mac restores on arm64 Linux, run once above;

[Moving a running machine from a Mac to Linux](../mac-to-linux.md) has the requirements. Linux to

Mac, and a stored capture rather than a file, were not run.

- **GPU and CUDA machines**, which a capture refuses, and `--share-weights`.

- **Branch pools** (`--hold`, `branch-release`), and `--freeze-source` outside Windows.

- **The API's restore route**, the `from` field on create; the `local-api` packet covers pause and

resume over HTTP.

## Related packets

- `dev-env` for state that only has to survive a stop and start.

- `pack` for moving a machine to another host as a file, including a restored one.

- `local-api` for pause and resume over HTTP.

- `teardown` for the wider cleanup, and for what a leak check must exclude.