running-resumable-sync-jobs git:20260816.dd21600A

Immutable. This exact content is served forever at /api/v1/blob/b6ca8b3e6a1b856e.

---
name: running-resumable-sync-jobs
description: Design and review long-running batch or sync jobs that process many items against a remote API and persist checkpoint state — exit codes that report partial failure, checkpoints that are safe to resume, per-item tolerance vs. fatal abort, telling "zero" apart from "couldn't fetch", bounded retries, and honest dry-runs. Use when writing or reviewing a mirror/sync command, a nightly cron job, an importer, a paginated fetcher, or a background worker.
---

# Running Resumable Sync Jobs

A sync job is a loop over N items against a service that will fail partway
through. Everything hard about these jobs is what happens on item K of N: what
the process reports, what it wrote down, and what the *next* run does with that.
Get the happy path right and you still ship a job that silently under-collects,
double-applies, or bills for work it never did.

## The exit code is the only thing automation reads

The most common failure: per-item errors are printed as warnings, the loop
`continue`s, and the run finishes by saving state, committing, printing
"Sync complete!" and returning success. A cron wrapper, a CI step, or a
supervisor sees exit 0 and reports a healthy job forever.

```python
# Bad — the warning goes to a log nobody reads; the process says "fine".
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        print(f"Warning: failed to mirror {user}: {e}")
        continue
save_state(); commit(); print("Sync complete!")
return 0

# Good — failures are counted and surfaced in the status the caller can act on.
failed = []
for user in users:
    try:
        mirror(user)
    except MirrorError as e:
        log.error("failed to mirror %s: %s", user, e)
        failed.append(user)
save_state()           # keep the work that succeeded
if failed:
    log.error("%d of %d targets failed: %s", len(failed), len(users), failed)
    return 1           # or 2, if you want "partial" distinguishable from "total"
return 0
```

Continuing past a failure is usually right — you want the other 99 items. What
is never right is continuing *and* claiming success. If partial success must be
acceptable to the caller, make it an explicit opt-in (`--allow-partial`), not the
default silence.

## Checkpoint what actually happened, not what you attempted

Resumable jobs record "item X is done" so the next run skips it. Two ways that
record goes wrong, and both are worse than no checkpoint at all:

**Attempted-but-unfinished recorded as done.** State is saved for everything
processed before the failure, so the next run skips those and the post-failure
items stay pending forever — and because the run exited 0, nothing ever says so.
This is the exit-code bug above compounding into permanent data loss.

**Finished-but-not-recorded.** A batch applies its units one at a time
(N commits, N rows, N API calls) and the helper returns only an error when unit
K fails — no count of the K−1 that landed. The caller leaves that item's state
untouched, so the next run recomputes the *whole* delta and re-applies the units
that already succeeded. Every retry over-applies.

```python
# Bad — the caller learns nothing about how far the batch got.
def apply_batch(units) -> None:
    for u in units:
        apply(u)            # raises on failure; K-1 side effects already landed

# Good — report progress alongside the failure so state can advance.
def apply_batch(units) -> int:
    """Returns the number of units successfully applied."""
    applied = 0
    for u in units:
        try:
            apply(u)
        except Exception:
            raise PartialBatch(applied) from None
        applied += 1
    return applied
```

Either report the partial count and check it in, or make the unit of work atomic
(one transaction, one commit) so "some of it happened" is not a reachable state.
Pick one deliberately; the default — raise and discard the count — is the bug.

## Per-item tolerance must be decided *before* the shared error handler

Jobs accumulate a shared `handle_api_error(response)` helper. These are almost
always **fatal by default**: unknown status → log and exit. That is the right
default for a single-target command and the wrong one for a loop over 900 items,
where one deleted account should skip one item.

The trap is that the skip branch looks present but is dead:

```python
# Bad — handle_api_error raises SystemExit on any non-200, so the skip below
# is unreachable. One 404 aborts the run after 800 successful calls and
# writes no output at all.
r = get(f"/users/{login}")
if r.status_code != 200:
    signal = handle_api_error(r)
    if signal == "retry":
        continue
    log.warning("Skipping user %s due to error", login)   # never runs
    skipped += 1
    continue
```

Branch on the cases you tolerate *before* delegating:

```python
if r.status_code != 200:
    if is_rate_limited(r):
        handle_api_error(r)      # shared wait/retry policy
        continue
    log.warning("Skipping %s: HTTP %s", login, r.status_code)
    skipped += 1
    continue
```

Then report `skipped` in the summary. Two more rules that fall out of this:

- **Grep for other callers** when you change a shared handler's contract. A
  helper that is fatal-by-default with one tolerant caller is a helper whose
  other callers are all one bad row away from discarding a full run.
- **Write output incrementally, or before enrichment.** A job that fetches a
  list, enriches every element, then writes one file at the end loses everything
  to a failure in the last element. Persist the cheap list first.

## Distinguish "zero" from "couldn't fetch"

Aggregators fold a failed fetch into the output as a plausible number. This is
the quietest bug in the whole category, because the artifact looks complete.

```python
# Bad — a rate-limited or transient clones fetch is written out as a confident 0
# and flows into the printed total.
row = {"views": views["count"], "clones": clones.get("count", 0) if clones else 0}

# Good — unknown stays unknown, and the run says how much it lost.
if clones is None:
    incomplete.append((repo, "clones"))
    row["clones"] = None          # or omit the row entirely
```

Related: a run that treats only *some* failures as decisive is inconsistent by
construction. If a `None` from fetcher A increments `skipped` and `continue`s,
but a `None` from fetchers B and C is silently coerced to 0 and dropped from the
aggregate, the trailing "skipped: M" line under-reports and the totals are wrong
in a direction no reader can detect.

**Cross-check totals against detail when the source gives you both.** If a page
publishes "1,759 contributions in 2024" and your parser extracts zero rows, that
is a parse failure, not an empty year. A positive summary with an empty detail
set must raise; zero-or-absent summary with an empty detail set is a valid empty
result.

## Bound every retry, and count per unit of work

A `continue` on a "retry" signal, inside a pagination loop, with no counter,
re-requests the same page forever against a persistent 403 or 429. The job does
not crash — it hangs, which no timeout-free supervisor will ever notice.

```python
MAX_RETRIES = 5

retries = 0                       # per page, not per run
while True:
    r = get(url, params={"page": page})
    if is_rate_limited(r):
        if retries >= MAX_RETRIES:
            return items, False   # partial + incomplete flag; warn loudly
        wait_for_reset(r)
        retries += 1
        continue
    retries = 0                   # reset after any success
    ...
```

Counting **per unit of work** and resetting after a success is what lets a long
multi-page fetch ride out repeated rate limits while still bounding retries of a
single stuck page. A per-run counter kills legitimate long jobs; no counter hangs
forever.

**Write down whose job the waiting is.** If the shared handler already slept
until the reset timestamp before returning `"retry"`, a caller that adds its own
`sleep(60)` double-waits — a bug that is invisible in output and only shows up as
a job that takes twice as long as it should. Document the contract once ("the
wait already happened; re-request now") and hold every new caller to it.

## Incremental cursors: re-include the boundary

A job that stores `last_sync` as a timestamp but compares at day granularity
loses the boundary day permanently:

```python
# Bad — skips days <= since. Anything that happened later on the same calendar
# day as the last sync is never re-fetched, and that day's count freezes.
if day_date <= since:
    continue

# Good — truncate the cursor to the comparison's granularity and re-include the
# boundary. Merging is keyed by date, so re-processing that day is idempotent.
if day_date < since.date():
    continue
```

The general rule: an incremental cursor must be exclusive only at the exact
granularity it is stored at. If your merge step is keyed and idempotent — and it
should be — re-processing the boundary unit costs nothing and closes the hole.

## Dry-run must perform every in-memory mutation the real run does

Gate persistence and external side effects. Never gate the in-memory state
changes the preview loop reads, or the preview describes a different job than the
one that will run.

```python
# Bad — under --dry-run the counters are never cleared, so the preview computes
# incremental deltas and reports "nothing to do", while a real --rebuild would
# re-apply the entire history.
if rebuild and not dry_run:
    state.clear_all_progress()

# Good — clear in memory always; only the write is gated.
if rebuild:
    state.clear_all_progress()
...
if not dry_run:
    save_state(state)
    rewrite_history()
```

Also make destructive scope explicit in the preview. A `--rebuild` that recreates
history by removing every tracked file is correct only in a dedicated
single-purpose repo; run it where other content lives and it deletes that too.
Say what will be removed, not just what will be added.

## Testing these paths

- **Assert the exit code**, not just stdout. A test that only checks for a
  warning string passes on the version that exits 0.
- **Record sleep durations, don't stub them away.** `monkeypatch.setattr(mod,
  "sleep", sleeps.append)` lets you assert the exact sequence
  (`assert sleeps == [reset_wait, 0.1]`), which is what catches a redundant extra
  wait. A no-op `lambda _s: None` cannot.
- **Make the error persistent** when testing a retry cap: serve a reusable
  error response and assert the request count for the stuck page. Also match on
  the response *body* if that is what your handler branches on — status alone may
  not select the branch you think.
- **A missing cap makes the suite hang, not fail.** Run the mutated suite under
  an external timeout; no output within N seconds *is* the reproduction.
- **Test the resume, not just the run.** Fail item K, then run again against the
  same state and assert the remaining items are attempted and the completed ones
  are not re-applied. A single-run test cannot see either checkpoint bug.

## Checklist

- [ ] Non-zero exit (or an explicit `--allow-partial`) when any item failed
- [ ] Failure count and the failing identifiers appear in the final summary
- [ ] Checkpoints record only completed work; partial batches report their count
- [ ] Per-item tolerance branches *before* the shared, fatal-by-default handler
- [ ] Skip/dead branches verified reachable (grep the handler's other callers)
- [ ] Output written incrementally, or the cheap pass persisted before enrichment
- [ ] Failed fetches are `None`/absent in the artifact, never a coerced `0`
- [ ] Summary-vs-detail contradiction raises instead of returning empty
- [ ] Every retry loop is bounded, counted per unit, and reset after a success
- [ ] The "who waits" contract is documented and no caller double-sleeps
- [ ] Incremental cursor re-includes the boundary unit; merge is idempotent
- [ ] Dry-run mutates in-memory state identically; only writes are gated
- [ ] Tests assert exit codes, sleep sequences, request counts, and a resumed run

Related: **no-silent-pass** for making in-process errors visible at the call site
and for proving a regression test can actually fail.