AGENTS.md git:20260902.a7bdafbA

Immutable. This exact content never changes and is served at /api/v1/blob/932c6c7c064ec4b3.

# AGENTS.md

## Project

translate-book is a Codex Skill that translates books (PDF/DOCX/EPUB) into any language using parallel subagents. Published on GitHub as `kcy4334-lgtm/translate-book-arxiv`.
A fork of [deusyu/translate-book](https://github.com/deusyu/translate-book); upstream also publishes on ClawHub, this fork does not.

## Structure

- `SKILL.md` — Skill definition, the orchestration logic that Codex / OpenClaw follows
- `KNOWLEDGE.md` — findings log: tool behaviour that surprised us, with the
  measurement that proved it. Read on unexpected output; append per SKILL.md
  Step 9. Anything a test now enforces is one line here pointing at the test
- `scripts/convert.py` — PDF/DOCX/EPUB → Markdown chunks (via Calibre HTMLZ)
- `scripts/manifest.py` — SHA-256 chunk tracking and merge validation
- `scripts/glossary.py` — Term-consistency glossary; per-chunk term tables injected into sub-agent prompts
- `scripts/chunk_context.py` — Read-only previous/next chunk excerpts injected into sub-agent prompts
- `scripts/meta.py` — Per-chunk sub-agent observation file schema
- `scripts/merge_meta.py` — Merges sub-agent observations into the canonical glossary as each chunk lands
- `scripts/run_state.py` — Selective re-translation planner and run_state.json recorder
- `scripts/repair.py` — In-place fixes for a book that is already translated: `sidecar` rewrites the raw LaTeX tables held in `chunk*.math.json`, `rehash` refreshes `manifest.json` after a chunk was edited by hand. Re-converting instead would move every chunk boundary and cost a full re-translation
- `scripts/arxiv_backend.py` — arXiv LaTeX source → Markdown; every ingest-side fix for constructs pandoc cannot read lives here
- `scripts/merge_and_build.py` — Merge translated chunks → HTML/DOCX/EPUB/PDF
- `scripts/layout.py` — Language font tables and print profiles (page size, margins, body size); the single source of truth both merge_and_build.py and calibre_html_publish.py read
- `scripts/chromium_pdf.py` — Headless-Chromium PDF renderer and PyMuPDF page-number stamping
- `scripts/calibre_html_publish.py` — Calibre format conversion wrapper (EPUB, and DOCX fallback)
- `scripts/template.html`, `scripts/template_ebook.html` — HTML templates

## How this pipeline breaks

Worth reading before adding a check, because it says where to point one.

**Nothing here fails. It disagrees.** Every defect this project has shipped
lived at a boundary between two tools that were each behaving correctly.
pandoc's markdown writer escapes a literal `[` as `\[`; our reader treats
`\[..\]` as display maths — both right, and a column's units rendered as
nothing. pandoc lays a grid table out by display width; a translator pads to
the same character count — both reasonable, and the table collapsed into
prose. pandoc expands `*{9}{r}` when reading and re-emits it beside the
expansion — internally consistent, and a 12-column table became 21. No
component logged a problem in any of these, because none had one.

**Counting agrees with itself.** When nine of twelve tables fell out of the
Word file, the table count, the image count and the value probes all passed:
the HTML had all twelve, and three is not zero. A count is only evidence when
it is compared against something that was not produced by the same code path
— the other formats, or the original PDF.

So checks are worth most in this order:

1. **Against the source PDF** (`source_probe`) — the one reference that cannot
   have drifted. It caught a numbering scheme that was wholly self-consistent
   and wholly wrong.
2. **Between the formats** (`format_probe`) — DOCX against the ebook HTML.
   Neither is authoritative; the disagreement is the signal.
3. **Inside the artifact** (`consistency_probe`, `layout_probe`) — open the
   built file and look at what a reader sees, not at what was emitted.

A check that reads only the thing it is checking will pass on a broken book.

**A table format you turn off does not become safe; it becomes absent.**
`grid_tables` is the only writer format pandoc has for a spanning multi-deck
header. Disabling it because grid layout is width-fragile made pandoc write the
literal string `[TABLE]` instead, and a whole results table left the book with
every count still passing. Formats that are unsafe get *repaired at the merge*,
where the translator is finished and nothing can drift further — never removed
from the writer.


## Testing changes

Use a small file for quick checks, or the checked-in baseline book for the repository's full-pipeline test.

Quick smoke test:

```bash
python3 scripts/convert.py /path/to/small.pdf --olang zh
# then run translation via the skill
python3 scripts/merge_and_build.py --temp-dir <name>_temp --title "test"
```

Full baseline test:

```bash
mkdir -p tests/.artifacts
cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# then run translation via the skill
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "test"
```

Verify: all output_chunk*.md files exist, manifest validation passes, output formats generate.

### Print layout

`python tests/layout_probe.py` builds `tests/fixtures/layout_ko.md` straight
through the build stage and measures the resulting PDF: page size, the four
margins from actual ink, modal body size, measured leading, embedded font
names, lines per page, characters per line and right-margin overflow. Add
`--strict` for a non-zero exit on drift, `--keep` to leave the artifacts in
`tests/.artifacts/`. It needs PyMuPDF, pandoc and a browser, so it is
deliberately not named `test_*.py` and CI never collects it.

A Type3 font in the probe's font list means a variable font failed to
subset-embed — replace it with a static face in `layout.LANG_CONFIG`.

### Output formats

`python tests/format_probe.py <temp_dir> --strict` opens book.docx,
book.epub and book.pdf and counts the tables, images and table values each
actually carries. Run it after any change to how content is injected: raw
HTML survives the HTML path alone, and a `<table>` or `<figure>` that looks
right in the PDF can leave the DOCX empty with every other check green.

`python tests/layout_probe.py --stress` uses a second fixture built for
pagination only: a 45-row table, a display equation positioned to land on a
page break, and a code block longer than a page. It reports which pages each
spans and whether the table header repeated, the equation stayed whole, and
every code line survived. Body-size and leading checks are skipped there —
that fixture is deliberately mostly table and code, so its modal glyph size
is not the body size.

### Against the original paper

`python tests/source_probe.py <temp_dir> --strict` compares the section,
figure, table, equation and cross-reference numbers against the numbers the
**source PDF** prints, located through `config.txt`. Everything else in the
suite checks the build against itself; this is the only check with an outside
reference, and it is the one that catches a numbering scheme that is wholly
self-consistent and wholly wrong. It reads `flat.tex` and the PDF only, never
the translation, so it gives the same answer before and after translating.

### Against the reader

`python tests/consistency_probe.py <temp_dir> --lang ko --strict` looks for
what a reader would trip over and no format check can see: LaTeX that reached
the page as literal text, unresolved `(sec:x)` markers, doubled reference
labels, one term glossed two ways, one English term rendered two ways, and
per-chunk invariants against the untranslated source. Term drift is structural
here — one sub-agent per chunk, none able to see the others — so it is a
standing check, not a one-off.

### Before translating

`python tests/dry_run.py <temp_dir> --lang ko` stages the untranslated chunks
as their own translations in `<temp_dir>_dryrun`, builds, and runs the source
and format probes. The real temp dir is untouched. Use it whenever the ingest
or build path changes: it exercises the whole pipeline on a real paper without
an API call, and every structural defect is decided before translation anyway.

### Table edits

`python scripts/verify_tables.py snapshot <temp_dir>` before the caption
sub-agents run, `check <temp_dir> --strict` after. Translation changes words;
the multiset of numbers, the row count, the `&` count per row and every
`\multicolumn` span must survive it untouched. `check` refuses to answer
without a snapshot, because a baseline rebuilt from the edited files is not a
baseline.

### Is this machine able to build the same book

`python scripts/doctor.py --strict` resolves pandoc, Chromium, Calibre and
PyMuPDF the same way the pipeline does, and reports the fonts separately: a
missing face is not cosmetic, the fallback has different metrics and the page
count changes. It reports what is installed; `tests/layout_probe.py --strict`
is what proves the output.

### Reviewing a finished book

The probes cover what is worth automating. Three sweeps of a finished book
found the rest by opening the artifacts, and these are the axes that paid:

| axis | what to do | what it found |
|---|---|---|
| floats | count figures and tables against the source PDF | 4 figures missing, 12 tables the wrong width |
| tables | read the built HTML's header cells | 4 collapsed into prose, 1 row label gone |
| maths | look for `<math>` that renders empty, `<merror>`, a stray `$` | a column's units drawn as nothing |
| formats | DOCX against the ebook HTML, not against zero | 9 tables missing from Word |
| pages | characters per page; a page with an image and nothing else | 7 near-empty pages, panels split |
| prose | paragraphs with no Hangul; length against the source chunk | (clean — all were the reference list) |
| ingest | re-convert one paper and dry-run it | proved the ingest fixes actually fire |

The last row is the one to remember: a fix to `sanitize_tex` is not tested by
rebuilding a book that was converted before it. Re-convert something.

## The two logs

`KNOWLEDGE.md` records how the tools behave — the surprise and the measurement
that proved it, numbered `K<n>`. `KNOWHOW.md` records how to work — the
practice, and the incident that shows what skipping it cost, numbered `H<n>`.
Before changing anything here, read KNOWHOW's "Which document, for which task"
table; it says which of these documents to open for the change you are making.
`tests/test_knowhow.py` keeps the two from drifting into each other.

## Conventions

- Only `chunk*.md` naming — no `page*` legacy support
- SKILL.md frontmatter must stay single-line per field (OpenClaw parser requirement)
- Script paths in SKILL.md use `{baseDir}` not hardcoded paths
- Subagent instructions in SKILL.md must be platform-neutral (work on Codex, OpenClaw, Codex)
- Checked-in baseline inputs live under `tests/baselines/<book-id>/`; generated full-pipeline outputs live under `tests/.artifacts/`
- There is ONE README. Upstream keeps a Chinese one; this fork dropped it rather than carry a translation nobody here can review — it would go stale on the first edit, and a stale translation is worse than none
- Releases follow `.claude/commands/release.md` — `git push origin main`, then `git tag vX.Y.Z && git push --tags`. Do not skip the git tag; it's the only version anchor in the repo. ClawHub publishing is upstream's, not this fork's

## Do not

- Do not put language font data or page geometry anywhere but `scripts/layout.py` — it used to be duplicated across two modules that had already drifted
- Do not list a variable font (`Noto Serif KR`, `Noto Sans KR`) in a CJK stack; Chromium cannot subset-embed one and emits a Type3 object per glyph instead
- Do not reintroduce `page*` file support — it was intentionally removed
- Do not hardcode `~/.Codex/skills/` paths in SKILL.md — use `{baseDir}`
- Do not put platform-specific tool names (Agent, sessions_spawn) in `allowed-tools` as the only option — keep the whitelist cross-platform
- Do not infer table structure from the rendered HTML. Header depth and body rules are read from `flat.tex` by `table_structures()` and applied by document order, so they are identical whether the table reached the page as raw LaTeX or as a pandoc-converted markdown table. Guessing from cell contents is what the Average/평균 heuristic is for, and it runs only when the source has no rules at all
- Do not centre a display `<math>` with `text-align` — Chromium's MathML Core ignores it on a block element. Use `display: flex; justify-content: center`, which also keeps the element full width so the `::after` equation number still lands in the margin
- Do not accept a sub-agent's report as evidence that its work is correct. Run `scripts/verify_chunk.py` on the file it wrote: a chunk that came back in the wrong language, lost a paragraph, or quoted evidence that is not in the source reports success exactly the way a correct one does
- Do not declare table or figure STRUCTURE inside `@media print`. The PDF is the file that gets looked at, so a print-only rule reads as working while the EPUB and the web page go without it -- and Calibre deletes a class that no active rule matches, so the EPUB loses the markup too, not just the styling
- Do not disable a pandoc table format to avoid a layout bug (see "How this pipeline breaks")
- Do not add mtime-based incremental rebuild for HTML/format generation — the current skip logic is intentionally simple (existence check). Metadata/template changes require manual cleanup. This is documented in the README.

## Cursor Cloud specific instructions

### Environment

- Python 3.12+ is pre-installed; no version manager needed.
- System dependencies (Calibre, Pandoc) and pip packages (pypandoc, beautifulsoup4) are installed by the update script.
- Unit tests only依赖 Python stdlib（不需要 pip 包或外部二进制，直接 `python3 -m unittest discover` 即可运行）。

### Running tests

- **Unit tests (CI-equivalent):** `python3 -m unittest discover -s tests -p 'test_*.py' -v` — runs from repo root, no setup needed.
- **Compile check:** `python3 -m compileall scripts tests`

### Full pipeline integration test

Run from `tests/.artifacts/` to keep generated files out of the repo root:

```bash
mkdir -p tests/.artifacts && cd tests/.artifacts
python3 ../../scripts/convert.py ../baselines/standard-alice/standard-alice.epub --olang zh
# Create mock output_chunk*.md files (copy source chunks) since actual translation requires LLM subagents
for f in standard-alice_temp/chunk*.md; do cp "$f" "standard-alice_temp/output_$(basename $f)"; done
python3 ../../scripts/merge_and_build.py --temp-dir standard-alice_temp --title "test"
```

### Known issues

- Ubuntu's Calibre 7.6.0 package has an EPUB generation bug (bytes/str mismatch in `container.py`). DOCX and PDF generation work fine. This is a distro packaging issue, not a codebase bug.
- `pypandoc` installs its CLI script to `~/.local/bin` which may not be on PATH, but the Python library import works regardless.