ccr git:20260726.12dea82A

Immutable. This exact content is served forever at /api/v1/blob/3d2166777dc1dac6.

---
name: ccr
description: >-
  ccr — reversible payload compression: shrink any blob (tool output, log, RAG
  chunk, file) BEFORE it enters an LLM's context, while caching the full
  original locally so it can be expanded byte-for-byte on demand. The
  payload-axis counterpart to a knowledge-graph loader (the retrieval axis):
  the model sees a compact lossy view + a handle (ccr://<sha256>), and calls
  retrieve(handle) only when it needs the bytes back. Content-routed
  deterministic compactors (json skeleton / code outline / text head-tail);
  stdlib-only, no ML. Lifted from the CCR component of github.com/chopratejas/headroom.
  USE WHEN — a tool output / log / RAG chunk / file is too large for context and
  you want to compress it reversibly; "compress this payload", "shrink this
  before the model", "reversible compression", "cache the original and give me a
  handle". NOT FOR — loading knowledge-graph entities (that's the kg loader,
  the retrieval axis); lossless whole-file compression (use gzip); semantic
  summarization that does not need byte-exact recovery.
---

# ccr — reversible payload compression (the payload axis)

`ccr compress <file|->` · `ccr retrieve <handle>` · `ccr stats`

Shrink any blob — a tool output, a log, a RAG chunk, a file — **before** it
enters context, while keeping the original recoverable. The **payload-axis**
counterpart to a knowledge-graph loader (the **retrieval axis**): kg shrinks
*which* entities reach the model; ccr shrinks *each blob* that does.

Lifted from the **CCR / reversible-compression** component of
[Headroom](https://github.com/chopratejas/headroom) (28.5k★, Apache-2.0). We
lifted the *pattern*, not the dependency. Anchor: **BRO-1521**.

## The one idea

> Compression is reversible because the **full original is cached locally**,
> keyed by content hash. The model sees a compact lossy *view* + a handle
> (`ccr://<sha256>`); it calls `retrieve(handle)` only when it needs the bytes.

So the view can be aggressively lossy without losing information — the loss is
recoverable on demand. Same shape a KG loader uses on the other axis: a lossy
catalog projection that expands to the full entity body on load.

## Mechanism

Dispatch by detected content type (a file-extension hint beats the content
heuristic, which beats nothing):

| Type | Compact view | Reversible? |
|---|---|---|
| `json` | type/shape skeleton (keys + value types + collection sizes; **no leaf data**) | yes (cache) |
| `code` | structural outline (imports + `def`/`class`/`fn` signatures w/ line numbers; **bodies elided**) | yes (cache) |
| `text` | head + tail **by line**, then head + tail **by character** on any over-long line | yes (cache) |

Deterministic, **stdlib-only, no ML** — the value is the reversible-cache
architecture, not the compression ratio. Identical payloads are
content-addressed (idempotent: one handle, one record). The view is never
emitted larger than the original (tiny inputs fall back to the payload).

**Two budgets, and they apply on both line-oriented paths.** `--head`/`--tail`
bound the number of lines; `--line-budget` (default **2000 chars**, `0`
disables) bounds each line. Without the second, a payload that is *one enormous
line* — a minified or **truncated** JSON blob, a single-line dump — compressed
to exactly 0%, because `len(lines) <= head + tail` returned the whole thing as
the "compact view". Truncated JSON lands here by construction: `detect_type`
needs a *successful* `json.loads`, so a streamed/cut-off document routes to
`text`.

The char budget applies to the **code** outline too. It initially did not:
`compact_code(payload, **_)` swallowed `line_budget`, so a minified JS/TS
bundle — one enormous line that still matches `_DEF_RE`, and the canonical
artifact this skill exists to shrink — emitted that whole line as its "outline"
and still compressed to 0.0%, with no flag able to work around it. The same
defect, surviving on the sibling path. Signature lines and the no-structure
`compact_text` fallback now both carry the caller's budgets.

## Guarantees

Each row names the mechanism that holds the invariant, and is backed by a
mutation-proven test — revert the fix and the named test fails.

One scope note, stated rather than glossed. `_is_confined`'s `is_symlink()`
guard is **defence-in-depth with no reachable behavioural difference**: delete
it and the suite stays green, because content-addressing means a symlink can
only ever serve content whose sha256 matches the requested handle, and no
content hashes to the link's own name. It is an *equivalent mutant*, not an
untested guard, so no test is claimed for it.

That distinction is the point. An earlier revision of this file asserted
"each is backed by a mutation-proven test" as a blanket claim; adversarial
review showed it was false for the hex-validation row, whose *outcome* was
upheld by a neighbouring guard while its stated **ordering** went unpinned.
That gap is now closed by
`test_malformed_handle_is_refused_BEFORE_the_filesystem_is_touched`, which
distinguishes the two by pointing the cache at a nonexistent directory so the
guards would give different errors. A blanket "everything is proven" line is
itself an unverified claim — the exact failure mode this skill's review process
exists to catch.

| Invariant | What holds it |
|---|---|
| **Byte-exact.** `retrieve` returns the input's exact bytes: CRLF, lone CR, missing trailing newline, NUL, BOM, 4-byte UTF-8, lone surrogates, and raw non-UTF-8 bytes all survive. | `compress` reads **bytes** (`read_bytes` + `surrogateescape`) — `read_text` applies universal-newline translation, which rewrote `\r\n` → `\n` *before hashing*, destroying the original. `retrieve` writes through `stdout.buffer`. |
| **Confined.** A handle is `[0-9a-f]{8,64}` and nothing else, validated **before** it touches the filesystem; the resolved record must be a real file inside the cache dir. | Handles come back from *model output* — the untrusted side. `Path.__truediv__` silently discards the cache dir for an absolute token and keeps `..` for a relative one, and a `<sha>.json` symlink can point anywhere. |
| **Atomic.** A concurrent reader never observes a partial record. | Records are published by temp-file + `os.replace`, never streamed into the final name. |
| **Verified.** Every read recomputes the sha256 and refuses a mismatch. | A content-addressed store that hands back unverified bytes is not content-addressed. |
| **Self-healing.** A truncated or tampered record is treated as *absent*, so the next `compress` of that payload rewrites it. | The old `if not rec_path.exists()` short-circuit made a partial record permanent. |
| **Private.** Cache dir `0700`, records `0600`. | The cache holds full plaintext originals. |
| **Linear.** Both code regexes are line-bounded (`[ \t]*`, `[^\n]*`). | `^\s*` matches `\n`, so `_IMPORT_RE` ran to EOF and backtracked at every line start — quadratic (32KB of blank lines took 5.6s), reachable from `--type auto`. |

`--json` is a **machine envelope**: always pure ASCII, so it stays pipeable even
when the payload holds bytes no JSON reader could otherwise decode.

**Addressing.** For valid-UTF-8 input the handle is exactly
`sha256(<the file's bytes>)`. Input that is *not* valid UTF-8 is addressed by
its `surrogateescape`-decoded form instead, so the handle is not
`sha256sum` of the file — this keeps the digest collision-free (a
`surrogateescape` digest maps a lone-surrogate string and a file holding the
corresponding raw bytes onto the same handle). Byte-exact recovery holds either
way.

## Usage

```bash
python3 scripts/ccr.py compress path/to/big.json          # view on stdout, savings on stderr
cat huge.log | python3 scripts/ccr.py compress - --type text --head 30 --tail 10
python3 scripts/ccr.py compress minified.json --line-budget 4000   # per-line char budget
python3 scripts/ccr.py retrieve ccr://<sha256> > out.bin  # byte-exact original (full handle or unique prefix)
python3 scripts/ccr.py stats --json                       # cache size + cumulative savings
```

Exit codes: `0` ok · `1` no such / malformed handle, bad payload, user error ·
`2` internal error.

As a library:

```python
import ccr
r = ccr.compress(payload, filename="server.ts")   # auto-detects code
context_blob = r["view"]                            # feed this to the model
original = ccr.retrieve(r["handle"])                # expand on demand
```

`BROOMVA_CCR_HOME` relocates the content-addressed cache (default
`~/.cache/broomva/ccr/`, created `0700`) for CI runners and non-standard layouts.

## Tests

```bash
python3 -m pytest skills/knowledge/ccr/tests/ -q     # 49 tests
python3 skills/knowledge/ccr/tests/test_ccr.py       # no pytest needed
```

Core invariant under test: the view is lossy, but `retrieve(handle)` returns the
original **byte-for-byte** — asserted **through the CLI** over a byte-level
fixture matrix (`\r\n`, lone `\r`, no trailing newline, NUL, BOM, 4-byte UTF-8,
lone surrogate, raw non-UTF-8) as well as through the Python API. Plus:
64-char traversal/absolute/symlink handle rejection, cache atomicity + digest
verification + self-healing, `0700`/`0600` permissions, ReDoS bounds on **both**
code regexes, the character budget, head/tail clamping, and deep-JSON recursion.

Tests assert at the layer that can actually break. `test_small_text_not_mangled`
asserts `compact_text` **directly** — routed through `compress()` it cannot
fail, because the `len(view) >= len(payload)` clamp rescues even a deleted
compactor. Likewise the traversal tests assert `_resolve_sha` directly, since
the downstream digest check would otherwise mask an escape.

## Install

```bash
npx skills add broomva/skills --skill ccr
```

## Provenance

- Origin: `/checkit github.com/chopratejas/headroom` → entity `tool/headroom`
- Pattern sibling: the `kg` loader (retrieval axis) / `llm-as-index-architecture`
- Ticket: BRO-1521