v0.18.1 to v0.19.0

+ version: "0.19.0"

+ upstream: "lance-format/lance@v12.0.0-beta.15"

+ This skill tracks **`v12.0.0-beta.15`** (the `lance-format/lance` git tag), the current

+ | `2.1` | previous default | Adaptive structural encodings; better integer/string compression; nulls in struct fields; better nested random access. Was the default from Lance 5.0.0 until `v12.0.0-beta.15` |

+ | `2.2` | **current default** (`stable`) | Map type, Blob v2, `VariablePackedStruct`, larger mini-blocks. Required for Map and Blob v2 |

+ **`stable` now resolves to 2.2, not 2.1** (#8657, beta.15), and `2.2` is the enum `#[default]`,

+ so a dataset created without an explicit `data_storage_version` is written as 2.2. The change

+ reaches new-dataset creation through `DataStorageFormat::default() -> stable_file_version()`,

+ and Python's `write_dataset` inherits it because its default routes through `stable`. **The docs

+ were not updated with it** - `format/file/versioning.md` still only says `stable` is an "alias

+ for the default version", so the code is the authority here. `next` resolves to 2.3. Pin an

+ explicit number for deterministic behavior across builds.

+

+ 2.3 is the only version the code flags unstable; 2.2 never was, and is now what you get by

+ default. The release *selectors* (`LanceFileVersion`) are a type distinct from the persisted

+ identity (`ConcreteFileVersion`). Details, plus the sparse auto-selection rules, in

+ change landed", not a redesign, and a `!` without the label bumps nothing. It has fired on

+ **three consecutive lines**, which is why **none of `v9.1.0`, `v10.1.0`, or `v11.1.0` was ever

+ released** - the 11.1 line never got even one beta tag, and `release-root/12.0.0-beta.N` points at

+ the same base commit `release-root/11.1.0-beta.N` did.

+ Both recent lines **did** ship a final: `v10.0.0` (2026-08-08) and `v11.0.0` (2026-08-30). Each

+ sits on a stabilization branch that is **not an ancestor of `main`** - normal for a Lance final,

+ not a sign the release is unofficial.

+ | **v12** (current, `v12.0.0-beta.15`) | `WrappingObjectStore` implementors must add `wrap_paginated` (no default); MemWAL `ShardManifestStore` renamed and narrowed; `lance-namespace` returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list. Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Delta below |

+ | **v11** (`v11.0.0`, 2026-08-30) | Fragment ids became a dataset-lifetime high-water mark; large internal reorganization of `lance-file` / `lance-encoding`; the first new manifest feature flag since v7 - which was then **reallocated before the final**. Net-new: covering indexes, `merge_insert` `write_mode`, row-address prefilter. Delta below |

+ **`v11.0.0` is the stable pin** and what GitHub Releases marks `Latest`. crates.io carries

+ **finals only** (newest `lance 11.0.0`, no 12.x); PyPI `pylance` is likewise at `11.0.0`. So a

+ beta pin means a git dependency - beta wheels publish to fury.io instead, under the renamed org

+ (`https://pypi.fury.io/lance-format`).

+ PRs** (14 through `beta.16`, plus #8407 and #8535 in the final). Most structural invariants held:

+ **26 crates**, **16 transaction ops**, `CommitConfig.num_retries` **20**, arrow 58 / datafusion 54,

+ MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at `v12.0.0-beta.15`.

+ **The manifest feature flags changed - and bit 128 was reallocated before the final.** v11 added

+ the first new bit since v7 and moved `FLAG_UNKNOWN` 128 -> 256. But the bit it added,

+ `index_catchup` now unconditionally means *unknown*. Both reader and writer must hold bit 128 or

+ refuse the table. Section 7.

+ **Do not pin anywhere in `v11.0.0-beta.4` through `beta.17`.** Those builds treat bit 128 as a

+ MemWAL flag they support, so they *open* a covering-index dataset instead of refusing it - wrong

+ neighbours, no error. The exposure is inherited by whichever flag takes the bit.

+ redefines `fields` as "keyed columns followed by carried ones", and widens index invalidation to

+ **any** index whose `fields` include the updated column, "whether the index is keyed on it or

+ merely carries it". But "no index builder writes carried values yet", so this is

+ capability-in-place, not a usable speedup - and because the flag is set only while some index

+ actually carries values, it is not set in practice. Section 11.

+ *equal to* `FLAG_UNKNOWN` (a compile-time assert pins them together) and carried by a

+ `STICKY_PAIRED_FLAGS` mechanism, so the supported set is unchanged and a manifest setting the bit

+ is still refused. Only the reservation (#8580) merged. Section 7.

+ budgets plus fragment exclusion (#8235, #8532). Landing in the final: `merge_insert` gained

+ `write_mode` (`Auto` / `RewriteRows` / `RewriteColumns`, the last patching columns in place

+ through a new `InPlaceMergeInsertExec`, #8423); `Scanner::with_row_addr_prefilter(RowAddrMask)`

+ (#7288); `get_deleted_row_ids` (#8589); and Python commit conflicts became

+ `lance.commit.CommitConflictError` with a typed `retryable` attribute - a subclass of `OSError`,

+ so existing `except OSError` handlers keep working (#8563).

+ onto the new fragment ids - including ZoneMap, whose stored addresses point into the fragments the

+ rewrite dropped. The Rewrite path now branches on `results_are_row_addrs()`: a row-id-domain index

+ follows its data via `recalculate_fragment_bitmap`, an address-domain one gets

+ (`can_remap() -> false`). **Heals only for new compactions**: an index already damaged under v10

+ or earlier must be recreated, and the damage does not self-heal through routine maintenance

+ because the refreshed `fragment_bitmap` also makes incremental folds a no-op. Section 11.

+ upgrade. These do **not** - they need data rewritten or repaired: #8382, #8669, #8509, #7703,

+ #8539, #8459, #8378, #8482, #8834 (rebuild HNSW - a persisted graph can hold edges to ids it does

+ not contain; lost recall stays lost), #8101 (**nullable primary keys silently duplicated rows** on

+ every repeat `merge_insert`; existing duplicates must be removed by hand), #8511, #8427, #8513,

+ #8839, #8904. Conditions for each in `references/changelog-v7-v12.md`.

+ 170 commits from `release-root/12.0.0-beta.N`, with **exactly 5 `breaking-change`-labeled PRs**.

+ No new index types and no new crates; every structural invariant above still holds. **The label is

+ a floor, not a ceiling** - the two biggest behavior changes in the line carry a conventional-commit

+ `!` but no label, so the bot never counted them: the `stable` -> 2.2 move (#8657, above) and the

+ IVF_RQ 5-bit default (below).

+ - **External manifest stores gained predecessor-conditioned publication** (#8800) -

+ `put_if_predecessor` reserves a version only while the predecessor still carries the identity

+ the writer observed, and `commit_after` refuses with `PrerequisiteFailed`, "never a conflict".

+ The hard compile break is the new `ManifestLocation.identity` field, not the trait methods

+ (all default-implemented). No built-in store implements it. Section 9.

+ - **Namespace merge-insert keys became a list** (#8915) - `on` moves from `Option<String>` to

+ `Option<Vec<String>>`, with **arity-dependent NULL semantics**: a single-column key treats NULL

+ as equal to NULL, a composite key uses SQL equality, "under which a NULL key matches nothing -

+ not even a byte-identical NULL".

+ **The `lance-namespace` pin is no longer one number.** #8915 moved the Rust client to **0.12.0**

+ while Java and Python deliberately stay on **0.11.1** - their generated models still send `on` as

+ a bare string. Quote a language-specific pin, never one number for all three.

+

+ **IVF_RQ now defaults to 5 bits per dimension, not 1** (#8936) - roughly a **4.4x index-size

+ increase** at the default (upstream's 100M x 768d example: ~10.8 GiB -> ~47.3 GiB). `Fast` search

+ mode "uses only the 1-bit sign code even when the index stores additional bits", so it pays the

+ storage without using it; set `num_bits=1` explicitly to opt out, at the cost of the multi-bit

+ distance estimate and some recall. Sizing formulas in `references/indexes.md`.

+

+ **Column slice stitching (#8660) was reverted** at beta.9 (#8926) - it "should not ship while the

+ caller-managed replacement in #8923 is being developed". `rust/lance-file/src/concat.rs` exists

+ again at beta.15, but holds #8923's caller-managed data file parts, not the reverted stitching.

+

+ **Two proto additions.** MemWAL `SsTable` gained `in_memory_bytes`, `physical_rows` and

+ `primary_key_bytes` (fields 3-5, #8981); all optional, and **absence must not be read as zero**.

+ `FilteredReadOptions` gained `materialization_readahead_bytes` and `batch_size_bytes` (13, 14) -

+ not a format change under a new rule in `protos/AGENTS.md`: execution-plan schemas "are wire

+ contracts, not persisted Lance formats". `transaction.proto` / `ann.proto` / `index.proto` are

+ untouched.

+

+ 13); Python `ObjectStoreProvider` registration (#8522); `BinaryView` in the packed blob writer

+ (#8700); caller-managed data file parts (#8923); cleanup of specific versions (#8617);

+ `LanceDataset.slice()` (#8059) and six more `LanceFragment.scanner` options (#8429); restored

+ Python index retraining (#8786); namespace-managed clone deprecated to a shim (#8964). Namespace

+ latest-version resolution no longer lists the whole `_versions/` prefix (#8679) - on a

+ **Fixes needing a rebuild or rewrite, not just an upgrade:** #8779 (rebuild NGRAM indexes), #8510

+ (rewrite data compacted from uniformly reordered fragments), #8984 (re-drop a resurrected index),

+ #8837 (repair a MemWAL shard below ~2.7KB/row - it cannot be reopened). Full per-PR conditions,

+ plus the much longer list that *does* heal on upgrade, in `references/changelog-v7-v12.md`.

+

+ **In flight, not landed - do not treat as shipped:** generic block v5 compression (still 1 of 10

+ PRs merged, #8324) and mixed data-file versions (still 1 of 6, #8580, the reservation only). Next

+ big dependency break in the queue: #8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" -

+ open at beta.15, so arrow 58 / datafusion 54 still hold.