v0.20.0 to v0.21.0

- description: Deep reference for Lance v13 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.

- version: "0.20.0"

- upstream: "lance-format/lance@v13.0.0-beta.4"

- # Lance v13 reference

- This skill tracks **`v13.0.0-beta.4`** (the `lance-format/lance` git tag), the current

- development frontier; **`v12.0.0`** is the stable pin, released 2026-09-17. Pin against tags, not

- | `format-file.md` | What Lance is, the 26 crates, file format, data types | 1-4 |

- | `changelog-v7-v13.md` | The full v7 -> v13 delta | 14 |

- 26 crate directories under `rust/`. **`lance` is the public entry point** - `Dataset`, scanner,

- beneath it. Edition 2024, MSRV 1.91.0, arrow 58, datafusion 54; Python bindings need 3.10+. Full

- table with roles, versions, and every workspace dep in `references/format-file.md` section 2.

-

- **If you depend on anything below `lance`, v11 will break you** - PRs #8020-#8026 deleted

- `lance-encoding::version` with no re-export (`LanceFileVersion` and `ConcreteFileVersion` both

- live in `lance-file::version` now), removed `lance_io::encodings` and the `previous` namespaces,

- and gave each current format its own `versions/v2_{0,1,2,3}` module. Section 2.1.

- The transaction code moved too (#8053/#8054/#8056): `rust/lance/src/dataset/transaction.rs` is

- **deleted**, replaced by a `rust/lance-table/src/transaction/` module tree (`builder`,

- `conflicts`, `operation`, `proto`, `manifest_build`, `validate`, `index_maintenance`,

- `row_version`, `update_map`). A `lance::dataset::transaction` shim still re-exports `Operation`,

- `Transaction`, `TransactionBuilder`, `RewriteGroup`, `UpdateMap` and friends, so the common

- surface is unbroken - but a symbol the shim omits, or a citation of the old path, needs

- retargeting.

- at creation - but as of `v12.0.0` it is **no longer fixed once the dataset exists**. It is the

- *default* for writes that omit a target, not a summary of what the dataset holds: "Create and

- overwrite establish this default; append, update, merge-insert, and compaction do not change it."

- An existing V2 dataset can take `"2.0"`, `"2.1"`, `"2.2"` or `"2.3"` per operation without

- rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2

- versions. **V1 and V2 still cannot be mixed.** Section 3.

- | `2.3` | unstable (`next`) | The current `next` alias target (`V2_3` in the enum). Ships **sparse structural pages**, which the 2.3 writer now auto-selects under a rep/def budget heuristic |

- **`stable` now resolves to 2.2, not 2.1** (#8657, beta.15), and `2.2` is the enum `#[default]`,

- so a dataset created without an explicit `data_storage_version` is written as 2.2. The change

- reaches new-dataset creation through `DataStorageFormat::default() -> stable_file_version()`,

- and Python's `write_dataset` inherits it because its default routes through `stable`. **The docs

- were not updated with it** - `format/file/versioning.md` still only says `stable` is an "alias

- for the default version", so the code is the authority here. `next` resolves to 2.3. Pin an

- explicit number for deterministic behavior across builds.

- 2.3 is the only version the code flags unstable; 2.2 never was, and is now what you get by

- default. The release *selectors* (`LanceFileVersion`) are a type distinct from the persisted

- identity (`ConcreteFileVersion`). Details, plus the sparse auto-selection rules, in

- `references/format-file.md` sections 3.1 and 3.6.

- change landed", not a redesign, and a `!` without the label bumps nothing. It has now fired on

- **four consecutive lines**, which is why **none of `v9.1.0`, `v10.1.0`, `v11.1.0`, or `v12.1.0`

- was ever released**. The 12.1 line is the clearest case: `main` took a

- `chore: bump main to 12.1.0-beta.0` commit, and four commits later the bot re-rooted to 13, so

- `release-root/12.1.0-beta.N` and `release-root/13.0.0-beta.N` are the **same base commit**

- (`c3c9632a2`) and no `v12.1.0-beta.*` tag exists.

- Three recent lines **did** ship a final: `v10.0.0` (2026-08-08), `v11.0.0` (2026-08-30) and

- `v12.0.0` (2026-09-17). Each sits on a stabilization branch that is **not an ancestor of `main`**

- - normal for a Lance final, not a sign the release is unofficial.

- | **v13** (current, `v13.0.0-beta.4`) | `WriteParams` gained `file_writer_options` (#9192 - the one labeled PR that re-rooted the major); lazy page-metadata init changed the `StructuralFieldScheduler` signature and the metadata **cache key shape** (#7465); `json_extract`/`json_get` no longer route to JSON indices (#9101). Delta below |

- | **v12** (`v12.0.0`, 2026-09-17) | `WrappingObjectStore` implementors must add `wrap_paginated` (no default); MemWAL `ShardManifestStore` renamed and narrowed; `lance-namespace` returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list; the caller-provided Writer / `open_part` flow was removed (#9072). Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Net-new format capability: mixed data-file versions. Delta below |

- | **v11** (`v11.0.0`, 2026-08-30) | Fragment ids became a dataset-lifetime high-water mark; large internal reorganization of `lance-file` / `lance-encoding`; the first new manifest feature flag since v7 - which was then **reallocated before the final**. Net-new: covering indexes, `merge_insert` `write_mode`, row-address prefilter. Delta below |

- | **v10** | Blob APIs preserve null selections; cache keys became opaque BLAKE3 digests (every warm or persisted cache cold-misses, no legacy fallback); async `create_remapper`; MemWAL renamed generation -> SSTable, merge -> compaction (wire-compatible, symbol-breaking) |

- | **v9.1** (never released; renamed into v10) | FTS/inverted creation took a `block_size` param. Net-new: Data Overlay Files (cell-level updates without base-file rewrite, unstable + env-gated), sparse structural pages, `lance-index-core` |

- | **v9** | Python 3.9 dropped; `alter_columns` fails fast when casting an indexed column; FM-Index proto rename made existing FM indexes unreadable; FTS/inverted defaults to on-disk format v2 |

- | **v8** | All index builds unified onto one segment-based lifecycle. Net-new: `lance-derive`, FM-Index, multi-bit IVF_RQ, public `approx_mode`, TOS + GooseFS object stores |

- **`v12.0.0` is the stable pin** and what GitHub Releases marks `Latest`. crates.io carries

- **finals only** (newest `lance 12.0.0`, no 13.x); PyPI `pylance` is likewise at `12.0.0`. So a

- beta pin means a git dependency - beta wheels publish to fury.io instead, under the renamed org

- (`https://pypi.fury.io/lance-format`), which currently carries `pylance-13.0.0b1` through `b4`.

-

- Full per-tag deltas with every PR citation: `references/changelog-v7-v13.md`.

-

- ## The v11 delta

-

- 357 commits from `v10.0.0-beta.7` to the `v11.0.0` final, with **16 `breaking-change`-labeled

- PRs** (14 through `beta.16`, plus #8407 and #8535 in the final). Most structural invariants held:

- **26 crates**, **16 transaction ops**, `CommitConfig.num_retries` **20**, arrow 58 / datafusion 54,

- MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at `v13.0.0-beta.4`.

-

- **`references/changelog-v7-v13.md` has the full delta** - every PR citation, the per-tag

- breakdown from v7 forward, the Python/Java surface, and each correctness fix with its trigger

- condition. Load it for any "what changed / will this break me" question. What follows is only

- what bites hardest.

-

- **Five things that break you at v11:**

-

- - **Fragment ids are a dataset-lifetime high-water mark** (#8206) - a *format* invariant, not

- just an API. Overwrite no longer restarts ids at 0, an overwrite fragment carrying a deletion

- file is rejected, and any commit producing duplicate ids is rejected - so datasets written by

- Lance 0.16 and earlier may still read but no longer commit. `dataset.get_fragment(0)` after an

- overwrite must read ids from the manifest. Section 5 - which also covers a resolution hazard on

- pre-0.10 unsorted manifests that can make a fragment-filtered index cover the wrong fragments.

- - **The file-version types and reader/writer composition moved** (#8020-#8026) -

- `lance-encoding::version` deleted with no re-export; `LanceFileVersion` lost `PartialOrd`/`Ord`

- (#8027, #8028), so `v >= LanceFileVersion::Next` no longer compiles. `FileWriter` is now an

- enum with all constructors removed. Most of these break silently at compile time. Section 3.6.

- - **Transaction code moved to `lance-table`** (#8053/#8054/#8056) - see the crate-workspace note

- above; the `lance::dataset::transaction` shim covers the common surface.

- - **`Operation::Project` / `Merge` gained `preserves_nullability`** (#8347) - a nullability

- *tightening* must not set it, and such a projection now conflicts with any concurrent

- value-write. This closed a real hole where `alter_columns` could let a racing write land nulls

- unreadable under the tightened schema. Section 9.2.

- - **The external-manifest protocol changed** (#8499) - object storage is authoritative, the

- external store's put-if-not-exists is a *reservation*, and a stored ETag must be **ignored**;

- a retained one makes readers reject a good manifest with `Manifest e_tag mismatch`. Section 9.

-

- **The manifest feature flags changed - and bit 128 was reallocated before the final.** v11 added

- the first new bit since v7 and moved `FLAG_UNKNOWN` 128 -> 256. But the bit it added,

- `FLAG_MEM_WAL_INDEX_CATCHUP`, was **retired again** (#8680) and the reclaimed bit handed to

- `FLAG_COVERED_INDEX_METADATA = 128` (#8535) before `v11.0.0` shipped. At the final and at v12

- there is no index-catchup flag and no `require_index_catchup` proto field; a shard absent from

- `index_catchup` now unconditionally means *unknown*. Both reader and writer must hold bit 128 or

- refuse the table. Section 7.

-

- **Do not pin anywhere in `v11.0.0-beta.4` through `beta.17`.** Those builds treat bit 128 as a

- MemWAL flag they support, so they *open* a covering-index dataset instead of refusing it - wrong

- neighbours, no error. The exposure is inherited by whichever flag takes the bit.

-

- **Covering indexes are the v11 net-new format feature** (#8535), **redefined at v13** (#8856).

- `IndexMetadata.covering_fields` (proto field 11) names the columns an index *carries* values for,

- so a query projecting only those columns is answered without a base-table take. It is **no longer

- a trailing suffix of `fields`**: it "must be a subset of `fields`, in the order the index emits

- them. A column is carried if and only if it is named here", including a column the index is also

- keyed on - and `fields[0]` remains a keyed column. Index invalidation stays wide: **any** index

- whose `fields` include the updated column, "whether the index is keyed on it or merely carries

- it".

-

- The old "no index builder writes carried values yet" no longer holds. V3 IVF auxiliary files can

- physically carry columns, and "a reader discovers carried columns by exclusion, not by position:

- any column in the auxiliary file's schema that is not one of the quantizer's internal columns is

- a carried column", bound to source fields by a new `covering_field_ids` metadata key. Coverage is

- now per-segment, not per-index: "one logical index may hold values for some of its segments and

- not others". `VectorQueryProto.covering_projection` (field 15) reserves the query-side tag, where

- absent / present-and-empty / present-and-non-empty are three distinct meanings. Section 11.

-

- **Bit 8 was spent in the `v12.0.0` final.** `FLAG_MIXED_DATA_FILE_VERSIONS = 1 << 8` (256) is no

- longer a reservation pinned equal to `FLAG_UNKNOWN`: the assert relaxed to

- `FLAG_MIXED_DATA_FILE_VERSIONS < FLAG_UNKNOWN`, `FLAG_UNKNOWN` moved `1 << 8` -> `1 << 9` (512),

- and the build now both reads and writes mixed-version datasets. It is still carried by

- `STICKY_PAIRED_FLAGS`, and a **half-set** manifest is now a hard error: "Manifest has only one of

- the mixed data-file-version reader and writer feature bits set, so its semantics are undefined".

- Section 7.

-

- **Bit 1024 is where the docs and the code disagree - trust the code.**

- `FLAG_FRAGMENT_REUSE_INDEX = 1 << 10` is declared at `rust/lance-table/src/feature_flags.rs:69`

- and, at `v13.0.0-beta.4`, **that declaration is its only occurrence in the entire tree**. It sits

- *above* `FLAG_UNKNOWN` (512), and the supported set is computed as `FLAG_UNKNOWN - 1`, so a

- manifest setting it is **refused**. The spec page meanwhile lists it as reader `Yes` / writer

- `Yes` and puts the unknown boundary at 2048. The docs describe the intended end state; the code

- has only reserved the constant. Anything you build against tagged FRI today is building against

- prose, not behavior.

-

- **Two `LANCE_*` env vars landed** (from the AMX work, #8540): `LANCE_DISABLE_AMX` (runtime kill

- switch) and `LANCE_AMX_FP16_CC` (build-time compiler override). Grep trap: `LANCE_AMX_CFG_*` and

- `LANCE_AMX_TILE_COUNT` are **C macros in `amx_fp16.c`, not env vars**, and `LANCE_FACTOR` is a

- substring of `BALANCE_FACTOR` - a plain `LANCE_*` grep reports all four as if they were real.

- **Worth knowing without reading the full delta:** FTS gained a document-boundary axis

- (`DocumentGranularity`, #7788) whose `list_element` mode is a third trigger requiring FTS on-disk

- format v3; transactions above **20 MiB** spill out of the manifest entirely (#7881); MemWAL

- catch-up became derived rather than declared (#8481); transaction proto field 9 is deprecated for

- field 10 (#7432); compaction gained row/byte budgets plus fragment exclusion (#8235, #8532);

- `merge_insert` gained `write_mode` (#8423); and Python commit conflicts became

- `lance.commit.CommitConflictError`, a subclass of `OSError`, so existing handlers keep working

- (#8563). Full list with citations in `references/changelog-v7-v13.md`.

- **Address-domain indexes stopped falsely claiming compacted fragments** (v11, `beta.16` or

- earlier). A rewrite used to advance *every* index's `fragment_bitmap` onto the new fragment ids -

- including ZoneMap, whose stored addresses point into the fragments the rewrite dropped. The

- Rewrite path now branches on `results_are_row_addrs()`: an address-domain index gets

- `drop_rewritten_fragments` and a full-scan fallback, correct-but-slower instead of stale

- addresses. **Heals only for new compactions**: an index already damaged under v10 or earlier must

- be recreated, and the damage does not self-heal through routine maintenance, because the

- refreshed `fragment_bitmap` also makes incremental folds a no-op. Section 11.

- **Correctness fixes split by whether upgrading is enough.** Most are read-path only and heal on

- upgrade. These do **not** - they need data rewritten or repaired: #8382, #8669, #8509, #7703,

- #8539, #8459, #8378, #8482, #8834 (rebuild HNSW - a persisted graph can hold edges to ids it does

- not contain; lost recall stays lost), #8101 (**nullable primary keys silently duplicated rows** on

- every repeat `merge_insert`; existing duplicates must be removed by hand), #8511, #8427, #8513,

- #8839, #8904. Conditions for each in `references/changelog-v7-v13.md`.

- ## The v12 delta

- **225 commits** from `release-root/12.0.0-beta.N` to the `v12.0.0` final, with **7

- `breaking-change`-labeled PRs** - the five visible at beta.15 plus **#9072** and **#9101** in the

- run-up to the final. No new index types and no new crates; every structural invariant above still

- holds. **The label is a floor, not a ceiling** - the two biggest behavior changes in the line

- carry a conventional-commit `!` but no label, so the bot never counted them: the `stable` -> 2.2

- move (#8657, above) and the IVF_RQ 5-bit default (below).

- - **`WrappingObjectStore` implementors must add `wrap_paginated`** (#8606) - "There is

- deliberately no default: getting this wrong is either a silent loss of speed or a silent loss

- of the wrapper, and neither announces itself." Return `Some` to keep listing pushdown through

- the wrapper, `None` to give it up and fall back through `inner`. One wrapper giving it up gives

- it up for the whole chain. Anything wrapping the object store fails to compile until updated.

- - **New paged listing: `ObjectStore::read_dir_page`** (#8606) - one page of a prefix's immediate

- children plus an opaque resume token. The trap: "One page is one request, so a page can hold

- fewer children than `limit` asked for and still be followed by more" - walk until the token is

- `None`, never until a page comes back short.

- - **MemWAL `ShardManifestStore` renamed and narrowed** (#8640) - `read_latest` -> `latest`,

- `read_latest_uncached` -> `refresh_latest`, and `write` is now crate-private (reach it through

- `commit_update`, `claim_epoch`, or `initialize_shard`). Existing `commit_update` closures need

- no change. Section 10.

- - **`lance-namespace` 0.8.5 -> 0.11.1** (#8903) - four `LanceNamespace` methods now return

- response objects instead of bare values: `count_table_rows` -> `CountTableRowsResponse`,

- `query_table` -> `QueryTableResponse`, `namespace_exists` / `table_exists` -> their own

- response types. Callers unwrap; anyone implementing the trait needs the same signature updates.

- - **External manifest stores gained predecessor-conditioned publication** (#8800) -

- `put_if_predecessor` reserves a version only while the predecessor still carries the identity

- the writer observed, and `commit_after` refuses with `PrerequisiteFailed`, "never a conflict".

- The hard compile break is the new `ManifestLocation.identity` field, not the trait methods

- (all default-implemented). No built-in store implements it. Section 9.

- - **Namespace merge-insert keys became a list** (#8915) - `on` moves from `Option<String>` to

- `Option<Vec<String>>`, with **arity-dependent NULL semantics**: a single-column key treats NULL

- as equal to NULL, a composite key uses SQL equality, "under which a NULL key matches nothing -

- not even a byte-identical NULL".

- **The `lance-namespace` pin is no longer one number.** #8915 moved the Rust client to **0.12.0**

- and #8979 moved **Java** to 0.12.0 as well; only **Python** still holds `>=0.11.1,<0.12`, because

- its generated models still send `on` as a bare string. Quote a language-specific pin, never one

- number for all three - and note this split moved once already, so re-check it rather than

- carrying the pairing forward.

- **IVF_RQ now defaults to 5 bits per dimension, not 1** (#8936) - roughly a **4.4x index-size

- increase** at the default (upstream's 100M x 768d example: ~10.8 GiB -> ~47.3 GiB). `Fast` search

- mode "uses only the 1-bit sign code even when the index stores additional bits", so it pays the

- storage without using it; set `num_bits=1` explicitly to opt out, at the cost of the multi-bit

- distance estimate and some recall. Sizing formulas in `references/indexes.md`.

- **Column slice stitching (#8660) was reverted** at beta.9 (#8926) - it "should not ship while the

- caller-managed replacement in #8923 is being developed". `rust/lance-file/src/concat.rs` exists

- again at beta.15, but holds #8923's caller-managed data file parts, not the reverted stitching.

- **Two proto additions.** MemWAL `SsTable` gained `in_memory_bytes`, `physical_rows` and

- `primary_key_bytes` (fields 3-5, #8981); all optional, and **absence must not be read as zero**.

- `FilteredReadOptions` gained `materialization_readahead_bytes` and `batch_size_bytes` (13, 14) -

- not a format change under a new rule in `protos/AGENTS.md`: execution-plan schemas "are wire

- contracts, not persisted Lance formats". `transaction.proto` / `ann.proto` / `index.proto` are

- untouched.

- **Net-new, non-breaking:** provider-native bulk copy and a deep-clone concurrency bound (section

- 13); Python `ObjectStoreProvider` registration (#8522); `BinaryView` in the packed blob writer

- (#8700); caller-managed data file parts (#8923); cleanup of specific versions (#8617);

- `LanceDataset.slice()` (#8059) and six more `LanceFragment.scanner` options (#8429); restored

- Python index retraining (#8786); namespace-managed clone deprecated to a shim (#8964). Namespace

- latest-version resolution no longer lists the whole `_versions/` prefix (#8679) - on a

- ~340k-version table that was ~344 list pages, "~25s of pure I/O wait", paid by every open.

- **Fixes needing a rebuild or rewrite, not just an upgrade:** #8779 (rebuild NGRAM indexes), #8510

- (rewrite data compacted from uniformly reordered fragments), #8984 (re-drop a resurrected index),

- #8837 (repair a MemWAL shard below ~2.7KB/row - it cannot be reopened). Full per-PR conditions,

- plus the much longer list that *does* heal on upgrade, in `references/changelog-v7-v13.md`.

- **Mixed data-file versions LANDED** - it is no longer "1 of 6". #8581-#8584 shipped in `v12.0.0`

- (validation, per-operation V2 write targets, propagation across dataset operations, compaction

- targeting) and #8585 exposed it in the bindings in the v13 line. The proto changed with it:

- `DataStorageFormat.version` is now "the default format version used when writing data files",

- and "each DataFile's version is authoritative for decoding" once the capability is set.

- **In flight, not landed - do not treat as shipped:** generic block v5 compression is **still 1 of

- 10** PRs merged (#8324; #8325-#8333 all remain open). Next big dependency break in the queue:

- #8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" - **still open** at `v13.0.0-beta.4`,

- so arrow 58 / datafusion 54 still hold. It also gates two outstanding PyO3 advisories

- (RUSTSEC-2026-0176/0177); rustls was separately patched to 0.23.45 for RUSTSEC-2026-0285 (#9212).

- ## The v13 delta

- **66 commits** from `release-root/13.0.0-beta.N` to `v13.0.0-beta.4`, with **2

- `breaking-change`-labeled PRs**. No new crates and no new index types; 26 crates, 16 transaction

- ops, `CommitConfig.num_retries` 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024 and

- Python 3.10+ all still hold.

- **The `!`-vs-label rule inverted in this window.** All three conventional-commit `!` commits

- (#7465, #9192, #9101) *do* carry the `breaking-change` label. Keep treating the label as a floor

- rather than a ceiling - but this window is the counter-example, not more evidence for the gap.

- - **`WriteParams` gained `file_writer_options`** (#9192) - the single labeled PR that re-rooted

- the major. `FileWriterOptions { data_cache_bytes, max_page_bytes, keep_original_array }` is now

- reachable from the dataset write APIs in Rust, Python and Java. A zero `max_page_bytes` is

- rejected before encoder construction rather than misbehaving later. Anything constructing

- `WriteParams` by struct literal fails to compile.

- - **Page metadata is initialized lazily, and the metadata cache key changed shape** (#7465).

- `StructuralFieldScheduler::initialize` now takes `requested_ranges`, and the page-scheduler

- `initialize` splits into `init_ranges()` and `init_from_buffers(buffers, io)` - any external

- implementor fails to compile. The public `DecodeBatchScheduler::try_new` kept its signature;

- the range-aware entry point is the crate-private `try_new_with_ranges`. The part that bites

- without a compile error: caching moved from a per-column `FieldDataCacheKey` to a per-page

- `PageDataCacheKey { column_index, page_index, view_tag }`, so **every warm or persisted

- metadata cache cold-misses** across this upgrade. The payoff is real - "a cold point/range

- read's metadata IO is invariant to the column's total page count".

- - **`json_extract` and `json_get` no longer route to JSON indices** (#9101). Only the four typed

- accessors (`json_get_int` / `_float` / `_bool` / `_string`) reach the index; everything else

- falls back to a full scan. This fixes three real wrong-answer bugs - a quoted-key mismatch that

- "searched for a quoted key and matched nothing", a `Utf8` literal driving an `Int64` btree into

- a panic, and an unsound range because "quoting is not order-preserving (`ab` < `ab!` but

- `"ab"` > `"ab!"`)". **The cost is silent**: a `json_extract` workload that used to hit an index

- now scans, with no error and no plan warning. Rewrite those predicates onto the typed

- accessors.

- **The Fragment Reuse Index gained a versioned on-disk contract** (#9136). `InlineContent` field 1

- was renamed `versions` -> `legacy_versions` and a tagged `transitions` list added at field 2,

- gated on `index_version >= 1`; mappings are now a oneof of `OrderedCompaction` or

- `StablePartition`. A stable partition "assigns source rows to destination fragments while

- preserving their relative source order within each destination", which lets FRI reuse existing

- indices after **reclustering** - a second use case the v0 model had no concept of. Its physical

- form is an immutable row-map Lance file with `uint16` labels and an `LSPC`-magic counts matrix.

- Two hard rules: **stable row IDs and tagged FRI are mutually exclusive** ("writers must not

- publish `index_version >= 1` on them"), and cleanup "must retain intermediate transitions still

- needed to translate old addresses". Upstream also softened the old claim - "FRI does not remove

- conflicts between overlapping rewrites". Section 11.

- **Net-new, non-breaking:** `Dataset::frag_reuse_index()` is public (#9112) and documented in the

- performance guide; `FileFragment::write_overlay` returns a real `OverlayWriter` (#8761, still

- env-gated); an `hf://` object store with `hf_enable_resolve_cache` (#9236); Python

- `lance.bitmap.Bitmap`, `deep_clone()` (#9181), `base_paths()` (#9191) and

- `update_columns(with_offsets=True)` (#8891); Java `DataStorageVersion`, `FileWriteOptions` and

- `ScanOptions.indexSegments`; and namespace table listing finally bounded by `read_dir_page`

- (#9165). `inline_optimization_enabled` **flipped `true` -> `false`** (#9180), which upstream

- justifies with a -49% write-p50 measurement at 1M entries.

- section (v11, unchanged at v12) gives a starting point for cross-region or public-internet

- access, where the cloud default of 64 concurrent requests is too aggressive: `LANCE_IO_THREADS=8`,

- **AMX-FP16** (#8540, beta.16) is the one v11 performance change that alters *results*, not just

- speed: where it engages, IVF partition assignment becomes **exact instead of approximate**, so

- recall improves *and* assignments differ from an older build. It is shape-gated (`float16` +

- `dot`, `dimension >= 32`, `num_centroids >= 32`); everything else keeps the previous path.

- `LANCE_DISABLE_AMX=1` disables it, but reverts assignment to the approximate path too - so an

- index built with it set is not equivalent to one built without it.

- remedy. Note that **#7465 changes the metadata cache key shape**, so the first run after a v13

- upgrade re-pays that paging even against a warm or persisted cache. Details and build-time

- requirements in `references/performance.md`.

- explicit archive if you need one.

- 45 markdown files plus 4 diagrams, all directly readable.

- | `format/table/` | 9 | Layout, schema, transactions (**conflict-resolution matrix**), versioning, row-id lineage, branch/tag, MemWAL, data overlay files |

- | `format/index/` | 1 + 4 svg | Index lifecycle, fragment coverage, compaction interplay |

- | `format/index/scalar/` | 9 | fts, fmindex, ngram, btree, bitmap, bloom_filter, label_list (`array_has_any/all`), zonemap, rtree |

- **Not mirrored:** `docs/src/images/` (PNG/GIF assets), so image links in the mirrored pages do

- not resolve - the prose is self-contained, and the four `.drawio.svg` diagrams *are* mirrored.

- Also out by design: `community/`, `examples/`, `integrations/{index,pytorch,tensorflow}.md`; and

- the landing stubs and contributor files (`format/AGENTS.md`, `format/CLAUDE.md`).