v0.19.0 to v0.20.0

+ description: Deep reference for Lance v13 columnar format, its Rust crates, and pylance - file encodings, table format, indexes, schema evolution, time travel. Use when building on the Lance crates or reading .lance datasets, not the LanceDB product.

+ version: "0.20.0"

+ upstream: "lance-format/lance@v13.0.0-beta.4"

+ # Lance v13 reference

+ This skill tracks **`v13.0.0-beta.4`** (the `lance-format/lance` git tag), the current

+ development frontier; **`v12.0.0`** is the stable pin, released 2026-09-17. Pin against tags, not

+ `main` - Lance ships beta tags every few days and `next`-format encodings can change. Version

+ landscape below.

+ | `changelog-v7-v13.md` | The full v7 -> v13 delta | 14 |

+ **The wrapper can hide format features.** LanceDB's `create_table` cannot enable stable row IDs;

+ only pylance's `write_dataset(enable_stable_row_ids=True)` can. If a format-level capability

+ matters to your design, check whether the wrapper exposes it before assuming the underlying

+ format settles the question - and reach for `pylance` directly when it does not.

+

+ The file format carries a single major.minor version. `data_storage_version` is set per dataset

+ at creation - but as of `v12.0.0` it is **no longer fixed once the dataset exists**. It is the

+ *default* for writes that omit a target, not a summary of what the dataset holds: "Create and

+ overwrite establish this default; append, update, merge-insert, and compaction do not change it."

+ An existing V2 dataset can take `"2.0"`, `"2.1"`, `"2.2"` or `"2.3"` per operation without

+ rewriting its other files (#8582-#8585), so one dataset can hold data files at several exact V2

+ versions. **V1 and V2 still cannot be mixed.** Section 3.

+ change landed", not a redesign, and a `!` without the label bumps nothing. It has now fired on

+ **four consecutive lines**, which is why **none of `v9.1.0`, `v10.1.0`, `v11.1.0`, or `v12.1.0`

+ was ever released**. The 12.1 line is the clearest case: `main` took a

+ `chore: bump main to 12.1.0-beta.0` commit, and four commits later the bot re-rooted to 13, so

+ `release-root/12.1.0-beta.N` and `release-root/13.0.0-beta.N` are the **same base commit**

+ (`c3c9632a2`) and no `v12.1.0-beta.*` tag exists.

+ Three recent lines **did** ship a final: `v10.0.0` (2026-08-08), `v11.0.0` (2026-08-30) and

+ `v12.0.0` (2026-09-17). Each sits on a stabilization branch that is **not an ancestor of `main`**

+ - normal for a Lance final, not a sign the release is unofficial.

+ | **v13** (current, `v13.0.0-beta.4`) | `WriteParams` gained `file_writer_options` (#9192 - the one labeled PR that re-rooted the major); lazy page-metadata init changed the `StructuralFieldScheduler` signature and the metadata **cache key shape** (#7465); `json_extract`/`json_get` no longer route to JSON indices (#9101). Delta below |

+ | **v12** (`v12.0.0`, 2026-09-17) | `WrappingObjectStore` implementors must add `wrap_paginated` (no default); MemWAL `ShardManifestStore` renamed and narrowed; `lance-namespace` returns response objects; external stores gained predecessor-conditioned publication; namespace merge-insert keys became a list; the caller-provided Writer / `open_part` flow was removed (#9072). Unlabeled but bigger: `stable` -> 2.2 and the IVF_RQ 5-bit default. Net-new format capability: mixed data-file versions. Delta below |

+ **`v12.0.0` is the stable pin** and what GitHub Releases marks `Latest`. crates.io carries

+ **finals only** (newest `lance 12.0.0`, no 13.x); PyPI `pylance` is likewise at `12.0.0`. So a

+ (`https://pypi.fury.io/lance-format`), which currently carries `pylance-13.0.0b1` through `b4`.

+ Full per-tag deltas with every PR citation: `references/changelog-v7-v13.md`.

+ MSRV 1.91.0, Edition 2024, Python 3.10+ - and all of them still hold at `v13.0.0-beta.4`.

+ **`references/changelog-v7-v13.md` has the full delta** - every PR citation, the per-tag

+ **Covering indexes are the v11 net-new format feature** (#8535), **redefined at v13** (#8856).

+ `IndexMetadata.covering_fields` (proto field 11) names the columns an index *carries* values for,

+ so a query projecting only those columns is answered without a base-table take. It is **no longer

+ a trailing suffix of `fields`**: it "must be a subset of `fields`, in the order the index emits

+ them. A column is carried if and only if it is named here", including a column the index is also

+ keyed on - and `fields[0]` remains a keyed column. Index invalidation stays wide: **any** index

+ whose `fields` include the updated column, "whether the index is keyed on it or merely carries

+ it".

+ The old "no index builder writes carried values yet" no longer holds. V3 IVF auxiliary files can

+ physically carry columns, and "a reader discovers carried columns by exclusion, not by position:

+ any column in the auxiliary file's schema that is not one of the quantizer's internal columns is

+ a carried column", bound to source fields by a new `covering_field_ids` metadata key. Coverage is

+ now per-segment, not per-index: "one logical index may hold values for some of its segments and

+ not others". `VectorQueryProto.covering_projection` (field 15) reserves the query-side tag, where

+ absent / present-and-empty / present-and-non-empty are three distinct meanings. Section 11.

+ **Bit 8 was spent in the `v12.0.0` final.** `FLAG_MIXED_DATA_FILE_VERSIONS = 1 << 8` (256) is no

+ longer a reservation pinned equal to `FLAG_UNKNOWN`: the assert relaxed to

+ `FLAG_MIXED_DATA_FILE_VERSIONS < FLAG_UNKNOWN`, `FLAG_UNKNOWN` moved `1 << 8` -> `1 << 9` (512),

+ and the build now both reads and writes mixed-version datasets. It is still carried by

+ `STICKY_PAIRED_FLAGS`, and a **half-set** manifest is now a hard error: "Manifest has only one of

+ the mixed data-file-version reader and writer feature bits set, so its semantics are undefined".

+ Section 7.

+

+ **Bit 1024 is where the docs and the code disagree - trust the code.**

+ `FLAG_FRAGMENT_REUSE_INDEX = 1 << 10` is declared at `rust/lance-table/src/feature_flags.rs:69`

+ and, at `v13.0.0-beta.4`, **that declaration is its only occurrence in the entire tree**. It sits

+ *above* `FLAG_UNKNOWN` (512), and the supported set is computed as `FLAG_UNKNOWN - 1`, so a

+ manifest setting it is **refused**. The spec page meanwhile lists it as reader `Yes` / writer

+ `Yes` and puts the unknown boundary at 2048. The docs describe the intended end state; the code

+ has only reserved the constant. Anything you build against tagged FRI today is building against

+ prose, not behavior.

+

+ catch-up became derived rather than declared (#8481); transaction proto field 9 is deprecated for

+ field 10 (#7432); compaction gained row/byte budgets plus fragment exclusion (#8235, #8532);

+ `merge_insert` gained `write_mode` (#8423); and Python commit conflicts became

+ `lance.commit.CommitConflictError`, a subclass of `OSError`, so existing handlers keep working

+ (#8563). Full list with citations in `references/changelog-v7-v13.md`.

+ earlier). A rewrite used to advance *every* index's `fragment_bitmap` onto the new fragment ids -

+ including ZoneMap, whose stored addresses point into the fragments the rewrite dropped. The

+ Rewrite path now branches on `results_are_row_addrs()`: an address-domain index gets

+ `drop_rewritten_fragments` and a full-scan fallback, correct-but-slower instead of stale

+ addresses. **Heals only for new compactions**: an index already damaged under v10 or earlier must

+ be recreated, and the damage does not self-heal through routine maintenance, because the

+ refreshed `fragment_bitmap` also makes incremental folds a no-op. Section 11.

+ #8839, #8904. Conditions for each in `references/changelog-v7-v13.md`.

+ **225 commits** from `release-root/12.0.0-beta.N` to the `v12.0.0` final, with **7

+ `breaking-change`-labeled PRs** - the five visible at beta.15 plus **#9072** and **#9101** in the

+ run-up to the final. No new index types and no new crates; every structural invariant above still

+ holds. **The label is a floor, not a ceiling** - the two biggest behavior changes in the line

+ carry a conventional-commit `!` but no label, so the bot never counted them: the `stable` -> 2.2

+ move (#8657, above) and the IVF_RQ 5-bit default (below).

+ and #8979 moved **Java** to 0.12.0 as well; only **Python** still holds `>=0.11.1,<0.12`, because

+ its generated models still send `on` as a bare string. Quote a language-specific pin, never one

+ number for all three - and note this split moved once already, so re-check it rather than

+ carrying the pairing forward.

+ plus the much longer list that *does* heal on upgrade, in `references/changelog-v7-v13.md`.

+ **Mixed data-file versions LANDED** - it is no longer "1 of 6". #8581-#8584 shipped in `v12.0.0`

+ (validation, per-operation V2 write targets, propagation across dataset operations, compaction

+ targeting) and #8585 exposed it in the bindings in the v13 line. The proto changed with it:

+ `DataStorageFormat.version` is now "the default format version used when writing data files",

+ and "each DataFile's version is authoritative for decoding" once the capability is set.

+ **In flight, not landed - do not treat as shipped:** generic block v5 compression is **still 1 of

+ 10** PRs merged (#8324; #8325-#8333 all remain open). Next big dependency break in the queue:

+ #8997, "upgrade to arrow 59, DataFusion 55, and pyo3 0.29" - **still open** at `v13.0.0-beta.4`,

+ so arrow 58 / datafusion 54 still hold. It also gates two outstanding PyO3 advisories

+ (RUSTSEC-2026-0176/0177); rustls was separately patched to 0.23.45 for RUSTSEC-2026-0285 (#9212).

+

+ ## The v13 delta

+

+ **66 commits** from `release-root/13.0.0-beta.N` to `v13.0.0-beta.4`, with **2

+ `breaking-change`-labeled PRs**. No new crates and no new index types; 26 crates, 16 transaction

+ ops, `CommitConfig.num_retries` 20, arrow 58 / datafusion 54, MSRV 1.91.0, Edition 2024 and

+ Python 3.10+ all still hold.

+

+ **The `!`-vs-label rule inverted in this window.** All three conventional-commit `!` commits

+ (#7465, #9192, #9101) *do* carry the `breaking-change` label. Keep treating the label as a floor

+ rather than a ceiling - but this window is the counter-example, not more evidence for the gap.

+

+ - **`WriteParams` gained `file_writer_options`** (#9192) - the single labeled PR that re-rooted

+ the major. `FileWriterOptions { data_cache_bytes, max_page_bytes, keep_original_array }` is now

+ reachable from the dataset write APIs in Rust, Python and Java. A zero `max_page_bytes` is

+ rejected before encoder construction rather than misbehaving later. Anything constructing

+ `WriteParams` by struct literal fails to compile.

+ - **Page metadata is initialized lazily, and the metadata cache key changed shape** (#7465).

+ `StructuralFieldScheduler::initialize` now takes `requested_ranges`, and the page-scheduler

+ `initialize` splits into `init_ranges()` and `init_from_buffers(buffers, io)` - any external

+ implementor fails to compile. The public `DecodeBatchScheduler::try_new` kept its signature;

+ the range-aware entry point is the crate-private `try_new_with_ranges`. The part that bites

+ without a compile error: caching moved from a per-column `FieldDataCacheKey` to a per-page

+ `PageDataCacheKey { column_index, page_index, view_tag }`, so **every warm or persisted

+ metadata cache cold-misses** across this upgrade. The payoff is real - "a cold point/range

+ read's metadata IO is invariant to the column's total page count".

+ - **`json_extract` and `json_get` no longer route to JSON indices** (#9101). Only the four typed

+ accessors (`json_get_int` / `_float` / `_bool` / `_string`) reach the index; everything else

+ falls back to a full scan. This fixes three real wrong-answer bugs - a quoted-key mismatch that

+ "searched for a quoted key and matched nothing", a `Utf8` literal driving an `Int64` btree into

+ a panic, and an unsound range because "quoting is not order-preserving (`ab` < `ab!` but

+ `"ab"` > `"ab!"`)". **The cost is silent**: a `json_extract` workload that used to hit an index

+ now scans, with no error and no plan warning. Rewrite those predicates onto the typed

+ accessors.

+

+ **The Fragment Reuse Index gained a versioned on-disk contract** (#9136). `InlineContent` field 1

+ was renamed `versions` -> `legacy_versions` and a tagged `transitions` list added at field 2,

+ gated on `index_version >= 1`; mappings are now a oneof of `OrderedCompaction` or

+ `StablePartition`. A stable partition "assigns source rows to destination fragments while

+ preserving their relative source order within each destination", which lets FRI reuse existing

+ indices after **reclustering** - a second use case the v0 model had no concept of. Its physical

+ form is an immutable row-map Lance file with `uint16` labels and an `LSPC`-magic counts matrix.

+ Two hard rules: **stable row IDs and tagged FRI are mutually exclusive** ("writers must not

+ publish `index_version >= 1` on them"), and cleanup "must retain intermediate transitions still

+ needed to translate old addresses". Upstream also softened the old claim - "FRI does not remove

+ conflicts between overlapping rewrites". Section 11.

+

+ **Net-new, non-breaking:** `Dataset::frag_reuse_index()` is public (#9112) and documented in the

+ performance guide; `FileFragment::write_overlay` returns a real `OverlayWriter` (#8761, still

+ env-gated); an `hf://` object store with `hf_enable_resolve_cache` (#9236); Python

+ `lance.bitmap.Bitmap`, `deep_clone()` (#9181), `base_paths()` (#9191) and

+ `update_columns(with_offsets=True)` (#8891); Java `DataStorageVersion`, `FileWriteOptions` and

+ `ScanOptions.indexSegments`; and namespace table listing finally bounded by `read_dir_page`

+ (#9165). `inline_optimization_enabled` **flipped `true` -> `false`** (#9180), which upstream

+ justifies with a -49% write-p50 measurement at 1M entries.

+

+ remedy. Note that **#7465 changes the metadata cache key shape**, so the first run after a v13

+ upgrade re-pays that paging even against a warm or persisted cache. Details and build-time

+ requirements in `references/performance.md`.

+ **Time travel is not an archive mechanism.** Versions look like free history, but the default

+ cleanup reclaims anything older than 7 days and cleanup is part of routine optimize - so a design

+ that treats old versions as the durable record loses it on the first maintenance pass. Keep an

+ explicit archive if you need one.

+

+ Also out by design: `community/`, `examples/`, `integrations/{index,pytorch,tensorflow}.md`; and

+ the landing stubs and contributor files (`format/AGENTS.md`, `format/CLAUDE.md`).

+

+ **A whole tier of docs is not in this repo at all**, so it cannot be mirrored and cannot be

+ enumerated from a clone. `docs/make-full-website.sh` assembles `format/catalog`,

+ `format/namespace`, and the `integrations/{duckdb,huggingface,spark,ray,trino,context}` sections

+ at build time from six sibling repos with their own version lines - the checked-in

+ `integrations/index.md` links `spark/`, `duckdb` and `trino` as if they were local, but those

+ paths do not exist in the tree. **Lance Context** and the **HuggingFace** integration docs are

+ whole nav sections that exist only on the built site. For any of those, read `lance.org` rather

+ than this mirror. Protobuf message bodies are likewise expanded at build time from `protos/` by

+ `mkdocs_protobuf`, so the mirrored spec pages show `%%% proto.message.X %%%` placeholders where

+ the site shows a rendered schema.