data-formats · git:20261026.6e0be1d · 2026-10-26 · sha256 f00faaa2a15a682f
data-formats git:20261026.6e0be1dA
Immutable. This exact content never changes and is served at /api/v1/blob/f00faaa2a15a682f.
---
name: data-formats
description: Parse and load RDF into a sparq Graph (Turtle/N-Triples/N-Quads/TriG via sparq-core, and HDT incl. compressed .hdt.gz/.hdt.zst/.hdt.bz2 via sparq-hdt), do streaming/parallel/external-memory ingest of compressed dumps, and take cheap immutable copy-on-write Graph snapshots. Use when ingesting RDF files, choosing a loader, wiring HDT, or snapshotting a graph for serving.
---
# sparq data formats
How to get RDF *into* a sparq `Graph` and how to snapshot one cheaply. Text formats
(Turtle / N-Triples / N-Quads / TriG) and the in-memory + streaming + external-memory
loaders live in `sparq-core`; the binary HDT archive format (including content-sniffed
`.hdt.gz` / `.hdt.zst` / `.hdt.bz2`) lives in the opt-in `sparq-hdt` crate.
> Direction note: these crates **parse RDF in**. To write RDF *out*, sparq-engine ships the
> **RDF writer matrix** behind its opt-in `serialize-rdf` feature — Turtle / TriG / N-Quads /
> JSON-LD 1.1 writers (`sparq_engine::serialize::*`), plus deterministic **pretty** (indented)
> variants — Turtle / TriG (`graph_to_turtle_pretty`, the long-term engine home for the site
> formatter) and JSON-LD (`graph_to_jsonld_pretty`); the N-Triples writer (`triples_to_ntriples`)
> is always on. See recipe 6.
## Literal validity and cache eligibility
RDF ingestion preserves ill-typed literals. To check numeric datatype
membership, call `sparq_core::numeric_literal_valid(value, datatype_iri)`: this
checks raw lexical grammar and integer subtype facets, rejecting all boundary
whitespace. This includes the
range of `xsd:byte` and unsigned integer types. It does not impose the evaluator's
finite mantissa capacity. `numeric_cache_value` can return `None` for a valid large
number; a missing cache value is not a datatype-validity result.
`temporal::Timeline::parse_datetime`, `parse_date`, `parse_civil_date` and
`parse_tz` reject raw boundary whitespace, malformed calendar/timezone values and timestamps outside their
representation, including the existing checked BCE range. For exact value decisions,
use `temporal::ExactTimeline`, `ExactTemporal`, or `Graph::exact_temporal_value(id)`.
These borrow the original fraction and compare integer seconds plus lexical digits,
without allocation; work is linear in literal length. Legacy `Timeline`/`Temporal`
and `Graph::temporal_value` contain approximate floating values; vector/cache consumers
retain them, but query equality/order must use the exact keys. `year_within_capacity`
checks an explicit year range separately from ordinary datatype validity.
Malformed Unicode returns `None` without slicing panics. `dateTimeStamp` requires a timezone.
Raw RDF temporal literals undergo no XML preprocessing; string casts are separate constructors.
`Graph::open` ignores legacy `numerics.bin`/`temporals.bin` and both v2 caches,
which could contain padded raw literals, and recomputes values from the dictionary.
It does not rewrite old caches or drop ill-typed RDF terms. Current writers use
`numerics-v3.bin` and `temporals-v4.bin` across ordinary, compressed and external
builds; absent or wrong-sized current caches are rebuilt in memory. To persist a
migrated archive, call `Graph::open(old)?.save(new)?` with a separate destination.
Until saved, legacy opens repeat the dictionary scan and cache allocation.
Cache versions signal semantic compatibility; they do not authenticate archive
bytes or change the existing trusted-storage assumption.
## Named graph access
`Graph::named` is private. Read `graph.named()` (an ordered slice) or
`graph.named_graph(&name)`. Use `add_named`, `extend_named`, `set_named`,
`remove_named`, `retain_named`, and `take_named` for in-memory members; these
return `Result` and keep the graph-name prefix index current. Store-backed
members cannot be replaced, extracted, or shifted through those methods. A directory-backed
root refuses `add_named`, `set_named`, and nonempty `extend_named` batches of in-memory
graphs: use a dataset `fork()` for views, or logged insertion through `ensure_named`.
Use `ensure_named` and `apply_delta_to_named(index, inserts, deletes)` for logged
updates, and `clear_named_durable`, `drop_named_durable`, or
`drop_all_named_durable` for durable removal. Frozen and read-only graphs refuse
mutations. `rebuild_default` and `replace_default` keep named members and mapping
locks while replacing the default graph with an in-memory view; that view's later
updates are not logged. See [the accessor example](named-accessors.md).
## Quickstart
Add the dependency (HDT is a separate, native-only crate):
```toml
# Cargo.toml
[dependencies]
sparq-core = "0.1" # default features include `parallel` (native)
sparq-hdt = "0.1" # OPTIONAL — only if you load .hdt archives
# To filter an HDT during its SPO decode instead of building the full Graph:
# sparq-hdt = { version = "0.1", features = ["load-filter"] }
oxrdf = { version = "0.3", features = ["rdf-12"] } # for Term/NamedNode at call sites
```
```rust
use sparq_core::Graph;
// `format` (plus media-type aliases): "turtle"|"ttl"|"text/turtle"|"application/turtle" |
// "ntriples"|"n-triples"|"nt"|"application/n-triples" | "nquads"|"n-quads"|"nq"|"application/n-quads" |
// "trig"|"application/trig". An UNRECOGNISED format string is an `Err` — NOT silently
// parsed as Turtle (sq-m2pc). JSON-LD ("jsonld"/"json-ld"/"application/ld+json") needs the
// `jsonld` feature on the `sparq-core` LIBRARY dep (OFF by default — the library stays lean);
// without it those strings also error rather than mis-parsing. sq-oy1f.4: the
// `sparq-cli` and `sparq-server` BINARIES enable `jsonld` by DEFAULT (a maintainer-directed
// exception), so they read/write JSON-LD out of the box; a library embedder opts in explicitly.
// sq-f47w1 (survey §B1): RDF/XML ("rdfxml"/"rdf-xml"/"application/rdf+xml")
// likewise needs the OPT-IN `rdfxml` feature on `sparq-core` (OFF by default — it links
// `oxrdfxml`/`quick-xml`, kept off the lean wasm bundle); without it those strings error
// rather than mis-parsing. RDF/XML has no named-graph syntax, so it loads via `load_str` /
// `parse_to_triples` (not the dataset path), serial-only (XML is not line-delimited).
let ttl = r#"@prefix ex: <http://ex/> .
ex:alice ex:knows ex:bob ."#;
let g = Graph::load_str(ttl, "turtle").expect("parse");
assert_eq!(g.len(), 1);
// HDT archive (sniffs .hdt.gz/.hdt.zst/.hdt.bz2 by magic bytes, not file name):
// let g = sparq_hdt::load("dataset.hdt").unwrap();
```
Or from the CLI:
```bash
sparq-cli query data.ttl turtle 'SELECT * WHERE { ?s ?p ?o } LIMIT 5'
# HDT needs the opt-in `hdt` CLI feature:
cargo build -p sparq-cli --features hdt
./target/.../sparq-cli query dataset.hdt hdt 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
```
## Key APIs
`sparq_core::Graph` — construction / ingest (all take a `format: &str`):
```rust
// In-memory, whole document in a &str. Parallel-chunked for N-Triples & Turtle
// when the `parallel` feature is on (default native); on ≥2 threads both fan the
// per-chunk dictionaries into one sharded-dict merge (the serial `merge_remap`
// loop is kept only on a single thread). Output is identical to the serial parse.
pub fn load_str(text: &str, format: &str) -> Result<Graph, String>
pub fn load_str_with_base(text: &str, format: &str, base: &str) -> Result<Graph, String>
// Dataset load (N-Quads / TriG) preserving NAMED GRAPHS as separate sub-graphs
// (so `GRAPH ?g {…}` works). Other formats defer to load_str. In-memory only.
// N-Quads is parallel-chunked (per-graph routing + sharded-dict merge) when the
// `parallel` feature is on (default native); TriG uses the serial path. Named
// graphs come out in first-occurrence document order (deterministic).
pub fn load_dataset(text: &str, format: &str) -> Result<Graph, String>
// sq-tonhr.2 — load_dataset with a base IRI for the document's relative
// IRIs (the DATASET companion to load_str_with_base): named graphs preserved,
// SERIAL parse (base-relative docs are small — manifests, W3C test actions; the
// rdf-trig conformance lane drives this). N-Quads has no relative IRIs (base
// ignored); non-dataset formats defer to load_str_with_base.
pub fn load_dataset_with_base(text: &str, format: &str, base: &str) -> Result<Graph, String>
// Streaming from any reader (e.g. a decompression stream) — serial.
pub fn load_reader<R: std::io::Read>(reader: R, format: &str) -> Result<Graph, String>
// Streaming + PARALLEL (needs `parallel`): pipelined 32 MiB block parse, never
// materializes the whole decompressed doc. N-Triples gets this parallel pipelined
// path; other formats fall back to serial load_reader. (This is the PARALLEL-LOADER
// fast path — distinct from the byte-level IRI-validation fast path noted below, which
// is a separate, opt-in mechanism that covers N-Triples AND N-Quads.)
pub fn load_reader_parallel<R: std::io::Read + Send>(reader: R, format: &str) -> Result<Graph, String>
// sq-7d3dj.18 — the byte-level N-Triples/N-Quads fast path does NOT validate IRIs against
// RFC-3987 by default (the serial oxttl path DOES). It does enforce the N-Triples grammar
// (#2716): IRIREF characters (a `\u`/`\U` escape is re-checked after decoding, so an
// escaped space, `<`, `>` or other forbidden character is a parse error in every IRI
// position), UCHAR-only IRI escapes, an absolute scheme, BLANK_NODE_LABEL
// (full PN_CHARS ranges; a label may abut the next term, `_:s<p>`), string ECHAR/UCHAR and
// no raw line breaks, LANGTAG (direction ltr/rtl), IRI-only predicates and no literal
// subjects, one statement per line (no line break inside a statement; only WS and a comment
// after its `.`), and it accepts RDF 1.2 `VERSION "..."` directives. The
// OPT-IN `iri-fast` feature on `sparq-core` (OFF by default) turns that validation on for the
// byte parser — subject, predicate, object, datatype and graph IRIs alike — via a
// prefix-memoized fast path IN FRONT of `oxiri` — a last-N validated
// `scheme://authority/` memo + a one-pass ASCII `iunreserved`/sub-delims suffix scan, falling
// back to the full `oxiri` automaton on any non-ASCII / `%`-escape / `#` / delimiter anomaly.
// It accepts EXACTLY what `oxiri::Iri::parse` accepts (an absolute IRI) — a MANDATORY
// differential-fuzz gate (`sparq_core::iri`, run under `--features iri-fast`) asserts
// `fast_accept ⇒ oxiri` and `validate == oxiri` over a committed adversarial corpus + a
// deterministic generator, so a malformed IRI can never be wrong-accepted into the store.
// The `oxiri` dep is already in-tree (via oxttl), so the feature adds zero new compilation.
// sq-jocpn — the OPT-IN `native-ttl` feature on `sparq-core` (OFF by default) swaps the
// oxttl Turtle path for a hand-rolled byte-level tokenizer/parser (`sparq_core`'s `ttl` module) that
// interns S/P/O directly into the Dict — the Turtle analogue of the byte-level N-Triples parser. It
// handles the FULL Turtle grammar (prefixes/@base, collections `()`, blank-node property lists `[]`,
// predicate-object lists `; ,`, `a`, numeric/boolean/triple-quoted literals, escapes) PLUS the RDF
// 1.2 surface the workspace oxttl enables (reifiers `<< … >>`, triple terms `<<( … )>>`, `~` and
// `{| … |}` annotations, all desugared to `rdf:reifies`). It is a BYTE-IDENTICAL drop-in: IRI base
// resolution delegates to the same `oxiri` automaton oxttl uses, so resolved terms match exactly,
// and it passes an IDENTICAL pass/fail set on the whole W3C rdf-turtle suite. Correctness is pinned
// by (a) a native-vs-oxttl DIFFERENTIAL over many constructs + malformed-input parity (both must
// Err), and (b) the conformance ratchet run with `--features sparq-core/native-ttl`
// (`sparq-conformance`'s `native-ttl-suite`). Only anonymous blank-node labels differ (as between
// two oxttl runs), so comparisons are blank-node-isomorphic. It is opt-in so it can be A/B'd against
// oxttl before any thought of defaulting on; on a quiet box the native single-thread parse is
// meaningfully faster (it removes oxttl's tokenizer/state-machine, ~53% of the incumbent parse per
// the sq-wrn61 profile). Reproduce: `cargo test -p sparq-core --features native-ttl`.
// Parse WITHOUT building indexes — the seam reasoning/transform hooks use.
pub fn parse_to_triples(text: &str, format: &str) -> Result<(Dict, Vec<[Id; 3]>), String>
pub fn from_parts(dict: Dict, triples: Vec<[Id; 3]>) -> Graph
// EXTERNAL-MEMORY build (needs `mmap`): stream-parse, spill sorted runs to disk,
// k-way merge — bounded RAM for datasets larger than memory. `chunk` = triples per run.
pub fn build_external<R: std::io::Read + Send>(reader: R, format: &str, dir: &Path, chunk: usize) -> Result<(), String>
// Empty, in-memory graph — the trivial infallible constructor (no `load_str("", …)`
// workaround, no error handling). `default()` and `new()` are interchangeable.
pub fn new() -> Graph // also: Graph::default()
```
`sparq_core::Dict` — compact FILTERED rebuild on the `(Dict, triples)` seam:
```rust
// sq-eiv — consume the Dict, keep only the real ids `retain` accepts, and get
// back a dense rebuilt Dict plus an old→new remap (`remap[(old_id - 1) as usize]`; NO_ID
// = dropped) for rewriting triple ids before `from_parts`. Inline integer ids are not
// dictionary records: never passed to `retain`, unchanged under remapping. Dependency-aware
// for RDF 1.2 triple terms — retaining a triple term implicitly retains its s/p/o and any
// recursively nested components (their remap entries are populated even if `retain` said
// false). Never materialises an `oxrdf::Term` and never re-interns survivors (arena
// payloads MOVE; blob/mmap/frozen records copy compactly); dead prefix/datatype metadata
// is filtered too, so it does not accumulate across rebuilds.
pub fn rebuild_filtered(self, retain: impl FnMut(Id) -> bool) -> (Dict, Vec<Id>)
```
`sparq_core::Graph` — single-triple mutation from `oxrdf` terms (the ergonomic one-triple
case over `apply_delta`; each position takes anything that converts into a `Term`:
`NamedNode` / `Literal` / `BlankNode` / a `Term`). Set-valued (re-insert / absent-remove is a
no-op) and, for a directory-backed graph, WAL-logged + fsync'd — same semantics as `apply_delta`.
For several triples at once prefer ONE `apply_delta` batch (one WAL append) over a loop:
```rust
use sparq_core::Graph;
use oxrdf::{NamedNode, Literal, Term};
let mut g = Graph::new();
g.insert_triple(
NamedNode::new_unchecked("http://ex/alice"),
NamedNode::new_unchecked("http://schema.org/name"),
Term::Literal(Literal::new_simple_literal("Alice")),
)?;
g.remove_triple(
NamedNode::new_unchecked("http://ex/alice"),
NamedNode::new_unchecked("http://schema.org/name"),
Term::Literal(Literal::new_simple_literal("Alice")),
)?; // absent ⇒ no-op, never an error
```
`sparq_core::Graph` — cheap snapshots / forks (the serving pattern):
```rust
pub fn snapshot(&self) -> GraphSnapshot // immutable COW view; O(pending delta), not O(triples)
pub fn fork(&self) -> Graph // mutable structural fork (Arc-shares base storage)
```
`sparq_core::GraphSnapshot` — `Send + Sync`, `Deref<Target = Graph>` (every read method:
`len`, `id_of`, `pattern`, `iter_ids`, `store`/`dict`), **no** mutating methods.
`into_graph(self) -> Graph` / `as_graph(&self) -> &Graph`.
`sparq_hdt` (native-only; opt-in):
```rust
pub fn load(path: impl AsRef<Path>) -> Result<Graph, Error> // sniffs .hdt.gz/.hdt.zst/.hdt.bz2
pub fn load_reader<R: BufRead>(reader: R) -> Result<Graph, Error> // any buffered source
// sq-lsp7k.24 — opt-in `load-filter`; None means wildcard. Filtering
// runs during the one-shot SPO walk and only accepted triples' terms enter the
// returned Graph. An all-None pattern is identical to load_reader.
#[cfg(feature = "load-filter")]
pub type TriplePattern = (
Option<oxrdf::NamedOrBlankNode>, // subject
Option<oxrdf::NamedNode>, // predicate
Option<oxrdf::Term>, // object
);
#[cfg(feature = "load-filter")]
pub fn load_reader_filtered<R: BufRead>(reader: R, pattern: &TriplePattern) -> Result<Graph, Error>
#[cfg(feature = "load-filter")]
pub fn stats_reader_filtered<R: BufRead>(reader: R, pattern: &TriplePattern) -> Result<HdtStats, Error>
pub fn header(path: impl AsRef<Path>) -> Result<Graph, Error> // HDT header metadata as a Graph
pub fn header_reader<R: BufRead>(reader: R) -> Result<Graph, Error>
pub fn stats(path: impl AsRef<Path>) -> Result<HdtStats, Error> // dictionary counts + exact triples
pub fn stats_reader<R: BufRead>(reader: R) -> Result<HdtStats, Error>
pub struct HdtStats {
pub triples: usize, pub shared: usize, pub subjects_only: usize,
pub objects_only: usize, pub predicates: usize,
}
// HdtStats also provides distinct_subjects() and distinct_objects().
// also: load_reader_via_upstream (differential oracle), graph_from_hdt, graph_from_reader
// MEASUREMENT-ONLY (bench/parse's HDT stage split — NOT a production loader):
// identical decode to graph_from_reader, but records a per-stage wall-clock split.
// Production callers use the plain loaders above; the decode is byte-for-byte the
// same (timing only reads Instant::now() at the existing stage boundaries), so
// decoder behaviour is unchanged.
// COARSE three: dict (control/header + 4-section PFC decode+intern+merge), scan
// (triples-section read + SPO id-translation walk), build (Graph::from_parts).
// FINER (sq-7ge0): dict_{shared,subjects,objects,predicates} are the four per-PFC-
// section decode walls — in a multi-threaded pool they run CONCURRENTLY so they
// OVERLAP and need NOT sum to `dict`; dict_merge is the section-order merge. scan
// splits into scan_read (triples bitmaps/sequences) + scan_walk (the SPO walk),
// which sum EXACTLY to `scan`.
pub struct StageTimings {
pub dict: Duration, pub scan: Duration, pub build: Duration,
pub dict_shared: Duration, pub dict_subjects: Duration, pub dict_objects: Duration,
pub dict_predicates: Duration, pub dict_merge: Duration,
pub scan_read: Duration, pub scan_walk: Duration,
}
pub fn graph_from_reader_timed<R: BufRead>(reader: R, t: &mut StageTimings) -> Result<Graph, Error>
// WRITE (opt-in `write` feature): Graph -> .hdt (honours .gz/.zst/.bz2 by extension)
// CLI: `sparq-cli to-hdt <file> <in-fmt> <out.hdt[.gz|.zst|.bz2]>` (opt-in `hdt-write` feature)
#[cfg(feature = "write")]
pub fn save(graph: &Graph, path: impl AsRef<Path>) -> Result<(), Error>
```
## Common recipes
**1. Load a compressed N-Triples dump with the fast parallel streaming path.**
Decompress on the fly and parse in parallel without ever holding the full text:
```rust
use sparq_core::Graph;
let file = std::fs::File::open("dump.nt.gz")?;
let reader = flate2::read::MultiGzDecoder::new(file); // .bz2 -> bzip2::read::MultiBzDecoder
// .zst -> zstd::stream::read::Decoder::new
let g = Graph::load_reader_parallel(reader, "ntriples")?; // N-Triples gets the pipelined parser
```
**2. Load a dataset with named graphs (TriG / N-Quads).**
```rust
let trig = r#"<http://ex/g> { <http://ex/a> <http://ex/p> <http://ex/b> . }"#;
let g = sparq_core::Graph::load_dataset(trig, "trig")?;
assert_eq!(g.named().len(), 1); // each named graph is its own sub-Graph
```
**3. Load an HDT archive (and read its header metadata).**
```rust
let g = sparq_hdt::load("dataset.hdt")?; // .hdt / .hdt.gz / .hdt.zst / .hdt.bz2 (by magic bytes)
let meta = sparq_hdt::header("dataset.hdt")?; // VoID stats / provenance as a queryable Graph
let stats = sparq_hdt::stats("dataset.hdt")?; // exact counts without a full graph decode
assert_eq!(stats.distinct_subjects(), stats.shared + stats.subjects_only);
let bytes = std::fs::read("dataset.hdt")?;
// from memory: sparq_hdt::load_reader(std::io::Cursor::new(bytes.clone()))?
// Predicate-filtered from memory (needs `load-filter`; None is a wildcard):
let pattern: sparq_hdt::TriplePattern = (
None,
Some(oxrdf::NamedNode::new("http://www.w3.org/2000/01/rdf-schema#label")?),
None,
);
// sq-obhf1: count matches without constructing a result Graph/Dict.
let label_stats = sparq_hdt::stats_reader_filtered(std::io::Cursor::new(bytes.clone()), &pattern)?;
let labels = sparq_hdt::load_reader_filtered(std::io::Cursor::new(bytes), &pattern)?;
assert_eq!(label_stats.triples, labels.len());
// WRITE back (opt-in `write` feature): sparq_hdt::save(&g, "out.hdt.gz")?;
// (currently a temp-N-Triples round-trip through the wrapped builder — see
// sparq-hdt/UPSTREAM.md for the queued in-memory-builder contribution)
```
**4. Out-of-core build for a dataset larger than RAM** (requires `sparq-core` `mmap` feature):
```rust
use std::path::Path;
let reader = flate2::read::MultiGzDecoder::new(std::fs::File::open("huge.nt.gz")?);
// stream-parse + external-sort the six permutations onto disk; ~16M triples per spill run:
sparq_core::Graph::build_external(reader, "ntriples", Path::new("/data/idx"), 16_000_000)?;
let g = sparq_core::Graph::open(Path::new("/data/idx"))?; // query with indexes memory-mapped
```
`g.is_directory_backed()` is `true` for a graph opened this way, or one holding a named graph
opened this way (updates are written through to the directory), and `false` for a wholly
in-memory graph or without the `mmap` feature.
**5. Cheap snapshot for serving (one mutable master + immutable readers per commit).**
`snapshot()` Arc-shares the base indexes/dictionary — it copies neither, and later
mutation of the master is invisible to the snapshot:
```rust
let mut master = sparq_core::Graph::load_str(doc, "turtle")?;
let snap = master.snapshot(); // immutable, Send+Sync, point-in-time
master.apply_delta(&inserts, &deletes)?; // master moves on
assert_eq!(snap.len(), original_len); // snapshot frozen at snapshot time
// publish `snap` to query threads; it Derefs to &Graph
let mutable_copy = master.snapshot().into_graph(); // a snapshot you can then mutate
```
**6. Serialize a graph back out (the RDF writer matrix).** The Turtle / TriG / N-Quads /
JSON-LD writers live in `sparq-engine` behind the **`serialize-rdf`** feature (it pulls
in ZERO new dependencies — the default dep graph is byte-for-byte unchanged when off, *and*
unchanged even when on: the JSON-LD writer emits JSON by hand, no json-ld/serde crate). For a
LIBRARY embedder it is opt-in (the engine library default stays lean): enable with
`sparq-engine = { version = "0.1", features = ["serialize-rdf"] }`. The `sparq-cli` and
`sparq-server` BINARIES pull it into their default build via the default-on `jsonld` feature
(sq-oy1f.4), so `dump …` and the server's `application/ld+json` work out of the box.
The N-Triples writer (`triples_to_ntriples`) is always on. Internally the writer matrix is
housed in the `sparq-engine-serialize` sub-crate (published only to satisfy `sparq-engine`'s
crates.io dependency closure, sq-6vshe.4) and
re-exported verbatim — the `sparq_engine::serialize::*` paths and feature names above are
unchanged and remain the supported surface.
```rust
use sparq_engine::serialize::{graph_to_turtle, graph_to_trig, graph_to_nquads,
graph_to_jsonld, graph_to_jsonld_pretty, JsonLdForm};
let g = sparq_core::Graph::load_dataset(trig_src, "trig")?;
let ttl = graph_to_turtle(&g); // Turtle: @prefix header, `a` for rdf:type, predicate-object
// lists; DEFAULT graph only.
let tg = graph_to_trig(&g); // TriG: default graph + `GRAPH <g> { … }` blocks (whole dataset).
let nq = graph_to_nquads(&g); // N-Quads: default graph (3 cols) + named graphs (4th column).
let jx = graph_to_jsonld(&g, JsonLdForm::Expanded); // JSON-LD 1.1, fully-expanded node-object
// array, no @context (whole dataset).
let jf = graph_to_jsonld(&g, JsonLdForm::Flattened); // node-merged, `@graph`-framed.
let jc = graph_to_jsonld(&g, JsonLdForm::Compacted); // basic prefix `@context` (default_prefixes).
let jp = graph_to_jsonld_pretty(&g, JsonLdForm::Expanded); // same docs, indented (multi-line).
```
Lower-level entry points take `&[oxrdf::Triple]` (e.g. CONSTRUCT output) directly:
`write_turtle(triples, &prefixes)`, `write_trig(&named_graphs, &prefixes)`,
`write_nquads(&named_graphs)`, `write_jsonld(&named_graphs, JsonLdForm::Expanded, &prefixes)`;
`default_prefixes()` supplies the common namespaces, or pass your own `Prefixes` (a
`BTreeMap<String, String>`) — `prefixes_from_pairs([(prefix, iri), …])` builds one from an
ordered `(prefix, iri)` pair list (e.g. a query's parsed `PREFIX` lines or a `[[prefix, iri], …]`
array). The `graph_to_*_with` convenience wrappers (`graph_to_turtle_with`, `graph_to_trig_with`,
`graph_to_jsonld_with`, and the pretty `*_with`) take that same map, so the whole-graph path can
also serialise under an explicit prefix policy. When several prefixes match an IRI, the longest
namespace wins (equal lengths: first label in map order). Only prefixes actually used are emitted (the
Turtle/TriG header, or the JSON-LD compacted `@context`). Round-trip (parse → serialize → re-parse) is isomorphic
for every form. **JSON-LD specifics:** `xsd:string`/`rdf:langString` stay implicit
(`@value` + optional `@language`); every other datatype is preserved as `@type`; canonical
`xsd:integer`/`xsd:boolean` literals coerce to native JSON scalars only when lossless (leading
zeros, `xsd:double`/`decimal`, etc. stay typed strings). A *well-formed, single-referenced*
`rdf:first`/`rdf:rest`/`rdf:nil` blank-node chain is collapsed into a native `@list` array
(nested lists collapse recursively); anything that fails the conservative safety predicate —
a list cell referenced more than once, carrying an extra predicate, cyclic, or not terminated
by `rdf:nil` — is left as ordinary `rdf:first`/`rdf:rest` triples, so the round-trip stays
lossless either way (the empty list `()` stays an `rdf:nil` reference, never `@list`).
A same-box Turtle-only comparison against oxttl's `TurtleSerializer` (Oxigraph 0.5's writer)
on a document-shaped graph is `cargo run --release -p sparq-engine-serialize --features
serialize-rdf --example turtle_vs_oxttl` (#4898). **Comparative throughput** for the writer matrix is measured by `bench/serialize/run.sh`
(registered `serialize-bench`, sq-hmd7l.14): sparq's buffered/streaming/pretty
regimes in-process, plus a cross-engine pipeline panel vs serd/rapper/Jena riot/oxrdfio —
every emitted document round-trip-gated (re-parse == source store) before its timing row
is trusted. First-read analysis: `research/gap-serialize-2026-07.md`.
**Pretty Turtle / TriG (idiomatic, deterministic).** Alongside the plain writers, the same
feature exposes a *pretty* variant — `graph_to_turtle_pretty(&g)` / `graph_to_trig_pretty(&g)`,
or the lower-level `write_turtle_pretty(triples, &prefixes, &opts)` /
`write_trig_pretty(&named_graphs, &prefixes, &opts)` with a `PrettyOptions { indent, abbreviate }`
(default: two-space indent, abbreviation on). Two differences from `graph_to_turtle`: (1) output
is **emission-order-independent** — subjects, predicates and objects are sorted by their
canonical N-Triples spelling and `rdf:type` (`a`) sorts first in each block, so the same triple
SET always renders to the same bytes (the plain writer preserves the store's `iter_ids()` order,
which is thread-count-dependent); (2) the **output shape** matches the site's `prettyTurtle`
reshaper (`packages/sparq-client/src/pretty-turtle.ts`, sq-gb4o / #805) — a prefix-alphabetical
used-only `@prefix` header, blank-line-separated subject blocks, the subject on its own line,
predicate lines at one indent (`;`-joined, `.`-terminated), and object lists wrapped onto
continuation lines (`,\n` + two indents). It is round-trip-correct (re-parses to the same triple
set) and reuses the same escaping/term helpers, so literals (datatype/lang, implicit `xsd:string`
dropped), blank nodes, and RDF 1.2 triple terms `<<( s p o )>>` are all reproduced losslessly.
This is the long-term engine home for the site-side TS formatter; the wasm-surface exposure
(so the site can drop its formatter) is a deferred follow-up.
**Pretty (indented) JSON-LD.** The JSON-LD writers have a matching pretty variant —
`graph_to_jsonld_pretty(&g, form)` / `write_jsonld_pretty(&named_graphs, form, &prefixes, &opts)`,
or the `graph_to_jsonld_pretty_with(&g, form, &prefixes, &opts)` wrapper — taking
`JsonLdPrettyOptions { indent }` (default: two-space indent). Unlike the Turtle pretty writer
this is **whitespace-only**: it produces the *exact* minified JSON-LD document and re-indents it
(each `{`/`[` opens an indented level, each member/element on its own line; empty `{}`/`[]` stay
compact). So it reuses the minified writer's already-deterministic first-seen subject/predicate
ordering unchanged, round-trips to the identical RDF, and stays dependency-free (a small
hand-written JSON re-indenter — no `serde_json`, which is dev-only). Indentation is presentation
only; the document content is byte-for-byte the minified form plus newlines/indent.
**Streaming Turtle / TriG (opt-in `streaming-serialization`, implies `serialize-rdf`).** The plain
`write_turtle` / `write_trig` build a *full* in-memory `String` for the whole graph, so a large
CONSTRUCT/DESCRIBE holds both the materialised triples and the fully-rendered string at once. The
opt-in `streaming-serialization` feature adds `write_turtle_streaming(triples, &prefixes, &mut w)`
and `write_trig_streaming(&named_graphs, &prefixes, &mut w)` (plus the whole-graph
`graph_to_turtle_streaming(&g, &prefixes, &mut w)` / `graph_to_trig_streaming`) that render the body
directly into any `W: std::io::Write`, buffering only **one subject block at a time** (emitting on
subject change) — so the whole rendered output is never materialised. <!-- sq-0kq6k -->
These are the writers behind `sparq-cli dump <file> <in> turtle|trig` under the CLI feature of the
same name. They are **not** what serves an HTTP CONSTRUCT/DESCRIBE: `sparq-server` renders a graph
response with `oxttl` / `oxrdfxml` (a different writer with a different output shape) and streams
through *those* serialisers' `io::Write` seam, so its response bytes are unchanged — see
`skills/http-server/SKILL.md`. The
streamed bytes are **byte-identical** to the buffered `write_turtle` / `write_trig` for the same
graph (same used-prefix header, same subject grouping, same ordering): both share the prefix-header
and per-subject-block rendering, and graph-sourced triples are subject-contiguous (`iter_ids()` walks
the SPO permutation), which is the precondition for that equality. The memory / time-to-first-byte
payoff is a measurable hypothesis on the canonical host, not a baked-in number. Zero new deps (std
`io::Write` only). Reproduce: `cargo test -p sparq-engine --features streaming-serialization`.
**Full W3C JSON-LD 1.1 Compaction (caller `@context`).** `JsonLdForm::Compacted` above is the
*lighter* prefix-only `@context` (it just abbreviates IRIs to `prefix:local` CURIEs). For the
real **W3C JSON-LD 1.1 Compaction Algorithm** against a caller-supplied `@context`, use
`graph_to_jsonld_compact(&g, &ctx)` (or the slice-level `write_jsonld_compact(&named_graphs, &ctx)`)
— still hand-rolled and **dependency-free** (a tiny internal `Json` AST — `JsonLdValue` — no
`serde_json`, no json-ld crate). It implements: **term definitions** (`{"name":"http://…/name"}`
or expanded `{"@id":…,"@type":…,"@language":…,"@container":…}`), **`@vocab`** (bare vocab-relative
terms for predicates and `@type` values), **type coercion** (a term `@type` matching a value's
datatype collapses the value object to a bare scalar; `@type:@id`/`@vocab` collapse a node
reference to a bare IRI string), **language coercion** (a term `@language` or the document default
`@language` drops a matching `@language`), **`@container`** — `@set` (always an array, no
compact-arrays), `@list` (strips the `{"@list":…}` wrapper to a bare ordered array — but only the
*pure* `{"@list":…}` value: any co-located non-list sibling stays under the property IRI, never
dropped, sq-oy1f.8), `@language` (a `{lang: value(s)}` map — a value with **no** language tag falls
under the reserved `@none` member rather than being dropped, sq-oy1f.9; several values that share
one language slot accumulate into an **array** so none is overwritten, and a **non-string** value —
a typed/numeric literal — routes to a separate non-language key so a strict processor keeps its
datatype rather than rejecting an invalid language-map value, sq-oy1f.12/.14) and `@index` (a
`{index: value(s)}` map; fromRdf has no per-value index, so values share the reserved `@none`
slot, accumulating into an array, sq-oy1f.14). `@id` / `@graph` containers — which the fromRdf
model cannot losslessly populate — **fall back to the default (no-container) framing** rather than
emit a container map a strict processor rejects (sq-oy1f.14). **`@reverse`** terms (forward edges
whose predicate a reverse term maps relocate onto the object node and are emitted as a **forward
member keyed by the reverse term**, NOT inside an `@reverse` block — a reverse-term key inside an
`@reverse` block would double-invert and a strict processor reads the edge backwards, sq-oy1f.12;
an edge whose object is *not* itself a subject is kept as a forward property, never stripped,
sq-oy1f.10), **`@id`/`@type` keyword aliasing** (`{"id":"@id","type":"@type"}`), and **value + node
+ IRI compaction** against the active context (vocab-relative compaction emits the `@vocab`-stripped
form even when the suffix contains `/` or `#` — only a `:` suffix is kept full, since it is ambiguous
with a compact IRI on read-back, sq-oy1f.11; a plain **literal** value under a `@type:@id`-coerced
term moves to a non-coerced key so it does not read back as a node IRI, sq-oy1f.13). Build the
context with `parse_context_json(r#"{…}"#)` (a string → `JsonLdValue`, returns `None` if not a JSON
object) or construct the `JsonLdValue` directly; the parsed `ActiveContext` drives compaction. The
output is the W3C Compaction Algorithm's document: `@context` merged into the single top-level node,
or `@context` plus a `@graph` array when there are several, with nodes in code-point order of their
expanded `@id`. The compaction is **lossless** — every coercion is invertible against the same
`@context`, so a JSON-LD-to-RDF round-trip reconstructs the original dataset (list cells typed
`rdf:List`, referenced from another graph, naming a graph or used as a type stay explicit nodes;
`rdf:JSON` literals stay typed strings with their exact lexical form; a predicate whose `@vocab`
suffix holds a `:` keeps its full IRI; an empty named graph is kept as `{"@id":…,"@graph":[]}`; when a `@type` map would hold a node object, which sparq's oxjsonld reader cannot load, the output is redone with that context's `@type` containers removed, and the emitted `@context` shows it). *Scope:* this is the fromRdf-then-compact (serialise) path — sparq always
emits RDF, so the input is a `Graph`, not an arbitrary remote document. Scoped and type-scoped
contexts, `@propagate` and `@protected` follow the W3C algorithm; a remote `@context` or `@import` is
never fetched, so such a context yields the lossless expanded document instead (JSON-LD
**Framing** is its own recipe below, sq-oy1f.17). *Strict third-party faithfulness:* the compaction
round-trip is verified both against sparq's own JSON-LD→RDF reader **and** differentially against the
**pyld** W3C reference processor (`expand`/`toRdf`) for the `@reverse`, language-map, `@type:@id`,
and multi-value-container shapes (sq-oy1f.12/.13/.14), so the emitted document re-expands to the
original triples in a strict external processor, not only in sparq. (An `@id`/`@graph`-container
*input* — i.e. compacting an already-indexed document, as distinct from the fromRdf graph sparq
produces — remains out of scope and falls back to default framing as noted above.)
*Native document-level pipeline (the Compaction and Framing writers now run on it).* The
algorithm gaps above (scoped/typed contexts, `@propagate`, `@protected`) are closed by a
from-scratch, dependency-free **document-level** pipeline in
the `sparq-jsonld` crate (design record `research/jsonld-1.1-design.md`, epic sq-oy1f). Its
Context Processing foundation (sq-oy1f.24) ships today: `sparq_jsonld::context::ActiveContext`
with `process()` (Context Processing + Create Term Definition, fallible with exact spec error
codes), `expand_iri()` (IRI Expansion), RFC 3986 reference resolution, and **deny-by-default**
remote-context loading via `DocumentLoader` (`NoopLoader` refuses every fetch — no ambient
network — while `FsLoader` serves trusted local fixtures). The document-level **Expansion
Algorithm** (sq-oy1f.25) ships too: `sparq_jsonld::expand(&input, &opts, &loader)` returns the
canonical expanded array — Value Expansion, scoped (property/type) contexts,
`@index`/`@id`/`@type`/`@language`/`@graph` container maps, `@nest`, `@reverse`, `@included`,
`@json` literals, keyword aliases, and the drop-null + array-normalisation rules — raising the
exact spec error code on invalid input and threading `frameExpansion`. **Node Map Generation +
Flattening** (sq-oy1f.26) ship next: `sparq_jsonld::generate_node_map()` (§7.2, deterministic
`_:bN` blank-node issuer) and `sparq_jsonld::flatten(&input, &opts, &loader)` (§7.1 = expand ∘
node-map ∘ named-graph fold, sorted by `@id`, empty nodes dropped), with the `flatten`
conformance lane moved to the native document oracle. The document-level **Compaction
Algorithm + Value Compaction** (sq-oy1f.27) ship too:
`sparq_jsonld::compact::compact(&input, &ctx, &opts, &loader)` (and `compact_expanded` over an
already-expanded document) runs the spec algorithm — scoped (property/type) contexts with
previous-context reversion, `@list`/`@language`/`@index`/`@id`/`@type`/`@graph` container
reshaping, `@nest`, `@reverse` redistribution, keyword aliasing, Value Compaction, and the
`compactArrays`/`compactToRelative`/`ordered` options — and the `compact` conformance lane now
compares against the W3C **expected** documents (the normative oracle; floor re-pinned 186 →
228, then raised to 243 by sq-gzsky RUNNING the negatives, see `floors::compact`). The
`expand` and `compact` lanes no longer SKIP `NegativeEvaluationTest`s: a negative passes iff
the algorithm raises EXACTLY the manifest's `expectErrorCode`, so a wrong code is a FAIL and
error-code completeness is now gated (expand floor 276 → 381). The document-level **Framing
Algorithm** (sq-oy1f.29) ships on the same substrate, and the engine-serialize cutover
(sq-oy1f.41) has landed for the two full-algorithm writers: `graph_to_jsonld_compact` is a
`from_rdf ∘ compact_expanded` adapter and `graph_to_jsonld_framed` a
`from_rdf ∘ frame_match_from_rdf ∘ compact_framed` adapter
over this pipeline, both under default options and the deny-by-default `NoopLoader`, so a
remote `@context` IRI is still never fetched (a network-fetching loader is sq-oy1f.32 and is
not in the tree). The writers stay total: a `@context` or frame the processor rejects yields
the lossless **expanded** document instead. The `JsonLdForm::{Expanded, Flattened, Compacted}`
writers (`graph_to_jsonld`, the prefix-only `@context` form) are unchanged and remain the
engine's own RDF-first emit path.
```rust
use sparq_engine::serialize::{graph_to_jsonld_compact, parse_context_json};
let ctx = parse_context_json(r#"{
"@vocab": "http://schema.org/", "id": "@id", "type": "@type",
"knows": {"@id": "http://schema.org/knows", "@type": "@id"}
}"#).expect("a JSON object");
let doc = graph_to_jsonld_compact(&g, &ctx); // term defs + @vocab + @type:@id coercion applied
// Indented (multi-line) variant — whitespace-only over the minified document, same triples:
// graph_to_jsonld_compact_pretty(&g, &ctx, &JsonLdPrettyOptions::default()).
```
Exposed on the wasm `Store` as `serializeCompact(context, pretty, indent?)` (the JSON-LD
SKILL note) and on the CLI as the `jsonld-compact[-pretty]` `dump` out-format (sq-oy1f.5).
**Document-level JSON-LD 1.1 pipeline — the `sparq-jsonld` crate (epic sq-oy1f, in progress).**
The RDF-first writers above have a hard conformance ceiling (scoped/typed contexts, `@nest`,
`@index` maps, negative tests — structures an RDF projection has already erased). The full
document-level pipeline (Context Processing, Expansion, Flattening, Compaction, Framing,
from/to-RDF) is being built in the new **opt-in, dependency-free `sparq-jsonld` crate**
(`crates/sparq-jsonld`, design record `research/jsonld-1.1-design.md`). Phase A (sq-oy1f.23)
scaffolds it: the writer's `Json` AST moved out of `sparq-engine` into `sparq_jsonld::Json`
(public API preserved — `sparq_engine::serialize::JsonLdValue` re-exports it, byte-identical
output), alongside the full `JsonLdErrorCode` registry, `JsonLdOptions`, and a **deny-by-default**
`DocumentLoader` (the default `NoopLoader` refuses every remote load — no ambient network). The
crate is `serialize-rdf`-gated on the engine side, so the lean default + wasm builds are unchanged.
The algorithms land in the follow-on beads (sq-oy1f.24+); until then the writer above is the JSON-LD
emit path.
**W3C JSON-LD 1.1 Framing (caller frame).** `graph_to_jsonld_framed(&g, &frame)` (or the
slice-level `write_jsonld_framed(&named_graphs, &frame)`) reshapes the graph into a deterministic
tree matching a caller **frame** document, then compacts it against the frame's `@context`. Build
the frame with `parse_context_json` (a frame is a JSON-LD document). Hand-rolled and
dependency-free, same `Json` AST as compaction — no `serde_json`, no `json-ld` crate (no Rust
JSON-LD crate ships framing). It implements:
* **Node-pattern matching** — `@type` (explicit list / wildcard `{}` / match-none `[]`), property
*presence* (`{}`), *absence* (`[]`), specific `@value` / `@id` match, and `@id` selection.
`@requireAll: true` combines the frame's properties with AND; the default with OR (a `{}` frame matches all).
* **`@embed`** — `@once` (default: embed a node the first time it is reached result-wide, a
`{"@id":…}` reference thereafter), `@always` (re-embed every reference), `@never` (always a
reference); `@link` is treated as `@always`. A blank-node cycle (`a→b→a`) terminates via the
embed link table (the back-edge becomes a reference).
* **`@explicit: true`** — keep only the properties the frame names. **`@default`** — the value
emitted for a framed property absent from the matched node (`@default: @null` → a preserve-`null`
marker). **`@omitDefault: true`** — drop an absent framed property instead of emitting its
default/null. **List framing** keeps the `{"@list":…}` wrapper, framing each element.
Output is a `{"@context":…,"@graph":[…]}` document, collapsing to the bare framed node merged with
`@context` for a single matched root (the `omitGraph` default). Nodes in named graphs are matched
against the frame together with the default graph (the W3C merged node map), so a matched node is
emitted bare, without a graph wrapper. The framed shape is
**differential-tested byte-for-byte against the pyld reference processor's `frame`** across the
flag matrix (sq-oy1f.17), and **ratcheted against the official `w3c/json-ld-framing` suite**
(sq-oy1f.19; see the conformance bullet below) — each `jld:FrameTest` expanded input is framed and
compared by RDF-equivalence to the normative expected output, which quantifies the real gap. *Scope:*
the serialise (fromRdf-then-frame) path — sparq supplies the graph + an inline frame. The W3C suite
documents the current below-floor divergences honestly (value-pattern matching over `@value`
alternative arrays, some `@explicit`/`@default` and named-graph `@graph`-frame shapes); `@reverse`
frames and scoped/typed frame contexts remain follow-up beads (children of sq-oy1f).
```rust
use sparq_engine::serialize::{graph_to_jsonld_framed, parse_context_json};
let frame = parse_context_json(r#"{
"@context": {"@vocab": "http://schema.org/"},
"@type": "Person", "knows": {"@embed": "@never"}
}"#).expect("a JSON object");
let doc = graph_to_jsonld_framed(&g, &frame); // selects Person nodes; knows → bare {"@id":…}
```
From the CLI (opt-in `serialize-rdf` feature) — re-serialize a loaded document to stdout:
```bash
cargo build -p sparq-cli --features serialize-rdf
./target/.../sparq-cli dump data.trig trig nquads
# out-format: turtle | turtle-pretty | trig | trig-pretty | nquads | ntriples
# | jsonld[-expanded|-flattened|-compacted]
# | jsonld-pretty[-expanded|-flattened|-compacted]
# | jsonld-compact[-pretty] (FULL W3C Compaction; needs --context <ctx.jsonld>)
# (bare `jsonld` == jsonld-expanded; `turtle-pretty`/`trig-pretty` emit the deterministic,
# idiomatic Turtle/TriG from recipe 6 — sorted, blank-line-separated subject blocks;
# the `jsonld-pretty*` forms emit indented JSON-LD, bare `jsonld-pretty` == expanded)
# Full 1.1 Compaction against your own @context (term defs / @vocab / coercion / @reverse):
./target/.../sparq-cli dump data.ttl turtle jsonld-compact --context ctx.jsonld
```
## Gotchas / feature flags / prerequisites
**One holder per directory:** `Graph::open`, `open_or_create`, `save`, `save_compressed`,
`restore_into_durable`, `restore_in_place` and the `build_external*` builders lock the
directory exclusively (`dir_lock`: advisory `flock` on Unix, `LockFileEx` on Windows; the lock
files are below) before touching it, all through one ownership check. A store's files are reachable only through its
`Graph` (`g.dict()` and `g.store()` read them; the store, dictionary and permutation openers
are crate-internal), and every memory-mapped part of a graph carries the lock: it is held while
the graph, any mapped `fork()`/`snapshot()`, or a graph rebuilt
around its dictionary (`Graph::into_dict` + `from_parts`) lives, and by the graphs `compact`,
`vacuum` and `restore_in_place` swap in. A second holder, in another process or this one, gets
`WouldBlock` at once. Rules that follow:
- **No save into the graph's own directory.** `g.save(dir)` into the directory `g` is open
from (compared by file identity, so `dir/`, `x/../dir` or a link to it count) is refused
with `InvalidInput`: `g` still maps and logs into the files a save would truncate. That
directory already holds `g`'s state: `g.restore_in_place(other)` replaces it with another
image (the supported in-place rewrite), `g.compact()` folds the log into its base, and
`g.is_backed_by(dir)` tells the case apart. A fork of `g` cannot save over `g`'s directory
either.
- **Ownership is per root, checked when a directory is claimed.** Every store directory
carries a marker file, `sparq-store` (`root <id>` or `child <id>`, the id a random identity
kept for the store's life), written before anything else. Opening, saving or building at a
path is refused (`InvalidInput`) when any ancestor of it is a store, so a named graph's
directory (`<root>/named/<i>`) belongs to its root. Only what creates a store (`save`, the
builders, `open_or_create`) creates missing parent directories; `open`, `open_read_only` and
`compact` of a missing store fail with `NotFound` and create nothing. Saving, building or
`restore_into_durable` into a directory is then refused when it holds another root store, a
store without a marker, a marker that cannot be read, or a held lock file, and only after
those checks recovers an interrupted swap of it. A store saved before markers counts as a
root by its dictionary or manifest file and is marked on its next writable open. These
checks see what is on disk when they run; they reserve nothing. Separate processes creating
nested stores (one inside the other's directory) at the same time are not supported.
- **Recovery touches only its own staging.** A swap (`compact`, `vacuum`, the restores, a
named-graph drop) stages beside the directory under names appended to its whole name
(`<dir>.compact-new`/`-old`, so `g.a` and `g.b` never share one; `named.drop-new`/`-old`),
taking those paths' lock files first (`<staging>.lock`; a named graph's are inside the
store, `named/<i>.compact-new.lock`). A staging directory is made under a scratch name
(`<staging>.sparq-part`), given its record `sparq-staging` (`<kind> <store id>`), and only
then renamed to its staging name, so it never exists without one. Through the renames one
half always carries a record: the old base is marked once it is aside, the new one loses its
record just before it is renamed in; a rollback marks the new half again before renaming the
old one back, and clears the old one's record last. Cleanup removes the other entries,
moves the record-only directory to its `.sparq-part` name, then removes its record and
directory; recovery can discard an empty scratch after a stop between those removals.
A named drop saves its matching manifest with the staged survivor tree and recovery
publishes that manifest before removing the old tree.
Recovery, and the next swap, promote or remove a staging directory only when its record
names this store, take a half caught between a rename and its record only when its own marker
carries that store's identity, and remove a record a crash left in the live directory.
Anything else at those names (another store, a scratch directory holding more than a record,
or staging an earlier release left, which records nothing) is refused with an error and left
in place, never removed (it may hold the only copy of a store): move it by hand, then retry.
No open, save or build claims a directory carrying `sparq-staging` on its own: open the store
it belongs to. A writable open checks the store's internals before recovering anything.
- **Advisory, over a namespace you trust.** All of this guards against mistakes and other
well-behaved processes, not against someone racing the filesystem: paths are checked and
then opened by name (no `O_NOFOLLOW`/`openat`), so a change between the check and the open
is not caught. Each file is published on its own, so a save or build is not atomic across
a store's files; the swaps are what replace a store as a whole.
- **A store's internals are not links.** An open, and a save or build into an existing
directory, is refused when a file or named graph inside the store is a symbolic link, or a
file written in place (`wal.log`, `txn.log`) has another hard link. Every other file is
written to a fresh `<name>.tmp` sibling, synced and renamed over its target (taking the
replaced file's permissions on Unix), so a hard-linked snapshot of a store (`cp -al`,
`rsync --link-dest`) is allowed and never written through: on Unix, `rename` and then an
fsync of the directory; on Windows, `MoveFileExW` with `MOVEFILE_WRITE_THROUGH` (there is no
directory fsync). A crash leaves the old file or the new one, never a missing or truncated
one. A build's side files (sort runs, dictionary spills, the per-graph spill of a quads
build) go in a private `.sparq-*` scratch directory made afresh in the store directory and
removed when the build ends; scratch a killed build left is removed by the next save,
build or restore into that directory.
- **Tool outputs stay out of stores.** `sparq_core::dir_lock::is_inside_store(path)` says
whether a path is a store directory or inside one (`store_root(path)` names the store), and
fails closed: a path that cannot be resolved, such as a dangling link, is an error.
`dir_lock::is_reserved_store_path(path)` also covers what a store reserves beside itself (its
`<dir>.lock`, a `<name>.lock` of anything at `<name>`, any lock file held now, swap staging
names and their `.sparq-part` scratch, `sparq-store`, `sparq-staging`, `sparq.lock`,
`.sparq-*` scratch), checking the entry itself (so a symbolic link inside a store counts
even when it points out) and where it leads. Reserved names and named-staging suffixes
match in any letter case, including Unicode aliases of reserved ASCII letters (and, on
Windows, ignoring trailing dots and spaces), since a case-insensitive file system resolves
`data.LOCK` to `data.lock`. `sparq-cli` refuses such a path for `reason`, `classify`, tabular
and benchmark `--json` outputs (exit 2, before any work), and writes every output to a fresh,
randomly named owner-only (`0600`) sibling it renames over the path, so a hard link to a
store's file is replaced, never written through; just before the rename it takes the
replaced file's mode, or a new file's default under the umask. A symbolic link is followed,
link by link, to the file it names, and that file is the one checked and replaced (the link
stays); a dangling link is an error. A device, pipe or descriptor path (`/dev/stdout`,
`/dev/fd/N`, `/proc/<pid>/fd/N`) and anything else that is not a regular file is written in
place. `probe-compress` maps a store's file only under the store's shared lock.
- **One path per directory.** The directory is resolved once (symbolic links and `.`/`..`;
a dangling symbolic link is an error) and that path is used for the lock, the log, crash
recovery and compaction swaps, so compacting through a link rewrites the real directory and
leaves the link alone. `open_or_create` recovers an interrupted swap before deciding to
seed. Two bind mounts of one directory are two paths: not excluded.
- **Read-only opens:** `Graph::open_read_only(dir)` takes available outside and inside locks
SHARED (each created when missing and creatable). The inside lock is required; a reader
that cannot open or create it returns an error.
Read-only opens coexist with each other, never with a writable holder (`WouldBlock` either
way), write nothing into the directory but a missing lock file, and refuse every write (also
the durable clears, and a rebuilding `sparq_engine::update`; `Graph::check_writable` says
so). Pending WAL and transaction records, including named graphs, are replayed in memory;
the logs stay unchanged. Interrupted swaps or named-graph drops require a writable open
for directory recovery first. This also works with only the inside lock under a read-only
parent, where root-directory compaction is unavailable. It can be saved to another directory. A store made under a `077` umask has lock files only its owner can open, so other
users cannot open it, even read-only.
- **A swap releases the graph's files first.** `compact`, `vacuum` and `restore_in_place`
stage the new base, then the graph lets go of every file it has in the directory (maps, logs,
named graphs) before the renames, since Windows refuses to rename a directory with a file in
it open or mapped. On Windows, a fork or snapshot that still maps the store makes the swap
fail (the directory is left as it was). Durable named graphs stay within their root store;
the named accessors refuse to take them out.
- **A failed swap** (`compact`, `vacuum`, `restore_in_place`) leaves the directory old-or-new,
never lost: it is recovered at once, under the graph's lock. Completed, the call succeeds and
the graph is the new store; rolled back, the error is returned and the graph keeps logging to
the old store. Only when neither works is the graph frozen (`Graph::is_frozen()`): every
later write is refused, and it reads the directory's contents opened read-only with its logs
replayed in memory, or nothing if they do not open. Unwinding during a swap also keeps
it frozen, with an empty placeholder for the graph being swapped. Report it as failed, drop it
and re-open the directory: the embedded LWS store and the Python `Graph` (`frozen`) raise an
error rather than serve one.
- **Special files:** store paths must name directories, and existing lock targets must be
regular files. Devices, pipes, and sockets are refused before they are opened as stores
or locks, including special files reached through symbolic links. Windows DOS device
aliases are refused by basename before metadata lookup.
- **The lock files** are `<resolved dir>.lock` beside the directory and `sparq.lock` inside
it (reserved). Every opener takes the outside one first, when it exists or can be created (an
existing one is opened read-only when it cannot be written: a lock needs no write access),
then the inside one, always. A store whose parent is not writable (a mount root, say) and has
no `<dir>.lock` is held by its inside lock alone: it opens writable or read-only and keeps one
writer, but `compact`, `vacuum` and the restores, which rename the directory, are refused
with an error naming the parent (create `<dir>.lock` once, make the parent writable, or keep
the store in a subdirectory).
Filesystem and drive roots have no sibling lock location and also use their inside lock
alone. They support ordinary opens and updates; place the store in a subdirectory for
compaction, vacuum, or restore operations that rename it.
A swap holds the old directory's inside lock until the old directory is gone and locks the
new one's before putting it in place (on Windows, which
cannot rename a directory with a file open in it, it lets go for the renames and refuses
writes afterwards if someone took the lock in between). A swap also leaves the lock files of
its staging paths (`<dir>.compact-new.lock`, `<dir>.compact-old.lock`). Anyone who can open a
lock file can take a shared lock on it and so keep a writer out; a child made by `fork()`
shares its parent's locks (they belong to the open file) and holds them while it lives.
Never delete a lock file as stale while the directory may be in use. Network file systems
are not supported.
- **Format strings are matched literally — an unknown format is REJECTED, not
guessed.** Accepted aliases:
`"turtle"`/`"ttl"`/`"text/turtle"`/`"application/turtle"`,
`"ntriples"`/`"n-triples"`/`"nt"`/`"application/n-triples"`,
`"nquads"`/`"n-quads"`/`"nq"`/`"application/n-quads"`, `"trig"`/`"application/trig"`
(and the `"jsonld"`/`"json-ld"`/`"application/ld+json"` set when the `jsonld` feature is
on). `parse_to_triples` (and the `_with_base` variants), `load_dataset`/`load_dataset_with_base`/`load_dataset_serial`
gate on these explicit alias sets: any unknown/typo'd string returns
`Err("unknown RDF format \"…\" (known: …)")` naming the bad format — it does **not**
silently fall back to Turtle/TriG (the old `_ =>` catch-all was removed in sq-m2pc /
sq-01yr; verify: `cargo test -p sparq-core parse_to_triples_rejects_unknown_format
load_dataset_rejects_unknown_format`). Pass the exact string.
- **Parallel paths need the `parallel` feature** (on by default natively). The parallel
fast path applies to N-Triples and Turtle in `load_str`; `load_reader_parallel`'s
pipelined parser is **N-Triples only** (other formats silently fall back to serial
`load_reader`). With `parallel` off (e.g. the wasm build, `--no-default-features`),
everything parses serially. The parallel and serial parses are pinned **result-equivalent**
by a differential test (`crates/sparq-core/tests/parallel_serial_load_differential.rs`,
sq-bif.13): the same Turtle + N-Triples document loaded under `--features parallel` and under
`--no-default-features` answers every term-level probe (triple/term counts, full dump,
pattern scans) identically to a feature-independent reference graph.
- **RDF 1.2 triple terms are first-class.** A triple term `<<( s p o )>>` is a
real `oxrdf::Term::Triple` (object position only — RDF 1.2 makes triple terms object-only;
a `<<( … )>>` in subject/predicate position is rejected with a precise error). Triple terms
may nest, take blank-node/literal components, and are content-addressed (an identical triple
term shares one dict id). They load through **every** path — serial, parallel-chunked,
streaming-pipelined, the sharded external builder, and the **dict-spill** external builder
(sq-jvbr: triple-term occurrences take an in-RAM arena finalised after the spilled leaf
consolidation, content-addressed by their components' final ids and assigned ids after every
leaf) — at full parallelism (the in-memory parallel merge no longer drops to a serial
fallback when they are present). In **Turtle/TriG**, the SPARQL-1.2
reification sugar is supported via the Turtle parser: the reifying triple `<< s p o >>`
(subject or object position, optionally `<< s p o ~ reifier >>`) and the annotation block
`s p o {| … |}` desugar to the standard `rdf:reifies <<( s p o )>>` form (the annotation block
also **asserts** the base triple; a bare `<< … >>` reifier does not). N-Triples/N-Quads carry
only the desugared `<<( … )>>` triple-term form (no `<<>>`/`{| |}` sugar — per the line-format
grammar). Nested triple terms are **depth-bounded** (`MAX_TRIPLE_TERM_DEPTH = 128` in
`sparq-core::nt`): the byte-level N-Triples/N-Quads parser is the only native-recursion RDF
parse path, so a pathologically nested `<<( … )>>` chain returns a clean parse error rather
than overflowing the stack (ASVS V5.5.2 / sq-53s1). The Turtle/TriG/N-Quads path via `oxttl`
is a heap-stack pushdown automaton and cannot recurse the native stack. The SPARQL parser has
the matching `MAX_RECURSION_DEPTH = 128` cap (groups/expressions/paths/collections/triple
terms). 128 is far deeper than any real data/query nests.
- **Language tags are normalised to lowercase on ingest.** A `@lang` tag is stored ASCII-lowercased
across **every** format and path — Turtle/TriG/N-Quads via `oxttl`, and the byte-level
N-Triples/N-Quads fast parser (`sparq-core::nt`). RDF 1.1/1.2 language tags are case-insensitive,
so `"x"@en-US` and `"x"@en-us` are the **same** RDF term and intern to one dict id (matching
`oxrdf::Literal::new_language_tagged_literal`). This makes ingest format-INDEPENDENT: the same
triple written as N-Triples or as Turtle yields the identical stored term (it did **not** before —
the byte path used to preserve the written case while every `oxttl` path lowercased it; KamiQuasi
#1119 / sq-langcase). The RDF 1.2 `--ltr`/`--rtl` direction is unaffected (already lowercase).
- **`build_external` / `open` / `save` require the `mmap` feature** (native only). The
N-Triples external path also honors `SPARQ_SHARDED_DICT` (default on with ≥2 threads)
and, with the `dict-spill` feature + `SPARQ_DICT_SPILL` env, bounds peak build RSS by
spilling the term dictionary to disk (byte-identical output). The spill path's activation
+ mmap-reload edges are pinned by `crates/sparq-core/tests/dict_spill_activation.rs`
(sq-bif.13): a tiny `mem_budget` (constant cache eviction across many epochs + many spilled
sort runs) yields a build byte-identical to a comfortable-budget build, a `disk_floor` above
free disk aborts cleanly through the spill pipeline's resource gate, and a spill-built store
reloads via `Graph::open` (and re-saves raw/compressed) to identical content. External build
folds N-Quads/TriG named graphs into the default graph (only `load_dataset` preserves them).
All store writers emit `predstats.bin` in ascending predicate-ID order, so equal
statistics serialize identically after reload. Older unordered files remain readable.
- **HDT is opt-in and native-only.** `sparq-hdt` declares MSRV **1.87** (the wrapped `hdt` crate;
the workspace floor is now 1.88) — in the CLI it is gated behind `--features hdt`. It carries
zero code into the wasm build. Compression containers are detected by **magic bytes, not
file extension**, so a mislabeled `.hdt` still loads; all three (`gz`/`zst`/`bz2`) decode
in a streaming fashion.
- **HDT triple-pattern loading is separately opt-in.** Enable `sparq-hdt`'s
`load-filter` feature for `TriplePattern`, `load_reader_filtered`, and
`stats_reader_filtered`. Both filtered operations share one resolved-pattern
SPO walk. The stats call counts matches without building a result graph; its
dictionary cardinalities remain archive-wide. The loader does not accumulate,
intern, or index rejected triples, so result memory follows the matches.
- **HDT format coverage:** standard v1.0 layout only (FourSectionDictionary / Plain Front
Coding + BitmapTriples, SPO order) as emitted by hdt-cpp / hdt-java. Exotic submission
layouts are rejected with an error.
- **Writing HDT is opt-in** (`sparq-hdt`'s `write` feature): `save(&graph, path)` emits the
same standard v1.0 layout. The path currently round-trips through a **temporary
N-Triples file** via the wrapped crate's builder (`Hdt::read_nt` + `Hdt::write`, behind
its `sophia` feature) — correct and interoperable, but it re-serialises + re-parses the
whole graph, so it is NOT free. A direct in-memory builder (no text round-trip) is queued
upstream (`sparq-hdt/UPSTREAM.md`). HDT carries a single default graph, so `save` ignores
named graphs.
- **`load_dataset` is in-memory only.** Numeric/temporal filter caches are built on load
in all in-memory paths; `into_compressed()` / `load_str_compressed()` trade a small
per-scan decode for ~2.5× more triples per byte of RAM (browser target).
- **`block-bloom` (opt-in, OFF by default).** A block-compressed permutation already has an
implicit min/max zone map (the directory's first-triple) that prunes RANGE scans, but for an
EQUALITY-BOUND leading column (a point/prefix lookup on a constant subject/object) the zone map
still leaves a candidate block that must be decoded just to discover it holds no matching row.
The `block-bloom` feature builds a tiny per-block Bloom bitset over each block's distinct
leading ids (built only for high-NDV columns past a density gate; pure-std, no new dep) and
probes it in `range` before decoding a block. Zero false negatives by construction, so results
are IDENTICAL to the feature-off path — only fewer blocks decode. The filter is in-RAM/build-time
only: it is NEVER written to the on-disk `SPQCPRM1` format, so a perm built with the feature on
persists byte-identically to one built with it off, and a memory-mapped perm carries no filter.
**The measured skip opportunity is on ABSENT-key lookups.** On the SYNTHETIC high-NDV columns
the sq-v8ixk harness generates (`measure_bloom_*` in `compress.rs`), the original "the id
overlaps several blocks" rationale does not show up: because those columns are high-NDV, a
PRESENT id's rows are contiguous and the zone map already narrows the candidate window to about
one block, so there is little left for the filter to skip. The larger skip opportunity is an
equality-bound id that is ABSENT from the permutation — a subject/object bound by an earlier
pattern with no rows here — where the candidate block can be dropped undecoded. Those are
block-SKIP COUNTS on a synthetic distribution: host-independent, but they establish no latency
result and do not settle the question for real workloads. Run the harness for figures; none is
asserted here, and the canonical run against a real dataset on the perf host is bead `sq-0g6g`
(PENDING).
- **`elias-fano` (opt-in, OFF by default) — compressed-seek codecs, PROTOTYPE, not wired in.**
`sparq-core = { …, features = ["elias-fano"] }` exposes `sparq_core::eliasfano` with
`EliasFano` and `PartitionedEliasFano` (plus `EliasFanoIter` and
`PartitionedEliasFano::DEFAULT_PARTITION`). Their `next_geq(target)` answers a successor query
*directly on the compressed data* — no whole-block decode, unlike the varint block codec. Pure
`std`, no new dependency.
- **Input MUST be non-decreasing.** `encode` returns `None` on a non-monotone slice (EF is
undefined there — that means the caller mis-segmented the column), and also `None` when the
high-bits vector would exceed `u32::MAX` bits. An empty slice encodes fine. A permutation
column is monotone only *within* one leading-prefix group, so you segment first.
- **Nothing in the store calls this yet.** It is a measurement-gated spike: not a `PermData`
variant, no join uses it, and its A/B against the incumbent codec
(`eliasfano::tests::measure_ef_vs_block_varint`, `#[ignore]`d) is **native-only** — it times
with `std::time::Instant`, so the `wasm32` half of the comparison is unresolved. Treat
adoption as an open question; no performance number is asserted.
```rust
use sparq_core::eliasfano::EliasFano;
let ef = EliasFano::encode(&[3, 3, 9, 40, 41]).expect("non-decreasing");
assert_eq!(ef.get(2), Some(9));
assert_eq!(ef.next_geq(10), Some((3, 40))); // first (index, value) >= target
assert_eq!(ef.next_geq(99), None); // every element is smaller
assert_eq!(ef.iter().collect::<Vec<_>>(), vec![3, 3, 9, 40, 41]);
assert!(EliasFano::encode(&[5, 1]).is_none()); // not non-decreasing
```
- **SPQCPRM2 frame-of-reference col2 encoding (versioned on-disk format; V2 emission opt-in).**
A second block-stream version, `SPQCPRM2`, alongside the shipped `SPQCPRM1`. Where `SPQCPRM1`
writes a middle-column-reset col2 id ABSOLUTE, `SPQCPRM2` writes it as a zigzag signed delta from
the block's first-row col2 (the frame origin the decoder recovers at block start), so clustered
objects encode as short varints. A V2 block with no middle-column reset is byte-identical to its
V1 block.
- **The V2 READER ships with `mmap`.** `open`/`from_mmap` auto-detect the 8-byte file magic —
`compress::FILE_MAGIC` (`SPQCPRM1`) or `compress::FILE_MAGIC_V2` (`SPQCPRM2`) — and pick the
matching block reader; a `CompressedPerm` carries its `format` (`enum Format { V1, V2 }`). **An
existing `SPQCPRM1` file on disk keeps decoding byte-identically forever** (the backward-compat
soundness invariant). ANY OTHER 8-byte magic is a loud, clean `Err` — never a silent misdecode
(mutation-witnessed in `mmap_corruption_oracle`).
- **A build EMITS `SPQCPRM1` by DEFAULT.** V2 is NOT defaulted on (a separate decision pending a
clearly-positive real-data B/triple measurement — the col2-clustered synthetic win did NOT hold
on WatDiv). The `spqcprm2` cargo feature (which implies `mmap`) exposes the emit config gate:
`compress::with_emit_format(EmitFormat::V2, || …)` on a thread, or `SPARQ_EMIT_FORMAT=v2` for a
process, routes the store's `save_compressed`, the streaming `CompressedPermWriter`, AND the
in-RAM compressed profile (`TripleStore::from_triples_compressed` / `Graph::into_compressed`,
i.e. `SPARQ_STORE_PROFILE=compressed` — sq-559dp) through the V2 encoder. On the CLI
that same choice is a per-invocation FLAG — `sparq-cli save …
compressed --format-v2` / `sparq-cli recompress … --v2` (sq-kmve2) — which maps
onto the per-thread override (so it beats the env var) and is a hard exit-2 error on a build
without `spqcprm2`, never a silent V1 write. `CompressedPerm::encode_v2` builds a `V2` perm directly; `encode_emit` honours
the gate in one place. With the feature OFF, `emit_format()` is a `const V1`, so the default
build cannot emit V2 and every shipped index stays `SPQCPRM1` bit-for-bit.
- **Public API (`sparq_core::compress`).** Always: `fn CompressedPerm::encode_emit` (a plain
inlined `encode` when the gate is compiled out, so the `mmap`-off wasm in-RAM stream is
byte-identical). With `mmap`: `enum Format`, `const FILE_MAGIC_V2`.
With `spqcprm2`: `enum EmitFormat`, `fn with_emit_format`,
`fn CompressedPerm::encode_v2`, `fn CompressedPermWriter::create_with`.
- **JSON-LD W3C conformance is RATCHETED (honest baseline, not 100%).** A ratcheted W3C
JSON-LD 1.1 conformance gate (sq-oy1f.2 + sq-3uos5 + sq-oy1f.19 + sq-oy1f) drives the official
`w3c/json-ld-api` suite AND the SEPARATE `w3c/json-ld-framing` suite through the real paths:
**toRdf** through the `jsonld` parser (oxjsonld), and **fromRdf**, **expand**, **flatten**,
**compact** + **frame** through the **native `sparq-jsonld` document pipeline**
(`sparq_jsonld::from_rdf()` sq-oy1f.28 / `expand()` sq-kk1mq / `flatten()` sq-oy1f.26 /
`compact::compact()` sq-oy1f.27 / `frame::frame()` sq-oy1f.29). **toRdf** compares the parsed
dataset against the suite's NORMATIVE expected N-Quads by canonical RDF-dataset equality
(a negative case passes when the parse fails); the five native lanes are compared **document-to-document**
against the suite's NORMATIVE expected output via the `json_ld_equal` comparator (key order
insignificant; array order significant only inside `@list` and a `@json` payload) — these
are JSON-LD normal forms / a SELECT+RESHAPE (they drop/merge/prune/fill), so the oracle anchors on
the expected document, not the input; **fromRdf** additionally keeps the round-trip leg where
§8.1 is lossless. The floors only RISE; they reflect ACTUAL current pass
counts, not full conformance — the
remaining toRdf divergences are documented oxjsonld limits (remote/`@import` `@context` needs a
`LoadDocumentCallback`; `expandContext`/`rdfDirection` options not applied; a few
base-normalization edge cases + leniently-accepted negative tests), and fromRdf skips the
one JSON-LD-1.0-only positive. The **compact** lane was RE-PINNED off the old engine-writer
self-reparse oracle (which measured RDF losslessness of `graph_to_jsonld_compact`, not the
Compaction Algorithm) onto the spec algorithm against the expected document, a strictly
harder oracle, and sq-gzsky then RAN the negatives (a case passes iff `compact()` raises
EXACTLY the manifest's `expectErrorCode`). Its 2 remaining below-floor FAILS are itemised,
not skipped, in `crates/sparq-conformance/src/floors/compact.rs`: `#te001` expects a
JSON-LD 1.0 error code the 1.1 REC removed, and `#te002` (`IRI confused with prefix`) is a
real 1.1 gap — `context::inverse::compact_iri` is infallible, so raising it needs a fallible
signature threaded through the compaction walk. The **frame** lane (over the separate framing suite) moved to the
**native `sparq-jsonld` document-level framer** (`sparq_jsonld::frame::frame()`, sq-oy1f.29 —
frame matching over the expanded document + node map: value patterns over `@value` alternative
arrays, `@explicit`/`@default` fill, named-graph `@graph` framing, `@list` re-emit, blank-node
`@embed` with 1.1 pruning, `omitGraph`/`frameDefault`, and the framing error codes). It holds a
FULL score at the pinned suite revision under the normative document oracle (`json_ld_equal`
against the suite's expected document), the 3 NegativeEvaluationTests now RUN (pass iff
`frame()` raises the manifest's exact `expectErrorCode`: `invalid frame`,
`invalid @embed value`), and the old-oracle→new-oracle re-pin (61/92 RDF-answer-equivalence →
92/92 document oracle) is documented side-by-side in the lib-side floor
(`sparq-conformance/src/floors/frame.rs`, rise-only after). The engine-serialize cutover
(sq-oy1f.41) has landed for these two forms: `graph_to_jsonld_compact` and
`graph_to_jsonld_framed` are now thin `from_rdf ∘ compact` / `from_rdf ∘ frame` adapters over
the same `sparq-jsonld` algorithms the lanes measure, so the writer and the conformance lane
are no longer separate code. The `@reverse` double-inversion / non-string-language interop
gap documented above (the compaction interop caveat) is tracked separately.
The **expand** + **flatten** lanes (sq-oy1f) GRADUATED out of the NOT-IMPLEMENTED bucket and
BOTH now run the **native `sparq-jsonld` document oracle**: the input is passed to
`sparq_jsonld::expand()` / `sparq_jsonld::flatten()` and the result is deep-compared to the
normative expected document via `json_ld_equal`. The `flatten` floor was RE-PINNED 50→53
(sq-oy1f.26) when it moved off the RDF-writer oracle: on the first cut the native lane
measured below the writer oracle purely through inherited native-`expand()` gaps
(empty-property retention, a few `invalid typed value` cases), and sq-oy1f.37 fixing those in
`expand()` flipped all of them, so the re-pin is a net rise. It is rise-only after the
re-pin. Below-floor cases are honest
divergences; the SKIP buckets are the documented ones (negatives, JSON-LD-1.0-only, the single
post-flatten-compaction case deferred to sq-oy1f.27). Only **html** (script extraction) +
**remote-doc** (a `LoadDocumentCallback`)
remain NOT-IMPLEMENTED buckets the runner reports separately and never fails on (they grow the
ratchet as those land). The lane is the opt-in
`jsonld-suite` feature on `sparq-conformance` (forwards to `sparq-core/jsonld` +
`sparq-engine/serialize-rdf`); OFF it compiles to a self-skip. Reproduce with
`scripts/fetch-jsonld-tests.sh` + `scripts/fetch-jsonld-framing-tests.sh` then
`cargo test -p sparq-conformance --features jsonld-suite --test jsonld_suite` (self-skips if
the gitignored suites are absent). It is registered in the central conformance scoreboard
(`sparq-conformance-scoreboard`).
- **JSON-LD comparative benchmarking lives in `bench/jsonld/`** (sq-hmd7l.15): the W3C
pass-rate table vs jsonld.js / titanium-json-ld's PUBLISHED results (pinned with provenance
in `bench/jsonld/conformance-peers.json` — sparq's compact/frame lanes use a round-trip
oracle, so those two rows are NOT 1:1 comparable) + an output-equality-gated
expand/flatten/compact/toRdf throughput harness (`bench/jsonld/run.sh --smoke|--gather`).
The document-level comparator + canonical-dataset helpers the ratchet AND the bench share
live lib-side in `sparq_conformance::jsonld_bench` (behind `jsonld-suite`); the sparq runner
is the `bench_jsonld` example on `sparq-conformance`. Gap record:
`research/gap-jsonld-2026-07.md`.
## See also
- `fused-decompress-parse` — choosing gzip vs zstd vs bzip2 and fusing decode with parse (measured numbers).
- `rust-parallel-parsing` — how the chunk-parallel N-Triples/Turtle scanners work, and when NOT to parallelize.
- `hdt-format` — the HDT binary layout internals and the `hdt`-crate wrapping/decode performance.