209 added, 209 removed. Audit A to A.
---
name: gtars
- description: High-performance toolkit for genomic interval analysis in Rust with Python bindings. Use when working with genomic regions, BED files, coverage tracks, overlap detection, tokenization for ML models, or fragment analysis in computational genomics and machine learning applications.
- license: Unknown
- metadata: {"version": "1.0", "skill-author": "K-Dense Inc."}
+ description: Use Gtars for local genomic interval models and set algebra, overlaps and counts, consensus and coverage, tokenization, fragment processing, and refget/BEDbase planning across Python, Rust, and the CLI.
+ license: MIT
+ compatibility: Python bindings require Python 3.10+ and gtars 0.9.2. The Rust meta-crate and gtars-cli are 0.9.0 and require a Rust toolchain supporting Edition 2024; upstream declares no rust-version. Bundled audit CLIs use only Python 3.10+ standard library and are local/network-free. Remote constructors, pretrained tokenizers, refget, and BEDbase caching require explicit network and storage approval.
+ allowed-tools: Read, Write, Edit, Bash, Glob
+ metadata:
+ version: "1.1"
+ skill-author: K-Dense Inc.
---
- # Gtars: Genomic Tools and Algorithms in Rust
+ # Gtars
- ## Overview
+ Gtars provides native Rust implementations, Python bindings, and a feature-gated
+ `gtars` binary for genomic interval and reference-sequence work. Start with the
+ bundled local inspectors; call upstream code only after the data contract,
+ provenance, resource bounds, and side effects are explicit.
- Gtars is a high-performance Rust toolkit for manipulating, analyzing, and processing genomic interval data. It provides specialized tools for overlap detection, coverage analysis, tokenization for machine learning, and reference sequence management.
+ ## Verified snapshot (2026-07-23)
- Use this skill when working with:
- - Genomic interval files (BED format)
- - Overlap detection between genomic regions
- - Coverage track generation (WIG, BigWig)
- - Genomic ML preprocessing and tokenization
- - Fragment analysis in single-cell genomics
- - Reference sequence retrieval and validation
+ - Python: [`gtars==0.9.2`](https://pypi.org/project/gtars/), released
+ 2026-06-17, `Requires-Python >=3.10`.
+ - Rust meta-crate: [`gtars=0.9.0`](https://crates.io/crates/gtars), released
+ 2026-06-15. Its default feature set is empty.
+ - CLI crate/binary: [`gtars-cli=0.9.0`](https://crates.io/crates/gtars-cli);
+ the installed binary is named `gtars`.
+ - Direct refget crate: [`gtars-refget=0.9.1`](https://crates.io/crates/gtars-refget),
+ released 2026-06-17. `gtars=0.9.0` itself pins its component release set, which
+ includes refget 0.9.0.
+ - Upstream intentionally versions workspace crates, Python bindings, and CLI
+ independently. Do not assume matching numbers mean matching artifacts.
+ - The published docs changelog stops at 0.5.1. API examples here were checked
+ against the 0.9.2 Python stubs/runtime and the `v0.9.0` CLI/Rust source.
- ## Installation
+ The `license: MIT` field covers this skill. Published `gtars` crates declare MIT,
+ while the GitHub repository currently displays BSD-2-Clause at the root; verify
+ the exact artifact's license before redistribution.
- ### Python Installation
+ ## Native-code trust gate and exact pins
- Install gtars Python bindings:
+ The Python wheel contains a PyO3 native extension. Cargo installation compiles a
+ native binary and can run dependency build scripts. Treat either path as code
+ execution:
+ 1. Confirm the official PyPI/crates.io/GitHub owner and immutable version.
+ 2. Review filenames, platform tags, release provenance, license, and SHA-256.
+ GitHub's v0.9.0 binary release includes per-archive `.sha256` sidecars.
+ 3. Never run an untrusted prebuilt binary, wheel, source tree, Cargo build script,
+ or archive installer. Use isolation and CPU/RAM/disk/time limits.
+ 4. Keep a lockfile and artifact hashes with the analysis manifest.
+
+ After that review, create an isolated Python environment:
+
```bash
- uv pip install gtars
+ uv venv --python 3.11 .venv-gtars
+ uv pip install --dry-run --python .venv-gtars/bin/python "gtars==0.9.2"
+ uv pip install --python .venv-gtars/bin/python "gtars==0.9.2"
+ .venv-gtars/bin/python -c \
+ "import gtars; assert gtars.__version__ == '0.9.2'; print(gtars.__version__)"
```
- ### CLI Installation
-
- Install command-line tools (requires Rust/Cargo):
+ For the reviewed CLI source release:
```bash
- # Install with all features
- cargo install gtars-cli --features "uniwig overlaprs igd bbcache scoring fragsplit"
-
- # Or install specific features only
- cargo install gtars-cli --features "uniwig overlaprs"
+ cargo install gtars-cli --version 0.9.0 --locked
+ gtars --version
+ gtars --help
```
- ### Rust Library
-
- Add to Cargo.toml for Rust projects:
+ For a Rust project, pin the wrapper exactly and enable only required features:
```toml
[dependencies]
- gtars = { version = "0.1", features = ["tokenizers", "overlaprs"] }
+ gtars = { version = "=0.9.0", default-features = false, features = [
+ "core", "overlaprs", "uniwig", "tokenizers", "refget"
+ ] }
```
- ## Core Capabilities
-
- Gtars is organized into specialized modules, each focused on specific genomic analysis tasks:
-
- ### 1. Overlap Detection and IGD Indexing
-
- Efficiently detect overlaps between genomic intervals using the Integrated Genome Database (IGD) data structure.
-
- **When to use:**
- - Finding overlapping regulatory elements
- - Variant annotation
- - Comparing ChIP-seq peaks
- - Identifying shared genomic features
-
- **Quick example:**
- ```python
- import gtars
-
- # Build IGD index and query overlaps
- igd = gtars.igd.build_index("regions.bed")
- overlaps = igd.query("chr1", 1000, 2000)
- ```
+ Use `gtars-refget = "=0.9.1"` directly only when the newer direct component API is
+ required and compatibility has been tested. Do not replace these pins with a Git
+ branch or an unreviewed release.
- See `references/overlap.md` for comprehensive overlap detection documentation.
+ ## Genomic data contract
- ### 2. Coverage Track Generation
+ Apply this contract before every operation:
- Generate coverage tracks from sequencing data with the uniwig module.
+ 1. **Coordinates:** BED intervals are 0-based and half-open: `[start, end)`.
+ Require `0 <= start < end <= contig_length`. Gtars coordinates are `u32`, so
+ reject values above `4,294,967,295`.
+ 2. **Assembly:** record an assembly accession/version and the SHA-256 of the exact
+ chromosome-sizes or refget sequence-collection metadata. Never infer assembly
+ from filenames or `chr` prefixes.
+ 3. **Contigs:** compare names exactly. `1` and `chr1`, alternate loci, decoys, and
+ mitochondrial aliases are not interchangeable. Rename or liftover only as a
+ separately reviewed transformation.
+ 4. **Sorting:** preserve the original file, then sort a copy by chromosome-sizes
+ order and numeric start/end when the operation requires it. Python
+ `RegionSet(path)` currently sorts lexicographically by contig and start while
+ loading; do not rely on original row order afterward.
+ 5. **Strand:** BED6 uses `+`, `-`, or `.`. `Region.rest` retains trailing BED
+ fields, but a file-backed Python `RegionSet` currently initializes its separate
+ `strands` vector to `*`. Several set operations drop strand. Preserve and
+ validate strand externally when it is scientifically meaningful.
+ 6. **Duplicates/adjacency:** choose policies explicitly. `reduce()` and consensus
+ merge overlapping **and adjacent** intervals; ordinary half-open overlap does
+ not treat `[0,10)` and `[10,20)` as overlapping.
- **When to use:**
- - ATAC-seq accessibility profiles
- - ChIP-seq coverage visualization
- - RNA-seq read coverage
- - Differential coverage analysis
+ Run the local validator first:
- **Quick example:**
```bash
- # Generate BigWig coverage track
- gtars uniwig generate --input fragments.bed --output coverage.bw --format bigwig
+ python3 -B scripts/bed_validator.py \
+ --input data.bed.gz \
+ --assembly GRCh38.p14 \
+ --chrom-sizes GRCh38.p14.chrom.sizes \
+ --require-sorted
```
- See `references/coverage.md` for detailed coverage analysis workflows.
+ ## Safe local workflow
- ### 3. Genomic Tokenization
+ 1. Inventory local files, checksums, assembly, contig dictionary, coordinate
+ system, strand policy, patient/replicate groups, and intended outputs.
+ 2. Validate BED/fragments and estimate work. Pilot a small synthetic file.
+ 3. Choose Python, CLI, or Rust from the documented surface; do not translate API
+ names by guesswork.
+ 4. Set hard limits for input bytes/records/files, threads/jobs, memory, temporary
+ disk, output size, and wall time.
+ 5. Run in a dedicated output directory. Refuse collisions unless overwrite was
+ explicitly approved.
+ 6. Revalidate output sorting, bounds, row counts, checksums, and provenance.
- Convert genomic regions into discrete tokens for machine learning applications, particularly for deep learning models on genomic data.
+ ## Current Python core
- **When to use:**
- - Preprocessing for genomic ML models
- - Integration with geniml library
- - Creating position encodings
- - Training transformer models on genomic sequences
+ Imports are from submodules, not the `gtars` top level:
- **Quick example:**
```python
- from gtars.tokenizers import TreeTokenizer
-
- tokenizer = TreeTokenizer.from_bed_file("training_regions.bed")
- token = tokenizer.tokenize("chr1", 1000, 2000)
- ```
+ from gtars.models import Region, RegionSet
- See `references/tokenizers.md` for tokenization documentation.
+ query = RegionSet.from_regions(
+ [
+ Region(chr="chr1", start=100, end=200, rest=None),
+ Region(chr="chr1", start=300, end=400, rest=None),
+ ],
+ strands=["+", "-"],
+ )
+ universe = RegionSet.from_vectors(
+ ["chr1", "chr1"],
+ [150, 500],
+ [350, 600],
+ )
- ### 4. Reference Sequence Management
+ counts = query.count_overlaps(universe) # one count per query region
+ flags = query.any_overlaps(universe) # one bool per query region
+ indices = query.find_overlaps(universe) # indices into universe
+ pieces = query.intersect_all(universe) # all intersection fragments
+ fraction = query.coverage(universe) # fraction of query bp covered
+ ```
- Handle reference genome sequences and compute digests following the GA4GH refget protocol.
+ `RegionSet.sort()` mutates and returns `None`. Set algebra includes `reduce`,
+ `setdiff`, `pintersect` (pairs by index), `concat`, `union`, `jaccard`,
+ `coverage`, `overlap_coefficient`, `intersect_all`, `closest`, `cluster`, and
+ `gaps`. Read `references/python-api.md` before relying on ordering or strand.
- **When to use:**
- - Validating reference genome integrity
- - Extracting specific genomic sequences
- - Computing sequence digests
- - Cross-reference comparisons
+ Consensus is a Python binding in a different module:
- **Quick example:**
```python
- # Load reference and extract sequences
- store = gtars.RefgetStore.from_fasta("hg38.fa")
- sequence = store.get_subsequence("chr1", 1000, 2000)
+ from gtars.genomic_distributions import consensus
+
+ rows = consensus([query, universe])
+ # rows: [{"chr": ..., "start": ..., "end": ..., "count": ...}, ...]
```
- See `references/refget.md` for reference sequence operations.
+ Signal-track generation is **not** exposed as `gtars.uniwig` in Python 0.9.2;
+ use the reviewed CLI or Rust API. `RegionSet.coverage()` is a base-pair set metric,
+ not a WIG/bigWig generator.
- ### 5. Fragment Processing
+ ## Tokenizers, fragments, and reference stores
- Split and analyze fragment files, particularly useful for single-cell genomics data.
+ Use only local constructors by default:
- **When to use:**
- - Processing single-cell ATAC-seq data
- - Splitting fragments by cell barcodes
- - Cluster-based fragment analysis
- - Fragment quality control
+ ```python
+ from gtars.models import RegionSet
+ from gtars.tokenizers import Tokenizer
- **Quick example:**
- ```bash
- # Split fragments by clusters
- gtars fragsplit cluster-split --input fragments.tsv --clusters clusters.txt --output-dir ./by_cluster/
+ tokenizer = Tokenizer.from_bed("reviewed-universe.bed")
+ regions = RegionSet("local-query.bed")
+ tokens = tokenizer.tokenize(regions)
+ encoding = tokenizer(regions)
+ ids = encoding["input_ids"]
```
- See `references/cli.md` for fragment processing commands.
-
- ### 6. Fragment Scoring
-
- Score fragment overlaps against reference datasets.
+ `Tokenizer.from_pretrained(name)` contacts Hugging Face and writes its cache when
+ the argument is not an existing local directory; it exposes no revision or cache
+ argument. Obtain explicit approval, fetch an immutable revision through a reviewed
+ mechanism, verify checksums, then pass the local snapshot directory. See
+ `references/tokenizers.md`.
- **When to use:**
- - Evaluating fragment enrichment
- - Comparing experimental data to references
- - Quality metrics computation
- - Batch scoring across samples
+ For refget, prefer `RefgetStore.in_memory()` or `RefgetStore.open_local(path)`.
+ `open_remote(cache_path, remote_url)` contacts a remote service, creates/uses a
+ local cache, and performs on-demand range reads. See `references/refget.md`.
- **Quick example:**
- ```bash
- # Score fragments against reference
- gtars scoring score --fragments fragments.bed --reference reference.bed --output scores.txt
- ```
+ ## Network and cache gate
- ## Common Workflows
+ No download or cache write is implicit in this skill. Before any network-capable
+ upstream call:
- ### Workflow 1: Peak Overlap Analysis
+ - obtain explicit user approval for the exact host, endpoint, data, and cache;
+ - allowlist HTTPS hosts and reject unreviewed redirects;
+ - record immutable revision/identifier, retrieval time, expected SHA-256 and
+ domain digest, assembly accession, size quota, and provenance;
+ - disclose sensitive BED coordinates, barcodes, sample labels, and reference
+ choices that could leave the approved environment;
+ - validate downloaded content as untrusted before using it.
- Identify overlapping genomic features:
+ Important side effects:
- ```python
- import gtars
+ - `RegionSet(path)` has HTTP support; a nonexistent local string may be treated as
+ a URL. Check that the local path exists before construction.
+ - `Tokenizer.from_pretrained` may download `universe.bed.gz` into the Hugging Face
+ cache.
+ - `RefgetStore.on_disk` creates/writes a store. `open_remote` loads remote metadata
+ and enables persistence by default.
+ - `gtars bbcache` creates cache directories even when constructing the client.
+ Cache/download commands use `BBCLIENT_CACHE` (default `~/.bbcache`) and
+ `BEDBASE_API` (default `https://api.bedbase.org`).
- # Load two region sets
- peaks = gtars.RegionSet.from_bed("chip_peaks.bed")
- promoters = gtars.RegionSet.from_bed("promoters.bed")
+ ## Sensitive metadata and leakage
- # Find overlaps
- overlapping_peaks = peaks.filter_overlapping(promoters)
+ Genomic intervals, rare loci, barcodes, sample names, phenotypes, and assembly
+ choices can be identifying. Keep full paths and raw coordinates out of logs;
+ default bundled reports redact paths and emit only counts/checksums.
- # Export results
- overlapping_peaks.to_bed("peaks_in_promoters.bed")
- ```
+ Freeze splits by patient/donor first, then keep all technical and biological
+ replicates in the same split. Fit consensus sets, universes, tokenizers, scaling,
+ thresholds, and QC rules on training data only. Do not create a universe from all
+ samples and then split: that leaks validation/test locus support. Record excluded
+ samples and replicate aggregation separately.
- ### Workflow 2: Coverage Track Pipeline
+ ## Bundled deterministic CLIs
- Generate coverage tracks for visualization:
+ All six helpers reject URLs, traversal, symlinks, and special files; apply byte,
+ record, file, coordinate, and worker caps; use no network or gtars import; and
+ write no output files. Plans contain fixed argv templates and never launch them.
```bash
- # Step 1: Generate coverage
- gtars uniwig generate --input atac_fragments.bed --output coverage.wig --resolution 10
-
- # Step 2: Convert to BigWig for genome browsers
- gtars uniwig generate --input atac_fragments.bed --output coverage.bw --format bigwig
+ python3 -B scripts/bed_validator.py --help
+ python3 -B scripts/execution_plan.py --help
+ python3 -B scripts/tokenizer_manifest.py --help
+ python3 -B scripts/refget_digest_plan.py --help
+ python3 -B scripts/coverage_preflight.py --help
+ python3 -B scripts/artifact_inspector.py --help
```
- ### Workflow 3: ML Preprocessing
-
- Prepare genomic data for machine learning:
-
- ```python
- from gtars.tokenizers import TreeTokenizer
- import gtars
-
- # Step 1: Load training regions
- regions = gtars.RegionSet.from_bed("training_peaks.bed")
-
- # Step 2: Create tokenizer
- tokenizer = TreeTokenizer.from_bed_file("training_peaks.bed")
-
- # Step 3: Tokenize regions
- tokens = [tokenizer.tokenize(r.chromosome, r.start, r.end) for r in regions]
+ Run synthetic tests without bytecode:
- # Step 4: Use tokens in ML pipeline
- # (integrate with geniml or custom models)
+ ```bash
+ PYTHONDONTWRITEBYTECODE=1 python3 -B -m unittest discover \
+ -s skills/gtars/tests -p 'test_*.py' -v
```
- ## Python vs CLI Usage
-
- **Use Python API when:**
- - Integrating with analysis pipelines
- - Need programmatic control
- - Working with NumPy/Pandas
- - Building custom workflows
-
- **Use CLI when:**
- - Quick one-off analyses
- - Shell scripting
- - Batch processing files
- - Prototyping workflows
-
- ## Reference Documentation
-
- Comprehensive module documentation:
-
- - **`references/python-api.md`** - Complete Python API reference with RegionSet operations, NumPy integration, and data export
- - **`references/overlap.md`** - IGD indexing, overlap detection, and set operations
- - **`references/coverage.md`** - Coverage track generation with uniwig
- - **`references/tokenizers.md`** - Genomic tokenization for ML applications
- - **`references/refget.md`** - Reference sequence management and digests
- - **`references/cli.md`** - Command-line interface complete reference
-
- ## Integration with geniml
-
- Gtars serves as the foundation for the geniml Python package, providing core genomic interval operations for machine learning workflows. When working on geniml-related tasks, use gtars for data preprocessing and tokenization.
-
- ## Performance Characteristics
-
- - **Native Rust performance**: Fast execution with low memory overhead
- - **Parallel processing**: Multi-threaded operations for large datasets
- - **Memory efficiency**: Streaming and memory-mapped file support
- - **Zero-copy operations**: NumPy integration with minimal data copying
-
- ## Data Formats
-
- Gtars works with standard genomic formats:
-
- - **BED**: Genomic intervals (3-column or extended)
- - **WIG/BigWig**: Coverage tracks
- - **FASTA**: Reference sequences
- - **Fragment TSV**: Single-cell fragment files with barcodes
-
- ## Error Handling and Debugging
+ ## Migration traps removed in 1.1
- Enable verbose logging for troubleshooting:
+ Do not use stale examples containing `gtars.RegionSet`,
+ `RegionSet.from_bed`, `TreeTokenizer`, `gtars.igd.build_index`,
+ `gtars.uniwig.coverage_from_bed`, `gtars.RefgetStore`, global
+ `set_option`/`set_log_level`, `parallel_apply`, or invented exception classes.
+ CLI forms such as `uniwig generate`, `igd build`, `scoring score`, and
+ `fragsplit cluster-split` are also stale for 0.9.0.
- ```python
- import gtars
+ Upstream's published docs and stubs have some drift (for example the older
+ `GlobalRefgetStore` tutorial and incomplete 0.9.2 stubs). Prefer installed
+ signature smoke tests plus immutable tagged source when they conflict.
- # Enable debug logging
- gtars.set_log_level("DEBUG")
- ```
+ ## Bundled references
- ```bash
- # CLI verbose mode
- gtars --verbose <command>
- ```
+ These are the only six bundled references; all links are local and present:
+ - `references/python-api.md` — exact Python 0.9.2 imports and behavior
+ - `references/overlap.md` — overlap/count/set algebra and consensus semantics
+ - `references/coverage.md` — uniwig, bigWig, coverage, sorting, and resources
+ - `references/tokenizers.md` — tokenizer/universe and fragment compatibility
+ - `references/refget.md` — digests, stores, BEDbase, network/cache controls
+ - `references/cli.md` — CLI 0.9.0 commands, features, and migrations