geniml · diff
git:20260611.1b8fae3 to v1.1
236 added, 236 removed. Audit A to A.
---
name: geniml
- description: This skill should be used when working with genomic interval data (BED files) for machine learning tasks. Use for training region embeddings (Region2Vec, BEDspace), single-cell ATAC-seq analysis (scEmbed), building consensus peaks (universes), or any ML-based analysis of genomic regions. Applies to BED file collections, scATAC-seq data, chromatin accessibility datasets, and region-based genomic feature learning.
- license: BSD-2-Clause license
- metadata: {"version": "1.0", "skill-author": "K-Dense Inc."}
+ description: "Use Geniml for audited local genomic-interval workflows: validate BED and universe contracts, plan Region2Vec or scEmbed runs, inspect model/tokenizer compatibility, and assess consensus universes."
+ license: MIT
+ compatibility: Requires Python 3.10+ and uv. Guidance targets geniml 0.8.4 with gtars 0.9.2; ML workflows need the pinned ml extra and compatible native wheels. Bundled planners and inspectors are dependency-free, local-only, and make no network requests.
+ allowed-tools:
+ - Read
+ - Write
+ - Edit
+ - Bash
+ - Glob
+ metadata:
+ version: "1.1"
+ skill-author: "K-Dense Inc."
+ upstream-version: "0.8.4"
+ last-reviewed: "2026-07-23"
---
- # Geniml: Genomic Interval Machine Learning
+ # Geniml
- ## Overview
+ Use Geniml for machine learning and statistical workflows over genomic interval
+ sets. Treat coordinates, assemblies, token vocabularies, model artifacts, and
+ sample grouping as explicit contracts. The bundled scripts validate or plan;
+ they do not import Geniml, contact services, deserialize models, or execute
+ training.
- Geniml is a Python package for building machine learning models on genomic interval data from BED files. It provides unsupervised methods for learning embeddings of genomic regions, single cells, and metadata labels, enabling similarity searches, clustering, and downstream ML tasks.
+ `Bash` is declared only for explicit, user-approved `uv`, Python, Geniml,
+ Gtars, Git, and native CLI commands shown in this guide; bundled Python helpers
+ do not spawn subprocesses. Example paths under `data/`, `refs/`, `work/`, and
+ `models/` are user-provided project placeholders, not missing bundled files.
- ## Installation
+ ## Verified release snapshot
- Install geniml using uv:
+ - Latest stable PyPI release on 2026-07-23: `geniml==0.8.4` (2026-01-14).
+ - PyPI does not declare `Requires-Python`; its classifiers list Python
+ 3.10-3.14. Prefer Python 3.11 or 3.12 where all native/ML wheels resolve.
+ - `geniml==0.8.4` accepts `gtars>=0.2.5`; the verified base smoke used current
+ `gtars==0.9.2` (2026-06-17, Python >=3.10).
+ - Extras are `ml` and `test`. The base install omits Torch, Gensim, Scanpy,
+ Hugging Face Hub, pyBigWig, and HMM dependencies.
+ - Upstream documentation contains stale examples. Release source and installed
+ `--help` output take precedence where they conflict.
- ```bash
- uv pip install geniml
- ```
+ ## Install reproducibly
- For ML dependencies (PyTorch, etc.):
+ Use a project environment and commit its generated lockfile:
```bash
- uv pip install 'geniml[ml]'
+ uv venv --python 3.12
+ uv pip install "geniml==0.8.4" "gtars==0.9.2"
```
- Development version from GitHub:
+ For Region2Vec, scEmbed, evaluation, or universe methods needing ML libraries:
```bash
- uv pip install git+https://github.com/databio/geniml.git
+ uv pip install "geniml[ml]==0.8.4" "gtars==0.9.2"
```
- ## Core Capabilities
-
- Geniml provides five primary capabilities, each detailed in dedicated reference files:
-
- ### 1. Region2Vec: Genomic Region Embeddings
-
- Train unsupervised embeddings of genomic regions using word2vec-style learning.
-
- **Use for:** Dimensionality reduction of BED files, region similarity analysis, feature vectors for downstream ML.
-
- **Workflow:**
- 1. Tokenize BED files using a universe reference
- 2. Train Region2Vec model on tokens
- 3. Generate embeddings for regions
-
- **Reference:** See `references/region2vec.md` for detailed workflow, parameters, and examples.
-
- ### 2. BEDspace: Joint Region and Metadata Embeddings
-
- Train shared embeddings for region sets and metadata labels using StarSpace.
-
- **Use for:** Metadata-aware searches, cross-modal queries (region→label or label→region), joint analysis of genomic content and experimental conditions.
-
- **Workflow:**
- 1. Preprocess regions and metadata
- 2. Train BEDspace model
- 3. Compute distances
- 4. Query across regions and labels
-
- **Reference:** See `references/bedspace.md` for detailed workflow, search types, and examples.
-
- ### 3. scEmbed: Single-Cell Chromatin Accessibility Embeddings
+ For a durable project, prefer:
- Train Region2Vec models on single-cell ATAC-seq data for cell-level embeddings.
+ ```bash
+ uv add "geniml[ml]==0.8.4" "gtars==0.9.2"
+ uv lock
+ ```
- **Use for:** scATAC-seq clustering, cell-type annotation, dimensionality reduction of single cells, integration with scanpy workflows.
+ Do not install an unpinned Git branch. Record Python, OS/architecture, the
+ resolved lockfile, and the PyPI artifact digest. Geniml itself is BSD-2-Clause;
+ the `MIT` frontmatter value licenses this skill's content.
- **Workflow:**
- 1. Prepare AnnData with peak coordinates
- 2. Pre-tokenize cells
- 3. Train scEmbed model
- 4. Generate cell embeddings
- 5. Cluster and visualize with scanpy
+ ## Start with the safety gate
- **Reference:** See `references/scembed.md` for detailed workflow, parameters, and examples.
+ Before importing Geniml or running an external binary:
- ### 4. Consensus Peaks: Universe Building
+ 1. Work only with explicit local regular files. Reject URLs, FIFOs, devices,
+ and symlinks unless the user deliberately changes that policy.
+ 2. Validate BED structure and the declared assembly against a trusted local
+ chromosome-sizes file.
+ 3. Bound file count, bytes, rows, workers, epochs, and output size.
+ 4. Separate train/validation/test by patient, donor, biological replicate, or
+ other independent unit—not by BED row or cell alone.
+ 5. Inventory and checksum the universe, tokenizer, model, config, inputs,
+ metadata manifest, and native binaries.
+ 6. Obtain explicit approval before any BEDbase or Hugging Face download. Never
+ infer approval from a model ID or BEDbase identifier.
+ 7. Keep logs aggregate and bounded. BED filenames, sample IDs, phenotypes,
+ labels, barcodes, and genomic intervals may be sensitive.
- Build reference peak sets (universes) from BED file collections using multiple statistical methods.
+ ## Coordinate and assembly contract
- **Use for:** Creating tokenization references, standardizing regions across datasets, defining consensus features with statistical rigor.
+ BED intervals are normally **0-based, half-open** `[start, end)`: start is
+ included, end is excluded, and length is `end - start`. Do not mix them with
+ 1-based closed coordinates from VCF/GFF or user-facing genome browsers.
- **Workflow:**
- 1. Combine BED files
- 2. Generate coverage tracks
- 3. Build universe using CC, CCF, ML, or HMM method
+ For every corpus and artifact, record:
- **Methods:**
- - **CC (Coverage Cutoff)**: Simple threshold-based
- - **CCF (Coverage Cutoff Flexible)**: Confidence intervals for boundaries
- - **ML (Maximum Likelihood)**: Probabilistic modeling of positions
- - **HMM (Hidden Markov Model)**: Complex state modeling
+ - assembly and patch/accession where possible (for example GRCh38 versus
+ GRCh38.p14), plus the chromosome-sizes checksum;
+ - contig naming convention (`chr1` versus `1`), alt/random/decoy policy, and
+ mitochondrial naming;
+ - coordinate convention, sorting order, duplicate/overlap policy, and whether
+ BED strand is meaningful;
+ - liftover tool, chain digest, source/target assemblies, unmapped fraction, and
+ post-liftover validation.
- **Reference:** See `references/consensus_peaks.md` for method comparison, parameters, and examples.
+ Reject negative coordinates, `end <= start`, integer overflow, unknown
+ contigs, ends beyond contig length, malformed columns, mixed assemblies, and
+ silent contig renaming. Sorting and normalization never repair an assembly
+ mismatch. BED3 has no strand; when column 6 is present, preserve `+`, `-`, or
+ `.` unless the assay contract says otherwise.
- ### 5. Utilities: Supporting Tools
+ Run a bounded validation and normalization **plan** before analysis:
- Additional tools for caching, randomization, evaluation, and search.
+ ```bash
+ python skills/geniml/scripts/bed_validator.py \
+ --input data/peaks.bed \
+ --assembly GRCh38 \
+ --chrom-sizes refs/GRCh38.chrom.sizes
+ ```
- **Available utilities:**
- - **BBClient**: BED file caching for repeated access
- - **BEDshift**: Randomization preserving genomic context
- - **Evaluation**: Metrics for embedding quality (silhouette, Davies-Bouldin, etc.)
- - **Tokenization**: Region tokenization utilities (hard, soft, universe-based)
- - **Text2BedNN**: Neural search backends for genomic queries
+ The validator reports proposed actions but never rewrites the BED file.
- **Reference:** See `references/utilities.md` for detailed usage of each utility.
+ ## Current API map
- ## Common Workflows
+ ### Region and tokenizer I/O
- ### Basic Region Embedding Pipeline
+ Prefer Gtars for new interval/tokenizer code:
```python
- from geniml.tokenization import hard_tokenization
- from geniml.region2vec import region2vec
- from geniml.evaluation import evaluate_embeddings
-
- # Step 1: Tokenize BED files
- hard_tokenization(
- src_folder='bed_files/',
- dst_folder='tokens/',
- universe_file='universe.bed',
- p_value_threshold=1e-9
- )
-
- # Step 2: Train Region2Vec
- region2vec(
- token_folder='tokens/',
- save_dir='model/',
- num_shufflings=1000,
- embedding_dim=100
- )
+ from gtars.models import Region, RegionSet
+ from gtars.tokenizers import Tokenizer
- # Step 3: Evaluate
- metrics = evaluate_embeddings(
- embeddings_file='model/embeddings.npy',
- labels_file='metadata.csv'
- )
+ regions = RegionSet("data/peaks.bed")
+ tokenizer = Tokenizer.from_bed("refs/universe.bed")
+ encoded = tokenizer(regions)
+ input_ids = encoded["input_ids"]
```
- ### scATAC-seq Analysis Pipeline
-
- ```python
- import scanpy as sc
- from geniml.scembed import ScEmbed
- from geniml.io import tokenize_cells
-
- # Step 1: Load data
- adata = sc.read_h5ad('scatac_data.h5ad')
-
- # Step 2: Tokenize cells
- tokenize_cells(
- adata='scatac_data.h5ad',
- universe_file='universe.bed',
- output='tokens.parquet'
- )
-
- # Step 3: Train scEmbed
- model = ScEmbed(embedding_dim=100)
- model.train(dataset='tokens.parquet', epochs=100)
-
- # Step 4: Generate embeddings
- embeddings = model.encode(adata)
- adata.obsm['scembed_X'] = embeddings
+ `RegionSet` and `Tokenizer` also accept remote inputs in some constructors;
+ this skill permits local paths only unless network access is explicitly
+ approved. `geniml.io.RegionSet(regions, backed=False)` remains available as a
+ legacy Python implementation; backed sets are iterable but not indexable.
+ `geniml.io.Region` uses `stop`, while `gtars.models.Region` uses `end`.
- # Step 5: Cluster with scanpy
- sc.pp.neighbors(adata, use_rep='scembed_X')
- sc.tl.leiden(adata)
- sc.tl.umap(adata)
- ```
+ With gtars 0.9.2, seven special tokens are added to a BED vocabulary. Therefore
+ `len(tokenizer)` is not simply the number of universe rows. Preserve universe
+ row order and the exact special-token map.
- ### Universe Building and Evaluation
+ ### Region2Vec
- ```bash
- # Generate coverage
- cat bed_files/*.bed > combined.bed
- uniwig -m 25 combined.bed chrom.sizes coverage/
+ The modern class lives at a concrete module path:
- # Build universe with coverage cutoff
- geniml universe build cc \
- --coverage-folder coverage/ \
- --output-file universe.bed \
- --cutoff 5 \
- --merge 100 \
- --filter-size 50
+ ```python
+ from geniml.region2vec.main import Region2VecExModel
+ from geniml.region2vec.utils import Region2VecDataset
+ from gtars.tokenizers import Tokenizer
- # Evaluate universe quality
- geniml universe evaluate \
- --universe universe.bed \
- --coverage-folder coverage/ \
- --bed-folder bed_files/
+ tokenizer = Tokenizer.from_bed("refs/universe.bed")
+ dataset = Region2VecDataset("work/tokens.parquet", shuffle=True)
+ model = Region2VecExModel(tokenizer=tokenizer, embedding_dim=100)
+ model.train(dataset, epochs=10, window_size=5, num_cpus=4, seed=42)
```
- ## CLI Reference
+ The Parquet input must contain one list-valued `tokens` column, one document
+ per row. See [references/region2vec.md](references/region2vec.md) for export,
+ encoding, legacy CLI, and evaluation details.
- Geniml provides command-line interfaces for major operations:
+ ### scEmbed
- ```bash
- # Region2Vec training
- geniml region2vec --token-folder tokens/ --save-dir model/ --num-shuffle 1000
+ Import `ScEmbed` from `geniml.scembed.main`. AnnData `.var` must contain
+ `chr`, `start`, and `end`; rows are cells and nonzero features identify
+ accessible regions. Pre-tokenize to a Parquet `tokens` column and use the same
+ Tokenizer for training and inference. See
+ [references/scembed.md](references/scembed.md).
- # BEDspace preprocessing
- geniml bedspace preprocess --input regions/ --metadata labels.csv --universe universe.bed
+ ### BEDspace
- # BEDspace training
- geniml bedspace train --input preprocessed.txt --output model/ --dim 100
+ BEDspace remains in 0.8.4 and invokes an external StarSpace executable.
+ StarSpace is archived and upstream Geniml does not pin a compatible revision.
+ Treat BEDspace as a legacy reproduction path, not the default for new systems.
+ See [references/bedspace.md](references/bedspace.md) for the exact stable CLI
+ spelling and an immutable, explicitly unverified build baseline.
- # BEDspace search
- geniml bedspace search -t r2l -d distances.pkl -q query.bed -n 10
+ ### Consensus universes and assessment
- # Universe building
- geniml universe build cc --coverage-folder coverage/ --output universe.bed --cutoff 5
+ The installed 0.8.4 CLI uses:
- # BEDshift randomization
- geniml bedshift --input peaks.bed --genome hg38 --preserve-chrom --iterations 100
+ ```text
+ geniml build-universe {cc,ccf,ml,hmm} ...
+ geniml assess-universe ...
+ geniml eval {gdst,npt,ctt,rct,bin-gen} ...
```
- ## When to Use Which Tool
+ CC/CCF/ML/HMM consume precomputed coverage bigWigs. Do not concatenate or
+ generate coverage until all BED files pass the same assembly contract.
+ Assessment and embedding metrics are distinct: `assess-universe` measures fit
+ of a universe to interval collections, while `eval` implements CTT, RCT, GDST,
+ and NPT for embeddings. See
+ [references/consensus_peaks.md](references/consensus_peaks.md) and
+ [references/utilities.md](references/utilities.md).
- **Use Region2Vec when:**
- - Working with bulk genomic data (ChIP-seq, ATAC-seq, etc.)
- - Need unsupervised embeddings without metadata
- - Comparing region sets across experiments
- - Building features for downstream supervised learning
+ ## Important 0.8.4 migration notes
- **Use BEDspace when:**
- - Metadata labels available (cell types, tissues, conditions)
- - Need to query regions by metadata or vice versa
- - Want joint embedding space for regions and labels
- - Building searchable genomic databases
+ - The 0.7.0 changelog moved new RegionSet/tokenizer work toward Gtars.
+ - The 0.4.0 names `TreeTokenizer` and `AnnDataTokenizer` are historical; the
+ current Gtars API exposes `Tokenizer`.
+ - In the 0.8.4 wheel, `geniml.region2vec` and `geniml.scembed` do not re-export
+ their modern classes/functions. Use the concrete module paths above.
+ - `geniml tokenize` and `geniml region2vec` call names no longer exported by
+ their package `__init__` files; do not build new workflows around those CLI
+ paths without an installed-version smoke test.
+ - `geniml scembed` parses legacy MatrixMarket options but its command body is a
+ no-op in 0.8.4. Use `geniml.scembed.main.ScEmbed`.
+ - Official pages still show `geniml assess`; the release command is
+ `geniml assess-universe`.
+ - `.gtok` remains present in legacy datasets, but upstream issue #14 proposes
+ deprecating many-file `.gtok` workflows. Prefer one bounded Parquet corpus.
+ - Config key `embedding_size` is accepted only for backward compatibility;
+ use `embedding_dim`.
- **Use scEmbed when:**
- - Analyzing single-cell ATAC-seq data
- - Clustering cells by chromatin accessibility
- - Annotating cell types from scATAC-seq
- - Integration with scanpy is desired
+ ## Model and universe compatibility
- **Use Universe Building when:**
- - Need reference peak sets for tokenization
- - Combining multiple experiments into consensus
- - Want statistically rigorous region definitions
- - Building standard references for a project
+ A Region2Vec/scEmbed inference bundle is valid only when these agree:
- **Use Utilities when:**
- - Need to cache remote BED files (BBClient)
- - Generating null models for statistics (BEDshift)
- - Evaluating embedding quality (Evaluation)
- - Building search interfaces (Text2BedNN)
+ - model `config.yaml` `vocab_size` and `embedding_dim`;
+ - exact `universe.bed` bytes/order and assembly;
+ - tokenizer implementation/version and special-token IDs;
+ - checkpoint tensor shapes and pooling policy;
+ - Geniml/Gtars versions and any tokenization parameters.
- ## Best Practices
+ Geniml 0.8.4 defaults to `checkpoint.pt`, `config.yaml`, and `universe.bed`.
+ Its loader uses `torch.load(..., weights_only=True)`, but `.pt`, Gensim
+ `.model`, pickle, joblib, and native binaries remain untrusted inputs. Inspect
+ and checksum artifacts before loading; use an isolated environment and never
+ load a checkpoint merely to discover its metadata.
- ### General Guidelines
+ ```bash
+ python skills/geniml/scripts/model_artifact_inspector.py \
+ --model-dir models/region2vec
- - **Universe quality is critical**: Invest time in building comprehensive, well-constructed universes
- - **Tokenization validation**: Check coverage (>80% ideal) before training
- - **Parameter tuning**: Experiment with embedding dimensions, learning rates, and training epochs
- - **Evaluation**: Always validate embeddings with multiple metrics and visualizations
- - **Documentation**: Record parameters and random seeds for reproducibility
+ python skills/geniml/scripts/tokenizer_compatibility.py \
+ --model-dir models/region2vec \
+ --universe refs/universe.bed \
+ --assembly GRCh38
+ ```
- ### Performance Considerations
+ `Region2VecExModel(model_path="org/repo")`, `ScEmbed(model_path="org/repo")`,
+ and Gtars `Tokenizer.from_pretrained(...)` can download from Hugging Face.
+ Local `from_pretrained("models/local")` loads a local bundle. Pin Hub revision
+ and expected hashes when a user approves download; then work offline from the
+ verified cache.
- - **Pre-tokenization**: For scEmbed, always pre-tokenize cells for faster training
- - **Memory management**: Large datasets may require batch processing or downsampling
- - **Computational resources**: ML/HMM universe methods are computationally intensive
- - **Model caching**: Use BBClient to avoid repeated downloads
+ ## BEDbase downloads and caches
- ### Integration Patterns
+ `BBClient.load_bed`, `load_bedset`, and token-cache operations may contact
+ `https://api.bedbase.org`. The default cache is
+ `$BBCLIENT_CACHE` or `~/.bbcache`; `BEDBASE_API` changes the endpoint. Do not
+ read unrelated environment variables. Set an explicit project cache, estimate
+ size, approve identifiers/endpoints, and verify returned checksums before use.
- - **With scanpy**: scEmbed embeddings integrate seamlessly as `adata.obsm` entries
- - **With BEDbase**: Use BBClient for accessing remote BED repositories
- - **With Hugging Face**: Export trained models for sharing and reproducibility
- - **With R**: Use reticulate for R integration (see utilities reference)
+ Local inspection commands are safer:
- ## Related Projects
+ ```text
+ geniml bbclient seek ID --cache-folder /absolute/project/cache
+ geniml bbclient inspect-bedfiles --cache-folder /absolute/project/cache
+ geniml bbclient inspect-bedsets --cache-folder /absolute/project/cache
+ ```
- Geniml is part of the BEDbase ecosystem:
+ The `cache-bed`, `cache-bedset`, and `cache-tokens` subcommands may use the
+ network. Do not run them implicitly or include sensitive local BED files in an
+ upload/cache workflow.
- - **BEDbase**: Unified platform for genomic regions
- - **BEDboss**: Processing pipeline for BED files
- - **Gtars**: Genomic tools and utilities
- - **BBClient**: Client for BEDbase repositories
+ ## Local audit and planning CLIs
- ## Additional Resources
+ All scripts are standard-library-only and default to redacted JSON:
- - **Documentation**: https://docs.bedbase.org/geniml/
- - **GitHub**: https://github.com/databio/geniml
- - **Pre-trained models**: Available on Hugging Face (databio organization)
- - **Publications**: Cited in documentation for methodological details
+ ```bash
+ # Audit manifest paths, checksums, assemblies, and patient/donor leakage
+ python skills/geniml/scripts/corpus_auditor.py \
+ --manifest data/manifest.tsv --assembly-column assembly \
+ --group-column patient_id --split-column split
- ## Troubleshooting
+ # Plan tokenizer/model compatibility checks
+ python skills/geniml/scripts/tokenizer_compatibility.py \
+ --model-dir models/r2v --universe refs/universe.bed --assembly GRCh38
- **"Tokenization coverage too low":**
- - Check universe quality and completeness
- - Adjust p-value threshold (try 1e-6 instead of 1e-9)
- - Ensure universe matches genome assembly
+ # Plan consensus construction; does not execute Geniml or coverage tools
+ python skills/geniml/scripts/consensus_plan.py \
+ --manifest data/manifest.tsv --chrom-sizes refs/GRCh38.chrom.sizes \
+ --assembly GRCh38 --method cc --output-dir work/consensus
- **"Training not converging":**
- - Adjust learning rate (try 0.01-0.05 range)
- - Increase training epochs
- - Check data quality and preprocessing
+ # Plan an embedding run; does not import ML libraries
+ python skills/geniml/scripts/embedding_plan.py \
+ --mode region2vec --data work/tokens.parquet \
+ --universe refs/universe.bed --output-dir work/r2v \
+ --assembly GRCh38
+ ```
- **"Out of memory errors":**
- - Reduce batch size for scEmbed
- - Process data in chunks
- - Use pre-tokenization for single-cell data
+ Use `--help` for resource limits and explicit path-disclosure controls.
- **"StarSpace not found" (BEDspace):**
- - Install StarSpace separately: https://github.com/facebookresearch/StarSpace
- - Set `--path-to-starspace` parameter correctly
+ ## References
- For detailed troubleshooting and method-specific issues, consult the appropriate reference file.
+ - [Region2Vec](references/region2vec.md): modern API, artifacts, CLI drift,
+ training, encoding, and evaluation.
+ - [scEmbed](references/scembed.md): AnnData/token preparation, training,
+ inference, annotation, privacy, and leakage.
+ - [BEDspace](references/bedspace.md): metadata schema, exact legacy CLI,
+ StarSpace status, artifacts, and retrieval.
+ - [Consensus peaks](references/consensus_peaks.md): coverage prerequisites,
+ CC/CCF/ML/HMM, assessment, and assembly safeguards.
+ - [Utilities](references/utilities.md): I/O, Gtars tokenizers, BBClient,
+ evaluation, model safety, migration, and dated sources.
+ Source snapshot and primary-paper links are dated in
+ [references/utilities.md](references/utilities.md). Re-check release metadata
+ and installed signatures before changing the pinned versions.