tiledbvcf · diff
v1.2 to v1.3
208 added, 381 removed. Audit A to A.
---
name: tiledbvcf
- description: Efficient storage and retrieval of genomic variant data using TileDB. Scalable VCF/BCF ingestion, incremental sample addition, compressed storage, parallel queries, and export capabilities for population genomics.
+ description: Stores and retrieves genomic variant calls with TileDB-VCF. Use for indexed single-sample VCF/BCF ingestion, incremental cohorts, region and sample queries, streaming results, allele statistics, QC, and VCF/BCF export locally or through TileDB Cloud.
license: MIT license
+ compatibility: Requires a native TileDB-VCF installation and Python 3.9-3.12 for the reviewed Conda builds. bcftools compresses/indexes input fixtures. Installation and remote storage need network access; TileDB Cloud needs its Python client, account token, and storage permissions.
metadata:
- version: "1.2"
+ version: "1.3"
skill-author: Jeremy Leipzig
+ last-reviewed: "2026-10-01"
+ upstream-version: "TileDB-VCF 0.40.3; tiledb-cloud 0.14.4"
---
# TileDB-VCF
- ## Overview
+ ## When to use
- TileDB-VCF is a high-performance C++ library with Python and CLI interfaces for efficient storage and retrieval of genomic variant-call data. Built on TileDB's sparse array technology, it enables scalable ingestion of VCF/BCF files, incremental sample addition without expensive merging operations, and efficient parallel queries of variant data stored locally or in the cloud.
+ Use for cohort variant storage, incremental sample ingestion, interval queries, and
+ exporting subsets for downstream genomics. TileDB-VCF stores records; it does not
+ perform joint variant calling, association testing, normalization, or population
+ structure adjustment. Dataset size alone does not determine whether cloud execution
+ is appropriate: benchmark the intended sample, region, and attribute workload.
- ## When to Use This Skill
+ This skill targets released **TileDB-VCF 0.40.3**. Native Python/CLI checks used two
+ synthetic single-sample VCFs on macOS ARM64. Cloud client **0.14.4** contracts were
+ checked against its released wheel and mocked dispatch; hosted queries and jobs
+ were not executed. See [installation and verification notes](references/verification.md).
- This skill should be used when:
- - Learning TileDB-VCF concepts and workflows
- - Prototyping genomics analyses and pipelines
- - Working with small-to-medium datasets (< 1000 samples)
- - Need incremental addition of new samples to existing datasets
- - Require efficient querying of specific genomic regions across many samples
- - Working with cloud-stored variant data (S3, Azure, GCS)
- - Need to export subsets of large VCF datasets
- - Building variant databases for cohort studies
- - Educational projects and method development
- - Performance is critical for variant data operations
+ ## Install and verify the native stack
- ## Quick Start
+ The `tiledbvcf` Python distribution is not published on PyPI at this review. The
+ official **`tiledb` Conda channel** supplies `tiledbvcf-py` 0.40.3 for macOS ARM64,
+ macOS x86-64, and Linux x86-64, with Python 3.9-3.12 builds. `tiledb` on PyPI is
+ TileDB-Py and does not install TileDB-VCF. Native Apple Silicon no longer requires
+ forcing `CONDA_SUBDIR=osx-64`.
- ### Installation
+ For the tested macOS ARM64 environment, pins below avoid versioned-library import
+ failures in the otherwise successful current Conda solve:
- **Preferred Method: Conda/Mamba**
```bash
- # Enter the following two lines if you are on a M1 Mac
- CONDA_SUBDIR=osx-64
- conda config --env --set subdir osx-64
-
- # Create the conda environment
- conda create -n tiledb-vcf "python<3.10"
+ conda create -n tiledb-vcf -c tiledb -c conda-forge \
+ python=3.12 tiledbvcf-py=0.40.3 \
+ azure-core-cpp=1.16.2 azure-storage-blobs-cpp=12.16.0 \
+ azure-storage-files-datalake-cpp=12.14.0 capnproto=1.4.0 c-blosc2=2.23.1
conda activate tiledb-vcf
-
- # Mamba is a faster and more reliable alternative to conda
- conda install -c conda-forge mamba
-
- # Install TileDB-Py and TileDB-VCF, align with other useful libraries
- mamba install -y -c conda-forge -c bioconda -c tiledb tiledb-py tiledbvcf-py pandas pyarrow numpy
+ python -c 'import tiledbvcf; print(tiledbvcf.version)'
+ tiledbvcf version
```
- **Alternative: Docker Images**
- ```bash
- docker pull tiledb/tiledbvcf-py # Python interface
- docker pull tiledb/tiledbvcf-cli # Command-line interface
- ```
+ The equivalent Micromamba solve/install was executed. These extra ABI pins are a
+ verified macOS ARM64 workaround, not a claim about all platforms. Preserve the
+ resolved environment for production. Other-platform installation and official
+ Docker images are alternatives documented upstream, not tested here.
- ### Basic Examples
+ ## Prepare and create a cohort
- **Create and populate a dataset:**
- ```python
- import tiledbvcf
+ 1. Record reference assembly, contig naming/lengths, callers, normalization rules,
+ sample identity, and file checksums. Do not mix assemblies or assume `1` and
+ `chr1` are equivalent. Confirm compatible headers across inputs.
+ 2. Each input must contain **one sample**, be coordinate sorted, and have an index.
+ Use BGZF-compressed VCF or BCF with a matching `.csi`/`.tbi`. Inspect
+ `bcftools query -l sample1.vcf.gz`; filenames are not sample identifiers.
+ 3. Create the dataset explicitly, then ingest. Opening `mode="w"` alone does not
+ create its schema. Materialize frequently read fields at creation time.
+ 4. Validate a small known interval and a round-trip export before scaling.
- # Create a new dataset
- ds = tiledbvcf.Dataset(uri="my_dataset", mode="w",
- cfg=tiledbvcf.ReadConfig(memory_budget=1024))
+ For existing sorted plain-text VCFs (run once per input; output names must be new):
- # Ingest VCF files (must be single-sample with indexes)
- # Requirements:
- # - VCFs must be single-sample (not multi-sample)
- # - Must have indexes: .csi (bcftools) or .tbi (tabix)
- ds.ingest_samples(["sample1.vcf.gz", "sample2.vcf.gz"])
+ ```bash
+ bcftools view -Oz -o sample1.vcf.gz sample1.vcf
+ bcftools index -c sample1.vcf.gz
+ bcftools view -Oz -o sample2.vcf.gz sample2.vcf
+ bcftools index -c sample2.vcf.gz
```
- **Query variant data:**
- ```python
- # Open existing dataset for reading
- ds = tiledbvcf.Dataset(uri="my_dataset", mode="r")
-
- # Query specific regions and samples
- df = ds.read(
- attrs=["sample_name", "pos_start", "pos_end", "alleles", "fmt_GT"],
- regions=["chr1:1000000-2000000", "chr2:500000-1500000"],
- samples=["sample1", "sample2", "sample3"]
- )
- print(df.head())
- ```
+ The following local examples were exercised with sample IDs `S1` and `S2`, contig
+ `chr1`, and coordinates 1-100; substitute the verified IDs and intervals in real data.
- **Export to VCF:**
```python
- import os
-
- # Export two VCF samples
- ds.export(
- regions=["chr21:8220186-8405573"],
- samples=["HG00101", "HG00097"],
- output_format="v",
- output_dir=os.path.expanduser("~"),
- )
- ```
-
- ## Core Capabilities
-
- ### 1. Dataset Creation and Ingestion
-
- Create TileDB-VCF datasets and incrementally ingest variant data from multiple VCF/BCF files. This is appropriate for building population genomics databases and cohort studies.
-
- **Requirements:**
- - **Single-sample VCFs only**: Multi-sample VCFs are not supported
- - **Index files required**: VCF/BCF files must have indexes (.csi or .tbi)
-
- **Common operations:**
- - Create new datasets with optimized array schemas
- - Ingest single or multiple VCF/BCF files in parallel
- - Add new samples incrementally without re-processing existing data
- - Configure memory usage and compression settings
- - Handle various VCF formats and INFO/FORMAT fields
- - Resume interrupted ingestion processes
- - Validate data integrity during ingestion
-
-
- ### 2. Efficient Querying and Filtering
-
- Query variant data with high performance across genomic regions, samples, and variant attributes. This is appropriate for association studies, variant discovery, and population analysis.
-
- **Common operations:**
- - Query specific genomic regions (single or multiple)
- - Filter by sample names or sample groups
- - Extract specific variant attributes (position, alleles, genotypes, quality)
- - Access INFO and FORMAT fields efficiently
- - Combine spatial and attribute-based filtering
- - Stream large query results
- - Perform aggregations across samples or regions
-
-
- ### 3. Data Export and Interoperability
-
- Export data in various formats for downstream analysis or integration with other genomics tools. This is appropriate for sharing datasets, creating analysis subsets, or feeding other pipelines.
-
- **Common operations:**
- - Export to standard VCF/BCF formats
- - Generate TSV files with selected fields
- - Create sample/region-specific subsets
- - Maintain data provenance and metadata
- - Lossless data export preserving all annotations
- - Compressed output formats
- - Streaming exports for large datasets
-
-
- ### 4. Population Genomics Workflows
-
- TileDB-VCF excels at large-scale population genomics analyses requiring efficient access to variant data across many samples and genomic regions.
-
- **Common workflows:**
- - Genome-wide association studies (GWAS) data preparation
- - Rare variant burden testing
- - Population stratification analysis
- - Allele frequency calculations across populations
- - Quality control across large cohorts
- - Variant annotation and filtering
- - Cross-population comparative analysis
-
-
- ## Key Concepts
-
- ### Array Schema and Data Model
+ import tiledbvcf
- **TileDB-VCF Data Model:**
- - Variants stored as sparse arrays with genomic coordinates as dimensions
- - Samples stored as attributes allowing efficient sample-specific queries
- - INFO and FORMAT fields preserved with original data types
- - Automatic compression and chunking for optimal storage
+ uri = "cohort"
+ config = {"sm.compute_concurrency_level": "2", "sm.io_concurrency_level": "2"}
+ with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
+ ds.create_dataset(extra_attrs=["fmt_GT", "fmt_DP"])
+ ds.ingest_samples(
+ ["sample1.vcf.gz"], threads=2, total_memory_budget_mb=512
+ )
- **Schema Configuration:**
- ```python
- # Custom schema with specific tile extents
- config = tiledbvcf.ReadConfig(
- memory_budget=2048, # MB
- region_partition=(0, 3095677412), # Full genome
- sample_partition=(0, 10000) # Up to 10k samples
- )
+ # Incremental ingestion uses the existing dataset; do not call create_dataset again.
+ with tiledbvcf.Dataset(uri, mode="w", tiledb_config=config) as ds:
+ ds.ingest_samples(
+ ["sample2.vcf.gz"], threads=2, total_memory_budget_mb=512
+ )
```
- ### Coordinate Systems and Regions
+ The schema-v4 data array has `contig`, start-coordinate, and sample **dimensions**;
+ headers and optional statistics are separate arrays in the dataset group. Fields
+ not explicitly materialized remain in the INFO/FORMAT payloads. Query names such
+ as `pos_start` need not match raw storage column names.
- **Critical:** TileDB-VCF uses **1-based genomic coordinates** following VCF standard:
- - Positions are 1-based (first base is position 1)
- - Ranges are inclusive on both ends
- - Region "chr1:1000-2000" includes positions 1000-2000 (1001 bases total)
+ Upstream supports parallel thread/process ingestion. Assign distinct sample work
+ and coordinate lifecycle/maintenance operations; do not blindly re-ingest samples.
+ `resume=True` supports interrupted ingestion, not arbitrary duplicate correction.
- **Region specification formats:**
- ```python
- # Single region
- regions = ["chr1:1000000-2000000"]
+ ## Query completely and interpret coordinates correctly
- # Multiple regions
- regions = ["chr1:1000000-2000000", "chr2:500000-1500000"]
+ | Surface | Coordinate convention |
+ | --- | --- |
+ | Python `regions=["chr1:10-14"]` | 1-based, closed interval |
+ | Returned `pos_start`, `pos_end` | 1-based, inclusive record endpoints |
+ | BED input and `query_bed_start`, `query_bed_end` | 0-based, half-open |
+ | `read_variant_stats()` / `read_allele_count()` column `pos` in 0.40.3 | **0-based**; add one before joining to VCF POS |
- # Whole chromosome
- regions = ["chr1"]
+ Queries return **overlapping records**, not just records starting inside the
+ interval. A deletion spanning positions 10-14 appears in `chr1:14-14`, with
+ `pos_start=10`. For BED `[13,14)`, the equivalent string is `chr1:14-14`.
+ The Python region parser requires explicit `contig:start-end`; bare `"chr1"`
+ is rejected in this release. Use a known contig length for a whole-contig query.
- # Convert BED [999999, 2000000) to a 1-based inclusive region explicitly:
- regions = ["chr1:1000000-2000000"] # start + 1; end unchanged
+ ```python
+ cfg = tiledbvcf.ReadConfig(memory_budget_mb=128, tiledb_config=config)
+ attrs = ["sample_name", "contig", "pos_start", "pos_end", "alleles", "fmt_GT"]
+ with tiledbvcf.Dataset(uri, cfg=cfg) as ds:
+ assert {"S1", "S2"}.issubset(ds.samples())
+ assert set(attrs).issubset(ds.attributes())
+ for batch in ds.read_iter(
+ attrs=attrs, regions=["chr1:1-100"], samples=["S1", "S2"]
+ ):
+ print(batch) # Replace with a bounded consumer or partitioned output.
```
- ### Memory Management
-
- **Performance considerations:**
- 1. **Set appropriate memory budget** based on available system memory
- 2. **Use streaming queries** for very large result sets
- 3. **Partition large ingestions** to avoid memory exhaustion
- 4. **Configure tile cache** for repeated region access
- 5. **Use parallel ingestion** for multiple files
- 6. **Optimize region queries** by combining nearby regions
+ - `read()` returns a pandas DataFrame; `read_arrow()` returns a PyArrow Table.
+ Either can return only the first memory-limited batch. Continue with
+ `continue_read()` / `continue_read_arrow()` until `read_completed()`, or use
+ `read_iter()` for ordinary DataFrame queries. Avoid concatenating all batches
+ when the result cannot fit in memory.
+ - `ReadConfig(memory_budget_mb=...)` controls reads; use
+ `ingest_samples(total_memory_budget_mb=...)` for writes. A read budget is not
+ a process RSS cap. `limit` truncates total output; it is not a pagination size.
+ - `sample_partition=(0, 2)` selects partition zero of two. Likewise,
+ `region_partition=(index, number_of_partitions)` partitions query regions;
+ these tuples are not coordinate bounds, tile extents, or sample capacities.
+ - **0.40.3 retains BED selection state** when a later call omits `bed_file`.
+ Open a fresh `Dataset` for an independent query when changing BED/region inputs.
+ - `fmt_GT` contains allele indices: 0 refers to REF, positive values index ALT,
+ and -1 means missing. Preserve multiallelic identity and ploidy; do not treat
+ every positive integer as a biallelic dosage or missing calls as reference.
+ The integer list alone does not encode the original phased GT string.
- ### Cloud Storage Integration
+ ## Export and validate
- TileDB-VCF seamlessly works with cloud storage:
```python
- # S3 dataset
- ds = tiledbvcf.Dataset(uri="s3://bucket/dataset", mode="r")
-
- # Azure Blob Storage
- ds = tiledbvcf.Dataset(uri="azure://container/dataset", mode="r")
+ from pathlib import Path
- # Google Cloud Storage
- ds = tiledbvcf.Dataset(uri="gcs://bucket/dataset", mode="r")
+ Path("exported").mkdir(exist_ok=True)
+ with tiledbvcf.Dataset(uri, cfg=cfg) as ds:
+ ds.export(
+ samples=["S1"], regions=["chr1:1-100"],
+ output_format="v", output_dir="exported",
+ )
```
- ## Common Pitfalls
-
- 1. **Memory exhaustion during ingestion:** Use appropriate memory budget and batch processing for large VCF files
- 2. **Inefficient region queries:** Combine nearby regions instead of many separate queries
- 3. **Missing sample names:** Ensure sample names in VCF headers match query sample specifications
- 4. **Coordinate system confusion:** Remember TileDB-VCF uses 1-based coordinates like VCF standard
- 5. **Large result sets:** Use streaming or pagination for queries returning millions of variants
- 6. **Cloud permissions:** Ensure proper authentication for cloud storage access
- 7. **Concurrent access:** Multiple writers to the same dataset can cause corruption—use appropriate locking
-
- ## CLI Usage
-
- TileDB-VCF provides a command-line interface with the following subcommands:
+ Python export formats: `v` VCF, `z` compressed VCF, `u` uncompressed BCF, `b`
+ compressed BCF. `merge=False` writes per-sample files; `merge=True` requires a
+ combined `output_path`. A combined export is not joint calling. Re-index exported
+ files before indexed downstream access. Compare sample IDs, contigs, record
+ counts, alleles, INFO/FORMAT values, and missing/phased genotypes; semantic
+ round-trip preservation does not imply byte-identical compression or headers.
- **Available Subcommands:**
- - `create` - Creates an empty TileDB-VCF dataset
- - `store` - Ingests samples into a TileDB-VCF dataset
- - `export` - Exports data from a TileDB-VCF dataset
- - `list` - Lists all sample names present in a TileDB-VCF dataset
- - `stat` - Prints high-level statistics about a TileDB-VCF dataset
- - `utils` - Utils for working with a TileDB-VCF dataset
- - `version` - Print the version information and exit
+ The CLI accepts positional sample paths or `--samples-file` (one URI per line),
+ not a comma-separated `--samples` argument:
```bash
- # Create empty dataset
- tiledbvcf create --uri my_dataset
-
- # Ingest samples (requires single-sample VCFs with indexes)
- tiledbvcf store --uri my_dataset --samples sample1.vcf.gz,sample2.vcf.gz
-
- # Export data
- tiledbvcf export --uri my_dataset \
- --regions "chr1:1000000-2000000" \
- --sample-names "sample1,sample2"
-
- # List all samples
- tiledbvcf list --uri my_dataset
-
- # Show dataset statistics
- tiledbvcf stat --uri my_dataset
+ tiledbvcf create --uri cli_cohort
+ tiledbvcf store --uri cli_cohort --threads 2 --total-memory-budget-mb 512 \
+ -- sample1.vcf.gz sample2.vcf.gz
+ tiledbvcf list --uri cli_cohort
+ tiledbvcf stat --uri cli_cohort
+ tiledbvcf export --uri cli_cohort --regions chr1:1-100 \
+ --sample-names S1,S2 -Ot --tsv-fields 'SAMPLE,CHR,POS,REF,ALT,F:GT' \
+ --output-path variants.tsv
```
- ## Advanced Features
+ `--` protects positional paths from variable-length options such as
+ `--tiledb-config`. Check expected artifacts and sample counts as well as the exit
+ code: an invalid store command returned exit zero while printing an error in the
+ reviewed CLI. TSV fields use `F:GT` for FORMAT/GT and `I:DP` for INFO/DP.
- ### Allele Frequency Analysis
- ```python
- # Calculate allele frequencies
- af_df = tiledbvcf.read_allele_frequency(
- uri="my_dataset",
- regions=["chr1:1000000-2000000"],
- samples=["sample1", "sample2", "sample3"]
- )
- ```
+ ## Cohort statistics and QC
- ### Sample Quality Control
```python
- # Perform sample QC
- qc_results = tiledbvcf.sample_qc(
- uri="my_dataset",
- samples=["sample1", "sample2"]
- )
- ```
+ with tiledbvcf.Dataset(uri, cfg=cfg) as ds:
+ stats = ds.read_variant_stats(regions=["chr1:1-100"], drop_ref=True)
+ stats["vcf_pos"] = stats["pos"] + 1
+ print(stats[["contig", "vcf_pos", "alleles", "ac", "an", "af"]])
- ### Custom Configurations
- ```python
- # Advanced configuration
- config = tiledbvcf.ReadConfig(
- memory_budget=4096,
- tiledb_config={
- "sm.tile_cache_size": "1000000000",
- "vfs.s3.region": "us-east-1"
- }
- )
+ qc = tiledbvcf.sample_qc(uri, samples=["S1"], config=config)
+ print(qc)
```
-
- ## Resources
-
- ## Getting Help
-
- ### Open Source TileDB-VCF Resources
-
- **Open Source Documentation:**
- - TileDB Academy: https://cloud.tiledb.com/academy/
- - Population Genomics Guide: https://cloud.tiledb.com/academy/structure/life-sciences/population-genomics/
- - TileDB-VCF GitHub: https://github.com/TileDB-Inc/TileDB-VCF
-
- ### TileDB-Cloud Resources
-
- **For Large-Scale/Production Genomics:**
- - TileDB-Cloud Platform: https://cloud.tiledb.com
- - TileDB Academy (All Documentation): https://cloud.tiledb.com/academy/
-
- **Getting Started:**
- - Free account signup: https://cloud.tiledb.com
- - Contact: sales@tiledb.com for enterprise needs
-
- ## Scaling to TileDB-Cloud
-
- When your genomics workloads outgrow single-node processing, TileDB-Cloud provides enterprise-scale capabilities for production genomics pipelines.
-
- **Note**: This section covers TileDB-Cloud capabilities based on available documentation. For complete API details and current functionality, consult the official TileDB-Cloud documentation and API reference.
-
- ### Setting Up TileDB-Cloud
+ Statistics arrays are enabled by default at creation; older/disabled arrays may
+ not support these operations. `read_variant_stats` has no `samples` argument.
+ The older `read_allele_frequency(dataset_uri, region)` wrapper accepts a single
+ region and delegates to the deprecated singular argument; prefer the Dataset
+ method above. `sample_qc` uses `dataset_uri`, not `uri`, as its first argument.
- **1. Create Account and Get API Token**
- ```bash
- # Sign up at https://cloud.tiledb.com
- # Generate API token in your account settings
- ```
+ The `ac`, `an`, and `af` values summarize the ingested cohort, not a selected
+ ancestry or query sample subset. A `read(samples=[...], set_af_filter=">0.6")`
+ filter still used cohort AF in the tested release. Compute subgroup frequencies
+ from correctly selected calls with explicit missingness, ploidy, callable-region,
+ and gVCF reference-block policies. Use an appropriate called-allele denominator,
+ not universally `2 * number_of_samples`; a missing VCF record is not proof of a
+ homozygous-reference call. QC metrics and sparse storage do not establish GWAS
+ readiness or scientific validity.
- **2. Install TileDB-Cloud Python Client**
- ```bash
- # Base installation
- uv pip install tiledb-cloud
+ ## Object storage and TileDB Cloud
- # With genomics-specific functionality
- uv pip install tiledb-cloud[life-sciences]
- ```
+ Direct storage uses `s3://bucket/path`, `azure://container/path`, or
+ `gcs://bucket/path`, supported by the installed TileDB backend and its provider
+ credentials. This does not automatically distribute the computation. Use the
+ provider's credential chain or scoped configuration, and verify access to both
+ the dataset and source indexes. A TileDB Cloud token is distinct from bucket
+ credentials. Remote storage examples below are illustrative, source-verified,
+ and not authenticated end-to-end tests.
- **3. Configure Authentication**
- ```bash
- # Set environment variable with your API token
- export TILEDB_REST_TOKEN="your_api_token"
- ```
+ Install `tiledb-cloud==0.14.4` in a compatible environment. Its `life-sciences`
+ extra adds TileDB-SOMA, **not TileDB-VCF**. Supply `TILEDB_REST_TOKEN` through the
+ execution environment before importing the client; `TILEDB_REST_HOST` selects a
+ custom deployment if required. Do not embed tokens in code or print configuration.
```python
+ # Illustrative: requires an accessible registered cohort and a billing namespace.
import tiledb.cloud
-
- # Authentication is automatic via TILEDB_REST_TOKEN
- # No explicit login required in code
- ```
-
- ### Migrating from Open Source to TileDB-Cloud
-
- **Large-Scale Ingestion**
- ```python
- # TileDB-Cloud: Distributed VCF ingestion
import tiledb.cloud.vcf
- # Use specialized VCF ingestion module
- # Note: Exact API requires TileDB-Cloud documentation
- # This represents the available functionality structure
- tiledb.cloud.vcf.ingestion.ingest_vcf_dataset(
- source="s3://my-bucket/vcf-files/",
- output="tiledb://my-namespace/large-dataset",
- namespace="my-namespace",
- acn="my-s3-credentials",
- ingest_resources={"cpu": "16", "memory": "64Gi"}
- )
- ```
-
- **Distributed Query Processing**
- ```python
- # TileDB-Cloud: VCF querying across distributed storage
- import tiledb.cloud.vcf
- import tiledbvcf
-
- # Define the dataset URI
- dataset_uri = "tiledb://TileDB-Inc/gvcf-1kg-dragen-v376"
-
- # Get all samples from the dataset
- ds = tiledbvcf.Dataset(dataset_uri, tiledb_config=cfg)
- samples = ds.samples()
-
- # Define attributes and ranges to query on
- attrs = ["sample_name", "fmt_GT", "fmt_AD", "fmt_DP"]
- regions = ["chr13:32396898-32397044", "chr13:32398162-32400268"]
+ cloud_cfg = tiledb.cloud.Config()
+ with tiledbvcf.Dataset("tiledb://my-namespace/cohort", tiledb_config=cloud_cfg) as ds:
+ sample_names = ds.samples()
- # Perform the read, which is executed in a distributed fashion
- df = tiledb.cloud.vcf.read(
- dataset_uri=dataset_uri,
- regions=regions,
- samples=samples,
- attrs=attrs,
- namespace="my-namespace", # specifies which account to charge
+ result = tiledb.cloud.vcf.read(
+ dataset_uri="tiledb://my-namespace/cohort",
+ config=cloud_cfg, attrs=["sample_name", "pos_start", "fmt_GT"],
+ regions=["chr1:1-100"], samples=sample_names[:2],
+ num_region_partitions=1, namespace="my-namespace", max_workers=2,
)
- df.to_pandas()
+ frame = result.to_pandas() # read returns an Arrow table, not a DataFrame.
```
- ### Enterprise Features
+ Distributed reads assemble an Arrow result and may use significant worker/client
+ memory. Bound the requested regions/samples and worker count; partitioning does
+ not make the final concatenated result memory-free. This is SDK task execution,
+ not a paginated VCF REST-list API.
- **Data Sharing and Collaboration**
```python
- # TileDB-Cloud provides enterprise data sharing capabilities
- # through namespace-based permissions and group management
-
- # Access shared datasets via TileDB-Cloud URIs
- dataset_uri = "tiledb://shared-namespace/population-study"
-
- # Collaborate through shared notebooks and compute resources
- # (Specific API requires TileDB-Cloud documentation)
+ # Illustrative, mutating cloud job: use only for an authorized ingestion task.
+ submission = tiledb.cloud.vcf.ingest(
+ dataset_uri="s3://my-bucket/cohort",
+ sample_list_uri="s3://my-bucket/inputs/sample-uris.txt",
+ namespace="my-namespace", acn="registered-storage-role",
+ register_name="cohort", max_samples=2,
+ ingest_resources={"cpu": "2", "memory": "4Gi"},
+ )
+ print(submission["graph_id"])
```
- **Cost Optimization**
- - **Serverless Compute**: Pay only for actual compute time
- - **Auto-scaling**: Automatically scale up/down based on workload
- - **Spot Instances**: Use cost-optimized compute for batch jobs
- - **Data Tiering**: Automatic hot/cold storage management
-
- **Security and Compliance**
- - **End-to-end Encryption**: Data encrypted in transit and at rest
- - **Access Controls**: Fine-grained permissions and audit logs
- - **HIPAA/SOC2 Compliance**: Enterprise security standards
- - **VPC Support**: Deploy in private cloud environments
-
- ### When to Migrate Checklist
-
- ✅ **Migrate to TileDB-Cloud if you have:**
- - [ ] Datasets > 1000 samples
- - [ ] Need to process > 100GB of VCF data
- - [ ] Require distributed computing
- - [ ] Multiple team members need access
- - [ ] Need enterprise security/compliance
- - [ ] Want cost-optimized serverless compute
- - [ ] Require 24/7 production uptime
+ Use exactly one of `search_uri`, `sample_list_uri`, or `metadata_uri`; file-search
+ patterns apply to `search_uri`. Registration requires the access credential name
+ (`acn`). `vcf.ingest` submits asynchronously and returns
+ `{"status": "started", "graph_id": ...}`; track that graph to terminal success
+ and verify ingested samples before claiming completion. There is no released
+ `ingest_vcf_dataset(source=..., output=...)` API. Pricing, availability, security
+ controls, and deployment obligations require the current account/service terms.
- ### Getting Started with TileDB-Cloud
+ ## Official references
- 1. **Start Free**: TileDB-Cloud offers free tier for evaluation
- 2. **Migration Support**: TileDB team provides migration assistance
- 3. **Training**: Access to genomics-specific tutorials and examples
- 4. **Professional Services**: Custom deployment and optimization
+ - [TileDB-VCF 0.40.3 release](https://github.com/TileDB-Inc/TileDB-VCF/releases/tag/0.40.3)
+ - [Dataset API](https://tiledb-inc.github.io/TileDB-VCF/documentation/reference/Dataset.html)
+ - [Ingestion](https://tiledb-inc.github.io/TileDB-VCF/documentation/how-to/ingest-samples.html)
+ - [Large queries](https://tiledb-inc.github.io/TileDB-VCF/documentation/how-to/handle-large-queries.html)
+ - [Cloud storage](https://tiledb-inc.github.io/TileDB-VCF/documentation/how-to/work-with-cloud-object-stores.html)
+ - [TileDB Cloud client release](https://pypi.org/project/tiledb-cloud/0.14.4/)
+ - [Verification details and known release issues](references/verification.md)
- **Next Steps:**
- - Visit https://cloud.tiledb.com to create account
- - Review documentation at https://cloud.tiledb.com/academy/
- - Contact sales@tiledb.com for enterprise needs