imaging-data-commons · diff
v1.4 to v1.5
347 added, 258 removed. Audit A to A.
---
name: imaging-data-commons
- description: Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses.
+ description: Query and download public cancer imaging data from NCI Imaging Data Commons. Invoke for any question about IDC collections, cancer imaging datasets, DICOM data access, radiology (CT, MR, PET) or pathology AI training sets, metadata queries, visualization, or license checks — even when the user doesn't explicitly mention "IDC". No authentication required.
license: This skill is provided under the MIT License. IDC data itself has individual licensing (mostly CC-BY, some CC-NC) that must be respected when using the data.
metadata:
- version: "1.4"
- source-skill-version: 1.4.0
+ version: "1.5"
+ source-skill-version: 1.8.1
skill-author: Andrey Fedorov, @fedorov
- idc-index: 0.11.14
- idc-data-version: v23
- repository: https://github.com/ImagingDataCommons/idc-claude-skill
+ idc-index: "0.12.5"
+ idc-data-version: "v24"
+ repository: https://github.com/ImagingDataCommons/imaging-data-commons-skill
---
# Imaging Data Commons
## Overview
- Use the `idc-index` Python package to query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
+ Query and download public cancer imaging data from the National Cancer Institute Imaging Data Commons (IDC). No authentication required for data access.
- **Current IDC Data Version: v23** (always verify with `IDCClient().get_idc_version()`)
+ **Expected network access:** IDC metadata is reachable three ways — a local DuckDB index shipped with the `idc-index` Python package (no network), or the hosted IDC service over MCP or REST (`api.imaging.datacommons.cancer.gov`, no authentication). File downloads use public GCS (`storage.googleapis.com`) and AWS S3 (`s3.amazonaws.com`) — no authentication required. DICOMweb access uses either the public IDC proxy (`proxy.imaging.datacommons.cancer.gov`, no auth) or the Google Cloud Healthcare API (`healthcare.googleapis.com`, requires GCP authentication). Optional BigQuery queries (`bigquery.googleapis.com`) also require GCP authentication. No credentials or environment variables are accessed by this skill.
- **Primary tool:** `idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index))
+ **Current IDC Data Version: v24** (always verify — see *Best Practices*)
- **CRITICAL - Check package version before anything else (run this FIRST):**
+ **Choose the access path first.** There is no single default: the cheapest correct path depends
+ on the session and the task.
- This block only *reports*. It never installs. If the version is too old, show the user the
- suggested command and wait for them to approve it — do not run an install on their behalf.
+ 1. **Session already has the IDC MCP server?** Route discovery and metadata there — see *IDC
+ MCP Server*.
+ 2. **Otherwise, is `idc-index` installed?** Run `python scripts/check_version.py`. If it passes,
+ use `idc-index` for everything.
+ 3. **Not installed, and the task is read-only metadata** — counts, attribute values, collection
+ lookups, SQL under 10 000 rows, licenses, citations, viewer URLs? **Use the REST API over
+ `curl`; do not install anything.** Installing costs ~77 MB of packaged index data plus
+ pandas, pyarrow, and duckdb, which a metadata question does not need. See *Data Access
+ Options*.
+ 4. **Not installed, and the task needs more than metadata** — downloading files, pandas or
+ plotting, pydicom/SimpleITK, pathology tiling, results past 10 000 rows, or a version-pinned
+ script the user re-runs? Install `idc-index`: `check_version.py` exits non-zero and prints
+ the exact install command for the running interpreter. Prefer a virtual environment, then
+ restart Python.
- ```python
- import idc_index
+ `idc-index` ([GitHub](https://github.com/imagingdatacommons/idc-index)) is still the most
+ capable path and the only one that moves image bytes; the rule is just not to pay for it before
+ the task calls for it. `check_version.py` never installs anything itself — it also flags a newer
+ `idc-index` or skill release when one exists.
- REQUIRED_VERSION = "0.11.14" # Must match metadata.idc-index in this file
- installed = idc_index.__version__
+ **Setup for the `idc-index` path:**
- def _parts(version):
- # Compare numerically: "0.9.0" < "0.11.14" is False as a string comparison.
- return tuple(int(p) if p.isdigit() else 0 for p in version.split(".")[:3])
+ ```python
+ from idc_index import IDCClient
+ client = IDCClient()
- if _parts(installed) < _parts(REQUIRED_VERSION):
- print(f"idc-index {installed} is older than the tested {REQUIRED_VERSION}.")
- print("Ask the user before installing. Suggested command, in a virtual environment:")
- print(f" uv pip install 'idc-index=={REQUIRED_VERSION}'")
- else:
- print(f"idc-index {installed} meets requirement ({REQUIRED_VERSION})")
+ # Verify IDC data version (should be "v24")
+ print(f"IDC data version: {client.get_idc_version()}")
```
- **Never** install into a system-managed Python with `--break-system-packages`. That flag exists to
- override a protection the distribution put there deliberately, and a skill has no business
- switching it off unattended. Install into a virtual environment, and pin the version you tested
- against so a later IDC release cannot silently change query results underneath a saved analysis.
+ **Core workflow:** query metadata with `client.sql_query()` → download with
+ `client.download_from_selection()` → visualize with `client.get_viewer_URL()`. Python examples
+ below assume this `client`; *Data Access Options* has the REST equivalents. For current data
+ scale, run the summary query in `references/sql_patterns.md` or `GET /v3/stats`.
- **Verify IDC data version and check current data scale:**
+ ## IDC MCP Server
- ```python
- from idc_index import IDCClient
- client = IDCClient()
+ IDC operates a hosted MCP server at `https://api.imaging.datacommons.cancer.gov/mcp`
+ (streamable HTTP, no authentication). Where it is available it complements — it does not
+ replace — the `idc-index` workflow below.
- # Verify IDC data version (should be "v23")
- print(f"IDC data version: {client.get_idc_version()}")
+ **Identify it** by the MCP resource `idc://guide`, or by three or more of the tool names
+ `build_cohort`, `get_cohort_urls`, `list_analysis_results`, and `get_idc_version`. Generic
+ names such as `run_sql` are not evidence on their own. If identification is ambiguous, use
+ `idc-index`.
- # Get collection count and total series
- stats = client.sql_query("""
- SELECT
- COUNT(DISTINCT collection_id) as collections,
- COUNT(DISTINCT analysis_result_id) as analysis_results,
- COUNT(DISTINCT PatientID) as patients,
- COUNT(DISTINCT StudyInstanceUID) as studies,
- COUNT(DISTINCT SeriesInstanceUID) as series,
- SUM(instanceCount) as instances,
- SUM(series_size_MB)/1000000 as size_TB
- FROM index
- """)
- print(stats)
- ```
+ **If this session has the server**, treat it as authoritative for discovery and metadata —
+ IDC version, counts, attribute values, cohort building, metadata SQL — and follow the
+ server's own instructions rather than re-deriving them from this file. Its data version is
+ whatever the server reports: call `get_idc_version` instead of relying on the version pinned
+ in this file.
- **Core workflow:**
- 1. Query metadata → `client.sql_query()`
- 2. Download DICOM files → `client.download_from_selection()`
- 3. Visualize in browser → `client.get_viewer_URL(seriesInstanceUID=...)`
+ Return here for what the server does not do: downloading files, local pandas/notebook
+ analysis, DICOMweb, BigQuery, digital pathology tiling, and reproducible scripts. Hand off by
+ passing SeriesInstanceUIDs from the server to `client.download_from_selection(...)`, and run
+ `scripts/check_version.py` at that point.
+ **If it is not available**, the identical service is reachable with no configuration as a REST
+ API at `https://api.imaging.datacommons.cancer.gov/v3` — use it for read-only metadata rather
+ than installing `idc-index`, per the routing gate in *Overview*. Suggest connecting the MCP
+ server at most once, only for repeated interactive discovery, and never change the user's
+ configuration yourself.
+
+ See `references/mcp_guide.md` for the tool inventory, handoff patterns, and per-host notes.
+
## When to Use This Skill
- Finding publicly available radiology (CT, MR, PET) or pathology (slide microscopy) images
- Selecting image subsets by cancer type, modality, anatomical site, or other metadata
- Downloading DICOM data from IDC
- Checking data licenses before use in research or commercial applications
- Visualizing medical images in a browser without local DICOM viewer software
## Quick Navigation
- **Core Sections (inline):**
- - IDC Data Model - Collection and analysis result hierarchy
- - Index Tables - Available tables and joining patterns
- - Installation - Package setup and version verification
- - Core Capabilities - Essential API patterns (query, download, visualize, license, citations, batch)
- - Best Practices - Usage guidelines
- - Troubleshooting - Common issues and solutions
+ Inline below: the MCP/REST routing rules, the IDC data model, the index tables and how they
+ join, the core API patterns (query, download, visualize, license, cite), best practices, and
+ troubleshooting.
**Reference Guides (load on demand):**
| Guide | When to Load |
|-------|--------------|
| `index_tables_guide.md` | Complex JOINs, schema discovery, DataFrame access |
- | `use_cases.md` | End-to-end workflow examples (training datasets, batch downloads) |
+ | `use_cases.md` | End-to-end workflows: training datasets, batch downloads, DICOM reading with pydicom/SimpleITK, pipeline integration |
| `sql_patterns.md` | Quick SQL patterns for filter discovery, annotations, size estimation |
| `clinical_data_guide.md` | Clinical/tabular data, imaging+clinical joins, value mapping |
+ | `licensing_and_citation.md` | Commercial-use questions, mixed-license cohorts, citation formats |
| `cloud_storage_guide.md` | Direct S3/GCS access, versioning, UUID mapping |
| `dicomweb_guide.md` | DICOMweb endpoints, PACS integration |
| `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows |
| `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) |
| `cli_guide.md` | Command-line tools (`idc download`, manifest files) |
| `parquet_access_guide.md` | Direct Parquet queries via GCS (no idc-index install needed) |
+ | `mcp_guide.md` | Hosted IDC MCP server: tool inventory, identification, handoff to `idc-index` |
+ | `rest_api_guide.md` | Hosted IDC REST API: endpoints, filter syntax, SQL over HTTP, manifests |
## IDC Data Model
IDC adds two grouping levels above the standard DICOM hierarchy (Patient → Study → Series → Instance):
- **collection_id**: Groups patients by disease, modality, or research focus (e.g., `tcga_luad`, `nlst`). A patient belongs to exactly one collection.
- - **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections.
-
- Use `collection_id` to find original imaging data, may include annotations deposited along with the images; use `analysis_result_id` to find AI-generated or expert annotations.
+ - **analysis_result_id**: Identifies derived objects (segmentations, annotations, radiomics features) across one or more original collections. Use it to find AI-generated or expert annotations, while `collection_id` finds original imaging data (which may itself include deposited annotations).
**Key identifiers for queries:**
| Identifier | Scope | Use for |
|------------|-------|---------|
| `collection_id` | Dataset grouping | Filtering by project/study |
| `PatientID` | Patient | Grouping images by patient |
| `StudyInstanceUID` | DICOM study | Grouping of related series, visualization |
| `SeriesInstanceUID` | DICOM series | Grouping of related series, visualization |
## Index Tables
- The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames.
-
- **Complete index table documentation:** Use https://idc-index.readthedocs.io/en/latest/indices_reference.html for quick check of available tables and columns without executing any code.
+ The `idc-index` package provides multiple metadata index tables, accessible via SQL or as pandas DataFrames. The REST API exposes the same tables through `GET /tables` and `POST /sql`.
- **Important:** Use `client.indices_overview` to get current table descriptions and column schemas. This is the authoritative source for available columns and their types — always query it when writing SQL or exploring data structure.
+ **Important:** `client.indices_overview` is the authoritative source for current table descriptions, available columns, and their types — query it when writing SQL or exploring data structure. It also answers "which table contains column X"; see `references/index_tables_guide.md` for that search pattern and full schema discovery.
### Available Tables
- | Table | Row Granularity | Loaded | Description |
- |-------|-----------------|--------|-------------|
- | `index` | 1 row = 1 DICOM series | Auto | Primary metadata for all current IDC data |
- | `prior_versions_index` | 1 row = 1 DICOM series | Auto | Series from previous IDC releases; for downloading deprecated data |
- | `collections_index` | 1 row = 1 collection | fetch_index() | Collection-level metadata and descriptions |
- | `analysis_results_index` | 1 row = 1 analysis result collection | fetch_index() | Metadata about derived datasets (annotations, segmentations) |
- | `clinical_index` | 1 row = 1 clinical data column | fetch_index() | Dictionary mapping clinical table columns to collections |
- | `sm_index` | 1 row = 1 slide microscopy series | fetch_index() | Slide Microscopy (pathology) series metadata |
- | `sm_instance_index` | 1 row = 1 slide microscopy instance | fetch_index() | Instance-level (SOPInstanceUID) metadata for slide microscopy |
- | `seg_index` | 1 row = 1 DICOM Segmentation series | fetch_index() | Segmentation metadata: algorithm, segment count, reference to source image series |
- | `ann_index` | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
- | `ann_group_index` | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
- | `contrast_index` | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
- | `volume_geometry_index` | 1 row = 1 CT/MR/PT series | fetch_index() | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag |
- | `rtstruct_index` | 1 row = 1 RTSTRUCT series | fetch_index() | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
+ Always call `client.fetch_index("table_name")` before querying any index table — it is safe and idempotent for all tables, including those loaded automatically at startup.
- **Auto** = loaded automatically when `IDCClient()` is instantiated
- **fetch_index()** = requires `client.fetch_index("table_name")` to load
+ | Family | Tables | Granularity |
+ |--------|--------|-------------|
+ | Core | `index` (primary metadata for all current data), `collections_index`, `analysis_results_index` | series / collection / analysis result |
+ | Modality acquisition parameters | `ct_index`, `mr_index`, `pt_index`, `contrast_index` | 1 row = 1 series of that modality |
+ | Derived objects | `seg_index`, `rtstruct_index`, `ann_index`, `ann_group_index` | 1 row = 1 series (or annotation group) |
+ | Microscopy | `sm_index`, `sm_instance_index` | 1 row = 1 SM series / instance |
+ | Geometry, clinical, history | `volume_geometry_index`, `clinical_index`, `version_metadata_index`, `prior_versions_index` | see guide |
+ `references/index_tables_guide.md` has the full inventory with each table's columns and
+ contents — load it when you need to know what a specialized table actually holds.
+
+ **`prior_versions_index` is for reproducibility only.** It contains series permanently *removed*
+ from IDC, with zero overlap with `index`. Use it only to reproduce work against a prior IDC
+ version. Do NOT use it for version history or "what's new" questions — those use
+ `series_init_idc_version` / `series_revised_idc_version` in the main `index` table, which are
+ not equivalent to this table's `min_idc_version` / `max_idc_version`.
+
### Joining Tables
- **Key columns are not explicitly labeled, the following is a subset that can be used in joins.**
+ **`SeriesInstanceUID` is the universal join key** for all series-level specialized tables: `sm_index`, `sm_instance_index`, `seg_index`, `ann_index`, `ann_group_index`, `contrast_index`, `volume_geometry_index`, `rtstruct_index`, `ct_index`, `mr_index`, `pt_index`. Always join these to `index` on `SeriesInstanceUID`. The exceptions below use different column names.
| Join Column | Tables | Use Case |
|-------------|--------|----------|
| `collection_id` | index, prior_versions_index, collections_index, clinical_index | Link series to collection metadata or clinical data |
- | `SeriesInstanceUID` | index, prior_versions_index, sm_index, sm_instance_index | Link series across tables; connect to slide microscopy details |
- | `StudyInstanceUID` | index, prior_versions_index | Link studies across current and historical data |
- | `PatientID` | index, prior_versions_index | Link patients across current and historical data |
| `analysis_result_id` | index, analysis_results_index | Link series to analysis result metadata (annotations, segmentations) |
| `source_DOI` | index, analysis_results_index | Link by publication DOI |
- | `crdc_series_uuid` | index, prior_versions_index | Link by CRDC unique identifier |
- | `Modality` | index, prior_versions_index | Filter by imaging modality |
- | `SeriesInstanceUID` | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation/annotation/contrast series to its index metadata |
- | `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |
- | `referenced_SeriesInstanceUID` | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |
- | `SeriesInstanceUID` | index, volume_geometry_index | Link series to its 3D geometry validation result (join index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID) |
- | `SeriesInstanceUID` / `referenced_SeriesInstanceUID` | index, rtstruct_index | Join RTSTRUCT series to its metadata (index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID); use rtstruct_index.referenced_SeriesInstanceUID to find the source image series |
+ | `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (`seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID`) |
+ | `referenced_SeriesInstanceUID` | ann_index → index, rtstruct_index → index | Link annotation or RTSTRUCT to its source image series |
- **Note:** `Subjects`, `Updated`, and `Description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).
+ **Note:** `subjects`, `updated`, and `description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts). Joining `prior_versions_index` to `index` on `SeriesInstanceUID` always returns zero rows — see the warning above.
For detailed join examples, schema discovery patterns, key columns reference, and DataFrame access, see `references/index_tables_guide.md`.
### Clinical Data Access
- ```python
- # Fetch clinical index (also downloads clinical data tables)
- client.fetch_index("clinical_index")
+ Clinical (non-imaging) attributes — staging, demographics, therapy — live in per-collection
+ tables. `client.fetch_index("clinical_index")` loads the dictionary mapping columns to
+ collections; `client.get_clinical_table(name)` returns one table as a DataFrame.
- # Query clinical index to find available tables and their columns
- tables = client.sql_query("SELECT DISTINCT table_name, column_label FROM clinical_index")
+ See `references/clinical_data_guide.md` for the discovery workflow, coded-value mapping, and
+ joining clinical data with imaging.
- # Load a specific clinical table as DataFrame
- clinical_df = client.get_clinical_table("table_name")
- ```
+ ## Data Access Options
- See `references/clinical_data_guide.md` for detailed workflows including value mapping patterns and joining clinical data with imaging.
+ | Method | Auth | Best For | Reference |
+ |--------|------|----------|-----------|
+ | `idc-index` | No | Downloads, pandas analysis, unbounded queries — the most capable path | This document |
+ | IDC MCP server | No | Discovery, cohort building, metadata when the session already has it | `mcp_guide.md` |
+ | IDC REST API | No | Metadata with no install, from any language or shell — the default when `idc-index` is absent | `rest_api_guide.md` |
+ | Direct Parquet (GCS) | No | Version-pinned queries, or results past the REST row cap | `parquet_access_guide.md` |
+ | Cloud storage (S3/GCS) | No | Direct file access, bulk transfer, custom pipelines | `cloud_storage_guide.md` |
+ | DICOMweb via IDC proxy | No | Tool and PACS integration; daily quota, so testing and moderate use | `dicomweb_guide.md` |
+ | DICOMweb via Google Healthcare | Yes (GCP) | The same DICOMweb API at production volume, without the proxy quota | `dicomweb_guide.md` |
+ | SlicerIDCBrowser | No | 3D visualization and analysis in 3D Slicer | https://github.com/ImagingDataCommons/SlicerIDCBrowser |
+ | BigQuery | Yes (GCP) | Full DICOM metadata, private elements, SR measurements — last resort | `bigquery_guide.md` |
- ## Data Access Options
+ **The IDC Portal (https://portal.imaging.datacommons.cancer.gov/) is interactive only** —
+ browser-based exploration, manual cohort selection, and download. Unlike every option above it
+ has no programmatic interface, so point a user there to browse or click through data
+ themselves; never use it as a step in a script or workflow.
- | Method | Auth Required | Best For |
- |--------|---------------|----------|
- | `idc-index` | No | Key queries and downloads (recommended) |
- | Direct Parquet (GCS) | No | Quick queries without installing idc-index; always uses latest data |
- | IDC Portal | No | Interactive exploration, manual selection, browser-based download |
- | BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |
- | DICOMweb proxy | No | Tool integration via DICOMweb API |
- | Cloud storage (S3/GCS) | No | Direct file access, bulk downloads, custom pipelines |
+ **REST API — the no-install metadata path**
- **Cloud storage organization**
+ `https://api.imaging.datacommons.cancer.gov/v3`, no authentication: discovery, cohort counts and
+ manifests, read-only SQL, clinical tables, viewer URLs, licenses, citations. It is the same
+ service as the MCP server over plain HTTP, so it needs no configuration. It never moves image
+ bytes — switch to `idc-index` to download, to get a DataFrame, or for results past 10 000 rows.
- IDC maintains all DICOM files in public cloud storage buckets mirrored between AWS S3 and Google Cloud Storage. Files are organized by CRDC UUIDs (not DICOM UIDs) to support versioning.
+ ```bash
+ B=https://api.imaging.datacommons.cancer.gov/v3
+ curl -s $B/version # idc_version, idc_index_data_version, api_version
+ curl -s $B/stats # collections, patients, studies, series, instances, size_TB
+ curl -s "$B/attributes/Modality/values?limit=5" # real filter values, with counts
+ curl -s $B/sql -H 'content-type: application/json' \
+ -d '{"sql":"SELECT collection_id, COUNT(*) n FROM index GROUP BY 1 ORDER BY n DESC LIMIT 3"}'
+ curl -s $B/cohort/counts -H 'content-type: application/json' \
+ -d '{"filters":{"terms":{"collection_id":["rider_pilot"]}}}'
+ ```
- | Bucket (AWS / GCS) | License | Content |
- |--------------------|---------|---------|
- | `idc-open-data` / `idc-open-data` | No commercial restriction | >90% of IDC data |
- | `idc-open-data-two` / `idc-open-idc1` | No commercial restriction | Collections with potential head scans |
- | `idc-open-data-cr` / `idc-open-cr` | Commercial use restricted (CC BY-NC) | ~4% of data |
+ **The filter object always goes under `filters`** — on `cohort/counts`, `cohort/manifest`,
+ `cohort/manifest.txt`, `licenses`, and `citations` alike. A bare filter or an unrecognized key is
+ a 422 naming the fix; an unfiltered series-enumerating request is a 400, not the whole archive.
+ Every filtered response echoes `filters_applied` and `warnings` — read them, because they name
+ any predicate the server dropped. A zero count with empty `warnings` therefore means the filter
+ matched nothing, not that a value was miscased; miscasing produces a warning that says so.
- Files are stored as `<crdc_series_uuid>/<crdc_instance_uuid>.dcm`. Access is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access. Use `series_aws_url` column from the index for S3 URLs; GCS uses the same path structure.
+ `POST /sql` takes one read-only `SELECT`/`WITH` over the tables `idc-index` exposes plus
+ `clinical.<table>`; `max_rows` defaults to 5 000, caps at 10 000, and `truncated` flags clipping.
+ `GET /attributes` lists the 19 filterable attributes — clinical values, segmented anatomy, and
+ acquisition parameters are not among them and need SQL. There is no rate limit or quota. **Use
+ v3 only:** V1 and V2 are superseded and scheduled for shutdown, so port any `/v1/`- or
+ `Modality_btw`-style example a user brings rather than extending it.
- See `references/cloud_storage_guide.md` for bucket details, access commands, UUID mapping, and versioning.
+ Both sides build on `idc-index-data`, so compare the API's `idc_index_data_version` against local
+ `idc_index_data.__version__` before mixing them: the **major is the IDC data release** (`24.x.y`
+ serves `v24`), so differing minor/patch means the series are identical. If the API is a whole
+ release ahead, `idc-index` **cannot download the extra series** — it silently skips what its own
+ index does not list — so either upgrade it (run `scripts/check_version.py` for the right command)
+ or transfer directly from the bucket with `s5cmd --no-sign-request`.
- **DICOMweb access**
+ See `references/rest_api_guide.md` for the endpoint reference, filter grounding, limits, and the
+ manifest-based download flow.
- IDC data is available via DICOMweb interface (Google Cloud Healthcare API implementation) for integration with PACS systems and DICOMweb-compatible tools.
+ **Cloud storage organization**
- | Endpoint | Auth | Use Case |
- |----------|------|----------|
- | Public proxy | No | Testing, moderate queries, daily quota |
- | Google Healthcare | Yes (GCP) | Production use, higher quotas |
+ All DICOM files live in public buckets mirrored between AWS S3 and GCS, organized by CRDC UUIDs
+ (not DICOM UIDs) to support versioning, as `<crdc_series_uuid>/<crdc_instance_uuid>.dcm`. Access
+ is free (no egress fees) via AWS CLI, gsutil, or s5cmd with anonymous access; use the
+ `series_aws_url` column for S3 URLs. Note that `idc-open-data-cr` / `idc-open-cr` (~4% of data)
+ is commercial-use restricted (CC BY-NC). See `references/cloud_storage_guide.md` for the full
+ bucket list and UUID mapping.
- See `references/dicomweb_guide.md` for endpoint URLs, code examples, supported operations, and implementation details.
+ **DICOMweb access**
+ IDC data is available via DICOMweb (Google Cloud Healthcare API) for PACS integration and
+ DICOMweb-compatible tools: a public proxy (no auth, daily quota) for testing and moderate
+ queries, or Google Healthcare (GCP auth) for production volumes. See
+ `references/dicomweb_guide.md`.
+
**Direct Parquet access**
- All idc-index metadata tables are published as Parquet files to a public GCS bucket (`idc-index-data-artifacts`) with unrestricted CORS. This enables DuckDB or pandas queries without installing idc-index, including cross-table joins and queries against `volume_geometry_index` and `rtstruct_index`.
+ The idc-index metadata tables are also published as Parquet on a public GCS bucket
+ (`idc-index-data-artifacts`), queryable with DuckDB or pandas. This needs DuckDB installed
+ and cannot reach the per-collection clinical tables, so prefer REST `/sql` for ad-hoc metadata;
+ choose Parquet to pin a data version or for results past the REST row cap. See
+ `references/parquet_access_guide.md`.
- See `references/parquet_access_guide.md` for URL patterns, available files, and DuckDB query examples.
+ ## Core Capabilities
- ## Installation and Setup
+ The patterns below are the ones that go wrong when recalled from memory rather than checked.
+ Worked examples for each area live in the reference guides named inline.
- **Required (for basic access):** install into a virtual environment, pinned to the tested release:
- ```bash
- uv pip install 'idc-index==0.11.14'
- ```
+ ### 1. Discovery — enumerate values before filtering on them
- **Important:** every new IDC data release ships a new `idc-index`. Moving to a newer version
- changes which data your queries see, so treat it as a deliberate step: check the release notes,
- then pin the new version here. An unpinned `--upgrade` makes the data version a moving target and
- silently breaks reproducibility of an analysis you ran last month.
+ Filtering on a guessed `Modality` or `BodyPartExamined` string is the most common cause of an
+ empty result set. Enumerate first:
- **IMPORTANT:** IDC data version v23 is current. Always verify your version:
```python
- print(client.get_idc_version()) # Should return "v23"
+ modalities = client.sql_query("""
+ SELECT DISTINCT Modality, COUNT(*) as series_count
+ FROM index
+ GROUP BY Modality
+ ORDER BY series_count DESC
+ """)
+ print(modalities)
```
- If it returns an older version, tell the user which version they have and which one this skill was
- tested against, and let them decide whether to upgrade.
- **Tested with:** idc-index 0.11.14 (IDC data version v23)
+ The same pattern works for any filter column, optionally narrowed by another —
+ `BodyPartExamined` within a `Modality`, `Manufacturer`, `collection_id`. On the REST path this
+ grounding is a single call — `GET /attributes/{attr}/values` returns values with counts — and the
+ cohort endpoints report a miscased value in `warnings` rather than as an empty result.
- **Optional (for data analysis):**
- ```bash
- uv pip install pandas numpy pydicom
+ Two indices carry curated collection-level metadata the primary `index` does not, both
+ requiring `client.fetch_index(...)` first: `collections_index` (cancer types, tumor locations,
+ species, subject counts) and `analysis_results_index` (derived datasets — AI segmentations,
+ expert annotations, radiomics — with their source collections and modalities).
+
+ **Cancer type lives in `collections_index.cancer_types`, not in `index`** — filtering by
+ cancer type requires a join:
+
+ ```python
+ client.fetch_index("collections_index")
+ results = client.sql_query("""
+ SELECT i.collection_id, i.PatientID, i.SeriesInstanceUID, i.Modality
+ FROM index i
+ JOIN collections_index c ON i.collection_id = c.collection_id
+ WHERE c.cancer_types LIKE '%Breast%'
+ AND i.Modality = 'MR'
+ LIMIT 20
+ """)
```
- ## Core Capabilities
+ `client.sql_query()` returns a pandas DataFrame. Confirm column names with
+ `client.get_index_schema('index')` or `client.indices_overview` before writing a query rather
+ than assuming them.
- Nine capability areas, each with worked code, are documented in
- [references/core_capabilities.md](references/core_capabilities.md):
+ See `references/sql_patterns.md` for filter-value discovery, annotation and segmentation
+ queries, size estimation, clinical linking, and version tracking ("what's new in vX" — use
+ `series_init_idc_version` / `series_revised_idc_version` in `index`, never
+ `prior_versions_index`).
- 1. **Data discovery and exploration** — summary statistics, and enumerating the actual
- `Modality` and `BodyPartExamined` values before filtering on them.
- 2. **Querying metadata with SQL** — against `index`, `collections_index`, and
- `analysis_results_index`, returning pandas DataFrames.
- 3. **Downloading DICOM files** — Python and CLI, by collection, series UID, or manifest,
- with control over directory hierarchy (full, simplified, or flat).
- 4. **Visualizing IDC images** — single series or a whole study in the OHIF viewer.
- 5. **Licenses and citations** — per-collection license checks, and citations in APA or
- BibTeX.
- 6. **Batch processing and filtering** — scanner- and protocol-level filters, manifests,
- and batched downloads that avoid timeouts.
- 7. **Advanced BigQuery queries** — for joins and aggregations beyond the index API.
- 8. **Tool selection guide** — which access path fits which task.
- 9. **Integration with analysis pipelines** — reading series with pydicom, processing with
- SimpleITK, and converting to NIfTI.
+ ### 2. Downloading DICOM files
- Always explore the real column values first: filtering on a guessed `Modality` or
- `BodyPartExamined` string is the most common cause of an empty result set.
+ **The two download methods take their first two arguments in opposite order.** This is the
+ most common source of broken IDC code — check it rather than recalling it:
- ## Common Use Cases
+ | Method | First arg | Second arg | Use when |
+ |--------|-----------|------------|----------|
+ | `download_from_selection` | `downloadDir` (required) | filter kwargs (optional) | Filtering by collection, patient, study, or series |
+ | `download_dicom_series` | `seriesInstanceUID` (required) | `downloadDir` (required) | Downloading specific series by UID only |
- See `references/use_cases.md` for complete end-to-end workflow examples including:
- - Building deep learning training datasets from lung CT scans
- - Comparing image quality across scanner manufacturers
- - Previewing data in browser before downloading
- - License-aware batch downloads for commercial use
+ **`download_from_selection` takes filter keyword arguments, NOT a DataFrame.** The name
+ "from_selection" refers to filtering the IDC index by criteria — not to accepting a pandas
+ DataFrame. To download query results, extract the UIDs into a list first:
- ## Best Practices
+ ```python
+ # Step 1: Query for series UIDs
+ series_df = client.sql_query("""
+ SELECT SeriesInstanceUID
+ FROM index
+ WHERE Modality = 'CT'
+ AND BodyPartExamined = 'CHEST'
+ AND collection_id = 'nlst'
+ LIMIT 5
+ """)
- - **Verify IDC version before generating responses** - Always call `client.get_idc_version()` at the start of a session to confirm you're using the expected data version (currently v23). If using an older version, report it and let the user decide whether to install a newer pinned release; never install on their behalf
- - **Check licenses before use** - Always query the `license_short_name` field and respect licensing terms (CC BY vs CC BY-NC)
- - **Generate citations for attribution** - Use `citations_from_selection()` to get properly formatted citations from `source_DOI` values; include these in publications
- - **Start with small queries** - Use `LIMIT` clause when exploring to avoid long downloads and understand data structure
- - **Use mini-index for simple queries** - Only use BigQuery when you need comprehensive metadata or complex JOINs
- - **Organize downloads with dirTemplate** - Use meaningful directory structures like `%collection_id/%PatientID/%Modality`
- - **Cache query results** - Save DataFrames to CSV files to avoid re-querying and ensure reproducibility
- - **Estimate size first** - Check collection size before downloading - some collection sizes are in terabytes!
- - **Save manifests** - Always save query results with Series UIDs for reproducibility and data provenance
- - **Read documentation** - IDC data structure and metadata fields are documented at https://learn.canceridc.dev/
- - **Use IDC forum** - Search for questions/answers and ask your questions to the IDC maintainers and users at https://discourse.canceridc.dev/
+ # Step 2: Extract UIDs as a list from the DataFrame
+ uids = list(series_df['SeriesInstanceUID'].values)
- ## Troubleshooting
+ # Step 3: Pass the list to download_from_selection (NOT the DataFrame itself)
+ client.download_from_selection(
+ downloadDir="./data/lung_ct",
+ seriesInstanceUID=uids # list of strings, not a DataFrame
+ )
- **Issue: `ModuleNotFoundError: No module named 'idc_index'`**
- - **Cause:** idc-index package not installed
- - **Solution:** with the user's agreement, `uv pip install 'idc-index==0.11.14'` in a virtual environment
+ # Alternative: download_dicom_series has seriesInstanceUID as FIRST arg (different order!)
+ client.download_dicom_series(
+ seriesInstanceUID=uids, # FIRST arg here
+ downloadDir="./data/lung_ct"
+ )
- **Issue: Download fails with connection timeout**
- - **Cause:** Network instability or large download size
- - **Solution:**
- - Download smaller batches (e.g., 10-20 series at a time)
- - Check network connection
- - Use `dirTemplate` to organize downloads by batch
- - Implement retry logic with delays
+ # Whole collection: downloadDir is still the FIRST positional argument
+ client.download_from_selection(downloadDir="./data/rider", collection_id="rider_pilot")
+ ```
- **Issue: `BigQuery quota exceeded` or billing errors**
- - **Cause:** BigQuery requires billing-enabled GCP project
- - **Solution:** Use idc-index mini-index for simple queries (no billing required), or see `references/bigquery_guide.md` for cost optimization tips
+ Both methods default to AWS; pass `source_bucket_location="gcs"` to pull from Google Storage.
- **Issue: Series UID not found or no data returned**
- - **Cause:** Typo in UID, data not in current IDC version, or wrong field name
- - **Solution:**
- - Check if data is in current IDC version (some old data may be deprecated)
- - Use `LIMIT 5` to test query first
- - Check field names against metadata schema documentation
+ **Downloaded files are named `<crdc_instance_uuid>.dcm`, not by SOPInstanceUID.** The DICOM
+ UIDs are preserved inside the file metadata, not in the filename. Use the `crdc_instance_uuid`
+ column to map files back to the series they came from.
- **Issue: Downloaded DICOM files won't open**
- - **Cause:** Corrupted download or incompatible viewer
- - **Solution:**
- - Check DICOM object type (Modality and SOPClassUID attributes) - some object types require specialized tools
- - Verify file integrity (check file sizes)
- - Use pydicom to validate: `pydicom.dcmread(file, force=True)`
- - Try different DICOM viewer (3D Slicer, Horos, RadiAnt, QuPath)
- - Re-download the series
+ `idc download <collection|series-uid|manifest> --download-dir ./data` does the same from a
+ shell. See `references/cli_guide.md` for the `dirTemplate` hierarchy options (Python default:
+ `%collection_id/%PatientID/%StudyInstanceUID/%Modality_%SeriesInstanceUID`; `dirTemplate=""`
+ flattens), manifest downloads with resume, and dry-run size estimation.
- ## Common SQL Query Patterns
+ ### 3. Visualizing IDC images
- See `references/sql_patterns.md` for quick-reference SQL patterns including:
- - Filter value discovery (modalities, body parts, manufacturers)
- - Annotation and segmentation queries (including seg_index, ann_index joins)
- - Slide microscopy queries (sm_index patterns)
- - Download size estimation
- - Clinical data linking
+ ```python
+ viewer_url = client.get_viewer_URL(seriesInstanceUID=uid) # one series
+ viewer_url = client.get_viewer_URL(studyInstanceUID=study_uid) # all series in a study
+ ```
- For segmentation and annotation details, also see `references/digital_pathology_guide.md`.
+ Returns a browser URL — nothing is downloaded. The method selects OHIF v3 for radiology or
+ SLIM for slide microscopy automatically. Viewing by study is useful when a single DICOM Study
+ holds several Series (T1, T2, and DWI from one MRI session).
- ## Related Skills
+ ### 4. Licenses and citations — obligations, not optional steps
- The following skills complement IDC workflows for downstream analysis and visualization:
+ IDC data carries license terms and attribution requirements that follow it into any downstream
+ publication or product, and neither is inferable from the pixel data. **Check the license
+ before use, and generate citations for whatever you download.**
- ### DICOM Processing
- - **pydicom** - Read, write, and manipulate downloaded DICOM files. Use for extracting pixel data, reading metadata, anonymization, and format conversion. Essential for working with IDC radiology data (CT, MR, PET).
+ ```python
+ # License breakdown for a selection
+ licenses = client.sql_query("""
+ SELECT DISTINCT collection_id, license_short_name,
+ COUNT(DISTINCT SeriesInstanceUID) as series_count
+ FROM index GROUP BY collection_id, license_short_name
+ """)
- ### Pathology and Slide Microscopy
- See `references/digital_pathology_guide.md` for DICOM-compatible tools (highdicom, wsidicom, TIA-Toolbox, Slim viewer).
+ # Citations for the same selection you downloaded (APA by default)
+ for citation in client.citations_from_selection(collection_id="rider_pilot"):
+ print(citation)
+ ```
- ### Metadata Visualization
- - **matplotlib** - Low-level plotting for full customization. Use for creating static figures summarizing IDC query results (bar charts of modalities, histograms of series counts, etc.).
- - **seaborn** - Statistical visualization with pandas integration. Use for quick exploration of IDC metadata distributions, relationships between variables, and categorical comparisons with attractive defaults.
- - **plotly** - Interactive visualization. Use when you need hover info, zoom, and pan for exploring IDC metadata, or for creating web-embeddable dashboards of collection statistics.
+ About 97% of IDC data is CC BY (commercial use allowed with attribution) and about 3% is
+ CC BY-NC (non-commercial only). **Licenses attach to series, not collections** — 39 of 176
+ collections carry more than one — so check the selection you actually intend to use, and note
+ that the most restrictive term governs a mixed cohort.
- ### Data Exploration
- - **exploratory-data-analysis** - Comprehensive EDA on scientific data files. Use after downloading IDC data to understand file structure, quality, and characteristics before analysis.
+ Both tasks are available from all three access paths, so stay on whichever one the session is
+ already using: `idc-index` as above, `POST /v3/licenses` and `POST /v3/citations` over REST,
+ or the `get_licenses` and `get_citations` MCP tools. See
+ `references/licensing_and_citation.md` for the full license inventory, all three routes, the
+ citation formats (APA, BibTeX, CSL JSON, RDF Turtle), and what to include when publishing.
- ## Resources
+ ### 5. Reaching past the index
- ### Schema Reference (Primary Source)
+ Pick the access path with the routing gate in *Overview*; *Data Access Options* above is the
+ full routing table.
- **Always use `client.indices_overview` for current column schemas.** This ensures accuracy with the installed idc-index version:
+ Before reaching for BigQuery (which needs a billing-enabled GCP account), check whether a
+ specialized index table already has the column you want: search `client.indices_overview`,
+ then `client.fetch_index(...)` and query locally for free. BigQuery is required only for
+ private DICOM elements, per-segment anatomy (`segmentations`), and pre-extracted SR
+ measurements (`quantitative_measurements`, `qualitative_measurements`) — these have no
+ idc-index equivalent.
- ```python
- # Get all column names and types for any table
- schema = client.indices_overview["index"]["schema"]
- columns = [(c['name'], c['type'], c.get('description', '')) for c in schema['columns']]
- ```
+ ## Best Practices
- ### Reference Documentation
+ - **Check schema before writing queries** — Use `client.get_index_schema('index')` (reads cached metadata, no SQL executed) or `client.indices_overview` to see all available columns and their descriptions. The version-tracking columns `series_init_idc_version` and `series_revised_idc_version` in the main `index` table directly answer "what's new / when was this added" questions without touching `prior_versions_index`.
+ - **Never use web search for IDC data content questions** - Always query the IDC index directly, via `client.sql_query()` locally or `POST /v3/sql` over HTTP. Web sources (release notes, blog posts, documentation pages) are frequently out of date and will produce incorrect answers. The index is the authoritative source; use it even when web search is available.
+ - **Verify the IDC data version at the start of a session** - `client.get_idc_version()`, `GET /v3/version`, or the MCP `get_idc_version` tool, depending on the path in use (currently v24). For a stale local index, run `scripts/check_version.py` and use the upgrade command it prints
+ - **Check licenses and generate citations** - Query `license_short_name` and respect CC BY vs CC BY-NC terms; use `citations_from_selection()` to produce citations from `source_DOI` for publications
+ - **Explore small, then commit** - Use `LIMIT` (or a low `max_rows`) while exploring, and check collection size before downloading — some collections are terabytes. See `references/cli_guide.md`
+ - **Keep downloads reproducible** - Organize with `dirTemplate` (e.g. `%collection_id/%PatientID/%Modality`) and save the Series UIDs or manifest behind any dataset you build
- See the Quick Navigation section at the top for the full list of reference guides with decision triggers.
+ ## Troubleshooting
- - **[indices_reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html)** - External documentation for index tables (may be ahead of the installed version)
+ **Issue: `ModuleNotFoundError: No module named 'idc_index'`**
+ - **Cause:** idc-index package not installed
+ - **Solution:** If the task is read-only metadata, do not install it — use the REST API instead (*Data Access Options*). Otherwise run `scripts/check_version.py` and use the install command it prints, which targets the running interpreter and pins the vetted version. For data analysis also add pandas, numpy, and pydicom (tested with pandas>=1.5, numpy>=1.23, pydicom>=2.3)
- ### External Links
+ **Issue: Download fails with connection timeout**
+ - **Cause:** Network instability or large download size
+ - **Solution:** Download in smaller batches (10-20 series); see `references/cli_guide.md` for
+ `--use-s5cmd-sync` resume and retry guidance
- - **IDC Portal**: https://portal.imaging.datacommons.cancer.gov/explore/
- - **Documentation**: https://learn.canceridc.dev/
- - **Tutorials**: https://github.com/ImagingDataCommons/IDC-Tutorials
- - **User Forum**: https://discourse.canceridc.dev/
- - **idc-index GitHub**: https://github.com/ImagingDataCommons/idc-index
- - **Citation**: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180
+ **Issue: `BigQuery quota exceeded` or billing errors**
+ - **Cause:** BigQuery requires billing-enabled GCP project
+ - **Solution:** Use idc-index mini-index for simple queries (no billing required), or see `references/bigquery_guide.md` for cost optimization tips
- ### Skill Updates
+ **Issue: Series UID not found or no data returned**
+ - **Cause:** Typo in UID, data not in the current IDC version, or wrong field name
+ - **Solution:** Test with `LIMIT 5` first, check field names against `client.indices_overview`,
+ and confirm the series is in the current version (some old data is deprecated)
- This skill version is available in skill metadata. To check for updates:
- - Visit the [releases page](https://github.com/ImagingDataCommons/idc-claude-skill/releases)
- - Watch the repository on GitHub (Watch → Custom → Releases)
+ **Issue: Column not found in `index` table (e.g., `SliceThickness`, `PixelSpacing`, `KVP`, `EchoTime`, `InjectedDose`)**
+ - **Cause:** The `index` table contains series-level metadata only; modality-specific acquisition and reconstruction parameters live in dedicated tables (`ct_index`, `mr_index`, `pt_index`)
+ - **Solution:** Search `client.indices_overview` for the column to find its table — the loop is under *Finding which table contains a column* in `references/index_tables_guide.md` — then fetch and join on `SeriesInstanceUID`:
+ ```python
+ client.fetch_index("ct_index")
+ result = client.sql_query("""
+ SELECT i.SeriesInstanceUID, i.Modality, c.SliceThickness, c.KVP, c.PixelSpacing_row_mm
+ FROM index i
+ JOIN ct_index c USING (SeriesInstanceUID)
+ WHERE i.collection_id = 'your_collection'
+ """)
+ ```
+
+ **Issue: Downloaded DICOM files won't open**
+ - **Cause:** Corrupted download, or an object type the viewer does not handle — SEG, RTSTRUCT,
+ SR, and slide microscopy all need specialized tools
+ - **Solution:** Check `Modality` and `SOPClassUID` first, validate with
+ `pydicom.dcmread(file, force=True)`, try another viewer (3D Slicer, QuPath for pathology),
+ then re-download
+
+ ## Resources
+
+ Reference guides and their decision triggers are listed in *Quick Navigation* above.
+
+ - **IDC Portal**: https://portal.imaging.datacommons.cancer.gov/explore/
+ - **Documentation**: https://learn.canceridc.dev/ — **Tutorials**: https://github.com/ImagingDataCommons/IDC-Tutorials
+ - **User Forum**: https://discourse.canceridc.dev/ — **idc-index**: https://github.com/ImagingDataCommons/idc-index
+ - **[indices_reference](https://idc-index.readthedocs.io/en/latest/indices_reference.html)** — external index-table docs (may be ahead of the installed version)
+ - **Citation**: Fedorov, A., et al. "National Cancer Institute Imaging Data Commons: Toward Transparency, Reproducibility, and Scalability in Imaging Artificial Intelligence." RadioGraphics 43.12 (2023). https://doi.org/10.1148/rg.230180
+ - **Skill updates**: [releases page](https://github.com/ImagingDataCommons/imaging-data-commons-skill/releases); watch the repository (Watch → Custom → Releases)