IDC CGCI-HTMCP-CC: Cervical Squamous Cell Carcinoma whole-slide microscopy (211 patients): sample notebook
A worked notebook for IDC CGCI-HTMCP-CC: Cervical Squamous Cell Carcinoma whole-slide microscopy (211 patients), with 5 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- NCI Imaging Data Commons; CGCI-HTMCP-CC contributors; tables and sample by Scigantic
- License
- Per series (series.parquet license_short_name): CC BY 4.0 525. Cite the source DOI(s) and the IDC.
- Size
- 5 parquet tables plus 3 sample DICOM files (4 MB)
- Stored at
- Amazon S3, s3://scigantic-idc-open/cgci_htmcp_cc (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
- DOI
- 10.5281/zenodo.17381404
The notebook
Cervical cancer slides from IDC, joined to GDC clinical and RNA-seq data
The HIV+ Tumor Molecular Characterization Project cervical cancer cohort (HTMCP-CC) is split across two NCI repositories. The slides are on the Imaging Data Commons (IDC); the clinical records, RNA-seq and mutations are on the Genomic Data Commons (GDC). idc-index carries no clinical table for this collection, so the two halves have to be joined by hand.
Both halves are mounted here. ~/archive is the IDC collection index (one row per slide), and ~/additional-datasets holds the GDC project as parquet tables. The clinical and expression data are read from disk with no API calls, and only the slide thumbnails are fetched from IDC's public bucket.
Start here. Click the grey cell below and press Shift+Enter. It counts the patients who have both a slide and a clinical record. Then change STAIN and run it again.
STAIN = "P16 T" # slide stain to count; try "HE T", "P63 T" or "KI67 T"
import glob, io, os
import pandas as pd
import requests
def _find(base, names, must_have):
"""First directory under base whose name contains all of names and that holds must_have."""
for root in [base] + sorted(glob.glob(os.path.join(base, "*"))):
low = os.path.basename(root).lower()
if root != base and not all(n in low for n in names):
continue
for cand in (root, os.path.join(root, "cgci_htmcp_cc"), os.path.join(root, "CGCI-HTMCP-CC")):
if os.path.exists(os.path.join(cand, must_have)):
return cand
return None
IDC_ROOT = None
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
IDC_ROOT = IDC_ROOT or (_find(base, [], "series.parquet") if base and os.path.isdir(base) else None)
GDC_ROOT = None
for base in ("/mnt/additional-mounts", os.path.expanduser("~/additional-datasets")):
GDC_ROOT = GDC_ROOT or (_find(base, ["htmcp", "cc"], "clinical/cases.parquet") if os.path.isdir(base) else None)
def _read(root, url, name, **kw):
if root:
return pd.read_parquet(os.path.join(root, name), **kw)
r = requests.get(f"{url}/{name}", timeout=120) # same files, public mirror bucket
r.raise_for_status()
return pd.read_parquet(io.BytesIO(r.content), **kw)
def read_idc(name, **kw):
return _read(IDC_ROOT, "https://scigantic-idc-open.s3.amazonaws.com/cgci_htmcp_cc", name, **kw)
def read_gdc(name, **kw):
return _read(GDC_ROOT, "https://scigantic-gdc-open.s3.amazonaws.com/CGCI-HTMCP-CC", name, **kw)
series = read_idc("series.parquet")
cases = read_gdc("clinical/cases.parquet")
with_stain = set(series.loc[series.SeriesDescription == STAIN, "PatientID"])
both = with_stain & set(cases.submitter_id)
print(f"{series.PatientID.nunique()} imaged patients, {len(cases)} GDC cases")
print(f"{len(with_stain)} patients have a {STAIN!r} slide; {len(both)} of them have a GDC clinical record")
print("stains available:", ", ".join(series.SeriesDescription.value_counts().index[:8]))
1. Read both archives
The two readers below look for the mounts first and fall back to the same files on the public mirror buckets (s3://scigantic-idc-open, s3://scigantic-gdc-open) when a mount is missing, so this notebook also runs outside Scigantic. The first cell defines the same readers; they are repeated here so that this section runs on its own.
GDC records several diagnoses for some cases (a prior cancer at another site, a recurrence). diagnosis_is_primary_disease marks the one this project is about.
import glob, io, os
import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt
plt.rcParams["figure.dpi"] = 72
try:
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats("jpeg") # slide thumbnails are photographs; JPEG keeps the notebook small
except ImportError:
pass
def _find(base, names, must_have):
"""First directory under base whose name contains all of names and that holds must_have."""
for root in [base] + sorted(glob.glob(os.path.join(base, "*"))):
low = os.path.basename(root).lower()
if root != base and not all(n in low for n in names):
continue
for cand in (root, os.path.join(root, "cgci_htmcp_cc"), os.path.join(root, "CGCI-HTMCP-CC")):
if os.path.exists(os.path.join(cand, must_have)):
return cand
return None
IDC_ROOT = None
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
IDC_ROOT = IDC_ROOT or (_find(base, [], "series.parquet") if base and os.path.isdir(base) else None)
GDC_ROOT = None
for base in ("/mnt/additional-mounts", os.path.expanduser("~/additional-datasets")):
GDC_ROOT = GDC_ROOT or (_find(base, ["htmcp", "cc"], "clinical/cases.parquet") if os.path.isdir(base) else None)
def _read(root, url, name, **kw):
if root:
return pd.read_parquet(os.path.join(root, name), **kw)
r = requests.get(f"{url}/{name}", timeout=120) # same files, public mirror bucket
r.raise_for_status()
return pd.read_parquet(io.BytesIO(r.content), **kw)
def read_idc(name, **kw):
return _read(IDC_ROOT, "https://scigantic-idc-open.s3.amazonaws.com/cgci_htmcp_cc", name, **kw)
def read_gdc(name, **kw):
return _read(GDC_ROOT, "https://scigantic-gdc-open.s3.amazonaws.com/CGCI-HTMCP-CC", name, **kw)
print("IDC index:", IDC_ROOT or "public mirror over HTTPS")
print("GDC tables:", GDC_ROOT or "public mirror over HTTPS")
series = read_idc("series.parquet")
cases = read_gdc("clinical/cases.parquet")
dx = read_gdc("clinical/diagnoses.parquet")
dx = dx[dx.diagnosis_is_primary_disease.astype(str).str.lower() == "true"]
print(f"IDC: {len(series)} slides from {series.PatientID.nunique()} patients")
print(f"GDC: {len(cases)} cases, {len(dx)} primary diagnoses")
Output:
IDC index: /mnt/archive GDC tables: /mnt/additional-mounts/s3-scigantic-gdc-open-cgci-htmcp-cc IDC: 525 slides from 211 patients GDC: 212 cases, 212 primary diagnoses
2. Join slides to clinical records
In this collection the IDC PatientID is the GDC case submitter_id, so the join is a plain merge. The IDC side is first collapsed to one row per patient listing the stains of that patient's slides (SeriesDescription: HE T is hematoxylin and eosin, the others are immunohistochemistry). FIGO stages are grouped to their Roman numeral for the table.
per_patient = (series.groupby("PatientID")
.agg(slides=("SeriesInstanceUID", "nunique"),
stains=("SeriesDescription", lambda s: sorted(set(s))))
.reset_index())
clin = (cases[["submitter_id", "demographic_vital_status", "demographic_age_at_index"]]
.merge(dx[["case_submitter_id", "primary_diagnosis", "figo_stage", "age_at_diagnosis"]],
left_on="submitter_id", right_on="case_submitter_id", how="left"))
joined = per_patient.merge(clin, left_on="PatientID", right_on="submitter_id", how="inner")
print(f"{len(joined)} of {len(per_patient)} imaged patients matched a GDC case")
top = joined.primary_diagnosis.value_counts().index[:3]
joined["histology"] = joined.primary_diagnosis.where(joined.primary_diagnosis.isin(top), "Other")
joined["stage_group"] = joined.figo_stage.str.extract(r"Stage (IV|III|II|I)")[0].fillna("not reported")
pd.crosstab(joined.histology, joined.stage_group, margins=True)
Output:
211 of 211 imaged patients matched a GDC case stage_group I II III IV All histology Adenocarcinoma, NOS 3 9 6 1 19 Other 2 10 1 2 15 Squamous cell carcinoma, keratinizing, NOS 8 16 22 2 48 Squamous cell carcinoma, nonkeratinizing, NOS 21 56 46 6 129 All 34 91 75 11 211
3. p16 gene expression by histologic type
p16 is the protein encoded by CDKN2A. Pathologists stain for it because high-risk HPV infection drives it up, so the p16 slides in section 4 and the RNA-seq here measure the same thing in two ways. The GDC tables hold STAR gene-level TPM for every tumor aliquot in one parquet file (genes as rows, aliquots as columns), and rna_seq/samples.parquet maps each column to its case. A few cases have more than one aliquot; their values are averaged.
genes = read_gdc("rna_seq/genes.parquet")
gene_id = genes.loc[genes.gene_name == "CDKN2A", "gene_id"].iloc[0]
tpm = read_gdc("rna_seq/tpm_unstranded.parquet", filters=[("gene_id", "==", gene_id)])
rna = read_gdc("rna_seq/samples.parquet")
expr = (tpm.drop(columns="gene_id").T.rename(columns={0: "tpm"})
.join(rna.set_index("column")["case_submitter_id"])
.groupby("case_submitter_id").tpm.mean())
joined["CDKN2A_tpm"] = joined.PatientID.map(expr)
has = joined.dropna(subset=["CDKN2A_tpm"])
print(f"CDKN2A ({gene_id}) TPM for {len(has)} of {len(joined)} joined patients")
print(has.groupby("histology").CDKN2A_tpm.describe()[["count", "50%", "min", "max"]].round(1).to_string())
order = list(top) + (["Other"] if (has.histology == "Other").any() else [])
fig, ax = plt.subplots(figsize=(8, 3.6))
for i, h in enumerate(order):
v = np.log2(has.loc[has.histology == h, "CDKN2A_tpm"] + 1)
ax.scatter(np.random.default_rng(0).normal(i, 0.07, len(v)), v, s=12, alpha=0.6, color="#2b6cb0")
ax.hlines(v.median(), i - 0.3, i + 0.3, color="black")
ax.set_xticks(range(len(order)), [h.replace(", NOS", "").replace("Squamous cell carcinoma", "SCC") for h in order])
ax.set_ylabel("CDKN2A log2(TPM + 1)")
ax.set_title("p16 (CDKN2A) expression by histologic type, one dot per patient")
plt.tight_layout(); plt.show()
Output:
CDKN2A (ENSG00000147889.18) TPM for 123 of 211 joined patients
count 50% min max
histology
Adenocarcinoma, NOS 10.0 92.1 14.3 166.7
Other 8.0 169.7 1.4 293.7
Squamous cell carcinoma, keratinizing, NOS 33.0 150.5 51.5 305.3
Squamous cell carcinoma, nonkeratinizing, NOS 72.0 123.8 1.3 395.1
4. The slides next to the record
Each slide in IDC is a DICOM series with one instance per resolution level, and every slide in this collection has a small uncompressed THUMBNAIL instance. The IDC index on the mount gives each slide's bucket and crdc_series_uuid; listing that prefix in s3://idc-open-data returns the instances, and reading the first few kilobytes of each is enough to find the thumbnail from its ImageType. Only the thumbnails are downloaded, a few MB in all, without credentials.
The patients shown are the first (by PatientID) of each of the three most common histologic types who have an H&E slide, a p16 slide, an age at diagnosis and an RNA-seq value.
import struct
import xml.etree.ElementTree as ET
import pydicom
S3 = "https://{bucket}.s3.amazonaws.com"
def instance_keys(bucket, uuid):
r = requests.get(S3.format(bucket=bucket), params={"list-type": "2", "prefix": uuid + "/"}, timeout=60)
r.raise_for_status()
ns = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"}
return [k.text for k in ET.fromstring(r.content).findall(".//s3:Key", ns) if k.text.endswith(".dcm")]
def image_type(bucket, key):
head = requests.get(f"{S3.format(bucket=bucket)}/{key}", headers={"Range": "bytes=0-8191"}, timeout=60).content
i = head.find(b"\x08\x00\x08\x00CS") # (0008,0008) ImageType, explicit VR little endian
if i < 0:
return ""
n = struct.unpack("<H", head[i + 6:i + 8])[0]
return head[i + 8:i + 8 + n].decode("ascii", "replace")
def thumbnail(patient, stain):
s = series[(series.PatientID == patient) & (series.SeriesDescription == stain)].iloc[0]
for key in instance_keys(s.aws_bucket, s.crdc_series_uuid):
if "THUMBNAIL" in image_type(s.aws_bucket, key):
ds = pydicom.dcmread(io.BytesIO(requests.get(f"{S3.format(bucket=s.aws_bucket)}/{key}", timeout=120).content))
img = ds.pixel_array
step = max(1, int(np.ceil(max(img.shape[:2]) / 600)))
return img[::step, ::step]
raise LookupError(f"no THUMBNAIL instance in {s.series_aws_url}")
ok = has[has.stains.apply(lambda s: "HE T" in s and "P16 T" in s) & has.age_at_diagnosis.notna()]
examples = ok[ok.histology.isin(top)].sort_values("PatientID").groupby("histology", sort=False).head(1)
examples = examples.set_index("histology").loc[[h for h in top if h in set(examples.histology)]].reset_index()
fig, axes = plt.subplots(2, len(examples), figsize=(4.6 * len(examples), 8.4))
for col, ex in enumerate(examples.itertuples()):
for row, (stain, label) in enumerate([("HE T", "H&E"), ("P16 T", "p16 IHC")]):
ax = axes[row, col]
ax.imshow(thumbnail(ex.PatientID, stain))
ax.set_xticks([]); ax.set_yticks([])
if col == 0:
ax.set_ylabel(label, fontsize=13)
age = int(ex.age_at_diagnosis // 365.25)
axes[0, col].set_title(f"{ex.PatientID}\n{ex.primary_diagnosis.replace(', NOS', '')}\n"
f"{ex.figo_stage}, age {age}, {ex.demographic_vital_status.lower()}\n"
f"CDKN2A {ex.CDKN2A_tpm:.0f} TPM", fontsize=10)
plt.tight_layout(); plt.show()
5. Where to go next
The GDC mount holds more than clinical data: somatic_mutations/masked_somatic_mutations.parquet, copy_number/, mirna/ and the full rna_seq/ tables, all keyed by the same case ids. clinical/follow_ups.parquet and clinical/treatments.parquet add outcomes and therapy.
The same join works for the other IDC collections whose clinical data is only in GDC, among them cgci_htmcp_dlbcl, cgci_htmcp_lc, cgci_blgsp and hcmi_cmdc. Each is its own archive on Scigantic, under the same names.
The chat panel is prefilled with the first of these:
- *Compare CDKN2A expression between patients who are alive and dead within each FIGO stage group, and say whether the difference holds after accounting for stage.*
- *Find the patients whose p16 slide and CDKN2A expression disagree, and show their thumbnails.*
- *Join somatic_mutations to the histology table and list the genes mutated more often in adenocarcinoma than in squamous cell carcinoma.*
The slides are CC BY 4.0 (license_short_name in series.parquet). When you publish, cite the collection DOI (source_DOI), the IDC paper (Fedorov et al., RadioGraphics 2023) and the GDC (Heath et al., Nature Genetics 2021).
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.