IDC CGCI-HTMCP-CC: Cervical Squamous Cell Carcinoma whole-slide microscopy (211 patients): sample notebook

A worked notebook for IDC CGCI-HTMCP-CC: Cervical Squamous Cell Carcinoma whole-slide microscopy (211 patients), with 5 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
NCI Imaging Data Commons; CGCI-HTMCP-CC contributors; tables and sample by Scigantic
License
Per series (series.parquet license_short_name): CC BY 4.0 525. Cite the source DOI(s) and the IDC.
Size
5 parquet tables plus 3 sample DICOM files (4 MB)
Stored at
Amazon S3, s3://scigantic-idc-open/cgci_htmcp_cc (us-east-1)
Access
Free and public. No account with the data provider is needed.
DOI
10.5281/zenodo.17381404

The notebook

Cervical cancer slides from IDC, joined to GDC clinical and RNA-seq data

The HIV+ Tumor Molecular Characterization Project cervical cancer cohort (HTMCP-CC) is split across two NCI repositories. The slides are on the Imaging Data Commons (IDC); the clinical records, RNA-seq and mutations are on the Genomic Data Commons (GDC). idc-index carries no clinical table for this collection, so the two halves have to be joined by hand.

Both halves are mounted here. ~/archive is the IDC collection index (one row per slide), and ~/additional-datasets holds the GDC project as parquet tables. The clinical and expression data are read from disk with no API calls, and only the slide thumbnails are fetched from IDC's public bucket.

Start here. Click the grey cell below and press Shift+Enter. It counts the patients who have both a slide and a clinical record. Then change STAIN and run it again.

STAIN = "P16 T"  # slide stain to count; try "HE T", "P63 T" or "KI67 T"

import glob, io, os
import pandas as pd
import requests

def _find(base, names, must_have):
    """First directory under base whose name contains all of names and that holds must_have."""
    for root in [base] + sorted(glob.glob(os.path.join(base, "*"))):
        low = os.path.basename(root).lower()
        if root != base and not all(n in low for n in names):
            continue
        for cand in (root, os.path.join(root, "cgci_htmcp_cc"), os.path.join(root, "CGCI-HTMCP-CC")):
            if os.path.exists(os.path.join(cand, must_have)):
                return cand
    return None

IDC_ROOT = None
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
    IDC_ROOT = IDC_ROOT or (_find(base, [], "series.parquet") if base and os.path.isdir(base) else None)
GDC_ROOT = None
for base in ("/mnt/additional-mounts", os.path.expanduser("~/additional-datasets")):
    GDC_ROOT = GDC_ROOT or (_find(base, ["htmcp", "cc"], "clinical/cases.parquet") if os.path.isdir(base) else None)

def _read(root, url, name, **kw):
    if root:
        return pd.read_parquet(os.path.join(root, name), **kw)
    r = requests.get(f"{url}/{name}", timeout=120)  # same files, public mirror bucket
    r.raise_for_status()
    return pd.read_parquet(io.BytesIO(r.content), **kw)

def read_idc(name, **kw):
    return _read(IDC_ROOT, "https://scigantic-idc-open.s3.amazonaws.com/cgci_htmcp_cc", name, **kw)

def read_gdc(name, **kw):
    return _read(GDC_ROOT, "https://scigantic-gdc-open.s3.amazonaws.com/CGCI-HTMCP-CC", name, **kw)

series = read_idc("series.parquet")
cases = read_gdc("clinical/cases.parquet")
with_stain = set(series.loc[series.SeriesDescription == STAIN, "PatientID"])
both = with_stain & set(cases.submitter_id)
print(f"{series.PatientID.nunique()} imaged patients, {len(cases)} GDC cases")
print(f"{len(with_stain)} patients have a {STAIN!r} slide; {len(both)} of them have a GDC clinical record")
print("stains available:", ", ".join(series.SeriesDescription.value_counts().index[:8]))

1. Read both archives

The two readers below look for the mounts first and fall back to the same files on the public mirror buckets (s3://scigantic-idc-open, s3://scigantic-gdc-open) when a mount is missing, so this notebook also runs outside Scigantic. The first cell defines the same readers; they are repeated here so that this section runs on its own.

GDC records several diagnoses for some cases (a prior cancer at another site, a recurrence). diagnosis_is_primary_disease marks the one this project is about.

import glob, io, os
import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt
plt.rcParams["figure.dpi"] = 72
try:
    from matplotlib_inline.backend_inline import set_matplotlib_formats
    set_matplotlib_formats("jpeg")  # slide thumbnails are photographs; JPEG keeps the notebook small
except ImportError:
    pass

def _find(base, names, must_have):
    """First directory under base whose name contains all of names and that holds must_have."""
    for root in [base] + sorted(glob.glob(os.path.join(base, "*"))):
        low = os.path.basename(root).lower()
        if root != base and not all(n in low for n in names):
            continue
        for cand in (root, os.path.join(root, "cgci_htmcp_cc"), os.path.join(root, "CGCI-HTMCP-CC")):
            if os.path.exists(os.path.join(cand, must_have)):
                return cand
    return None

IDC_ROOT = None
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
    IDC_ROOT = IDC_ROOT or (_find(base, [], "series.parquet") if base and os.path.isdir(base) else None)
GDC_ROOT = None
for base in ("/mnt/additional-mounts", os.path.expanduser("~/additional-datasets")):
    GDC_ROOT = GDC_ROOT or (_find(base, ["htmcp", "cc"], "clinical/cases.parquet") if os.path.isdir(base) else None)

def _read(root, url, name, **kw):
    if root:
        return pd.read_parquet(os.path.join(root, name), **kw)
    r = requests.get(f"{url}/{name}", timeout=120)  # same files, public mirror bucket
    r.raise_for_status()
    return pd.read_parquet(io.BytesIO(r.content), **kw)

def read_idc(name, **kw):
    return _read(IDC_ROOT, "https://scigantic-idc-open.s3.amazonaws.com/cgci_htmcp_cc", name, **kw)

def read_gdc(name, **kw):
    return _read(GDC_ROOT, "https://scigantic-gdc-open.s3.amazonaws.com/CGCI-HTMCP-CC", name, **kw)

print("IDC index:", IDC_ROOT or "public mirror over HTTPS")
print("GDC tables:", GDC_ROOT or "public mirror over HTTPS")

series = read_idc("series.parquet")
cases = read_gdc("clinical/cases.parquet")
dx = read_gdc("clinical/diagnoses.parquet")
dx = dx[dx.diagnosis_is_primary_disease.astype(str).str.lower() == "true"]
print(f"IDC: {len(series)} slides from {series.PatientID.nunique()} patients")
print(f"GDC: {len(cases)} cases, {len(dx)} primary diagnoses")

Output:

IDC index: /mnt/archive
GDC tables: /mnt/additional-mounts/s3-scigantic-gdc-open-cgci-htmcp-cc
IDC: 525 slides from 211 patients
GDC: 212 cases, 212 primary diagnoses

2. Join slides to clinical records

In this collection the IDC PatientID is the GDC case submitter_id, so the join is a plain merge. The IDC side is first collapsed to one row per patient listing the stains of that patient's slides (SeriesDescription: HE T is hematoxylin and eosin, the others are immunohistochemistry). FIGO stages are grouped to their Roman numeral for the table.

per_patient = (series.groupby("PatientID")
               .agg(slides=("SeriesInstanceUID", "nunique"),
                    stains=("SeriesDescription", lambda s: sorted(set(s))))
               .reset_index())
clin = (cases[["submitter_id", "demographic_vital_status", "demographic_age_at_index"]]
        .merge(dx[["case_submitter_id", "primary_diagnosis", "figo_stage", "age_at_diagnosis"]],
               left_on="submitter_id", right_on="case_submitter_id", how="left"))
joined = per_patient.merge(clin, left_on="PatientID", right_on="submitter_id", how="inner")
print(f"{len(joined)} of {len(per_patient)} imaged patients matched a GDC case")

top = joined.primary_diagnosis.value_counts().index[:3]
joined["histology"] = joined.primary_diagnosis.where(joined.primary_diagnosis.isin(top), "Other")
joined["stage_group"] = joined.figo_stage.str.extract(r"Stage (IV|III|II|I)")[0].fillna("not reported")
pd.crosstab(joined.histology, joined.stage_group, margins=True)

Output:

211 of 211 imaged patients matched a GDC case
stage_group                                     I  II  III  IV  All
histology                                                          
Adenocarcinoma, NOS                             3   9    6   1   19
Other                                           2  10    1   2   15
Squamous cell carcinoma, keratinizing, NOS      8  16   22   2   48
Squamous cell carcinoma, nonkeratinizing, NOS  21  56   46   6  129
All                                            34  91   75  11  211

3. p16 gene expression by histologic type

p16 is the protein encoded by CDKN2A. Pathologists stain for it because high-risk HPV infection drives it up, so the p16 slides in section 4 and the RNA-seq here measure the same thing in two ways. The GDC tables hold STAR gene-level TPM for every tumor aliquot in one parquet file (genes as rows, aliquots as columns), and rna_seq/samples.parquet maps each column to its case. A few cases have more than one aliquot; their values are averaged.

genes = read_gdc("rna_seq/genes.parquet")
gene_id = genes.loc[genes.gene_name == "CDKN2A", "gene_id"].iloc[0]
tpm = read_gdc("rna_seq/tpm_unstranded.parquet", filters=[("gene_id", "==", gene_id)])
rna = read_gdc("rna_seq/samples.parquet")
expr = (tpm.drop(columns="gene_id").T.rename(columns={0: "tpm"})
        .join(rna.set_index("column")["case_submitter_id"])
        .groupby("case_submitter_id").tpm.mean())
joined["CDKN2A_tpm"] = joined.PatientID.map(expr)
has = joined.dropna(subset=["CDKN2A_tpm"])
print(f"CDKN2A ({gene_id}) TPM for {len(has)} of {len(joined)} joined patients")
print(has.groupby("histology").CDKN2A_tpm.describe()[["count", "50%", "min", "max"]].round(1).to_string())

order = list(top) + (["Other"] if (has.histology == "Other").any() else [])
fig, ax = plt.subplots(figsize=(8, 3.6))
for i, h in enumerate(order):
    v = np.log2(has.loc[has.histology == h, "CDKN2A_tpm"] + 1)
    ax.scatter(np.random.default_rng(0).normal(i, 0.07, len(v)), v, s=12, alpha=0.6, color="#2b6cb0")
    ax.hlines(v.median(), i - 0.3, i + 0.3, color="black")
ax.set_xticks(range(len(order)), [h.replace(", NOS", "").replace("Squamous cell carcinoma", "SCC") for h in order])
ax.set_ylabel("CDKN2A log2(TPM + 1)")
ax.set_title("p16 (CDKN2A) expression by histologic type, one dot per patient")
plt.tight_layout(); plt.show()

Output:

CDKN2A (ENSG00000147889.18) TPM for 123 of 211 joined patients
                                               count    50%   min    max
histology                                                               
Adenocarcinoma, NOS                             10.0   92.1  14.3  166.7
Other                                            8.0  169.7   1.4  293.7
Squamous cell carcinoma, keratinizing, NOS      33.0  150.5  51.5  305.3
Squamous cell carcinoma, nonkeratinizing, NOS   72.0  123.8   1.3  395.1

4. The slides next to the record

Each slide in IDC is a DICOM series with one instance per resolution level, and every slide in this collection has a small uncompressed THUMBNAIL instance. The IDC index on the mount gives each slide's bucket and crdc_series_uuid; listing that prefix in s3://idc-open-data returns the instances, and reading the first few kilobytes of each is enough to find the thumbnail from its ImageType. Only the thumbnails are downloaded, a few MB in all, without credentials.

The patients shown are the first (by PatientID) of each of the three most common histologic types who have an H&E slide, a p16 slide, an age at diagnosis and an RNA-seq value.

import struct
import xml.etree.ElementTree as ET
import pydicom

S3 = "https://{bucket}.s3.amazonaws.com"

def instance_keys(bucket, uuid):
    r = requests.get(S3.format(bucket=bucket), params={"list-type": "2", "prefix": uuid + "/"}, timeout=60)
    r.raise_for_status()
    ns = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"}
    return [k.text for k in ET.fromstring(r.content).findall(".//s3:Key", ns) if k.text.endswith(".dcm")]

def image_type(bucket, key):
    head = requests.get(f"{S3.format(bucket=bucket)}/{key}", headers={"Range": "bytes=0-8191"}, timeout=60).content
    i = head.find(b"\x08\x00\x08\x00CS")  # (0008,0008) ImageType, explicit VR little endian
    if i < 0:
        return ""
    n = struct.unpack("<H", head[i + 6:i + 8])[0]
    return head[i + 8:i + 8 + n].decode("ascii", "replace")

def thumbnail(patient, stain):
    s = series[(series.PatientID == patient) & (series.SeriesDescription == stain)].iloc[0]
    for key in instance_keys(s.aws_bucket, s.crdc_series_uuid):
        if "THUMBNAIL" in image_type(s.aws_bucket, key):
            ds = pydicom.dcmread(io.BytesIO(requests.get(f"{S3.format(bucket=s.aws_bucket)}/{key}", timeout=120).content))
            img = ds.pixel_array
            step = max(1, int(np.ceil(max(img.shape[:2]) / 600)))
            return img[::step, ::step]
    raise LookupError(f"no THUMBNAIL instance in {s.series_aws_url}")

ok = has[has.stains.apply(lambda s: "HE T" in s and "P16 T" in s) & has.age_at_diagnosis.notna()]
examples = ok[ok.histology.isin(top)].sort_values("PatientID").groupby("histology", sort=False).head(1)
examples = examples.set_index("histology").loc[[h for h in top if h in set(examples.histology)]].reset_index()

fig, axes = plt.subplots(2, len(examples), figsize=(4.6 * len(examples), 8.4))
for col, ex in enumerate(examples.itertuples()):
    for row, (stain, label) in enumerate([("HE T", "H&E"), ("P16 T", "p16 IHC")]):
        ax = axes[row, col]
        ax.imshow(thumbnail(ex.PatientID, stain))
        ax.set_xticks([]); ax.set_yticks([])
        if col == 0:
            ax.set_ylabel(label, fontsize=13)
    age = int(ex.age_at_diagnosis // 365.25)
    axes[0, col].set_title(f"{ex.PatientID}\n{ex.primary_diagnosis.replace(', NOS', '')}\n"
                           f"{ex.figo_stage}, age {age}, {ex.demographic_vital_status.lower()}\n"
                           f"CDKN2A {ex.CDKN2A_tpm:.0f} TPM", fontsize=10)
plt.tight_layout(); plt.show()

5. Where to go next

The GDC mount holds more than clinical data: somatic_mutations/masked_somatic_mutations.parquet, copy_number/, mirna/ and the full rna_seq/ tables, all keyed by the same case ids. clinical/follow_ups.parquet and clinical/treatments.parquet add outcomes and therapy.

The same join works for the other IDC collections whose clinical data is only in GDC, among them cgci_htmcp_dlbcl, cgci_htmcp_lc, cgci_blgsp and hcmi_cmdc. Each is its own archive on Scigantic, under the same names.

The chat panel is prefilled with the first of these:

The slides are CC BY 4.0 (license_short_name in series.parquet). When you publish, cite the collection DOI (source_DOI), the IDC paper (Fedorov et al., RadioGraphics 2023) and the GDC (Heath et al., Nature Genetics 2021).

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.