IDC CMMD: Breast Cancer mammography (1,775 patients): sample notebook

A worked notebook for IDC CMMD: Breast Cancer mammography (1,775 patients), with 5 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
NCI Imaging Data Commons; CMMD contributors; tables and sample by Scigantic
License
Per series (series.parquet license_short_name): CC BY 4.0 1,775. Cite the source DOI(s) and the IDC.
Size
6 parquet tables plus 4 sample DICOM files (18 MB)
Stored at
Amazon S3, s3://scigantic-idc-open/cmmd (us-east-1)
Access
Free and public. No account with the data provider is needed.
DOI
10.7937/tcia.eqde-4b16

The notebook

Chinese Mammography Database (CMMD): every clinical record, then the images behind it

CMMD holds the mammograms of 1,775 patients (5,202 images in the IDC index) with biopsy-confirmed breast lesions, plus a clinical table with the side of the lesion, its type (mass, calcification or both), benign or malignant, age, and for 749 patients the molecular subtype. The images are on the NCI Imaging Data Commons (IDC); the clinical table comes from The Cancer Imaging Archive (TCIA).

Both are mounted here at ~/archive: the series index and the clinical table as parquet. That means the whole-cohort analysis below reads all 1,872 clinical rows from disk with no download, and only the four example patients' images are fetched from IDC's public bucket.

Start here. Click the grey cell below and press Shift+Enter. It counts lesions by subtype from the mounted clinical table. Then change SUBTYPE and run it again.

SUBTYPE = "triple negative"  # try "Luminal A", "Luminal B" or "HER2-enriched"

import io, os
import pandas as pd
import requests

def _root():
    for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
        for cand in (base, os.path.join(base, "cmmd")):
            if cand and os.path.exists(os.path.join(cand, "series.parquet")):
                return cand
    return None

ROOT = _root()

def read(name):
    if ROOT:
        return pd.read_parquet(os.path.join(ROOT, name))
    r = requests.get(f"https://scigantic-idc-open.s3.amazonaws.com/cmmd/{name}", timeout=120)  # same files, public mirror
    r.raise_for_status()
    return pd.read_parquet(io.BytesIO(r.content))

clin = read("clinical/cmmd_clinical.parquet")
rows = clin[clin.subtype == SUBTYPE]
print(f"{len(clin):,} lesions from {clin.dicom_patient_id.nunique():,} patients; {len(rows)} are {SUBTYPE!r}")
print("lesion type:", rows.abnormality.value_counts().to_dict())
print(f"age: median {rows.age.median():.0f}, range {rows.age.min()}-{rows.age.max()}")

Output:

1,872 lesions from 1,775 patients; 86 are 'triple negative'
lesion type: {'mass': 65, 'both': 14, 'calcification': 7}
age: median 49, range 28-85

1. Read the mounted tables

series.parquet is the IDC index for the collection: one row per patient's mammography series, with its public S3 location. clinical/cmmd_clinical.parquet is TCIA's clinical spreadsheet with one row per affected breast, so a patient with lesions in both breasts has two rows. The reader falls back to the same files on the public mirror bucket when the mount is missing, so this notebook also runs outside Scigantic.

import io, os
import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt
plt.rcParams["figure.dpi"] = 72
try:
    from matplotlib_inline.backend_inline import set_matplotlib_formats
    set_matplotlib_formats("jpeg")  # mammograms are photographic; JPEG keeps the notebook small
except ImportError:
    pass

def _root():
    for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
        for cand in (base, os.path.join(base, "cmmd")):
            if cand and os.path.exists(os.path.join(cand, "series.parquet")):
                return cand
    return None

ROOT = _root()

def read(name):
    if ROOT:
        return pd.read_parquet(os.path.join(ROOT, name))
    r = requests.get(f"https://scigantic-idc-open.s3.amazonaws.com/cmmd/{name}", timeout=120)  # same files, public mirror
    r.raise_for_status()
    return pd.read_parquet(io.BytesIO(r.content))

print("tables from:", ROOT or "public mirror over HTTPS")
series = read("series.parquet")
clin = read("clinical/cmmd_clinical.parquet")
clin["subtype"] = clin.subtype.replace("", np.nan)
print(f"{len(series):,} series, {series.instanceCount.sum():,} images, {series.series_size_MB.sum() / 1024:.1f} GB on IDC")
print(f"{len(clin):,} clinical rows for {clin.dicom_patient_id.nunique():,} patients; "
      f"{clin.dicom_patient_id.duplicated(keep=False).sum()} rows belong to patients with lesions in both breasts")
missing = set(clin.dicom_patient_id) - set(series.PatientID)
print("clinical patients with no image series:", len(missing))

Output:

tables from: public mirror over HTTPS
1,775 series, 5,202 images, 22.3 GB on IDC
1,872 clinical rows for 1,775 patients; 194 rows belong to patients with lesions in both breasts
clinical patients with no image series: 0

2. The whole cohort, with no image download

Because every clinical row is on disk, cohort questions take a line each. Malignant lesions outnumber benign ones for every lesion type; the benign share is highest for a mass alone (36%) and lowest when a mass and calcifications occur together (17%). The molecular subtype is recorded for 749 lesions, all from D2 patients and all malignant.

print(pd.crosstab(clin.abnormality, clin.classification, margins=True).to_string(), "\n")
sub = clin.dropna(subset=["subtype"])
print(sub.groupby("subtype").agg(lesions=("age", "size"), median_age=("age", "median"),
                                 mass_share=("abnormality", lambda a: round((a != "calcification").mean(), 2))).to_string())

order = ["Luminal A", "Luminal B", "HER2-enriched", "triple negative"]
fig, axes = plt.subplots(1, 2, figsize=(11, 3.6))
for i, (label, ages) in enumerate([("Benign", clin.loc[clin.classification == "Benign", "age"]),
                                   ("Malignant", clin.loc[clin.classification == "Malignant", "age"])]):
    axes[0].hist(ages, bins=range(15, 95, 5), alpha=0.6, label=f"{label} (n={len(ages)})", color=["#7fa7d9", "#1f4e8c"][i])
axes[0].set_xlabel("age"); axes[0].set_ylabel("lesions"); axes[0].legend(); axes[0].set_title("Age by diagnosis, all 1,872 lesions")
axes[1].boxplot([sub.loc[sub.subtype == s, "age"] for s in order], showfliers=False)
axes[1].set_xticks(range(1, 5), [s.replace("triple negative", "triple\nnegative").replace("HER2-enriched", "HER2-\nenriched") for s in order])
axes[1].set_ylabel("age"); axes[1].set_title("Age by molecular subtype (D2 patients)")
plt.tight_layout(); plt.show()

Output:

classification  Benign  Malignant   All
abnormality                            
both                80        381   461
calcification       64        198   262
mass               412        737  1149
All                556       1316  1872 

                 lesions  median_age  mass_share
subtype                                         
HER2-enriched        135        50.0        0.77
Luminal A            152        50.0        0.88
Luminal B            376        48.0        0.89
triple negative       86        49.0        0.92

3. The images behind four records

Each series row gives the bucket and crdc_series_uuid; listing that prefix in s3://idc-open-data returns the DICOM files, which are uncompressed and about 4 MB each. We take the first patient (by id) of each molecular subtype who has a single clinical row and a mass, download their images anonymously, and join on patient and breast side using the ImageLaterality tag, because the clinical row describes one breast only.

import xml.etree.ElementTree as ET
import pydicom

def series_files(row):
    base = f"https://{row.aws_bucket}.s3.amazonaws.com"
    xml = requests.get(base, params={"list-type": "2", "prefix": row.crdc_series_uuid + "/"}, timeout=60).content
    ns = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"}
    keys = [k.text for k in ET.fromstring(xml).findall(".//s3:Key", ns) if k.text.endswith(".dcm")]
    return [pydicom.dcmread(io.BytesIO(requests.get(f"{base}/{k}", timeout=120).content)) for k in keys]

single = clin[~clin.dicom_patient_id.duplicated(keep=False) & (clin.abnormality == "mass")]
examples = single.dropna(subset=["subtype"]).sort_values("dicom_patient_id").groupby("subtype").head(1)
examples = examples.set_index("subtype").loc[order].reset_index()

images = []
for ex in examples.itertuples():
    for ds in series_files(series.set_index("PatientID").loc[ex.dicom_patient_id]):
        view = ds.ViewCodeSequence[0].CodeMeaning if "ViewCodeSequence" in ds else ""
        view = "CC" if "cranio-caudal" in view.lower() else "MLO" if "oblique" in view.lower() else view
        images.append({"PatientID": ex.dicom_patient_id, "side": ds.get("ImageLaterality", ""), "view": view, "ds": ds})
imgs = pd.DataFrame(images)
joined = imgs.merge(clin, left_on=["PatientID", "side"], right_on=["dicom_patient_id", "leftright"], how="left")
print(joined[["PatientID", "side", "view", "subtype", "abnormality", "classification", "age"]].to_string(index=False))

Output:

PatientID side view         subtype abnormality classification  age
  D2-0337    L   CC             NaN         NaN            NaN  NaN
  D2-0337    R   CC       Luminal A        mass      Malignant 54.0
  D2-0337    L  MLO             NaN         NaN            NaN  NaN
  D2-0337    R  MLO       Luminal A        mass      Malignant 54.0
  D2-0333    L   CC             NaN         NaN            NaN  NaN
  D2-0333    L  MLO             NaN         NaN            NaN  NaN
  D2-0333    R   CC       Luminal B        mass      Malignant 85.0
  D2-0333    R  MLO       Luminal B        mass      Malignant 85.0
  D2-0342    L   CC             NaN         NaN            NaN  NaN
  D2-0342    L  MLO             NaN         NaN            NaN  NaN
  D2-0342    R   CC   HER2-enriched        mass      Malignant 52.0
  D2-0342    R  MLO   HER2-enriched        mass      Malignant 52.0
  D2-0344    R  MLO triple negative        mass      Malignant 64.0
  D2-0344    R   CC triple negative        mass      Malignant 64.0
...
pids = list(examples.dicom_patient_id)
fig, axes = plt.subplots(len(pids), 4, figsize=(12, 3.6 * len(pids)))
for r, pid in enumerate(pids):
    row = joined[joined.PatientID == pid].sort_values(["side", "view"], key=lambda c: c.map({"R": 0, "L": 1, "CC": 0, "MLO": 1}))
    for c, x in enumerate(row.itertuples()):
        px = x.ds.pixel_array
        step = max(1, int(np.ceil(max(px.shape) / 500)))
        ax = axes[r, c]; ax.imshow(px[::step, ::step], cmap="gray"); ax.set_xticks([]); ax.set_yticks([])
        labeled = isinstance(x.subtype, str)
        ax.set_title(f"{pid} {x.side} {x.view}\n" + (f"{x.subtype}, {x.abnormality}, age {int(x.age)}" if labeled else "no clinical row"),
                     fontsize=10, color="black" if labeled else "gray", fontweight="bold" if labeled else "normal")
plt.tight_layout(); plt.show()

4. Where to go next

The mount carries patients.parquet, studies.parquet and clinical/dictionary.parquet (the column definitions), and sample/ holds one full series on disk. Every other series is one anonymous S3 read away through its crdc_series_uuid, as above.

The chat panel is prefilled with the first of these:

CMMD is CC BY 4.0. When you publish, cite the collection (Cui et al., https://doi.org/10.7937/tcia.eqde-4b16), TCIA (Clark et al., J Digit Imaging 2013) and IDC (Fedorov et al., RadioGraphics 2023).

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.