IDC CMMD: Breast Cancer mammography (1,775 patients): sample notebook
A worked notebook for IDC CMMD: Breast Cancer mammography (1,775 patients), with 5 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- NCI Imaging Data Commons; CMMD contributors; tables and sample by Scigantic
- License
- Per series (series.parquet license_short_name): CC BY 4.0 1,775. Cite the source DOI(s) and the IDC.
- Size
- 6 parquet tables plus 4 sample DICOM files (18 MB)
- Stored at
- Amazon S3, s3://scigantic-idc-open/cmmd (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
- DOI
- 10.7937/tcia.eqde-4b16
The notebook
Chinese Mammography Database (CMMD): every clinical record, then the images behind it
CMMD holds the mammograms of 1,775 patients (5,202 images in the IDC index) with biopsy-confirmed breast lesions, plus a clinical table with the side of the lesion, its type (mass, calcification or both), benign or malignant, age, and for 749 patients the molecular subtype. The images are on the NCI Imaging Data Commons (IDC); the clinical table comes from The Cancer Imaging Archive (TCIA).
Both are mounted here at ~/archive: the series index and the clinical table as parquet. That means the whole-cohort analysis below reads all 1,872 clinical rows from disk with no download, and only the four example patients' images are fetched from IDC's public bucket.
Start here. Click the grey cell below and press Shift+Enter. It counts lesions by subtype from the mounted clinical table. Then change SUBTYPE and run it again.
SUBTYPE = "triple negative" # try "Luminal A", "Luminal B" or "HER2-enriched"
import io, os
import pandas as pd
import requests
def _root():
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
for cand in (base, os.path.join(base, "cmmd")):
if cand and os.path.exists(os.path.join(cand, "series.parquet")):
return cand
return None
ROOT = _root()
def read(name):
if ROOT:
return pd.read_parquet(os.path.join(ROOT, name))
r = requests.get(f"https://scigantic-idc-open.s3.amazonaws.com/cmmd/{name}", timeout=120) # same files, public mirror
r.raise_for_status()
return pd.read_parquet(io.BytesIO(r.content))
clin = read("clinical/cmmd_clinical.parquet")
rows = clin[clin.subtype == SUBTYPE]
print(f"{len(clin):,} lesions from {clin.dicom_patient_id.nunique():,} patients; {len(rows)} are {SUBTYPE!r}")
print("lesion type:", rows.abnormality.value_counts().to_dict())
print(f"age: median {rows.age.median():.0f}, range {rows.age.min()}-{rows.age.max()}")
Output:
1,872 lesions from 1,775 patients; 86 are 'triple negative'
lesion type: {'mass': 65, 'both': 14, 'calcification': 7}
age: median 49, range 28-85
1. Read the mounted tables
series.parquet is the IDC index for the collection: one row per patient's mammography series, with its public S3 location. clinical/cmmd_clinical.parquet is TCIA's clinical spreadsheet with one row per affected breast, so a patient with lesions in both breasts has two rows. The reader falls back to the same files on the public mirror bucket when the mount is missing, so this notebook also runs outside Scigantic.
import io, os
import numpy as np
import pandas as pd
import requests
import matplotlib.pyplot as plt
plt.rcParams["figure.dpi"] = 72
try:
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats("jpeg") # mammograms are photographic; JPEG keeps the notebook small
except ImportError:
pass
def _root():
for base in (os.environ.get("SCIGANTIC_MOUNT_PATH", ""), os.path.expanduser("~/archive")):
for cand in (base, os.path.join(base, "cmmd")):
if cand and os.path.exists(os.path.join(cand, "series.parquet")):
return cand
return None
ROOT = _root()
def read(name):
if ROOT:
return pd.read_parquet(os.path.join(ROOT, name))
r = requests.get(f"https://scigantic-idc-open.s3.amazonaws.com/cmmd/{name}", timeout=120) # same files, public mirror
r.raise_for_status()
return pd.read_parquet(io.BytesIO(r.content))
print("tables from:", ROOT or "public mirror over HTTPS")
series = read("series.parquet")
clin = read("clinical/cmmd_clinical.parquet")
clin["subtype"] = clin.subtype.replace("", np.nan)
print(f"{len(series):,} series, {series.instanceCount.sum():,} images, {series.series_size_MB.sum() / 1024:.1f} GB on IDC")
print(f"{len(clin):,} clinical rows for {clin.dicom_patient_id.nunique():,} patients; "
f"{clin.dicom_patient_id.duplicated(keep=False).sum()} rows belong to patients with lesions in both breasts")
missing = set(clin.dicom_patient_id) - set(series.PatientID)
print("clinical patients with no image series:", len(missing))
Output:
tables from: public mirror over HTTPS 1,775 series, 5,202 images, 22.3 GB on IDC 1,872 clinical rows for 1,775 patients; 194 rows belong to patients with lesions in both breasts clinical patients with no image series: 0
2. The whole cohort, with no image download
Because every clinical row is on disk, cohort questions take a line each. Malignant lesions outnumber benign ones for every lesion type; the benign share is highest for a mass alone (36%) and lowest when a mass and calcifications occur together (17%). The molecular subtype is recorded for 749 lesions, all from D2 patients and all malignant.
print(pd.crosstab(clin.abnormality, clin.classification, margins=True).to_string(), "\n")
sub = clin.dropna(subset=["subtype"])
print(sub.groupby("subtype").agg(lesions=("age", "size"), median_age=("age", "median"),
mass_share=("abnormality", lambda a: round((a != "calcification").mean(), 2))).to_string())
order = ["Luminal A", "Luminal B", "HER2-enriched", "triple negative"]
fig, axes = plt.subplots(1, 2, figsize=(11, 3.6))
for i, (label, ages) in enumerate([("Benign", clin.loc[clin.classification == "Benign", "age"]),
("Malignant", clin.loc[clin.classification == "Malignant", "age"])]):
axes[0].hist(ages, bins=range(15, 95, 5), alpha=0.6, label=f"{label} (n={len(ages)})", color=["#7fa7d9", "#1f4e8c"][i])
axes[0].set_xlabel("age"); axes[0].set_ylabel("lesions"); axes[0].legend(); axes[0].set_title("Age by diagnosis, all 1,872 lesions")
axes[1].boxplot([sub.loc[sub.subtype == s, "age"] for s in order], showfliers=False)
axes[1].set_xticks(range(1, 5), [s.replace("triple negative", "triple\nnegative").replace("HER2-enriched", "HER2-\nenriched") for s in order])
axes[1].set_ylabel("age"); axes[1].set_title("Age by molecular subtype (D2 patients)")
plt.tight_layout(); plt.show()
Output:
classification Benign Malignant All
abnormality
both 80 381 461
calcification 64 198 262
mass 412 737 1149
All 556 1316 1872
lesions median_age mass_share
subtype
HER2-enriched 135 50.0 0.77
Luminal A 152 50.0 0.88
Luminal B 376 48.0 0.89
triple negative 86 49.0 0.92
3. The images behind four records
Each series row gives the bucket and crdc_series_uuid; listing that prefix in s3://idc-open-data returns the DICOM files, which are uncompressed and about 4 MB each. We take the first patient (by id) of each molecular subtype who has a single clinical row and a mass, download their images anonymously, and join on patient and breast side using the ImageLaterality tag, because the clinical row describes one breast only.
import xml.etree.ElementTree as ET
import pydicom
def series_files(row):
base = f"https://{row.aws_bucket}.s3.amazonaws.com"
xml = requests.get(base, params={"list-type": "2", "prefix": row.crdc_series_uuid + "/"}, timeout=60).content
ns = {"s3": "http://s3.amazonaws.com/doc/2006-03-01/"}
keys = [k.text for k in ET.fromstring(xml).findall(".//s3:Key", ns) if k.text.endswith(".dcm")]
return [pydicom.dcmread(io.BytesIO(requests.get(f"{base}/{k}", timeout=120).content)) for k in keys]
single = clin[~clin.dicom_patient_id.duplicated(keep=False) & (clin.abnormality == "mass")]
examples = single.dropna(subset=["subtype"]).sort_values("dicom_patient_id").groupby("subtype").head(1)
examples = examples.set_index("subtype").loc[order].reset_index()
images = []
for ex in examples.itertuples():
for ds in series_files(series.set_index("PatientID").loc[ex.dicom_patient_id]):
view = ds.ViewCodeSequence[0].CodeMeaning if "ViewCodeSequence" in ds else ""
view = "CC" if "cranio-caudal" in view.lower() else "MLO" if "oblique" in view.lower() else view
images.append({"PatientID": ex.dicom_patient_id, "side": ds.get("ImageLaterality", ""), "view": view, "ds": ds})
imgs = pd.DataFrame(images)
joined = imgs.merge(clin, left_on=["PatientID", "side"], right_on=["dicom_patient_id", "leftright"], how="left")
print(joined[["PatientID", "side", "view", "subtype", "abnormality", "classification", "age"]].to_string(index=False))
Output:
PatientID side view subtype abnormality classification age D2-0337 L CC NaN NaN NaN NaN D2-0337 R CC Luminal A mass Malignant 54.0 D2-0337 L MLO NaN NaN NaN NaN D2-0337 R MLO Luminal A mass Malignant 54.0 D2-0333 L CC NaN NaN NaN NaN D2-0333 L MLO NaN NaN NaN NaN D2-0333 R CC Luminal B mass Malignant 85.0 D2-0333 R MLO Luminal B mass Malignant 85.0 D2-0342 L CC NaN NaN NaN NaN D2-0342 L MLO NaN NaN NaN NaN D2-0342 R CC HER2-enriched mass Malignant 52.0 D2-0342 R MLO HER2-enriched mass Malignant 52.0 D2-0344 R MLO triple negative mass Malignant 64.0 D2-0344 R CC triple negative mass Malignant 64.0 ...
pids = list(examples.dicom_patient_id)
fig, axes = plt.subplots(len(pids), 4, figsize=(12, 3.6 * len(pids)))
for r, pid in enumerate(pids):
row = joined[joined.PatientID == pid].sort_values(["side", "view"], key=lambda c: c.map({"R": 0, "L": 1, "CC": 0, "MLO": 1}))
for c, x in enumerate(row.itertuples()):
px = x.ds.pixel_array
step = max(1, int(np.ceil(max(px.shape) / 500)))
ax = axes[r, c]; ax.imshow(px[::step, ::step], cmap="gray"); ax.set_xticks([]); ax.set_yticks([])
labeled = isinstance(x.subtype, str)
ax.set_title(f"{pid} {x.side} {x.view}\n" + (f"{x.subtype}, {x.abnormality}, age {int(x.age)}" if labeled else "no clinical row"),
fontsize=10, color="black" if labeled else "gray", fontweight="bold" if labeled else "normal")
plt.tight_layout(); plt.show()
4. Where to go next
The mount carries patients.parquet, studies.parquet and clinical/dictionary.parquet (the column definitions), and sample/ holds one full series on disk. Every other series is one anonymous S3 read away through its crdc_series_uuid, as above.
The chat panel is prefilled with the first of these:
- *For each molecular subtype, download the affected-side CC view for 10 patients and compare the mean breast density inside the breast region.*
- *Build a table of patients with lesions in both breasts and show both sides for three of them.*
- *Train a quick logistic regression on age and lesion type to predict benign versus malignant, and report how well it does.*
CMMD is CC BY 4.0. When you publish, cite the collection (Cui et al., https://doi.org/10.7937/tcia.eqde-4b16), TCIA (Clark et al., J Digit Imaging 2013) and IDC (Fedorov et al., RadioGraphics 2023).
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.