Raw flow cytometry FCS files from a BD FACSAria II: 29 specimens and 9 compensation controls (Caerin 1.1/1.9 antitumour immunity study): sample notebook

A worked notebook for Raw flow cytometry FCS files from a BD FACSAria II: 29 specimens and 9 compensation controls (Caerin 1.1/1.9 antitumour immunity study), with 4 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Tianfang Wang (University of the Sunshine Coast); mirrored by Scigantic
License
CC-BY-4.0
Size
38 files, 231,279,051 bytes
Stored at
Amazon S3, s3://scigantic-zenodo-13976956 (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Raw flow cytometry: FCS files from a BD FACSAria II

38 event-level FCS 3.0 files written by BD FACSDiva Software Version 8.0.2: 29 specimen tubes of mouse tumor-infiltrating immune cells (two staining panels, CD24 and Ly6C, under three conditions F, P3 and PBS, four replicates each, plus single-marker controls) and 9 single-stain compensation tubes. Each file holds 13 channels as big-endian 32-bit floats: forward and side scatter, eight fluorescence detectors and time.

Source: Wang, *Raw flow cytometry data files (fcs files) part 1*, Zenodo 10.5281/zenodo.13976956, CC BY 4.0, mirrored unchanged and mounted read-only at /mnt/archive.

Start here. Click the grey cell below and press Shift+Enter. It reads one tube's header (a few kilobytes, not the events) and prints what the cytometer recorded. Then change TUBE to another tube, for example LJJLy6c PBS_2, and run it again.

import os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
TUBE = "LJJCD24F_1"

path = glob.glob(f"{ROOT}/*{TUBE}_*.fcs")[0]
with open(path, "rb") as f:
    h = f.read(8192)
t0, t1 = int(h[10:18]), int(h[18:26]); txt = h[t0:t1 + 1].decode("latin-1")
kv = txt[1:].strip(txt[0]).split(txt[0]); meta = dict(zip(kv[0::2], kv[1::2]))
print(os.path.basename(path))
print(f"{int(meta['$TOT']):,} events x {meta['$PAR']} channels, {meta['$CYT']}, {meta['$DATE']} {meta['$BTIM']}")
print("compensation stored in file:", meta["APPLY COMPENSATION"], "| lasers:", meta["LASER1NAME"], meta["LASER2NAME"], meta["LASER3NAME"])

A reader and the inventory

No FCS library ships in this image, and none is needed: the format is a fixed header, a keyword segment and a block of floats. The reader checks the data type and byte order before decoding. The inventory reads headers only.

import os, glob
import numpy as np, pandas as pd

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")

def read_fcs(path, events=True):
    """FCS 3.0 list-mode reader for these files: HEADER offsets, TEXT keywords, DATA as float32."""
    with open(path, "rb") as f:
        h = f.read(58)
        t0, t1, d0, d1 = (int(h[i:i + 8]) for i in (10, 18, 26, 34))
        f.seek(t0); txt = f.read(t1 - t0 + 1).decode("latin-1")
        kv = txt[1:].strip(txt[0]).split(txt[0]); meta = dict(zip(kv[0::2], kv[1::2]))
        if not events:
            return meta, None
        npar, ntot = int(meta["$PAR"]), int(meta["$TOT"])
        assert meta["$DATATYPE"] == "F" and meta["$BYTEORD"] == "4,3,2,1"
        f.seek(d0); raw = f.read(npar * ntot * 4)
    names = [meta[f"$P{i}N"] for i in range(1, npar + 1)]
    return meta, pd.DataFrame(np.frombuffer(raw, ">f4").reshape(ntot, npar).astype(np.float32), columns=names)

def spill(meta):
    """FACSDiva's spillover matrix from the SPILL keyword: (channels, matrix)."""
    s = meta["SPILL"].split(","); n = int(s[0])
    return s[1:1 + n], np.array(s[1 + n:], float).reshape(n, n)

inventory = []
for p in sorted(glob.glob(f"{ROOT}/*.fcs")):
    meta, _ = read_fcs(p, events=False)
    inventory.append({"tube": meta["TUBE NAME"], "source": meta["$SRC"], "date": meta["$DATE"], "events": int(meta["$TOT"])})
inventory = pd.DataFrame(inventory)
inventory.groupby(["source", "date"]).agg(tubes=("tube", "size"), events=("events", "sum"))

Output:

                                   tubes   events
source                date                       
Compensation Controls 27-APR-2022      9    45000
Specimen_002          02-MAY-2022     29  4400000

Compensation is already in the files

FACSDiva wrote its spillover matrix into every header (SPILL, 8 x 8 over FITC-A, PE-A, PerCP-Cy5-5-A, PE-Cy7-A, APC-A, APC-Cy7-A, BV421-A, BV510-A). The 29 specimen tubes share the real matrix; the compensation tubes carry the identity, because they were acquired before it was computed. Compensate the specimens with the inverse of their own matrix, then transform: the stored values are linear.

meta, ev = read_fcs(glob.glob(f"{ROOT}/*LJJCD24F_1_*.fcs")[0])
chans, S = spill(meta)
print("largest spillover:", f"{S[~np.eye(len(chans), dtype=bool)].max():.3f}")
comp = ev.copy()
comp[chans] = ev[chans].values.astype(np.float64) @ np.linalg.inv(S)
asinh = np.arcsinh(comp[chans] / 150)
asinh.describe().T[["mean", "50%", "max"]].round(2)

Output:

largest spillover: 0.628
               mean   50%   max
FITC-A         2.70  2.52  8.15
PE-A           2.28  2.33  8.16
PerCP-Cy5-5-A  1.54  1.31  8.16
PE-Cy7-A       2.26  2.28  8.15
APC-A          2.62  2.49  8.16
APC-Cy7-A      1.79  1.84  8.15
BV421-A        0.63  0.63  8.15
BV510-A        3.41  3.49  8.15
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(8, 3.2), dpi=68)
axes[0].hexbin(ev["FSC-A"], ev["SSC-A"], gridsize=60, bins="log", cmap="Blues")
axes[0].set_xlabel("FSC-A"); axes[0].set_ylabel("SSC-A"); axes[0].set_title("Scatter")
axes[1].hexbin(asinh["APC-A"], asinh["BV421-A"], gridsize=60, bins="log", cmap="Oranges")
axes[1].set_xlabel("APC-A (asinh)"); axes[1].set_ylabel("BV421-A (asinh)"); axes[1].set_title("Compensated")
plt.tight_layout(); plt.show()

Notes

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.