Raw flow cytometry FCS files from a BD FACSAria II: 29 specimens and 9 compensation controls (Caerin 1.1/1.9 antitumour immunity study): sample notebook
A worked notebook for Raw flow cytometry FCS files from a BD FACSAria II: 29 specimens and 9 compensation controls (Caerin 1.1/1.9 antitumour immunity study), with 4 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Tianfang Wang (University of the Sunshine Coast); mirrored by Scigantic
- License
- CC-BY-4.0
- Size
- 38 files, 231,279,051 bytes
- Stored at
- Amazon S3, s3://scigantic-zenodo-13976956 (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Raw flow cytometry: FCS files from a BD FACSAria II
38 event-level FCS 3.0 files written by BD FACSDiva Software Version 8.0.2: 29 specimen tubes of mouse tumor-infiltrating immune cells (two staining panels, CD24 and Ly6C, under three conditions F, P3 and PBS, four replicates each, plus single-marker controls) and 9 single-stain compensation tubes. Each file holds 13 channels as big-endian 32-bit floats: forward and side scatter, eight fluorescence detectors and time.
Source: Wang, *Raw flow cytometry data files (fcs files) part 1*, Zenodo 10.5281/zenodo.13976956, CC BY 4.0, mirrored unchanged and mounted read-only at /mnt/archive.
Start here. Click the grey cell below and press Shift+Enter. It reads one tube's header (a few kilobytes, not the events) and prints what the cytometer recorded. Then change TUBE to another tube, for example LJJLy6c PBS_2, and run it again.
import os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
TUBE = "LJJCD24F_1"
path = glob.glob(f"{ROOT}/*{TUBE}_*.fcs")[0]
with open(path, "rb") as f:
h = f.read(8192)
t0, t1 = int(h[10:18]), int(h[18:26]); txt = h[t0:t1 + 1].decode("latin-1")
kv = txt[1:].strip(txt[0]).split(txt[0]); meta = dict(zip(kv[0::2], kv[1::2]))
print(os.path.basename(path))
print(f"{int(meta['$TOT']):,} events x {meta['$PAR']} channels, {meta['$CYT']}, {meta['$DATE']} {meta['$BTIM']}")
print("compensation stored in file:", meta["APPLY COMPENSATION"], "| lasers:", meta["LASER1NAME"], meta["LASER2NAME"], meta["LASER3NAME"])
A reader and the inventory
No FCS library ships in this image, and none is needed: the format is a fixed header, a keyword segment and a block of floats. The reader checks the data type and byte order before decoding. The inventory reads headers only.
import os, glob
import numpy as np, pandas as pd
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
def read_fcs(path, events=True):
"""FCS 3.0 list-mode reader for these files: HEADER offsets, TEXT keywords, DATA as float32."""
with open(path, "rb") as f:
h = f.read(58)
t0, t1, d0, d1 = (int(h[i:i + 8]) for i in (10, 18, 26, 34))
f.seek(t0); txt = f.read(t1 - t0 + 1).decode("latin-1")
kv = txt[1:].strip(txt[0]).split(txt[0]); meta = dict(zip(kv[0::2], kv[1::2]))
if not events:
return meta, None
npar, ntot = int(meta["$PAR"]), int(meta["$TOT"])
assert meta["$DATATYPE"] == "F" and meta["$BYTEORD"] == "4,3,2,1"
f.seek(d0); raw = f.read(npar * ntot * 4)
names = [meta[f"$P{i}N"] for i in range(1, npar + 1)]
return meta, pd.DataFrame(np.frombuffer(raw, ">f4").reshape(ntot, npar).astype(np.float32), columns=names)
def spill(meta):
"""FACSDiva's spillover matrix from the SPILL keyword: (channels, matrix)."""
s = meta["SPILL"].split(","); n = int(s[0])
return s[1:1 + n], np.array(s[1 + n:], float).reshape(n, n)
inventory = []
for p in sorted(glob.glob(f"{ROOT}/*.fcs")):
meta, _ = read_fcs(p, events=False)
inventory.append({"tube": meta["TUBE NAME"], "source": meta["$SRC"], "date": meta["$DATE"], "events": int(meta["$TOT"])})
inventory = pd.DataFrame(inventory)
inventory.groupby(["source", "date"]).agg(tubes=("tube", "size"), events=("events", "sum"))
Output:
tubes events source date Compensation Controls 27-APR-2022 9 45000 Specimen_002 02-MAY-2022 29 4400000
Compensation is already in the files
FACSDiva wrote its spillover matrix into every header (SPILL, 8 x 8 over FITC-A, PE-A, PerCP-Cy5-5-A, PE-Cy7-A, APC-A, APC-Cy7-A, BV421-A, BV510-A). The 29 specimen tubes share the real matrix; the compensation tubes carry the identity, because they were acquired before it was computed. Compensate the specimens with the inverse of their own matrix, then transform: the stored values are linear.
meta, ev = read_fcs(glob.glob(f"{ROOT}/*LJJCD24F_1_*.fcs")[0])
chans, S = spill(meta)
print("largest spillover:", f"{S[~np.eye(len(chans), dtype=bool)].max():.3f}")
comp = ev.copy()
comp[chans] = ev[chans].values.astype(np.float64) @ np.linalg.inv(S)
asinh = np.arcsinh(comp[chans] / 150)
asinh.describe().T[["mean", "50%", "max"]].round(2)
Output:
largest spillover: 0.628
mean 50% max
FITC-A 2.70 2.52 8.15
PE-A 2.28 2.33 8.16
PerCP-Cy5-5-A 1.54 1.31 8.16
PE-Cy7-A 2.26 2.28 8.15
APC-A 2.62 2.49 8.16
APC-Cy7-A 1.79 1.84 8.15
BV421-A 0.63 0.63 8.15
BV510-A 3.41 3.49 8.15
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(8, 3.2), dpi=68)
axes[0].hexbin(ev["FSC-A"], ev["SSC-A"], gridsize=60, bins="log", cmap="Blues")
axes[0].set_xlabel("FSC-A"); axes[0].set_ylabel("SSC-A"); axes[0].set_title("Scatter")
axes[1].hexbin(asinh["APC-A"], asinh["BV421-A"], gridsize=60, bins="log", cmap="Oranges")
axes[1].set_xlabel("APC-A (asinh)"); axes[1].set_ylabel("BV421-A (asinh)"); axes[1].set_title("Compensated")
plt.tight_layout(); plt.show()
Notes
- Stain names ($PnS) are empty in every file, so columns are detector names. Markers come from the tube names and the paper.
- The trailing _NNN in a file name is the acquisition counter; the replicate is the digit before it.
- Do not compensate the compensation tubes: their stored matrix is the identity.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.