Flow cytometry instrument output with ASM: 5 exports from BD Biosciences FACSDiva, FlowJo: sample notebook

A worked notebook for Flow cytometry instrument output with ASM: 5 exports from BD Biosciences FACSDiva, FlowJo, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Benchling (allotropy contributors); mirrored by Scigantic
License
MIT
Size
10 objects, 5,282,575 bytes (5 instrument exports, 5 ASM documents)
Format
JSON (in sampled files)
Stored at
Amazon S3, s3://scigantic-allotrope-instrument-data/flow-cytometry (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Flow cytometry: instrument output beside its ASM

Flow cytometry analysis exports: gated population statistics (counts, percentages, medians per fluorochrome) from FACSDiva experiment XML and FlowJo workspaces. Event-level .fcs files are a different thing; see the FCS archive on this portal. This archive holds 5 exports from 2 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.

folderinstrument softwareallotropy Vendorexportsmeasurement rows
flowjo/FlowJoFLOWJO315
bd_biosciences_facsdiva/BD Biosciences FACSDivaBD_BIOSCIENCES_FACSDIVA26

The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.

Start here. Click the grey cell below and press Shift+Enter. It lists the FlowJo exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (bd_biosciences_facsdiva) and run it again.

import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "flowjo"

def count(node):
    if isinstance(node, dict):
        return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
    return sum(count(v) for v in node) if isinstance(node, list) else 0

asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
    doc = json.load(open(p))
    print(f"  {count(doc):>6} measurements  {os.path.basename(p)}  ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")

Helpers

Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.

import json, os, glob
import pandas as pd

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")

def measurement_rows(doc):
    """Every 'measurement document' / 'calculated data document' entry in an ASM document."""
    rows = []
    def walk(node):
        if isinstance(node, dict):
            for k, v in node.items():
                if k in ("measurement document", "calculated data document") and isinstance(v, list):
                    rows.extend(v)
                else:
                    walk(v)
        elif isinstance(node, list):
            for v in node:
                walk(v)
    walk(doc)
    return rows

def measurement_table(doc):
    """One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
    rows = measurement_rows(doc)
    return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()

def asm_files(root=ROOT):
    """ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
    return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
                  if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)

def export_for(asm_path):
    """The instrument export that shares the ASM file's stem."""
    stem = asm_path[:-5].replace("/output/", "/input/")
    hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
    return hits[0] if hits else None

Convert an instrument export to ASM with allotropy

allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8.

import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
                    "allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
                    "python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
                    "rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
    raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file

EXPORT = f"{ROOT}/flowjo/flowjo_example_2.wsp"
fresh = allotrope_from_file(EXPORT, Vendor.FLOWJO, encoding="chardet")
committed = json.load(open(f"{ROOT}/flowjo/flowjo_example_2.json"))

print("manifest  :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))

Output:

manifest  : flow-cytometry/BENCHLING/2025/03/flow-cytometry.manifest
measurements, fresh vs committed: 1 vs 1

The ASM as a table

Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.

table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()

Output:

1 rows, 36 columns, 0 quantities with units
  sample document.sample identifier
0                                33

Every ASM document in this folder

The same two helpers work on every file, whatever the instrument.

import matplotlib.pyplot as plt

rows = []
for p in asm_files():
    doc = json.load(open(p))
    rel = os.path.relpath(p, ROOT).split("/")
    rows.append({"software": rel[0], "file": rel[-1],
                 "release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
                 "measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("software").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")

fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)

Output:

                         documents  measurements
software                                        
flowjo                           3            15
bd_biosciences_facsdiva          2             6

Reading the raw exports yourself

What the instruments actually write, measured from every file in this archive:

Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.

Where to go next

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.