Plate reader instrument output with ASM: 140 exports from Agilent Gen5, Agilent Gen5 Image, BMG Labtech SMART Control, BMG Labtech MARS and 10 more: sample notebook

A worked notebook for Plate reader instrument output with ASM: 140 exports from Agilent Gen5, Agilent Gen5 Image, BMG Labtech SMART Control, BMG Labtech MARS and 10 more, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Benchling (allotropy contributors); mirrored by Scigantic
License
MIT
Size
281 objects, 115,462,558 bytes (140 instrument exports, 141 ASM documents)
Format
JSON, CSV (in sampled files)
Stored at
Amazon S3, s3://scigantic-allotrope-instrument-data/plate-reader (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Plate reader: instrument output beside its ASM

Microplate reader runs: absorbance, fluorescence and luminescence endpoints, kinetics and spectra, plus imaging-based readers (ImmunoSpot, Apex) and the Lunatic/Stunner DLS and A260 readers. A microplate is read well by well and the export is the plate of values with the read settings around it. This archive holds 140 exports from 14 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.

folderinstrument softwareallotropy Vendorexportsmeasurement rows
agilent_gen5/Agilent Gen5AGILENT_GEN53712,166
moldev_softmax_pro/Molecular Devices SoftMax ProMOLDEV_SOFTMAX_PRO3118,771
unchained_labs_lunatic_stunner/Unchained Labs Lunatic and StunnerUNCHAINED_LABS_LUNATIC121,868
perkin_elmer_envision/PerkinElmer EnVisionPERKIN_ELMER_ENVISION95,888
revvity_kaleido/Revvity KaleidoREVVITY_KALEIDO81,632
agilent_gen5_image/Agilent Gen5 ImageAGILENT_GEN5_IMAGE7695
bmg_mars/BMG Labtech MARSBMG_MARS71,251
methodical_mind/MSD Methodical MindMETHODICAL_MIND72,310
ctl_immunospot/CTL ImmunoSpotCTL_IMMUNOSPOT4384
mabtech_apex/Mabtech ApexMABTECH_APEX41,028
msd_workbench/MSD Discovery WorkbenchMSD_WORKBENCH4500
tecan_magellan/Tecan MagellanTECAN_MAGELLAN41,636
thermo_skanit/Thermo Fisher SkanItTHERMO_SKANIT4462
bmg_labtech_smart_control/BMG Labtech SMART ControlBMG_LABTECH_SMART_CONTROL2338

The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.

Start here. Click the grey cell below and press Shift+Enter. It lists the Agilent Gen5 exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (moldev_softmax_pro, unchained_labs_lunatic_stunner, perkin_elmer_envision) and run it again.

import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "agilent_gen5"

def count(node):
    if isinstance(node, dict):
        return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
    return sum(count(v) for v in node) if isinstance(node, list) else 0

asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
    doc = json.load(open(p))
    print(f"  {count(doc):>6} measurements  {os.path.basename(p)}  ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")

Helpers

Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.

import json, os, glob
import pandas as pd

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")

def measurement_rows(doc):
    """Every 'measurement document' / 'calculated data document' entry in an ASM document."""
    rows = []
    def walk(node):
        if isinstance(node, dict):
            for k, v in node.items():
                if k in ("measurement document", "calculated data document") and isinstance(v, list):
                    rows.extend(v)
                else:
                    walk(v)
        elif isinstance(node, list):
            for v in node:
                walk(v)
    walk(doc)
    return rows

def measurement_table(doc):
    """One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
    rows = measurement_rows(doc)
    return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()

def asm_files(root=ROOT):
    """ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
    return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
                  if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)

def export_for(asm_path):
    """The instrument export that shares the ASM file's stem."""
    stem = asm_path[:-5].replace("/output/", "/input/")
    hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
    return hits[0] if hits else None

Convert an instrument export to ASM with allotropy

allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8 and fails on the latin-1, utf-16, utf-8-sig exports in this archive.

import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
                    "allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
                    "python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
                    "rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
    raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file

EXPORT = f"{ROOT}/agilent_gen5/luminescence/endpoint_singleplate_without_bandwidth.txt"
fresh = allotrope_from_file(EXPORT, Vendor.AGILENT_GEN5, encoding="chardet")
committed = json.load(open(f"{ROOT}/agilent_gen5/luminescence/endpoint_singleplate_without_bandwidth.json"))

print("manifest  :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))

Output:

manifest  : plate-reader/REC/2025/03/plate-reader.manifest
measurements, fresh vs committed: 12 vs 12

The ASM as a table

Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.

table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()

Output:

12 rows, 9 columns, 1 quantities with units
  sample document.location identifier sample document.sample identifier  \
0                                  A1                        Plate 1 A1   
1                                  A2                        Plate 1 A2   
2                                  A3                        Plate 1 A3   
3                                  A4                        Plate 1 A4   
4                                  A5                        Plate 1 A5   

   luminescence.value luminescence.unit  
0            210530.0               RLU  
1            213380.0               RLU  
2            216200.0               RLU  
3            219000.0               RLU  
4            221780.0               RLU

Every ASM document in this folder

The same two helpers work on every file, whatever the instrument.

import matplotlib.pyplot as plt

rows = []
for p in asm_files():
    doc = json.load(open(p))
    rel = os.path.relpath(p, ROOT).split("/")
    rows.append({"software": rel[0], "file": rel[-1],
                 "release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
                 "measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("software").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")

fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)

Output:

                                documents  measurements
software                                               
moldev_softmax_pro                     31         18771
agilent_gen5                           38         13606
perkin_elmer_envision                   9          5888
methodical_mind                         7          2310
unchained_labs_lunatic_stunner         12          1868
tecan_magellan                          4          1636
revvity_kaleido                         8          1632
bmg_mars                                7          1251
mabtech_apex                            4          1028
agilent_gen5_image                      7           695
msd_workbench                           4           500
thermo_skanit                           4           462
ctl_immunospot                          4           384
...

Reading the raw exports yourself

What the instruments actually write, measured from every file in this archive:

Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.

Where to go next

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.