Plate reader instrument output with ASM: 140 exports from Agilent Gen5, Agilent Gen5 Image, BMG Labtech SMART Control, BMG Labtech MARS and 10 more: sample notebook
A worked notebook for Plate reader instrument output with ASM: 140 exports from Agilent Gen5, Agilent Gen5 Image, BMG Labtech SMART Control, BMG Labtech MARS and 10 more, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Benchling (allotropy contributors); mirrored by Scigantic
- License
- MIT
- Size
- 281 objects, 115,462,558 bytes (140 instrument exports, 141 ASM documents)
- Format
- JSON, CSV (in sampled files)
- Stored at
- Amazon S3, s3://scigantic-allotrope-instrument-data/plate-reader (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Plate reader: instrument output beside its ASM
Microplate reader runs: absorbance, fluorescence and luminescence endpoints, kinetics and spectra, plus imaging-based readers (ImmunoSpot, Apex) and the Lunatic/Stunner DLS and A260 readers. A microplate is read well by well and the export is the plate of values with the read settings around it. This archive holds 140 exports from 14 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.
| folder | instrument software | allotropy Vendor | exports | measurement rows |
| agilent_gen5/ | Agilent Gen5 | AGILENT_GEN5 | 37 | 12,166 |
| moldev_softmax_pro/ | Molecular Devices SoftMax Pro | MOLDEV_SOFTMAX_PRO | 31 | 18,771 |
| unchained_labs_lunatic_stunner/ | Unchained Labs Lunatic and Stunner | UNCHAINED_LABS_LUNATIC | 12 | 1,868 |
| perkin_elmer_envision/ | PerkinElmer EnVision | PERKIN_ELMER_ENVISION | 9 | 5,888 |
| revvity_kaleido/ | Revvity Kaleido | REVVITY_KALEIDO | 8 | 1,632 |
| agilent_gen5_image/ | Agilent Gen5 Image | AGILENT_GEN5_IMAGE | 7 | 695 |
| bmg_mars/ | BMG Labtech MARS | BMG_MARS | 7 | 1,251 |
| methodical_mind/ | MSD Methodical Mind | METHODICAL_MIND | 7 | 2,310 |
| ctl_immunospot/ | CTL ImmunoSpot | CTL_IMMUNOSPOT | 4 | 384 |
| mabtech_apex/ | Mabtech Apex | MABTECH_APEX | 4 | 1,028 |
| msd_workbench/ | MSD Discovery Workbench | MSD_WORKBENCH | 4 | 500 |
| tecan_magellan/ | Tecan Magellan | TECAN_MAGELLAN | 4 | 1,636 |
| thermo_skanit/ | Thermo Fisher SkanIt | THERMO_SKANIT | 4 | 462 |
| bmg_labtech_smart_control/ | BMG Labtech SMART Control | BMG_LABTECH_SMART_CONTROL | 2 | 338 |
The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.
Start here. Click the grey cell below and press Shift+Enter. It lists the Agilent Gen5 exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (moldev_softmax_pro, unchained_labs_lunatic_stunner, perkin_elmer_envision) and run it again.
import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "agilent_gen5"
def count(node):
if isinstance(node, dict):
return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
return sum(count(v) for v in node) if isinstance(node, list) else 0
asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
doc = json.load(open(p))
print(f" {count(doc):>6} measurements {os.path.basename(p)} ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")
Helpers
Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.
import json, os, glob
import pandas as pd
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
def measurement_rows(doc):
"""Every 'measurement document' / 'calculated data document' entry in an ASM document."""
rows = []
def walk(node):
if isinstance(node, dict):
for k, v in node.items():
if k in ("measurement document", "calculated data document") and isinstance(v, list):
rows.extend(v)
else:
walk(v)
elif isinstance(node, list):
for v in node:
walk(v)
walk(doc)
return rows
def measurement_table(doc):
"""One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
rows = measurement_rows(doc)
return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()
def asm_files(root=ROOT):
"""ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)
def export_for(asm_path):
"""The instrument export that shares the ASM file's stem."""
stem = asm_path[:-5].replace("/output/", "/input/")
hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
return hits[0] if hits else None
Convert an instrument export to ASM with allotropy
allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8 and fails on the latin-1, utf-16, utf-8-sig exports in this archive.
import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
"allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
"python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
"rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file
EXPORT = f"{ROOT}/agilent_gen5/luminescence/endpoint_singleplate_without_bandwidth.txt"
fresh = allotrope_from_file(EXPORT, Vendor.AGILENT_GEN5, encoding="chardet")
committed = json.load(open(f"{ROOT}/agilent_gen5/luminescence/endpoint_singleplate_without_bandwidth.json"))
print("manifest :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))
Output:
manifest : plate-reader/REC/2025/03/plate-reader.manifest measurements, fresh vs committed: 12 vs 12
The ASM as a table
Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.
table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()
Output:
12 rows, 9 columns, 1 quantities with units sample document.location identifier sample document.sample identifier \ 0 A1 Plate 1 A1 1 A2 Plate 1 A2 2 A3 Plate 1 A3 3 A4 Plate 1 A4 4 A5 Plate 1 A5 luminescence.value luminescence.unit 0 210530.0 RLU 1 213380.0 RLU 2 216200.0 RLU 3 219000.0 RLU 4 221780.0 RLU
Every ASM document in this folder
The same two helpers work on every file, whatever the instrument.
import matplotlib.pyplot as plt
rows = []
for p in asm_files():
doc = json.load(open(p))
rel = os.path.relpath(p, ROOT).split("/")
rows.append({"software": rel[0], "file": rel[-1],
"release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
"measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("software").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")
fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)
Output:
documents measurements software moldev_softmax_pro 31 18771 agilent_gen5 38 13606 perkin_elmer_envision 9 5888 methodical_mind 7 2310 unchained_labs_lunatic_stunner 12 1868 tecan_magellan 4 1636 revvity_kaleido 8 1632 bmg_mars 7 1251 mabtech_apex 4 1028 agilent_gen5_image 7 695 msd_workbench 4 500 thermo_skanit 4 462 ctl_immunospot 4 384 ...
Reading the raw exports yourself
What the instruments actually write, measured from every file in this archive:
- Agilent Gen5: 35 x .txt: UTF-8; tab-delimited; block markers Results x1; first line Software Version ⇥ 3.12.08
- Agilent Gen5: 1 x .txt: UTF-8 with BOM; tab-delimited; block markers Layout x1, Results x1; first line Software Version ⇥ 3.12.08
- Agilent Gen5: 1 x .txt: Latin-1 (a raw 0xB5 for micro, or similar); tab-delimited; block markers Layout x1; first line Software Version ⇥ 3.15.15
- Molecular Devices SoftMax Pro: 7 x .txt: UTF-8; tab-delimited; block markers ##BLOCKS x1, Plate: x1; first line ##BLOCKS= 1
- Molecular Devices SoftMax Pro: 1 x .txt: Latin-1 (a raw 0xB5 for micro, or similar); tab-delimited; block markers ~End x6, Group: x3, ##BLOCKS x1, Plate: x1; first line ##BLOCKS= 6 ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥
- Molecular Devices SoftMax Pro: 23 x .txt: UTF-16 LE (BOM); tab-delimited; block markers Plate: x2, ~End x2, ##BLOCKS x1; first line ##BLOCKS= 2
- Unchained Labs Lunatic and Stunner: 5 x .csv: UTF-8; comma-delimited; first line Plate ID,"Plate
- Unchained Labs Lunatic and Stunner: 7 x .xlsx: workbook, 1 sheet(s): 'Example_Lunatic_Plate_Reader (1' 51x12
- PerkinElmer EnVision: 9 x .csv: UTF-8; comma-delimited; block markers Results x2; first line Plate information
- Revvity Kaleido: 8 x .csv: UTF-8; comma-delimited; block markers Well x3; first line EnSight Results from
- Agilent Gen5 Image: 3 x .txt: UTF-8; tab-delimited; first line Software Version ⇥ 3.12.08
- Agilent Gen5 Image: 4 x .txt: Latin-1 (a raw 0xB5 for micro, or similar); tab-delimited; block markers Results x1; first line Software Version ⇥ 3.12.08
- BMG Labtech MARS: 7 x .csv: UTF-8; comma-delimited; first line User: USER,Path: C:\Program Files (x86)\BMG\PHERAstar\User\Data,Test run no.: 65
- MSD Methodical Mind: 7 x .txt: UTF-8; tab-delimited; block markers Well x2; first line FileName : ⇥ Z:\MSD-WP\Export\Data\22M0TALA85_2023-09-08-1=Data=33933.txt
- CTL ImmunoSpot: 4 x .txt: UTF-8; tab-delimited; block markers Well x1; first line Software version: ImmunoSpot 7.0.30.5
- Mabtech Apex: 4 x .xlsx: workbook, 15 sheet(s): 'Plate Info' 6x2, 'Plate Database' 97x26, 'Spot Forming Units (SFU)' 20x7, 'Average RSV490' 20x7, 'Average RSV550' 20x7, 'Average RSV640' 20x7
- MSD Discovery Workbench: 2 x .csv: UTF-8; comma-delimited; first line Plate_2BO40AW287,,,,,,,,,,,,,,,,
- MSD Discovery Workbench: 1 x .txt: UTF-8; tab-delimited; block markers Well x4; first line FileName : ⇥ Z:\MSD-WP\Export\22M0TALA85_2023-09-08-133933.txt
- MSD Discovery Workbench: 1 x .xlsx: workbook, 3 sheet(s): 'MSD Data' 72x13, 'Workbench data' 26x16, 'Chart' 53x19
- Tecan Magellan: 4 x .xlsx: workbook, 1 sheet(s): 'Magellan Pro Sheet 1' 25x2
- Thermo Fisher SkanIt: 4 x .xlsx: workbook, 1 sheet(s): 'Luminescence 1_01' 20x13
- BMG Labtech SMART Control: 2 x .xlsx: workbook, 3 sheet(s): 'Microplate End point' 69x15, 'Table End point' 18x97, 'Protocol Information' 46x2
Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.
Where to go next
- Validate an ASM document against its schema with the Allotrope Simple Models (ASM) archive on this portal: every $asm.manifest PURL maps to a file in that mirror.
- Ask the assistant in the sidebar, for example: *"Which quantities and units does each instrument software report?"*
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.