Laboratory instrument output files with their ASM conversions: 343 exports from 53 instrument softwares across 12 techniques: sample notebook
A worked notebook for Laboratory instrument output files with their ASM conversions: 343 exports from 53 instrument softwares across 12 techniques, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Benchling (allotropy contributors); mirrored by Scigantic
- License
- MIT
- Size
- 687 objects, 769,830,126 bytes
- Format
- JSON, CSV, TSV (in sampled files)
- Stored at
- Amazon S3, s3://scigantic-allotrope-instrument-data (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Laboratory instrument output and its ASM
A repository of real laboratory instrument exports, one folder per technique and one per instrument software. This archive holds 343 exports from 53 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.
| technique folder | exports | instrument softwares |
| binding-affinity/ | 15 | 3 |
| cell-counting/ | 32 | 6 |
| electrophoresis/ | 3 | 1 |
| flow-cytometry/ | 5 | 2 |
| liquid-chromatography/ | 14 | 4 |
| liquid-handler/ | 10 | 3 |
| multi-analyte-profiling/ | 12 | 3 |
| plate-reader/ | 140 | 14 |
| solution-analyzer/ | 22 | 3 |
| spectrophotometry/ | 28 | 8 |
| dpcr/ | 15 | 2 |
| qpcr/ | 47 | 4 |
The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.
Start here. Click the grey cell below and press Shift+Enter. It lists the Molecular Devices SoftMax Pro exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (agilent_gen5, appbio_quantstudio_designandanalysis, appbio_quantstudio) and run it again.
import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "moldev_softmax_pro"
def count(node):
if isinstance(node, dict):
return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
return sum(count(v) for v in node) if isinstance(node, list) else 0
asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
doc = json.load(open(p))
print(f" {count(doc):>6} measurements {os.path.basename(p)} ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")
Helpers
Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.
import json, os, glob
import pandas as pd
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
def measurement_rows(doc):
"""Every 'measurement document' / 'calculated data document' entry in an ASM document."""
rows = []
def walk(node):
if isinstance(node, dict):
for k, v in node.items():
if k in ("measurement document", "calculated data document") and isinstance(v, list):
rows.extend(v)
else:
walk(v)
elif isinstance(node, list):
for v in node:
walk(v)
walk(doc)
return rows
def measurement_table(doc):
"""One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
rows = measurement_rows(doc)
return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()
def asm_files(root=ROOT):
"""ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)
def export_for(asm_path):
"""The instrument export that shares the ASM file's stem."""
stem = asm_path[:-5].replace("/output/", "/input/")
hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
return hits[0] if hits else None
Convert an instrument export to ASM with allotropy
allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8 and fails on the latin-1, utf-16, utf-8-sig exports in this archive.
import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
"allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
"python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
"rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file
EXPORT = f"{ROOT}/plate-reader/moldev_softmax_pro/no_measurements.txt"
fresh = allotrope_from_file(EXPORT, Vendor.MOLDEV_SOFTMAX_PRO, encoding="chardet")
committed = json.load(open(f"{ROOT}/plate-reader/moldev_softmax_pro/no_measurements.json"))
print("manifest :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))
Output:
manifest : plate-reader/REC/2025/03/plate-reader.manifest measurements, fresh vs committed: 6 vs 6
The ASM as a table
Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.
table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()
Output:
6 rows, 10 columns, 2 quantities with units sample document.location identifier sample document.sample identifier \ 0 A1 PLate1 A1 1 A2 PLate1 A2 2 A3 PLate1 A3 3 B1 PLate1 B1 4 B2 PLate1 B2 compartment temperature.value compartment temperature.unit \ 0 21.9 degC 1 21.9 degC 2 21.9 degC 3 21.9 degC 4 21.9 degC ...
Every ASM document in the archive
The same two helpers work on every file, across all twelve techniques, whatever the instrument.
import matplotlib.pyplot as plt
rows = []
for p in asm_files():
doc = json.load(open(p))
rel = os.path.relpath(p, ROOT).split("/")
rows.append({"technique": rel[0], "software": rel[1],
"release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
"measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("technique").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")
fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)
Output:
documents measurements technique plate-reader 141 50369 qpcr 47 17978 binding-affinity 15 16599 multi-analyte-profiling 12 2139 dpcr 15 1063 liquid-handler 10 625 solution-analyzer 22 573 cell-counting 32 540 spectrophotometry 28 471 electrophoresis 3 344 liquid-chromatography 14 104 flow-cytometry 5 21
Reading the raw exports yourself
What the instruments actually write, measured from every file in this archive:
- Agilent Gen5: 35 x .txt: UTF-8; tab-delimited; block markers Results x1; first line Software Version ⇥ 3.12.08
- Agilent Gen5: 1 x .txt: UTF-8 with BOM; tab-delimited; block markers Layout x1, Results x1; first line Software Version ⇥ 3.12.08
- Agilent Gen5: 1 x .txt: Latin-1 (a raw 0xB5 for micro, or similar); tab-delimited; block markers Layout x1; first line Software Version ⇥ 3.15.15
- Molecular Devices SoftMax Pro: 7 x .txt: UTF-8; tab-delimited; block markers ##BLOCKS x1, Plate: x1; first line ##BLOCKS= 1
- Molecular Devices SoftMax Pro: 1 x .txt: Latin-1 (a raw 0xB5 for micro, or similar); tab-delimited; block markers ~End x6, Group: x3, ##BLOCKS x1, Plate: x1; first line ##BLOCKS= 6 ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥ ⇥
- Molecular Devices SoftMax Pro: 23 x .txt: UTF-16 LE (BOM); tab-delimited; block markers Plate: x2, ~End x2, ##BLOCKS x1; first line ##BLOCKS= 2
- AppBio QuantStudio Design & Analysis: 21 x .xlsx: workbook, 1 sheet(s): 'Results' 121x7
- AppBio QuantStudio Design & Analysis: 1 x .xls: workbook, 6 sheet(s): 'Raw Data' 15382x15, 'Amplification Data' 8662x8, 'Multicomponent' 8662x7, 'Results' 238x26, 'Replicate Group Result' 94x9, 'Target Call' 212x11 (OOXML bytes under a .xls name: open with engine="openpyxl" from a file object)
- AppBio QuantStudio RT-PCR: 16 x .txt: UTF-8; tab-delimited; block markers * x14, Well x3, [Sample Setup] x1, [Amplification Data] x1, [Results] x1; first line * Block Type = 96-Well Block (0.2mL)
- AppBio QuantStudio RT-PCR: 3 x .xlsx: workbook, 3 sheet(s): 'Sample Setup' 38x13, 'Amplification Data' 350x5, 'Results' 42x34
- Unchained Labs Lunatic and Stunner: 5 x .csv: UTF-8; comma-delimited; first line Plate ID,"Plate
- Unchained Labs Lunatic and Stunner: 7 x .xlsx: workbook, 1 sheet(s): 'Example_Lunatic_Plate_Reader (1' 51x12
- AppBio AbsoluteQ: 1 x .zip: zip of 3 members (.csv x3), e.g. Appbio_AbsoluteQ_fluorescence.csv, Appbio_AbsoluteQ_multichannel.csv, Appbio_AbsoluteQ_summary.csv
- AppBio AbsoluteQ: 10 x .csv: UTF-8; comma-delimited; first line Run,Date,Instrument,Plate,Group,Name,Well,Total,Dye,Target,Conc. cp/uL,95%CI,Positives
Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.
Where to go next
- Validate an ASM document against its schema with the Allotrope Simple Models (ASM) archive on this portal: every $asm.manifest PURL maps to a file in that mirror.
- Ask the assistant in the sidebar, for example: *"Which techniques report concentration, and in which units?"*
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.