Laboratory instrument output files with their ASM conversions: 343 exports from 53 instrument softwares across 12 techniques: sample notebook

A worked notebook for Laboratory instrument output files with their ASM conversions: 343 exports from 53 instrument softwares across 12 techniques, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Benchling (allotropy contributors); mirrored by Scigantic
License
MIT
Size
687 objects, 769,830,126 bytes
Format
JSON, CSV, TSV (in sampled files)
Stored at
Amazon S3, s3://scigantic-allotrope-instrument-data (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Laboratory instrument output and its ASM

A repository of real laboratory instrument exports, one folder per technique and one per instrument software. This archive holds 343 exports from 53 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.

technique folderexportsinstrument softwares
binding-affinity/153
cell-counting/326
electrophoresis/31
flow-cytometry/52
liquid-chromatography/144
liquid-handler/103
multi-analyte-profiling/123
plate-reader/14014
solution-analyzer/223
spectrophotometry/288
dpcr/152
qpcr/474

The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.

Start here. Click the grey cell below and press Shift+Enter. It lists the Molecular Devices SoftMax Pro exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (agilent_gen5, appbio_quantstudio_designandanalysis, appbio_quantstudio) and run it again.

import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "moldev_softmax_pro"

def count(node):
    if isinstance(node, dict):
        return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
    return sum(count(v) for v in node) if isinstance(node, list) else 0

asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
    doc = json.load(open(p))
    print(f"  {count(doc):>6} measurements  {os.path.basename(p)}  ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")

Helpers

Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.

import json, os, glob
import pandas as pd

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")

def measurement_rows(doc):
    """Every 'measurement document' / 'calculated data document' entry in an ASM document."""
    rows = []
    def walk(node):
        if isinstance(node, dict):
            for k, v in node.items():
                if k in ("measurement document", "calculated data document") and isinstance(v, list):
                    rows.extend(v)
                else:
                    walk(v)
        elif isinstance(node, list):
            for v in node:
                walk(v)
    walk(doc)
    return rows

def measurement_table(doc):
    """One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
    rows = measurement_rows(doc)
    return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()

def asm_files(root=ROOT):
    """ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
    return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
                  if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)

def export_for(asm_path):
    """The instrument export that shares the ASM file's stem."""
    stem = asm_path[:-5].replace("/output/", "/input/")
    hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
    return hits[0] if hits else None

Convert an instrument export to ASM with allotropy

allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8 and fails on the latin-1, utf-16, utf-8-sig exports in this archive.

import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
                    "allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
                    "python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
                    "rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
    raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file

EXPORT = f"{ROOT}/plate-reader/moldev_softmax_pro/no_measurements.txt"
fresh = allotrope_from_file(EXPORT, Vendor.MOLDEV_SOFTMAX_PRO, encoding="chardet")
committed = json.load(open(f"{ROOT}/plate-reader/moldev_softmax_pro/no_measurements.json"))

print("manifest  :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))

Output:

manifest  : plate-reader/REC/2025/03/plate-reader.manifest
measurements, fresh vs committed: 6 vs 6

The ASM as a table

Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.

table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()

Output:

6 rows, 10 columns, 2 quantities with units
  sample document.location identifier sample document.sample identifier  \
0                                  A1                         PLate1 A1   
1                                  A2                         PLate1 A2   
2                                  A3                         PLate1 A3   
3                                  B1                         PLate1 B1   
4                                  B2                         PLate1 B2   

   compartment temperature.value compartment temperature.unit  \
0                           21.9                         degC   
1                           21.9                         degC   
2                           21.9                         degC   
3                           21.9                         degC   
4                           21.9                         degC   

...

Every ASM document in the archive

The same two helpers work on every file, across all twelve techniques, whatever the instrument.

import matplotlib.pyplot as plt

rows = []
for p in asm_files():
    doc = json.load(open(p))
    rel = os.path.relpath(p, ROOT).split("/")
    rows.append({"technique": rel[0], "software": rel[1],
                 "release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
                 "measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("technique").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")

fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)

Output:

                         documents  measurements
technique                                       
plate-reader                   141         50369
qpcr                            47         17978
binding-affinity                15         16599
multi-analyte-profiling         12          2139
dpcr                            15          1063
liquid-handler                  10           625
solution-analyzer               22           573
cell-counting                   32           540
spectrophotometry               28           471
electrophoresis                  3           344
liquid-chromatography           14           104
flow-cytometry                   5            21

Reading the raw exports yourself

What the instruments actually write, measured from every file in this archive:

Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.

Where to go next

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.