Electrophoresis instrument output with ASM: 3 exports from Agilent TapeStation Analysis: sample notebook

A worked notebook for Electrophoresis instrument output with ASM: 3 exports from Agilent TapeStation Analysis, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Benchling (allotropy contributors); mirrored by Scigantic
License
MIT
Size
6 objects, 444,464 bytes (3 instrument exports, 3 ASM documents)
Format
JSON (in sampled files)
Stored at
Amazon S3, s3://scigantic-allotrope-instrument-data/electrophoresis (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Electrophoresis: instrument output beside its ASM

Automated electrophoresis exports from the Agilent TapeStation: per-sample peak tables, sizes, concentrations and integrity scores (RIN, DIN) with the electropherogram regions. This archive holds 3 exports from 1 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.

folderinstrument softwareallotropy Vendorexportsmeasurement rows
agilent_tapestation_analysis/Agilent TapeStation AnalysisAGILENT_TAPESTATION_ANALYSIS3344

The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.

Start here. Click the grey cell below and press Shift+Enter. It lists the Agilent TapeStation Analysis exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name () and run it again.

import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "agilent_tapestation_analysis"

def count(node):
    if isinstance(node, dict):
        return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
    return sum(count(v) for v in node) if isinstance(node, list) else 0

asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
    doc = json.load(open(p))
    print(f"  {count(doc):>6} measurements  {os.path.basename(p)}  ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")

Helpers

Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.

import json, os, glob
import pandas as pd

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")

def measurement_rows(doc):
    """Every 'measurement document' / 'calculated data document' entry in an ASM document."""
    rows = []
    def walk(node):
        if isinstance(node, dict):
            for k, v in node.items():
                if k in ("measurement document", "calculated data document") and isinstance(v, list):
                    rows.extend(v)
                else:
                    walk(v)
        elif isinstance(node, list):
            for v in node:
                walk(v)
    walk(doc)
    return rows

def measurement_table(doc):
    """One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
    rows = measurement_rows(doc)
    return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()

def asm_files(root=ROOT):
    """ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
    return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
                  if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)

def export_for(asm_path):
    """The instrument export that shares the ASM file's stem."""
    stem = asm_path[:-5].replace("/output/", "/input/")
    hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
    return hits[0] if hits else None

Convert an instrument export to ASM with allotropy

allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8.

import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
    subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
                    "allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
                    "python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
                    "rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
    raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file

EXPORT = f"{ROOT}/agilent_tapestation_analysis/agilent_tapestation_analysis_example_03.xml"
fresh = allotrope_from_file(EXPORT, Vendor.AGILENT_TAPESTATION_ANALYSIS, encoding="chardet")
committed = json.load(open(f"{ROOT}/agilent_tapestation_analysis/agilent_tapestation_analysis_example_03.json"))

print("manifest  :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))

Output:

manifest  : electrophoresis/BENCHLING/2024/09/electrophoresis.manifest
measurements, fresh vs committed: 116 vs 116

The ASM as a table

Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.

table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()

Output:

116 rows, 14 columns, 2 quantities with units
  sample document.sample identifier sample document.location identifier  \
0                               NaN                                 NaN   
1                               NaN                                 NaN   
2                               NaN                                 NaN   
3                               NaN                                 NaN   
4                               NaN                                 NaN   

   calculated result.value calculated result.unit  \
0                   2970.0             (unitless)   
1                    275.0             (unitless)   
2                    275.0             (unitless)   
3                     82.4             (unitless)   
4                  12100.0             (unitless)   

...

Every ASM document in this folder

The same two helpers work on every file, whatever the instrument.

import matplotlib.pyplot as plt

rows = []
for p in asm_files():
    doc = json.load(open(p))
    rel = os.path.relpath(p, ROOT).split("/")
    rows.append({"software": rel[0], "file": rel[-1],
                 "release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
                 "measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("software").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")

fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)

Output:

                              documents  measurements
software                                             
agilent_tapestation_analysis          3           344

Reading the raw exports yourself

What the instruments actually write, measured from every file in this archive:

Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.

Where to go next

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.