Spectrophotometry instrument output with ASM: 28 exports from Thermo Fisher GENESYS 30, Thermo Fisher GENESYS On-Board, Thermo Fisher NanoDrop 8000, Thermo Fisher NanoDrop Eight and 4 more: sample notebook
A worked notebook for Spectrophotometry instrument output with ASM: 28 exports from Thermo Fisher GENESYS 30, Thermo Fisher GENESYS On-Board, Thermo Fisher NanoDrop 8000, Thermo Fisher NanoDrop Eight and 4 more, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Benchling (allotropy contributors); mirrored by Scigantic
- License
- MIT
- Size
- 56 objects, 1,222,747 bytes (28 instrument exports, 28 ASM documents)
- Format
- TSV, JSON, CSV (in sampled files)
- Stored at
- Amazon S3, s3://scigantic-allotrope-instrument-data/spectrophotometry (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Spectrophotometry: instrument output beside its ASM
UV-Vis spectrophotometer and fluorometer exports: nucleic acid and protein concentration with A260/A280 and A260/A230 ratios, full absorbance spectra, and Qubit fluorometric assays. This archive holds 28 exports from 8 instrument softwares, each beside the Allotrope Simple Model (ASM) document that the allotropy library produces from it. Files that share a name are a pair: the non-JSON file is what the instrument wrote, and the .json is its ASM.
| folder | instrument software | allotropy Vendor | exports | measurement rows |
| thermo_fisher_visionlite/ | Thermo Fisher VISIONlite | THERMO_FISHER_VISIONLITE | 6 | 26 |
| thermo_fisher_nanodrop_8000/ | Thermo Fisher NanoDrop 8000 | THERMO_FISHER_NANODROP_8000 | 5 | 163 |
| thermo_fisher_nanodrop_eight/ | Thermo Fisher NanoDrop Eight | THERMO_FISHER_NANODROP_EIGHT | 4 | 87 |
| thermo_fisher_qubit4/ | Thermo Fisher Qubit 4 | THERMO_FISHER_QUBIT4 | 4 | 18 |
| thermo_fisher_qubit_flex/ | Thermo Fisher Qubit Flex | THERMO_FISHER_QUBIT_FLEX | 4 | 71 |
| thermo_fisher_genesys30/ | Thermo Fisher GENESYS 30 | THERMO_FISHER_GENESYS30 | 2 | 2 |
| thermo_fisher_nanodrop_one/ | Thermo Fisher NanoDrop One | THERMO_FISHER_NANODROP_ONE | 2 | 48 |
| thermo_fisher_genesys_on_board/ | Thermo Fisher GENESYS On-Board | THERMO_FISHER_GENESYS_ON_BOARD | 1 | 56 |
The archive is mounted read-only at /mnt/archive. The data are allotropy's test fixtures (Benchling, MIT), mirrored unchanged.
Start here. Click the grey cell below and press Shift+Enter. It lists the Thermo Fisher VISIONlite exports with the number of measurements in each one's ASM. Then change SOFTWARE to another folder name (thermo_fisher_nanodrop_8000, thermo_fisher_nanodrop_eight, thermo_fisher_qubit4) and run it again.
import json, os, glob
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SOFTWARE = "thermo_fisher_visionlite"
def count(node):
if isinstance(node, dict):
return sum(len(v) if k in ("measurement document", "calculated data document") and isinstance(v, list) else count(v) for k, v in node.items())
return sum(count(v) for v in node) if isinstance(node, list) else 0
asm = sorted(p for p in glob.glob(f"{ROOT}/**/{SOFTWARE}/**/*.json", recursive=True) if "/input/" not in p)
print(f"{len(asm)} ASM documents for {SOFTWARE}")
for p in asm[:5]:
doc = json.load(open(p))
print(f" {count(doc):>6} measurements {os.path.basename(p)} ({doc['$asm.manifest'].rsplit('/manifests/', 1)[-1]})")
Helpers
Two small functions do most of the work below: measurement_rows finds every measurement entry in an ASM document wherever the technique nests it, and measurement_table flattens them, splitting each {"value", "unit"} quantity into a .value and a .unit column.
import json, os, glob
import pandas as pd
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
def measurement_rows(doc):
"""Every 'measurement document' / 'calculated data document' entry in an ASM document."""
rows = []
def walk(node):
if isinstance(node, dict):
for k, v in node.items():
if k in ("measurement document", "calculated data document") and isinstance(v, list):
rows.extend(v)
else:
walk(v)
elif isinstance(node, list):
for v in node:
walk(v)
walk(doc)
return rows
def measurement_table(doc):
"""One row per measurement; {"value", "unit"} quantities become <name>.value / <name>.unit columns."""
rows = measurement_rows(doc)
return pd.json_normalize(rows, sep=".") if rows else pd.DataFrame()
def asm_files(root=ROOT):
"""ASM documents under root (input/*.json of the Empower and Chromeleon adapters are exports, not ASM)."""
return sorted(p for p in glob.glob(f"{root}/**/*.json", recursive=True)
if "/input/" not in p and os.path.relpath(p, root).count("/") >= 1)
def export_for(asm_path):
"""The instrument export that shares the ASM file's stem."""
stem = asm_path[:-5].replace("/output/", "/input/")
hits = [p for p in glob.glob(glob.escape(stem) + ".*") if p != asm_path]
return hits[0] if hits else None
Convert an instrument export to ASM with allotropy
allotropy is not preinstalled. The next cell installs it into a private folder (about 30 seconds the first time): a plain pip install allotropy would downgrade jsonschema and break this notebook server. allotrope_from_file takes the export and a Vendor member, which is the folder name in upper case for every folder here except biorad_bioplex_manager (BIORAD_BIOPLEX) and unchained_labs_lunatic_stunner (UNCHAINED_LABS_LUNATIC). Pass encoding="chardet": without it allotropy assumes UTF-8 and fails on the utf-16 exports in this archive.
import os, subprocess, sys
# allotropy pins jsonschema<4.18, but this image's Jupyter server needs jsonschema>=4.18.
# A plain `pip install allotropy` downgrades it and breaks jupyter-events and
# jupyterlab-server, so install into a private folder that only this kernel uses.
LIB = os.path.expanduser("~/.allotropy-lib")
if not os.path.isdir(os.path.join(LIB, "allotropy")):
subprocess.run([sys.executable, "-m", "pip", "install", "-q", "--no-deps", "--target", LIB,
"allotropy==0.1.148", "rainbow-api==1.0.10", "jsonschema==4.17.3", "pyrsistent",
"python-calamine", "olefile", "xmltodict", "chardet", "defusedxml", "babel",
"rfc3339-validator", "lxml"], check=True)
if "jsonschema" in sys.modules and not sys.modules["jsonschema"].__file__.startswith(LIB):
raise RuntimeError("jsonschema is already imported in this kernel; restart the kernel and run this cell first")
sys.path.insert(0, LIB)
from allotropy.parser_factory import Vendor
from allotropy.to_allotrope import allotrope_from_file
EXPORT = f"{ROOT}/thermo_fisher_visionlite/Thermo_VISIONlite_example_fixed.csv"
fresh = allotrope_from_file(EXPORT, Vendor.THERMO_FISHER_VISIONLITE, encoding="chardet")
committed = json.load(open(f"{ROOT}/thermo_fisher_visionlite/Thermo_VISIONlite_example_fixed.json"))
print("manifest :", fresh["$asm.manifest"].rsplit("/manifests/", 1)[-1])
print("measurements, fresh vs committed:", len(measurement_rows(fresh)), "vs", len(measurement_rows(committed)))
Output:
manifest : spectrophotometry/BENCHLING/2023/12/spectrophotometry.manifest measurements, fresh vs committed: 3 vs 3
The ASM as a table
Every quantity carries its unit. Keep the unit column next to the value when you aggregate: the same technique can report different units from different instruments.
table = measurement_table(fresh)
ids = [c for c in table.columns if c.endswith(("location identifier", "sample identifier", "well plate identifier"))]
quantities = [c for c in table.columns if c.endswith((".value", ".unit"))]
print(table.shape[0], "rows,", table.shape[1], "columns,", len(quantities) // 2, "quantities with units")
table[(ids[:2] + quantities[:6]) or list(table.columns[:8])].head()
Output:
3 rows, 5 columns, 1 quantities with units sample document.sample identifier absorbance.value absorbance.unit 0 x 0.027 mAU 1 y 0.037 mAU 2 z 0.047 mAU
Every ASM document in this folder
The same two helpers work on every file, whatever the instrument.
import matplotlib.pyplot as plt
rows = []
for p in asm_files():
doc = json.load(open(p))
rel = os.path.relpath(p, ROOT).split("/")
rows.append({"software": rel[0], "file": rel[-1],
"release": doc["$asm.manifest"].rsplit("/manifests/", 1)[-1].rsplit("/", 1)[0],
"measurements": len(measurement_rows(doc))})
inventory = pd.DataFrame(rows)
summary = inventory.groupby("software").agg(documents=("release", "size"), measurements=("measurements", "sum")).sort_values("measurements")
fig, ax = plt.subplots(figsize=(7, 0.32 * len(summary) + 0.8), dpi=80)
summary["measurements"].plot.barh(ax=ax, color="#043874")
ax.set_xscale("log"); ax.set_xlabel("ASM measurement rows (log scale)"); ax.set_ylabel("")
plt.tight_layout(); plt.show()
summary.sort_values("measurements", ascending=False)
Output:
documents measurements software thermo_fisher_nanodrop_8000 5 163 thermo_fisher_nanodrop_eight 4 87 thermo_fisher_qubit_flex 4 71 thermo_fisher_genesys_on_board 1 56 thermo_fisher_nanodrop_one 2 48 thermo_fisher_visionlite 6 26 thermo_fisher_qubit4 4 18 thermo_fisher_genesys30 2 2
Reading the raw exports yourself
What the instruments actually write, measured from every file in this archive:
- Thermo Fisher VISIONlite: 5 x .csv: UTF-8; comma-delimited; block markers Sample Name x1; first line Sample Name,500 nm [A]
- Thermo Fisher VISIONlite: 1 x .csv: UTF-16 LE (BOM); tab-delimited; first line Sample name ⇥ A260 ⇥ Result
- Thermo Fisher NanoDrop 8000: 5 x .txt: UTF-8; tab-delimited; first line Plate ID ⇥ Well ⇥ Sample ID ⇥ User ID ⇥ Date ⇥ Time ⇥ Conc. ⇥ Units ⇥ nm 1 ⇥ Abs 1 ⇥…
- Thermo Fisher NanoDrop Eight: 4 x .txt: UTF-8; tab-delimited; first line Application: ⇥ RNA
- Thermo Fisher Qubit 4: 2 x .csv: UTF-8; comma-delimited; first line Run ID,Assay Name,Test Name,Test Date,Qubit® tube conc.,Units,Original sample conc.,Units…
- Thermo Fisher Qubit 4: 2 x .xlsx: workbook, 1 sheet(s): 'QubitData_13-12-2023_03-15-30' 3x17
- Thermo Fisher Qubit Flex: 2 x .csv: UTF-8; comma-delimited; first line Run ID,Test Date,Assay Name,Sample Name,Original Sample Conc.,Original sample conc. units…
- Thermo Fisher Qubit Flex: 2 x .xlsx: workbook, 1 sheet(s): 'QubitData_10-10-2023_12-34-44' 17x24
- Thermo Fisher GENESYS 30: 1 x .csv: UTF-8; comma-delimited; first line Scan,Scan_20230914_160142.csv
- Thermo Fisher GENESYS 30: 1 x .tsv: UTF-8; tab-delimited; first line Scan ⇥ Scan_20230914_160142.tsv ⇥
- Thermo Fisher NanoDrop One: 1 x .csv: UTF-8; comma-delimited; first line Date,Sample Name,Nucleic Acid(ng/uL),A260/A280,A260/A230,A260,A280,Nucleic Acid Factor,Ba…
- Thermo Fisher NanoDrop One: 1 x .xlsx: workbook, 1 sheet(s): 'dsDNA 7_19_2023 3_27_29 PM' 7x24
- Thermo Fisher GENESYS On-Board: 1 x .csv: UTF-16 LE (BOM); tab-delimited; block markers Well x1; first line Well Position ⇥ Absorbance ⇥ Result
Biacore .blr and .bme files are OLE2 containers of binary run data that only allotropy reads; spreadsheets open with pd.read_excel, and a .xls that is really a zip needs engine="openpyxl" on a file object.
Where to go next
- Validate an ASM document against its schema with the Allotrope Simple Models (ASM) archive on this portal: every $asm.manifest PURL maps to a file in that mirror.
- Ask the assistant in the sidebar, for example: *"Which quantities and units does each instrument software report?"*
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.