Allotrope Simple Models (ASM): analytical instrument data schemas and sample data: sample notebook
A worked notebook for Allotrope Simple Models (ASM): analytical instrument data schemas and sample data, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Allotrope Foundation
- License
- CC-BY-NC-4.0 (non-commercial) / CC-BY-ND-4.0 (commercial) / Allotrope member license
- Size
- 2,441 files
- Format
- JSON (in sampled files)
- Stored at
- Amazon S3, s3://scigantic-allotrope-asm (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Allotrope Simple Models (ASM)
ASM is the JSON standard for analytical laboratory instrument data, published by the Allotrope Foundation. It gives plate readers, chromatographs, mass spectrometers and 60-odd other instrument classes one vendor-neutral document shape.
A verbatim mirror of the repository is mounted read-only at ~/archive, so files stream on demand and nothing is copied at launch. Four directories:
| Directory | What it holds |
| json-schemas/ | The JSON Schemas that define each technique, plus shared core and QUDT unit schemas |
| test/ | Real sample instrument data. One or more validated example documents per technique |
| manifests/ | Binds an instance to its schema, SHACL shapes, and vocabulary |
| diagrams/ | Excel tabular models (retired as a modeling tool in 2024, kept for reference) |
This notebook walks the path a parser author actually takes: find a document, resolve which schema governs it, validate it offline, then pull the measurements into a DataFrame.
import json, os, re
from pathlib import Path
ARCHIVE = Path(os.environ.get("SCIGANTIC_MOUNT_PATH", Path.home() / "archive"))
# The archive is an S3 FUSE mount, so reading one file is quick but walking the
# whole tree is not: a recursive glob over test/ costs ~30s, and stat()-ing every
# sample another ~35s. Listing a single directory is ~0.07s. So list what you
# need and open files directly; never rglob the archive.
#
# `core` is shared building blocks (quantity values, datacubes, document
# hierarchy), not an instrument, so it is excluded from the technique list.
techniques = sorted(p.name for p in (ARCHIVE / "json-schemas/adm").iterdir()
if p.is_dir() and p.name != "core")
print(f"archive root: {ARCHIVE}")
print(f"{len(techniques)} instrument techniques, 490 sample data files\n")
for i in range(0, len(techniques), 4):
print(" " + "".join(f"{t:<27}" for t in techniques[i:i + 4]))
Output:
archive root: /mnt/archive 66 instrument techniques, 490 sample data files absorbance automated-reactors balance bga binding-affinity bulk-density cell-counting cell-culture-analyzer chromatography code-reader conductance conductivity disintegration dsc dvs electronic-lab-notebook electronic-spectrometry electrophoresis flow-cytometry fluorescence foam-height foam-qualification fplc fraction-collection ftir gas-chromatography gc-ms gloss hot-tack impedance lc-ms light-obscuration liquid-chromatography loss-on-drying luminescence mass-spectrometry mercury-porosimetry metabolite-analyzer multi-analyte-profiling nephelometry nmr optical-imaging optical-microscopy osmolality oven-kf pcr ph plate-reader ...
Every purl.allotrope.org URL is mirrored in this repo
ASM documents and schemas refer to each other by PURL, for example http://purl.allotrope.org/json-schemas/adm/core/REC/2026/07/core.schema. Those look like web addresses, but every one of them corresponds to a file in the tree you already have.
The rule is one line: drop the http://purl.allotrope.org/ prefix and append .json.
That single mapping is what lets you resolve and validate ASM entirely offline. It is also how you answer "which schema governs this document": read the instance's $asm.manifest, resolve it, and follow its json-schemas entry.
PURL = "http://purl.allotrope.org/"
def purl_to_path(uri: str) -> Path:
"""Map an Allotrope PURL to its file in this repository."""
if not uri.startswith(PURL):
raise ValueError(f"not an Allotrope PURL: {uri}")
return ARCHIVE / (uri[len(PURL):].split("#")[0] + ".json")
instance_path = ARCHIVE / "test/adm/plate-reader/REC/2025/12/luminescence-profile.json"
instance = json.loads(instance_path.read_text())
manifest_uri = instance["$asm.manifest"]
manifest = json.loads(purl_to_path(manifest_uri).read_text())
schema_uri = manifest["json-schemas"][0]
schema_path = purl_to_path(schema_uri)
embed_path = schema_path.with_name(schema_path.name.replace(".schema.json", ".embed.schema.json"))
print(f"instance {instance_path.relative_to(ARCHIVE)}")
print(f" $asm.manifest -> {manifest_uri}")
print(f"manifest {purl_to_path(manifest_uri).relative_to(ARCHIVE)}")
print(f" json-schemas[0] -> {schema_uri}")
print()
print(f"referencing schema {schema_path.relative_to(ARCHIVE)}")
print(f"embedded schema {embed_path.relative_to(ARCHIVE)}")
Output:
instance test/adm/plate-reader/REC/2025/12/luminescence-profile.json $asm.manifest -> http://purl.allotrope.org/manifests/plate-reader/REC/2025/12/plate-reader.manifest manifest manifests/plate-reader/REC/2025/12/plate-reader.manifest.json json-schemas[0] -> http://purl.allotrope.org/json-schemas/adm/plate-reader/REC/2025/12/plate-reader.schema referencing schema json-schemas/adm/plate-reader/REC/2025/12/plate-reader.schema.json embedded schema json-schemas/adm/plate-reader/REC/2025/12/plate-reader.embed.schema.json
Validating a document
Each release ships the same schema twice:
- .schema.json keeps its references to other files as external $refs.
- .embed.schema.json inlines them into one self-contained document.
Reach for the .embed variant. Across the sample corpus it validates without dereferencing a single external $ref, so it works offline with no extra setup.
import jsonschema
from referencing import Registry, Resource
from referencing.jsonschema import DRAFT202012
import referencing.exceptions
def retrieve(uri: str) -> Resource:
"""Resolve an Allotrope PURL from the local clone instead of over the network."""
try:
contents = json.loads(purl_to_path(uri).read_text())
except (ValueError, OSError):
raise referencing.exceptions.NoSuchResource(ref=uri)
return Resource.from_contents(contents, default_specification=DRAFT202012)
registry = Registry(retrieve=retrieve)
def validate(instance: dict, schema_file: Path, registry=registry):
schema = json.loads(schema_file.read_text())
cls = jsonschema.validators.validator_for(schema)
validator = cls(schema, registry=registry) if registry is not None else cls(schema)
return list(validator.iter_errors(instance))
errors = validate(instance, embed_path)
print(f"{instance_path.name} against {embed_path.name}")
print(f"validation errors: {len(errors)}")
for e in errors[:5]:
print(f" at {list(e.absolute_path)}: {e.message[:120]}")
Output:
luminescence-profile.json against plate-reader.embed.schema.json validation errors: 0
Why pass a registry at all
If you use the .schema.json variant, its external $refs have to be resolved somehow, and what jsonschema does by default is the trap:
- With the local registry the refs resolve from ~/archive. Fast and offline.
- With an empty Registry() it raises Unresolvable, naming the ref it could not find.
- With no registry at all it quietly downloads each ref from purl.allotrope.org. It
still returns the right answer, but it needs network access, takes seconds instead of milliseconds, and jsonschema warns that the behavior is deprecated and "will shortly become an error".
The cell below shows the first two. The third is left out on purpose so this notebook never depends on outbound network access.
import time
t0 = time.perf_counter()
errs = validate(instance, schema_path)
local_ms = (time.perf_counter() - t0) * 1000
print(f"local registry : {len(errs)} errors in {local_ms:.0f} ms")
try:
validate(instance, schema_path, registry=Registry())
print("empty registry : resolved (this schema needed no external refs)")
except Exception as e:
print(f"empty registry : {type(e).__name__}")
print(f" {str(e).splitlines()[0][:100]}")
external = sorted({r for r in re.findall(r'"\$ref"\s*:\s*"([^"]+)"', schema_path.read_text())
if not r.startswith("#")})
print(f"\n{schema_path.name} declares {len(external)} external $refs, e.g.")
for r in external[:3]:
print(f" {r}")
Output:
local registry : 0 errors in 2617 ms
empty registry : _WrappedReferencingError
Unresolvable: http://purl.allotrope.org/json-schemas/adm/core/REC/2025/12/hierarchy.schema#/$defs/te
plate-reader.schema.json declares 26 external $refs, e.g.
http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-area-scan-detector.schema#/$defs/measurementDocumentItems
http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-cube-detector.schema#/$defs/measurementDocumentItems
http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-point-detector.schema#/$defs/measurementDocumentItems
Measurements: datacubes and quantity values
Two ASM shapes surprise people writing their first parser.
Datacubes are not tabular. A cube is {cube-structure, data}, where cube-structure names and units each axis and data holds parallel arrays in the same order. You zip them together to get columns.
Scalars are objects, not bare numbers: {"value": 4.5, "unit": "mm"}. A plain pd.json_normalize will happily flatten those into ...value / ...unit column pairs and lose the association, so pull them out deliberately.
import pandas as pd
def cube_to_frame(cube: dict) -> pd.DataFrame:
"""Zip an ASM datacube's parallel arrays into a labelled, unit-carrying DataFrame."""
cols = {}
for kind in ("dimensions", "measures"):
for spec, values in zip(cube["cube-structure"][kind], cube["data"][kind]):
label = spec.get("concept", kind)
unit = spec.get("unit")
cols[f"{label} ({unit})" if unit else label] = values
return pd.DataFrame(cols)
doc = instance["plate reader aggregate document"]["plate reader document"][0]
meas = doc["measurement aggregate document"]["measurement document"][0]
cube_key = next(k for k in meas if k.endswith("data cube"))
df = cube_to_frame(meas[cube_key])
print(f"{cube_key}: {len(df)} rows")
print(df.head(8).to_string(index=False))
device = meas["device control aggregate document"]["device control document"][0]
print("\ninstrument settings (quantity values):")
for k, v in device.items():
if isinstance(v, dict) and "value" in v:
print(f" {k}: {v['value']} {v.get('unit', '')}")
Output:
luminescence profile data cube: 11 rows
elapsed time (s) luminescence (RLU)
0 91115
60 131640
120 168250
180 197430
240 219830
300 236950
360 250150
430 260460
instrument settings (quantity values):
detector distance setting (plate reader): 4.5 mm
integration time: 1 s
Finding your way around the rest
Artifacts are versioned as / / / /, where maturity is REC (Recommended, public), CR (Candidate Release, requires Allotrope Foundation membership), or WD (Working Draft). Several dated releases of one technique coexist, so pin the version you target rather than assuming the newest.
Two layout details will bite generic globbing code: gc-ms-mrm omits the maturity level entirely, and moisture-analysis has sample data but no schema directory.
# Build the index from directory NAMES only — no rglob, no stat() per file.
# Same answer as walking every file, but ~7s instead of ~65s over FUSE.
rows = []
for tech_dir in sorted(p for p in (ARCHIVE / "test/adm").iterdir() if p.is_dir()):
# gc-ms-mrm stores releases directly under the technique, with no maturity level.
rec = tech_dir / "REC"
base = rec if rec.is_dir() else tech_dir
years = [y for y in base.iterdir() if y.is_dir()]
if not years:
continue
year = max(years, key=lambda p: p.name)
months = [m for m in year.iterdir() if m.is_dir()]
if not months:
continue
month = max(months, key=lambda p: p.name)
files = sorted(f.name for f in month.iterdir() if f.suffix == ".json")
if files:
rows.append({"technique": tech_dir.name,
"latest release": f"{year.name}/{month.name}",
"sample file": files[0]})
index = pd.DataFrame(rows)
print(f"{len(index)} techniques with sample data\n")
print(index.head(20).to_string(index=False))
# The outliers, so you don't open one by accident. These are the largest files
# in the corpus across all releases, not just the latest.
print("""
Largest files. Stream these with ijson rather than json.load:
148.8 MB test/adm/automated-reactors/REC/2022/03/automated-reactors.json
135.7 MB test/adm/automated-reactors/REC/2026/07/automated-reactors.json
18.5 MB test/adm/mass-spectrometry/REC/2022/06/mass-spectrometry.tabular.json
The 8 automated-reactors releases are ~1.2 GB together; upstream stores them via
git-lfs and this archive materializes them. The median sample is only ~7 KB.""")
Output:
62 techniques with sample data
technique latest release sample file
absorbance 2026/07 absorbance-area-scan.json
automated-reactors 2026/07 automated-reactors.json
balance 2026/07 balance.json
bga 2024/12 bga.json
binding-affinity 2026/07 binding-affinity-analyzer.json
bulk-density 2024/12 bulk-density.json
cell-counting 2026/07 cell-counting.json
cell-culture-analyzer 2024/12 cell-culture-analyzer.json
chromatography 2026/07 chromatography-column.json
code-reader 2024/12 code-reader.json
conductivity 2024/12 conductivity.json
disintegration 2024/12 disintegration.json
...
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.