Allotrope Simple Models (ASM): analytical instrument data schemas and sample data: sample notebook

A worked notebook for Allotrope Simple Models (ASM): analytical instrument data schemas and sample data, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Allotrope Foundation
License
CC-BY-NC-4.0 (non-commercial) / CC-BY-ND-4.0 (commercial) / Allotrope member license
Size
2,441 files
Format
JSON (in sampled files)
Stored at
Amazon S3, s3://scigantic-allotrope-asm (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Allotrope Simple Models (ASM)

ASM is the JSON standard for analytical laboratory instrument data, published by the Allotrope Foundation. It gives plate readers, chromatographs, mass spectrometers and 60-odd other instrument classes one vendor-neutral document shape.

A verbatim mirror of the repository is mounted read-only at ~/archive, so files stream on demand and nothing is copied at launch. Four directories:

DirectoryWhat it holds
json-schemas/The JSON Schemas that define each technique, plus shared core and QUDT unit schemas
test/Real sample instrument data. One or more validated example documents per technique
manifests/Binds an instance to its schema, SHACL shapes, and vocabulary
diagrams/Excel tabular models (retired as a modeling tool in 2024, kept for reference)

This notebook walks the path a parser author actually takes: find a document, resolve which schema governs it, validate it offline, then pull the measurements into a DataFrame.

import json, os, re
from pathlib import Path

ARCHIVE = Path(os.environ.get("SCIGANTIC_MOUNT_PATH", Path.home() / "archive"))

# The archive is an S3 FUSE mount, so reading one file is quick but walking the
# whole tree is not: a recursive glob over test/ costs ~30s, and stat()-ing every
# sample another ~35s. Listing a single directory is ~0.07s. So list what you
# need and open files directly; never rglob the archive.
#
# `core` is shared building blocks (quantity values, datacubes, document
# hierarchy), not an instrument, so it is excluded from the technique list.
techniques = sorted(p.name for p in (ARCHIVE / "json-schemas/adm").iterdir()
                    if p.is_dir() and p.name != "core")

print(f"archive root: {ARCHIVE}")
print(f"{len(techniques)} instrument techniques, 490 sample data files\n")
for i in range(0, len(techniques), 4):
    print("  " + "".join(f"{t:<27}" for t in techniques[i:i + 4]))

Output:

archive root: /mnt/archive
66 instrument techniques, 490 sample data files

  absorbance                 automated-reactors         balance                    bga                        
  binding-affinity           bulk-density               cell-counting              cell-culture-analyzer      
  chromatography             code-reader                conductance                conductivity               
  disintegration             dsc                        dvs                        electronic-lab-notebook    
  electronic-spectrometry    electrophoresis            flow-cytometry             fluorescence               
  foam-height                foam-qualification         fplc                       fraction-collection        
  ftir                       gas-chromatography         gc-ms                      gloss                      
  hot-tack                   impedance                  lc-ms                      light-obscuration          
  liquid-chromatography      loss-on-drying             luminescence               mass-spectrometry          
  mercury-porosimetry        metabolite-analyzer        multi-analyte-profiling    nephelometry               
  nmr                        optical-imaging            optical-microscopy         osmolality                 
  oven-kf                    pcr                        ph                         plate-reader               
...

Every purl.allotrope.org URL is mirrored in this repo

ASM documents and schemas refer to each other by PURL, for example http://purl.allotrope.org/json-schemas/adm/core/REC/2026/07/core.schema. Those look like web addresses, but every one of them corresponds to a file in the tree you already have.

The rule is one line: drop the http://purl.allotrope.org/ prefix and append .json.

That single mapping is what lets you resolve and validate ASM entirely offline. It is also how you answer "which schema governs this document": read the instance's $asm.manifest, resolve it, and follow its json-schemas entry.

PURL = "http://purl.allotrope.org/"

def purl_to_path(uri: str) -> Path:
    """Map an Allotrope PURL to its file in this repository."""
    if not uri.startswith(PURL):
        raise ValueError(f"not an Allotrope PURL: {uri}")
    return ARCHIVE / (uri[len(PURL):].split("#")[0] + ".json")

instance_path = ARCHIVE / "test/adm/plate-reader/REC/2025/12/luminescence-profile.json"
instance = json.loads(instance_path.read_text())

manifest_uri = instance["$asm.manifest"]
manifest = json.loads(purl_to_path(manifest_uri).read_text())
schema_uri = manifest["json-schemas"][0]

schema_path = purl_to_path(schema_uri)
embed_path = schema_path.with_name(schema_path.name.replace(".schema.json", ".embed.schema.json"))

print(f"instance   {instance_path.relative_to(ARCHIVE)}")
print(f"  $asm.manifest -> {manifest_uri}")
print(f"manifest   {purl_to_path(manifest_uri).relative_to(ARCHIVE)}")
print(f"  json-schemas[0] -> {schema_uri}")
print()
print(f"referencing schema  {schema_path.relative_to(ARCHIVE)}")
print(f"embedded schema     {embed_path.relative_to(ARCHIVE)}")

Output:

instance   test/adm/plate-reader/REC/2025/12/luminescence-profile.json
  $asm.manifest -> http://purl.allotrope.org/manifests/plate-reader/REC/2025/12/plate-reader.manifest
manifest   manifests/plate-reader/REC/2025/12/plate-reader.manifest.json
  json-schemas[0] -> http://purl.allotrope.org/json-schemas/adm/plate-reader/REC/2025/12/plate-reader.schema

referencing schema  json-schemas/adm/plate-reader/REC/2025/12/plate-reader.schema.json
embedded schema     json-schemas/adm/plate-reader/REC/2025/12/plate-reader.embed.schema.json

Validating a document

Each release ships the same schema twice:

Reach for the .embed variant. Across the sample corpus it validates without dereferencing a single external $ref, so it works offline with no extra setup.

import jsonschema
from referencing import Registry, Resource
from referencing.jsonschema import DRAFT202012
import referencing.exceptions

def retrieve(uri: str) -> Resource:
    """Resolve an Allotrope PURL from the local clone instead of over the network."""
    try:
        contents = json.loads(purl_to_path(uri).read_text())
    except (ValueError, OSError):
        raise referencing.exceptions.NoSuchResource(ref=uri)
    return Resource.from_contents(contents, default_specification=DRAFT202012)

registry = Registry(retrieve=retrieve)

def validate(instance: dict, schema_file: Path, registry=registry):
    schema = json.loads(schema_file.read_text())
    cls = jsonschema.validators.validator_for(schema)
    validator = cls(schema, registry=registry) if registry is not None else cls(schema)
    return list(validator.iter_errors(instance))

errors = validate(instance, embed_path)
print(f"{instance_path.name} against {embed_path.name}")
print(f"validation errors: {len(errors)}")
for e in errors[:5]:
    print(f"  at {list(e.absolute_path)}: {e.message[:120]}")

Output:

luminescence-profile.json against plate-reader.embed.schema.json
validation errors: 0
Why pass a registry at all

If you use the .schema.json variant, its external $refs have to be resolved somehow, and what jsonschema does by default is the trap:

still returns the right answer, but it needs network access, takes seconds instead of milliseconds, and jsonschema warns that the behavior is deprecated and "will shortly become an error".

The cell below shows the first two. The third is left out on purpose so this notebook never depends on outbound network access.

import time

t0 = time.perf_counter()
errs = validate(instance, schema_path)
local_ms = (time.perf_counter() - t0) * 1000
print(f"local registry   : {len(errs)} errors in {local_ms:.0f} ms")

try:
    validate(instance, schema_path, registry=Registry())
    print("empty registry   : resolved (this schema needed no external refs)")
except Exception as e:
    print(f"empty registry   : {type(e).__name__}")
    print(f"                   {str(e).splitlines()[0][:100]}")

external = sorted({r for r in re.findall(r'"\$ref"\s*:\s*"([^"]+)"', schema_path.read_text())
                   if not r.startswith("#")})
print(f"\n{schema_path.name} declares {len(external)} external $refs, e.g.")
for r in external[:3]:
    print(f"  {r}")

Output:

local registry   : 0 errors in 2617 ms
empty registry   : _WrappedReferencingError
                   Unresolvable: http://purl.allotrope.org/json-schemas/adm/core/REC/2025/12/hierarchy.schema#/$defs/te

plate-reader.schema.json declares 26 external $refs, e.g.
  http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-area-scan-detector.schema#/$defs/measurementDocumentItems
  http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-cube-detector.schema#/$defs/measurementDocumentItems
  http://purl.allotrope.org/json-schemas/adm/absorbance/REC/2025/12/absorbance-point-detector.schema#/$defs/measurementDocumentItems

Measurements: datacubes and quantity values

Two ASM shapes surprise people writing their first parser.

Datacubes are not tabular. A cube is {cube-structure, data}, where cube-structure names and units each axis and data holds parallel arrays in the same order. You zip them together to get columns.

Scalars are objects, not bare numbers: {"value": 4.5, "unit": "mm"}. A plain pd.json_normalize will happily flatten those into ...value / ...unit column pairs and lose the association, so pull them out deliberately.

import pandas as pd

def cube_to_frame(cube: dict) -> pd.DataFrame:
    """Zip an ASM datacube's parallel arrays into a labelled, unit-carrying DataFrame."""
    cols = {}
    for kind in ("dimensions", "measures"):
        for spec, values in zip(cube["cube-structure"][kind], cube["data"][kind]):
            label = spec.get("concept", kind)
            unit = spec.get("unit")
            cols[f"{label} ({unit})" if unit else label] = values
    return pd.DataFrame(cols)

doc = instance["plate reader aggregate document"]["plate reader document"][0]
meas = doc["measurement aggregate document"]["measurement document"][0]

cube_key = next(k for k in meas if k.endswith("data cube"))
df = cube_to_frame(meas[cube_key])
print(f"{cube_key}: {len(df)} rows")
print(df.head(8).to_string(index=False))

device = meas["device control aggregate document"]["device control document"][0]
print("\ninstrument settings (quantity values):")
for k, v in device.items():
    if isinstance(v, dict) and "value" in v:
        print(f"  {k}: {v['value']} {v.get('unit', '')}")

Output:

luminescence profile data cube: 11 rows
 elapsed time (s)  luminescence (RLU)
                0               91115
               60              131640
              120              168250
              180              197430
              240              219830
              300              236950
              360              250150
              430              260460

instrument settings (quantity values):
  detector distance setting (plate reader): 4.5 mm
  integration time: 1 s

Finding your way around the rest

Artifacts are versioned as / / / /, where maturity is REC (Recommended, public), CR (Candidate Release, requires Allotrope Foundation membership), or WD (Working Draft). Several dated releases of one technique coexist, so pin the version you target rather than assuming the newest.

Two layout details will bite generic globbing code: gc-ms-mrm omits the maturity level entirely, and moisture-analysis has sample data but no schema directory.

# Build the index from directory NAMES only — no rglob, no stat() per file.
# Same answer as walking every file, but ~7s instead of ~65s over FUSE.
rows = []
for tech_dir in sorted(p for p in (ARCHIVE / "test/adm").iterdir() if p.is_dir()):
    # gc-ms-mrm stores releases directly under the technique, with no maturity level.
    rec = tech_dir / "REC"
    base = rec if rec.is_dir() else tech_dir
    years = [y for y in base.iterdir() if y.is_dir()]
    if not years:
        continue
    year = max(years, key=lambda p: p.name)
    months = [m for m in year.iterdir() if m.is_dir()]
    if not months:
        continue
    month = max(months, key=lambda p: p.name)
    files = sorted(f.name for f in month.iterdir() if f.suffix == ".json")
    if files:
        rows.append({"technique": tech_dir.name,
                     "latest release": f"{year.name}/{month.name}",
                     "sample file": files[0]})

index = pd.DataFrame(rows)
print(f"{len(index)} techniques with sample data\n")
print(index.head(20).to_string(index=False))

# The outliers, so you don't open one by accident. These are the largest files
# in the corpus across all releases, not just the latest.
print("""
Largest files. Stream these with ijson rather than json.load:
   148.8 MB  test/adm/automated-reactors/REC/2022/03/automated-reactors.json
   135.7 MB  test/adm/automated-reactors/REC/2026/07/automated-reactors.json
    18.5 MB  test/adm/mass-spectrometry/REC/2022/06/mass-spectrometry.tabular.json
The 8 automated-reactors releases are ~1.2 GB together; upstream stores them via
git-lfs and this archive materializes them. The median sample is only ~7 KB.""")

Output:

62 techniques with sample data

              technique latest release                            sample file
             absorbance        2026/07              absorbance-area-scan.json
     automated-reactors        2026/07                automated-reactors.json
                balance        2026/07                           balance.json
                    bga        2024/12                               bga.json
       binding-affinity        2026/07         binding-affinity-analyzer.json
           bulk-density        2024/12                      bulk-density.json
          cell-counting        2026/07                     cell-counting.json
  cell-culture-analyzer        2024/12             cell-culture-analyzer.json
         chromatography        2026/07             chromatography-column.json
            code-reader        2024/12                       code-reader.json
           conductivity        2024/12                      conductivity.json
         disintegration        2024/12                    disintegration.json
...

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.