Enveda-180: 1.16 million timsTOF MS/MS spectra with CCS and retention time for 184,330 drug-like molecules: sample notebook

A worked notebook for Enveda-180: 1.16 million timsTOF MS/MS spectra with CCS and retention time for 184,330 drug-like molecules, with 11 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Enveda (Krettler C.A., Kind T., DeBloois E., Killian M., Prince J. et al.); mirrored by Scigantic
License
CC-BY-4.0
Size
9 objects (8 data files + MIRROR.md), 8,920 MB, sizes and md5 identical to the Zenodo record
Format
JSON Lines (in sampled files)
Stored at
Amazon S3, s3://scigantic-zenodo-20436851 (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Start here. Click the grey cell below and press Shift+Enter. It looks one molecule up in four of the five archives in a few seconds. Then change INCHIKEY to another compound (try caffeine, RYYVLZVUVIJVGH-UHFFFAOYSA-N) and run it again.

INCHIKEY = 'REFJWTPEDVJJIY-UHFFFAOYSA-N'   # quercetin; change me

import pyarrow.parquet as pq, pyarrow.fs as pafs
S3 = pafs.S3FileSystem(anonymous=True, region='us-east-1')
def lookup(path, key_col, cols):
    return pq.read_table(path, filesystem=S3, columns=cols, filters=[(key_col, '=', INCHIKEY)]).to_pylist()

coco = lookup('scigantic-coconut/coconut-10-2026.parquet', 'standard_inchi_key',
              ['identifier', 'name', 'molecular_formula', 'np_likeness', 'np_classifier_class', 'organisms'])
mb = lookup('scigantic-massbank/massbank_open_spectra.parquet', 'inchikey', ['accession', 'precursor_type'])
gn = lookup('scigantic-gnps-libraries/gnps_community_spectra.parquet', 'inchikey', ['spectrum_id', 'adduct'])
for r in coco:
    print(f"COCONUT  {r['identifier']}  {r['name']}  {r['molecular_formula']}  NP-likeness {r['np_likeness']}"
          f"  class {r['np_classifier_class']}  organisms {len((r['organisms'] or '').split('|')) if r['organisms'] else 0}")
print(f'MassBank {len(mb)} open-licence reference spectra;  GNPS {len(gn)} community-library spectra')

One natural product through five open mass-spec libraries

Identifying a small molecule from its MS/MS spectrum means matching it against something: a reference spectrum, a structure database, a list of what organisms make. This notebook follows quercetin, a flavonol found in thousands of plants, through five open archives, then turns to Enveda-180 and measures how little of it overlaps with natural-product chemical space.

archivewhat it holdshere as
Enveda-180 (~/archive)1,158,293 timsTOF MS/MS spectra of 184,330 Enamine compounds, CCS and RT, CC BY 4.0enveda-180-filtered.parquet.gz
COCONUT 2.0738,794 natural-product structures, organisms, NPClassifier classes, CC0s3://scigantic-coconut/coconut-10-2026.parquet
LOTUS674,454 structure-organism-reference triples from Wikidata, CC BY 4.0s3://scigantic-zenodo-19360665/.csv.gz
MassBank (open records)96,949 reference spectra, licence per record, NC records excludeds3://scigantic-massbank/massbank_open_spectra.parquet
GNPS community libraries581,722 spectra from 65 directly contributed libraries, CC0s3://scigantic-gnps-libraries/gnps_community_spectra.parquet

RDKit does the chemistry. Spectral similarity is the greedy matched-peak cosine that matchms calls CosineGreedy, written out below in numpy because matchms is not installed in this image.

import os, gzip, shutil, time
from pathlib import Path
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.compute as pc, pyarrow.csv as pacsv
from rdkit import Chem, RDLogger
from rdkit.Chem import Descriptors, rdMolDescriptors, Draw, DataStructs
from rdkit.Chem import rdFingerprintGenerator
RDLogger.DisableLog('rdApp.*')
import pyarrow.fs as pafs
S3 = pafs.S3FileSystem(anonymous=True, region='us-east-1')   # the other four archives' public mirrors

ARCHIVE = Path(os.environ.get('SCIGANTIC_MOUNT_PATH') or Path.home() / 'archive')   # Enveda-180
QUERCETIN = 'REFJWTPEDVJJIY-UHFFFAOYSA-N'
PROTON = 1.007276

def cosine_greedy(mz1, i1, mz2, i2, tol=0.01):
    """Greedy matched-peak cosine (matchms CosineGreedy, mz_power 0, intensity_power 1)."""
    mz1, i1, mz2, i2 = (np.asarray(x, float) for x in (mz1, i1, mz2, i2))
    a, b = np.nonzero((mz2[None, :] >= mz1[:, None] - tol) & (mz2[None, :] <= mz1[:, None] + tol))
    if len(a) == 0:
        return 0.0, 0
    score = i1[a] * i2[b]
    used1, used2, total, n = set(), set(), 0.0, 0
    for k in np.argsort(score, kind='mergesort')[::-1]:
        if a[k] in used1 or b[k] in used2:
            continue
        used1.add(a[k]); used2.add(b[k]); total += score[k]; n += 1
    return total / (np.linalg.norm(i1) * np.linalg.norm(i2)), n

def clean(mz, inten, precursor_mz=None, keep=0.01):
    """Drop peaks under 1% of the base peak and anything at or above the precursor."""
    mz, inten = np.asarray(mz, float), np.asarray(inten, float)
    if len(mz) == 0:
        return mz, inten
    ok = inten >= keep * inten.max()
    if precursor_mz:
        ok &= mz < precursor_mz - 0.5
    if not ok.any():
        return mz[:0], inten[:0]
    return mz[ok], inten[ok] / inten[ok].max()

print('Enveda-180 mount:', ARCHIVE, '| enveda-180-filtered.parquet.gz present:', (ARCHIVE / 'enveda-180-filtered.parquet.gz').exists())

Output:

Enveda-180 mount: /mnt/archive | enveda-180-filtered.parquet.gz present: True

1. The structure, from COCONUT, checked with RDKit

COCONUT stores a canonical SMILES, InChI and InChIKey per molecule plus computed properties. Two things to know before joining on it: identifiers carry a stereo-variant suffix (CNP0103504.0, .1, .2, ...), and the InChIKey is not unique (9,920 keys appear on more than one row, mostly tautomers such as an imide and its iminol). Join on the key, then pick a row.

coco = pq.read_table('scigantic-coconut/coconut-10-2026.parquet', filesystem=S3,
                     filters=[('standard_inchi_key', '=', QUERCETIN)]).to_pylist()
q = coco[0]
mol = Chem.MolFromSmiles(q['canonical_smiles'])
assert Chem.MolToInchiKey(mol) == QUERCETIN      # COCONUT's key agrees with RDKit's
mono = Descriptors.ExactMolWt(mol)
print(q['identifier'], q['name'], '|', q['np_classifier_pathway'], '>', q['np_classifier_superclass'], '>', q['np_classifier_class'])
print('formula', rdMolDescriptors.CalcMolFormula(mol), '(COCONUT', q['molecular_formula'] + ')',
      f'| monoisotopic {mono:.5f} (COCONUT {q["exact_molecular_weight"]})')
print(f'[M+H]+ {mono + PROTON:.4f}   [M-H]- {mono - PROTON:.4f}   NP-likeness {q["np_likeness"]}')
print('in collections:', ', '.join(q['collections'].split('|')[:8]), '...')
Draw.MolToImage(mol, size=(320, 240))

Output:

CNP0103504.0 quercetin | Shikimates and Phenylpropanoids > Flavonoids > Flavonols
formula C15H10O7 (COCONUT C15H10O7) | monoisotopic 302.04265 (COCONUT 302.04265)
[M+H]+ 303.0499   [M-H]- 301.0354   NP-likeness 1.7
in collections: ANPDB, AfroCancer, AfroDB, AnalytiCon Discovery NPs, Australian natural products, BIOFACQUIM, BitterDB, CMAUP (cCollective molecular activities of useful plants) ...

2. Who makes it, from LOTUS

LOTUS is the Wikidata-backed record of which organism a structure was reported in, and by which paper. The core file is one row per (structure, organism, reference) triple; the metadata file adds the organism's taxonomy, so a family breakdown is one filter away. Only 178 of 674,454 triples carry manual_validation = Y; the rest are curated but unreviewed literature mining.

LOTUS = 'scigantic-zenodo-19360665/260413_frozen_metadata.csv.gz'
with S3.open_input_stream(LOTUS) as f:      # 90 MB gzip, 709,740 rows; read only 4 columns
    lm = pacsv.read_csv(f, convert_options=pacsv.ConvertOptions(include_columns=[
        'structure_inchikey', 'organism_name', 'organism_taxonomy_06family', 'reference_doi']))
ql = lm.filter(pc.equal(lm['structure_inchikey'], QUERCETIN)).to_pandas()
print(f'{len(ql)} LOTUS rows for quercetin: {ql.organism_name.nunique()} organisms, {ql.reference_doi.nunique()} references')
coco_orgs = set(q['organisms'].split('|'))
print(f'COCONUT lists {len(coco_orgs)} organism names; {len(coco_orgs & set(ql.organism_name))} match a LOTUS name exactly')
fam = ql[ql.organism_taxonomy_06family.fillna('') != ''].groupby('organism_taxonomy_06family').organism_name.nunique()
fam.sort_values().tail(12).plot.barh(figsize=(6, 3.6), color='#3b6ea8')
plt.xlabel('organisms reporting quercetin'); plt.ylabel(''); plt.title('Quercetin in LOTUS, top plant families')
plt.tight_layout(); plt.show()

Output:

2421 LOTUS rows for quercetin: 1745 organisms, 1369 references
COCONUT lists 3320 organism names; 1715 match a LOTUS name exactly

3. Reference spectra: MassBank against GNPS

Both libraries are stored here with peaks as mz / intensity lists, so pulling every quercetin spectrum is a filtered Parquet read. MassBank's record licence is on every row. Spectra are compared in positive mode at [M+H]+ after dropping peaks under 1% and the precursor itself, at 0.01 Da.

mb = pq.read_table('scigantic-massbank/massbank_open_spectra.parquet', filesystem=S3,
                   filters=[('inchikey', '=', QUERCETIN)]).to_pandas()
gn = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
                   filters=[('inchikey', '=', QUERCETIN)]).to_pandas()
print('MassBank', len(mb), 'spectra |', mb.precursor_type.value_counts(dropna=False).to_dict())
print('  licences', mb.license.value_counts().to_dict())
print('GNPS    ', len(gn), 'spectra |', gn.adduct.value_counts(dropna=False).to_dict())
print('  libraries', gn.library.value_counts().to_dict())

ref = []
for r in mb[(mb.precursor_type == '[M+H]+') & (mb.spectrum_type == 'MS2')].itertuples():
    ref.append(('MassBank', r.accession, r.instrument_type, r.collision_energy, *clean(r.mz, r.intensity, r.precursor_mz)))
for r in gn[gn.adduct.isin(['M+H', '[M+H]+']) & (gn.ion_mode.str.lower() == 'positive')].itertuples():
    ref.append(('GNPS', r.spectrum_id, r.instrument, r.library, *clean(r.mz, r.intensity, r.precursor_mz)))
ref = [x for x in ref if len(x[4]) >= 3]
n = len(ref)
M = np.eye(n)
for i in range(n):
    for j in range(i + 1, n):
        M[i, j] = M[j, i] = cosine_greedy(ref[i][4], ref[i][5], ref[j][4], ref[j][5])[0]
src = np.array([x[0] for x in ref])
iu = np.triu_indices(n, 1)
same = src[iu[0]] == src[iu[1]]
for name, sel in [('MassBank x MassBank', same & (src[iu[0]] == 'MassBank')), ('GNPS x GNPS', same & (src[iu[0]] == 'GNPS')), ('MassBank x GNPS', ~same)]:
    v = M[iu][sel]
    print(f'{name:20s} {len(v):4d} pairs  median cosine {np.median(v):.2f}  share >= 0.7: {np.mean(v >= 0.7):.0%}')
order = np.argsort(src, kind='stable')
fig, ax = plt.subplots(figsize=(5.2, 4.4))
im = ax.imshow(M[np.ix_(order, order)], vmin=0, vmax=1, cmap='Blues')
k = int((src == 'GNPS').sum())
ax.axhline(k - 0.5, color='k', lw=0.8); ax.axvline(k - 0.5, color='k', lw=0.8)
ax.set_title(f'Quercetin [M+H]+: GNPS ({k}) then MassBank ({n - k})'); ax.set_xticks([]); ax.set_yticks([])
fig.colorbar(im, ax=ax, label='cosine'); plt.tight_layout(); plt.show()

Output:

MassBank 76 spectra | {'[M-H]-': 42, '[M+H]+': 22, None: 12}
  licences {'CC BY-SA 4.0': 36, 'CC BY 4.0': 23, 'dl-de/by-2-0': 17}
GNPS     100 spectra | {'M+H': 38, 'M-H': 24, '[M-H]-': 18, '[M+H]+': 10, '2M-H': 8, '[M+H-H2O]+': 1, '[M+NH4]+': 1}
  libraries {'GNPS-LIBRARY': 19, 'BERKELEY-LAB': 18, 'MSNLIB-NEGATIVE': 18, 'WINE-DB-QTOF': 13, 'CMMC-FOOD-BIOMARKERS': 9, 'LEAFBOT': 7, 'GNPS-NIST14-MATCHES': 6, 'WINE-DB-ORBITRAP': 4, 'PHENOLICSDB': 2, 'DMIM-DRUG-METABOLITE-LIBRARY': 2, 'GNPS-NIH-NATURALPRODUCTSLIBRARY_ROUND2_POSITIVE': 1, 'GNPS-ION-MOBILITY-LIBRARY': 1}
MassBank x MassBank    36 pairs  median cosine 0.50  share >= 0.7: 39%
GNPS x GNPS           861 pairs  median cosine 0.81  share >= 0.7: 60%
MassBank x GNPS       378 pairs  median cosine 0.70  share >= 0.7: 50%
# The best cross-library pair, drawn as a mirror plot
cross = [(M[i, j], i, j) for i in range(n) for j in range(n) if src[i] == 'GNPS' and src[j] == 'MassBank']
s, i, j = max(cross)
fig, ax = plt.subplots(figsize=(7, 3.4))
ax.vlines(ref[i][4], 0, ref[i][5], color='#3b6ea8', lw=1.2, label=f'GNPS {ref[i][1]} ({ref[i][2]})')
ax.vlines(ref[j][4], 0, -ref[j][5], color='#c0504d', lw=1.2, label=f'MassBank {ref[j][1]} ({ref[j][2]}, {ref[j][3]})')
ax.axhline(0, color='k', lw=0.6); ax.set_xlabel('m/z'); ax.set_ylabel('relative intensity')
ax.set_title(f'Quercetin [M+H]+ = {mono + PROTON:.4f}, cosine {s:.2f}'); ax.legend(fontsize=8, loc='lower left')
plt.tight_layout(); plt.show()

4. Enveda-180: what it covers, and what it does not

Enveda-180 profiles 184,330 compounds on a Bruker timsTOF Pro 2 at three collision energies (20, 40, 60 eV, plus a merged spectrum) in both polarities, with retention time on one C18 method and ion-mobility CCS. The compounds are synthetic Enamine screening molecules, not natural products. The mount holds the Zenodo files unchanged, and .parquet.gz is a gzip wrapped around a Parquet file, so it has to be decompressed once (about 10 s, 428 MB) before pyarrow can open it.

Two encoding traps: every numeric column is a string (pepmass, ccs, mzs, ...), and a missing CCS is the sentinel '-1.00', not a null.

SRC = ARCHIVE / 'enveda-180-filtered.parquet.gz'
LOCAL = Path('/tmp/enveda-180-filtered.parquet')
if not LOCAL.exists():
    t0 = time.time()
    with gzip.open(SRC, 'rb') as fin, open(LOCAL, 'wb') as fout:
        shutil.copyfileobj(fin, fout, 16 << 20)
    print(f'decompressed in {time.time() - t0:.0f} s')
ev = pq.ParquetFile(LOCAL)
meta = ev.read(columns=['inchikey', 'smiles', 'ionmode', 'collision_energies', 'ccs'])
print(f'{ev.metadata.num_rows:,} spectra, {len(pc.unique(meta["inchikey"])):,} InChIKeys, '
      f'{ev.metadata.num_row_groups} row groups (so a filter still decodes half the file)')
print('collision energies:', pc.value_counts(meta['collision_energies']).to_pylist()[:6])
print('CCS missing (-1.00):', pc.sum(pc.equal(meta['ccs'], '-1.00')).as_py())

ekeys = set(pc.unique(meta['inchikey']).to_pylist())
ckeys = set(pq.read_table('scigantic-coconut/coconut-10-2026.parquet', filesystem=S3,
                          columns=['standard_inchi_key'])['standard_inchi_key'].to_pylist())
print(f'quercetin in Enveda-180: {QUERCETIN in ekeys};  Enveda-180 InChIKeys also in COCONUT: {len(ekeys & ckeys)} of {len(ekeys):,}')

Output:

decompressed in 4 s
1,158,293 spectra, 184,330 InChIKeys, 2 row groups (so a filter still decodes half the file)
collision energies: [{'values': '-40', 'counts': 83416}, {'values': '-60', 'counts': 70379}, {'values': '20,40,60', 'counts': 241197}, {'values': '40', 'counts': 245346}, {'values': '60', 'counts': 230342}, {'values': '20', 'counts': 173851}]
CCS missing (-1.00): 42401
quercetin in Enveda-180: False;  Enveda-180 InChIKeys also in COCONUT: 33 of 184,330

So no library match is possible for quercetin in Enveda-180. What the library does offer is chemical neighbours. RDKit finds every flavone-like core (a 4H-chromen-4-one with a 2-aryl group) among the 184,330 structures, then ranks them by Morgan-fingerprint Tanimoto similarity to quercetin.

uniq = pa.table({'inchikey': meta['inchikey'], 'smiles': meta['smiles']}).group_by('inchikey').aggregate([('smiles', 'min')])
core = Chem.MolFromSmarts('O=c1cc(-c2ccccc2)oc2ccccc12')
gen = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=2048)
qfp = gen.GetFingerprint(mol)
t0, hits = time.time(), []
for ik, smi in zip(uniq['inchikey'].to_pylist(), uniq['smiles_min'].to_pylist()):
    m = Chem.MolFromSmiles(smi)
    if m is not None and m.HasSubstructMatch(core):
        hits.append((DataStructs.TanimotoSimilarity(qfp, gen.GetFingerprint(m)), ik, smi))
hits.sort(reverse=True)
print(f'{len(hits)} flavone-core compounds in Enveda-180 (scanned {len(uniq):,} in {time.time() - t0:.0f} s)')
for s, ik, smi in hits[:5]:
    print(f'  Tanimoto {s:.2f}  {ik}  {smi}  {"(also in COCONUT)" if ik in ckeys else ""}')
Draw.MolsToGridImage([Chem.MolFromSmiles(h[2]) for h in hits[:4]], molsPerRow=4, subImgSize=(200, 170),
                     legends=[f'{h[0]:.2f}' for h in hits[:4]])

Output:

11 flavone-core compounds in Enveda-180 (scanned 184,330 in 22 s)
  Tanimoto 0.19  YTSXWTNKPYQBGN-UHFFFAOYSA-N  COC1=CC=C(C2=CC(=O)C3=C(C=CC(C)=C3)O2)C=C1  (also in COCONUT)
  Tanimoto 0.19  AGBYYVMLHAYADK-UHFFFAOYSA-N  CC(C)N(C1CCC1)C(=O)C1=C2C(=CC=C1)C(=O)C(C)=C(C1=CC=CC=C1)O2  
  Tanimoto 0.18  WRGAGHDXEIFJOG-UHFFFAOYSA-N  COC1=C(OC)C=C(C2=CC(=O)C3=C(C=CC(C)=C3)O2)C=C1  (also in COCONUT)
  Tanimoto 0.17  USZKFWPUUMPUBH-UHFFFAOYSA-N  CC1=C(C2=CC=CC=C2)OC2=C(C=CC=C2C(=O)N2CCCC2)C1=O  
  Tanimoto 0.16  HQGGRZZPVOBAOV-UHFFFAOYSA-N  CC1=C(C2=CC=CC=C2)OC2=C(C=CC=C2C(=O)N2CCCCC2)C1=O

5. One molecule that is in four of the five: 2'-methoxyflavone

YEHDMSUNJUONMW-UHFFFAOYSA-N is a plant flavone (in COCONUT and LOTUS) that Enamine also sells, so Enveda-180 measured it, and the GNPS community libraries hold reference spectra of it. MassBank's open records do not have it. That makes it a direct test of the question every CASMI entrant has: does a timsTOF spectrum match an Orbitrap reference of the same molecule?

FLAVONE = 'YEHDMSUNJUONMW-UHFFFAOYSA-N'
def enveda_spectra(keys, adduct='[M+H]+'):
    vs, rows = pa.array(sorted(keys)), []
    for b in ev.iter_batches(batch_size=100_000, columns=['title', 'inchikey', 'adduct', 'collision_energies', 'pepmass', 'ccs', 'mzs', 'intensities']):
        t = pa.Table.from_batches([b])
        rows += t.filter(pc.and_(pc.is_in(t['inchikey'], value_set=vs), pc.equal(t['adduct'], adduct))).to_pylist()
    return rows

def gnps_spectra(keys):
    t = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
                      filters=[('inchikey', 'in', sorted(keys)), ('adduct', 'in', ['M+H', '[M+H]+'])]).to_pylist()
    return [r for r in t if (r['ion_mode'] or '').lower().startswith('positive')]

fe, fg = enveda_spectra({FLAVONE}), gnps_spectra({FLAVONE})
lot = lm.filter(pc.equal(lm['structure_inchikey'], FLAVONE)).to_pandas()
print(f"Enveda-180 {len(fe)} [M+H]+ spectra; GNPS {len(fg)} ({', '.join(sorted({r['library'] for r in fg}))}); "
      f"LOTUS organisms: {', '.join(sorted(set(lot.organism_name)))}")
G = [clean(r['mz'], r['intensity'], r['precursor_mz']) for r in fg]
fig, axes = plt.subplots(len(fe), 1, figsize=(7, 1.9 * len(fe)), sharex=True)
for ax, r in zip(np.atleast_1d(axes), sorted(fe, key=lambda r: r['collision_energies'])):
    mz, it = clean(np.array(r['mzs'], float), np.array(r['intensities'], float), float(r['pepmass']))
    scores = [cosine_greedy(mz, it, *g)[0] for g in G]
    k = int(np.argmax(scores))
    ax.vlines(mz, 0, it, color='#3b6ea8', lw=1.2)
    ax.vlines(G[k][0], 0, -G[k][1], color='#c0504d', lw=1.2)
    ax.axhline(0, color='k', lw=0.5); ax.set_yticks([])
    ax.set_title(f"timsTOF CE {r['collision_energies']} (CCS {r['ccs']}) vs best GNPS Orbitrap spectrum: cosine {scores[k]:.2f}", fontsize=8)
plt.xlabel('m/z'); plt.tight_layout(); plt.show()

Output:

Enveda-180 2 [M+H]+ spectra; GNPS 12 (BERKELEY-LAB, GNPS-NIH-NATURALPRODUCTSLIBRARY_ROUND2_POSITIVE, GNPS-NIST14-MATCHES, MSNLIB-POSITIVE); LOTUS organisms: Primula veris

6. The same check on 150 shared molecules

2,077 Enveda-180 compounds have a positive-mode [M+H]+ spectrum in the GNPS community libraries, almost all from the MSnLib Orbitrap libraries. Take 150 at random; for each Enveda [M+H]+ spectrum, score its best cosine against GNPS spectra of the same molecule and, as a control, against GNPS spectra of a different molecule from the sample.

gk = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
                   columns=['inchikey', 'adduct', 'ion_mode']).to_pandas()
gk = gk[gk.adduct.isin(['M+H', '[M+H]+']) & gk.ion_mode.str.lower().str.startswith('positive', na=False)]
shared = sorted(set(gk.inchikey.dropna()) & ekeys)
sample = sorted(np.random.default_rng(0).choice(shared, 150, replace=False))
G = {}
for r in gnps_spectra(sample):
    G.setdefault(r['inchikey'], []).append(clean(r['mz'], r['intensity'], r['precursor_mz']))
res = []
for r in enveda_spectra(sample):
    mz, it = clean(np.array(r['mzs'], float), np.array(r['intensities'], float), float(r['pepmass']))
    if len(mz) < 3 or r['inchikey'] not in G:
        continue
    other = sample[(sample.index(r['inchikey']) + 1) % len(sample)]
    best = lambda refs: max((cosine_greedy(mz, it, *g)[0] for g in refs if len(g[0])), default=0.0)
    res.append({'ce': r['collision_energies'], 'same': best(G[r['inchikey']]), 'other': best(G.get(other, []))})
res = pd.DataFrame(res)
print(f'{len(shared):,} shared compounds; scored {len(res)} Enveda [M+H]+ spectra of {len(sample)} of them')
print(res.groupby('ce')[['same', 'other']].median().round(2).rename(columns={'same': 'same molecule', 'other': 'different molecule'}))
plt.figure(figsize=(6, 3))
plt.hist(res.same, bins=40, range=(0, 1), alpha=0.75, color='#3b6ea8', label='same molecule')
plt.hist(res.other, bins=40, range=(0, 1), alpha=0.75, color='#c0504d', label='different molecule')
plt.xlabel('best cosine, Enveda-180 timsTOF vs GNPS'); plt.ylabel('spectra'); plt.legend(); plt.tight_layout(); plt.show()

Output:

2,077 shared compounds; scored 484 Enveda [M+H]+ spectra of 150 of them
          same molecule  different molecule
ce                                         
20                 0.98                0.00
20,40,60           0.96                0.00
40                 0.91                0.00
60                 0.81                0.01

Where the files are, for your own analysis

the unfiltered enveda-180.*.gz, byte-identical to Zenodo 20436851. The unfiltered files hold the same spectra with unfiltered peak lists (median about 139 peaks against 14); read that Parquet with iter_batches, since read_row_group fails on its nested columns.

Every bucket has a MIRROR.md at its root with the source, licence and per-file md5. Cite the upstream resource and authors, not the mirror. Raw tables, if you want them:

pd.set_option('display.max_colwidth', 40)
pq.read_table('scigantic-massbank/massbank_open_spectra.parquet', filesystem=S3,
              columns=['accession', 'name', 'inchikey', 'precursor_type', 'instrument_type', 'license', 'num_peaks']).slice(0, 5).to_pandas()

Output:

             accession             name                     inchikey  \
0  MSBNK-AAFC-AC000857       Mevastatin  AJLFOPYRIVGYMJ-INTXDZFKSA-N   
1  MSBNK-AAFC-AC000845  Ctrinin hydrate  MBTAFMZUDOUGPS-CGKQSRJASA-N   
2  MSBNK-AAFC-AC000182          Kotanin  CSJOUDOXDHMIAH-UHFFFAOYSA-N   
3  MSBNK-AAFC-AC000230  Stipitatic acid  ZGKNMKBZOSTFCB-UHFFFAOYSA-N   
4  MSBNK-AAFC-AC000237        T-2 Toxin  BXFOFFBJRFZBQZ-QYWOHJEZSA-N   

  precursor_type instrument_type       license  num_peaks  
0         [M+H]+     LC-ESI-ITFT  CC BY-SA 4.0         16  
1         [M-H]-     LC-ESI-ITFT  CC BY-SA 4.0         12  
2        [M+Na]+     LC-ESI-ITFT  CC BY-SA 4.0          1  
3         [M+H]+     LC-ESI-ITFT  CC BY-SA 4.0          1  
4         [M+H]+     LC-ESI-ITFT  CC BY-SA 4.0         74

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.