Enveda-180: 1.16 million timsTOF MS/MS spectra with CCS and retention time for 184,330 drug-like molecules: sample notebook
A worked notebook for Enveda-180: 1.16 million timsTOF MS/MS spectra with CCS and retention time for 184,330 drug-like molecules, with 11 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Enveda (Krettler C.A., Kind T., DeBloois E., Killian M., Prince J. et al.); mirrored by Scigantic
- License
- CC-BY-4.0
- Size
- 9 objects (8 data files + MIRROR.md), 8,920 MB, sizes and md5 identical to the Zenodo record
- Format
- JSON Lines (in sampled files)
- Stored at
- Amazon S3, s3://scigantic-zenodo-20436851 (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Start here. Click the grey cell below and press Shift+Enter. It looks one molecule up in four of the five archives in a few seconds. Then change INCHIKEY to another compound (try caffeine, RYYVLZVUVIJVGH-UHFFFAOYSA-N) and run it again.
INCHIKEY = 'REFJWTPEDVJJIY-UHFFFAOYSA-N' # quercetin; change me
import pyarrow.parquet as pq, pyarrow.fs as pafs
S3 = pafs.S3FileSystem(anonymous=True, region='us-east-1')
def lookup(path, key_col, cols):
return pq.read_table(path, filesystem=S3, columns=cols, filters=[(key_col, '=', INCHIKEY)]).to_pylist()
coco = lookup('scigantic-coconut/coconut-10-2026.parquet', 'standard_inchi_key',
['identifier', 'name', 'molecular_formula', 'np_likeness', 'np_classifier_class', 'organisms'])
mb = lookup('scigantic-massbank/massbank_open_spectra.parquet', 'inchikey', ['accession', 'precursor_type'])
gn = lookup('scigantic-gnps-libraries/gnps_community_spectra.parquet', 'inchikey', ['spectrum_id', 'adduct'])
for r in coco:
print(f"COCONUT {r['identifier']} {r['name']} {r['molecular_formula']} NP-likeness {r['np_likeness']}"
f" class {r['np_classifier_class']} organisms {len((r['organisms'] or '').split('|')) if r['organisms'] else 0}")
print(f'MassBank {len(mb)} open-licence reference spectra; GNPS {len(gn)} community-library spectra')
One natural product through five open mass-spec libraries
Identifying a small molecule from its MS/MS spectrum means matching it against something: a reference spectrum, a structure database, a list of what organisms make. This notebook follows quercetin, a flavonol found in thousands of plants, through five open archives, then turns to Enveda-180 and measures how little of it overlaps with natural-product chemical space.
| archive | what it holds | here as |
| Enveda-180 (~/archive) | 1,158,293 timsTOF MS/MS spectra of 184,330 Enamine compounds, CCS and RT, CC BY 4.0 | enveda-180-filtered.parquet.gz |
| COCONUT 2.0 | 738,794 natural-product structures, organisms, NPClassifier classes, CC0 | s3://scigantic-coconut/coconut-10-2026.parquet |
| LOTUS | 674,454 structure-organism-reference triples from Wikidata, CC BY 4.0 | s3://scigantic-zenodo-19360665/.csv.gz |
| MassBank (open records) | 96,949 reference spectra, licence per record, NC records excluded | s3://scigantic-massbank/massbank_open_spectra.parquet |
| GNPS community libraries | 581,722 spectra from 65 directly contributed libraries, CC0 | s3://scigantic-gnps-libraries/gnps_community_spectra.parquet |
RDKit does the chemistry. Spectral similarity is the greedy matched-peak cosine that matchms calls CosineGreedy, written out below in numpy because matchms is not installed in this image.
import os, gzip, shutil, time
from pathlib import Path
import numpy as np, pandas as pd, matplotlib.pyplot as plt
import pyarrow as pa, pyarrow.parquet as pq, pyarrow.compute as pc, pyarrow.csv as pacsv
from rdkit import Chem, RDLogger
from rdkit.Chem import Descriptors, rdMolDescriptors, Draw, DataStructs
from rdkit.Chem import rdFingerprintGenerator
RDLogger.DisableLog('rdApp.*')
import pyarrow.fs as pafs
S3 = pafs.S3FileSystem(anonymous=True, region='us-east-1') # the other four archives' public mirrors
ARCHIVE = Path(os.environ.get('SCIGANTIC_MOUNT_PATH') or Path.home() / 'archive') # Enveda-180
QUERCETIN = 'REFJWTPEDVJJIY-UHFFFAOYSA-N'
PROTON = 1.007276
def cosine_greedy(mz1, i1, mz2, i2, tol=0.01):
"""Greedy matched-peak cosine (matchms CosineGreedy, mz_power 0, intensity_power 1)."""
mz1, i1, mz2, i2 = (np.asarray(x, float) for x in (mz1, i1, mz2, i2))
a, b = np.nonzero((mz2[None, :] >= mz1[:, None] - tol) & (mz2[None, :] <= mz1[:, None] + tol))
if len(a) == 0:
return 0.0, 0
score = i1[a] * i2[b]
used1, used2, total, n = set(), set(), 0.0, 0
for k in np.argsort(score, kind='mergesort')[::-1]:
if a[k] in used1 or b[k] in used2:
continue
used1.add(a[k]); used2.add(b[k]); total += score[k]; n += 1
return total / (np.linalg.norm(i1) * np.linalg.norm(i2)), n
def clean(mz, inten, precursor_mz=None, keep=0.01):
"""Drop peaks under 1% of the base peak and anything at or above the precursor."""
mz, inten = np.asarray(mz, float), np.asarray(inten, float)
if len(mz) == 0:
return mz, inten
ok = inten >= keep * inten.max()
if precursor_mz:
ok &= mz < precursor_mz - 0.5
if not ok.any():
return mz[:0], inten[:0]
return mz[ok], inten[ok] / inten[ok].max()
print('Enveda-180 mount:', ARCHIVE, '| enveda-180-filtered.parquet.gz present:', (ARCHIVE / 'enveda-180-filtered.parquet.gz').exists())
Output:
Enveda-180 mount: /mnt/archive | enveda-180-filtered.parquet.gz present: True
1. The structure, from COCONUT, checked with RDKit
COCONUT stores a canonical SMILES, InChI and InChIKey per molecule plus computed properties. Two things to know before joining on it: identifiers carry a stereo-variant suffix (CNP0103504.0, .1, .2, ...), and the InChIKey is not unique (9,920 keys appear on more than one row, mostly tautomers such as an imide and its iminol). Join on the key, then pick a row.
coco = pq.read_table('scigantic-coconut/coconut-10-2026.parquet', filesystem=S3,
filters=[('standard_inchi_key', '=', QUERCETIN)]).to_pylist()
q = coco[0]
mol = Chem.MolFromSmiles(q['canonical_smiles'])
assert Chem.MolToInchiKey(mol) == QUERCETIN # COCONUT's key agrees with RDKit's
mono = Descriptors.ExactMolWt(mol)
print(q['identifier'], q['name'], '|', q['np_classifier_pathway'], '>', q['np_classifier_superclass'], '>', q['np_classifier_class'])
print('formula', rdMolDescriptors.CalcMolFormula(mol), '(COCONUT', q['molecular_formula'] + ')',
f'| monoisotopic {mono:.5f} (COCONUT {q["exact_molecular_weight"]})')
print(f'[M+H]+ {mono + PROTON:.4f} [M-H]- {mono - PROTON:.4f} NP-likeness {q["np_likeness"]}')
print('in collections:', ', '.join(q['collections'].split('|')[:8]), '...')
Draw.MolToImage(mol, size=(320, 240))
Output:
CNP0103504.0 quercetin | Shikimates and Phenylpropanoids > Flavonoids > Flavonols formula C15H10O7 (COCONUT C15H10O7) | monoisotopic 302.04265 (COCONUT 302.04265) [M+H]+ 303.0499 [M-H]- 301.0354 NP-likeness 1.7 in collections: ANPDB, AfroCancer, AfroDB, AnalytiCon Discovery NPs, Australian natural products, BIOFACQUIM, BitterDB, CMAUP (cCollective molecular activities of useful plants) ...
2. Who makes it, from LOTUS
LOTUS is the Wikidata-backed record of which organism a structure was reported in, and by which paper. The core file is one row per (structure, organism, reference) triple; the metadata file adds the organism's taxonomy, so a family breakdown is one filter away. Only 178 of 674,454 triples carry manual_validation = Y; the rest are curated but unreviewed literature mining.
LOTUS = 'scigantic-zenodo-19360665/260413_frozen_metadata.csv.gz'
with S3.open_input_stream(LOTUS) as f: # 90 MB gzip, 709,740 rows; read only 4 columns
lm = pacsv.read_csv(f, convert_options=pacsv.ConvertOptions(include_columns=[
'structure_inchikey', 'organism_name', 'organism_taxonomy_06family', 'reference_doi']))
ql = lm.filter(pc.equal(lm['structure_inchikey'], QUERCETIN)).to_pandas()
print(f'{len(ql)} LOTUS rows for quercetin: {ql.organism_name.nunique()} organisms, {ql.reference_doi.nunique()} references')
coco_orgs = set(q['organisms'].split('|'))
print(f'COCONUT lists {len(coco_orgs)} organism names; {len(coco_orgs & set(ql.organism_name))} match a LOTUS name exactly')
fam = ql[ql.organism_taxonomy_06family.fillna('') != ''].groupby('organism_taxonomy_06family').organism_name.nunique()
fam.sort_values().tail(12).plot.barh(figsize=(6, 3.6), color='#3b6ea8')
plt.xlabel('organisms reporting quercetin'); plt.ylabel(''); plt.title('Quercetin in LOTUS, top plant families')
plt.tight_layout(); plt.show()
Output:
2421 LOTUS rows for quercetin: 1745 organisms, 1369 references COCONUT lists 3320 organism names; 1715 match a LOTUS name exactly
3. Reference spectra: MassBank against GNPS
Both libraries are stored here with peaks as mz / intensity lists, so pulling every quercetin spectrum is a filtered Parquet read. MassBank's record licence is on every row. Spectra are compared in positive mode at [M+H]+ after dropping peaks under 1% and the precursor itself, at 0.01 Da.
mb = pq.read_table('scigantic-massbank/massbank_open_spectra.parquet', filesystem=S3,
filters=[('inchikey', '=', QUERCETIN)]).to_pandas()
gn = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
filters=[('inchikey', '=', QUERCETIN)]).to_pandas()
print('MassBank', len(mb), 'spectra |', mb.precursor_type.value_counts(dropna=False).to_dict())
print(' licences', mb.license.value_counts().to_dict())
print('GNPS ', len(gn), 'spectra |', gn.adduct.value_counts(dropna=False).to_dict())
print(' libraries', gn.library.value_counts().to_dict())
ref = []
for r in mb[(mb.precursor_type == '[M+H]+') & (mb.spectrum_type == 'MS2')].itertuples():
ref.append(('MassBank', r.accession, r.instrument_type, r.collision_energy, *clean(r.mz, r.intensity, r.precursor_mz)))
for r in gn[gn.adduct.isin(['M+H', '[M+H]+']) & (gn.ion_mode.str.lower() == 'positive')].itertuples():
ref.append(('GNPS', r.spectrum_id, r.instrument, r.library, *clean(r.mz, r.intensity, r.precursor_mz)))
ref = [x for x in ref if len(x[4]) >= 3]
n = len(ref)
M = np.eye(n)
for i in range(n):
for j in range(i + 1, n):
M[i, j] = M[j, i] = cosine_greedy(ref[i][4], ref[i][5], ref[j][4], ref[j][5])[0]
src = np.array([x[0] for x in ref])
iu = np.triu_indices(n, 1)
same = src[iu[0]] == src[iu[1]]
for name, sel in [('MassBank x MassBank', same & (src[iu[0]] == 'MassBank')), ('GNPS x GNPS', same & (src[iu[0]] == 'GNPS')), ('MassBank x GNPS', ~same)]:
v = M[iu][sel]
print(f'{name:20s} {len(v):4d} pairs median cosine {np.median(v):.2f} share >= 0.7: {np.mean(v >= 0.7):.0%}')
order = np.argsort(src, kind='stable')
fig, ax = plt.subplots(figsize=(5.2, 4.4))
im = ax.imshow(M[np.ix_(order, order)], vmin=0, vmax=1, cmap='Blues')
k = int((src == 'GNPS').sum())
ax.axhline(k - 0.5, color='k', lw=0.8); ax.axvline(k - 0.5, color='k', lw=0.8)
ax.set_title(f'Quercetin [M+H]+: GNPS ({k}) then MassBank ({n - k})'); ax.set_xticks([]); ax.set_yticks([])
fig.colorbar(im, ax=ax, label='cosine'); plt.tight_layout(); plt.show()
Output:
MassBank 76 spectra | {'[M-H]-': 42, '[M+H]+': 22, None: 12}
licences {'CC BY-SA 4.0': 36, 'CC BY 4.0': 23, 'dl-de/by-2-0': 17}
GNPS 100 spectra | {'M+H': 38, 'M-H': 24, '[M-H]-': 18, '[M+H]+': 10, '2M-H': 8, '[M+H-H2O]+': 1, '[M+NH4]+': 1}
libraries {'GNPS-LIBRARY': 19, 'BERKELEY-LAB': 18, 'MSNLIB-NEGATIVE': 18, 'WINE-DB-QTOF': 13, 'CMMC-FOOD-BIOMARKERS': 9, 'LEAFBOT': 7, 'GNPS-NIST14-MATCHES': 6, 'WINE-DB-ORBITRAP': 4, 'PHENOLICSDB': 2, 'DMIM-DRUG-METABOLITE-LIBRARY': 2, 'GNPS-NIH-NATURALPRODUCTSLIBRARY_ROUND2_POSITIVE': 1, 'GNPS-ION-MOBILITY-LIBRARY': 1}
MassBank x MassBank 36 pairs median cosine 0.50 share >= 0.7: 39%
GNPS x GNPS 861 pairs median cosine 0.81 share >= 0.7: 60%
MassBank x GNPS 378 pairs median cosine 0.70 share >= 0.7: 50%
# The best cross-library pair, drawn as a mirror plot
cross = [(M[i, j], i, j) for i in range(n) for j in range(n) if src[i] == 'GNPS' and src[j] == 'MassBank']
s, i, j = max(cross)
fig, ax = plt.subplots(figsize=(7, 3.4))
ax.vlines(ref[i][4], 0, ref[i][5], color='#3b6ea8', lw=1.2, label=f'GNPS {ref[i][1]} ({ref[i][2]})')
ax.vlines(ref[j][4], 0, -ref[j][5], color='#c0504d', lw=1.2, label=f'MassBank {ref[j][1]} ({ref[j][2]}, {ref[j][3]})')
ax.axhline(0, color='k', lw=0.6); ax.set_xlabel('m/z'); ax.set_ylabel('relative intensity')
ax.set_title(f'Quercetin [M+H]+ = {mono + PROTON:.4f}, cosine {s:.2f}'); ax.legend(fontsize=8, loc='lower left')
plt.tight_layout(); plt.show()
4. Enveda-180: what it covers, and what it does not
Enveda-180 profiles 184,330 compounds on a Bruker timsTOF Pro 2 at three collision energies (20, 40, 60 eV, plus a merged spectrum) in both polarities, with retention time on one C18 method and ion-mobility CCS. The compounds are synthetic Enamine screening molecules, not natural products. The mount holds the Zenodo files unchanged, and .parquet.gz is a gzip wrapped around a Parquet file, so it has to be decompressed once (about 10 s, 428 MB) before pyarrow can open it.
Two encoding traps: every numeric column is a string (pepmass, ccs, mzs, ...), and a missing CCS is the sentinel '-1.00', not a null.
SRC = ARCHIVE / 'enveda-180-filtered.parquet.gz'
LOCAL = Path('/tmp/enveda-180-filtered.parquet')
if not LOCAL.exists():
t0 = time.time()
with gzip.open(SRC, 'rb') as fin, open(LOCAL, 'wb') as fout:
shutil.copyfileobj(fin, fout, 16 << 20)
print(f'decompressed in {time.time() - t0:.0f} s')
ev = pq.ParquetFile(LOCAL)
meta = ev.read(columns=['inchikey', 'smiles', 'ionmode', 'collision_energies', 'ccs'])
print(f'{ev.metadata.num_rows:,} spectra, {len(pc.unique(meta["inchikey"])):,} InChIKeys, '
f'{ev.metadata.num_row_groups} row groups (so a filter still decodes half the file)')
print('collision energies:', pc.value_counts(meta['collision_energies']).to_pylist()[:6])
print('CCS missing (-1.00):', pc.sum(pc.equal(meta['ccs'], '-1.00')).as_py())
ekeys = set(pc.unique(meta['inchikey']).to_pylist())
ckeys = set(pq.read_table('scigantic-coconut/coconut-10-2026.parquet', filesystem=S3,
columns=['standard_inchi_key'])['standard_inchi_key'].to_pylist())
print(f'quercetin in Enveda-180: {QUERCETIN in ekeys}; Enveda-180 InChIKeys also in COCONUT: {len(ekeys & ckeys)} of {len(ekeys):,}')
Output:
decompressed in 4 s
1,158,293 spectra, 184,330 InChIKeys, 2 row groups (so a filter still decodes half the file)
collision energies: [{'values': '-40', 'counts': 83416}, {'values': '-60', 'counts': 70379}, {'values': '20,40,60', 'counts': 241197}, {'values': '40', 'counts': 245346}, {'values': '60', 'counts': 230342}, {'values': '20', 'counts': 173851}]
CCS missing (-1.00): 42401
quercetin in Enveda-180: False; Enveda-180 InChIKeys also in COCONUT: 33 of 184,330
So no library match is possible for quercetin in Enveda-180. What the library does offer is chemical neighbours. RDKit finds every flavone-like core (a 4H-chromen-4-one with a 2-aryl group) among the 184,330 structures, then ranks them by Morgan-fingerprint Tanimoto similarity to quercetin.
uniq = pa.table({'inchikey': meta['inchikey'], 'smiles': meta['smiles']}).group_by('inchikey').aggregate([('smiles', 'min')])
core = Chem.MolFromSmarts('O=c1cc(-c2ccccc2)oc2ccccc12')
gen = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=2048)
qfp = gen.GetFingerprint(mol)
t0, hits = time.time(), []
for ik, smi in zip(uniq['inchikey'].to_pylist(), uniq['smiles_min'].to_pylist()):
m = Chem.MolFromSmiles(smi)
if m is not None and m.HasSubstructMatch(core):
hits.append((DataStructs.TanimotoSimilarity(qfp, gen.GetFingerprint(m)), ik, smi))
hits.sort(reverse=True)
print(f'{len(hits)} flavone-core compounds in Enveda-180 (scanned {len(uniq):,} in {time.time() - t0:.0f} s)')
for s, ik, smi in hits[:5]:
print(f' Tanimoto {s:.2f} {ik} {smi} {"(also in COCONUT)" if ik in ckeys else ""}')
Draw.MolsToGridImage([Chem.MolFromSmiles(h[2]) for h in hits[:4]], molsPerRow=4, subImgSize=(200, 170),
legends=[f'{h[0]:.2f}' for h in hits[:4]])
Output:
11 flavone-core compounds in Enveda-180 (scanned 184,330 in 22 s) Tanimoto 0.19 YTSXWTNKPYQBGN-UHFFFAOYSA-N COC1=CC=C(C2=CC(=O)C3=C(C=CC(C)=C3)O2)C=C1 (also in COCONUT) Tanimoto 0.19 AGBYYVMLHAYADK-UHFFFAOYSA-N CC(C)N(C1CCC1)C(=O)C1=C2C(=CC=C1)C(=O)C(C)=C(C1=CC=CC=C1)O2 Tanimoto 0.18 WRGAGHDXEIFJOG-UHFFFAOYSA-N COC1=C(OC)C=C(C2=CC(=O)C3=C(C=CC(C)=C3)O2)C=C1 (also in COCONUT) Tanimoto 0.17 USZKFWPUUMPUBH-UHFFFAOYSA-N CC1=C(C2=CC=CC=C2)OC2=C(C=CC=C2C(=O)N2CCCC2)C1=O Tanimoto 0.16 HQGGRZZPVOBAOV-UHFFFAOYSA-N CC1=C(C2=CC=CC=C2)OC2=C(C=CC=C2C(=O)N2CCCCC2)C1=O
5. One molecule that is in four of the five: 2'-methoxyflavone
YEHDMSUNJUONMW-UHFFFAOYSA-N is a plant flavone (in COCONUT and LOTUS) that Enamine also sells, so Enveda-180 measured it, and the GNPS community libraries hold reference spectra of it. MassBank's open records do not have it. That makes it a direct test of the question every CASMI entrant has: does a timsTOF spectrum match an Orbitrap reference of the same molecule?
FLAVONE = 'YEHDMSUNJUONMW-UHFFFAOYSA-N'
def enveda_spectra(keys, adduct='[M+H]+'):
vs, rows = pa.array(sorted(keys)), []
for b in ev.iter_batches(batch_size=100_000, columns=['title', 'inchikey', 'adduct', 'collision_energies', 'pepmass', 'ccs', 'mzs', 'intensities']):
t = pa.Table.from_batches([b])
rows += t.filter(pc.and_(pc.is_in(t['inchikey'], value_set=vs), pc.equal(t['adduct'], adduct))).to_pylist()
return rows
def gnps_spectra(keys):
t = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
filters=[('inchikey', 'in', sorted(keys)), ('adduct', 'in', ['M+H', '[M+H]+'])]).to_pylist()
return [r for r in t if (r['ion_mode'] or '').lower().startswith('positive')]
fe, fg = enveda_spectra({FLAVONE}), gnps_spectra({FLAVONE})
lot = lm.filter(pc.equal(lm['structure_inchikey'], FLAVONE)).to_pandas()
print(f"Enveda-180 {len(fe)} [M+H]+ spectra; GNPS {len(fg)} ({', '.join(sorted({r['library'] for r in fg}))}); "
f"LOTUS organisms: {', '.join(sorted(set(lot.organism_name)))}")
G = [clean(r['mz'], r['intensity'], r['precursor_mz']) for r in fg]
fig, axes = plt.subplots(len(fe), 1, figsize=(7, 1.9 * len(fe)), sharex=True)
for ax, r in zip(np.atleast_1d(axes), sorted(fe, key=lambda r: r['collision_energies'])):
mz, it = clean(np.array(r['mzs'], float), np.array(r['intensities'], float), float(r['pepmass']))
scores = [cosine_greedy(mz, it, *g)[0] for g in G]
k = int(np.argmax(scores))
ax.vlines(mz, 0, it, color='#3b6ea8', lw=1.2)
ax.vlines(G[k][0], 0, -G[k][1], color='#c0504d', lw=1.2)
ax.axhline(0, color='k', lw=0.5); ax.set_yticks([])
ax.set_title(f"timsTOF CE {r['collision_energies']} (CCS {r['ccs']}) vs best GNPS Orbitrap spectrum: cosine {scores[k]:.2f}", fontsize=8)
plt.xlabel('m/z'); plt.tight_layout(); plt.show()
Output:
Enveda-180 2 [M+H]+ spectra; GNPS 12 (BERKELEY-LAB, GNPS-NIH-NATURALPRODUCTSLIBRARY_ROUND2_POSITIVE, GNPS-NIST14-MATCHES, MSNLIB-POSITIVE); LOTUS organisms: Primula veris
6. The same check on 150 shared molecules
2,077 Enveda-180 compounds have a positive-mode [M+H]+ spectrum in the GNPS community libraries, almost all from the MSnLib Orbitrap libraries. Take 150 at random; for each Enveda [M+H]+ spectrum, score its best cosine against GNPS spectra of the same molecule and, as a control, against GNPS spectra of a different molecule from the sample.
gk = pq.read_table('scigantic-gnps-libraries/gnps_community_spectra.parquet', filesystem=S3,
columns=['inchikey', 'adduct', 'ion_mode']).to_pandas()
gk = gk[gk.adduct.isin(['M+H', '[M+H]+']) & gk.ion_mode.str.lower().str.startswith('positive', na=False)]
shared = sorted(set(gk.inchikey.dropna()) & ekeys)
sample = sorted(np.random.default_rng(0).choice(shared, 150, replace=False))
G = {}
for r in gnps_spectra(sample):
G.setdefault(r['inchikey'], []).append(clean(r['mz'], r['intensity'], r['precursor_mz']))
res = []
for r in enveda_spectra(sample):
mz, it = clean(np.array(r['mzs'], float), np.array(r['intensities'], float), float(r['pepmass']))
if len(mz) < 3 or r['inchikey'] not in G:
continue
other = sample[(sample.index(r['inchikey']) + 1) % len(sample)]
best = lambda refs: max((cosine_greedy(mz, it, *g)[0] for g in refs if len(g[0])), default=0.0)
res.append({'ce': r['collision_energies'], 'same': best(G[r['inchikey']]), 'other': best(G.get(other, []))})
res = pd.DataFrame(res)
print(f'{len(shared):,} shared compounds; scored {len(res)} Enveda [M+H]+ spectra of {len(sample)} of them')
print(res.groupby('ce')[['same', 'other']].median().round(2).rename(columns={'same': 'same molecule', 'other': 'different molecule'}))
plt.figure(figsize=(6, 3))
plt.hist(res.same, bins=40, range=(0, 1), alpha=0.75, color='#3b6ea8', label='same molecule')
plt.hist(res.other, bins=40, range=(0, 1), alpha=0.75, color='#c0504d', label='different molecule')
plt.xlabel('best cosine, Enveda-180 timsTOF vs GNPS'); plt.ylabel('spectra'); plt.legend(); plt.tight_layout(); plt.show()
Output:
2,077 shared compounds; scored 484 Enveda [M+H]+ spectra of 150 of them
same molecule different molecule
ce
20 0.98 0.00
20,40,60 0.96 0.00
40 0.91 0.00
60 0.81 0.01
Where the files are, for your own analysis
- Enveda-180 (~/archive): enveda-180-filtered.{parquet,jsonl,mgf,msp}.gz (1,158,293 spectra) and
the unfiltered enveda-180.*.gz, byte-identical to Zenodo 20436851. The unfiltered files hold the same spectra with unfiltered peak lists (median about 139 peaks against 14); read that Parquet with iter_batches, since read_row_group fails on its nested columns.
- COCONUT: coconut-10-2026.parquet plus the two upstream zips under downloads/.
- LOTUS: 260413_frozen.csv.gz (triples) and 260413_frozen_metadata.csv.gz (taxonomy, classes).
- MassBank: massbank_open_spectra.parquet and MassBank_NISTformat.open.msp (upstream blocks).
- GNPS: gnps_community_spectra.parquet, libraries/ .mgf, libraries.json.
Every bucket has a MIRROR.md at its root with the source, licence and per-file md5. Cite the upstream resource and authors, not the mirror. Raw tables, if you want them:
pd.set_option('display.max_colwidth', 40)
pq.read_table('scigantic-massbank/massbank_open_spectra.parquet', filesystem=S3,
columns=['accession', 'name', 'inchikey', 'precursor_type', 'instrument_type', 'license', 'num_peaks']).slice(0, 5).to_pandas()
Output:
accession name inchikey \ 0 MSBNK-AAFC-AC000857 Mevastatin AJLFOPYRIVGYMJ-INTXDZFKSA-N 1 MSBNK-AAFC-AC000845 Ctrinin hydrate MBTAFMZUDOUGPS-CGKQSRJASA-N 2 MSBNK-AAFC-AC000182 Kotanin CSJOUDOXDHMIAH-UHFFFAOYSA-N 3 MSBNK-AAFC-AC000230 Stipitatic acid ZGKNMKBZOSTFCB-UHFFFAOYSA-N 4 MSBNK-AAFC-AC000237 T-2 Toxin BXFOFFBJRFZBQZ-QYWOHJEZSA-N precursor_type instrument_type license num_peaks 0 [M+H]+ LC-ESI-ITFT CC BY-SA 4.0 16 1 [M-H]- LC-ESI-ITFT CC BY-SA 4.0 12 2 [M+Na]+ LC-ESI-ITFT CC BY-SA 4.0 1 3 [M+H]+ LC-ESI-ITFT CC BY-SA 4.0 1 4 [M+H]+ LC-ESI-ITFT CC BY-SA 4.0 74
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.