MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning

Provider
Gao Laboratory
License
CC-BY-4.0 (source)
Size
411 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M)
Access
Free and public. No account with the data provider is needed.
Category
Proteomics
Source
https://doi.org/10.7910/DVN/NUCT4N

About this dataset

An ML-ready release of MSConsensus from the Gao laboratory: 102,000,000 consensus tandem mass spectra (100M train, 1M validation, 1M test) with peptide labels and precursor charge, drawn from the full 110,209,043-spectrum corpus. The corpus was built from 1.01 PB of public raw mass-spectrometry data in 1,500 PRIDE repositories covering Orbitrap and timsTOF platforms. A Weighted-Score Binning (WSBIN) consensus algorithm produces several consensus spectra per precursor, which suppresses instrument and batch effects so models generalize across vendors. Files are Parquet shards (train-00000-of-00400.parquet and so on, test and validation shards, manifest.json), about 190 GB, readable with pyarrow, polars or Hugging Face datasets. The complete corpus with per-peak annotations, provenance metadata and the random-access msz format is on Harvard Dataverse (doi:10.7910/DVN/NUCT4N), with the MSDatasets loader and a three-task benchmark (spectrum embedding retrieval, fragment intensity prediction, de novo peptide sequencing). Released under CC BY 4.0; every ingested PRIDE submission is CC BY 4.0 or CC0. Cite the MSConsensus paper.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("Gaolaboratory/MSConsensus-100M")
print(ds)

Data structure

File types in sampled files: Parquet

Files

Example file: test-00000-of-00004.parquet

Analyze this dataset without downloading it

On Scigantic, MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Proteomics datasets