MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning
- Provider
- Gao Laboratory
- License
- CC-BY-4.0 (source)
- Size
- 411 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Proteomics
- Source
- https://doi.org/10.7910/DVN/NUCT4N
About this dataset
An ML-ready release of MSConsensus from the Gao laboratory: 102,000,000 consensus tandem mass spectra (100M train, 1M validation, 1M test) with peptide labels and precursor charge, drawn from the full 110,209,043-spectrum corpus. The corpus was built from 1.01 PB of public raw mass-spectrometry data in 1,500 PRIDE repositories covering Orbitrap and timsTOF platforms. A Weighted-Score Binning (WSBIN) consensus algorithm produces several consensus spectra per precursor, which suppresses instrument and batch effects so models generalize across vendors. Files are Parquet shards (train-00000-of-00400.parquet and so on, test and validation shards, manifest.json), about 190 GB, readable with pyarrow, polars or Hugging Face datasets. The complete corpus with per-peak annotations, provenance metadata and the random-access msz format is on Harvard Dataverse (doi:10.7910/DVN/NUCT4N), with the MSDatasets loader and a three-task benchmark (spectrum embedding retrieval, fragment intensity prediction, de novo peptide sequencing). Released under CC BY 4.0; every ingested PRIDE submission is CC BY 4.0 or CC0. Cite the MSConsensus paper.
How to load this dataset in Python
from datasets import load_dataset
ds = load_dataset("Gaolaboratory/MSConsensus-100M")
print(ds)Data structure
File types in sampled files: Parquet
Files
Example file: test-00000-of-00004.parquet
Analyze this dataset without downloading it
On Scigantic, MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Proteomics datasets
- Data from: Growth of maize on mixed nitrate and ammonium promotes auxin synthesis and biomass accumulation
- Chemical and Random Additive Noise Elimination (CRANE)
- Dysregulation in mTOR/HIF-1 signaling identified by proteo-transcriptomics of SARS- CoV-2 infected cells - Proteomic data obtained with Huh-7 cells
- Escape from NK cell tumor surveillance by NGFR-induced lipid remodeling in melanoma
- Lemonade Creek, Yellowstone National Park, USA - Microbial Community Analysis - Metabolomics Data
- Changes to the sebum lipidome upon COVID-19 infection observed via rapid sampling from the skin
- EATRIS-Plus multi-omics data of a human reference cohort
- AntiRef: reference clusters of human antibody sequences