# MSConsensus-100M: 102 million consensus MS/MS spectra from 1,500 PRIDE projects for proteomics machine learning

## Facts

- Provider: Gao Laboratory
- License: CC-BY-4.0 (https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M)
- Size: 411 files
- Format: Parquet (in sampled files)
- Stored at: Hugging Face, https://huggingface.co/datasets/Gaolaboratory/MSConsensus-100M
- Access: public, readable without an account
- Category: Proteomics
- Source: https://doi.org/10.7910/DVN/NUCT4N
- Page: https://scigantic.com/archive/163295e6-3ecd-58bf-92d7-23ecc8a1a3d7
- Keywords: Proteomics, spectroscopy, benchmark, Parquet, Hugging Face

## About

An ML-ready release of MSConsensus from the Gao laboratory: 102,000,000 consensus tandem mass spectra (100M train, 1M validation, 1M test) with peptide labels and precursor charge, drawn from the full 110,209,043-spectrum corpus. The corpus was built from 1.01 PB of public raw mass-spectrometry data in 1,500 PRIDE repositories covering Orbitrap and timsTOF platforms. A Weighted-Score Binning (WSBIN) consensus algorithm produces several consensus spectra per precursor, which suppresses instrument and batch effects so models generalize across vendors. Files are Parquet shards (train-00000-of-00400.parquet and so on, test and validation shards, manifest.json), about 190 GB, readable with pyarrow, polars or Hugging Face datasets. The complete corpus with per-peak annotations, provenance metadata and the random-access msz format is on Harvard Dataverse (doi:10.7910/DVN/NUCT4N), with the MSDatasets loader and a three-task benchmark (spectrum embedding retrieval, fragment intensity prediction, de novo peptide sequencing). Released under CC BY 4.0; every ingested PRIDE submission is CC BY 4.0 or CC0. Cite the MSConsensus paper.

## Load the data

```python
from datasets import load_dataset
ds = load_dataset("Gaolaboratory/MSConsensus-100M")
print(ds)
```

## Data structure

File types in sampled files: Parquet

Example file: `test-00000-of-00004.parquet`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/163295e6-3ecd-58bf-92d7-23ecc8a1a3d7
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/163295e6-3ecd-58bf-92d7-23ecc8a1a3d7
- Over MCP: `get_schema_card` with id `163295e6-3ecd-58bf-92d7-23ecc8a1a3d7` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Benchmark datasets](https://scigantic.com/topics/benchmark)
- [Spectroscopy datasets](https://scigantic.com/topics/spectroscopy)
- [Proteomics datasets](https://scigantic.com/topics/proteomics)
