ESM-2 UniRef Pretraining Data: sample notebook
A worked notebook for ESM-2 UniRef Pretraining Data, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- NVIDIA
- License
- cc-by-4.0
- Size
- 515 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/nvidia/esm2_uniref_pretraining_data)
- Access
- Free and public. No account with the data provider is needed.
- DOI
- 10.1126/science.ade2574
The notebook
ESM-2 UniRef pretraining data
This is the sequence corpus NVIDIA published for reproducing ESM-2 pretraining: UniRef50 cluster representatives sampled through their UniRef90 members, as plain amino-acid strings. No structures, no alignments. Everything ESM-2 knows about protein structure it inferred from sequences like these.
The dataset is a Hugging Face repository FUSE-mounted read-only at SCIGANTIC_MOUNT_PATH:
train/000.parquet ... train/511.parquet 512 shards, about 75 MB each
validation/000.parquet one shard, 55 MB
Three columns per row: ur50_id, ur90_id, sequence. Parquet reads are column-pruned and row-group-bounded, so a single shard opens in seconds over the mount. Reading all 512 is a 38 GB job you would plan for, not run by accident.
import os
from pathlib import Path
import pyarrow.parquet as pq
mount = Path(os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive'))
train = sorted((mount / 'train').glob('*.parquet'))
val = sorted((mount / 'validation').glob('*.parquet'))
print(f'Mount: {mount}')
print(f'Train shards: {len(train)} ({train[0].name} ... {train[-1].name})')
print(f'Validation shard: {val[0].name} {val[0].stat().st_size / 1e6:.0f} MB')
pf = pq.ParquetFile(val[0])
print()
print(pf.schema_arrow)
print(f'\n{pf.metadata.num_rows:,} rows in {pf.metadata.num_row_groups} row groups')
Output:
Mount: /mnt/archive Train shards: 512 (000.parquet ... 511.parquet) Validation shard: 000.parquet 55 MB ur50_id: large_string ur90_id: large_string sequence: large_string 328,360 rows in 2 row groups
1. Look at a few sequences
The validation shard is the cheap one to explore. Each row is one UniRef90 representative tagged with the UniRef50 cluster it belongs to.
import pandas as pd
df = pf.read().to_pandas()
df['length'] = df['sequence'].str.len()
pd.set_option('display.max_colwidth', 60)
print(df[['ur50_id', 'ur90_id', 'length']].head(8).to_string())
print()
print('Example sequence:')
print(df.loc[0, 'sequence'][:120] + ('...' if df.loc[0, 'length'] > 120 else ''))
Output:
ur50_id ur90_id length 0 UniRef50_Q8WZ42 UniRef90_Q8WZ42 34350 1 UniRef50_UPI00077DB535 UniRef90_UPI00077DB535 18696 2 UniRef50_A0A5A8EDT7 UniRef90_A0A5A8EDT7 17602 3 UniRef50_UPI00292E1904 UniRef90_UPI00292E1904 14705 4 UniRef50_UPI00265BFB45 UniRef90_UPI00265BFB45 14452 5 UniRef50_A0A2E2VNL9 UniRef90_A0A2E2VNL9 14148 6 UniRef50_A0A954PR65 UniRef90_A0A954PR65 13469 7 UniRef50_UPI00237D796B UniRef90_UPI00237D796B 13709 Example sequence: MTTQAPTFTQPLQSVVVLEGSTATFEAHISGFPVPEVSWFRDGQVISTSTLPGVQISFSDGRAKLTIPAVTKANSGRYSLKATNGSGQATSTAELLVKAETAPPNFVQRLQSMTVRQGSQ...
2. Length distribution
ESM-2 was trained with a context of 1,024 tokens, so what fraction of the corpus fits in one window matters. The long tail is real: titin-class proteins run past 30,000 residues.
import numpy as np
import matplotlib.pyplot as plt
L = df['length'].to_numpy()
print(f'{len(L):,} sequences')
print(f'median {np.median(L):.0f} mean {L.mean():.0f} max {L.max():,}')
print(f'{100 * (L <= 1024).mean():.1f}% fit in a 1,024-token window')
fig, ax = plt.subplots(figsize=(8, 4))
ax.hist(np.clip(L, 0, 2000), bins=100, color='#2563eb')
ax.axvline(1024, color='k', ls='--', lw=1, label='1,024 (ESM-2 context)')
ax.set_xlabel('sequence length (clipped at 2,000)')
ax.set_ylabel('sequences')
ax.set_title('Validation shard: sequence length')
ax.legend()
plt.show()
Output:
328,360 sequences median 195 mean 285 max 34,350 97.3% fit in a 1,024-token window
3. Amino-acid composition
The 20 standard residues plus the ambiguity codes UniRef carries (X, B, Z, U, O). ESM-2's vocabulary keeps those as separate tokens rather than dropping the sequences.
from collections import Counter
counts = Counter()
for s in df['sequence'].sample(20000, random_state=0):
counts.update(s)
total = sum(counts.values())
order = 'ACDEFGHIKLMNPQRSTVWY'
rare = {k: v for k, v in counts.items() if k not in order}
print('Residue frequency (20,000 sequences):')
for aa in order:
print(f' {aa} {100 * counts[aa] / total:5.2f}% ' + '#' * int(200 * counts[aa] / total))
print('Non-standard codes:', {k: f'{100 * v / total:.4f}%' for k, v in sorted(rare.items())})
Output:
Residue frequency (20,000 sequences): A 8.71% ################# C 1.42% ## D 5.53% ########### E 6.17% ############ F 3.84% ####### G 6.95% ############# H 2.19% #### I 5.38% ########## K 5.02% ########## L 9.54% ################### M 2.15% #### N 4.06% ######## P 5.12% ########## Q 3.84% ####### ...
4. One row per UniRef50 cluster
Sampling is by UniRef50 cluster: for each cluster one UniRef90 member is drawn as its representative. In the validation shard that is exactly one row per ur50_id, so there is no redundancy inside the shard to deduplicate. Where the ur90_id accession differs from the ur50_id accession, the drawn member is not the cluster's own representative sequence.
print(f'{df.ur50_id.nunique():,} UniRef50 clusters among {len(df):,} rows')
same = (df.ur50_id.str.replace('UniRef50_', '') == df.ur90_id.str.replace('UniRef90_', ''))
print(f'{100 * same.mean():.1f}% of rows use the UniRef50 representative itself as the UniRef90 member')
print()
print('Rows where the drawn UniRef90 member is a different accession:')
print(df.loc[~same, ['ur50_id', 'ur90_id', 'length']].head(5).to_string())
Output:
328,360 UniRef50 clusters among 328,360 rows
99.8% of rows use the UniRef50 representative itself as the UniRef90 member
Rows where the drawn UniRef90 member is a different accession:
ur50_id ur90_id length
691 UniRef50_UPI0022576F03 UniRef90_UPI002251CCAC 2700
2227 UniRef50_A0A8D1NC59 UniRef90_A0A8D0MPH2 1518
2834 UniRef50_A0A913YWA3 UniRef90_A0A913Y8U4 1579
3272 UniRef50_A0A1B1A065 UniRef90_A0A1B1A023 1507
3463 UniRef50_A0A0V1P5Q0 UniRef90_A0A0V1CRB1 1312
5. Reading the training shards
Same code, different directory. Read a column or two rather than the whole row when you only need lengths or ids, and iterate shards rather than concatenating them.
lengths = []
for shard in train[:4]: # 4 of 512
tbl = pq.read_table(shard, columns=['sequence'])
lengths.append(tbl['sequence'].to_pandas().str.len())
For model work, datasets.load_dataset('parquet', data_files=str(mount / 'train/*.parquet'), streaming=True) streams straight off the mount without a hub download.
6. OpenProteinSet, the other half of the story
ESM-2 learned from sequences alone. OpenFold trained on OpenProteinSet: multiple sequence alignments and templates for the same proteins. The homepage comparison card launches both archives side by side; this starter only mounts ESM-2. The cell below shows the MSA view when the companion is present and says so when it is not.
a3m = Path('~/additional-datasets/openfold/pdb/101m_A/a3m/uniref90_hits.a3m').expanduser()
if not a3m.exists():
print('OpenProteinSet is not mounted in this session.')
print('Launch the "ESM-2 vs OpenFold" comparison from the homepage to get both archives,')
print('or open the OpenProteinSet archive on its own.')
else:
names, seqs, cur = [], [], []
with a3m.open() as fh:
for line in fh:
line = line.rstrip()
if line.startswith('>'):
if cur:
seqs.append(''.join(cur))
names.append(line[1:]); cur = []
else:
cur.append(''.join(ch for ch in line if not ch.islower()))
if cur:
seqs.append(''.join(cur))
print(f'myoglobin 101m_A: {len(seqs):,} aligned homologs, query length {len(seqs[0])}')
arr = np.array([list(s[:len(seqs[0])]) for s in seqs[:300]])
fig, ax = plt.subplots(figsize=(9, 4))
ax.imshow(arr != '-', aspect='auto', cmap='Greys', interpolation='nearest')
ax.set_xlabel('alignment column'); ax.set_ylabel('homolog')
ax.set_title('OpenProteinSet MSA for 101m_A (300 of the hits, filled = residue present)')
plt.show()
Output:
OpenProteinSet is not mounted in this session. Launch the "ESM-2 vs OpenFold" comparison from the homepage to get both archives, or open the OpenProteinSet archive on its own.
References
- NVIDIA dataset card: https://huggingface.co/datasets/nvidia/esm2_uniref_pretraining_data
- ESM-2: Lin et al. 2023, Science, "Evolutionary-scale prediction of atomic-level protein structure with a language model"
- UniRef: https://www.uniprot.org/help/uniref
- OpenProteinSet: Ahdritz et al. 2023, https://registry.opendata.aws/openfold/
Other libraries in this image: pyarrow, pandas, numpy, matplotlib, biopython, torch.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.