ESM-2 UniRef Pretraining Data: sample notebook

A worked notebook for ESM-2 UniRef Pretraining Data, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
NVIDIA
License
cc-by-4.0
Size
515 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/nvidia/esm2_uniref_pretraining_data)
Access
Free and public. No account with the data provider is needed.
DOI
10.1126/science.ade2574

The notebook

ESM-2 UniRef pretraining data

This is the sequence corpus NVIDIA published for reproducing ESM-2 pretraining: UniRef50 cluster representatives sampled through their UniRef90 members, as plain amino-acid strings. No structures, no alignments. Everything ESM-2 knows about protein structure it inferred from sequences like these.

The dataset is a Hugging Face repository FUSE-mounted read-only at SCIGANTIC_MOUNT_PATH:

train/000.parquet ... train/511.parquet    512 shards, about 75 MB each
validation/000.parquet                     one shard, 55 MB

Three columns per row: ur50_id, ur90_id, sequence. Parquet reads are column-pruned and row-group-bounded, so a single shard opens in seconds over the mount. Reading all 512 is a 38 GB job you would plan for, not run by accident.

import os
from pathlib import Path
import pyarrow.parquet as pq

mount = Path(os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive'))
train = sorted((mount / 'train').glob('*.parquet'))
val = sorted((mount / 'validation').glob('*.parquet'))
print(f'Mount:            {mount}')
print(f'Train shards:     {len(train)}  ({train[0].name} ... {train[-1].name})')
print(f'Validation shard: {val[0].name}  {val[0].stat().st_size / 1e6:.0f} MB')

pf = pq.ParquetFile(val[0])
print()
print(pf.schema_arrow)
print(f'\n{pf.metadata.num_rows:,} rows in {pf.metadata.num_row_groups} row groups')

Output:

Mount:            /mnt/archive
Train shards:     512  (000.parquet ... 511.parquet)
Validation shard: 000.parquet  55 MB

ur50_id: large_string
ur90_id: large_string
sequence: large_string

328,360 rows in 2 row groups

1. Look at a few sequences

The validation shard is the cheap one to explore. Each row is one UniRef90 representative tagged with the UniRef50 cluster it belongs to.

import pandas as pd

df = pf.read().to_pandas()
df['length'] = df['sequence'].str.len()
pd.set_option('display.max_colwidth', 60)
print(df[['ur50_id', 'ur90_id', 'length']].head(8).to_string())
print()
print('Example sequence:')
print(df.loc[0, 'sequence'][:120] + ('...' if df.loc[0, 'length'] > 120 else ''))

Output:

                  ur50_id                 ur90_id  length
0         UniRef50_Q8WZ42         UniRef90_Q8WZ42   34350
1  UniRef50_UPI00077DB535  UniRef90_UPI00077DB535   18696
2     UniRef50_A0A5A8EDT7     UniRef90_A0A5A8EDT7   17602
3  UniRef50_UPI00292E1904  UniRef90_UPI00292E1904   14705
4  UniRef50_UPI00265BFB45  UniRef90_UPI00265BFB45   14452
5     UniRef50_A0A2E2VNL9     UniRef90_A0A2E2VNL9   14148
6     UniRef50_A0A954PR65     UniRef90_A0A954PR65   13469
7  UniRef50_UPI00237D796B  UniRef90_UPI00237D796B   13709

Example sequence:
MTTQAPTFTQPLQSVVVLEGSTATFEAHISGFPVPEVSWFRDGQVISTSTLPGVQISFSDGRAKLTIPAVTKANSGRYSLKATNGSGQATSTAELLVKAETAPPNFVQRLQSMTVRQGSQ...

2. Length distribution

ESM-2 was trained with a context of 1,024 tokens, so what fraction of the corpus fits in one window matters. The long tail is real: titin-class proteins run past 30,000 residues.

import numpy as np
import matplotlib.pyplot as plt

L = df['length'].to_numpy()
print(f'{len(L):,} sequences')
print(f'median {np.median(L):.0f}  mean {L.mean():.0f}  max {L.max():,}')
print(f'{100 * (L <= 1024).mean():.1f}% fit in a 1,024-token window')

fig, ax = plt.subplots(figsize=(8, 4))
ax.hist(np.clip(L, 0, 2000), bins=100, color='#2563eb')
ax.axvline(1024, color='k', ls='--', lw=1, label='1,024 (ESM-2 context)')
ax.set_xlabel('sequence length (clipped at 2,000)')
ax.set_ylabel('sequences')
ax.set_title('Validation shard: sequence length')
ax.legend()
plt.show()

Output:

328,360 sequences
median 195  mean 285  max 34,350
97.3% fit in a 1,024-token window

3. Amino-acid composition

The 20 standard residues plus the ambiguity codes UniRef carries (X, B, Z, U, O). ESM-2's vocabulary keeps those as separate tokens rather than dropping the sequences.

from collections import Counter

counts = Counter()
for s in df['sequence'].sample(20000, random_state=0):
    counts.update(s)
total = sum(counts.values())
order = 'ACDEFGHIKLMNPQRSTVWY'
rare = {k: v for k, v in counts.items() if k not in order}
print('Residue frequency (20,000 sequences):')
for aa in order:
    print(f'  {aa}  {100 * counts[aa] / total:5.2f}%  ' + '#' * int(200 * counts[aa] / total))
print('Non-standard codes:', {k: f'{100 * v / total:.4f}%' for k, v in sorted(rare.items())})

Output:

Residue frequency (20,000 sequences):
  A   8.71%  #################
  C   1.42%  ##
  D   5.53%  ###########
  E   6.17%  ############
  F   3.84%  #######
  G   6.95%  #############
  H   2.19%  ####
  I   5.38%  ##########
  K   5.02%  ##########
  L   9.54%  ###################
  M   2.15%  ####
  N   4.06%  ########
  P   5.12%  ##########
  Q   3.84%  #######
...

4. One row per UniRef50 cluster

Sampling is by UniRef50 cluster: for each cluster one UniRef90 member is drawn as its representative. In the validation shard that is exactly one row per ur50_id, so there is no redundancy inside the shard to deduplicate. Where the ur90_id accession differs from the ur50_id accession, the drawn member is not the cluster's own representative sequence.

print(f'{df.ur50_id.nunique():,} UniRef50 clusters among {len(df):,} rows')
same = (df.ur50_id.str.replace('UniRef50_', '') == df.ur90_id.str.replace('UniRef90_', ''))
print(f'{100 * same.mean():.1f}% of rows use the UniRef50 representative itself as the UniRef90 member')
print()
print('Rows where the drawn UniRef90 member is a different accession:')
print(df.loc[~same, ['ur50_id', 'ur90_id', 'length']].head(5).to_string())

Output:

328,360 UniRef50 clusters among 328,360 rows
99.8% of rows use the UniRef50 representative itself as the UniRef90 member

Rows where the drawn UniRef90 member is a different accession:
                     ur50_id                 ur90_id  length
691   UniRef50_UPI0022576F03  UniRef90_UPI002251CCAC    2700
2227     UniRef50_A0A8D1NC59     UniRef90_A0A8D0MPH2    1518
2834     UniRef50_A0A913YWA3     UniRef90_A0A913Y8U4    1579
3272     UniRef50_A0A1B1A065     UniRef90_A0A1B1A023    1507
3463     UniRef50_A0A0V1P5Q0     UniRef90_A0A0V1CRB1    1312

5. Reading the training shards

Same code, different directory. Read a column or two rather than the whole row when you only need lengths or ids, and iterate shards rather than concatenating them.

lengths = []
for shard in train[:4]:                       # 4 of 512
    tbl = pq.read_table(shard, columns=['sequence'])
    lengths.append(tbl['sequence'].to_pandas().str.len())

For model work, datasets.load_dataset('parquet', data_files=str(mount / 'train/*.parquet'), streaming=True) streams straight off the mount without a hub download.

6. OpenProteinSet, the other half of the story

ESM-2 learned from sequences alone. OpenFold trained on OpenProteinSet: multiple sequence alignments and templates for the same proteins. The homepage comparison card launches both archives side by side; this starter only mounts ESM-2. The cell below shows the MSA view when the companion is present and says so when it is not.

a3m = Path('~/additional-datasets/openfold/pdb/101m_A/a3m/uniref90_hits.a3m').expanduser()
if not a3m.exists():
    print('OpenProteinSet is not mounted in this session.')
    print('Launch the "ESM-2 vs OpenFold" comparison from the homepage to get both archives,')
    print('or open the OpenProteinSet archive on its own.')
else:
    names, seqs, cur = [], [], []
    with a3m.open() as fh:
        for line in fh:
            line = line.rstrip()
            if line.startswith('>'):
                if cur:
                    seqs.append(''.join(cur))
                names.append(line[1:]); cur = []
            else:
                cur.append(''.join(ch for ch in line if not ch.islower()))
        if cur:
            seqs.append(''.join(cur))
    print(f'myoglobin 101m_A: {len(seqs):,} aligned homologs, query length {len(seqs[0])}')
    arr = np.array([list(s[:len(seqs[0])]) for s in seqs[:300]])
    fig, ax = plt.subplots(figsize=(9, 4))
    ax.imshow(arr != '-', aspect='auto', cmap='Greys', interpolation='nearest')
    ax.set_xlabel('alignment column'); ax.set_ylabel('homolog')
    ax.set_title('OpenProteinSet MSA for 101m_A (300 of the hits, filled = residue present)')
    plt.show()

Output:

OpenProteinSet is not mounted in this session.
Launch the "ESM-2 vs OpenFold" comparison from the homepage to get both archives,
or open the OpenProteinSet archive on its own.

References

Other libraries in this image: pyarrow, pandas, numpy, matplotlib, biopython, torch.

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.