How to read prebuilt MSA databases for structure prediction (Thal-Kak) without unpacking them
Each database is one .tar.zst archive, and you can stream alignments from the front of it with zstandard and tarfile without extracting anything. In a Scigantic notebook the thal-kak-msa bucket is mounted at /mnt/archive, so the code reads a few megabytes of a multi-gigabyte archive in place.
The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is s3://thal-kak-msa (AWS us-east-1, public, no account needed).
Open Thal-Kak MSA databases on SciganticOpen the sample notebook: Nine MSA databases, read without unpacking them
Code
Stream the first 2,000 alignments from one HH-suite archive
import os, tarfile, time, zstandard
DB = 'mgnify_clusters' # or 'uniref100_2026_01', 'envhog', 'logan_human', 'logan_nonhuman'
N = 2000
root = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
def stream_alignments(db, n):
"""Yield the first n A3M entries of hhblits/<db>.tar.zst, extracting nothing."""
with open(f'{root}/hhblits/{db}.tar.zst', 'rb') as fh:
tar = tarfile.open(fileobj=zstandard.ZstdDecompressor().stream_reader(fh), mode='r|')
data = tar.extractfile(tar.next()) # <db>_a3m.ffdata, the alignments
rest, k = b'', 0
while k < n:
chunk = data.read(1 << 22)
if not chunk:
break
*entries, rest = (rest + chunk).split(b'\0') # each FFindex entry ends in a NUL byte
for e in entries:
if e.strip(): # two databases add an empty entry after each one
yield e.decode()
k += 1
if k == n:
break
stream_alignments.read_mb = fh.tell() / 1e6
def parse_a3m(text):
"""(name, rows), rows = [(header, aligned sequence)], the first row being the query."""
name, rows = None, []
for line in text.split('\n'):
if line.startswith('#'):
name = line[1:]
elif line.startswith('>'):
rows.append([line[1:], []])
elif line and rows:
rows[-1][1].append(line)
rows = [(h, ''.join(s)) for h, s in rows]
if rows and rows[0][0].endswith('_consensus'): # EnVhog opens with a consensus row; HH-suite skips it
rows = rows[1:]
return name, rows
def match_columns(seq):
return ''.join(c for c in seq if not c.islower()) # lowercase letters are insertions
t0 = time.time()
alignments = [parse_a3m(e) for e in stream_alignments(DB, N)]
depth = sorted(len(rows) for _, rows in alignments)
deepest = max(alignments, key=lambda a: len(a[1]))
gib = os.path.getsize(f'{root}/hhblits/{DB}.tar.zst') / 2**30
print(f'{DB}: {len(alignments):,} alignments in {time.time() - t0:.1f} s, '
f'{stream_alignments.read_mb:.1f} MB read of a {gib:.1f} GiB archive')
print(f'sequences per alignment: median {depth[len(depth) // 2]}, largest {depth[-1]}')
print(f'deepest: {deepest[1][0][0][:60]} ({len(deepest[1])} sequences, '
f'{len(match_columns(deepest[1][0][1]))} columns)')
Copied from this dataset's starter notebook on Scigantic, the one that opens when you start a notebook from the archive page (one cell, unchanged).
What the files are
Nine prebuilt multiple sequence alignment databases for protein structure prediction: five HH-suite databases under hhblits/ and four MMseqs2 databases under mmseqs/.
An HH-suite archive holds three FFindex pairs: the alignments (_a3m), profile HMMs (_hhm) and the column-state strings the HHblits prefilter scans (_cs219). The alignments come first in every HH-suite archive, which is why streaming from the start works. An MMseqs2 archive holds cluster representatives, member sequences, alignments and headers.
How the files are laid out
| Path under the mount | What it holds |
|---|---|
| hhblits/<key>.tar.zst | Five HH-suite databases: uniref100_2026_01, mgnify_clusters, envhog, logan_nonhuman, logan_human. Inside, in order: <key>_a3m.ffdata, <key>_a3m.ffindex, then the _cs219 and _hhm pairs. |
| mmseqs/<key>.tar.zst | Four MMseqs2 databases: mgnify_clusters, envhog_std, bfd_reduced, logan_human. |
| README.md, LICENSE.txt | Database sizes, cluster counts and the license of each database. |
Size, format and license
| Format | Zstandard-compressed tar archives of HH-suite (FFindex, A3M) and MMseqs2 databases |
|---|---|
| Size | 11 objects, 398.9 GiB: nine archives of 0.8 to 157.2 GiB. 1.77 TiB once extracted. |
| License | Per database, inherited from each upstream source; see LICENSE.txt at the bucket root. The three Logan databases carry no license grant. |
| Provider | Center for Structural Systems Biology (CSSB), Seoul National University, on AWS Open Data |
| Public source | s3://thal-kak-msa (AWS us-east-1, public, no account needed) |
| Notebook image | Cryo-EM and structural biology (zstandard preinstalled) |
Things to know before you read it
- Streaming reaches only what is at the front of an archive. Random access to one alignment needs the
.ffindexfile, which sits after 8.5 GiB or more of alignment data. - Running HHblits or MMseqs2 searches needs the archive extracted, and the MMseqs2 databases also need a search index built after extraction (about 0.70 TiB more).
- In an A3M alignment, lowercase letters are insertions relative to the query and are dropped to get match columns.
Run it
Start a notebook from the Thal-Kak MSA databases archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first. For a longer worked example, open the sample notebook Nine MSA databases, read without unpacking them.
Related guides
- How to open the OpenBind Zika virus blind challenge data in a Jupyter notebook
- How to browse CZ CryoET Data Portal datasets in a Jupyter notebook
- How to query ChEMBL bioactivity data as Parquet without installing the database
- How to query 1000 Genomes VCF files on S3 from a Jupyter notebook
Last checked 2026-11-21. All dataset guides.