SciganticGuidesTopicsExplore

How to read prebuilt MSA databases for structure prediction (Thal-Kak) without unpacking them

Each database is one .tar.zst archive, and you can stream alignments from the front of it with zstandard and tarfile without extracting anything. In a Scigantic notebook the thal-kak-msa bucket is mounted at /mnt/archive, so the code reads a few megabytes of a multi-gigabyte archive in place.

The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is s3://thal-kak-msa (AWS us-east-1, public, no account needed).

Open Thal-Kak MSA databases on SciganticOpen the sample notebook: Nine MSA databases, read without unpacking them

Code

Stream the first 2,000 alignments from one HH-suite archive

import os, tarfile, time, zstandard
DB = 'mgnify_clusters'   # or 'uniref100_2026_01', 'envhog', 'logan_human', 'logan_nonhuman'
N = 2000

root = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')

def stream_alignments(db, n):
    """Yield the first n A3M entries of hhblits/<db>.tar.zst, extracting nothing."""
    with open(f'{root}/hhblits/{db}.tar.zst', 'rb') as fh:
        tar = tarfile.open(fileobj=zstandard.ZstdDecompressor().stream_reader(fh), mode='r|')
        data = tar.extractfile(tar.next())              # <db>_a3m.ffdata, the alignments
        rest, k = b'', 0
        while k < n:
            chunk = data.read(1 << 22)
            if not chunk:
                break
            *entries, rest = (rest + chunk).split(b'\0')  # each FFindex entry ends in a NUL byte
            for e in entries:
                if e.strip():                           # two databases add an empty entry after each one
                    yield e.decode()
                    k += 1
                    if k == n:
                        break
        stream_alignments.read_mb = fh.tell() / 1e6

def parse_a3m(text):
    """(name, rows), rows = [(header, aligned sequence)], the first row being the query."""
    name, rows = None, []
    for line in text.split('\n'):
        if line.startswith('#'):
            name = line[1:]
        elif line.startswith('>'):
            rows.append([line[1:], []])
        elif line and rows:
            rows[-1][1].append(line)
    rows = [(h, ''.join(s)) for h, s in rows]
    if rows and rows[0][0].endswith('_consensus'):      # EnVhog opens with a consensus row; HH-suite skips it
        rows = rows[1:]
    return name, rows

def match_columns(seq):
    return ''.join(c for c in seq if not c.islower())   # lowercase letters are insertions

t0 = time.time()
alignments = [parse_a3m(e) for e in stream_alignments(DB, N)]
depth = sorted(len(rows) for _, rows in alignments)
deepest = max(alignments, key=lambda a: len(a[1]))
gib = os.path.getsize(f'{root}/hhblits/{DB}.tar.zst') / 2**30
print(f'{DB}: {len(alignments):,} alignments in {time.time() - t0:.1f} s, '
      f'{stream_alignments.read_mb:.1f} MB read of a {gib:.1f} GiB archive')
print(f'sequences per alignment: median {depth[len(depth) // 2]}, largest {depth[-1]}')
print(f'deepest: {deepest[1][0][0][:60]} ({len(deepest[1])} sequences, '
      f'{len(match_columns(deepest[1][0][1]))} columns)')

Copied from this dataset's starter notebook on Scigantic, the one that opens when you start a notebook from the archive page (one cell, unchanged).

What the files are

Nine prebuilt multiple sequence alignment databases for protein structure prediction: five HH-suite databases under hhblits/ and four MMseqs2 databases under mmseqs/.

An HH-suite archive holds three FFindex pairs: the alignments (_a3m), profile HMMs (_hhm) and the column-state strings the HHblits prefilter scans (_cs219). The alignments come first in every HH-suite archive, which is why streaming from the start works. An MMseqs2 archive holds cluster representatives, member sequences, alignments and headers.

How the files are laid out

Path under the mountWhat it holds
hhblits/<key>.tar.zstFive HH-suite databases: uniref100_2026_01, mgnify_clusters, envhog, logan_nonhuman, logan_human. Inside, in order: <key>_a3m.ffdata, <key>_a3m.ffindex, then the _cs219 and _hhm pairs.
mmseqs/<key>.tar.zstFour MMseqs2 databases: mgnify_clusters, envhog_std, bfd_reduced, logan_human.
README.md, LICENSE.txtDatabase sizes, cluster counts and the license of each database.

Size, format and license

FormatZstandard-compressed tar archives of HH-suite (FFindex, A3M) and MMseqs2 databases
Size11 objects, 398.9 GiB: nine archives of 0.8 to 157.2 GiB. 1.77 TiB once extracted.
LicensePer database, inherited from each upstream source; see LICENSE.txt at the bucket root. The three Logan databases carry no license grant.
ProviderCenter for Structural Systems Biology (CSSB), Seoul National University, on AWS Open Data
Public sources3://thal-kak-msa (AWS us-east-1, public, no account needed)
Notebook imageCryo-EM and structural biology (zstandard preinstalled)

Things to know before you read it

Run it

Start a notebook from the Thal-Kak MSA databases archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first. For a longer worked example, open the sample notebook Nine MSA databases, read without unpacking them.

Related guides

Last checked 2026-11-21. All dataset guides.