OpenFold3 tutorial dataset, prepared with AtomWorks: sample notebook

A worked notebook for OpenFold3 tutorial dataset, prepared with AtomWorks, with 10 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
OpenFold Consortium (pipeline), RCSB PDB (structures)
License
Pipeline Apache 2.0; input structures are from the PDB and in the public domain (license terms)
Stored at
Amazon S3, s3://scigantic-openfold3-tutorial (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

Your first dataset

This is the OpenFold Consortium's "Your first dataset" tutorial, as a notebook, with its output already prepared.

The tutorial builds an OpenFold3 training dataset from 8 structures and then reads its validation report. Here the dataset is already built, by the same AtomWorks pipeline, and mounted read-only. So you can read the report and look at what the pipeline produces straight away, and rebuild it yourself if you want to.

By the end you will know what the pipeline produces and how to read its report. You can then run it on your own structures.

%config InlineBackend.figure_formats = ["jpeg"]

import json
import os
import subprocess
from pathlib import Path

ARCHIVE = Path(os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive"))
INPUTS = ARCHIVE / "structures"       # the 8 mmCIFs the pipeline was run on
OUT = ARCHIVE / "dataset"             # what it produced
MSAS = ARCHIVE / "alignments"         # MSAs and template data, laid out for OpenFold3

print("archive:", ARCHIVE)
for d in (INPUTS, OUT, MSAS):
    print(f"  {d.name}/", "ok" if d.is_dir() else "MISSING")

Output:

archive: /mnt/archive
  structures/ ok
  dataset/ ok
  alignments/ ok

1. Look at the input

Each one is an mmCIF file from the PDB. The set mixes monomers, complexes, ligands and DNA, so you see several kinds of output. 4x1f also has a known issue in its deposited ligand, which you will inspect in the report.

for cif in sorted(INPUTS.glob("*.cif")):
    print(f"{cif.name}  {cif.stat().st_size / 1024:>6.0f} KB")

Output:

1a3n.cif     541 KB
1fdl.cif     456 KB
1ubq.cif     101 KB
2crb.cif    2618 KB
2q2k.cif     163 KB
3u8v.cif     262 KB
4x1f.cif     296 KB
7riv.cif     294 KB

2. What the pipeline produced

Everything is under dataset/.

OutputWhat it is
structures/The cleaned structures as mmCIF. Waters and crystallization aids are gone
structure_files/The same structures as NPZ, which OpenFold3 loads in training
reference_mols/One SDF per ligand and residue type, such as 3WF.sdf
dataset_cache.jsonThe index of every structure and chain
train.ymlA ready-to-run OpenFold3 training config
cleanup_steps/What each cleanup step did to each structure
report/index.htmlThe validation report

Find 7riv: its protein chain lists its MSA as alignment_representative_id, and its templates as template_ids. Then find 1ubq: it has no templates, because it was released in 1987 and by default a template must be older than the structure.

cache = json.loads((OUT / "dataset_cache.json").read_text())
entries = cache["structure_data"]

for entry in ("7riv", "1ubq"):
    for label, chain in entries[entry]["chains"].items():
        print(entry, label,
              "msa:", chain.get("alignment_representative_id"),
              "templates:", len(chain.get("template_ids") or []))

Output:

7riv 1 msa: 530239bfc44 templates: 12
7riv 2 msa: None templates: 0
1ubq 1 msa: 233b4b0b8c4 templates: 0

Two pictures of what the pipeline did, both read from its own output.

The first is where the atoms went. The table above says waters and crystallization aids are gone, which is true and not the biggest thing that happened.

from collections import Counter
import matplotlib.pyplot as plt

plt.rcParams.update({"figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb", "axes.labelcolor": "#52514e",
                     "text.color": "#0b0b0b", "xtick.color": "#898781", "ytick.color": "#52514e", "grid.color": "#e1e0d9"})
rows = []
for p in sorted((OUT / "cleanup_steps").glob("*.json")):
    steps = json.loads(p.read_text())["steps"]
    i = [s["name"] for s in steps].index("build_assembly_1")
    assembly = steps[i]["resolved_atoms"] - steps[i - 1]["resolved_atoms"]
    first, last = steps[0]["resolved_atoms"], steps[-1]["resolved_atoms"]
    rows.append((p.stem, assembly, last - first - assembly, first, last))
rows.sort(key=lambda r: r[1] + r[2])
ids, asm, rest, atoms_in, atoms_out = zip(*rows); y = range(len(ids))

fig, ax = plt.subplots(figsize=(8.2, 3.2))
ax.barh(y, asm, 0.52, color="#2a78d6", label="biological assembly")
ax.barh(y, rest, 0.52, left=[min(a, 0) for a in asm], color="#eb6834", label="cleanup and parse")
ax.annotate(f"+{max(asm):,} symmetry copies", (max(asm), asm.index(max(asm))), xytext=(-8, 0), textcoords="offset points", ha="right", va="center", fontsize=9, color="#ffffff")
ax.annotate(f"{-min(rest):,} removed, mostly waters", (0, rest.index(min(rest))), xytext=(8, 0), textcoords="offset points", ha="left", va="center", fontsize=9, color="#52514e")
ax.axvline(0, color="#c3c2b7", lw=1); ax.set_yticks(list(y), ids); ax.tick_params(length=0)
ax.set_xlabel("atoms added or removed, against the input file"); ax.set_title("atom count change by stage", loc="left", pad=10)
ax.grid(axis="x", lw=0.8); ax.set_axisbelow(True); ax.legend(frameon=False, fontsize=9, loc="lower right")
[s.set_visible(False) for s in ax.spines.values()]
fig.tight_layout(); plt.show()

why = Counter((f["entry"], f["detail"]["reason"]) for f in json.loads((OUT / "validation.json").read_text())["findings"] if f["check"] == "V1")
for pid, a, b in sorted(zip(ids, atoms_in, atoms_out)):
    print(f"{pid}  {a:>5,} -> {b:>5,} atoms  ", ", ".join(f"{n} {r}" for (e, r), n in why.items() if e == pid))

Output:

1a3n  4,997 -> 4,538 atoms   4 water
1fdl  4,309 -> 4,306 atoms   
1ubq    660 ->   601 atoms   1 water
2crb    759 ->   759 atoms   
2q2k  1,184 -> 2,338 atoms   1 crystallization_aid
3u8v  1,356 ->   623 atoms   2 not_in_assembly_1, 1 water
4x1f  2,366 -> 4,312 atoms   2 crystallization_aid, 1 water
7riv  2,283 -> 2,131 atoms   1 water

The second is templates. The text above says 1ubq has no templates because it was released in 1987. That is a gradient across the whole set, and it hides two different kinds of zero.

import pandas as pd

kept = {(e, c["label_asym_id"]): len(c["template_ids"] or []) for e, s in entries.items() for c in s["chains"].values()}
tf = pd.read_csv(MSAS / "of3_template_data" / "template_date_filter.tsv", sep="\t")
tf["kept"] = [kept[k] for k in zip(tf.pdb_id, tf.label_asym_id)]
tf = tf.sort_values(["release_date", "pdb_id", "label_asym_id"], ascending=False, ignore_index=True)
y = range(len(tf))

fig, ax = plt.subplots(figsize=(8.2, 4.6))
ax.hlines(y, tf.n_hits, tf.kept, color="#c3c2b7", lw=1.5, zorder=1)
ax.scatter(tf.n_hits, y, s=48, color="#86b6ef", ec="#fcfcfb", lw=1.5, zorder=2, label="candidates before the date filter")
ax.scatter(tf.kept, y, s=48, color="#1c5cab", ec="#fcfcfb", lw=1.5, zorder=3, label="templates kept, from dataset_cache.json")
i = tf.release_date.idxmin()
ax.annotate(f"{tf.n_hits[i]} found, {tf.kept[i]} kept", (tf.n_hits[i], i), xytext=(10, 0), textcoords="offset points", va="center", fontsize=9, color="#52514e")
ax.set_yticks(list(y), [f"{p} {c}   {d[:4]}" for p, c, d in zip(tf.pdb_id, tf.label_asym_id, tf.release_date)])
ax.set_xlabel("templates per protein chain"); ax.set_title("template candidates, before and after the date filter", loc="left", pad=10)
ax.grid(axis="x", lw=0.8); ax.set_axisbelow(True); ax.tick_params(length=0)
ax.legend(frameon=False, fontsize=9, loc="lower right")
[s.set_visible(False) for s in ax.spines.values()]
fig.tight_layout(); plt.show()

print(tf[["pdb_id", "label_asym_id", "release_date", "n_hits", "kept"]].to_string(index=False))

Output:

pdb_id label_asym_id release_date  n_hits  kept
  7riv             A   2021-08-25      12    12
  4x1f           A_2   2015-09-09      27    15
  4x1f           A_1   2015-09-09      27    15
  3u8v             A   2011-11-09       0     0
  2q2k           C_2   2008-02-05       0     0
  2q2k           C_1   2008-02-05       0     0
  2q2k           B_2   2008-02-05       0     0
  2q2k           B_1   2008-02-05       0     0
  2crb             A   2005-11-20       2     1
  1a3n             D   1998-04-29      67     9
  1a3n             C   1998-04-29      55     7
  1a3n             B   1998-04-29      67     9
  1a3n             A   1998-04-29      55     7
  1fdl             C   1991-10-15      42     0
...

3. Read the report

validate runs sanity checks before you spend GPU time. Most dataset problems do not crash a pipeline: a wrong bond order or a shifted residue trains silently and just makes a worse model.

Severity E is an error, W a warning, I informational. The informational rows are the chains cleanup removed on purpose.

report = json.loads((OUT / "validation.json").read_text())
print(f'{len(report["entries_in"])} in -> {len(report["entries_out"])} out')

for (check, severity), count in sorted(Counter((f["check"], f["severity"]) for f in report["findings"]).items()):
    print(f"  {check:<4} {severity}  {count}")

for f in report["findings"]:
    if f["severity"] != "I":
        print(f'\n{f["check"]} {f["severity"]} {f["entry"]}: {f["message"]}')

Output:

8 in -> 8 out
  V1   I  13
  V3   I  6

The HTML report is in the archive, so it opens now rather than after a run.

Processing should be green: red would mean the pipeline changed something it should not have. Input structures lists 4x1f, orange, meaning the issue is already in the input file. Click it: its ligand 3WF has a carbon-carbon bond of 0.893 A, under the 0.9 A limit, in both copies of the ligand. The pipeline did not cause that and keeps the structure, so you decide whether it matters for your training.

Dropped is the line to act on. A flagged structure is still in the training set and the call is yours; a dropped one is not, because the pipeline could not use it. Here it is 0 of 8.

from IPython.display import IFrame, display

# One URL, so this renders the same whether you are reading the preview page
# or running in a notebook. It used to branch on JUPYTERHUB_SERVICE_PREFIX and
# fall back to printing a path, which meant the copy published to the preview
# page was pre-rendered OUTSIDE a pod and froze the fallback: every reader who
# had not launched the notebook saw "Open /mnt/archive/..." where the report
# should be. This dataset's bucket is public, so there is a URL that works
# from anywhere.
REPORT_URL = "https://scigantic-openfold3-tutorial.s3.amazonaws.com/dataset/report/index.html"
display(IFrame(REPORT_URL, width="100%", height=700))
print("Also on the mount at", OUT / "report" / "index.html")

Output:

Also on the mount at /mnt/archive/dataset/report/index.html
The 3D cleanup view

The report above is live, so you can use it from here without launching anything. Pick a structure, then Open 3D cleanup view. It replays the pipeline one step at a time, so you can see what each filter removed instead of reading that it removed something. Arrow keys step; Before / After jumps to the ends.

The 14 steps come in four phases. Parse reads the file and picks one conformer. Assembly builds the biological assembly, which is where a monomer in the file can become a dimer. Cleanup filters is the long one: waters, crystallization aids, small polymers, clashing chains, atoms not in the CCD. Final trim drops dangling bonds and terminal atoms.

1a3n is the clearest one to watch: haemoglobin, four protein chains and four hemes. Step to remove_waters and 451 waters vanish at once, which is most of what cleanup does to a well-behaved X-ray structure.

The same page carries the checks themselves, under Ligand checks, Bonds, Covalent links and Polymer chains, plus Accepted exceptions and why for the things the pipeline chose to keep. That last one is where 4x1f explains itself.

4. Train on it

train.yml is a ready-to-run OpenFold3 training config. It needs a GPU and an OpenFold3 install, so it does not run in this notebook, and its paths point at where the dataset was built rather than at this mount.

config = (OUT / "train.yml").read_text()
print("\n".join(config.splitlines()[:18]))
print("...")
print(f"\n{OUT / 'train.yml'}")

Output:

experiment_settings:
  mode: train
  seed: 42
  restart_checkpoint_path: last
  output_dir: /private/tmp/claude-501/-Users-aaronkanzer-projects-scigantic/e50fbbd5-4505-449d-b5bb-e99ba8d170bb/scratchpad/of3data/home3/tutorial/training_output
data_module_args:
  batch_size: 1
  num_workers: 0
  num_workers_validation: 0
  epoch_len: 1
logging_config:
  log_lr: false
  wandb_config: null
checkpoint_config:
  save_top_k: 0
...

5. Run it on your own structures

First get your structures into this notebook. Use Compare my own data, on the archive page or beside this notebook in the viewer: it takes .cif, .mmcif and .pdb, ten files per upload and as many uploads as you need. They land in your home folder, next to this notebook, which is not a directory the pipeline can take, so the cell below gathers them into one.

Then point --cif-dir at that directory. The pipeline is installed here, and the Chemical Component Dictionary is already on disk at CCD_PATH, so there is nothing to download. Write the output somewhere under your home folder, since the archive is read-only.

subprocess.run([
    "finetune-pipeline",
    "--cif-dir", "<your structures>",
    "--output-dir", str(Path.home() / "my-dataset"),
    "--msa-dir", str(Path.home() / "my-msas"),
    "--ccd-path", os.environ["CCD_PATH"],
    "--name", "mine",
], check=True)

It runs ten stages and prints entries: N in -> N out. The MSA and template search calls the public ColabFold server, so that stage depends on its queue.

# Uploads land flat in your home folder; --cif-dir wants a directory.
import shutil

MINE = Path.home() / "my-structures"
MINE.mkdir(exist_ok=True)
for pattern in ("*.cif", "*.mmcif", "*.pdb"):
    for f in Path.home().glob(pattern):
        shutil.copy2(f, MINE / f.name)

found = sorted(p.name for p in MINE.iterdir() if p.is_file())
print(f"{len(found)} structure(s) in {MINE}")
for name in found[:10]:
    print(" ", name)
if not found:
    print("Nothing yet. Upload some with Compare my own data, then re-run this cell.")
print("pipeline:", subprocess.run(["which", "finetune-pipeline"], capture_output=True, text=True).stdout.strip())
print("CCD:     ", os.environ["CCD_PATH"], f"({os.path.getsize(os.environ['CCD_PATH']) / 1e6:.0f} MB)")

Output:

pipeline: /opt/conda/bin/finetune-pipeline
CCD:      /opt/openfold3/chemical_component_dictionary.cif (427 MB)

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.