OpenFold3 tutorial dataset, prepared with AtomWorks: sample notebook
A worked notebook for OpenFold3 tutorial dataset, prepared with AtomWorks, with 10 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- OpenFold Consortium (pipeline), RCSB PDB (structures)
- License
- Pipeline Apache 2.0; input structures are from the PDB and in the public domain (license terms)
- Stored at
- Amazon S3, s3://scigantic-openfold3-tutorial (us-east-1)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Your first dataset
This is the OpenFold Consortium's "Your first dataset" tutorial, as a notebook, with its output already prepared.
The tutorial builds an OpenFold3 training dataset from 8 structures and then reads its validation report. Here the dataset is already built, by the same AtomWorks pipeline, and mounted read-only. So you can read the report and look at what the pipeline produces straight away, and rebuild it yourself if you want to.
By the end you will know what the pipeline produces and how to read its report. You can then run it on your own structures.
%config InlineBackend.figure_formats = ["jpeg"]
import json
import os
import subprocess
from pathlib import Path
ARCHIVE = Path(os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive"))
INPUTS = ARCHIVE / "structures" # the 8 mmCIFs the pipeline was run on
OUT = ARCHIVE / "dataset" # what it produced
MSAS = ARCHIVE / "alignments" # MSAs and template data, laid out for OpenFold3
print("archive:", ARCHIVE)
for d in (INPUTS, OUT, MSAS):
print(f" {d.name}/", "ok" if d.is_dir() else "MISSING")
Output:
archive: /mnt/archive structures/ ok dataset/ ok alignments/ ok
1. Look at the input
Each one is an mmCIF file from the PDB. The set mixes monomers, complexes, ligands and DNA, so you see several kinds of output. 4x1f also has a known issue in its deposited ligand, which you will inspect in the report.
for cif in sorted(INPUTS.glob("*.cif")):
print(f"{cif.name} {cif.stat().st_size / 1024:>6.0f} KB")
Output:
1a3n.cif 541 KB 1fdl.cif 456 KB 1ubq.cif 101 KB 2crb.cif 2618 KB 2q2k.cif 163 KB 3u8v.cif 262 KB 4x1f.cif 296 KB 7riv.cif 294 KB
2. What the pipeline produced
Everything is under dataset/.
| Output | What it is |
| structures/ | The cleaned structures as mmCIF. Waters and crystallization aids are gone |
| structure_files/ | The same structures as NPZ, which OpenFold3 loads in training |
| reference_mols/ | One SDF per ligand and residue type, such as 3WF.sdf |
| dataset_cache.json | The index of every structure and chain |
| train.yml | A ready-to-run OpenFold3 training config |
| cleanup_steps/ | What each cleanup step did to each structure |
| report/index.html | The validation report |
Find 7riv: its protein chain lists its MSA as alignment_representative_id, and its templates as template_ids. Then find 1ubq: it has no templates, because it was released in 1987 and by default a template must be older than the structure.
cache = json.loads((OUT / "dataset_cache.json").read_text())
entries = cache["structure_data"]
for entry in ("7riv", "1ubq"):
for label, chain in entries[entry]["chains"].items():
print(entry, label,
"msa:", chain.get("alignment_representative_id"),
"templates:", len(chain.get("template_ids") or []))
Output:
7riv 1 msa: 530239bfc44 templates: 12 7riv 2 msa: None templates: 0 1ubq 1 msa: 233b4b0b8c4 templates: 0
Two pictures of what the pipeline did, both read from its own output.
The first is where the atoms went. The table above says waters and crystallization aids are gone, which is true and not the biggest thing that happened.
from collections import Counter
import matplotlib.pyplot as plt
plt.rcParams.update({"figure.facecolor": "#fcfcfb", "axes.facecolor": "#fcfcfb", "axes.labelcolor": "#52514e",
"text.color": "#0b0b0b", "xtick.color": "#898781", "ytick.color": "#52514e", "grid.color": "#e1e0d9"})
rows = []
for p in sorted((OUT / "cleanup_steps").glob("*.json")):
steps = json.loads(p.read_text())["steps"]
i = [s["name"] for s in steps].index("build_assembly_1")
assembly = steps[i]["resolved_atoms"] - steps[i - 1]["resolved_atoms"]
first, last = steps[0]["resolved_atoms"], steps[-1]["resolved_atoms"]
rows.append((p.stem, assembly, last - first - assembly, first, last))
rows.sort(key=lambda r: r[1] + r[2])
ids, asm, rest, atoms_in, atoms_out = zip(*rows); y = range(len(ids))
fig, ax = plt.subplots(figsize=(8.2, 3.2))
ax.barh(y, asm, 0.52, color="#2a78d6", label="biological assembly")
ax.barh(y, rest, 0.52, left=[min(a, 0) for a in asm], color="#eb6834", label="cleanup and parse")
ax.annotate(f"+{max(asm):,} symmetry copies", (max(asm), asm.index(max(asm))), xytext=(-8, 0), textcoords="offset points", ha="right", va="center", fontsize=9, color="#ffffff")
ax.annotate(f"{-min(rest):,} removed, mostly waters", (0, rest.index(min(rest))), xytext=(8, 0), textcoords="offset points", ha="left", va="center", fontsize=9, color="#52514e")
ax.axvline(0, color="#c3c2b7", lw=1); ax.set_yticks(list(y), ids); ax.tick_params(length=0)
ax.set_xlabel("atoms added or removed, against the input file"); ax.set_title("atom count change by stage", loc="left", pad=10)
ax.grid(axis="x", lw=0.8); ax.set_axisbelow(True); ax.legend(frameon=False, fontsize=9, loc="lower right")
[s.set_visible(False) for s in ax.spines.values()]
fig.tight_layout(); plt.show()
why = Counter((f["entry"], f["detail"]["reason"]) for f in json.loads((OUT / "validation.json").read_text())["findings"] if f["check"] == "V1")
for pid, a, b in sorted(zip(ids, atoms_in, atoms_out)):
print(f"{pid} {a:>5,} -> {b:>5,} atoms ", ", ".join(f"{n} {r}" for (e, r), n in why.items() if e == pid))
Output:
1a3n 4,997 -> 4,538 atoms 4 water 1fdl 4,309 -> 4,306 atoms 1ubq 660 -> 601 atoms 1 water 2crb 759 -> 759 atoms 2q2k 1,184 -> 2,338 atoms 1 crystallization_aid 3u8v 1,356 -> 623 atoms 2 not_in_assembly_1, 1 water 4x1f 2,366 -> 4,312 atoms 2 crystallization_aid, 1 water 7riv 2,283 -> 2,131 atoms 1 water
The second is templates. The text above says 1ubq has no templates because it was released in 1987. That is a gradient across the whole set, and it hides two different kinds of zero.
import pandas as pd
kept = {(e, c["label_asym_id"]): len(c["template_ids"] or []) for e, s in entries.items() for c in s["chains"].values()}
tf = pd.read_csv(MSAS / "of3_template_data" / "template_date_filter.tsv", sep="\t")
tf["kept"] = [kept[k] for k in zip(tf.pdb_id, tf.label_asym_id)]
tf = tf.sort_values(["release_date", "pdb_id", "label_asym_id"], ascending=False, ignore_index=True)
y = range(len(tf))
fig, ax = plt.subplots(figsize=(8.2, 4.6))
ax.hlines(y, tf.n_hits, tf.kept, color="#c3c2b7", lw=1.5, zorder=1)
ax.scatter(tf.n_hits, y, s=48, color="#86b6ef", ec="#fcfcfb", lw=1.5, zorder=2, label="candidates before the date filter")
ax.scatter(tf.kept, y, s=48, color="#1c5cab", ec="#fcfcfb", lw=1.5, zorder=3, label="templates kept, from dataset_cache.json")
i = tf.release_date.idxmin()
ax.annotate(f"{tf.n_hits[i]} found, {tf.kept[i]} kept", (tf.n_hits[i], i), xytext=(10, 0), textcoords="offset points", va="center", fontsize=9, color="#52514e")
ax.set_yticks(list(y), [f"{p} {c} {d[:4]}" for p, c, d in zip(tf.pdb_id, tf.label_asym_id, tf.release_date)])
ax.set_xlabel("templates per protein chain"); ax.set_title("template candidates, before and after the date filter", loc="left", pad=10)
ax.grid(axis="x", lw=0.8); ax.set_axisbelow(True); ax.tick_params(length=0)
ax.legend(frameon=False, fontsize=9, loc="lower right")
[s.set_visible(False) for s in ax.spines.values()]
fig.tight_layout(); plt.show()
print(tf[["pdb_id", "label_asym_id", "release_date", "n_hits", "kept"]].to_string(index=False))
Output:
pdb_id label_asym_id release_date n_hits kept 7riv A 2021-08-25 12 12 4x1f A_2 2015-09-09 27 15 4x1f A_1 2015-09-09 27 15 3u8v A 2011-11-09 0 0 2q2k C_2 2008-02-05 0 0 2q2k C_1 2008-02-05 0 0 2q2k B_2 2008-02-05 0 0 2q2k B_1 2008-02-05 0 0 2crb A 2005-11-20 2 1 1a3n D 1998-04-29 67 9 1a3n C 1998-04-29 55 7 1a3n B 1998-04-29 67 9 1a3n A 1998-04-29 55 7 1fdl C 1991-10-15 42 0 ...
3. Read the report
validate runs sanity checks before you spend GPU time. Most dataset problems do not crash a pipeline: a wrong bond order or a shifted residue trains silently and just makes a worse model.
Severity E is an error, W a warning, I informational. The informational rows are the chains cleanup removed on purpose.
report = json.loads((OUT / "validation.json").read_text())
print(f'{len(report["entries_in"])} in -> {len(report["entries_out"])} out')
for (check, severity), count in sorted(Counter((f["check"], f["severity"]) for f in report["findings"]).items()):
print(f" {check:<4} {severity} {count}")
for f in report["findings"]:
if f["severity"] != "I":
print(f'\n{f["check"]} {f["severity"]} {f["entry"]}: {f["message"]}')
Output:
8 in -> 8 out V1 I 13 V3 I 6
The HTML report is in the archive, so it opens now rather than after a run.
Processing should be green: red would mean the pipeline changed something it should not have. Input structures lists 4x1f, orange, meaning the issue is already in the input file. Click it: its ligand 3WF has a carbon-carbon bond of 0.893 A, under the 0.9 A limit, in both copies of the ligand. The pipeline did not cause that and keeps the structure, so you decide whether it matters for your training.
Dropped is the line to act on. A flagged structure is still in the training set and the call is yours; a dropped one is not, because the pipeline could not use it. Here it is 0 of 8.
from IPython.display import IFrame, display
# One URL, so this renders the same whether you are reading the preview page
# or running in a notebook. It used to branch on JUPYTERHUB_SERVICE_PREFIX and
# fall back to printing a path, which meant the copy published to the preview
# page was pre-rendered OUTSIDE a pod and froze the fallback: every reader who
# had not launched the notebook saw "Open /mnt/archive/..." where the report
# should be. This dataset's bucket is public, so there is a URL that works
# from anywhere.
REPORT_URL = "https://scigantic-openfold3-tutorial.s3.amazonaws.com/dataset/report/index.html"
display(IFrame(REPORT_URL, width="100%", height=700))
print("Also on the mount at", OUT / "report" / "index.html")
Output:
Also on the mount at /mnt/archive/dataset/report/index.html
The 3D cleanup view
The report above is live, so you can use it from here without launching anything. Pick a structure, then Open 3D cleanup view. It replays the pipeline one step at a time, so you can see what each filter removed instead of reading that it removed something. Arrow keys step; Before / After jumps to the ends.
The 14 steps come in four phases. Parse reads the file and picks one conformer. Assembly builds the biological assembly, which is where a monomer in the file can become a dimer. Cleanup filters is the long one: waters, crystallization aids, small polymers, clashing chains, atoms not in the CCD. Final trim drops dangling bonds and terminal atoms.
1a3n is the clearest one to watch: haemoglobin, four protein chains and four hemes. Step to remove_waters and 451 waters vanish at once, which is most of what cleanup does to a well-behaved X-ray structure.
The same page carries the checks themselves, under Ligand checks, Bonds, Covalent links and Polymer chains, plus Accepted exceptions and why for the things the pipeline chose to keep. That last one is where 4x1f explains itself.
4. Train on it
train.yml is a ready-to-run OpenFold3 training config. It needs a GPU and an OpenFold3 install, so it does not run in this notebook, and its paths point at where the dataset was built rather than at this mount.
config = (OUT / "train.yml").read_text()
print("\n".join(config.splitlines()[:18]))
print("...")
print(f"\n{OUT / 'train.yml'}")
Output:
experiment_settings: mode: train seed: 42 restart_checkpoint_path: last output_dir: /private/tmp/claude-501/-Users-aaronkanzer-projects-scigantic/e50fbbd5-4505-449d-b5bb-e99ba8d170bb/scratchpad/of3data/home3/tutorial/training_output data_module_args: batch_size: 1 num_workers: 0 num_workers_validation: 0 epoch_len: 1 logging_config: log_lr: false wandb_config: null checkpoint_config: save_top_k: 0 ...
5. Run it on your own structures
First get your structures into this notebook. Use Compare my own data, on the archive page or beside this notebook in the viewer: it takes .cif, .mmcif and .pdb, ten files per upload and as many uploads as you need. They land in your home folder, next to this notebook, which is not a directory the pipeline can take, so the cell below gathers them into one.
Then point --cif-dir at that directory. The pipeline is installed here, and the Chemical Component Dictionary is already on disk at CCD_PATH, so there is nothing to download. Write the output somewhere under your home folder, since the archive is read-only.
subprocess.run([
"finetune-pipeline",
"--cif-dir", "<your structures>",
"--output-dir", str(Path.home() / "my-dataset"),
"--msa-dir", str(Path.home() / "my-msas"),
"--ccd-path", os.environ["CCD_PATH"],
"--name", "mine",
], check=True)
It runs ten stages and prints entries: N in -> N out. The MSA and template search calls the public ColabFold server, so that stage depends on its queue.
- Have .pdb files or ligands named LIG? See Prepare a dataset from PDB files. Expect a few drops: the SMILES is the only source of a ligand's bond orders, and covalent compounds fail on an atom count because the warhead is shared with the cysteine.
- Want to hold out structures? See Add a validation split.
- Want to keep waters or short peptides? See Switch cleanup steps off.
# Uploads land flat in your home folder; --cif-dir wants a directory.
import shutil
MINE = Path.home() / "my-structures"
MINE.mkdir(exist_ok=True)
for pattern in ("*.cif", "*.mmcif", "*.pdb"):
for f in Path.home().glob(pattern):
shutil.copy2(f, MINE / f.name)
found = sorted(p.name for p in MINE.iterdir() if p.is_file())
print(f"{len(found)} structure(s) in {MINE}")
for name in found[:10]:
print(" ", name)
if not found:
print("Nothing yet. Upload some with Compare my own data, then re-run this cell.")
print("pipeline:", subprocess.run(["which", "finetune-pipeline"], capture_output=True, text=True).stdout.strip())
print("CCD: ", os.environ["CCD_PATH"], f"({os.path.getsize(os.environ['CCD_PATH']) / 1e6:.0f} MB)")
Output:
pipeline: /opt/conda/bin/finetune-pipeline CCD: /opt/openfold3/chemical_component_dictionary.cif (427 MB)
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.