Alchemistry 2027 Antiviral Blind Challenge: Test Set: sample notebook
A worked notebook for Alchemistry 2027 Antiviral Blind Challenge: Test Set, with 26 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- OpenADMET
- License
- cc-by-4.0
- Size
- 2 data files, 241 KB total
- Format
- CSV, PDB format (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/openadmet/alchemistry-2027-antiviral-blind-challenge)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Start here. Click the grey cell below and press Shift+Enter. It reads the challenge file off the mounted archive and draws the first few compounds, so about three seconds from now you will be looking at the actual molecules.
Change SHOW and run it again to see a different slice of the 54.
# Start here: click this cell, press Shift+Enter.
SHOW = 8 # how many compounds to draw
from pathlib import Path
import pandas as pd
from rdkit import Chem, RDLogger
from rdkit.Chem import Draw
RDLogger.DisableLog("rdApp.*")
archive = Path.home() / "archive"
test = pd.read_csv(archive / "alchemistry-2027-challenge_TEST_BLINDED.csv")
SMI = [c for c in test.columns if "SMILES" in c.upper()][0]
print(f"{len(test)} compounds to predict: {test.Molecule_Name.iloc[0]} through "
f"{test.Molecule_Name.iloc[-1]}")
print(f"columns in the file: {list(test.columns)}")
print("there is no potency column, in this file or anywhere else in the archive.")
print("that absence IS the challenge.")
Draw.MolsToGridImage([Chem.MolFromSmiles(s) for s in test[SMI][:SHOW]],
molsPerRow=4, subImgSize=(200, 170),
legends=list(test.Molecule_Name[:SHOW]))
Alchemistry 2027 Challenge: 54 compounds, no answer key
Someone has already measured how tightly each of those 54 compounds sticks to an enzyme that dengue virus needs in order to copy itself. Those measurements are not public, and will not be until the challenge closes. Your job is to predict them.
What you submit is one number per compound: a binding free energy in kcal/mol, plus an uncertainty. More negative means it binds more tightly. A number near -6 is a weak hit; near -12 is a very good drug candidate. Section 1 turns that hand-waving into a measured distribution you can aim at.
Why this one is unusual. Almost every benchmark in machine learning hands you a training set: examples with the answers attached, so you can fit something. This challenge hands you 54 molecules, one crystal structure, and nothing else. The organizers chose this target partly *because* there is no significant trainable public data on it. So the usual move, fit a model to known actives, is not available to anyone, which is what makes it a fair test of physics- based prediction.
The practical details. Opened 1 November 2026. Interim deadline 15 January 2027, interim leaderboard 25 January, closes 25 March 2027, results presented at the Alchemistry 2027 conference in May at Merck's South San Francisco labs. One submission per 48 hours. Ranking is by Kendall's tau, a rank correlation: getting the *order* right matters more than getting the absolute numbers right. Section 7 shows why that choice is almost forced.
What is mounted here. The challenge data at ~/archive, plus five more datasets chosen because each answers a question the challenge itself cannot. You do not have to download anything; they stream from S3 as you read them.
| mount | what it answers |
| ~/archive | the 54 compounds and the one crystal structure |
| ChEMBL 37 | what a real binding number looks like, and whether these are already published |
| OpenBind Zika | does a pocket model from a sibling virus transfer |
| OpenBind EV-A71 | is a pocket this small normal |
| Roche PDE10A | how much signal a series like this actually contains |
| PubChem | has anyone deposited these structures anywhere, ever |
Run the next cell to confirm what is actually attached.
# Where a companion lands depends on how it was mounted. A public S3 companion attached at
# spawn is a CSI volume at /mnt/archives/<bucket>; a GCS, HTTP or live-added mount is a
# subdirectory of /mnt/additional-mounts, which is symlinked to ~/additional-datasets.
# Search all three rather than hardcoding one.
CANDIDATE_ROOTS = [Path("/mnt/archives"), Path.home() / "additional-datasets",
Path("/mnt/additional-mounts")]
def find_mount(bucket):
for root in CANDIDATE_ROOTS:
p = root / bucket
if p.is_dir():
return p
return None
COMPANIONS = {
"chembl": "scigantic-chembl",
"zika": "scigantic-openbind-challenge-1",
"eva71": "scigantic-openbind",
"pde10a": "scigantic-pde10a-benchmark",
"pubchem": "scigantic-pubchem",
}
MOUNTS = {k: find_mount(v) for k, v in COMPANIONS.items()}
for k, v in MOUNTS.items():
print(f"{k:8s} {COMPANIONS[k]:32s} {v if v else 'NOT MOUNTED (its cells will skip)'}")
Output:
chembl scigantic-chembl /mnt/additional-mounts/scigantic-chembl zika scigantic-openbind-challenge-1 /mnt/additional-mounts/scigantic-openbind-challenge-1 eva71 scigantic-openbind /mnt/additional-mounts/scigantic-openbind pde10a scigantic-pde10a-benchmark /mnt/additional-mounts/scigantic-pde10a-benchmark pubchem scigantic-pubchem /mnt/additional-mounts/scigantic-pubchem
# Figures in this notebook are JPEG at 72 dpi on purpose: a starter notebook is capped at
# 512 KB in total and any single PNG output over 128 KB is refused outright by the uploader.
import io
import matplotlib.pyplot as plt
import matplotlib_inline.backend_inline as inline_backend
from IPython.display import Image, display
from PIL import Image as PILImage
inline_backend.set_matplotlib_formats("jpeg")
plt.rcParams.update({"figure.dpi": 72, "font.size": 9, "axes.grid": True,
"grid.alpha": 0.25, "axes.spines.top": False,
"axes.spines.right": False})
TEAL, GOLD, INK = "#2a8f8f", "#d19a2a", "#2b3a42"
def show_mols(mols, legends, cols=4, size=(200, 170), quality=78, **kw):
# Emit as JPEG, so the uploader's 128 KB PNG cap cannot bite.
#
# MolsToGridImage returns a DIFFERENT type depending on context: a PIL image in a
# plain script, but an IPython.display.Image once a kernel is attached. Checking
# this in a script is a false positive; handle all three shapes.
img = Draw.MolsToGridImage(list(mols), molsPerRow=cols, subImgSize=size,
legends=list(legends), useSVG=False, **kw)
if hasattr(img, "convert"):
pil = img
else:
pil = PILImage.open(io.BytesIO(getattr(img, "data", img)))
buf = io.BytesIO()
pil.convert("RGB").save(buf, format="JPEG", quality=quality, optimize=True)
display(Image(data=buf.getvalue(), format="jpeg"))
print("figure helpers ready")
Output:
figure helpers ready
1. The size of the answer
If you have never predicted a binding free energy before, this is the first real question, and it has a concrete answer: look at what has actually been measured on this family of enzymes.
ChEMBL is the curated record of published bioactivity, 24.5 million measurements in release 37. It is mounted here. The flavivirus proteases in it are the dengue-2 polyprotein, Zika NS3, West Nile NS3 and the dengue-3 protease. The file derived/activities_enriched.parquet is sorted by target, so pulling all four is a filtered read of a few hundred milliseconds rather than a scan of 4.7 million rows.
Potency is published as pIC50, pKi or pKd. One line converts any of them to a free energy:
$$\Delta G \approx -1.364 \times \mathrm{pChEMBL} \quad \text{kcal/mol at 298 K}$$
import numpy as np
import pyarrow.dataset as ds
FLAVI = {
"CHEMBL5980": "dengue-2 polyprotein",
"CHEMBL4523954": "Zika NS3",
"CHEMBL5705861": "West Nile NS3",
"CHEMBL5705863": "dengue-3 NS3 protease",
}
ch = MOUNTS["chembl"]
if ch is None:
print("ChEMBL not mounted; skipping.")
else:
acts = ds.dataset(ch / "chembl_37/derived/activities_enriched.parquet", format="parquet")
flavi = acts.to_table(
filter=ds.field("target_chembl_id").isin(list(FLAVI)),
columns=["target_chembl_id", "standard_type", "pchembl_value", "assay_type",
"standard_inchi_key", "doc_year", "confidence_score"],
).to_pandas()
flavi["label"] = flavi.target_chembl_id.map(FLAVI)
flavi["dG"] = -1.364 * flavi.pchembl_value # kcal/mol at 298 K
print(f"{len(flavi)} measured activities on four flavivirus proteases\n")
summary = (flavi.groupby("label")
.agg(n=("dG", "size"), compounds=("standard_inchi_key", "nunique"),
tightest=("dG", "min"), median=("dG", "median"), weakest=("dG", "max"))
.round(1).sort_values("n", ascending=False))
print(summary.to_string())
denv = flavi[flavi.label == "dengue-2 polyprotein"]
lo, hi = denv.dG.quantile(0.10), denv.dG.quantile(0.90)
print(f"\nanything ever measured on this enzyme family spans "
f"{flavi.dG.min():.1f} to {flavi.dG.max():.1f} kcal/mol.")
print(f"a prediction outside roughly -4 to -16 is a unit error, not a result.")
n_floor = int((denv.pchembl_value <= 4.0).sum())
print(f"\nBut the published dengue-2 literature is WEAK.")
print(f" 80% of it sits between {lo:.1f} and {hi:.1f}, with a median of {denv.dG.median():.1f}.")
print(f" {n_floor} of {len(denv)} rows are pinned at the assay floor of {-1.364*4:.1f},")
print(f" which means 'not active at the highest concentration we tried'. Censored,")
print(f" not measured. Never fit to those.")
print(f"\nThe 54 challenge compounds are an optimised series, so expect them at the")
print(f"tight end: nearer the {denv.dG.min():.1f} that is the best dengue-2 number anyone has")
print(f"published than the {denv.dG.median():.1f} median of the literature.")
Output:
3989 measured activities on four flavivirus proteases
n compounds tightest median weakest
label
Zika NS3 1790 1346 -12.0 -7.4 -5.5
dengue-2 polyprotein 1790 973 -14.0 -6.5 -5.5
West Nile NS3 316 292 -9.8 -7.0 -5.5
dengue-3 NS3 protease 93 82 -8.3 -6.7 -5.5
anything ever measured on this enzyme family spans -14.0 to -5.5 kcal/mol.
a prediction outside roughly -4 to -16 is a unit error, not a result.
But the published dengue-2 literature is WEAK.
80% of it sits between -8.0 and -5.7, with a median of -6.5.
11 of 1790 rows are pinned at the assay floor of -5.5,
...
if MOUNTS["chembl"] is not None:
fig, ax = plt.subplots(figsize=(7.2, 3.0))
bins = np.arange(-15.5, -4.9, 0.5)
ax.hist(denv.dG, bins=bins, color=TEAL, alpha=0.85, label="dengue-2 polyprotein")
others = flavi[flavi.label != "dengue-2 polyprotein"]
ax.hist(others.dG, bins=bins, histtype="step", linewidth=1.6, color=GOLD,
label="Zika / West Nile / dengue-3")
ax.axvspan(lo, hi, color=INK, alpha=0.07, zorder=0)
ax.axvline(denv.dG.median(), color=INK, linestyle="--", linewidth=1.2)
ax.axvline(-1.364 * 4, color=GOLD, linestyle=":", linewidth=1.5)
box = dict(boxstyle="round,pad=0.22", fc="white", ec="none", alpha=0.88)
ax.annotate(f"median {denv.dG.median():.1f}", (denv.dG.median(), ax.get_ylim()[1] * 0.96),
xytext=(-5, 0), textcoords="offset points", ha="right", va="top",
fontsize=8, color=INK, bbox=box)
ax.annotate("assay floor:\n'not active'", (-1.364 * 4, ax.get_ylim()[1] * 0.60),
xytext=(-5, 0), textcoords="offset points", ha="right", va="top",
fontsize=8, color=GOLD, bbox=box)
ax.set_xlabel("measured binding free energy (kcal/mol), tighter to the left")
ax.set_ylabel("published measurements")
ax.set_title("Every flavivirus protease measurement in ChEMBL 37",
fontsize=10, loc="left")
ax.legend(frameon=False, fontsize=8, loc="upper left")
fig.tight_layout()
plt.show()
Read that plot carefully, because the obvious reading of it is wrong.
The shaded band is where 80% of the *published* dengue-2 numbers sit, and it is weak: roughly -5.7 to -8.0. That is not your target zone. It is a statement about the literature, which on this enzyme is mostly early-stage compounds and outright inactives. The spike hard against the right-hand edge is not a measurement at all, it is the detection limit of the assay, compounds recorded as "not active up to the highest concentration we tried". Treat that edge as censored data, never as a value.
What the plot is genuinely good for is the outer bounds. Nothing in the entire flavivirus protease literature is tighter than -14 or weaker than -5.5, so a prediction of -25 or of +3 is a unit error or a sign error, and you now have the evidence to say so. The 54 challenge compounds are a lead-optimisation series, which means they belong at the tight tail of this distribution rather than in the middle of it.
Two honest caveats before you lean on this. These rows are every published dengue-2 polyprotein measurement, which mixes protease-inhibition assays with cell-based antiviral readouts, so read it as a plausibility range rather than as a training set. And public potency on this enzyme is known to be contaminated by colloidal aggregation and to shift with construct and buffer, which is exactly why the organizers generated fresh measurements instead of reusing the literature.
2. Prior measurements
Worth two seconds of checking before you spend a week of compute. ChEMBL covers 1.68 million distinct compounds with at least one measured activity against any target. If a challenge compound were already published, this is where it would show up.
# Defined here and not inside the mount check: section 9 needs `keys` even if
# ChEMBL happens to be unmounted.
mols = {r.Molecule_Name: Chem.MolFromSmiles(r[SMI]) for _, r in test.iterrows()}
assert all(m is not None for m in mols.values()), "a CXSMILES failed to parse"
names = sorted(mols)
keys = {n: Chem.MolToInchiKey(m) for n, m in mols.items()}
if MOUNTS["chembl"] is None:
print("ChEMBL not mounted; skipping the lookup.")
else:
known = set(acts.to_table(columns=["standard_inchi_key"]).column(0).to_pylist())
hits = {n: k for n, k in keys.items() if k in known}
print(f"ChEMBL 37 covers {len(known):,} distinct compounds with a measured activity.")
print(f"challenge compounds found there: {len(hits)} of {len(keys)}")
print()
# A control, so that "zero hits" means "absent" and not "my lookup is broken".
aspirin = Chem.MolToInchiKey(Chem.MolFromSmiles("CC(=O)Oc1ccccc1C(=O)O"))
print(f"control, aspirin ({aspirin}): {'FOUND' if aspirin in known else 'MISSING'}")
print("The control is the whole point of this cell. Without it, a typo in the")
print("column name also prints zero hits and looks like a result.")
Output:
ChEMBL 37 covers 1,684,012 distinct compounds with a measured activity. challenge compounds found there: 0 of 54 control, aspirin (BSYNRYMUTXBXSQ-UHFFFAOYSA-N): FOUND The control is the whole point of this cell. Without it, a typo in the column name also prints zero hits and looks like a result.
Zero of 54, with the control passing. These compounds have no published activity against anything, let alone against this protease. The challenge really is blind, and no amount of literature mining will shortcut it.
That also means something more useful: nobody has a trained model with an unfair advantage here. Section 9 is the slow, exhaustive version of this check against PubChem's full 125-million-compound registry, if you want the stronger statement.
3. One shared core
Relative free energy methods do not compute binding from scratch. They compute the *difference* between two similar molecules, which is a far easier sum, and then chain those differences into a network. That only works if the compounds really are small edits of each other.
Here is the test, and the picture.
from rdkit.Chem import Descriptors, rdMolDescriptors, rdFMCS
from rdkit.Chem.Scaffolds import MurckoScaffold
heavy = pd.Series({n: m.GetNumHeavyAtoms() for n, m in mols.items()})
# The maximum common substructure across ALL 54, rings kept whole.
mcs = rdFMCS.FindMCS(list(mols.values()), timeout=180, ringMatchesRingOnly=True,
completeRingsOnly=True,
atomCompare=rdFMCS.AtomCompare.CompareElements,
bondCompare=rdFMCS.BondCompare.CompareOrder)
core = Chem.MolFromSmarts(mcs.smartsString)
print(f"largest substructure common to all 54: {mcs.numAtoms} heavy atoms, {mcs.numBonds} bonds")
print(f"carried by {sum(m.HasSubstructMatch(core) for m in mols.values())} of {len(mols)} compounds")
print(f"that is {mcs.numAtoms / heavy.median():.0%} of a typical molecule here "
f"({heavy.median():.0f} heavy atoms)")
print()
props = pd.DataFrame({
"MW": {n: Descriptors.MolWt(m) for n, m in mols.items()},
"cLogP": {n: Descriptors.MolLogP(m) for n, m in mols.items()},
"TPSA": {n: Descriptors.TPSA(m) for n, m in mols.items()},
"HBD": {n: rdMolDescriptors.CalcNumHBD(m) for n, m in mols.items()},
"RotB": {n: rdMolDescriptors.CalcNumRotatableBonds(m) for n, m in mols.items()},
"heavy": heavy,
})
print(props.describe().loc[["min", "50%", "max"]].round(1).to_string())
print()
print("Lead-like and tightly clustered: this is one lead-optimisation series, not a screening")
print("library. Relative methods are the right tool.")
Output:
largest substructure common to all 54: 19 heavy atoms, 21 bonds
carried by 54 of 54 compounds
that is 66% of a typical molecule here (29 heavy atoms)
MW cLogP TPSA HBD RotB heavy
min 321.4 1.8 73.1 3.0 3.0 24.0
50% 411.7 3.7 81.8 3.0 5.0 29.0
max 478.9 5.0 119.2 5.0 8.0 33.0
Lead-like and tightly clustered: this is one lead-optimisation series, not a screening
library. Relative methods are the right tool.
# Draw eight of them with the shared core picked out, so "congeneric" stops being a word
# and becomes something you can see.
pick = [names[i] for i in (0, 6, 12, 20, 26, 33, 41, 52)]
show_mols([mols[n] for n in pick], pick, cols=4, size=(205, 180),
highlightAtomLists=[list(mols[n].GetSubstructMatch(core)) for n in pick])
murcko = pd.Series({n: MurckoScaffold.MurckoScaffoldSmiles(mol=m) for n, m in mols.items()})
vc = murcko.value_counts()
print(f"Bemis-Murcko scaffolds over the same 54: {len(vc)} distinct, "
f"largest cluster {vc.iloc[0]}, {(vc == 1).sum()} singletons")
print()
print(f"Both of these are true at once:")
print(f" one {mcs.numAtoms}-atom core is in every compound -> relative methods apply")
print(f" Bemis-Murcko splits them into {len(vc)} groups -> scaffold code shatters it")
Output:
Bemis-Murcko scaffolds over the same 54: 15 distinct, largest cluster 23, 9 singletons Both of these are true at once: one 19-atom core is in every compound -> relative methods apply Bemis-Murcko splits them into 15 groups -> scaffold code shatters it
That second number is a trap worth knowing about. MurckoScaffoldSmiles promotes every different saturated amine ring and every different anilide ring into a scaffold of its own, so anything that groups, splits, deduplicates or clusters this series by Murcko scaffold will report 15 unrelated groups and quietly destroy the thing that made the series tractable. Use the MCS core as the definition of the series.
4. The reference complex
You get a single crystal structure: the protease with one of the 54 compounds actually bound. That is a real gift, because it means nobody has to guess the binding mode. But the file has been stripped, and what is missing matters.
import collections
PDB = archive / "alchemistry-2027-challenge_reference_complex.pdb"
raw_pdb = PDB.read_text().splitlines()
kinds = collections.Counter(l[:6].strip() for l in raw_pdb)
print("record types in the file:", dict(kinds))
print()
print("waters :", sum(1 for l in raw_pdb if l[17:20].strip() == "HOH"))
print("hydrogens:", sum(1 for l in raw_pdb
if l.startswith(("ATOM", "HETATM")) and l[76:78].strip() == "H"))
print("metadata :", sum(kinds.get(r, 0) for r in
("HEADER", "TITLE", "REMARK", "SEQRES", "EXPDTA", "DBREF")), "records")
print()
print([l for l in raw_pdb if l.startswith("CRYST1")][0])
print()
print("No resolution, no R-free, no deposition code, no waters, no hydrogens. ANISOU records")
print("survive, so this came out of an anisotropic refinement, but the file will not tell you")
print("the resolution that implies. You protonate and solvate it yourself.")
Output:
record types in the file: {'CRYST1': 1, 'ATOM': 1435, 'ANISOU': 1462, 'TER': 2, 'HETATM': 28, 'CONECT': 27, 'END': 1}
waters : 0
hydrogens: 0
metadata : 0 records
CRYST1 93.317 93.317 61.281 90.00 90.00 120.00 P 63 1
No resolution, no R-free, no deposition code, no waters, no hydrogens. ANISOU records
survive, so this came out of an anisotropic refinement, but the file will not tell you
the resolution that implies. You protonate and solvate it yourself.
# Chain inventory, and the gap most preparation tools will not mention.
res = {}
for l in raw_pdb:
if l.startswith(("ATOM", "HETATM")):
res[(l[21], int(l[22:26]))] = l[17:20].strip()
for ch_id in sorted({c for c, _ in res}):
nums = sorted(n for c, n in res if c == ch_id)
aa = [n for n in nums if res[(ch_id, n)] not in ("LIG", "ZN", "HOH")]
hets = collections.Counter(res[(ch_id, n)] for n in nums
if res[(ch_id, n)] in ("LIG", "ZN", "HOH"))
label = {"A": "NS2B cofactor", "B": "NS3 protease", "C": "metal"}.get(ch_id, "")
print(f"chain {ch_id} ({label}): {len(nums)} residues, numbered {min(nums)}-{max(nums)}"
+ (f", hetero {dict(hets)}" if hets else ""))
if aa:
gaps = [(a, b) for a, b in zip(aa, aa[1:]) if b - a > 1]
print(f" amino acids {aa[0]}-{aa[-1]}, GAPS: {gaps if gaps else 'none'}")
print()
for want, num in (("HIS", 51), ("ASP", 75), ("SER", 135)):
print(f" catalytic {want}{num}: present as {res.get(('B', num))}")
Output:
chain A (NS2B cofactor): 38 residues, numbered 50-87
amino acids 50-87, GAPS: none
chain B (NS3 protease): 152 residues, numbered 17-201, hetero {'LIG': 1}
amino acids 17-172, GAPS: [(27, 33)]
chain C (metal): 1 residues, numbered 101-101, hetero {'ZN': 1}
catalytic HIS51: present as HIS
catalytic ASP75: present as ASP
catalytic SER135: present as SER
import math
atoms = []
for l in raw_pdb:
if l.startswith(("ATOM", "HETATM")):
try:
xyz = (float(l[30:38]), float(l[38:46]), float(l[46:54]))
except ValueError:
continue
# Columns 77-78 are the element. Take it from there, not from the atom NAME:
# chlorine is named "CL" and a name-based read counts it as carbon.
element = l[76:78].strip().capitalize() or l[12:16].strip()[0]
atoms.append((l[21], int(l[22:26]), l[17:20].strip(), l[12:16].strip(), xyz, element))
lig = [a for a in atoms if a[2] == "LIG"]
print(f"the bound ligand: {len(lig)} heavy atoms, composition "
f"{dict(collections.Counter(a[5] for a in lig))}")
near = {}
for p in atoms:
if p[2] == "LIG":
continue
d = min(math.dist(p[4], q[4]) for q in lig)
if d <= 4.5:
k = (p[0], p[1], p[2])
near[k] = min(near.get(k, 99), d)
print(f"\nresidues within 4.5 A of it: {len(near)}")
for (ch_id, num, nm), d in sorted(near.items(), key=lambda kv: kv[1]):
tag = " <-- on the NS2B COFACTOR chain" if ch_id == "A" else ""
print(f" {nm}{num:<4} chain {ch_id} {d:.2f} A{tag}")
by_chain = collections.Counter(ch for ch, _, _ in near)
print(f"\nby chain: {dict(by_chain)}")
print(f"-> {by_chain.get('A', 0)} of {len(near)} pocket residues are NOT on the protease chain.")
zn = [a for a in atoms if a[2] == "ZN"]
if zn:
print(f"\nthe zinc is {min(math.dist(zn[0][4], q[4]) for q in lig):.1f} A from the ligand: a")
print("surface site at the construct N-terminus, not a catalytic metal. Dengue-2 protease")
print("has no structural zinc, and no flavivirus protease entry in the PDB contains one.")
print("It is crystallisation additive. Safe to drop for a pocket-local calculation.")
Output:
the bound ligand: 27 heavy atoms, composition {'N': 6, 'C': 19, 'O': 1, 'Cl': 1}
residues within 4.5 A of it: 15
PHE130 chain B 2.41 A
GLY159 chain B 2.83 A
ASP129 chain B 2.83 A
GLY151 chain B 3.07 A
ASP81 chain A 3.19 A <-- on the NS2B COFACTOR chain
TYR150 chain B 3.19 A
ASN152 chain B 3.32 A
HIS51 chain B 3.38 A
GLY82 chain A 3.39 A <-- on the NS2B COFACTOR chain
SER135 chain B 3.42 A
TYR161 chain B 3.51 A
PRO132 chain B 3.57 A
...
# A picture of the thing. Project onto the protein's own two longest axes with a plain
# SVD, no extra dependencies. ONE DOT PER RESIDUE, taken from the CA atoms, rather than
# one per atom: at atom resolution the two chains merge into a single cloud and the
# picture stops saying anything.
ca = [a for a in atoms if a[3] == "CA" and a[2] not in ("LIG", "ZN", "HOH")]
prot = np.array([a[4] for a in ca])
ch_of = np.array([a[0] for a in ca])
centre = prot.mean(axis=0)
_, _, axes_ = np.linalg.svd(prot - centre, full_matrices=False)
def proj(pts):
return (np.atleast_2d(pts) - centre) @ axes_[:2].T
fig, ax = plt.subplots(figsize=(6.6, 4.3))
for chain, colour, label in (("B", TEAL, "NS3 protease"), ("A", GOLD, "NS2B cofactor")):
p = proj(prot[ch_of == chain])
ax.plot(p[:, 0], p[:, 1], "-", color=colour, linewidth=0.7, alpha=0.40, zorder=1)
ax.scatter(p[:, 0], p[:, 1], s=14, c=colour, linewidths=0, zorder=2,
label=f"{label}, {int((ch_of == chain).sum())} residues")
pocket_ca = np.array([a[4] for a in ca if (a[0], a[1], a[2]) in near])
if len(pocket_ca):
pp = proj(pocket_ca)
ax.scatter(pp[:, 0], pp[:, 1], s=78, facecolors="none", edgecolors=INK,
linewidths=1.1, zorder=3, label=f"pocket, {len(near)} residues")
lp = proj(np.array([a[4] for a in lig]))
ax.scatter(lp[:, 0], lp[:, 1], s=26, c="#b03a3a", linewidths=0, zorder=5,
label=f"the bound compound, {len(lig)} atoms")
if zn:
zp = proj(np.array(zn[0][4]))
ax.scatter(zp[:, 0], zp[:, 1], s=85, marker="X", c="#6b7a82", zorder=5,
label="zinc, a crystallisation additive")
ax.set_aspect("equal")
ax.set_xlabel("angstroms along the longest axis of the protein")
ax.set_ylabel("angstroms")
ax.set_title(f"One compound of {len(lig)} atoms, a {len(near)}-residue pocket, "
f"and a zinc nowhere near it", fontsize=10, loc="left")
# Outside the axes: every in-axes corner has data in it.
ax.legend(frameon=False, fontsize=8, loc="upper center", bbox_to_anchor=(0.5, -0.18),
ncol=2, markerscale=1.3, handletextpad=0.4, columnspacing=1.4)
fig.tight_layout()
plt.show()
That picture carries the two facts that cost people the most time.
The compound sits on the seam between two different chains. Three of the pocket residues are on chain A, which is the NS2B cofactor, not the protease itself. Build a system out of NS3 alone, or truncate NS2B to save atoms, and you have deleted part of your own binding site. This is the active, closed, cofactor-bound form and it has to stay that way.
NS3 residues 28 to 32 are simply absent, one disordered loop. Left alone that is a chain break with two charged, unnatural ends in the middle of your protein. It sits well away from the ligand, so capping both sides is normally enough, but an uncapped break is not.
The zinc is a red herring, 26 angstroms away and a leftover from the crystallisation buffer.
5. The bound compound
The dataset card says a series member is bound but does not say which. It is worth knowing, because that compound is your anchor: it is the only one whose pose you are not guessing. Two independent routes should agree.
# Route 1: heavy-atom composition. Exactly one of the 54 can match.
target_formula = collections.Counter(a[5] for a in lig)
comp_hits = [n for n, m in mols.items()
if collections.Counter(a.GetSymbol() for a in m.GetAtoms()) == target_formula]
print(f"ligand composition {dict(target_formula)}")
print(f"compounds with exactly that heavy-atom composition: {comp_hits}")
# Route 2: connectivity. Rebuild the ligand from its coordinates alone and compare
# heavy-atom skeletons, ignoring bond orders, which a stripped PDB never records.
# sanitize=False matters: the raw block has no bond orders to perceive.
from rdkit.Chem import rdmolfiles
lig_block = "\n".join(l for l in raw_pdb
if l.startswith("HETATM") and l[17:20].strip() == "LIG")
lig_mol = rdmolfiles.MolFromPDBBlock(lig_block, sanitize=False, proximityBonding=True)
def skeleton(m):
rw = Chem.RWMol(m)
for b in rw.GetBonds():
b.SetBondType(Chem.BondType.SINGLE)
b.SetIsAromatic(False)
for a in rw.GetAtoms():
a.SetIsAromatic(False)
a.SetNoImplicit(True)
a.SetNumExplicitHs(0)
a.SetFormalCharge(0)
out = rw.GetMol()
Chem.SanitizeMol(out, Chem.SanitizeFlags.SANITIZE_SYMMRINGS, catchErrors=True)
return Chem.MolToSmiles(out)
lig_skel = skeleton(lig_mol)
skel_hits = [n for n, m in mols.items() if skeleton(Chem.RemoveHs(m)) == lig_skel]
print(f"compounds whose heavy-atom skeleton matches: {skel_hits}")
print()
ANCHOR = comp_hits[0] if comp_hits else None
assert skel_hits == comp_hits, "the two routes disagree, which should be impossible"
print(f"two independent routes, one answer: {ANCHOR} is the compound in the crystal structure.")
show_mols([mols[ANCHOR]], [f"{ANCHOR}: the only pose you are given"], cols=1, size=(330, 250))
Output:
ligand composition {'N': 6, 'C': 19, 'O': 1, 'Cl': 1}
compounds with exactly that heavy-atom composition: ['A_027']
compounds whose heavy-atom skeleton matches: ['A_027']
two independent routes, one answer: A_027 is the compound in the crystal structure.
# You have one experimental pose and 53 compounds that need one. How far is the reach?
from rdkit import DataStructs
from rdkit.Chem import rdFingerprintGenerator
gen = rdFingerprintGenerator.GetMorganGenerator(radius=2, fpSize=2048)
fps = {n: gen.GetFingerprint(m) for n, m in mols.items()}
sim = pd.Series({n: DataStructs.TanimotoSimilarity(fps[ANCHOR], fps[n])
for n in names if n != ANCHOR}).sort_values(ascending=False)
print(f"similarity of the other 53 to {ANCHOR}: median {sim.median():.2f}, "
f"best {sim.max():.2f} ({sim.index[0]}), worst {sim.min():.2f} ({sim.index[-1]})")
print(f"within 0.6 of the anchor: {(sim >= 0.6).sum()} compounds")
print()
print("So every compound is reachable from the one known pose, but not in one hop.")
print("That is exactly what a perturbation network is for. Section 6 builds one.")
Output:
similarity of the other 53 to A_027: median 0.56, best 0.86 (A_053), worst 0.46 (A_023) within 0.6 of the anchor: 18 compounds So every compound is reachable from the one known pose, but not in one hop. That is exactly what a perturbation network is for. Section 6 builds one.
6. Two traps
Everything up to here is reading. These two are the ones that produce a plausible-looking answer that is wrong, which is worse than an error.
Trap one: three racemates
A trailing |&1:20| in CXSMILES is an AND1 enhanced-stereo group: the centre is drawn as a specific hand, but the physical sample is a mixture of both. Three compounds here carry that flag. If you model one enantiomer and the measurement was made on the mixture, you are predicting a different quantity than the one being scored.
The trap is specifically that nothing warns you.
raw = {r.Molecule_Name: r[SMI] for _, r in test.iterrows()}
flagged = sorted(n for n, s in raw.items() if "|" in s and "&" in s)
print("compounds whose raw CXSMILES carries an AND enhanced-stereo flag:", flagged)
print()
print(f"{'compound':10s} {'FindMolChiralCenters says':28s} {'GetStereoGroups() says':24s}")
for n in flagged:
m = mols[n]
centres = Chem.FindMolChiralCenters(m, includeUnassigned=True, useLegacyImplementation=False)
groups = [(str(g.GetGroupType()).split(".")[-1], [a.GetIdx() for a in g.GetAtoms()])
for g in m.GetStereoGroups()]
print(f"{n:10s} {str(centres):28s} {str(groups):24s}")
print()
one = flagged[0]
m = mols[one]
print(f"round-tripping {one}:")
print(f" MolToCXSmiles keeps the flag : ...{Chem.MolToCXSmiles(m)[-12:]!r}")
print(f" plain MolToSmiles drops it : {'|' not in Chem.MolToSmiles(m)}")
single = Chem.MolFromSmiles(raw[one].split()[0])
print(f" and the string it returns is byte-identical to the single enantiomer's: "
f"{Chem.MolToSmiles(m) == Chem.MolToSmiles(single)}")
Output:
compounds whose raw CXSMILES carries an AND enhanced-stereo flag: ['A_022', 'A_041', 'A_053']
compound FindMolChiralCenters says GetStereoGroups() says
A_022 [(16, 'S')] [('STEREO_AND', [16])]
A_041 [(17, 'S')] [('STEREO_AND', [17])]
A_053 [(20, 'S')] [('STEREO_AND', [20])]
round-tripping A_022:
MolToCXSmiles keeps the flag : ...'cc1C |&1:16|'
plain MolToSmiles drops it : True
and the string it returns is byte-identical to the single enantiomer's: True
Read the last line again. RDKit does not throw the information away: it is there in GetStereoGroups() as a STEREO_AND group, and MolToCXSmiles round-trips it faithfully. But FindMolChiralCenters reports a confident assignment with no hint that it is a mixture, and the moment anything in your pipeline passes through plain MolToSmiles, the flag is gone and the string is indistinguishable from the pure enantiomer.
Almost every file format in a modelling workflow goes through plain SMILES somewhere. Decide what you are going to do about these three before you build anything, not after.
Trap two: two charge states
# A sulfonamide N is not a base (pKa near 0), so it is excluded here along with
# amides. Without that exclusion A_025 and A_043 count as di-basic and land in
# the +2 set, which is wrong.
BASIC = Chem.MolFromSmarts(
"[NX3;H2,H1,H0;!$(N[#6]=[O,N,S]);!$(N[#16,#15]=O);!$(N[a]);!$(N=*);!$([N+])]")
ACID = Chem.MolFromSmarts("[CX3](=O)[OX2H1]")
# Beta-fluorination pulls an amine's pKa down by roughly 1.7 units per fluorine,
# so these nitrogens sit near or below the assay pH and are largely neutral.
# Flagged, not subtracted: the real number needs a pKa model.
WEAKENED = Chem.MolFromSmarts(
"[NX3;!$(N[#6]=[O,N,S]);!$(N[#16,#15]=O);!$(N[a]);!$(N=*)]~[CX4]~[CX4]F")
n_basic = {n: len(m.GetSubstructMatches(BASIC)) for n, m in mols.items()}
n_acid = {n: len(m.GetSubstructMatches(ACID)) for n, m in mols.items()}
print("formal charge as drawn in the file:",
pd.Series({n: Chem.GetFormalCharge(m) for n, m in mols.items()}).value_counts().to_dict())
print("carboxylic acids :", pd.Series(n_acid).value_counts().to_dict())
print("basic amines :", pd.Series(n_basic).value_counts().to_dict())
print()
charge = {n: 1 + (n_basic[n] > 1) for n in names} # +1, or +2 when di-basic
cls = pd.Series(charge)
print(f"basic-amine count (a pH-blind rule, not a pKa model): {(cls == 1).sum()} compounds "
f"at +1, {(cls == 2).sum()} at +2")
print("the +2 set:", ", ".join(sorted(cls[cls == 2].index)))
print()
print(f"Every compound is drawn neutral, so a pipeline that trusts the file sees one charge")
print(f"state. Counted this way they are all cations, and {(cls == 2).sum()} of them carry two.")
print(f"The anchor {ANCHOR} is in the {charge[ANCHOR]}-charge minority, which is worth knowing.")
print()
weak = sorted(n for n in cls[cls == 2].index if mols[n].HasSubstructMatch(WEAKENED))
print(f"of those {(cls == 2).sum()}, {len(weak)} carry a beta-fluorinated amine, whose pKa is")
print(f"pulled down by roughly 1.7 units per fluorine: {', '.join(weak)}.")
print("Near or below the assay pH those nitrogens are largely neutral, so treat them as +1")
print("unless a pKa model says otherwise. That is the whole point: this is a count, not a pKa.")
print()
print("Nothing above uses a pH. The DENV-2 assay protocol the challenge links (protocols.io")
print("kqdg32m87v25) uses 20 mM Tris-HCl pH 8.5, 10% glycerol, 0.01% Triton, no added salt.")
print("At 8.5 an amine of pKa near 9 is only about three-quarters protonated and a weaker one")
print("is mostly neutral, so +1 or +2 is a question per compound; settle it with a pKa tool.")
Output:
formal charge as drawn in the file: {0: 54}
carboxylic acids : {0: 54}
basic amines : {1: 43, 2: 11}
basic-amine count (a pH-blind rule, not a pKa model): 43 compounds at +1, 11 at +2
the +2 set: A_005, A_011, A_018, A_027, A_029, A_033, A_034, A_035, A_048, A_052, A_053
Every compound is drawn neutral, so a pipeline that trusts the file sees one charge
state. Counted this way they are all cations, and 11 of them carry two.
The anchor A_027 is in the 2-charge minority, which is worth knowing.
of those 11, 6 carry a beta-fluorinated amine, whose pKa is
pulled down by roughly 1.7 units per fluorine: A_005, A_018, A_029, A_033, A_034, A_052.
Near or below the assay pH those nitrogens are largely neutral, so treat them as +1
unless a pKa model says otherwise. That is the whole point: this is a count, not a pKa.
...
# The cheapest connected perturbation network: a minimum spanning tree over 1 - Tanimoto.
# Then count how many of its edges cross the charge boundary, because those are the expensive
# ones: a charge-changing perturbation needs extra machinery or it is simply wrong.
import itertools, heapq
N = len(names)
idx = {n: i for i, n in enumerate(names)}
dist = np.zeros((N, N))
for a, b in itertools.combinations(names, 2):
d = 1 - DataStructs.TanimotoSimilarity(fps[a], fps[b])
dist[idx[a], idx[b]] = dist[idx[b], idx[a]] = d
seen = [False] * N
seen[0] = True
pq = [(dist[0, j], 0, j) for j in range(1, N)]
heapq.heapify(pq)
edges = []
while len(edges) < N - 1:
d, i, j = heapq.heappop(pq)
if seen[j]:
continue
seen[j] = True
edges.append((d, names[i], names[j]))
for k in range(N):
if not seen[k]:
heapq.heappush(pq, (dist[j, k], j, k))
ed = pd.Series([e[0] for e in edges])
dh = pd.Series([abs(heavy[e[1]] - heavy[e[2]]) for e in edges])
print(f"cheapest connected network over the 54: {len(edges)} perturbations")
print(f" edge distance (1 - Tanimoto): median {ed.median():.3f}, worst {ed.max():.3f}")
print(f" heavy atoms changed per edge: median {dh.median():.0f}, max {dh.max():.0f}")
crossing = [e for e in edges if charge[e[1]] != charge[e[2]]]
print(f"\nedges crossing the +1/+2 boundary: {len(crossing)} of {len(edges)}")
for d, a, b in crossing:
print(f" {a} (+{charge[a]}) -- {b} (+{charge[b]}) distance {d:.3f}")
Output:
cheapest connected network over the 54: 53 perturbations edge distance (1 - Tanimoto): median 0.182, worst 0.316 heavy atoms changed per edge: median 1, max 5 edges crossing the +1/+2 boundary: 2 of 53 A_022 (+1) -- A_053 (+2) distance 0.234 A_011 (+2) -- A_013 (+1) distance 0.164
# Draw it. Classical MDS on the same distance matrix gives a deterministic layout, so this
# figure does not wander between runs the way a force-directed one would.
sq = dist ** 2
J = np.eye(N) - np.ones((N, N)) / N
w, V = np.linalg.eigh(-0.5 * J @ sq @ J)
order = np.argsort(w)[::-1][:2]
xy = V[:, order] * np.sqrt(np.maximum(w[order], 0))
fig, ax = plt.subplots(figsize=(6.2, 4.6))
for d, a, b in edges:
cross = charge[a] != charge[b]
ax.plot(*zip(xy[idx[a]], xy[idx[b]]), color=GOLD if cross else "#b9c6cc",
linewidth=2.0 if cross else 0.8, zorder=2 if cross else 1,
alpha=1.0 if cross else 0.9)
for n in names:
ax.scatter(*xy[idx[n]], s=58, c=GOLD if charge[n] == 2 else TEAL,
edgecolors="white", linewidths=0.8, zorder=3)
ax.scatter(*xy[idx[ANCHOR]], s=185, facecolors="none", edgecolors=INK, linewidths=1.8, zorder=4)
ax.annotate(f"{ANCHOR}\n(has a crystal structure)", xy[idx[ANCHOR]], xytext=(12, 10),
textcoords="offset points", fontsize=8, color=INK)
ax.set_xticks([]); ax.set_yticks([]); ax.grid(False)
for s in ax.spines.values():
s.set_visible(False)
ax.set_title(f"{len(edges)} perturbations covering all 54 compounds\n"
f"gold = the {(cls == 2).sum()} doubly-charged compounds and the "
f"{len(crossing)} edges that cross into them", fontsize=10, loc="left")
fig.tight_layout()
plt.show()
Three edges in gold. That is the whole decision: either you give those three the extra machinery a charge-changing perturbation needs, or you cut the network into two charge classes and anchor each one separately. Both are defensible. Running them as ordinary perturbations and not noticing is not, and it is the easiest mistake on this dataset to make.
The perturbations themselves are small, a median of one heavy atom per edge, so this is a well-behaved network apart from that seam.
7. The error budget
Two mounted datasets give you the honest answer, and it is the reason the challenge is scored the way it is.
The Roche PDE10A set is 1,162 inhibitors from a single real lead-optimisation programme, with a measured potency for every one. It is the same kind of object as the Alchemistry set, with the answers still attached. The EV-A71 set is 925 crystallographic complexes, so it tells you whether a 15-residue pocket is unusual.
pde = MOUNTS["pde10a"]
ev = MOUNTS["eva71"]
if pde is None or ev is None:
print("PDE10A or EV-A71 not mounted; skipping.")
else:
roche = pd.read_csv(pde / "pde10a_inhibitors_activity_splits.csv")
rdG = -1.364 * roche.pic50
print(f"Roche PDE10A: {len(roche)} inhibitors, one programme, one assay")
print(f" full range : {rdG.max() - rdG.min():.1f} kcal/mol")
print(f" 10th to 90th pct : {rdG.quantile(0.90) - rdG.quantile(0.10):.1f} kcal/mol")
print(f" interquartile range : {rdG.quantile(0.75) - rdG.quantile(0.25):.1f} kcal/mol")
print(f" standard deviation : {rdG.std():.1f} kcal/mol")
print()
base = ev / "ev-a71-2a" / "v2-20260828" / "parquet"
events = pd.read_parquet(base / "events.parquet")
contacts = pd.read_parquet(base / "contacts.parquet")
pocket_sizes = contacts[contacts.min_dist_a <= 4.5].groupby("complex_name").size()
pct = (pocket_sizes <= len(near)).mean()
print(f"EV-A71 2A protease: {len(events)} complexes, "
f"{int(events.has_affinity.sum())} with a measured affinity")
print(f" residues within 4.5 A of the ligand: median {pocket_sizes.median():.0f}, "
f"10th-90th {pocket_sizes.quantile(.1):.0f}-{pocket_sizes.quantile(.9):.0f}")
print(f" our dengue-2 reference complex : {len(near)} -> the {pct:.0%} percentile")
print(f" so a pocket this size is ordinary, not a sign the structure is wrong.")
Output:
Roche PDE10A: 1162 inhibitors, one programme, one assay full range : 7.3 kcal/mol 10th to 90th pct : 4.3 kcal/mol interquartile range : 2.4 kcal/mol standard deviation : 1.6 kcal/mol EV-A71 2A protease: 925 complexes, 649 with a measured affinity residues within 4.5 A of the ligand: median 17, 10th-90th 11-19 our dengue-2 reference complex : 15 -> the 18% percentile so a pocket this size is ordinary, not a sign the structure is wrong.
if pde is not None and ev is not None:
fig, (a1, a2) = plt.subplots(1, 2, figsize=(7.6, 2.9))
a1.hist(rdG, bins=28, color=TEAL, alpha=0.9)
q1, q3 = rdG.quantile(0.25), rdG.quantile(0.75)
a1.axvspan(q3, q1, color=GOLD, alpha=0.3, zorder=0)
a1.set_title(f"1,162 real inhibitors, one programme\nhalf of them inside "
f"{abs(q3 - q1):.1f} kcal/mol", fontsize=9, loc="left")
a1.set_xlabel("measured $\\Delta G$ (kcal/mol)")
a1.set_ylabel("compounds")
a2.hist(pocket_sizes, bins=range(int(pocket_sizes.min()), int(pocket_sizes.max()) + 2),
color=TEAL, alpha=0.9)
a2.axvline(len(near), color=GOLD, linewidth=2.2)
a2.annotate(f"ours: {len(near)}", (len(near), a2.get_ylim()[1] * 0.78),
xytext=(7, 0), textcoords="offset points", fontsize=8, color=GOLD,
bbox=dict(boxstyle="round,pad=0.25", fc="white", ec="none", alpha=0.9))
a2.set_title(f"925 crystallographic pockets\nours is the {pct:.0%} percentile",
fontsize=9, loc="left")
a2.set_xlabel("residues within 4.5 A of the ligand")
fig.tight_layout()
plt.show()
The gold band on the left is the error budget, and it is the single most useful number on this page. Half of the compounds in a mature, real-world congeneric series sit inside a window of about one and a half kcal/mol. A good alchemical free energy calculation has an uncertainty of roughly the same size.
So the signal you are trying to resolve and the noise of the method that resolves it are comparable. That is not a reason to give up; it is the reason the leaderboard ranks on Kendall's tau, a rank correlation, instead of on absolute error. Getting the ordering right is achievable. Getting every absolute number right is not, for anyone.
8. Transfer from Zika
The companion Zika archive holds two experimental structures of the *same enzyme* in a related flavivirus, with their headers and waters intact. The organizers of this challenge point at the Zika challenge by name and call the two targets close relatives. Worth checking how close.
AA3 = {"ALA": "A", "ARG": "R", "ASN": "N", "ASP": "D", "CYS": "C", "GLN": "Q", "GLU": "E",
"GLY": "G", "HIS": "H", "ILE": "I", "LEU": "L", "LYS": "K", "MET": "M", "PHE": "F",
"PRO": "P", "SER": "S", "THR": "T", "TRP": "W", "TYR": "Y", "VAL": "V"}
def ca_seq(path_or_lines):
src = (path_or_lines if isinstance(path_or_lines, list)
else Path(path_or_lines).read_text().splitlines())
out = {}
for l in src:
if l.startswith("ATOM") and l[12:16].strip() == "CA" and l[17:20].strip() in AA3:
out.setdefault(l[21], {})[int(l[22:26])] = AA3[l[17:20].strip()]
return out
denv_seq = ca_seq(raw_pdb)
zk = MOUNTS["zika"]
if zk is None:
print("Zika companion not mounted; skipping.")
else:
zika = {p.stem.upper(): ca_seq(p) for p in sorted((zk / "reference-structures").glob("*.pdb"))}
for lbl in zika:
n_wat = sum(1 for l in (zk / "reference-structures" / f"{lbl.lower()}.pdb")
.read_text().splitlines() if l[17:20].strip() == "HOH")
print(f" {lbl}: {n_wat} water atoms kept")
print(f" our dengue-2 reference: 0 water atoms. Only visible side by side.")
print()
rows = []
for ch_id, num in sorted([(c, n) for c, n, _ in near], key=lambda x: (x[0], x[1])):
d_aa = denv_seq.get(ch_id, {}).get(num, "-")
z_aa = [zika[k].get(ch_id, {}).get(num, "-") for k in zika]
rows.append({"chain": ch_id, "res": num, "DENV2": d_aa,
**dict(zip(zika, z_aa)),
"same": "yes" if all(a == d_aa for a in z_aa) and d_aa != "-" else "NO"})
tbl = pd.DataFrame(rows)
print("pocket residues, dengue-2 against Zika:")
print(tbl.to_string(index=False))
print()
for ch_id, label in (("B", "NS3 protease"), ("A", "NS2B cofactor")):
sub = tbl[tbl.chain == ch_id]
if len(sub):
print(f" {label}: {(sub['same'] == 'yes').sum()} of {len(sub)} pocket residues identical")
for k in zika:
for ch_id in ("A", "B"):
shared = set(denv_seq.get(ch_id, {})) & set(zika[k].get(ch_id, {}))
if shared:
ident = sum(1 for r in shared if denv_seq[ch_id][r] == zika[k][ch_id][r])
print(f" whole chain {ch_id} vs {k}: {ident}/{len(shared)} = "
f"{100 * ident / len(shared):.0f}% identical")
Output:
29HN: 119 water atoms kept
8PN6: 106 water atoms kept
our dengue-2 reference: 0 water atoms. Only visible side by side.
pocket residues, dengue-2 against Zika:
chain res DENV2 29HN 8PN6 same
A 81 D S S NO
A 82 G G G yes
A 83 S D D NO
B 51 H H H yes
B 75 D D D yes
B 129 D D D yes
B 130 F Y Y NO
B 131 S P P NO
B 132 P A A NO
...
This resolves more cleanly than the whole-chain numbers suggest. Overall identity is only about 56% on the protease chain and 42% on the cofactor, which sounds like a different protein. But 9 of the 12 protease pocket residues are identical, and the three that differ are substitutions whose contacts to the ligand come through the backbone, which every amino acid shares.
So the geometry transfers and the affinities do not. Use the Zika structures for pocket shape, water placement and sanity-checking a pose. Do not use Zika potency as a training signal for dengue-2.
9. Novelty against PubChem (the slow cell)
Section 2 proved no challenge compound has a published *activity*. The stronger question is whether the structures exist in the public record at all, which means checking PubChem: 124.9 million compounds, the registry rather than a bioactivity set.
This is the slow cell in the notebook, about two minutes. The file is sorted by compound id, not by InChIKey, and every row group spans the whole alphabet, so there is no way to skip ahead: the whole key column has to stream, 1.26 GB of a 24 GB file. Skip it if you are in a hurry, it changes nothing downstream.
import pyarrow.parquet as pq
pc_mount = MOUNTS["pubchem"]
if pc_mount is None:
print("PubChem not mounted; skipping.")
else:
f = pq.ParquetFile(pc_mount / "parquet" / "inchi_keys.parquet")
want = set(keys.values())
control = Chem.MolToInchiKey(Chem.MolFromSmiles("CC(=O)Oc1ccccc1C(=O)O"))
want.add(control)
found, scanned = {}, 0
for g in range(f.metadata.num_row_groups):
col = f.read_row_group(g, columns=["inchikey"]).column(0).to_pylist()
scanned += len(col)
for k in want.intersection(col):
found.setdefault(k, g)
if g % 250 == 0:
print(f" {scanned:,} of {f.metadata.num_rows:,} keys read...", flush=True)
by_key = {v: k for k, v in keys.items()}
print(f"\nscanned {scanned:,} PubChem InChIKeys")
print(f"control, aspirin: {'FOUND' if control in found else 'MISSING'}")
hits = sorted(by_key[k] for k in found if k in by_key)
print(f"challenge compounds present in PubChem: {len(hits)} of {len(keys)} {hits}")
print()
print("With the control found, zero hits means these 54 structures have never been")
print("deposited in the largest public chemical registry. Not published, not patented")
print("under a deposited structure, not in a vendor catalogue. Genuinely new matter.")
Output:
124,128 of 124,863,647 keys read... 31,073,090 of 124,863,647 keys read... 62,034,828 of 124,863,647 keys read... 93,005,911 of 124,863,647 keys read... 123,828,198 of 124,863,647 keys read... scanned 124,863,647 PubChem InChIKeys control, aspirin: FOUND challenge compounds present in PubChem: 0 of 54 [] With the control found, zero hits means these 54 structures have never been deposited in the largest public chemical registry. Not published, not patented under a deposited structure, not in a vendor catalogue. Genuinely new matter.
10. Leaderboard noise
One detail from the challenge's own configuration that is not in the dataset card: the live board scores 27 of the 54 compounds, not all of them. The rest are held back for the final evaluation. It is also worth knowing that absolute-error metrics are reported as not applicable for relative and edge-based methods, so for most entrants Kendall's tau is the only number that moves.
Scoring half the set makes the public tau noisy, and it helps to keep two questions apart. The 54 measured values are fixed, and so are your predictions once submitted; the only thing the live board adds is the choice of which 27 get scored. So: how far can one entry's live tau sit from its tau over all 54, and how often does that put two entries in the wrong order?
rng = np.random.default_rng(20270325)
N, K, TRIALS = 54, 27, 2000
def kendall(t, p):
i, j = np.triu_indices(len(t), 1)
return float(np.mean(np.sign(t[i] - t[j]) * np.sign(p[i] - p[j])))
def entry(truth, mix):
# a method whose predictions track the truth to a chosen degree
return mix * truth + np.sqrt(max(1 - mix ** 2, 0)) * rng.normal(size=truth.size)
def worse(pred, extra):
# a second entry built from the first plus its own extra error: two real entries on
# this series share most of their mistakes, which is what keeps their gap steady
return pred + extra * rng.normal(size=pred.size)
halves = [rng.choice(N, K, replace=False) for _ in range(TRIALS)]
truth = rng.normal(size=N) # ONE fixed set of 54 measurements, as in the challenge
print(f"one entry, predictions fixed, scored on {TRIALS:,} different random 27s:")
print(f" {'tau over all 54':>16s} {'live board, mean':>17s} {'sd':>6s}")
for mix in (0.95, 0.85, 0.7, 0.5):
pred = entry(truth, mix)
live = np.array([kendall(truth[h], pred[h]) for h in halves])
print(f" {kendall(truth, pred):>16.2f} {live.mean():>17.2f} {live.std():>6.3f}")
print()
print("two entries scored on the SAME 27:")
print(f" {'gap over all 54':>16s} {'sd of live gap':>15s} {'live board has them backwards':>31s}")
a = entry(truth, 0.85)
for extra in (0.15, 0.3, 0.5, 0.8):
b = worse(a, extra)
full_gap = kendall(truth, a) - kendall(truth, b)
gap = np.array([kendall(truth[h], a[h]) - kendall(truth[h], b[h]) for h in halves])
print(f" {abs(full_gap):>16.2f} {gap.std():>15.3f} "
f"{(np.sign(gap) == -np.sign(full_gap)).mean():>30.0%}")
print()
print("The live tau is an unbiased estimate of the tau over all 54; it is noisy, not skewed.")
print("To compare two entries, read their gap against the sd of the GAP, not against either")
print("entry's own sd. A gap well under that sd is not a ranking; one several times it is.")
Output:
one entry, predictions fixed, scored on 2,000 different random 27s:
tau over all 54 live board, mean sd
0.83 0.83 0.033
0.68 0.67 0.047
0.59 0.60 0.073
0.26 0.26 0.104
two entries scored on the SAME 27:
gap over all 54 sd of live gap live board has them backwards
0.02 0.023 13%
0.04 0.046 19%
0.01 0.066 41%
0.13 0.076 4%
The live tau is an unbiased estimate of the tau over all 54; it is noisy, not skewed.
...
11. The submission file
The format is strict and two of its rules cost people a resubmission, which matters when you only get one attempt per 48 hours.
The grader matches column names exactly, and it wants four of them, not two. Both identifier columns from the blinded file have to come back untouched:
| column | |
| Molecule_Name | all 54 ids, as spelled in the blinded file |
| CXSMILES (CDD Compatible) | the structure column, returned unchanged |
| denv2_ns2b_ns3_potency_predicted | your number, kcal/mol, more negative is tighter |
| denv2_ns2b_ns3_potency_predicted_uncertainty | required, not optional, same units |
.parquet is preferred and .csv is accepted. Every value must be a finite float: a single NaN or inf rejects the whole file. Relative and edge-based methods (FEP, TI, NEQ) must also attach a perturbation file with Molecule_Name_A, Molecule_Name_B, denv2_ns2b_ns3_potency_predicted_ddg and denv2_ns2b_ns3_potency_predicted_ddg_uncertainty, and are rejected without it. Absolute methods are not asked for one.
One trap worth stating: do not build the file by merging your predictions onto the blinded frame with overlapping column names. A _conf_low on one side and a _conf_high on the other silently become _x and _y suffixed columns, and the grader then cannot find what it needs. The cell below starts from the blinded file instead, so the ids and their order cannot drift, and it checks its own output against the real column list.
# Start from the blinded file so BOTH identifier columns go back exactly as they came.
sub = test.copy()
# A DELIBERATELY TRIVIAL placeholder, not a method: a ligand-efficiency rule of thumb at
# -0.3 kcal/mol per heavy atom. It exists so the file is real and the validator has
# something to check. Replace it with your own result before submitting anything.
sub["denv2_ns2b_ns3_potency_predicted"] = [-0.3 * heavy[n] for n in sub.Molecule_Name]
sub["denv2_ns2b_ns3_potency_predicted_uncertainty"] = 1.5
REQUIRED = ["Molecule_Name", "CXSMILES (CDD Compatible)",
"denv2_ns2b_ns3_potency_predicted",
"denv2_ns2b_ns3_potency_predicted_uncertainty"]
def validate(df, blinded):
errs = []
if list(df.columns) != REQUIRED:
errs.append(f"columns must be exactly {REQUIRED}, got {list(df.columns)}")
if len(df) != len(blinded):
errs.append(f"must have exactly {len(blinded)} rows, got {len(df)}")
if df.Molecule_Name.duplicated().any():
errs.append("duplicate Molecule_Name")
if set(df.Molecule_Name) != set(blinded.Molecule_Name):
errs.append("Molecule_Name set does not match the blinded test set")
for c in REQUIRED[2:]:
if c in df and not np.isfinite(df[c]).all():
errs.append(f"{c} contains NaN or inf, which rejects the whole file")
bad = [c for c in df.columns if c.endswith(("_conf_low", "_conf_high", "_x", "_y"))]
if bad:
errs.append(f"columns {bad} look like a merge collision; rename or drop them")
return errs
problems = validate(sub[REQUIRED], test)
print("validation:", "PASS" if not problems else "FAIL")
for p in problems:
print(" ", p)
OUT = Path.home() / "alchemistry_2027_submission.parquet"
sub[REQUIRED].to_parquet(OUT, index=False) # parquet preferred, csv accepted
print(f"\nwrote {OUT} ({OUT.stat().st_size:,} bytes)")
print(sub[REQUIRED].head(3).to_string(index=False))
print()
print("Submit on the Space's Submit tab. One submission per 48 hours per track;")
print("only your latest valid one counts. The live board scores 27 of the 54;")
print("the interim and final boards score all 54.")
Output:
validation: PASS
wrote /home/jovyan/alchemistry_2027_submission.parquet (5,013 bytes)
Molecule_Name CXSMILES (CDD Compatible) denv2_ns2b_ns3_potency_predicted denv2_ns2b_ns3_potency_predicted_uncertainty
A_001 CN1CC[C@@H](NC2=NC3=CC(C(=O)NC4=CC=C(C(C)(C)C)C(F)=C4)=CC(Cl)=C3N2)C1 -9.3 1.5
A_002 CCN1CC(NC2=NC3=CC(C(=O)NC4=CC=C(F)C=C4)=CC(Cl)=C3N2)C1 -8.1 1.5
A_003 CC(C)NC1=NC2=CC(C(=O)NC3=CC=C4CNCC4=C3)=CC=C2N1 -7.5 1.5
Submit on the Space's Submit tab. One submission per 48 hours per track;
only your latest valid one counts. The live board scores 27 of the 54;
the interim and final boards score all 54.
12. What a real alchemical campaign actually cost, and where its error came from
Everything above this point can be done before spending anything. This section is the other side: the numbers from one real relative-binding campaign on this exact target, run in two waves and submitted to the Regression track three times between 2026-11-04 and 2026-11-11. They are included because none of them are in the literature for this system, several of them contradicted the plan they replaced, and one was a post-processing bug that cost 0.08 Kendall tau on the live board before anyone noticed it.
The protocol. OpenFE 1.13.0 RelativeHybridTopologyProtocol, 11 lambda windows, Hamiltonian replica exchange, 1 ns equilibration and 1 ns production per window, one repeat, both legs per edge, on one NVIDIA A10G per leg. 90 edges over 64 nodes in the final network, built as 54 edges in a first wave and 36 more in a second.
What it cost. 1.51 GPU-hours per edge, both legs, which was $2.12 at on-demand rates. Four independent estimates of the hybrid-topology slowdown over plain MD had been made before running anything, spanning 2.73x to 8.64x. The measured factor was 1.31, so every one of them was wrong in the same direction and the campaign plan was roughly 2.7x too expensive. If you are sizing one of these, measure one edge before trusting any multiplier, including this one. The whole campaign came to 140.9 billed GPU-hours over 190 jobs, 134.0 of them production across both waves and 6.9 spent on the control experiment described below.
Where the error came from. The per-edge MBAR error had a median of 1.31 kcal/mol, which is about the size of the signal itself on a congeneric series, so the first instinct is to buy more sampling. That would have been the wrong purchase. The error is not uniform, it is bimodal, and it tracks the nearest-neighbour lambda overlap almost monotonically.
# 90 edges from the submitted network: (perturbation difficulty, MBAR error, min lambda overlap)
# Difficulty is a composite of heavy-atom change, dummy atom count and ring edits, computed
# before any simulation ran. Overlap is the smallest adjacent-window MBAR overlap on either leg.
MEASURED = [(0.0, 1.05, 0.1463), (1.3, 0.14, 0.0975), (1.3, 0.47, 0.0905),
(1.3, 0.63, 0.0912), (1.3, 0.65, 0.1018), (1.3, 0.65, 0.1099), (1.3, 0.78, 0.1023),
(1.3, 0.88, 0.0906), (1.3, 1.3, 0.1358), (1.3, 1.87, 0.0687), (1.7, 0.33, 0.1362),
(2.1, 0.83, 0.2274), (2.1, 0.88, 0.2266), (2.5, 0.18, 0.1247), (2.5, 0.46, 0.1138),
(2.5, 0.65, 0.0907), (2.5, 0.69, 0.0955), (2.5, 0.7, 0.0987), (2.5, 0.73, 0.0824),
(2.5, 0.77, 0.072), (3.3, 1.18, 0.113), (3.3, 1.45, 0.0836), (3.3, 1.63, 0.085),
(3.3, 1.64, 0.082), (3.3, 1.65, 0.081), (3.3, 1.69, 0.0816), (3.3, 1.97, 0.049),
(3.8, 1.19, 0.0464), (4.5, 0.55, 0.1097), (4.7, 0.95, 0.1689), (4.7, 1.19, 0.1114),
(5.0, 0.96, 0.0429), (5.4, 0.93, 0.1108), (5.4, 2.0, 0.0655), (5.4, 2.52, 0.027),
(5.5, 1.0, 0.2001), (5.8, 1.19, 0.0718), (5.8, 1.29, 0.0978), (5.8, 1.41, 0.0661),
(5.9, 0.81, 0.0876), (5.9, 0.86, 0.0703), (5.9, 1.5, 0.0413), (5.9, 1.67, 0.0294),
(6.3, 1.27, 0.0762), (6.7, 1.5, 0.0201), (7.0, 1.59, 0.0521), (7.1, 1.13, 0.0399),
(7.3, 0.89, 0.0982), (7.3, 1.26, 0.0759), (7.3, 1.28, 0.0938), (7.3, 1.4, 0.0675),
(8.0, 1.05, 0.1274), (8.0, 1.13, 0.077), (8.0, 1.87, 0.038), (8.2, 1.06, 0.1236),
(8.2, 1.38, 0.1573), (8.5, 1.25, 0.0511), (8.5, 2.19, 0.0517), (8.9, 0.83, 0.0936),
(9.0, 2.08, 0.0334), (9.1, 1.21, 0.0588), (9.1, 3.33, 0.0213), (9.3, 0.82, 0.0901),
(9.3, 1.4, 0.0588), (9.3, 2.08, 0.0314), (10.0, 1.54, 0.0686), (10.5, 1.64, 0.0332),
(11.2, 2.24, 0.0455), (11.7, 1.55, 0.0052), (12.0, 0.72, 0.1253), (12.0, 0.81, 0.1017),
(12.4, 2.35, 0.0253), (14.3, 1.51, 0.0436), (14.3, 1.61, 0.1056), (14.3, 1.7, 0.0284),
(14.3, 1.95, 0.0234), (14.9, 1.6, 0.0401), (14.9, 1.94, 0.0491), (14.9, 2.07, 0.0326),
(14.9, 2.24, 0.046), (15.1, 1.32, 0.0836), (15.1, 1.81, 0.0386), (15.9, 2.05, 0.0385),
(15.9, 2.12, 0.0394), (15.9, 2.17, 0.0255), (16.1, 3.0, 0.0185), (16.6, 2.17, 0.0068),
(16.7, 1.66, 0.0426), (17.1, 2.3, 0.057), (20.1, 2.89, 0.0107)]
OVERLAP_FLOOR = 0.05 # below this, published benchmarks degrade sharply
def kendall(x, y):
c = d = 0
for i in range(len(x)):
for j in range(i + 1, len(x)):
a, b = x[i] - x[j], y[i] - y[j]
if a == 0 or b == 0:
continue
if (a > 0) == (b > 0):
c += 1
else:
d += 1
return (c - d) / (c + d)
diff = [m[0] for m in MEASURED]
err = [m[1] for m in MEASURED]
ov = [m[2] for m in MEASURED]
print(f"n = {len(MEASURED)} edges, MBAR error median {np.median(err):.2f} kcal/mol")
print()
print(f" tau(lambda overlap, error) {kendall(ov, err):+.3f} worse overlap, bigger error")
print(f" tau(difficulty, error) {kendall(diff, err):+.3f} predictable BEFORE running")
print(f" tau(difficulty, overlap) {kendall(diff, ov):+.3f}")
print()
ok = [e for e, o in zip(err, ov) if o >= OVERLAP_FLOOR]
bad = [e for e, o in zip(err, ov) if o < OVERLAP_FLOOR]
print(f" overlap >= {OVERLAP_FLOOR}: n={len(ok):2d} median error {np.median(ok):.2f} kcal/mol")
print(f" overlap < {OVERLAP_FLOOR}: n={len(bad):2d} median error {np.median(bad):.2f} kcal/mol")
print(f" ratio {np.median(bad) / np.median(ok):.2f}x")
print()
print("So 31 of 90 edges carry nearly all of the error, and a descriptor computed before")
print("any simulation ran predicts which ones at tau +0.48. Spending uniformly on more")
print("sampling pays mostly for edges that were already converged.")
print()
print("That +0.48 was +0.63 over the first 54 edges alone. The second wave was selected for")
print("estimator variance rather than for difficulty, which deliberately buys harder edges,")
print("so the descriptor predicts less well on the set it helped choose.")
Output:
n = 90 edges, MBAR error median 1.31 kcal/mol tau(lambda overlap, error) -0.543 worse overlap, bigger error tau(difficulty, error) +0.477 predictable BEFORE running tau(difficulty, overlap) -0.445 overlap >= 0.05: n=59 median error 1.05 kcal/mol overlap < 0.05: n=31 median error 1.95 kcal/mol ratio 1.86x So 31 of 90 edges carry nearly all of the error, and a descriptor computed before any simulation ran predicts which ones at tau +0.48. Spending uniformly on more sampling pays mostly for edges that were already converged. That +0.48 was +0.63 over the first 54 edges alone. The second wave was selected for ...
Two consequences worth taking from this, both of which cost nothing to apply.
Buy production length, not repeats, when equilibration is charged per repeat. Three repeats pay the 1 ns equilibration three times and amortise it over nothing. At matched precision, 90 edges at 1 ns with 3 repeats and 90 edges at 3 ns with 1 repeat land within 0.003 Kendall tau of each other, and the second costs 33% less. Repeats do buy the run-to-run spread that MBAR cannot report, but a cycle-rich network buys that too, and buys accuracy at the same time.
Build the network for estimator variance, not for edge quality. The reflex is a minimum spanning tree over a LOMAP or difficulty score, which optimises per-edge convergence. The score depends on something else: the variance of each compound's network estimate, which is the centered diagonal of the pseudo-inverse of the weighted graph Laplacian. On a tree that equals the hop count, so a cheap-looking tree can put 8 hops between two scored compounds and let error accumulate instead of averaging. Re-selecting the same number of edges for variance instead of difficulty moved the predicted tau on this network from 0.39 to 0.47 for no extra compute.
The greedy selection is exact and cheap, because adding an edge updates the covariance by a rank-one formula rather than needing a fresh inverse:
v = C(e_i - e_j), R_ij = v_i - v_j, gain = w * sum_scored(v_k^2) / (1 + w * R_ij)
And one caution against over-applying the first result. Picking purely for variance selects long, chemically hard edges, which is exactly the population that fails the overlap floor. On this network an uncapped selection reached a nominal tau of 0.567 with only 44% of its new edges expected to pass the floor, against 0.513 at 73% when difficulty was capped. Since poor overlap can bias an edge rather than merely add noise, and a least-squares fit absorbs bias as if it were noise, the capped network is the more honest buy even though it reads worse on paper.
What the submitted entry looked like. 90 edges over 64 nodes, 3 disconnected components holding 54, 8 and 2 nodes, 29 independent cycles, per-compound uncertainty dominated by a 1.0 kcal/mol force-field term rather than by anything statistical. Accepted with a cosmetic warning about the disconnection and no penalty. Its honest weaknesses: 88 of the 90 edges have a single repeat, so there is no inter-replicate spread on them, and the offset between charge blocks is set rather than measured because no charge-neutral perturbation can cross it.
The most expensive mistake in this campaign was in four lines of post-processing, not in any simulation. Compounds that exist as several states, protonation variants, tautomers, rotamers, or the two enantiomers of a racemate, are run as separate nodes and combined afterwards in the partition function rather than by averaging dG. That is the right thing to do and it is what the reference implementations do. What none of them check is whether the states being combined sit in the same connected component.
Often they do not. Three compounds here had one state on the main component and one on an island, and the offset between two components is not measured by anything: it is a gauge, set by equalising component means. A Boltzmann sum is dominated by the tighter state, so combining across that boundary lets an arbitrary constant choose the answer, and because the sum takes the minimum the error is one-directional. It can only ever pull a compound tighter.
The first wave never hit this, because every multi-state compound happened to have all of its states inside one component. The second wave added edges, split the graph differently, and three compounds moved between 2.7 and 4.2 kcal/mol in the same direction for no physical reason. The live board went 0.44 to 0.36. Adding the guard, which keeps the state measured against the largest component instead of combining, took it to 0.49, above where it started.
Two things to take from that. The guard itself is cheap:
if len({component_of[n] for n in nodes}) > 1: # keep the node on the component holding the most nodes; do not combine return dg[argmax_component_size], sd[argmax_component_size]
The second is harder to apply and matters more. This error was invisible to every diagnostic in this notebook. Overlap was fine, cycle closure was fine, the MBAR bars were fine, and the submission validator passed it. It showed up only as a 0.08 drop on a board nobody outside the organisers can reproduce, which is exactly the size that is easy to dismiss as noise. If a step in your pipeline combines a measured quantity with a gauge, no convergence diagnostic will ever tell you, because nothing failed to converge.
What you have now
You now have, without downloading anything:
- a measured window for what a plausible answer looks like, from 3,989 published flavivirus
protease measurements
- proof that none of the 54 has a published activity, and the option of proving none has ever
been deposited at all
- the series definition that relative methods need, and the knowledge that Murcko scaffolds
will shatter it
- the identity of the one compound whose pose you are given, confirmed two independent ways
- the two traps: three racemates that survive only in GetStereoGroups(), and three
charge-changing edges in the cheapest network
- an error budget from a real programme with the answers attached, and a reason to trust the
rank-correlation metric
- a submission file that validates
- the measured cost and error profile of one real alchemical campaign on this target, including
the two purchases that turned out to be worth making and the one that did not
What is genuinely hard here is not the setup. It is that the signal and the method noise are the same size, the target has no alchemical prior art in the literature, and three of your perturbations change the net charge. Almost everything on this page is the part that can be done before any compute is spent.
If you are about to spend compute, section 12 is the part worth re-reading. The three findings that mattered most there were all corrections to a plan that looked reasonable on paper: the cost multiplier was 2.7x smaller than four independent estimates said, the error was concentrated in 31 of 90 edges that a pre-simulation descriptor could identify, and extra repeats were strictly worse value than longer production. Measure one edge first.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.