A scored blind-challenge entry, rebuilt
Drug discovery sample notebook on Scigantic.
Rebuild a scored OpenADMET CYP submission from twelve frozen models, see where its error sits, then refit the weights yourself.
Figure: Crystal structure of human cytochrome P450 3A4 (PDB 1TQN), the enzyme at the centre of the challenge, with its heme cofactor visible in the active site
The notebook
10 code cells, shown with the output each one printed.
Start here. The outputs below are from an earlier run. Nothing is live until you run a cell yourself: click the grey cell below and press Shift+Enter. It reads the challenge training set off the archive mounted at ~/archive and takes about two seconds. Then change ISOFORM to CYP2D6 and run it again.
# Start here: click this cell, press Shift+Enter.
import os
from pathlib import Path
import pandas as pd
ISOFORM = "CYP3A4" # try CYP1A2, CYP2C9 or CYP2D6, then run again
mount = Path(os.environ.get("SCIGANTIC_MOUNT_PATH") or Path.home() / "archive")
train = pd.read_csv(mount / "cyp-challenge-TRAIN_inhibition.csv")
col = f"{ISOFORM}_pIC50_direct_inhibition"
d = train[[col, f"{col}_conf_low", f"{col}_conf_high"]].dropna()
width = d[f"{col}_conf_high"] - d[f"{col}_conf_low"]
print(f"{len(train):,} training compounds, {len(d):,} of them with a measured {ISOFORM} pIC50")
print(f"median pIC50 {d[col].median():.2f}, strongest inhibitor {d[col].max():.2f}")
print(f"each label carries a 95% credible interval, {width.median():.2f} log units wide at the median")
print(f"{(width > 1).mean() * 100:.0f}% of them are wider than a whole log unit")
print()
print("A prediction anywhere inside a compound's interval scores zero error.")
print("The challenge is graded on that, and the recipe below is built around it.")
Rebuilding a scored submission to the OpenADMET CYP blind challenge
The OpenADMET CYP inhibition blind challenge asks for pIC50 predictions against four cytochrome P450 isoforms for 750 compounds whose labels are held back. The training data is mounted read-only at ~/archive. This notebook rebuilds our best scored entry from it.
That entry scored a macro ST-RAE of 0.4436 (CYP1A2 0.4739, CYP2C9 0.3802, CYP2D6 0.5442, CYP3A4 0.3762); lower is better. It was rank 11 of 219 on the 2026-09-25 interim leaderboard and would sit 24th of 177 on the live board as of 2026-10-19, where the top has since moved to 0.3537.
The whole recipe is open source, MIT-0, at Scigantic/open-admet-challenge-cyp: the submitted CSV, the frozen predictions of all twelve models behind it, the training tables, the training scripts, and the scored history of every upload. This notebook runs the combiner half of it. No training, no GPU, about ten seconds of CPU, and every stage is checked against the file that was actually uploaded, which ships in the bundle it downloads. On the environment pinned in the repo that check matches the sha256. On this image it matches to 3e-06 of a pIC50 unit instead, and a section near the end says why.
The last cell refits the weights yourself: drop members, change the ridge penalty, and see what it costs.
How the data is shaped, how to featurize it, and how to train a model from scratch are in OpenADMET's own two tutorials, which landed in this notebook's folder alongside it: 01-openadmet-activity-tutorial.ipynb and 02-tdi-prediction.ipynb.
The metric
Every training label comes with a 95% credible interval from the dose-response fit. The challenge scores Soft-Threshold Relative Absolute Error: a prediction inside a compound's interval costs nothing, and outside it you pay the distance to the nearer edge.
hinge(p) = max(0, lo - p) + max(0, p - hi)
The relative part divides that by the hinge a constant mean prediction would pay, so 1.0 is no better than guessing the mean and 0.0 is never outside an interval.
Squared error pushes every prediction toward its label. This one does not care where inside the interval a prediction lands, so all the available gain sits in the compounds that are currently missed, and a compound with a two-log-unit interval should be cheaper to miss than one measured tightly. Fitting the hinge directly, rather than fitting MAE and hoping, is most of what the recipe below does.
import hashlib, io, json, os, urllib.request, zipfile
from pathlib import Path
import numpy as np
import pandas as pd
import scipy
import matplotlib.pyplot as plt
from scipy.optimize import minimize
print(f"numpy {np.__version__} | pandas {pd.__version__} | scipy {scipy.__version__}")
EPS = ("CYP1A2", "CYP2C9", "CYP2D6", "CYP3A4")
COL = {e: f"{e}_pIC50_direct_inhibition" for e in EPS}
PRED = {e: f"{e.lower()}_pic50_direct_inhibition_pred" for e in EPS}
sha = lambda p: hashlib.sha256(Path(p).read_bytes()).hexdigest()
# the metric, ported from the organizers' rae_soft_threshold_absolute_error
hinge = lambda p, lo, hi: np.clip(lo - p, 0, None) + np.clip(p - hi, 0, None)
def st_rae(y_true, y_pred, lo, hi):
y_true, y_pred, lo, hi = (np.asarray(a, float) for a in (y_true, y_pred, lo, hi))
base = hinge(np.full_like(y_true, y_true.mean()), lo, hi)
return float(hinge(y_pred, lo, hi).sum() / base.sum())
def mm(b, c, w):
# mean-matched blend: shift c onto b's mean, then mix. Keeps b's level.
return (1 - w) * b + w * (c + (b.mean() - c.mean()))
def restore(raw, ref):
# rank-preserving: keep the reordering, put the mean and sd back to ref's
return (raw - raw.mean()) / max(raw.std(), 1e-9) * ref.std() + ref.mean()
print("metric check on a toy compound with interval [5.0, 6.0]:")
for p in (4.2, 5.0, 5.7, 6.0, 7.1):
print(f" prediction {p}: hinge {float(hinge(np.array([p]), 5.0, 6.0)[0]):.1f}")
Output:
numpy 2.0.2 | pandas 2.2.3 | scipy 1.14.1 metric check on a toy compound with interval [5.0, 6.0]: prediction 4.2: hinge 0.8 prediction 5.0: hinge 0.0 prediction 5.7: hinge 0.0 prediction 6.0: hinge 0.0 prediction 7.1: hinge 1.1
The inputs
Twelve models were trained for this: Chemprop directed message-passing backbones over the molecular graph, some initialized randomly and some from CheMeleon, all of them multi-task regressors over a 22-column table of organizer potency, single-concentration fold change, ChEMBL potency and PubChem qHTS signals, five scaffold folds each. 364 SageMaker jobs ran in total; these twelve are the ones the submission used.
Here is each one. The middle column is the training invocation, copied from training/README.md in the public repo, which carries the hyperparameters this table leaves out. The iv, ii and mu members are arms B and C again with only the objective or the sample weighting changed, which is most of why so many members predict nearly the same thing. Four of the twelve use the interval hinge rather than MAE.
| member | how it was trained | out-of-fold rows |
| armA | --arm A: d_h 700, depth 5, FFN 300, dropout 0.1, random init, MAE, inverse-variance sample weights | 31,669 |
| armB | --arm B: d_h 2048, depth 4, FFN 1024, dropout 0.2, random init | 31,669 |
| armC | --arm C: as B, message passing initialized from the CheMeleon checkpoint | 31,669 |
| armD | --arm D: as B, initialized from the organizers' CYP-finetuned CheMeleon checkpoint | 31,669 |
| mu2048 | --arm B --objective mae --weighting uniform | 31,669 |
| ii2048 | --arm B --objective interval --weighting invvar | 31,669 |
| ivB2048 | --arm B --objective interval --weighting uniform | 31,669 |
| ivC2048 | --arm C --objective interval --weighting uniform | 31,669 |
| union18 | the Workbench recipe, 18 heads, reproducing the public SuperCowPowers member | 31,670 |
| mt_aux8 | the same recipe on organizer data only, 8 heads | 4,905 |
| aux_ac50 | Chemprop: the union18 heads plus four qHTS AC50 heads, inverse-variance weights | 31,670 |
| cp_interval | as aux_ac50, with the scored heads on the interval hinge | 31,670 |
A thirteenth member, g26_tdi_armB, is arm B trained on the TDI-condition table. It has no out-of-fold predictions and is read only in step 3, so the combiner never weights it and it is not one of the twelve.
Retraining them needs a GPU and a day. Their predictions are frozen and published, so the combiner needs neither. The cell below pulls the slice of them this rebuild reads: 2.8 MB, one request. The full 113 MB of per-fold predictions, and the scripts that produced them, are in the public repo.
For each model you get its out-of-fold predictions on the training compounds (from the fold where that compound was held out) and its predictions on the 750 blind compounds. The weights are fitted on the first and applied to the second.
URL = "https://scigantic.com/openadmet-cyp/r25-inputs-v1.zip"
ROOT = Path.home() / "openadmet-r25-inputs"
D = ROOT / "data"
if not (D / "MANIFEST.json").exists():
with urllib.request.urlopen(URL, timeout=180) as r:
blob = r.read()
print(f"fetched {len(blob) / 1e6:.2f} MB")
zipfile.ZipFile(io.BytesIO(blob)).extractall(ROOT)
MANIFEST = json.loads((D / "MANIFEST.json").read_text())
MEMBERS = ["armA", "armB", "armC", "armD", "ivB2048", "ivC2048", "mu2048", "ii2048",
"mt_aux8", "union18", "aux_ac50", "cp_interval"]
def member(name, kind):
d = pd.read_csv(D / "members" / name / f"{kind}.csv")
d["Molecule_Name"] = d["Molecule_Name"].astype(str)
return d
def member_pred(name, kind, ep, names):
# per-compound mean over the five folds of that endpoint's prediction column
d = member(name, kind)
c = PRED[ep]
if c not in d.columns:
c = c[:-5] # cp_interval wrote its columns without the _pred suffix
return d.groupby("Molecule_Name")[c].mean().reindex(names)
def oof_base(ep):
# training compounds for one endpoint: label, credible interval, incumbent prediction
o = pd.read_csv(D / "oof_base" / f"r08_like_oof_{ep}.csv")
o["Molecule_Name"] = o["Molecule_Name"].astype(str)
return o.dropna(subset=["label", "conf_low", "conf_high", "r08_like"]).reset_index(drop=True)
sub = {k: pd.read_csv(D / "submissions" / f"{k}.csv") for k in ("R17", "R23", "R24", "R25")}
for k, v in sub.items():
v["Molecule_Name"] = v["Molecule_Name"].astype(str)
assert sha(D / "submissions" / f"{k}.csv") == MANIFEST["submissions"][k]["sha256"], k
IDS = sub["R17"]["Molecule_Name"].tolist()
assert len(IDS) == 750
print("out-of-fold rows per endpoint:", {e: len(oof_base(e)) for e in EPS})
print("blind compounds:", len(IDS))
print("members:", len(MEMBERS))
Output:
fetched 2.76 MB
out-of-fold rows per endpoint: {'CYP1A2': 1390, 'CYP2C9': 1275, 'CYP2D6': 1441, 'CYP3A4': 2326}
blind compounds: 750
members: 12
The incumbent
R17 is the incumbent: one 750-row CSV that was already on the leaderboard, holding for each endpoint the best column from any submission that had already been scored. Nothing below is built from scratch. Every step changes exactly one endpoint's column of that file and copies the rest, so each step's effect on the board is attributable to that one edit.
The same predictor appears in the out-of-fold tables as the r08_like column. That is what lets you measure a candidate change on training compounds before spending a submission on it.
Scored on out-of-fold rows, it looks like this. For three of the four endpoints these numbers are far worse than the live ones for the same file, because out-of-fold training compounds are harder than the blind set: the blind set was built by buying analogues of confirmed screening hits, so it sits closer to the training data. Read out-of-fold ST-RAE as a direction, not as a forecast of the board.
score = {}
for e in EPS:
o = oof_base(e)
score[(e, "incumbent")] = st_rae(o.label, o.r08_like, o.conf_low, o.conf_high)
print("out-of-fold ST-RAE of the incumbent, per endpoint (lower is better):")
for e in EPS:
print(f" {e} {score[(e, 'incumbent')]:.4f}")
Output:
out-of-fold ST-RAE of the incumbent, per endpoint (lower is better): CYP1A2 0.7132 CYP2C9 0.4862 CYP2D6 0.8327 CYP3A4 0.3776
Step 1. CYP3A4: blend an interval-trained model, then correct on the auxiliary heads
CYP3A4 is the most promiscuous of the four isoforms and has the most labels, and it is where the interval-objective models pay off most. Two moves, in order:
- Three passes of mean-matched blending over the candidate column, pulling in mt_aux8 and
the aux_ac50 potency head, then a small weight on ivB2048, the arm trained against the interval hinge, chosen on a coarse grid by the hinge it leaves.
- A hinge-fitted linear correction on four auxiliary heads: the qHTS AC50 head, the ChEMBL
potency head, the maximum-response head and the single-concentration fold-change head, each standardized, with an L2 penalty. None of the four is the scored quantity, so they carry information the potency head does not.
Powell minimization, four coefficients, free to go negative.
E = "CYP3A4"; e = E.lower()
AUX = [f"{e}_pic50_qhts_pred", f"{e}_pic50_chembl_pred", f"{e}_max_response_pred", f"{e}_log2fc_pred"]
W_GRID = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5]
L2_A4 = 0.1
o = oof_base(E).dropna(subset=["candidate"]).reset_index(drop=True)
ac = member("aux_ac50", "oof").set_index("Molecule_Name")
o = o[o.Molecule_Name.isin(ac.index)].reset_index(drop=True)
o["mt"] = o.Molecule_Name.map(member("mt_aux8", "oof").groupby("Molecule_Name")[PRED[E]].mean())
o["iv"] = o.Molecule_Name.map(member("ivB2048", "oof").groupby("Molecule_Name")[PRED[E]].mean())
o = o.dropna(subset=["mt", "iv"]).reset_index(drop=True)
A_ = ac.loc[o.Molecule_Name]
direct = A_[PRED[E]].to_numpy(float)
base = mm(mm(mm(o.candidate.to_numpy(), o.mt.to_numpy(), 0.18), direct, 0.20), o.mt.to_numpy(), 0.20)
lo, hi = o.conf_low.to_numpy(), o.conf_high.to_numpy()
best_w = min(W_GRID, key=lambda w: hinge(mm(base, o.iv.to_numpy(), w), lo, hi).sum())
stage_b = mm(base, o.iv.to_numpy(), best_w)
Xraw = np.nan_to_num(np.column_stack(
[(A_[c].to_numpy(float) - direct) if "pic50" in c else A_[c].to_numpy(float) for c in AUX]), nan=0.0)
mu, sd = Xraw.mean(0), Xraw.std(0) + 1e-9
X = (Xraw - mu) / sd
w_a4 = minimize(lambda v: hinge(stage_b + X @ v, lo, hi).sum() / len(base) + L2_A4 * np.sum(v ** 2),
np.zeros(4), method="Powell", options={"maxiter": 20000, "xtol": 1e-6, "ftol": 1e-8}).x
score[(E, "base")] = st_rae(o.label, base, lo, hi)
score[(E, "R23")] = st_rae(o.label, stage_b + X @ w_a4, lo, hi)
print(f"interval-member weight chosen on the grid: {best_w}")
print("auxiliary-head coefficients:")
for c, x in zip(AUX, w_a4):
print(f" {c:32s} {x:+.4f}")
print(f"out-of-fold ST-RAE CYP3A4: blend {score[(E,'base')]:.4f} -> after correction "
f"{score[(E,'R23')]:.4f} (fitted on {len(o):,} rows)")
Output:
interval-member weight chosen on the grid: 0.2 auxiliary-head coefficients: cyp3a4_pic50_qhts_pred +0.0297 cyp3a4_pic50_chembl_pred +0.0396 cyp3a4_max_response_pred -0.0333 cyp3a4_log2fc_pred -0.0487 out-of-fold ST-RAE CYP3A4: blend 0.3685 -> after correction 0.3603 (fitted on 2,326 rows)
Now the same transform on the 750 blind compounds, using the coefficients just fitted and the same standardization, then a check against the submitted file. Each stage is written out and read back before the next one starts, because the original build did exactly that and a CSV round trip moves the low bits.
ab = member("aux_ac50", "blind").set_index("Molecule_Name").loc[IDS]
iv_blind = member_pred("ivB2048", "blind", E, IDS).to_numpy(float)
direct_b = ab[PRED[E]].to_numpy(float)
base_b = sub["R17"][COL[E]].to_numpy(float)
Xb = (np.nan_to_num(np.column_stack(
[(ab[c].to_numpy(float) - direct_b) if "pic50" in c else ab[c].to_numpy(float) for c in AUX]),
nan=0.0) - mu) / sd
R23 = sub["R17"].copy()
R23[COL[E]] = mm(base_b, iv_blind, best_w) + Xb @ w_a4
BUILD = Path.home() / "openadmet-r25-build"; BUILD.mkdir(exist_ok=True)
R23.to_csv(BUILD / "R23_rebuilt.csv", index=False)
def agreement(df, key):
cols = [COL[e] for e in EPS]
d = np.abs(df[cols].to_numpy(float) - sub[key][cols].to_numpy(float)).max()
same = sha(BUILD / f"{key}_rebuilt.csv") == MANIFEST["submissions"][key]["sha256"]
print(f"vs the submitted {key}: largest disagreement {d:.1e} pIC50, identical bytes: {same}")
return d
agreement(R23, "R23")
R23 = pd.read_csv(BUILD / "R23_rebuilt.csv"); R23["Molecule_Name"] = R23["Molecule_Name"].astype(str)
Output:
vs the submitted R23: largest disagreement 4.4e-07 pIC50, identical bytes: False
Step 2. The other three endpoints: one joint fit over all twelve members
Most of the gain is here. For each of CYP1A2, CYP2C9 and CYP2D6, take every member's out-of-fold prediction, subtract the incumbent to get twelve delta columns, and fit weights on those deltas by minimizing the hinge directly, plus an L2 penalty whose strength was chosen per endpoint by nested inner folds. Negative weights are allowed: a member that is systematically wrong in a consistent direction still carries information.
The fitted column is then rescaled to the incumbent's mean and standard deviation, which throws away the fit's level and spread and keeps only its ordering. The hinge fit sees only out-of-fold compounds, so its absolute calibration is fitted to them, while its reordering generalizes. Keeping the reordering and borrowing the incumbent's calibration transferred to the board; keeping the raw fitted values did not.
L2 = {"CYP1A2": 1.0, "CYP2C9": 0.1, "CYP2D6": 0.1} # modal nested inner-fold choices
def joint_fit(ep, members, l2):
"""Hinge-fit weights over member deltas from the incumbent, out of fold."""
o = oof_base(ep)
M = {m: member_pred(m, "oof", ep, o.Molecule_Name) for m in members}
mk = [m for m in members if M[m].notna().all()]
keep = pd.DataFrame({m: M[m] for m in mk}).reset_index(drop=True)
ok = keep.notna().all(axis=1).to_numpy()
o = o[ok].reset_index(drop=True); keep = keep[ok].reset_index(drop=True)
b = o.r08_like.to_numpy()
X = np.column_stack([keep[m].to_numpy(float) - b for m in mk])
lo, hi = o.conf_low.to_numpy(), o.conf_high.to_numpy()
w = minimize(lambda v: hinge(b + X @ v, lo, hi).sum() / len(b) + l2 * np.sum(v ** 2),
np.zeros(X.shape[1]), method="Powell",
options={"maxiter": 60000, "xtol": 1e-6, "ftol": 1e-8}).x
return {"oof": o, "members": mk, "weights": w,
"st_rae": st_rae(o.label, b + X @ w, lo, hi),
"hinge_before": hinge(b, lo, hi), "hinge_after": hinge(b + X @ w, lo, hi)}
R24 = R23.copy(); weights = {}; hinge_curves = {}
for ep in ("CYP1A2", "CYP2C9", "CYP2D6"):
fit = joint_fit(ep, MEMBERS, L2[ep])
o, mk, w = fit["oof"], fit["members"], fit["weights"]
weights[ep] = dict(zip(mk, w))
score[(ep, "R24")] = fit["st_rae"]
hinge_curves[ep] = (fit["hinge_before"], fit["hinge_after"])
B = {m: member_pred(m, "blind", ep, pd.Index(IDS)) for m in mk}
base_b = R23[COL[ep]].to_numpy(float)
Xb = np.column_stack([B[m].to_numpy(float) - base_b for m in mk])
R24[COL[ep]] = restore(base_b + Xb @ w, base_b)
print(f"{ep}: {len(mk)} members over {len(o):,} rows, out-of-fold "
f"{score[(ep,'incumbent')]:.4f} -> {fit['st_rae']:.4f}")
R24.to_csv(BUILD / "R24_rebuilt.csv", index=False)
agreement(R24, "R24")
R24 = pd.read_csv(BUILD / "R24_rebuilt.csv"); R24["Molecule_Name"] = R24["Molecule_Name"].astype(str)
print()
print(pd.DataFrame(weights).round(3).to_string())
Output:
CYP1A2: 12 members over 1,390 rows, out-of-fold 0.7132 -> 0.7018
CYP2C9: 12 members over 1,275 rows, out-of-fold 0.4862 -> 0.4641
CYP2D6: 12 members over 1,441 rows, out-of-fold 0.8327 -> 0.7983
vs the submitted R24: largest disagreement 3.4e-06 pIC50, identical bytes: False
CYP1A2 CYP2C9 CYP2D6
armA 0.006 0.038 0.058
armB 0.008 0.049 0.062
armC 0.014 0.028 0.022
armD 0.009 0.018 -0.000
ivB2048 0.013 0.027 0.040
ivC2048 0.014 0.010 0.038
mu2048 0.019 0.021 0.052
ii2048 0.010 0.024 0.035
mt_aux8 0.018 0.049 0.063
...
Read the weights column by column. No member dominates: twelve models trained on the same table make heavily correlated predictions, so the fit spreads weight thinly instead of picking winners, and negative weights were allowed but barely used. The columns sum to 0.16, 0.40 and 0.51, so the fitted column moves the incumbent part of the way toward the members' consensus rather than onto it. CYP1A2's column is the flattest because its L2 penalty is ten times heavier, which was the choice the inner folds made, and CYP1A2 also had the least left to give.
Where the error sits
Hinge error is not spread evenly. Between a fifth and a half of compounds cost nothing, because the incumbent already lands inside their interval, and a quarter to a third of them carry 80% of everything left. Sorting by what each compound costs and accumulating shows it. Both curves are drawn as a share of the incumbent's total, so where the blue one stops short of 100% is what step 2 bought: 1.6%, 4.5% and 4.1% of the hinge. On the live board that step moved the macro from 0.4620 to 0.4453.
fig, axes = plt.subplots(1, 3, figsize=(12.5, 3.3), dpi=80)
for ax, ep in zip(axes, ("CYP1A2", "CYP2C9", "CYP2D6")):
h0, h1 = hinge_curves[ep]
n, total = len(h0), h0.sum()
x = np.arange(1, n + 1) / n * 100
c0 = np.cumsum(np.sort(h0)[::-1]) / total * 100
c1 = np.cumsum(np.sort(h1)[::-1]) / total * 100
ax.plot(x, c0, color="#94a3b8", lw=2, label=f"incumbent ({score[(ep,'incumbent')]:.3f})")
ax.plot(x, c1, color="#1565c0", lw=2, label=f"after the fit ({score[(ep,'R24')]:.3f})")
k80 = x[np.searchsorted(c0, 80)]
ax.axhline(80, color="#cbd5e1", lw=0.8, ls="--", zorder=0)
ax.axvline(k80, color="#cbd5e1", lw=0.8, ls="--", zorder=0)
ax.set_title(f"{ep}: the worst {k80:.0f}% of {n:,} compounds\ncarry 80% of the error", fontsize=10)
ax.set_xlabel("share of compounds, most expensive first (%)")
ax.set_ylim(0, 105)
ax.legend(fontsize=8, frameon=False, loc="lower right")
ax.spines[["top", "right"]].set_visible(False)
axes[0].set_ylabel("share of the incumbent's\ntotal hinge (%)")
plt.tight_layout()
plt.show()
for ep in ("CYP1A2", "CYP2C9", "CYP2D6"):
o = oof_base(ep)
r = np.corrcoef(hinge(o.r08_like.to_numpy(), o.conf_low.to_numpy(), o.conf_high.to_numpy()),
(o.conf_high - o.conf_low).to_numpy())[0, 1]
print(f"{ep}: correlation between hinge paid and interval width {r:+.2f}")
Output:
CYP1A2: correlation between hinge paid and interval width +0.25 CYP2C9: correlation between hinge paid and interval width -0.06 CYP2D6: correlation between hinge paid and interval width +0.25
Step 3. CYP1A2: borrow the ordering from a model trained on a different assay
g26_tdi_armB is arm B retrained with four extra heads for the organizers' *time-dependent inhibition* pIC50: a different assay, 30-minute pre-incubation, not the quantity being scored here. Blend 20% of it into CYP1A2, mean-matched, then restore the mean and sd again so only the reordering survives. On the board it moved 0.4453 to 0.4436.
E = "CYP1A2"; W_G26 = 0.20
g = member_pred("g26_tdi_armB", "blind", E, IDS).to_numpy(float)
b = R24[COL[E]].to_numpy(float)
R25 = R24.copy()
R25[COL[E]] = restore(mm(b, g, W_G26), b)
for other in ("CYP2C9", "CYP2D6", "CYP3A4"):
assert np.array_equal(R25[COL[other]].to_numpy(), R24[COL[other]].to_numpy())
out = BUILD / "R25_rebuilt.csv"
R25.to_csv(out, index=False)
worst = agreement(R25, "R25")
print(f" rebuilt sha256 {sha(out)}")
print(f" submitted sha256 {MANIFEST['submissions']['R25']['sha256']}")
assert worst < 1e-4, "the rebuild has drifted far enough to be a bug, not a tolerance"
# what the challenge platform checks on upload
assert len(R25) == 750 and R25.Molecule_Name.is_unique
assert list(R25.columns) == ["SMILES", "Molecule_Name"] + [COL[e] for e in EPS]
assert R25[[COL[e] for e in EPS]].notna().all().all()
print(f"\nwrote ~/{out.relative_to(Path.home())}, 750 unique compounds, four columns, no gaps")
preview = R25.set_index("Molecule_Name")[[COL[e] for e in EPS]].head(3).round(3)
preview.columns = list(EPS)
print(preview.to_string())
Output:
vs the submitted R25: largest disagreement 3.4e-06 pIC50, identical bytes: False
rebuilt sha256 a7e9a42f1b7f7163263c6c0756159c8cc78c1f87c99fc8083250055df3407289
submitted sha256 8fae881bfc3a343873d39dd1fc5dfd880f0de88a5da33da7c05232cb7896167f
wrote ~/openadmet-r25-build/R25_rebuilt.csv, 750 unique compounds, four columns, no gaps
CYP1A2 CYP2C9 CYP2D6 CYP3A4
Molecule_Name
OCNT-0010024 4.076 4.134 3.595 3.883
OCNT-0491471 4.208 3.982 3.288 3.429
OCNT-0493952 5.590 5.407 3.073 5.087
What it scored
| CYP1A2 | CYP2C9 | CYP2D6 | CYP3A4 | macro | |
| incumbent, live | 0.4909 | 0.3862 | 0.5694 | 0.4014 | 0.4620 |
| after steps 1 and 2 | 0.4806 | 0.3802 | 0.5442 | 0.3762 | 0.4453 |
| after step 3 | 0.4739 | 0.3802 | 0.5442 | 0.3762 | 0.4436 |
The out-of-fold numbers printed above are lower-is-better too, but they are not on the same scale as these: they are measured on the training compounds the weights were fitted on, and they read optimistic. docs/submission_history.md in the repo tabulates how often each endpoint's offline gain actually transferred to the board: CYP3A4 6 times out of 6, CYP2D6 2 out of 5.
Why the bytes do not match here
The three checks printed a disagreement around a millionth of a pIC50 unit and identical bytes: False. The same code on a Mac prints zero and True, under scipy 1.18 and under the same scipy 1.14 this image has. The library versions are not what decides it. The BLAS that numpy is linked against is: of the four environments measured, the two on Accelerate reproduce the bytes and the two on OpenBLAS, including this image, do not.
Powell is derivative-free. It walks a set of directions, line-searches along each, and stops when a pass stops paying more than xtol and ftol. Where it stops depends on the order the floating-point sums accumulate underneath it, which is what the BLAS changes. A member weight that lands on 0.0383 here and 0.0384 elsewhere moves 750 predictions by a millionth of a log unit, which is every byte after the sixth decimal.
Reproducing the numbers and reproducing the file are separate claims. The out-of-fold ST-RAE in step 2 is identical here to four decimal places and the fitted weights agree to three; the public repo's README tabulates all four environments.
Refit it yourself
The cell below refits one endpoint with the members and penalty you choose, and compares it against the shipped recipe. Two things worth trying:
- Drop members. DROP = ["mt_aux8", "union18"] removes the two reproductions of the
public Workbench recipe and keeps the ten that are ours. Does ten do as well as twelve?
- Set L2_PEN = 0.0. The out-of-fold score will improve. With no penalty the weights fit
the 1,275 to 1,441 compounds they can see, and that gain does not survive the trip to the blind set. The penalties in the recipe were picked on nested inner folds for that reason.
It ships set to drop those two members, so the output below is a worse fit than the shipped recipe.
ENDPOINT = "CYP2D6" # CYP1A2, CYP2C9 or CYP2D6
DROP = ["mt_aux8", "union18"] # members to leave out; [] keeps all twelve
L2_PEN = 0.1 # 0.0 fits harder and generalizes worse
use = [m for m in MEMBERS if m not in DROP]
fit = joint_fit(ENDPOINT, use, L2_PEN)
mk, w, fitted = fit["members"], fit["weights"], fit["st_rae"]
print(f"{ENDPOINT}: {len(mk)} members, L2 {L2_PEN}, fitted on {len(fit['oof']):,} out-of-fold rows")
print(f" incumbent {score[(ENDPOINT, 'incumbent')]:.4f}")
print(f" shipped recipe {score[(ENDPOINT, 'R24')]:.4f} (12 members, L2 {L2[ENDPOINT]})")
print(f" your fit {fitted:.4f} ({fitted - score[(ENDPOINT, 'R24')]:+.4f} vs shipped)")
print()
print(pd.Series(dict(zip(mk, w.round(3))), name="your weights").to_string())
Output:
CYP2D6: 10 members, L2 0.1, fitted on 1,441 out-of-fold rows incumbent 0.8327 shipped recipe 0.7983 (12 members, L2 0.1) your fit 0.8007 (+0.0024 vs shipped) armA 0.065 armB 0.068 armC 0.025 armD 0.004 ivB2048 0.044 ivC2048 0.045 mu2048 0.060 ii2048 0.037 aux_ac50 0.056 cp_interval 0.052
Submitting your own
~/openadmet-r25-build/R25_rebuilt.csv is a valid submission as it stands: 750 rows, the four *_pIC50_direct_inhibition columns, in the order the test set gives them. Upload it at the challenge Space and it will score what it scored for us. Three places to change it first:
- Nothing here looks at interval width. In the cost function a wide interval is cheap to
miss and a narrow one is expensive, so you would expect the error to pile up on narrow intervals. It does the opposite: the correlation printed above is positive on CYP1A2 and CYP2D6, so the compounds with uncertain labels are also the hard ones. Weighting the fit by width, or fitting narrow-interval compounds separately, is untried.
- CYP3A4 got a two-stage treatment and the other three did not. The auxiliary-head
correction in step 1 was never fitted for CYP1A2, CYP2C9 or CYP2D6. The data to try it is in the bundle you just downloaded: aux_ac50 carries the same four heads for every endpoint, not only CYP3A4.
- Train something new. OpenADMET's own tutorials are in this folder:
01-openadmet-activity-tutorial.ipynb builds a LightGBM baseline from RDKit descriptors end to end, and 02-tdi-prediction.ipynb does the time-dependent inhibition track. Both read the same mounted archive. The image already has rdkit, lightgbm, sklearn and useful_rdkit_utils, so nothing installs.
The submission deadline is 2026-11-03. The assistant panel on the right has this archive's schema card loaded, including the measured facts about sparsity and the traps that decide a submission. Ask it.
Everything here is MIT-0 at github.com/Scigantic/open-admet-challenge-cyp, including the training scripts for all twelve members and the full per-fold predictions.
Run it yourself
This sample opens as a hosted JupyterLab notebook with its datasets already mounted, so nothing has to be downloaded first. The notebook arrives with its outputs already rendered, and an AI assistant in the notebook can explain each step or change the analysis. Open the sample or browse the full catalog.