OpenFE Industry Benchmark 2024: calculated vs measured binding free energies

The paired calculated-and-measured corpus behind the field's working estimate of how accurate relative binding free energy calculations actually are. Fifteen pharmaceutical partners (ASAP, AstraZeneca, Bayer, Biogen, Boehringer Ingelheim, BMS, DeepOrigin, Eli Lilly, Genentech, GSK, Janssen, Merck, Odyssey and Roche) ran one fixed open-source protocol over 1,737 ligands and published the predictions next to the experimental numbers. The public half is 873 ligands across 58 systems; the private half is 864 ligands across 36 datasets from 10 partners, anonymised so that the target identities stay confidential while the error statistics remain usable. Every public ligand also carries an MM-GBSA score, and 918 ligands across 59 systems carry Schrodinger FEP+ predictions, so the release is simultaneously a head-to-head against the commercial standard and against a cheap baseline. The 1,145 public perturbation edges are reported per leg and per repeat: three independent repeats of the complex and solvent legs, each with its own free energy, MBAR uncertainty and smallest nearest-neighbour lambda overlap, which makes this one of the few places where inter-replicate spread and convergence diagnostics can be measured rather than assumed. The protocol is AMBER14SB protein with OpenFF Sage 2.2.0 ligands and AM1-BCC charges, run in OpenMM 8.1.1 through openfe 1.0.1 with Hamiltonian replica exchange, MBAR and a cinnabar maximum-likelihood estimator, at 11 lambda windows and 5 ns per window for neutral transformations and 22 windows at 20 ns for charge-changing ones. Headline accuracy, as published: edgewise ddG RMSE 1.37 kcal/mol on the public systems and 1.93 on the blinded ones, with all-to-all pairwise RMSE of 1.72 and 2.44 respectively. Anyone sizing or defending an alchemical campaign is currently citing those numbers from the paper; this archive makes them queryable per system, per edge and per repeat. Published as Large-scale collaborative assessment of binding free energy calculations for drug discovery using OpenFE, J. Chem. Inf. Model. (2026). Redistributed from the OpenFreeEnergy/IndustryBenchmarks2024 repository under the MIT license. Hosted by Scigantic for direct mounting.

Category
Drug Discovery
Provider
Open Free Energy consortium and 15 pharmaceutical partners
License
MIT
Hosted on
Amazon S3, bucket scigantic-openfe-benchmark, us-east-1
Access
Free and public. No account with the data provider is needed.
Source
https://doi.org/10.1021/acs.jcim.6c00089
Data structure and loading code

Fields

ligand namestring - ligand identifier, unique withi
DG (kcal/mol)float - CALCULATED absolute binding free
uncertainty (kcal/mol)float - the MLE statistical uncertainty
Exp DG (kcal/mol)float - MEASURED binding free energy. Ra
Exp dDG (kcal/mol)float - reported experimental error. Fre
partner_idstring - contributing company. Public se
dataset_namestring - the contributor's own path for
system groupstring - PUBLIC TABLE ONLY. One of 9: me
system namestring - PUBLIC TABLE ONLY. The target,
DG MMGBSAfloat - PUBLIC TABLE ONLY. MM-GBSA score
ligand_A, ligand_BEDGE TABLE - the two endpoints of the pe
{complex,solvent}_repeat_{0,1,2}_DG (kcal/mol)EDGE TABLE - 6 columns. The free energy
{complex,solvent}_repeat_{0,1,2}_dDG (kcal/mol)EDGE TABLE - 6 columns. MBAR statistical
{complex,solvent}_repeat_{0,1,2}_smallest_overlapEDGE TABLE - 6 columns. Smallest nearest

Load the data

# How accurate are relative binding free energy calculations, per target?
# This is the number everyone cites from the paper. Here it is computed from the data.
import pandas as pd, numpy as np
from scipy.stats import kendalltau

BASE = "/mnt/archive/industry_benchmarks/analysis"
dg = pd.read_csv(f"{BASE}/processed_results/combined_pymbar3_calculated_dg_data.csv")

# (group, name) is the system key: "system name" alone is not unique.
def per_system(g):
    return pd.Series({
        "n": len(g),
        "tau": kendalltau(g["DG (kcal/mol)"], g["Exp DG (kcal/mol)"]).statistic,
        "rmse": np.sqrt(((g["DG (kcal/mol)"] - g["Exp DG (kcal/mol)"]) ** 2).mean()),
        "exp_spread": g["Exp DG (kcal/mol)"].std(ddof=1),
    })

res = dg.groupby(["system group", "system name"]).apply(per_system)
res = res[res["n"] >= 8].sort_values("tau", ascending=False)
print(res.round(3).to_string())
print()
print(f"systems: {len(res)}")
print(f"Kendall tau   median {res.tau.median():.3f}  quartiles "
      f"{res.tau.quantile(.25):.3f}/{res.tau.quantile(.75):.3f}")
print(f"per-ligand RMSE  median {res.rmse.median():.2f} kcal/mol")
print(f"experimental spread  median {res.exp_spread.median():.2f} kcal/mol")

# The ratio of those last two is what sets how well any method can rank a series:
#   tau = (2/pi) * arcsin( s / sqrt(s^2 + m^2) )
# with s the experimental spread and m the per-ligand error. Check it against the data:
pred = (2 / np.pi) * np.arcsin(res.exp_spread / np.hypot(res.exp_spread, res.rmse))
print(f"\nanalytic vs observed tau: Kendall {kendalltau(pred, res.tau).statistic:+.3f}, "
      f"mean error {np.mean(pred - res.tau):+.3f}")

Written for a Scigantic notebook, where /mnt/archive is the mounted dataset.

Analyze this dataset without downloading it

On Scigantic, OpenFE Industry Benchmark 2024: calculated vs measured binding free energies opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

Related Drug Discovery datasets