Genomes to Fields: maize genotype by environment prediction competition data, 2014 to 2024: sample notebook

A worked notebook for Genomes to Fields: maize genotype by environment prediction competition data, 2014 to 2024, with 7 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Genomes to Fields Initiative
License
PDDL-1.0 (source)
Format
CSV (in sampled files)
Stored at
Amazon S3, s3://scigantic-g2f-maize (us-east-1)
Access
Free and public. No account with the data provider is needed.
DOI
10.25739/78mn-4394

The notebook

Start here. Click the grey cell below and press Shift+Enter. It reads the mounted G2F trial data and draws yield for one maize hybrid across every field site that grew it, so in a few seconds you will be looking at the genotype-by-environment effect this competition is about: the same genetics, a very different yield depending on where it was planted.

Change HYBRID to any other name the cell prints and run it again.

# Start here: click this cell, press Shift+Enter.
HYBRID = None                  # None picks the most widely grown hybrid

import os
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats("jpeg", pil_kwargs={"quality": 80})
plt.rcParams["figure.dpi"] = 72

ARCHIVE = Path(os.environ.get("SCIGANTIC_MOUNT_PATH", Path.home() / "archive"))
P2024 = ARCHIVE / "prediction-competition-2024"

trait = pd.read_csv(P2024 / "Training_data/1_Training_Trait_Data_2014_2023.csv",
                    low_memory=False).dropna(subset=["Yield_Mg_ha", "Hybrid"])

counts = trait.groupby("Hybrid").Env.nunique().sort_values(ascending=False)
if HYBRID is None:
    HYBRID = counts.index[0]
print(f"{len(trait):,} harvested plots, {trait.Env.nunique()} environments, "
      f"{trait.Hybrid.nunique():,} hybrids, {trait.Year.min()}-{trait.Year.max()}")
print(f"most widely grown hybrids: {', '.join(counts.index[:5])}")

one = (trait[trait.Hybrid == HYBRID].groupby("Env").Yield_Mg_ha.mean()
       .sort_values())
print(f"\n{HYBRID}: grown in {len(one)} environments, "
      f"{one.min():.1f} to {one.max():.1f} Mg/ha "
      f"({one.max() / max(one.min(), 0.1):.1f}x)")

fig, ax = plt.subplots(figsize=(11, max(3, len(one) * 0.22)))
ax.barh(one.index, one.values, color="#3b6ea5")
ax.axvline(one.mean(), color="#b4553b", lw=1.5,
           label=f"mean {one.mean():.1f} Mg/ha")
ax.set_xlabel("grain yield, Mg/ha at 15.5% moisture")
ax.set_title(f"One hybrid, many places: {HYBRID}")
ax.legend(loc="lower right")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()

What you are entering, and the two things that decide a good model

The 2026 G2F Genotype x Environment Prediction Competition asks for grain yield for each maize hybrid in each 2026 trial environment. Training data covers 2014 to 2025.

You register yourself on the official site.

to be an author on the joint results paper.

This archive holds the two previous open editions, each with its own held-out year and the organizers' published observed values for it, which is what makes local scoring possible.

Two properties of the task rule out the obvious approaches. The next cells demonstrate both on the real data.

The metric ignores the environment level. Scoring is the mean of per-environment Pearson r. Correlation is invariant to shifts and scales, so predicting how good a field was earns nothing. A model that nails the yield level everywhere but ranks hybrids randomly within each field scores zero.

Almost every test hybrid is new. In the 2024 edition only 9.8% of held-out hybrids had ever been grown in a training year. What does carry over is their parents, and their genotypes.

# the official metric, implemented once and used for everything below
def within_env_r(df, pred_col, obs_col="Yield_Mg_ha"):
    # Mean of per-environment Pearson r. This is the competition metric.
    rs = []
    for _, g in df.groupby("Env"):
        if len(g) > 2 and g[pred_col].std() > 0 and g[obs_col].std() > 0:
            rs.append(np.corrcoef(g[pred_col], g[obs_col])[0, 1])
    return float(np.mean(rs)), len(rs)


# Predict on the SUBMISSION TEMPLATE, which is what a real entry must fill, and keep the
# answer key strictly for scoring. The template has more rows than the key, so building
# features off the key would silently leave real submission rows unpredicted.
test = pd.read_csv(P2024 / "Testing_data/1_Submission_Template_2024.csv")[["Env", "Hybrid"]]
answer = pd.read_csv(P2024 / "Testing_data/7_Testing_Observed_Values.csv")
print(f"template rows {len(test):,} | answer-key rows {len(answer):,}")

# proof that the environment level is unscored: hand the metric perfect environment means
eg = answer.groupby("Env")["Yield_Mg_ha"]
answer["oracle_env_mean"] = eg.transform("mean")
r, _ = within_env_r(answer, "oracle_env_mean")
print(f"\nperfect knowledge of every environment mean, zero hybrid signal: r = {r:.4f}")
print("-> the environment level is worth nothing. Model hybrid order within an environment.")

# how much of the test set hybrid history can reach
seen = set(trait.Hybrid.unique())
rows_seen = answer.Hybrid.isin(seen)
print(f"\ntest hybrids ever grown in training: "
      f"{answer[rows_seen].Hybrid.nunique()} of {answer.Hybrid.nunique()}")
print(f"test ROWS whose hybrid was seen: {rows_seen.sum():,} ({rows_seen.mean():.1%})")

# the test files carry no parent columns, so split them out of the "P1/P2" hybrid name
parents = set(trait.Hybrid_Parent1.dropna()) | set(trait.Hybrid_Parent2.dropna())
for df in (test, answer):
    p = df.Hybrid.str.split("/", n=1, expand=True)
    df["Hybrid_Parent1"], df["Hybrid_Parent2"] = p[0], p[1]
known = answer.Hybrid_Parent1.isin(parents) | answer.Hybrid_Parent2.isin(parents)
print(f"unseen rows with at least one known parent: "
      f"{(known & ~rows_seen).sum():,} of {(~rows_seen).sum():,}")
print("-> parents carry what hybrids cannot.")

Output:

template rows 10,057 | answer-key rows 9,486

perfect knowledge of every environment mean, zero hybrid signal: r = -0.0000
-> the environment level is worth nothing. Model hybrid order within an environment.

test hybrids ever grown in training: 104 of 1063
test ROWS whose hybrid was seen: 1,172 (12.4%)
unseen rows with at least one known parent: 8,314 of 8,314
-> parents carry what hybrids cannot.

Features

Three blocks, in the order they contribute.

within-environment standardized yield across all training years. K pulls thinly observed parents toward zero. This one feature does most of the work.

to the training set and covering hybrids whose parents are also new.

organizers ship, plus a few explicit genotype-by-environment products. These only help a model that already ranks hybrids well.

K = 10.0        # combining-ability shrinkage: higher trusts thin parents less
N_GENO_PC = 40
N_ENV_PC = 15

# plot level -> Env x Hybrid, the unit we predict, then standardize within environment
trh = trait.groupby(["Env", "Hybrid"], as_index=False).agg(
    Yield_Mg_ha=("Yield_Mg_ha", "mean"), Year=("Year", "first"),
    Hybrid_Parent1=("Hybrid_Parent1", "first"), Hybrid_Parent2=("Hybrid_Parent2", "first"))
g = trh.groupby("Env")["Yield_Mg_ha"]
trh["z"] = ((trh.Yield_Mg_ha - g.transform("mean")) / g.transform("std")).fillna(0.0)

long = pd.concat([trh[[c, "z"]].rename(columns={c: "p"})
                  for c in ("Hybrid_Parent1", "Hybrid_Parent2")]).dropna(subset=["p"])
agg = long.groupby("p")["z"].agg(["sum", "count"])
gca = (agg["sum"] / (agg["count"] + K)).rename("gca")


def add_gca(df):
    out = df.copy()
    for c in ("Hybrid_Parent1", "Hybrid_Parent2"):
        out[f"gca_{c[-1]}"] = out[c].map(gca)
    out["gca_sum"] = out[["gca_1", "gca_2"]].fillna(0).sum(axis=1)
    out["gca_known"] = out[["gca_1", "gca_2"]].notna().sum(axis=1)
    return out


trh, test, answer = add_gca(trh), add_gca(test), add_gca(answer)
print(f"parents with an estimated effect: {len(gca):,}")
print(f"template rows with both parents known: {test.gca_known.eq(2).mean():.1%}, "
      f"one: {test.gca_known.eq(1).mean():.1%}, neither: {test.gca_known.eq(0).mean():.1%}")
r, _ = within_env_r(answer, "gca_sum")
print(f"parent combining ability alone: r = {r:.4f}")

Output:

parents with an estimated effect: 2,255
template rows with both parents known: 19.0%, one: 81.0%, neither: 0.0%
parent combining ability alone: r = 0.3079
from sklearn.decomposition import PCA


def pcs(df, n, prefix):
    df = df.apply(pd.to_numeric, errors="coerce")
    df = df.loc[:, df.notna().mean() > 0.8]
    df = df.fillna(df.mean())
    z = (df - df.mean()) / df.std().replace(0, 1)
    return pd.DataFrame(PCA(n_components=n, random_state=0).fit_transform(z),
                        index=df.index, columns=[f"{prefix}{i}" for i in range(n)]), df.shape


geno = pd.read_csv(P2024 / "Training_data/5_Genotype_Data_All_2014_2025_Hybrids_numerical.txt",
                   sep="\t", skiprows=1, index_col=0, low_memory=False)
gpc, gshape = pcs(geno, N_GENO_PC, "gpc")
print(f"genotypes: {gshape[0]:,} hybrids x {gshape[1]:,} markers -> {N_GENO_PC} PCs")

ec = pd.concat([pd.read_csv(P2024 / "Training_data/6_Training_EC_Data_2014_2023.csv").set_index("Env"),
                pd.read_csv(P2024 / "Testing_data/6_Testing_EC_Data_2024.csv").set_index("Env")])
epc, eshape = pcs(ec, N_ENV_PC, "epc")
print(f"environment covariates: {eshape[0]} environments x {eshape[1]} -> {N_ENV_PC} PCs")

GCOLS, ECOLS = list(gpc.columns), list(epc.columns)
FEAT = ["gca_sum", "gca_known"] + GCOLS + ECOLS + [f"gxe{i}" for i in range(6)]


def design(df):
    X = df.join(gpc, on="Hybrid").join(epc, on="Env")
    for c in GCOLS + ECOLS:
        if c not in X:
            X[c] = 0.0
    X[GCOLS + ECOLS] = X[GCOLS + ECOLS].fillna(0.0)
    for i in range(6):                    # cheapest usable GxE signal
        X[f"gxe{i}"] = X[f"gpc{i % 4}"] * X[f"epc{i // 4}"]
    return X[FEAT].astype(float).values


Xtr, ytr, Xte = design(trh), trh["z"].values, design(test)
print(f"design matrix: {Xtr.shape} -> template {Xte.shape}")

Output:

genotypes: 5,899 hybrids x 2,302 markers -> 40 PCs
environment covariates: 263 environments x 654 -> 15 PCs
design matrix: (107941, 63) -> template (10057, 63)
from sklearn.linear_model import RidgeCV
from sklearn.ensemble import HistGradientBoostingRegressor

ridge = RidgeCV(alphas=np.logspace(-2, 4, 25)).fit(Xtr, ytr)
test["pred_ridge"] = ridge.predict(Xte)

hgb = HistGradientBoostingRegressor(max_iter=400, learning_rate=0.06, max_depth=6,
                                    min_samples_leaf=40, l2_regularization=1.0,
                                    random_state=0).fit(Xtr, ytr)
test["pred_hgb"] = hgb.predict(Xte)

# Blend in standardized prediction space, NOT in rank space. Ranking flattens the prediction
# to uniform while observed yield is skewed, which costs Pearson r: the rank-space blend
# scores 0.2529, below both of its own components.
for c in ("pred_ridge", "pred_hgb"):
    gg = test.groupby("Env")[c]
    test[c + "_z"] = (test[c] - gg.transform("mean")) / gg.transform("std").replace(0, 1)
test["pred_blend"] = 0.35 * test.pred_ridge_z + 0.65 * test.pred_hgb_z

scored = test.merge(answer[["Env", "Hybrid", "Yield_Mg_ha", "oracle_env_mean"]],
                    on=["Env", "Hybrid"])
print(f"scoring on {len(scored):,} of {len(test):,} template rows with an observed value\n")
for name, col in [("perfect environment means (no hybrid signal)", "oracle_env_mean"),
                  ("parent combining ability only", "gca_sum"),
                  ("ridge", "pred_ridge"),
                  ("HistGradientBoosting", "pred_hgb"),
                  ("blend, z-space", "pred_blend")]:
    r, _ = within_env_r(scored, col)
    print(f"  {name:<44} r = {r:+.4f}")
print("\nFor reference, the first open edition's top-5 ensemble reached 0.369,")
print("and most entries landed between 0.30 and 0.37.")

Output:

scoring on 9,486 of 10,057 template rows with an observed value

  perfect environment means (no hybrid signal) r = -0.0000
  parent combining ability only                r = +0.3079
  ridge                                        r = +0.3088
  HistGradientBoosting                         r = +0.3192
  blend, z-space                               r = +0.3266

For reference, the first open edition's top-5 ensemble reached 0.369,
and most entries landed between 0.30 and 0.37.
# where the model is weak, which is where the gains are
per = (scored.groupby("Env")
       .apply(lambda g: np.corrcoef(g.pred_blend, g.Yield_Mg_ha)[0, 1], include_groups=False)
       .sort_values().rename("r").to_frame())
per["hybrids"] = scored.groupby("Env").size()

fig, ax = plt.subplots(figsize=(10, 6))
ax.barh(per.index, per.r, color=["#b4553b" if v < 0.25 else "#3b6ea5" for v in per.r])
ax.axvline(per.r.mean(), color="#444", lw=1.5, ls="--",
           label=f"mean {per.r.mean():.3f} (the score)")
ax.set_xlabel("within-environment Pearson r")
ax.set_title("Per-environment score: the average hides a wide spread")
ax.legend(loc="lower right")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()
print(per.to_string(float_format=lambda v: f"{v:+.3f}"))

Output:

               r  hybrids
Env                      
NYH3_2024 +0.140      370
GAH1_2024 +0.186      354
TXH1_2024 +0.207      237
IAH2_2024 +0.209      651
GAH2_2024 +0.220      369
WIH3_2024 +0.237      421
ONH3_2024 +0.246      422
NEH2_2024 +0.278      238
WIH1_2024 +0.316      354
DEH1_2024 +0.329      600
MNH1_2024 +0.346      397
NEH1_2024 +0.355      239
WIH2_2024 +0.355      741
...

Writing a submission

For the live competition, point this at the 2026 template and keep everything else. The format is three columns, Env,Hybrid,Yield_Mg_ha, no missing values, headers unaltered.

Because scoring is a within-environment correlation, the absolute scale of what you write cannot affect your score. The cell below maps the standardized prediction back onto a plausible yield scale anyway, so the file reads sensibly and survives inspection.

template = pd.read_csv(P2024 / "Testing_data/1_Submission_Template_2024.csv")
mu, sd = trait.Yield_Mg_ha.mean(), trait.Yield_Mg_ha.std()

sub = template[["Env", "Hybrid"]].merge(
    test[["Env", "Hybrid", "pred_blend"]], on=["Env", "Hybrid"], how="left")
sub["Yield_Mg_ha"] = (sub.pred_blend * sd * 0.45 + mu).clip(lower=0.0).round(4)
unreached = sub.Yield_Mg_ha.isna().sum()
sub["Yield_Mg_ha"] = sub.Yield_Mg_ha.fillna(round(mu, 4))
sub = sub[["Env", "Hybrid", "Yield_Mg_ha"]]

assert list(sub.columns) == ["Env", "Hybrid", "Yield_Mg_ha"], "headers must be unaltered"
assert len(sub) == len(template), "every template row must be present"
assert sub.Yield_Mg_ha.notna().all(), "no missing values allowed"
sub.to_csv("submission.csv", index=False)
print(f"wrote submission.csv: {len(sub):,} rows, "
      f"{sub.Yield_Mg_ha.min():.2f} to {sub.Yield_Mg_ha.max():.2f} Mg/ha")
print(f"template rows with no prediction, filled with the training mean: {unreached}")
sub.head()

Output:

wrote submission.csv: 10,057 rows, 0.00 to 15.04 Mg/ha
template rows with no prediction, filled with the training mean: 0
         Env        Hybrid  Yield_Mg_ha
0  DEH1_2024  01CSI6/LH287      10.3515
1  DEH1_2024  01DIB2/LH287      10.5950
2  DEH1_2024  01DIB2/PHP02       9.0055
3  DEH1_2024  2369/LH123HT      11.1120
4  DEH1_2024   2FACC/PHP02       9.9769

Where to gain

The blend sits a little above 0.32 against the 2024 answer key, with the published top-5 ensemble at 0.369. Each idea below is cheap to test here, because you can score locally instead of spending one of your five submissions.

environments by their covariate PCs and fit combining ability within each group. This is the genotype-by-environment signal the metric actually rewards.

GBLUP. The PCs add only about +0.011 over parent effects here, so the marker block is underused.

81% of template rows have only one parent seen in training, so the pair term is thin.

covariates are summaries, and growth-stage windows may hold more.

(Test_Set_Observed_Values_ANSWER.csv). Two independent held-out years is a far better guard against overfitting one season than any single split.

Data is the Genomes to Fields public release, ODC PDDL (public domain). Cite the G2F DOIs if you publish.

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.