Virtual Embryo Challenge Tutorial: sample notebook

A worked notebook for Virtual Embryo Challenge Tutorial, with 10 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Qiu Lab & Dr. Neil Chi's group
License
MIT
Access
Free and public. No account with the data provider is needed.

The notebook

Start here. The Challenge data cannot be hosted here (you bring your own copy in step 1 below), but the scorer this image ships can run right now on the two tiny 150-cell samples from the veckit repository. Click the grey cell below and press Shift+Enter: it downloads about 13 MB, scores the simplest possible Task 1 prediction (copy the E8.5 cells and call them E9.5) and prints the metrics the leaderboard uses. Expect de_score 0.0: that is the floor you have to beat.

# Start here: click this cell, press Shift+Enter. A few seconds, no Challenge data needed.
import urllib.request
from pathlib import Path
from veckit import score

SAMPLES = Path("data/veckit-samples")
SAMPLES.mkdir(parents=True, exist_ok=True)
# jsDelivr fronts the veckit GitHub repo on a CDN; raw.githubusercontent.com is the fallback.
SOURCES = ("https://cdn.jsdelivr.net/gh/aristoteleo/veckit@main/data/",
           "https://raw.githubusercontent.com/aristoteleo/veckit/main/data/")
for name in ("sample_8.5.h5ad", "sample_9.5.h5ad"):
    if (SAMPLES / name).exists():
        continue
    for base in SOURCES:
        try:
            urllib.request.urlretrieve(base + name, SAMPLES / name)
            print("fetched", name)
            break
        except Exception as err:  # try the next mirror
            print("retrying", name, "after", type(err).__name__)

# The floor: predict that E9.5 looks exactly like E8.5 ("copy_last").
floor = score(task="T1", input=str(SAMPLES / "sample_8.5.h5ad"),
              target=str(SAMPLES / "sample_9.5.h5ad"), reference=str(SAMPLES / "sample_8.5.h5ad"))

print(f"{'metric':<20}{'copy_last floor':>16}")
for k in ("de_score", "de_direction", "mmd_u", "variogram", "composition_JSD", "pseudobulk_pearson"):
    print(f"{k:<20}{floor['metrics'][k]:>16.4f}")
print("\nAnything above de_score 0 beats this. The rest of the notebook builds that step with the real data.")

Task 1: Your First Virtual Embryo Challenge Submission

This image bakes in veckit, the Virtual Embryo Challenge's own MIT-licensed offline scorer, plus anndata for working with the Challenge's .h5ad files. The official veckit tutorial shows you how to *score* a prediction once you have one, using tiny bundled samples. It deliberately stops at "swap --input for your own model's prediction." This notebook picks up there, for Task 1 (temporal scRNA-seq extrapolation): building an actual first prediction (a floor baseline and one honest step above it), then packaging and locally sanity-checking a submission before you spend a submission slot on it.

What Task 1 asks: predict the gene-expression distribution of mouse embryo cells at a developmental stage you have not observed, given earlier stages. Train stages are E8.5 and E9.5; the public validation leaderboard scores against E10.5; the hidden test ranking uses E12.5. Nothing is observed between E10.5 and E12.5, so this is genuine extrapolation, not interpolation. Full task definition: virtualembryo.ai/challenge/tasks.

This image contains no Challenge data. Per the Challenge Terms of Use (clause 14), the released datasets are unpublished and Scigantic cannot host, mirror, or redistribute them. You bring your own copy. See the next cell.

1. Get your data into this notebook

virtualembryo.ai/challenge/data.

upload button). They land in *this* private session's storage; Scigantic doesn't see or retain a separate copy of them.

Both files are AnnData .h5ad containers: 32,285 genes, log1p-normalized expression in .X, cell-type annotations in obs['celltype'], no spatial coordinates for Task 1.

2. Check the environment

import anndata as ad
import numpy as np
import scipy
import veckit

print("anndata", ad.__version__, "| scipy", scipy.__version__, "| veckit", veckit.__file__.split("/")[-1])
import json
import urllib.request
from pathlib import Path

from scipy import sparse

DATA = Path("data")


def fetch(url: str) -> bytes:
    # virtualembryo.ai 403s a bare urllib request with no User-Agent
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    return urllib.request.urlopen(req).read()


def dense(a: ad.AnnData) -> np.ndarray:
    return a.X.toarray() if sparse.issparse(a.X) else np.asarray(a.X)


a85 = ad.read_h5ad(DATA / "E8.5_RNA.h5ad")
a95 = ad.read_h5ad(DATA / "E9.5_RNA.h5ad")

print("E8.5:", a85.shape)
print("E9.5:", a95.shape)
print("same gene panel, same order:", list(a85.var_names) == list(a95.var_names))

3. Why this is hard

This is not a small population drift. Compare cell-type composition between the two training stages you actually have:

comp85 = a85.obs["celltype"].value_counts(normalize=True).mul(100).round(2)
comp95 = a95.obs["celltype"].value_counts(normalize=True).mul(100).round(2)
shared = sorted(set(comp85.index) & set(comp95.index))

print(f"E8.5: {a85.n_obs} cells, {comp85.shape[0]} cell types")
print(f"E9.5: {a95.n_obs} cells, {comp95.shape[0]} cell types")
print(f"shared cell types: {len(shared)} -> {shared}")
print()
print("cell types in E9.5 that did not exist (by this label) at E8.5:")
print(sorted(set(comp95.index) - set(comp85.index)))

Roughly half the cell-type labels at E9.5 are not present at E8.5. A model that only nudges existing cells around cannot account for that: some of what it needs to predict is cell-fate change, not just expression drift within a fixed population. Keep this in mind: the baseline below is deliberately simple and does not attempt to model this.

4. Baseline 0: the floor, copy_last

The simplest possible prediction: reuse the most recent real observation verbatim, unchanged. This is the official floor baseline ("floor_model": "copy_last" in the Task 1 panel spec). Every real model should beat it.

pred_floor = a85.copy()
pred_floor.write_h5ad(DATA / "pred_floor.h5ad")

5. Baseline 1: pseudobulk mean-shift extrapolation

One honest step up: measure the average per-gene shift between the two stages you have (E9.5 pseudobulk mean minus E8.5 pseudobulk mean, in the same log1p space the data already ships in), then apply that same shift to every cell. This captures *directional* change, which copy_last cannot, while staying simple enough to fully understand and audit.

It will not invent new cell types (see the composition gap above), and translating every cell by the same vector does not change the gene-gene covariance structure at all, so don't expect it to help metrics that specifically reward correct covariance. Watch for both effects in the scores below.

X85 = dense(a85)
X95 = dense(a95)

delta = X95.mean(axis=0) - X85.mean(axis=0)  # observed E8.5 -> E9.5 shift, one gene at a time

X_shift = np.clip(X85 + delta, 0, None).astype(np.float32)  # metrics require nonnegative log-normalized data
pred_shift = a85.copy()
pred_shift.X = X_shift
pred_shift.write_h5ad(DATA / "pred_shift.h5ad")

6. Scoring locally, honestly

The real E10.5 target is held out. You don't have it, and can't. So, exactly like veckit's own tutorial does, use a pseudo target: pretend E9.5 is the unknown stage, using the real E8.5-to-E9.5 pair you do have ground truth for. This tells you whether your *mechanism* works, not your real leaderboard score. The actual E9.5-to-E10.5 shift you'll extrapolate for the real submission is a different, unknown one, so don't over-trust these exact numbers.

from veckit import score

r_floor = score(task="T1", input=str(DATA / "pred_floor.h5ad"),
                 target=str(DATA / "E9.5_RNA.h5ad"), reference=str(DATA / "E8.5_RNA.h5ad"))
r_shift = score(task="T1", input=str(DATA / "pred_shift.h5ad"),
                 target=str(DATA / "E9.5_RNA.h5ad"), reference=str(DATA / "E8.5_RNA.h5ad"))

primary = ["de_score", "de_direction", "mmd_u", "variogram"]  # the metrics T1:val's official anchors normalize
print(f"{'metric':<14}{'floor (copy_last)':<20}{'mean-shift':<12}")
for k in primary:
    print(f"{k:<14}{r_floor['metrics'][k]:<20}{r_shift['metrics'][k]:<12}")

de_score and de_direction are 0 for the floor by construction. copy_last predicts no change at all, so there is no direction to get right or wrong. On a 150-cell toy calibration run using veckit's own public sample files (not real Challenge data, just to sanity-check this exact code), mean-shift took de_score from 0.0 to 0.58 and de_direction from 0.0 to 0.998, while variogram got slightly worse, consistent with the mechanism above: it fixes direction, it does nothing for covariance. Your real numbers will differ; what matters is the same pattern of trade-offs, not these exact figures.

The rest of what score() returns (energy_distance, pb_rel_err, library_size_ratio, variance_ratio, composition_JSD, pseudobulk_pearson) are useful diagnostics but aren't part of the four metrics the official T1:val anchors normalize. See virtualembryo.ai/challenge/evaluation.

7. Building the real submission: E9.5 to E10.5

Apply the *same* learned mechanism one stage forward, where you don't have ground truth: shift the real E9.5 cells by the same kind of delta, this time measured off the E8.5-to-E9.5 pair, to predict E10.5. Then shape the result to match the live Task 1 validation panel spec: gene order and a cell count the scorer will accept.

spec = json.loads(fetch("https://virtualembryo.ai/challenge/panels/index.json"))["T1:val"]
panel_genes = fetch(
    "https://virtualembryo.ai/challenge/panels/" + spec["genes_file"]
).decode().splitlines()

print(spec["label"], "-", spec["n_genes"], "genes, cells in [", spec["min_cells"], ",", spec["max_cells"], "]")
print("panel matches your files' gene order already:", panel_genes == list(a95.var_names))
X_next = np.clip(X95 + delta, 0, None).astype(np.float32)

# E9.5 has ~17k cells; the panel wants between min_cells and max_cells. Subsample down if needed.
n_target = min(spec["max_cells"], max(spec["min_cells"], a95.n_obs))
rng = np.random.default_rng(0)
idx = rng.choice(a95.n_obs, size=n_target, replace=False) if a95.n_obs > n_target else np.arange(a95.n_obs)

submission = ad.AnnData(X=X_next[idx], var=a95.var.copy(), obs=a95.obs.iloc[idx].copy())
if panel_genes != list(submission.var_names):
    submission = submission[:, panel_genes].copy()  # only triggers if the panel ever reorders

submission.write_h5ad(DATA / "submission_T1_val.h5ad")
print("wrote", DATA / "submission_T1_val.h5ad", submission.shape)

8. Format sanity check

Mirrors the same checks veckit's own score_h5ad.py runs on every file it loads: gene panel order, finite nonnegative values, cell count in range. This cannot tell you how good the prediction is (no ground truth exists for E10.5 outside the Challenge's own servers). It only confirms the file is shaped correctly.

check = ad.read_h5ad(DATA / "submission_T1_val.h5ad")
Xc = dense(check)

assert list(check.var_names) == panel_genes, "gene panel / order mismatch"
assert np.isfinite(Xc).all(), "non-finite values present"
assert (Xc >= 0).all(), "negative values present -- metrics expect log-normalized nonnegative data"
assert spec["min_cells"] <= check.n_obs <= spec["max_cells"], f"cell count {check.n_obs} outside allowed range"

print("all format checks passed:", check.shape)

9. Download your submission and submit it

Right-click data/submission_T1_val.h5ad in the file browser and choose Download to pull it out of this session onto your own machine. Then sign in at virtualembryo.ai/challenge/account/submissions and upload it there. Check virtualembryo.ai/challenge/rules for the current submission cadence limit before you use one.

A note on this session's storage: files here live in your own private notebook session. Scigantic does not host or retain a separate copy of anything you uploaded or produced. Download anything you want to keep before the session ends.

10. Where to go from here

This baseline moves the whole population by one shared vector. It cannot invent the cell types missing from the composition gap above, and it ignores gene-gene covariance entirely. Ideas that go further, roughly in order of effort:

the shared cell types, and handle emerging types some other way (e.g. via the frozen probe veckit already trains on the target's cell types).

gene space.

virtualembryo.ai/challenge/baselines.

Acknowledgments

Data, task design, and the veckit scorer are the work of Dr. Neil Chi's group and the Qiu Lab (virtualembryo.ai, github.com/aristoteleo). This notebook only adds a worked example on top of their public tooling and data.

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.