Tahoe-100M: sample notebook
A worked notebook for Tahoe-100M, with 3 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Vevo Therapeutics / Arc Institute
- License
- cc0-1.0
- Size
- 4,427 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/tahoebio/Tahoe-100M)
- Access
- Free and public. No account with the data provider is needed.
The notebook
# Start here: 28,000 drug-treated cancer cells from one file of Tahoe-100M,
# read off the mount and laid out by expression similarity.
import os, time, warnings
import numpy as np
import pandas as pd
import scipy.sparse as sp
import scanpy as sc
import matplotlib.pyplot as plt
warnings.filterwarnings("ignore")
%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 75}}
plt.rcParams["figure.dpi"] = 80
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SHARD = "data/train-00000-of-03388.parquet" # any of the 3,388 works
t = time.time()
cells = pd.read_parquet(os.path.join(ROOT, SHARD))
genes = pd.read_parquet(os.path.join(ROOT, "metadata/gene_metadata.parquet"))
lines = pd.read_parquet(os.path.join(ROOT, "metadata/cell_line_metadata.parquet")).drop_duplicates("Cell_ID_Cellosaur")
print(f"read {len(cells):,} cells in {time.time() - t:.1f} s")
# Each cell stores gene token ids and counts. The first entry is a -2 marker, not a gene.
lengths = cells["genes"].str.len().to_numpy()
cols = np.concatenate(cells["genes"].to_numpy())
vals = np.concatenate(cells["expressions"].to_numpy())
rows = np.repeat(np.arange(len(cells)), lengths)
keep = vals > 0
X = sp.csr_matrix((vals[keep], (rows[keep], cols[keep])), shape=(len(cells), int(genes.token_id.max()) + 1))
X = X[:, genes.token_id.to_numpy()]
adata = sc.AnnData(X.astype(np.float32), obs=cells[["drug", "cell_line_id", "plate"]].reset_index(drop=True))
adata.var_names = genes.gene_symbol.to_numpy()
adata.var_names_make_unique()
names = lines.set_index("Cell_ID_Cellosaur")
adata.obs["cell_line"] = adata.obs.cell_line_id.map(names.cell_name).fillna(adata.obs.cell_line_id)
adata.obs["organ"] = adata.obs.cell_line_id.map(names.Organ).fillna("unknown")
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, subset=True)
sc.pp.pca(adata, n_comps=30)
# transformer="sklearn": in this image scanpy's default nearest-neighbour search
# (pynndescent) fails on real data with "'csr_array' object has no attribute 'getnnz'".
sc.pp.neighbors(adata, n_neighbors=15, transformer="sklearn")
sc.tl.umap(adata)
fig, ax = plt.subplots(figsize=(9, 6))
sc.pl.umap(adata, color="organ", ax=ax, show=False, frameon=False, s=4,
title=f"{len(adata):,} cells from {SHARD.split('/')[-1]}, coloured by the cell line's organ")
plt.tight_layout(); plt.show()
print(f"{adata.obs.cell_line.nunique()} cell lines and {adata.obs.drug.nunique()} drugs in this one file; "
f"median {int(np.median(lengths - 1)):,} genes detected per cell")
Tahoe-100M: 100 million cells, 50 cancer lines, 1,100 drug treatments
The map above is one file out of 3,388. Each island is a cell line. Cancer lines differ from each other far more than a drug changes any one of them, so cells sort by line first, and drug effects are structure inside an island. Finding that structure is what Tahoe-100M is for.
Tahoe Therapeutics pooled 50 cancer cell lines, treated them with 379 compounds at three concentrations across 14 plates, and sequenced 100.6 million single cells. The repository is mounted read-only at /mnt/archive:
| Path | What it holds |
| data/train-NNNNN-of-03388.parquet | the cells, about 28,000 and 71 MB per file. Each row has genes (token ids), expressions (counts), drug, cell_line_id, plate, sample |
| metadata/gene_metadata.parquet | token id to gene symbol and Ensembl id |
| metadata/drug_metadata.parquet | targets, mechanism of action, approval status, SMILES |
| metadata/cell_line_metadata.parquet | organ and driver mutations per line (one row per mutation) |
| metadata/sample_metadata.parquet | one row per well: drug, dose, plate and mean QC counts |
| metadata/pseudobulk_differential_expression/ | DESeq2 results for every drug against DMSO, per line and plate |
The whole cell table is about 240 GB. Read one file at a time, or filter with pyarrow, rather than loading it all.
Which drugs change cells the most, before reading any cells
sample_metadata summarises each well. Comparing a well's mean genes per cell with the DMSO wells on the same plate is a quick screen. A drop usually means stressed or dying cells. A rise is common too and has other causes, for example cells that stop dividing but keep growing, so treat this as a pointer to where to look, not a toxicity measure.
samples = pd.read_parquet(os.path.join(ROOT, "metadata/sample_metadata.parquet"))
drugs = pd.read_parquet(os.path.join(ROOT, "metadata/drug_metadata.parquet"))
samples["dose_uM"] = samples.drugname_drugconc.str.extract(r",\s*([0-9.]+),")[0].astype(float)
dmso = samples[samples.drug.str.startswith("DMSO")].groupby("plate").mean_gene_count.mean()
samples["genes_vs_dmso"] = samples.mean_gene_count / samples.plate.map(dmso)
treated = samples[~samples.drug.str.startswith("DMSO")]
print(f"{len(samples):,} wells on {samples.plate.nunique()} plates, {treated.drug.nunique()} drugs at doses "
f"{', '.join(f'{d:g}' for d in sorted(treated.dose_uM.dropna().unique()))} uM")
print("\nlargest drops at the top dose:")
high = treated[treated.dose_uM == treated.dose_uM.max()].groupby("drug").genes_vs_dmso.mean().sort_values()
worst = high.head(12).rename("genes_vs_dmso").reset_index().merge(drugs[["drug", "moa-fine", "targets"]], on="drug", how="left")
print(worst.to_string(index=False, float_format=lambda v: f"{v:.2f}", max_colwidth=38))
fig, ax = plt.subplots(figsize=(8, 3))
ax.hist(high, bins=60, color="#1d4ed8")
ax.axvline(1, color="k", lw=1)
ax.set_xlabel(f"mean genes per cell at {treated.dose_uM.max():g} uM, relative to DMSO on the same plate")
ax.set_ylabel("drugs")
plt.tight_layout(); plt.show()
Output:
1,344 wells on 14 plates, 379 drugs at doses 0.05, 0.5, 5 uM
largest drops at the top dose:
drug genes_vs_dmso moa-fine targets
Digitoxin 0.60 unclear None
Dinaciclib 0.65 CDK inhibitor CDK1, CDK2, CDK5, CDK9
Sunitinib 0.73 Multi-TK inhibitor KDR, PDGFRB
Imiquimod (hydrochloride) 0.74 unclear TLR7
Saquinavir 0.77 unclear None
Nimesulide 0.77 Cyclooxygenase inhibitor PTGS2
Pentoxifylline 0.77 unclear None
Atazanavir (sulfate) 0.78 unclear CYP3A4
Medroxyprogesterone acetate 0.78 unclear PGR
Malotilate 0.79 unclear ALOX5
Pimozide 0.79 unclear DRD2, DRD3
...
A drug's effect on one cell line, already computed
Tahoe ships pseudobulk differential expression: for each drug, cell line, dose and plate, the treated cells summed and compared with DMSO by DESeq2. Each of the 1,026 files holds about 4 million rows; the first is one cell line (A549, lung) on plate 1, 64 drugs at 0.05 uM.
Check n_cells_trt before trusting a result. Ranked by genes changed, the top three drugs in this file each have one or two treated cells against nearly 5,000 controls. Their thousands of "significant" genes are noise from a comparison DESeq2 should not have been asked to make. The cell below shows them, then keeps only comparisons with at least 100 treated cells and draws the top one.
t = time.time()
de = pd.read_parquet(os.path.join(ROOT, "metadata/pseudobulk_differential_expression/train-00000-of-01026.parquet"),
columns=["gene_name", "log2FoldChange", "padj", "drug", "concentration", "Cell_Name_Vevo", "n_cells_trt", "n_cells_ctrl"])
print(f"read {len(de):,} rows in {time.time() - t:.1f} s: {de.Cell_Name_Vevo.unique().tolist()}, {de.drug.nunique()} drugs")
de = de.dropna(subset=["padj", "log2FoldChange"])
per_drug = de.groupby(["drug", "concentration"]).agg(treated_cells=("n_cells_trt", "first"), control_cells=("n_cells_ctrl", "first"),
genes_changed=("padj", lambda p: int((p < 0.05).sum())))
per_drug = per_drug.sort_values("genes_changed", ascending=False)
print("\nranked by genes changed (padj < 0.05), no filter:")
print(per_drug.head(5).to_string())
trusted = per_drug[per_drug.treated_cells >= 100]
print(f"\nwith at least 100 treated cells ({len(trusted)} of {len(per_drug)} drugs):")
print(trusted.head(5).to_string())
(drug, conc) = trusted.index[0]
one = de[(de.drug == drug) & (de.concentration == conc)].copy()
# padj is float32 and DESeq2 reports the strongest genes as exactly 0, so
# -log10 gives infinity and matplotlib silently drops the most significant
# points. Work in float64 and cap the axis instead.
one["nlp"] = np.minimum(-np.log10(one.padj.astype("float64").clip(lower=1e-300)), 50)
print(f"{int((one.padj == 0).sum()):,} genes have padj reported as exactly 0; drawn at the cap of 50")
sig = one.padj < 0.05
fig, ax = plt.subplots(figsize=(7, 5))
ax.scatter(one.log2FoldChange[~sig], one.nlp[~sig], s=3, c="#a3a3a3", rasterized=True)
ax.scatter(one.log2FoldChange[sig], one.nlp[sig], s=4, c="#1d4ed8", rasterized=True)
strongest = one[sig].sort_values(["nlp", "log2FoldChange"], ascending=[False, False])
up = strongest[(strongest.nlp >= 50) & (strongest.log2FoldChange > 0)].nlargest(6, "log2FoldChange")
down = strongest[(strongest.nlp >= 50) & (strongest.log2FoldChange < 0)].nsmallest(4, "log2FoldChange")
for k, (_, r) in enumerate(pd.concat([up, down]).iterrows()):
ax.annotate(r.gene_name, (r.log2FoldChange, r.nlp), fontsize=8, xytext=(0, -12 - 11 * (k % 3)), textcoords="offset points",
ha="center", arrowprops={"arrowstyle": "-", "color": "#737373", "lw": 0.5})
print("most raised at the cap:", ", ".join(f"{g} ({v:+.1f})" for g, v in zip(up.gene_name, up.log2FoldChange)))
print("most lowered at the cap:", ", ".join(f"{g} ({v:+.1f})" for g, v in zip(down.gene_name, down.log2FoldChange)))
ax.set_xlabel("log2 fold change vs DMSO"); ax.set_ylabel("-log10 adjusted p (capped at 50)")
ax.set_title(f"{drug} {conc:g} uM on A549: {int(sig.sum()):,} genes changed")
plt.tight_layout(); plt.show()
Output:
read 3,986,181 rows in 0.8 s: ['A549'], 64 drugs
ranked by genes changed (padj < 0.05), no filter:
treated_cells control_cells genes_changed
drug concentration
PF-06260933 0.05 1 4862 13012
Abemaciclib 0.05 1 4862 11677
DTP3 0.05 2 4862 9983
Bortezomib 0.05 1069 4862 5910
Elimusertib hydrochloride 0.05 1473 4862 2672
with at least 100 treated cells (59 of 64 drugs):
treated_cells control_cells genes_changed
drug concentration
Bortezomib 0.05 1069 4862 5910
...
The genes Bortezomib raises most are heat-shock chaperones (HSPA1A, HSPA1B, HSPH1, HSP90AA1, DNAJB1). Bortezomib blocks the proteasome, misfolded proteins build up, and this is the cell's response to them, which is a good sign the filtered comparison is measuring biology.
Where to go from here
- More cells. Loop over a few data/ files, or use pyarrow.dataset with a filter on drug or cell_line_id to pull only the cells you need from many files.
- Every cell's labels. metadata/obs_metadata.parquet (2.3 GB) lists all 100 million cells with their drug, line and plate, so you can find which files hold the cells you want before reading any.
- A model's view. The genes token ids are the vocabulary Tahoe's own foundation models use (metadata/gene_vocabulary.json).
- Ask the assistant. For example: "From the pseudobulk differential expression files, find the genes that Bortezomib changes consistently across every cell line it was tested on."
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.