Tahoe-100M: sample notebook

A worked notebook for Tahoe-100M, with 3 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Vevo Therapeutics / Arc Institute
License
cc0-1.0
Size
4,427 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/tahoebio/Tahoe-100M)
Access
Free and public. No account with the data provider is needed.

The notebook

# Start here: 28,000 drug-treated cancer cells from one file of Tahoe-100M,
# read off the mount and laid out by expression similarity.
import os, time, warnings
import numpy as np
import pandas as pd
import scipy.sparse as sp
import scanpy as sc
import matplotlib.pyplot as plt

warnings.filterwarnings("ignore")
%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 75}}
plt.rcParams["figure.dpi"] = 80

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SHARD = "data/train-00000-of-03388.parquet"      # any of the 3,388 works

t = time.time()
cells = pd.read_parquet(os.path.join(ROOT, SHARD))
genes = pd.read_parquet(os.path.join(ROOT, "metadata/gene_metadata.parquet"))
lines = pd.read_parquet(os.path.join(ROOT, "metadata/cell_line_metadata.parquet")).drop_duplicates("Cell_ID_Cellosaur")
print(f"read {len(cells):,} cells in {time.time() - t:.1f} s")

# Each cell stores gene token ids and counts. The first entry is a -2 marker, not a gene.
lengths = cells["genes"].str.len().to_numpy()
cols = np.concatenate(cells["genes"].to_numpy())
vals = np.concatenate(cells["expressions"].to_numpy())
rows = np.repeat(np.arange(len(cells)), lengths)
keep = vals > 0
X = sp.csr_matrix((vals[keep], (rows[keep], cols[keep])), shape=(len(cells), int(genes.token_id.max()) + 1))
X = X[:, genes.token_id.to_numpy()]

adata = sc.AnnData(X.astype(np.float32), obs=cells[["drug", "cell_line_id", "plate"]].reset_index(drop=True))
adata.var_names = genes.gene_symbol.to_numpy()
adata.var_names_make_unique()
names = lines.set_index("Cell_ID_Cellosaur")
adata.obs["cell_line"] = adata.obs.cell_line_id.map(names.cell_name).fillna(adata.obs.cell_line_id)
adata.obs["organ"] = adata.obs.cell_line_id.map(names.Organ).fillna("unknown")

sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, subset=True)
sc.pp.pca(adata, n_comps=30)
# transformer="sklearn": in this image scanpy's default nearest-neighbour search
# (pynndescent) fails on real data with "'csr_array' object has no attribute 'getnnz'".
sc.pp.neighbors(adata, n_neighbors=15, transformer="sklearn")
sc.tl.umap(adata)

fig, ax = plt.subplots(figsize=(9, 6))
sc.pl.umap(adata, color="organ", ax=ax, show=False, frameon=False, s=4,
           title=f"{len(adata):,} cells from {SHARD.split('/')[-1]}, coloured by the cell line's organ")
plt.tight_layout(); plt.show()
print(f"{adata.obs.cell_line.nunique()} cell lines and {adata.obs.drug.nunique()} drugs in this one file; "
      f"median {int(np.median(lengths - 1)):,} genes detected per cell")

Tahoe-100M: 100 million cells, 50 cancer lines, 1,100 drug treatments

The map above is one file out of 3,388. Each island is a cell line. Cancer lines differ from each other far more than a drug changes any one of them, so cells sort by line first, and drug effects are structure inside an island. Finding that structure is what Tahoe-100M is for.

Tahoe Therapeutics pooled 50 cancer cell lines, treated them with 379 compounds at three concentrations across 14 plates, and sequenced 100.6 million single cells. The repository is mounted read-only at /mnt/archive:

PathWhat it holds
data/train-NNNNN-of-03388.parquetthe cells, about 28,000 and 71 MB per file. Each row has genes (token ids), expressions (counts), drug, cell_line_id, plate, sample
metadata/gene_metadata.parquettoken id to gene symbol and Ensembl id
metadata/drug_metadata.parquettargets, mechanism of action, approval status, SMILES
metadata/cell_line_metadata.parquetorgan and driver mutations per line (one row per mutation)
metadata/sample_metadata.parquetone row per well: drug, dose, plate and mean QC counts
metadata/pseudobulk_differential_expression/DESeq2 results for every drug against DMSO, per line and plate

The whole cell table is about 240 GB. Read one file at a time, or filter with pyarrow, rather than loading it all.

Which drugs change cells the most, before reading any cells

sample_metadata summarises each well. Comparing a well's mean genes per cell with the DMSO wells on the same plate is a quick screen. A drop usually means stressed or dying cells. A rise is common too and has other causes, for example cells that stop dividing but keep growing, so treat this as a pointer to where to look, not a toxicity measure.

samples = pd.read_parquet(os.path.join(ROOT, "metadata/sample_metadata.parquet"))
drugs = pd.read_parquet(os.path.join(ROOT, "metadata/drug_metadata.parquet"))
samples["dose_uM"] = samples.drugname_drugconc.str.extract(r",\s*([0-9.]+),")[0].astype(float)
dmso = samples[samples.drug.str.startswith("DMSO")].groupby("plate").mean_gene_count.mean()
samples["genes_vs_dmso"] = samples.mean_gene_count / samples.plate.map(dmso)
treated = samples[~samples.drug.str.startswith("DMSO")]
print(f"{len(samples):,} wells on {samples.plate.nunique()} plates, {treated.drug.nunique()} drugs at doses "
      f"{', '.join(f'{d:g}' for d in sorted(treated.dose_uM.dropna().unique()))} uM")
print("\nlargest drops at the top dose:")

high = treated[treated.dose_uM == treated.dose_uM.max()].groupby("drug").genes_vs_dmso.mean().sort_values()
worst = high.head(12).rename("genes_vs_dmso").reset_index().merge(drugs[["drug", "moa-fine", "targets"]], on="drug", how="left")
print(worst.to_string(index=False, float_format=lambda v: f"{v:.2f}", max_colwidth=38))

fig, ax = plt.subplots(figsize=(8, 3))
ax.hist(high, bins=60, color="#1d4ed8")
ax.axvline(1, color="k", lw=1)
ax.set_xlabel(f"mean genes per cell at {treated.dose_uM.max():g} uM, relative to DMSO on the same plate")
ax.set_ylabel("drugs")
plt.tight_layout(); plt.show()

Output:

1,344 wells on 14 plates, 379 drugs at doses 0.05, 0.5, 5 uM

largest drops at the top dose:
                       drug  genes_vs_dmso                 moa-fine                targets
                  Digitoxin           0.60                  unclear                   None
                 Dinaciclib           0.65            CDK inhibitor CDK1, CDK2, CDK5, CDK9
                  Sunitinib           0.73       Multi-TK inhibitor            KDR, PDGFRB
  Imiquimod (hydrochloride)           0.74                  unclear                   TLR7
                 Saquinavir           0.77                  unclear                   None
                 Nimesulide           0.77 Cyclooxygenase inhibitor                  PTGS2
             Pentoxifylline           0.77                  unclear                   None
       Atazanavir (sulfate)           0.78                  unclear                 CYP3A4
Medroxyprogesterone acetate           0.78                  unclear                    PGR
                 Malotilate           0.79                  unclear                  ALOX5
                   Pimozide           0.79                  unclear             DRD2, DRD3
...

A drug's effect on one cell line, already computed

Tahoe ships pseudobulk differential expression: for each drug, cell line, dose and plate, the treated cells summed and compared with DMSO by DESeq2. Each of the 1,026 files holds about 4 million rows; the first is one cell line (A549, lung) on plate 1, 64 drugs at 0.05 uM.

Check n_cells_trt before trusting a result. Ranked by genes changed, the top three drugs in this file each have one or two treated cells against nearly 5,000 controls. Their thousands of "significant" genes are noise from a comparison DESeq2 should not have been asked to make. The cell below shows them, then keeps only comparisons with at least 100 treated cells and draws the top one.

t = time.time()
de = pd.read_parquet(os.path.join(ROOT, "metadata/pseudobulk_differential_expression/train-00000-of-01026.parquet"),
                     columns=["gene_name", "log2FoldChange", "padj", "drug", "concentration", "Cell_Name_Vevo", "n_cells_trt", "n_cells_ctrl"])
print(f"read {len(de):,} rows in {time.time() - t:.1f} s: {de.Cell_Name_Vevo.unique().tolist()}, {de.drug.nunique()} drugs")

de = de.dropna(subset=["padj", "log2FoldChange"])
per_drug = de.groupby(["drug", "concentration"]).agg(treated_cells=("n_cells_trt", "first"), control_cells=("n_cells_ctrl", "first"),
                                                     genes_changed=("padj", lambda p: int((p < 0.05).sum())))
per_drug = per_drug.sort_values("genes_changed", ascending=False)
print("\nranked by genes changed (padj < 0.05), no filter:")
print(per_drug.head(5).to_string())
trusted = per_drug[per_drug.treated_cells >= 100]
print(f"\nwith at least 100 treated cells ({len(trusted)} of {len(per_drug)} drugs):")
print(trusted.head(5).to_string())

(drug, conc) = trusted.index[0]
one = de[(de.drug == drug) & (de.concentration == conc)].copy()
# padj is float32 and DESeq2 reports the strongest genes as exactly 0, so
# -log10 gives infinity and matplotlib silently drops the most significant
# points. Work in float64 and cap the axis instead.
one["nlp"] = np.minimum(-np.log10(one.padj.astype("float64").clip(lower=1e-300)), 50)
print(f"{int((one.padj == 0).sum()):,} genes have padj reported as exactly 0; drawn at the cap of 50")
sig = one.padj < 0.05
fig, ax = plt.subplots(figsize=(7, 5))
ax.scatter(one.log2FoldChange[~sig], one.nlp[~sig], s=3, c="#a3a3a3", rasterized=True)
ax.scatter(one.log2FoldChange[sig], one.nlp[sig], s=4, c="#1d4ed8", rasterized=True)
strongest = one[sig].sort_values(["nlp", "log2FoldChange"], ascending=[False, False])
up = strongest[(strongest.nlp >= 50) & (strongest.log2FoldChange > 0)].nlargest(6, "log2FoldChange")
down = strongest[(strongest.nlp >= 50) & (strongest.log2FoldChange < 0)].nsmallest(4, "log2FoldChange")
for k, (_, r) in enumerate(pd.concat([up, down]).iterrows()):
    ax.annotate(r.gene_name, (r.log2FoldChange, r.nlp), fontsize=8, xytext=(0, -12 - 11 * (k % 3)), textcoords="offset points",
                ha="center", arrowprops={"arrowstyle": "-", "color": "#737373", "lw": 0.5})
print("most raised at the cap:", ", ".join(f"{g} ({v:+.1f})" for g, v in zip(up.gene_name, up.log2FoldChange)))
print("most lowered at the cap:", ", ".join(f"{g} ({v:+.1f})" for g, v in zip(down.gene_name, down.log2FoldChange)))
ax.set_xlabel("log2 fold change vs DMSO"); ax.set_ylabel("-log10 adjusted p (capped at 50)")
ax.set_title(f"{drug} {conc:g} uM on A549: {int(sig.sum()):,} genes changed")
plt.tight_layout(); plt.show()

Output:

read 3,986,181 rows in 0.8 s: ['A549'], 64 drugs

ranked by genes changed (padj < 0.05), no filter:
                                         treated_cells  control_cells  genes_changed
drug                      concentration                                             
PF-06260933               0.05                       1           4862          13012
Abemaciclib               0.05                       1           4862          11677
DTP3                      0.05                       2           4862           9983
Bortezomib                0.05                    1069           4862           5910
Elimusertib hydrochloride 0.05                    1473           4862           2672

with at least 100 treated cells (59 of 64 drugs):
                                         treated_cells  control_cells  genes_changed
drug                      concentration                                             
Bortezomib                0.05                    1069           4862           5910
...

The genes Bortezomib raises most are heat-shock chaperones (HSPA1A, HSPA1B, HSPH1, HSP90AA1, DNAJB1). Bortezomib blocks the proteasome, misfolded proteins build up, and this is the cell's response to them, which is a good sign the filtered comparison is measuring biology.

Where to go from here

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.