SciganticGuidesTopicsExplore

How to open Tahoe-100M single-cell perturbation data in a Jupyter notebook

Tahoe-100M is 3,388 Parquet files of about 28,000 cells each, and pandas reads one with pd.read_parquet. In a Scigantic notebook the Hugging Face repository tahoebio/Tahoe-100M is mounted at /mnt/archive, so you can read a single shard and its metadata tables without downloading the full atlas.

The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is the Hugging Face dataset tahoebio/Tahoe-100M (public).

Open Tahoe-100M on SciganticOpen the sample notebook: Tahoe-100M vs. SEA-AD: Drugs Against Alzheimer's

Code

Read one shard, build an AnnData and plot a UMAP

# Start here: 28,000 drug-treated cancer cells from one file of Tahoe-100M,
# read off the mount and laid out by expression similarity.
import os, time, warnings
import numpy as np
import pandas as pd
import scipy.sparse as sp
import scanpy as sc
import matplotlib.pyplot as plt

warnings.filterwarnings("ignore")
%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 75}}
plt.rcParams["figure.dpi"] = 80

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SHARD = "data/train-00000-of-03388.parquet"      # any of the 3,388 works

t = time.time()
cells = pd.read_parquet(os.path.join(ROOT, SHARD))
genes = pd.read_parquet(os.path.join(ROOT, "metadata/gene_metadata.parquet"))
lines = pd.read_parquet(os.path.join(ROOT, "metadata/cell_line_metadata.parquet")).drop_duplicates("Cell_ID_Cellosaur")
print(f"read {len(cells):,} cells in {time.time() - t:.1f} s")

# Each cell stores gene token ids and counts. The first entry is a -2 marker, not a gene.
lengths = cells["genes"].str.len().to_numpy()
cols = np.concatenate(cells["genes"].to_numpy())
vals = np.concatenate(cells["expressions"].to_numpy())
rows = np.repeat(np.arange(len(cells)), lengths)
keep = vals > 0
X = sp.csr_matrix((vals[keep], (rows[keep], cols[keep])), shape=(len(cells), int(genes.token_id.max()) + 1))
X = X[:, genes.token_id.to_numpy()]

adata = sc.AnnData(X.astype(np.float32), obs=cells[["drug", "cell_line_id", "plate"]].reset_index(drop=True))
adata.var_names = genes.gene_symbol.to_numpy()
adata.var_names_make_unique()
names = lines.set_index("Cell_ID_Cellosaur")
adata.obs["cell_line"] = adata.obs.cell_line_id.map(names.cell_name).fillna(adata.obs.cell_line_id)
adata.obs["organ"] = adata.obs.cell_line_id.map(names.Organ).fillna("unknown")

sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, subset=True)
sc.pp.pca(adata, n_comps=30)
# transformer="sklearn": in this image scanpy's default nearest-neighbour search
# (pynndescent) fails on real data with "'csr_array' object has no attribute 'getnnz'".
sc.pp.neighbors(adata, n_neighbors=15, transformer="sklearn")
sc.tl.umap(adata)

fig, ax = plt.subplots(figsize=(9, 6))
sc.pl.umap(adata, color="organ", ax=ax, show=False, frameon=False, s=4,
           title=f"{len(adata):,} cells from {SHARD.split('/')[-1]}, coloured by the cell line's organ")
plt.tight_layout(); plt.show()
print(f"{adata.obs.cell_line.nunique()} cell lines and {adata.obs.drug.nunique()} drugs in this one file; "
      f"median {int(np.median(lengths - 1)):,} genes detected per cell")

Copied from this dataset's starter notebook on Scigantic, the one that opens when you start a notebook from the archive page (one cell, unchanged).

What the files are

Tahoe-100M is a single-cell drug perturbation atlas: 50 cancer cell lines treated with 379 compounds at three concentrations across 14 plates, 100.6 million cells in total.

Each row is one cell. Expression is stored as two parallel lists, genes (token IDs) and expressions (counts), next to the drug, cell line and plate. The first entry of each list is a marker, not a gene, which the code handles by keeping only positive counts.

How the files are laid out

Path under the mountWhat it holds
data/train-NNNNN-of-03388.parquetThe cells, about 28,000 and 71 MB per file. Columns include genes, expressions, drug, cell_line_id, plate and sample.
metadata/gene_metadata.parquetToken ID to gene symbol and Ensembl ID.
metadata/drug_metadata.parquetTargets, mechanism of action, approval status and SMILES for each drug.
metadata/cell_line_metadata.parquetOrgan and driver mutations per cell line, one row per mutation.
metadata/sample_metadata.parquetOne row per well: drug, dose, plate and mean QC counts.
metadata/pseudobulk_differential_expression/Precomputed differential expression per drug, cell line, dose and plate, in 1,026 Parquet files.

Size, format and license

FormatParquet
Size4,427 files; 3,388 of them hold the cells at about 71 MB each
LicenseCC0 1.0
ProviderTahoe Therapeutics (listed on the dataset as Vevo Therapeutics) and the Arc Institute
Public sourcethe Hugging Face dataset tahoebio/Tahoe-100M (public)
Notebook imageGenomics (scanpy, anndata and pyarrow preinstalled)

Things to know before you read it

Run it

Start a notebook from the Tahoe-100M archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first. For a longer worked example, open the sample notebook Tahoe-100M vs. SEA-AD: Drugs Against Alzheimer's.

Related guides

Last checked 2026-11-21. All dataset guides.