How to open Tahoe-100M single-cell perturbation data in a Jupyter notebook
Tahoe-100M is 3,388 Parquet files of about 28,000 cells each, and pandas reads one with pd.read_parquet. In a Scigantic notebook the Hugging Face repository tahoebio/Tahoe-100M is mounted at /mnt/archive, so you can read a single shard and its metadata tables without downloading the full atlas.
The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is the Hugging Face dataset tahoebio/Tahoe-100M (public).
Open Tahoe-100M on SciganticOpen the sample notebook: Tahoe-100M vs. SEA-AD: Drugs Against Alzheimer's
Code
Read one shard, build an AnnData and plot a UMAP
# Start here: 28,000 drug-treated cancer cells from one file of Tahoe-100M,
# read off the mount and laid out by expression similarity.
import os, time, warnings
import numpy as np
import pandas as pd
import scipy.sparse as sp
import scanpy as sc
import matplotlib.pyplot as plt
warnings.filterwarnings("ignore")
%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 75}}
plt.rcParams["figure.dpi"] = 80
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
SHARD = "data/train-00000-of-03388.parquet" # any of the 3,388 works
t = time.time()
cells = pd.read_parquet(os.path.join(ROOT, SHARD))
genes = pd.read_parquet(os.path.join(ROOT, "metadata/gene_metadata.parquet"))
lines = pd.read_parquet(os.path.join(ROOT, "metadata/cell_line_metadata.parquet")).drop_duplicates("Cell_ID_Cellosaur")
print(f"read {len(cells):,} cells in {time.time() - t:.1f} s")
# Each cell stores gene token ids and counts. The first entry is a -2 marker, not a gene.
lengths = cells["genes"].str.len().to_numpy()
cols = np.concatenate(cells["genes"].to_numpy())
vals = np.concatenate(cells["expressions"].to_numpy())
rows = np.repeat(np.arange(len(cells)), lengths)
keep = vals > 0
X = sp.csr_matrix((vals[keep], (rows[keep], cols[keep])), shape=(len(cells), int(genes.token_id.max()) + 1))
X = X[:, genes.token_id.to_numpy()]
adata = sc.AnnData(X.astype(np.float32), obs=cells[["drug", "cell_line_id", "plate"]].reset_index(drop=True))
adata.var_names = genes.gene_symbol.to_numpy()
adata.var_names_make_unique()
names = lines.set_index("Cell_ID_Cellosaur")
adata.obs["cell_line"] = adata.obs.cell_line_id.map(names.cell_name).fillna(adata.obs.cell_line_id)
adata.obs["organ"] = adata.obs.cell_line_id.map(names.Organ).fillna("unknown")
sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)
sc.pp.highly_variable_genes(adata, n_top_genes=2000, subset=True)
sc.pp.pca(adata, n_comps=30)
# transformer="sklearn": in this image scanpy's default nearest-neighbour search
# (pynndescent) fails on real data with "'csr_array' object has no attribute 'getnnz'".
sc.pp.neighbors(adata, n_neighbors=15, transformer="sklearn")
sc.tl.umap(adata)
fig, ax = plt.subplots(figsize=(9, 6))
sc.pl.umap(adata, color="organ", ax=ax, show=False, frameon=False, s=4,
title=f"{len(adata):,} cells from {SHARD.split('/')[-1]}, coloured by the cell line's organ")
plt.tight_layout(); plt.show()
print(f"{adata.obs.cell_line.nunique()} cell lines and {adata.obs.drug.nunique()} drugs in this one file; "
f"median {int(np.median(lengths - 1)):,} genes detected per cell")
Copied from this dataset's starter notebook on Scigantic, the one that opens when you start a notebook from the archive page (one cell, unchanged).
What the files are
Tahoe-100M is a single-cell drug perturbation atlas: 50 cancer cell lines treated with 379 compounds at three concentrations across 14 plates, 100.6 million cells in total.
Each row is one cell. Expression is stored as two parallel lists, genes (token IDs) and expressions (counts), next to the drug, cell line and plate. The first entry of each list is a marker, not a gene, which the code handles by keeping only positive counts.
How the files are laid out
| Path under the mount | What it holds |
|---|---|
| data/train-NNNNN-of-03388.parquet | The cells, about 28,000 and 71 MB per file. Columns include genes, expressions, drug, cell_line_id, plate and sample. |
| metadata/gene_metadata.parquet | Token ID to gene symbol and Ensembl ID. |
| metadata/drug_metadata.parquet | Targets, mechanism of action, approval status and SMILES for each drug. |
| metadata/cell_line_metadata.parquet | Organ and driver mutations per cell line, one row per mutation. |
| metadata/sample_metadata.parquet | One row per well: drug, dose, plate and mean QC counts. |
| metadata/pseudobulk_differential_expression/ | Precomputed differential expression per drug, cell line, dose and plate, in 1,026 Parquet files. |
Size, format and license
| Format | Parquet |
|---|---|
| Size | 4,427 files; 3,388 of them hold the cells at about 71 MB each |
| License | CC0 1.0 |
| Provider | Tahoe Therapeutics (listed on the dataset as Vevo Therapeutics) and the Arc Institute |
| Public source | the Hugging Face dataset tahoebio/Tahoe-100M (public) |
| Notebook image | Genomics (scanpy, anndata and pyarrow preinstalled) |
Things to know before you read it
- Read one shard at a time. A single file becomes about 28,000 cells in memory; the full atlas does not fit in a notebook.
- The code passes
transformer="sklearn"tosc.pp.neighborsbecause the default nearest-neighbor search fails on this data in the hosted image.
Run it
Start a notebook from the Tahoe-100M archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first. For a longer worked example, open the sample notebook Tahoe-100M vs. SEA-AD: Drugs Against Alzheimer's.
Related guides
- How to query 1000 Genomes VCF files on S3 from a Jupyter notebook
- How to open Sid Sijbrandij's osteosarcoma multi-omic dataset in a Jupyter notebook
- How to query ChEMBL bioactivity data as Parquet without installing the database
- How to query 122 million PubChem compounds with ADMET predictions in a Jupyter notebook
Last checked 2026-11-21. All dataset guides.