Brain Image Library (BIL): BICCN whole-brain microscopy: sample notebook

A worked notebook for Brain Image Library (BIL): BICCN whole-brain microscopy, with 7 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Brain Image Library (Pittsburgh Supercomputing Center / University of Pittsburgh)
License
CC BY-SA 4.0 (some datasets additionally under Allen Institute Terms of Use)
Stored at
download.brainimagelibrary.org
Access
Free and public. No account with the data provider is needed.
DOI
10.1038/s41597-024-03761-8

The notebook

Brain Image Library: whole-brain microscopy, read in place

The Brain Image Library (BIL) is the BRAIN Initiative's repository for whole-brain microscopy: about 14,000 datasets and 6 PB from BICCN, BICAN and ASAP labs, the largest public collection of cleared-tissue light-sheet brains, plus fMOST and serial two-photon tomography volumes, spatial transcriptomics and Patch-seq morphologies.

Nothing here downloads a dataset. scigantic_bil (preinstalled in this image) loads BIL's own daily inventory into a catalog, finds datasets by technique or free text, and reads one slice, one thumbnail or one OME-Zarr level over HTTP. The same tree is also mounted read-only at $SCIGANTIC_MOUNT_PATH if you prefer paths.

Data license: CC BY-SA 4.0. Cite each dataset's DOI (retrieve(id).doi).

import time
import numpy as np
import scigantic_bil as bil
import matplotlib.pyplot as plt
%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 80}}
plt.rcParams["figure.dpi"] = 72

cat = bil.BilCatalog.load()          # one 5 MB GET, cached on disk afterwards
s = cat.summary()
print(f"inventory {s['inventory_date']}: {s['datasets']:,} datasets, {s['files']:,} files, {s['terabytes']:,} TB")
print("technique:", ", ".join(f"{k} ({v})" for k, v in list(s["technique"].items())[:8]))
print("species:  ", ", ".join(f"{k} ({v})" for k, v in list(s["species"].items())[:6]))

Output:

inventory 20260731: 14,224 datasets, 144,335,214 files, 6,034.6 TB
technique: other (6074), Patch-seq (2959), fMOST (985), retrograde transsynaptic tracing (914), STPT (757), light sheet microscopy (347), transsynaptic (341), EM (301)
species:   mouse (11350), human (1177), marmoset (308), macaca nemestrina (291), rat (231), homo sapiens (206)

1. Find the light-sheet data

The inventory's technique field says "light sheet microscopy" for a few hundred datasets. Many more are tagged other and only say light sheet in their abstract or instrument record. light_sheet() unions the structured field with BIL's fulltext index, which is the honest count.

by_field = cat.filter(light_sheet=True)
ls = cat.light_sheet()
tb = sum(d.size_bytes or 0 for d in ls) / 1e12
print(f"{len(by_field)} datasets by technique field, {len(ls)} with the fulltext union, {tb:.0f} TB")

df = cat.to_dataframe(ls)
top = (df.groupby(["contributor", "affiliation"])
         .agg(datasets=("bildid", "size"), TB=("size_gb", lambda g: g.sum() / 1000))
         .sort_values("datasets", ascending=False)
         .head(8))
top

Output:

348 datasets by technique field, 808 with the fulltext union, 278 TB
                                                                     datasets  \
contributor      affiliation                                                    
Zhang, Jiangyang New York University                                      193   
Kim, Yongsoo     Pennsylvania State University                            149   
Betty, Rebecca   Pennsylvania State University                            116   
Svoboda, Karel   Allen Institute for Neural Dynamics                       82   
Yongsoo Kim Lab  Pennsylvania State University                             53   
Hampel, Stefanie University of Puerto Rico, Medical Sciences Campus        51   
Kronman, Fae     Pennsylvania State University                             46   
Tasic, Bosiljka  Allen Institute for Brain Science                         40   

                                                                            TB  
contributor      affiliation                                                    
Zhang, Jiangyang New York University                                 15.220843  
...
fig, ax = plt.subplots(figsize=(7, 3.2))
sizes = df["size_gb"].dropna()
cap = sizes.quantile(0.98)                      # a few multi-TB outliers would flatten the plot
ax.hist(sizes.clip(upper=cap), bins=40, color="#3b6fb6")
ax.set_xlabel(f"dataset size (GB, clipped at the 98th percentile = {cap:.0f} GB)")
ax.set_ylabel("light-sheet datasets")
ax.set_title(f"median light-sheet dataset: {sizes.median():.0f} GB, largest {sizes.max() / 1000:.0f} TB")
plt.tight_layout()

2. Read one slice, not the stack

A light-sheet dataset here is usually one folder of a few thousand single-slice TIFFs, one per z-plane and channel, about 16 MB each. Listing the dataset is one request; reading a slice is one more. thumbnail() reads the middle slice and stride-downsamples it.

stack = bil.slices("ace-bin-run")               # Kim lab (Penn State) LSFM, nothing fetched yet
d = cat["ace-bin-run"]
print(f"{d.bildid}: {len(stack)} slices, {d.size_gb:.0f} GB, {d.technique}, {d.contributor}")
print("first / middle / last:", stack[0].name, stack[len(stack) // 2].name, stack[-1].name)

fig, axes = plt.subplots(1, 4, figsize=(11, 3.4))
for ax, z in zip(axes, (300, 700, 1100, 1500)):
    img = bil.downsample(bil.read_tiff(stack[z]), 384)   # one HTTP read per panel
    lo, hi = np.percentile(img, (1, 99.7))
    ax.imshow(img, cmap="gray", vmin=lo, vmax=hi)
    ax.set_title(f"z = {z}  ({img.shape[1]}x{img.shape[0]} preview)", fontsize=9)
    ax.axis("off")
plt.suptitle("E13.5 mouse embryo, NeuroTrace, four of 1,923 planes", fontsize=10)
plt.tight_layout()

Output:

ace-bin-run: 1923 slices, 29 GB, light sheet microscopy, Kim, Yongsoo
first / middle / last: Z00001_ch02.tif Z00962_ch02.tif Z01923_ch02.tif

3. OME-Zarr stores, when a dataset ships one

A zarr store is a directory, so it never shows up in the inventory's file-extension histogram. find_zarr() checks a dataset's manifest; open_zarr() opens it lazily and zarr_levels() reports the pyramid levels the server actually has (one BIL store declares eight and serves seven).

stores = bil.find_zarr("ace-cup-eel")            # UC Irvine iDISCO+ light sheet, technique tagged "other"
print(stores)
g = bil.open_zarr("ace-cup-eel")
levels = bil.zarr_levels(g)
print("levels:", levels, "  full-resolution shape:", tuple(g[levels[0]].shape), "(t, c, z, y, x)")

plane = bil.zarr_thumbnail(g, max_size=512)
ys, xs = np.nonzero(plane > np.percentile(plane, 50))   # the array is padded; crop to the tissue
plane = plane[ys.min():ys.max() + 1, xs.min():xs.max() + 1]
plt.figure(figsize=(6, 4))
plt.imshow(plane, cmap="magma", vmax=np.percentile(plane, 99.5))
plt.title(f"coarsest level, middle z plane ({plane.shape[1]}x{plane.shape[0]} px)"); plt.axis("off")

Output:

['https://download.brainimagelibrary.org/42/e4/42e4274e0579397f/subject_5/subject_5.zarr/']
levels: ['0', '1', '2', '3', '4', '5', '6']   full-resolution shape: (1, 1, 848, 6300, 9600) (t, c, z, y, x)
(np.float64(-0.5), np.float64(149.5), np.float64(97.5), np.float64(-0.5))

4. Two more layouts: JPEG 2000 sections and fMOST trees

Most STPT sections and nearly all viral tracing brains are JPEG 2000 (5,787 datasets). They decode in place through imagecodecs' OpenJPEG. fMOST brains are TeraFly trees with a full resolution pyramid; thumbnail() reads the coarsest level, about 25 small blocks, and never lists the full-resolution folder (1.6 million files on this one).

sec = bil.slices("ace-bit-wig")                   # STPT, 267 JPEG 2000 coronal sections
t0 = time.time(); stpt = bil.read_image(sec[137]); t_stpt = time.time() - t0
t0 = time.time(); fmost = bil.thumbnail("ace-cap-cop", max_size=512); t_fmost = time.time() - t0
print(f"STPT section {stpt.shape} {stpt.dtype} in {t_stpt:.1f}s; fMOST coarsest level {fmost.shape} in {t_fmost:.1f}s")

def tissue(img, pct=60, pad=0.03):
    # crop to the bounding box of the brighter pixels, so the panel is tissue, not black frame
    ys, xs = np.nonzero(img > np.percentile(img, pct))
    dy, dx = int((ys.max() - ys.min()) * pad), int((xs.max() - xs.min()) * pad)
    return img[max(0, ys.min() - dy):ys.max() + dy, max(0, xs.min() - dx):xs.max() + dx]

fig, axes = plt.subplots(1, 2, figsize=(11, 4.2))
for ax, img, title in ((axes[0], tissue(bil.downsample(stpt, 512), 70), "STPT, one JPEG 2000 section"), (axes[1], tissue(fmost, 70), "fMOST, coarsest TeraFly level")):
    lo, hi = np.percentile(img[img > 0], (2, 97))    # labelled cells are far brighter than tissue; do not let them set the range
    ax.imshow(img, cmap="gray", vmin=lo, vmax=hi); ax.set_title(title, fontsize=10); ax.axis("off")
plt.tight_layout()

Output:

STPT section (11377, 8557) uint16 in 1.9s; fMOST coarsest level (265, 423) in 4.1s

5. Provenance: the full record

The inventory row is what BIL indexes. The full record (title, abstract, instrument, specimen, rights, DOI, publications) lives on BIL's metadata API and is one retrieve() away.

r = bil.retrieve("ace-cup-eel")
print(r.title)
print("microscope:", r.instrument.get("microscopetype"), "|", r.instrument.get("microscopemanufacturerandmodel"))
print("specimen:  ", r.species, r.specimen.get("genotype") or "", "| technique:", r.technique, f"({r.technique_other})")
print("rights:    ", r.rights_identifier, "|", r.rights_uri)
for p in r.publications:
    print("cites:     ", p.doi or p.citation[:120])
print()
print(r.abstract[:500])

Output:

Light-sheet imaged brain follwing T.Gondii injection: Subject 5
microscope: Light-sheet | Zeiss Z.1
specimen:   mouse  | technique: other (iDISCO+ light sheet)
rights:     CC-BY-SA-4.0 | https://creativecommons.org/licenses/by-sa/4.0/, https://alleninstitute.org/legal/terms-use/
cites:      https://doi.org/10.1073/pnas.1915778116

This dataset includes light-sheet imaged brain follwing T.Gondii injection. We analyzed monocyte recruitment to the brain during infection with Toxoplasma gondii, a global foodborne parasite that establishes chronic infection in the CNS. By using 2-photon imaging through cranial windows, we dynamically tracked fluorescent monocytes at the blood–brain barrier from the acute to chronic stage of infection in mice. Whole-brain imaging revealed distinct localization patterns of monocytes in specific

Next

Package docs and source: https://github.com/Scigantic/scigantic-bil

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.