BossDB Open Neuroimagery Datasets: sample notebook

A worked notebook for BossDB Open Neuroimagery Datasets, with 10 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Johns Hopkins University Applied Physics Laboratory
License
Creative Commons 4.0 International (CC BY 4.0); Creative Commons CC0 1.0 Universal (CC0-1.0); Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)
Stored at
Amazon S3, s3://bossdb-open-data (us-east-1)
Access
Free and public. No account with the data provider is needed.

The notebook

# Start here: BossDB holds dozens of neuroimaging projects. This lists them,
# then zooms into the largest, the MICrONS cubic millimetre of mouse
# visual cortex, from the whole block down to individual cells.
import os, time
import numpy as np
import matplotlib.pyplot as plt
from cloudvolume import CloudVolume

%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 82}}
plt.rcParams["figure.dpi"] = 78

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
projects = sorted(d for d in os.listdir(ROOT) if os.path.isdir(os.path.join(ROOT, d)) and not d.startswith(("_", ".")))
print(f"{len(projects)} project folders at the top of the bucket, for example:")
print("  " + ", ".join(projects[:24]) + ", ...")

EM = os.path.join(ROOT, "iarpa_microns", "minnie", "minnie65", "em")
coarse = CloudVolume(f"precomputed://file://{EM}", mip=8, progress=False, fill_missing=True)   # 2 um voxels
fine = CloudVolume(f"precomputed://file://{EM}", mip=5, progress=False, fill_missing=True)     # 256 nm voxels
b = coarse.bounds
z8 = (b.minpt[2] + b.maxpt[2]) // 2
t = time.time()
block = np.squeeze(coarse[b.minpt[0]:b.maxpt[0], b.minpt[1]:b.maxpt[1], z8:z8 + 1])
# Centre of the tissue in this section, then the same point at the finer level.
ys, xs = np.nonzero(block.T > 0)
scale = np.array(coarse.resolution) / np.array(fine.resolution)
cx, cy, cz = (np.array([b.minpt[0] + xs.mean(), b.minpt[1] + ys.mean(), z8]) * scale).astype(int)
H = 400
crop = np.squeeze(fine[cx - H:cx + H, cy - H:cy + H, cz:cz + 1])
print(f"read the whole block at 2 um and an 800 x 800 crop at 256 nm in {time.time() - t:.1f} s")

fig, axes = plt.subplots(1, 2, figsize=(12, 5.4))
axes[0].imshow(block.T, cmap="gray")
x0, y0 = cx / scale[0] - b.minpt[0] - H / scale[0], cy / scale[1] - b.minpt[1] - H / scale[1]
axes[0].add_patch(plt.Rectangle((x0, y0), 2 * H / scale[0], 2 * H / scale[1], fill=False, ec="#1d4ed8", lw=2))
axes[0].set_title("The MICrONS block, one section (2 um voxels)")
lo, hi = np.percentile(crop[crop > 0], [1, 99])
axes[1].imshow(crop.T, cmap="gray", vmin=lo, vmax=hi)
axes[1].set_title(f"The boxed {2 * H * fine.resolution[0] / 1000:.0f} um square at 256 nm")
for ax in axes: ax.axis("off")
plt.tight_layout(); plt.show()
print("Pale discs are cell bodies, white ovals are blood vessels, dark lines are folds and cracks in the cut sections.")

BossDB: open neuroimaging volumes, from synapses to whole brains

BossDB is the Johns Hopkins APL archive for large neuroscience imaging: electron microscopy, X-ray microtomography and light microscopy volumes, with the segmentations, synapse tables and skeletons derived from them. The public bucket bossdb-open-data is mounted read-only at /mnt/archive, one folder per project, usually named after the first author and year (kasthuri2015/, hildebrand2017/) or the consortium (iarpa_microns/, flywire/).

Most volumes are stored in neuroglancer precomputed format: a folder with one info file and chunk files for each resolution level. Read them with cloud-volume, which is installed, as the cell above does, and never as image files. Many folders are tens of terabytes, so open one project at a time and ask for coarse resolution levels first.

The rest of this notebook walks through the richest project here, MICrONS, the only cortical dataset where the same neurons have both an EM reconstruction and a record of how they responded to visual stimuli. Everything below reads from iarpa_microns/minnie/. The same CloudVolume call opens the other projects' volumes once you point it at their folders.

MICrONS: structure and function in one cubic millimeter

MICrONS imaged about 1 mm³ of mouse visual cortex with electron microscopy after recording the same neurons with two-photon calcium imaging. The result is roughly 75,000 neurons, 500 million synapses, and, for 12,894 of those neurons, a matched in vivo response record. It is the only cortical dataset where you can ask "does the wiring follow the tuning" on the same cells.

MICrONS lives under iarpa_microns/ in this mount. Three kinds of data live under minnie/ (the minnie65 specimen is the one with the full derived stack):

Everything below reads from the mount. Nothing is downloaded and no CAVE token is needed.

Citations: MICrONS Consortium et al. 2025 (doi:10.1038/s41586-025-08790-w) for the volume; Ding, Fahey, Papadopoulos et al. 2025 (doi:10.1038/s41586-025-08840-3) for the functional-connectomics tables; Wang et al. 2025 (doi:10.1038/s41586-025-08829-y) for the digital twin.

import os, time
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%config InlineBackend.figure_formats = ["jpeg"]
%config InlineBackend.print_figure_kwargs = {"pil_kwargs": {"quality": 82}}
plt.rcParams["figure.dpi"] = 78

ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
M = os.path.join(ROOT, "iarpa_microns", "minnie")
print("mount:", ROOT)
print("minnie/:", sorted(os.listdir(M)))
print("minnie/minnie65/:", sorted(os.listdir(os.path.join(M, "minnie65"))))

Output:

mount: /mnt/archive
minnie/: ['functional_coregistration', 'functional_data', 'microCT', 'minnie35', 'minnie65', 'proofreading_status']
minnie/minnie65/: ['autoseg', 'autoseg-sharded', 'cell_types', 'clefts', 'clefts-sharded', 'em', 'nuclei', 'nuclei-sharded', 'nucleus_detection', 'nucleus_neuron_classification', 'seg', 'skeletons', 'synapse_graph', 'ws']

1. The volume, through cloud-volume

minnie65/em is a precomputed image layer: one info manifest plus chunk objects at nine resolution levels (mips), from 8 x 8 x 40 nm up to 2 x 2 x 1.3 µm. cloud-volume reads the manifest and returns numpy arrays for any cutout at any mip. Ask for the coarsest mip to see the whole block; zoom to fine mips only for the cutouts you need.

The file:// URL below points cloud-volume at the mount. If you run this off the platform, swap it for precomputed://s3://bossdb-open-data/iarpa_microns/minnie/minnie65/em with use_https=True.

from cloudvolume import CloudVolume

EM_PATH = os.path.join(M, "minnie65", "em")
vol = CloudVolume(f"precomputed://file://{EM_PATH}", mip=0, progress=False, fill_missing=True)
print("dtype", vol.dtype, "| mips", len(vol.available_mips), "| mip0 voxel size (nm)", [int(r) for r in vol.resolution])
for s in vol.scales:
    print(f"  {s['key']:>16}  {str(s['size']):>24} voxels  encoding={s['encoding']}")

# Whole-block overview at the coarsest mip: 2 x 2 x 1.28 um voxels, so the
# 832 x 704 slice below is 1.7 x 1.4 mm of tissue. Only 143 small chunks.
coarse = CloudVolume(f"precomputed://file://{EM_PATH}", mip=8, progress=False, fill_missing=True)
b = coarse.bounds
zmid = (b.minpt[2] + b.maxpt[2]) // 2
t = time.time()
overview = np.squeeze(coarse[b.minpt[0]:b.maxpt[0], b.minpt[1]:b.maxpt[1], zmid:zmid + 1])
print(f"overview {overview.shape} at {[int(r) for r in coarse.resolution]} nm voxels in {time.time() - t:.1f}s")

Output:

dtype uint8 | mips 9 | mip0 voxel size (nm) [8, 8, 40]
            8_8_40   [212992, 180224, 13088] voxels  encoding=raw
          16_16_40    [106496, 90112, 13088] voxels  encoding=jpeg
          32_32_40     [53248, 45056, 13088] voxels  encoding=jpeg
          64_64_40     [26624, 22528, 13088] voxels  encoding=jpeg
        128_128_80      [13312, 11264, 6544] voxels  encoding=jpeg
       256_256_160        [6656, 5632, 3272] voxels  encoding=jpeg
       512_512_320        [3328, 2816, 1636] voxels  encoding=jpeg
     1024_1024_640         [1664, 1408, 818] voxels  encoding=jpeg
    2048_2048_1280           [832, 704, 409] voxels  encoding=jpeg
overview (832, 704) at [2048, 2048, 1280] nm voxels in 3.7s

The EM bounding box is larger than the tissue. Black is outside the sample; the gray rectangle is the imaged block, about 1.3 x 0.9 mm in this section, with the montage tiles visible along its edges. The y axis of this volume runs with cortical depth, pia at the top, so a horizontal line across the block is a laminar position.

tissue = overview[overview > 0]
lo, hi = np.percentile(tissue, [1, 99])
fig, ax = plt.subplots(figsize=(7, 5.6))
ax.imshow(overview.T, cmap="gray", vmin=lo, vmax=hi)
ax.set_title(f"minnie65 EM, one section at mip 8 (z = {zmid}), {overview.shape[0]} x {overview.shape[1]} voxels of 2 um")
ax.set_xlabel("x (voxels of 2.05 um)"); ax.set_ylabel("y, cortical depth (voxels of 2.05 um)")
plt.tight_layout(); plt.show()

2. The neurons that have both a reconstruction and a response

Ding et al. 2025 asked whether cortical neurons connect like-to-like: do pairs with similar visual responses synapse onto each other more often? Their release ships as two pandas DataFrames in this bucket.

Both load in under ten seconds off the mount.

FC = os.path.join(M, "functional_data", "functional_connectomics", "node_and_edge_properties", "v1")
t = time.time()
node = pd.read_pickle(os.path.join(FC, "node_data_v1.pkl")).set_index("nucleus_id")
edge = pd.read_pickle(os.path.join(FC, "edge_data_v1.pkl"))
print(f"loaded in {time.time() - t:.1f}s: {len(node):,} neurons, {len(edge):,} pairs")

print("\nneurons by area and layer:")
print(pd.crosstab(node["brain_area"], node["layer"]).to_string())
print("\npairs by population:")
print(edge["population"].value_counts().to_string())
print("\nnode columns:", ", ".join(node.columns))

Output:

loaded in 7.3s: 12,894 neurons, 1,693,276 pairs

neurons by area and layer:
layer       L2/3    L4    L5
brain_area                  
V1          4056  2906  1527
RL          1536  1059   512
LM            11     8     4
AL           436   247   109

pairs by population:
population
Same region    1397905
ADP             287243
Connected         8128
...
Connection probability rises with functional similarity

Among pairs whose axon and dendrite come close enough to touch (ADP plus Connected), bin by in vivo signal correlation and count how often a synapse was actually found. This is the headline result of the paper, recomputed here from the release tables in one groupby.

touch = edge[edge["population"].isin(["Connected", "ADP"])].dropna(subset=["in_vivo_sig_corr", "delta_ori_iv"]).copy()
touch["connected"] = touch["population"] == "Connected"

def p_connected_by(col, nbins=8):
    q = np.quantile(touch[col], np.linspace(0, 1, nbins + 1))
    b = pd.cut(touch[col], q, include_lowest=True)
    g = touch.groupby(b, observed=True)["connected"].agg(["mean", "count"])
    g["center"] = [iv.mid for iv in g.index]
    g["sem"] = np.sqrt(g["mean"] * (1 - g["mean"]) / g["count"])
    return g

corr = p_connected_by("in_vivo_sig_corr")
dori = p_connected_by("delta_ori_iv")

fig, axes = plt.subplots(1, 2, figsize=(9.5, 3.6))
for ax, g, xl in [(axes[0], corr, "in vivo signal correlation (pair)"), (axes[1], dori, "orientation preference difference (deg)")]:
    ax.errorbar(g["center"], 100 * g["mean"], yerr=100 * g["sem"], fmt="o-", color="#1d4ed8", lw=1.5, capsize=3)
    ax.set_xlabel(xl); ax.set_ylabel("P(synapse | axon-dendrite proximity), %")
    ax.grid(alpha=0.3)
axes[0].set_title(f"Like-to-like wiring, {len(touch):,} proximate pairs")
axes[1].set_title("Same test against orientation difference")
plt.tight_layout(); plt.show()

lo, hi = corr.iloc[0], corr.iloc[-1]
print(f"connection probability: {100 * lo['mean']:.2f}% in the least-correlated octile, {100 * hi['mean']:.2f}% in the most-correlated ({hi['mean'] / lo['mean']:.2f}x)")

Output:

connection probability: 2.34% in the least-correlated octile, 3.72% in the most-correlated (1.59x)

The orientation-difference version is far weaker. A single tuning parameter captures less of what the wiring follows than whole-response similarity does, which is the paper's point: the digital twin's learned feature weights, not the classical tuning curve, are what predict who connects.

3. One connected pair, seen every way the dataset allows

Pick a connected pair in V1 layer 2/3 with at least three synapses, similar orientation preference, and the strongest in vivo signal correlation, then look at the same two cells as EM, as morphology, and as activity. Every panel below comes from this mount.

Skeleton files are named _ .swc under minnie65/skeletons/v661/skeletons/, one per proofread cell at CAVE version 661. Listing the 76,980 files takes a few seconds on the mount.

SK = os.path.join(M, "minnie65", "skeletons", "v661", "skeletons")
t = time.time()
skel_files = {int(f.split("_")[1].split(".")[0]): f for f in os.listdir(SK) if f.endswith(".swc")}
print(f"{len(skel_files):,} skeletons listed in {time.time() - t:.1f}s")

cand = touch[touch["connected"] & (touch["proj_hva_layer"] == "V1L2/3->V1L2/3") & (touch["n_synapses"] >= 3) & (touch["delta_ori_iv"] < 20)]
cand = cand[cand["pre_nucleus_id"].isin(skel_files) & cand["post_nucleus_id"].isin(skel_files)]
pair = cand.sort_values("in_vivo_sig_corr", ascending=False).iloc[0]
pre_id, post_id = int(pair["pre_nucleus_id"]), int(pair["post_nucleus_id"])
pre, post = node.loc[pre_id], node.loc[post_id]

print(f"pre  nucleus {pre_id}: {pre['brain_area']} {pre['layer']}, pref ori {pre['pref_ori_iv']:.0f} deg, OSI {pre['osi_iv']:.2f}, proofreading {pre['proofread_status']}")
print(f"post nucleus {post_id}: {post['brain_area']} {post['layer']}, pref ori {post['pref_ori_iv']:.0f} deg, OSI {post['osi_iv']:.2f}")
print(f"pair: {int(pair['n_synapses'])} synapses, summed cleft size {int(pair['synapse_size'])} voxels, in vivo signal corr {pair['in_vivo_sig_corr']:.2f}, orientation difference {pair['delta_ori_iv']:.1f} deg")

Output:

76,980 skeletons listed in 7.0s
pre  nucleus 292685: V1 L2/3, pref ori 61 deg, OSI 0.65, proofreading whole_cell
post nucleus 258429: V1 L2/3, pref ori 68 deg, OSI 0.46
pair: 3 synapses, summed cleft size 21044 voxels, in vivo signal corr 0.21, orientation difference 7.0 deg
import navis

def em_cutout(n, half_um=25, mip=3):
    # nucleus_x/y/z are nm and sit in the EM frame (they equal the skeleton
    # soma locations). mip 3 is 64 x 64 x 40 nm, so 25 um is 390 voxels.
    v = CloudVolume(f"precomputed://file://{EM_PATH}", mip=mip, progress=False, fill_missing=True)
    rx, ry, rz = v.resolution
    cx, cy, cz = int(n["nucleus_x"] // rx), int(n["nucleus_y"] // ry), int(n["nucleus_z"] // rz)
    h = int(half_um * 1000 // rx)
    return np.squeeze(v[cx - h:cx + h, cy - h:cy + h, cz:cz + 1]).T, half_um

pre_sk = navis.read_swc(os.path.join(SK, skel_files[pre_id]), units="um")
post_sk = navis.read_swc(os.path.join(SK, skel_files[post_id]), units="um")
print(f"pre skeleton: {pre_sk.n_nodes:,} nodes, {pre_sk.cable_length / 1000:.1f} mm of cable | post skeleton: {post_sk.n_nodes:,} nodes, {post_sk.cable_length / 1000:.2f} mm")

fig = plt.figure(figsize=(10, 7.4))
gs = fig.add_gridspec(2, 3, height_ratios=[1, 1.15])
for i, (n, nid, label) in enumerate([(pre, pre_id, "presynaptic"), (post, post_id, "postsynaptic")]):
    img, half = em_cutout(n)
    lo, hi = np.percentile(img[img > 0], [0.5, 99.5])
    ax = fig.add_subplot(gs[0, i])
    ax.imshow(img, cmap="gray", vmin=lo, vmax=hi, extent=[-half, half, half, -half])
    ax.plot([0], [0], "+", color="#f97316", ms=14, mew=1.2)
    ax.set_title(f"{label} soma, nucleus {nid}\nEM at 64 nm, pref ori {n['pref_ori_iv']:.0f} deg", fontsize=9)
    ax.set_xlabel("um (cross = nucleus centroid)"); ax.set_yticks([])

ax = fig.add_subplot(gs[0, 2])
tt = np.arange(len(pre["in_vivo_mean_resp"])) / 30.0
ax.plot(tt, pre["in_vivo_mean_resp"], color="#1d4ed8", lw=1.2, label="pre")
ax.plot(tt, post["in_vivo_mean_resp"], color="#dc2626", lw=1.2, label="post")
ax.set_title(f"trial-averaged in vivo response\nsignal corr {pair['in_vivo_sig_corr']:.2f}", fontsize=9)
ax.set_xlabel("time (s, 30 Hz frames)"); ax.set_ylabel("dF/F, a.u."); ax.legend(frameon=False, fontsize=8)

ax = fig.add_subplot(gs[1, :])
navis.plot2d([pre_sk, post_sk], color=["#1d4ed8", "#dc2626"], method="2d", view=("x", "y"), ax=ax, linewidth=0.5)
ax.invert_yaxis()
ax.set_title(f"skeletons at CAVE v661: pre (blue, {pre_sk.cable_length / 1000:.0f} mm incl. proofread axon) and post (red)", fontsize=9)
ax.set_xlabel("x (um)"); ax.set_ylabel("y, cortical depth (um)")
plt.tight_layout(); plt.show()

Output:

pre skeleton: 18,120 nodes, 26.9 mm of cable | post skeleton: 2,407 nodes, 3.65 mm

4. The V1 column: cell types by depth

allen_visp_column_soma_coarse_types_v1.csv labels every soma in a 100 µm column through V1 with a coarse class (excitatory, inhibitory, non-neuronal). The y coordinate is cortical depth, so a depth histogram by class is the laminar structure of the column. The larger allen_soma_ei_class_model_v1.csv extends the excitatory/inhibitory call to 69,957 somata across the volume. Both files ship without a header; the columns are id, valid, classification system, class, x, y, z in 4 x 4 x 40 nm voxels.

CT = os.path.join(M, "minnie65", "cell_types")
cols = ["id", "valid", "system", "cell_type", "x", "y", "z"]
column = pd.read_csv(os.path.join(CT, "allen_visp_column_soma_coarse_types_v1.csv"), header=None, names=cols)
ei_all = pd.read_csv(os.path.join(CT, "allen_soma_ei_class_model_v1.csv"), header=None, names=cols)
column["depth_um"] = column["y"] * 4 / 1000
print(f"column: {len(column):,} somata; volume-wide E/I model: {len(ei_all):,} somata")
print(column["cell_type"].value_counts().to_string())

fig, ax = plt.subplots(figsize=(7, 3.8))
bins = np.arange(column["depth_um"].min(), column["depth_um"].max() + 25, 25)
for ct, c in [("excitatory", "#1d4ed8"), ("inhibitory", "#dc2626"), ("Unsure", "#94a3b8")]:
    ax.hist(column.loc[column["cell_type"] == ct, "depth_um"], bins=bins, histtype="step", lw=1.6, color=c, label=ct if ct != "Unsure" else "non-neuronal or unsure")
ax.set_xlabel("cortical depth (um, volume y axis)"); ax.set_ylabel("somata per 25 um")
ax.set_title("V1 column soma classes by depth (allen_visp_column_soma_coarse_types_v1)")
ax.legend(frameon=False); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()

Output:

column: 2,204 somata; volume-wide E/I model: 69,957 somata
cell_type
excitatory    1185
Unsure         857
inhibitory     162

5. Raw access: the synapse graph and the rest of the bucket

The tables above are curated subsets. The full graph is here too, and it is large:

The cell below reads the head of the cell-pair export to show its shape, and defines the one-pass filter that turns it into a nucleus-to-nucleus graph. That pass takes a few minutes on the mount, so it is left for you to run.

CE = os.path.join(M, "minnie65", "synapse_graph", "cave_exports", "v1300")
header = pd.read_csv(os.path.join(CE, "connections_with_nuclei_header.csv"), header=None)[0].tolist()
print("columns:", header)

t = time.time()
head = pd.read_csv(os.path.join(CE, "connections_with_nuclei.csv.gz"), names=header, nrows=300_000, compression="gzip")
orphan = (head["post_pt_root_id"] == 0) | (head["pre_nuc_id"] < 0) | (head["post_nuc_id"] < 0)
print(f"first {len(head):,} rows read in {time.time() - t:.1f}s; {100 * orphan.mean():.0f}% are synapses onto background (root id 0) or onto segments with no detected nucleus")
print("The export is sorted with those rows first, so a head sample says nothing about the cell-to-cell graph. Stream the whole file for that:")
print(head.head(3).to_string(index=False))

def cell_to_cell_graph(chunksize=2_000_000):
    # One pass over the 3 GB export keeping only pairs where both sides have a
    # nucleus. About 3 to 4 minutes on the mount; returns a DataFrame of
    # (pre_nuc_id, post_nuc_id, n_syn, sum_size).
    parts = []
    for chunk in pd.read_csv(os.path.join(CE, "connections_with_nuclei.csv.gz"), names=header, chunksize=chunksize, compression="gzip"):
        keep = chunk[(chunk["post_pt_root_id"] != 0) & (chunk["pre_nuc_id"] > 0) & (chunk["post_nuc_id"] > 0)]
        parts.append(keep[["pre_nuc_id", "post_nuc_id", "n_syn", "sum_size"]])
    return pd.concat(parts, ignore_index=True)

# graph = cell_to_cell_graph()   # uncomment for the full nucleus-to-nucleus graph

Output:

columns: ['pre_pt_root_id', 'post_pt_root_id', 'n_syn', 'sum_size', 'pre_nuc_id', 'post_nuc_id']
first 300,000 rows read in 0.6s; 100% are synapses onto background (root id 0) or onto segments with no detected nucleus
The export is sorted with those rows first, so a head sample says nothing about the cell-to-cell graph. Stream the whole file for that:
    pre_pt_root_id  post_pt_root_id  n_syn  sum_size  pre_nuc_id  post_nuc_id
864691135608184522                0      1      1168          -1           -1
864691135609445383                0      1      1048          -1           -1
864691135635603908                0      1       532          -1           -1

Where to go next

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.