FlyEM Hemibrain: sample notebook
A worked notebook for FlyEM Hemibrain, with 9 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Janelia FlyEM Team (HHMI Janelia Research Campus)
- License
- Creative Commons Attribution 4.0 International (CC BY 4.0)
- Size
- 14.6 GB across 15 objects under v1.2/ (13.1 GB without tmp/)
- Stored at
- Google Cloud Storage, gs://hemibrain/v1.2
- Access
- Free and public. No account with the data provider is needed.
The notebook
FlyEM Hemibrain — the fly connectome as a graph
Roughly half the *Drosophila* central brain, reconstructed from FIB-SEM at 8 nm isotropic: 21,740 traced neurons and 3,550,404 neuron-to-neuron connections.
The reason this dataset is unusually approachable is that Janelia ships the connectivity as flat tables. You do not need to read a single voxel, or a segmentation, or an API token, to do real circuit analysis. The archive is mounted read-only (see SCIGANTIC_MOUNT_PATH below) and bytes are fetched on access, so nothing was copied to get here.
Read this before you draw a conclusion: it is a *hemi*-brain. Neurons whose arbors cross the cut plane are truncated, so their in/out degree is systematically undercounted. That is the single most common way people get hemibrain analyses wrong. The tables below cover only *non-cropped* traced neurons, which mitigates it but does not eliminate it.
1. What's in the mount
import os, tarfile, io
import pandas as pd
# The platform tells you where the archive is mounted. Do NOT hardcode a
# path: it differs by source type (/mnt/archive for S3, /mnt/gcs-archive/data
# for public GCS, /mnt/http-archive/data for HTTP-served archives).
MOUNT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
print("archive mounted at:", MOUNT)
for f in sorted(os.listdir(MOUNT)):
p = os.path.join(MOUNT, f)
if os.path.isfile(p):
print(f"{os.path.getsize(p)/1e6:9.1f} MB {f}")
Output:
archive mounted at: /mnt/gcs-archive/data
45.9 MB exported-traced-adjacencies-v1.2.tar.gz
80.8 MB hemibrain-v1.2-all-traced-adjacencies.tar.gz
45.6 MB hemibrain-v1.2-body-mean-neurotransmitters.feather
6330.2 MB hemibrain-v1.2-ftr.tar.gz
603.4 MB hemibrain-v1.2-neuprint-mito-stats-filtered-20201207.feather
1881.2 MB hemibrain-v1.2-skeletons.tar.gz
859.4 MB hemibrain-v1.2-synapse-partners.feather
1838.0 MB hemibrain-v1.2-synapse-points-with-body-roi-type-status.feather
442.7 MB hemibrain-v1.2-tbar-neurotransmitters.feather.bz2
632.7 MB hemibrain-v1.2.1-traced-synapse-to-nearest-mito-distances.feather
193.0 MB roi-meshes.tar.gz
161.3 MB traced-uncropped-contact-sizes.csv.gz
The connectome lives in exported-traced-adjacencies-v1.2.tar.gz (~46 MB). The much larger files here — skeletons, synapse points, mitochondria stats — are for morphology and ultrastructure work and are not needed for connectivity.
2. Load the connectome
# Three CSVs inside one tarball. Read them straight out of the mounted archive.
tar_path = os.path.join(MOUNT, "exported-traced-adjacencies-v1.2.tar.gz")
def read_member(name):
with tarfile.open(tar_path) as tf:
return pd.read_csv(tf.extractfile(f"exported-traced-adjacencies-v1.2/{name}"))
neurons = read_member("traced-neurons.csv") # bodyId, type, instance
conns = read_member("traced-total-connections.csv") # bodyId_pre, bodyId_post, weight
print(f"{len(neurons):,} traced neurons")
print(f"{len(conns):,} connections")
neurons.head()
Output:
21,739 traced neurons
3,550,403 connections
bodyId type instance
0 200326126 CL141 CL141_L
1 202916528 FS4C FS4C(FQ12)_C4_R
2 203253072 FS4A FS4A(AB)(FQ12)_C3_R
3 203253253 SMP505 SMP505(PDM22)_L
4 203257652 FS4C FS4C(FQ12)_C6_R
# `weight` is a SYNAPSE COUNT, not a physiological strength.
# Converting counts to functional weight is a modelling assumption you must defend.
conns.head()
Output:
bodyId_pre bodyId_post weight 0 294437328 295470623 1 1 294437328 295133902 1 2 294437328 448260940 1 3 294437328 294783423 1 4 294437328 5812979995 1
3. Cell types
type is the cell-type name (e.g. CL141); instance identifies the individual cell within that type. Not every traced body has an assigned type.
typed = neurons["type"].notna().sum()
print(f"{typed:,} of {len(neurons):,} neurons have an assigned type "
f"({100*typed/len(neurons):.1f}%)")
neurons["type"].value_counts().head(15)
Output:
19,723 of 21,739 neurons have an assigned type (90.7%) type KCg-m 590 LC10 448 KCab-m 354 MC61 346 KCab-c 252 LC12 226 KCab-s 223 LC17 186 LC9 135 KCa'b'-ap2 127 MC64 121 KCa'b'-m 119 PFNp_b 115 ...
4. Build the graph
3.55 M edges is small enough to hold in memory, so the whole connectome becomes a networkx DiGraph in one call.
import networkx as nx
G = nx.from_pandas_edgelist(
conns, "bodyId_pre", "bodyId_post",
edge_attr="weight", create_using=nx.DiGraph,
)
print(f"{G.number_of_nodes():,} nodes / {G.number_of_edges():,} edges")
# Attach type labels so results are readable rather than bare integers.
name_of = dict(zip(neurons["bodyId"], neurons["type"]))
Output:
21,739 nodes / 3,550,403 edges
5. Ask a circuit question
Mushroom body output neurons (MBON*) are the readout of the fly's learning centre. Here is what drives one of them most strongly.
# Pick an MBON that exists in this volume.
mbons = neurons[neurons["type"].fillna("").str.startswith("MBON")]
print(f"{len(mbons)} MBONs in the volume")
target = int(mbons.iloc[0]["bodyId"])
print(f"\ntarget: {target} ({name_of.get(target)})\n")
# Strongest presynaptic partners, by synapse count.
upstream = (
conns[conns["bodyId_post"] == target]
.nlargest(10, "weight")
.assign(partner_type=lambda d: d["bodyId_pre"].map(name_of))
)
upstream[["bodyId_pre", "partner_type", "weight"]]
Output:
68 MBONs in the volume
target: 300972942 (MBON14)
bodyId_pre partner_type weight
172656 331662710 PPL106 248
3091924 5813019607 PPL106 214
316027 422725634 MBON06 116
3404922 5813068729 MBON14 97
361013 424789697 MBON02 95
3518777 5813105172 DPM 82
345749 424767514 MBON11 40
442512 482071823 NaN 27
289801 393387426 KCab-s 26
3125916 5813021263 KCab-s 26
6. Where the synapses actually are
traced-roi-connections.csv splits each connection by brain region (ROI), so you can ask *where* two neurons talk, not just whether they do.
Per the bundled README: only primary ROIs appear. Synapses outside any primary ROI are omitted, so the per-ROI weights for a pair can sum to less than that pair's total weight. The difference is the unassigned remainder — don't read it as missing data.
roi = read_member("traced-roi-connections.csv")
print(f"{len(roi):,} rows, {roi['roi'].nunique()} primary ROIs")
# Verify the README's claim on our target rather than taking it on faith.
pair_total = conns[conns["bodyId_post"] == target]["weight"].sum()
pair_roi = roi[roi["bodyId_post"] == target]["weight"].sum()
print(f"\ninputs to {target}: total={pair_total:,} sum-over-ROIs={pair_roi:,} "
f"unassigned={pair_total - pair_roi:,}")
roi[roi["bodyId_post"] == target].groupby("roi")["weight"].sum().nlargest(10)
Output:
4,259,624 rows, 62 primary ROIs inputs to 300972942: total=12,099 sum-over-ROIs=12,099 unassigned=0 roi aL(R) 11789 SIP(R) 101 SMP(R) 98 SLP(R) 84 a'L(R) 24 NotPrimary 3 Name: weight, dtype: int64
7. Degree distribution
The truncation caveat bites hardest here: cells cut by the volume boundary have artificially low degree. Treat the left tail with suspicion.
import matplotlib.pyplot as plt
import numpy as np
indeg = np.array([d for _, d in G.in_degree()])
outdeg = np.array([d for _, d in G.out_degree()])
fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
for a, data, lab in ((ax[0], indeg, "in-degree"), (ax[1], outdeg, "out-degree")):
a.hist(data[data > 0], bins=60, log=True, color="#3b6ea5")
a.set_xlabel(f"{lab} (partners)"); a.set_ylabel("neurons (log)")
a.set_title(f"median {np.median(data):.0f}, max {data.max():,}")
fig.tight_layout()
8. Predicted neurotransmitters
Feather format (pyarrow is already in this image). Two things will trip you up:
- body is the index, not a column. nt.columns does not contain it.
- This table covers ~838,000 bodies — not the 21,739 traced neurons. The vast
majority are fragments carrying statusLabel values like Unimportant or Leaves. Joining it to the connectome without filtering silently drags that junk in. Always restrict to the traced set first.
The values are predictions from EM ultrastructure, not measurements. They carry real error rates — treat them as a prior, especially for any individual cell.
nt_path = os.path.join(MOUNT, "hemibrain-v1.2-body-mean-neurotransmitters.feather")
nt = pd.read_feather(nt_path)
print(f"{len(nt):,} bodies in the table (index name: {nt.index.name!r})")
print(f"columns: {list(nt.columns)}\n")
# Restrict to the traced neurons — this is the join you actually want.
traced_nt = nt.loc[nt.index.intersection(neurons["bodyId"])]
print(f"{len(traced_nt):,} of {len(neurons):,} traced neurons have a prediction")
traced_nt["predicted_nt"].value_counts()
Output:
837,710 bodies in the table (index name: 'body') columns: ['type', 'instance', 'statusLabel', 'gaba', 'acetylcholine', 'glutamate', 'serotonin', 'octopamine', 'dopamine', 'neither', 'predicted_nt'] 21,709 of 21,739 traced neurons have a prediction predicted_nt acetylcholine 9577 glutamate 5342 dopamine 3183 gaba 3009 serotonin 282 octopamine 191 neither 125 Name: count, dtype: int64
Where to go next
- bodyId is stable within v1.2 only. v1.0 and v1.1 renumber. This archive pins
v1.2 — state that in your methods.
- Optic lobes are largely excluded. For visual circuits use FlyWire (full adult
brain) instead.
- Morphology? hemibrain-v1.2-skeletons.tar.gz is in this mount (~1.9 GB).
- Targeted queries? neuPrint (neuprint.janelia.org) exposes the same data over
a Cypher API, often better than the flat files for narrow questions. It needs pip install neuprint-python and a free token — neither is in this image, so install into your own environment rather than this pod.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.