FlyEM Hemibrain: sample notebook

A worked notebook for FlyEM Hemibrain, with 9 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Janelia FlyEM Team (HHMI Janelia Research Campus)
License
Creative Commons Attribution 4.0 International (CC BY 4.0)
Size
14.6 GB across 15 objects under v1.2/ (13.1 GB without tmp/)
Stored at
Google Cloud Storage, gs://hemibrain/v1.2
Access
Free and public. No account with the data provider is needed.

The notebook

FlyEM Hemibrain — the fly connectome as a graph

Roughly half the *Drosophila* central brain, reconstructed from FIB-SEM at 8 nm isotropic: 21,740 traced neurons and 3,550,404 neuron-to-neuron connections.

The reason this dataset is unusually approachable is that Janelia ships the connectivity as flat tables. You do not need to read a single voxel, or a segmentation, or an API token, to do real circuit analysis. The archive is mounted read-only (see SCIGANTIC_MOUNT_PATH below) and bytes are fetched on access, so nothing was copied to get here.

Read this before you draw a conclusion: it is a *hemi*-brain. Neurons whose arbors cross the cut plane are truncated, so their in/out degree is systematically undercounted. That is the single most common way people get hemibrain analyses wrong. The tables below cover only *non-cropped* traced neurons, which mitigates it but does not eliminate it.

1. What's in the mount

import os, tarfile, io
import pandas as pd

# The platform tells you where the archive is mounted. Do NOT hardcode a
# path: it differs by source type (/mnt/archive for S3, /mnt/gcs-archive/data
# for public GCS, /mnt/http-archive/data for HTTP-served archives).
MOUNT = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
print("archive mounted at:", MOUNT)

for f in sorted(os.listdir(MOUNT)):
    p = os.path.join(MOUNT, f)
    if os.path.isfile(p):
        print(f"{os.path.getsize(p)/1e6:9.1f} MB  {f}")

Output:

archive mounted at: /mnt/gcs-archive/data
     45.9 MB  exported-traced-adjacencies-v1.2.tar.gz
     80.8 MB  hemibrain-v1.2-all-traced-adjacencies.tar.gz
     45.6 MB  hemibrain-v1.2-body-mean-neurotransmitters.feather
   6330.2 MB  hemibrain-v1.2-ftr.tar.gz
    603.4 MB  hemibrain-v1.2-neuprint-mito-stats-filtered-20201207.feather
   1881.2 MB  hemibrain-v1.2-skeletons.tar.gz
    859.4 MB  hemibrain-v1.2-synapse-partners.feather
   1838.0 MB  hemibrain-v1.2-synapse-points-with-body-roi-type-status.feather
    442.7 MB  hemibrain-v1.2-tbar-neurotransmitters.feather.bz2
    632.7 MB  hemibrain-v1.2.1-traced-synapse-to-nearest-mito-distances.feather
    193.0 MB  roi-meshes.tar.gz
    161.3 MB  traced-uncropped-contact-sizes.csv.gz

The connectome lives in exported-traced-adjacencies-v1.2.tar.gz (~46 MB). The much larger files here — skeletons, synapse points, mitochondria stats — are for morphology and ultrastructure work and are not needed for connectivity.

2. Load the connectome

# Three CSVs inside one tarball. Read them straight out of the mounted archive.
tar_path = os.path.join(MOUNT, "exported-traced-adjacencies-v1.2.tar.gz")

def read_member(name):
    with tarfile.open(tar_path) as tf:
        return pd.read_csv(tf.extractfile(f"exported-traced-adjacencies-v1.2/{name}"))

neurons = read_member("traced-neurons.csv")        # bodyId, type, instance
conns   = read_member("traced-total-connections.csv")  # bodyId_pre, bodyId_post, weight

print(f"{len(neurons):,} traced neurons")
print(f"{len(conns):,} connections")
neurons.head()

Output:

21,739 traced neurons
3,550,403 connections
      bodyId    type             instance
0  200326126   CL141              CL141_L
1  202916528    FS4C      FS4C(FQ12)_C4_R
2  203253072    FS4A  FS4A(AB)(FQ12)_C3_R
3  203253253  SMP505      SMP505(PDM22)_L
4  203257652    FS4C      FS4C(FQ12)_C6_R
# `weight` is a SYNAPSE COUNT, not a physiological strength.
# Converting counts to functional weight is a modelling assumption you must defend.
conns.head()

Output:

   bodyId_pre  bodyId_post  weight
0   294437328    295470623       1
1   294437328    295133902       1
2   294437328    448260940       1
3   294437328    294783423       1
4   294437328   5812979995       1

3. Cell types

type is the cell-type name (e.g. CL141); instance identifies the individual cell within that type. Not every traced body has an assigned type.

typed = neurons["type"].notna().sum()
print(f"{typed:,} of {len(neurons):,} neurons have an assigned type "
      f"({100*typed/len(neurons):.1f}%)")

neurons["type"].value_counts().head(15)

Output:

19,723 of 21,739 neurons have an assigned type (90.7%)
type
KCg-m         590
LC10          448
KCab-m        354
MC61          346
KCab-c        252
LC12          226
KCab-s        223
LC17          186
LC9           135
KCa'b'-ap2    127
MC64          121
KCa'b'-m      119
PFNp_b        115
...

4. Build the graph

3.55 M edges is small enough to hold in memory, so the whole connectome becomes a networkx DiGraph in one call.

import networkx as nx

G = nx.from_pandas_edgelist(
    conns, "bodyId_pre", "bodyId_post",
    edge_attr="weight", create_using=nx.DiGraph,
)
print(f"{G.number_of_nodes():,} nodes / {G.number_of_edges():,} edges")

# Attach type labels so results are readable rather than bare integers.
name_of = dict(zip(neurons["bodyId"], neurons["type"]))

Output:

21,739 nodes / 3,550,403 edges

5. Ask a circuit question

Mushroom body output neurons (MBON*) are the readout of the fly's learning centre. Here is what drives one of them most strongly.

# Pick an MBON that exists in this volume.
mbons = neurons[neurons["type"].fillna("").str.startswith("MBON")]
print(f"{len(mbons)} MBONs in the volume")

target = int(mbons.iloc[0]["bodyId"])
print(f"\ntarget: {target}  ({name_of.get(target)})\n")

# Strongest presynaptic partners, by synapse count.
upstream = (
    conns[conns["bodyId_post"] == target]
    .nlargest(10, "weight")
    .assign(partner_type=lambda d: d["bodyId_pre"].map(name_of))
)
upstream[["bodyId_pre", "partner_type", "weight"]]

Output:

68 MBONs in the volume

target: 300972942  (MBON14)

         bodyId_pre partner_type  weight
172656    331662710       PPL106     248
3091924  5813019607       PPL106     214
316027    422725634       MBON06     116
3404922  5813068729       MBON14      97
361013    424789697       MBON02      95
3518777  5813105172          DPM      82
345749    424767514       MBON11      40
442512    482071823          NaN      27
289801    393387426       KCab-s      26
3125916  5813021263       KCab-s      26

6. Where the synapses actually are

traced-roi-connections.csv splits each connection by brain region (ROI), so you can ask *where* two neurons talk, not just whether they do.

Per the bundled README: only primary ROIs appear. Synapses outside any primary ROI are omitted, so the per-ROI weights for a pair can sum to less than that pair's total weight. The difference is the unassigned remainder — don't read it as missing data.

roi = read_member("traced-roi-connections.csv")
print(f"{len(roi):,} rows, {roi['roi'].nunique()} primary ROIs")

# Verify the README's claim on our target rather than taking it on faith.
pair_total = conns[conns["bodyId_post"] == target]["weight"].sum()
pair_roi   = roi[roi["bodyId_post"] == target]["weight"].sum()
print(f"\ninputs to {target}: total={pair_total:,}  sum-over-ROIs={pair_roi:,}  "
      f"unassigned={pair_total - pair_roi:,}")

roi[roi["bodyId_post"] == target].groupby("roi")["weight"].sum().nlargest(10)

Output:

4,259,624 rows, 62 primary ROIs

inputs to 300972942: total=12,099  sum-over-ROIs=12,099  unassigned=0
roi
aL(R)         11789
SIP(R)          101
SMP(R)           98
SLP(R)           84
a'L(R)           24
NotPrimary        3
Name: weight, dtype: int64

7. Degree distribution

The truncation caveat bites hardest here: cells cut by the volume boundary have artificially low degree. Treat the left tail with suspicion.

import matplotlib.pyplot as plt
import numpy as np

indeg  = np.array([d for _, d in G.in_degree()])
outdeg = np.array([d for _, d in G.out_degree()])

fig, ax = plt.subplots(1, 2, figsize=(11, 3.6))
for a, data, lab in ((ax[0], indeg, "in-degree"), (ax[1], outdeg, "out-degree")):
    a.hist(data[data > 0], bins=60, log=True, color="#3b6ea5")
    a.set_xlabel(f"{lab} (partners)"); a.set_ylabel("neurons (log)")
    a.set_title(f"median {np.median(data):.0f}, max {data.max():,}")
fig.tight_layout()

8. Predicted neurotransmitters

Feather format (pyarrow is already in this image). Two things will trip you up:

majority are fragments carrying statusLabel values like Unimportant or Leaves. Joining it to the connectome without filtering silently drags that junk in. Always restrict to the traced set first.

The values are predictions from EM ultrastructure, not measurements. They carry real error rates — treat them as a prior, especially for any individual cell.

nt_path = os.path.join(MOUNT, "hemibrain-v1.2-body-mean-neurotransmitters.feather")
nt = pd.read_feather(nt_path)
print(f"{len(nt):,} bodies in the table (index name: {nt.index.name!r})")
print(f"columns: {list(nt.columns)}\n")

# Restrict to the traced neurons — this is the join you actually want.
traced_nt = nt.loc[nt.index.intersection(neurons["bodyId"])]
print(f"{len(traced_nt):,} of {len(neurons):,} traced neurons have a prediction")
traced_nt["predicted_nt"].value_counts()

Output:

837,710 bodies in the table (index name: 'body')
columns: ['type', 'instance', 'statusLabel', 'gaba', 'acetylcholine', 'glutamate', 'serotonin', 'octopamine', 'dopamine', 'neither', 'predicted_nt']

21,709 of 21,739 traced neurons have a prediction
predicted_nt
acetylcholine    9577
glutamate        5342
dopamine         3183
gaba             3009
serotonin         282
octopamine        191
neither           125
Name: count, dtype: int64
Where to go next

v1.2 — state that in your methods.

brain) instead.

a Cypher API, often better than the flat files for narrow questions. It needs pip install neuprint-python and a free token — neither is in this image, so install into your own environment rather than this pod.

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.