ExA-SPIM test acquisition 2026-08-04: four light-sheet tiles as OME-NGFF 0.5 / Zarr v3, sharded: sample notebook
A worked notebook for ExA-SPIM test acquisition 2026-08-04: four light-sheet tiles as OME-NGFF 0.5 / Zarr v3, sharded, with 5 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Allen Institute for Neural Dynamics (AIND)
- License
- CC-BY-4.0
- Size
- 282 objects, 16,519,295,393 B (16.5 GB)
- Stored at
- Amazon S3, s3://aind-open-data/exaSPIM_2026-08-04_10-27-41 (us-west-2)
- Access
- Free and public. No account with the data provider is needed.
The notebook
Start here. The outputs below are from an earlier run. Nothing is live until you run a cell yourself: click the grey cell below and press Shift+Enter. It opens one of the four Zarr v3 tiles mounted read-only at ~/archive, prints the codec pipeline that makes it a *sharded* store, and reads the coarsest level (one 251 KB shard) in about a second. Then change TILE to tile_000000_ch_561 and run it again.
# Start here: click this cell, press Shift+Enter.
import os, time
import zarr
TILE = "tile_000000_ch_488" # tile_000000 or tile_000001, channel 488 or 561
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH") or os.path.expanduser("~/archive")
g = zarr.open_group(f"{ROOT}/SPIM/{TILE}.ome.zarr", mode="r")
ome = g.attrs["ome"]
ms = ome["multiscales"][0]
print(f"{TILE}: Zarr v{g.metadata.zarr_format}, OME-NGFF {ome['version']}, axes {''.join(a['name'] for a in ms['axes'])}, {len(ms['datasets'])} levels")
for d in ms["datasets"]:
a = g[d["path"]]
z, y, x = d["coordinateTransformations"][0]["scale"][2:]
print(f" level {d['path']}: shape {a.shape[2:]} voxel {x:.2f} x {y:.2f} x {z:.0f} um shard {a.shards[2:]} inner chunk {a.chunks[2:]}")
sh = g["0"].metadata.codecs[0].to_dict()
print("codecs: sharding_indexed ->", " -> ".join(c["name"] for c in sh["configuration"]["codecs"]),
"| shard index:", " + ".join(c["name"] for c in sh["configuration"]["index_codecs"]))
t0 = time.time()
coarse = g[ms["datasets"][-1]["path"]][0, 0]
print(f"coarsest level read in {time.time() - t0:.1f} s: {coarse.shape} {coarse.dtype}, min {coarse.min()}, max {coarse.max()}, mean {coarse.mean():.1f}")
A sharded Zarr v3 light-sheet acquisition
This archive is the prefix exaSPIM_2026-08-04_10-27-41 of the public aind-open-data bucket: a small ExA-SPIM acquisition from the Allen Institute for Neural Dynamics, two stage positions by two channels, 16.5 GB in 282 objects. What makes it worth a notebook of its own is the container, not the sample: it is OME-NGFF 0.5 in Zarr v3, with sharding_indexed so a full-resolution tile is 84 shard objects of 70 to 110 MB instead of hundreds of thousands of small chunk files. That is the layout current light-sheet acquisition software writes, and the one to test a reader against.
What is in the mount:
| path | what it is |
| SPIM/tile_00000{0,1}_ch_488.ome.zarr/ | 6-level pyramids from (1024, 2660, 3548) voxels at 15.04 x 15.04 x 20 um (as recorded); 7.1 and 7.3 GB |
| SPIM/tile_00000{0,1}_ch_561.ome.zarr/ | 5-level pyramids from (512, 1330, 1774) at 30.08 x 30.08 x 40 um; 1.1 GB each |
| /zarr.json, / /zarr.json | the only metadata: no acquisition.json, subject.json or data_description.json at this prefix |
Every cell opens what it needs through the read-only mount at ~/archive (SCIGANTIC_MOUNT_PATH); the same code runs anywhere if ROOT is the bucket's HTTPS URL, which is how these outputs were made.
import json, os, time, urllib.request
import numpy as np
import zarr
import matplotlib.pyplot as plt
ROOT = os.environ.get("SCIGANTIC_MOUNT_PATH") or os.path.expanduser("~/archive")
HTTPS = "https://aind-open-data.s3.us-west-2.amazonaws.com/exaSPIM_2026-08-04_10-27-41"
tiles = {}
for name in ("tile_000000_ch_488", "tile_000000_ch_561", "tile_000001_ch_488", "tile_000001_ch_561"):
g = zarr.open_group(f"{ROOT}/SPIM/{name}.ome.zarr", mode="r")
ms = g.attrs["ome"]["multiscales"][0]
rows = []
for d in ms["datasets"]:
a = g[d["path"]]
grid = tuple(-(-s // c) for s, c in zip(a.shape[2:], a.shards[2:]))
rows.append((d["path"], a.shape[2:], a.shards[2:], grid, int(np.prod(grid))))
d0 = ms["datasets"][0]["coordinateTransformations"]
tiles[name] = dict(levels=rows, scale=d0[0]["scale"][2:], translation=d0[1]["translation"][2:], method=ms.get("type"), name=ms.get("name"),
window=g.attrs["ome"].get("omero", {}).get("channels", [{}])[0].get("window"))
print(f"{name}: downsampling '{tiles[name]['method']}', level-0 voxel {tiles[name]['scale'][2]:.2f} x {tiles[name]['scale'][1]:.2f} x {tiles[name]['scale'][0]:.0f} um, "
f"translation (z, y, x) {tuple(round(v, 1) for v in tiles[name]['translation'])} um, omero window {tiles[name]['window']}")
for path, shape, shard, grid, n in rows:
print(f" level {path}: {shape} shard grid {grid} = {n} object(s)")
print("\nboth stage positions report the same translation:",
tiles["tile_000000_ch_488"]["translation"] == tiles["tile_000001_ch_488"]["translation"])
Output:
tile_000000_ch_488: downsampling 'mode', level-0 voxel 15.04 x 15.04 x 20 um, translation (z, y, x) (19520.0, -60276.8, -37412.7) um, omero window {'end': 1.0, 'max': 1.0, 'min': 0.0, 'start': 0.0}
level 0: (1024, 2660, 3548) shard grid (2, 6, 7) = 84 object(s)
level 1: (512, 1330, 1774) shard grid (1, 3, 4) = 12 object(s)
level 2: (256, 665, 887) shard grid (1, 2, 2) = 4 object(s)
level 3: (128, 333, 444) shard grid (1, 2, 2) = 4 object(s)
level 4: (64, 167, 222) shard grid (1, 1, 1) = 1 object(s)
level 5: (32, 84, 111) shard grid (1, 1, 1) = 1 object(s)
tile_000000_ch_561: downsampling 'mode', level-0 voxel 30.08 x 30.08 x 40 um, translation (z, y, x) (19520.0, -60276.8, -37412.7) um, omero window {'end': 1.0, 'max': 1.0, 'min': 0.0, 'start': 0.0}
level 0: (512, 1330, 1774) shard grid (1, 3, 4) = 12 object(s)
level 1: (256, 665, 887) shard grid (1, 2, 2) = 4 object(s)
level 2: (128, 333, 444) shard grid (1, 2, 2) = 4 object(s)
level 3: (64, 167, 222) shard grid (1, 1, 1) = 1 object(s)
level 4: (32, 84, 111) shard grid (1, 1, 1) = 1 object(s)
tile_000001_ch_488: downsampling 'mode', level-0 voxel 15.04 x 15.04 x 20 um, translation (z, y, x) (19520.0, -60276.8, -37412.7) um, omero window {'end': 1.0, 'max': 1.0, 'min': 0.0, 'start': 0.0}
level 0: (1024, 2660, 3548) shard grid (2, 6, 7) = 84 object(s)
...
Inside one shard
A shard is one object holding a grid of inner chunks plus an index at its end: for every inner chunk, a 64-bit offset and a 64-bit byte count, then a CRC32C of the index. A reader fetches the last 16 * n + 4 bytes of the object, then exactly the inner chunk it wants. The cell below does this by hand with one HTTP range request on the first level-0 shard of tile_000000_ch_488 (a 72 MB object), to show what zarr does for you.
shard_key = "SPIM/tile_000000_ch_488.ome.zarr/0/c/0/0/0/0/0"
a0 = zarr.open_group(f"{ROOT}/SPIM/tile_000000_ch_488.ome.zarr", mode="r")["0"]
n_inner = int(np.prod([s // c for s, c in zip(a0.shards, a0.chunks)]))
index_bytes = 16 * n_inner + 4
req = urllib.request.Request(f"{HTTPS}/{shard_key}", headers={"Range": f"bytes=-{index_bytes}"})
with urllib.request.urlopen(req, timeout=60) as r:
total = int(r.headers["Content-Range"].split("/")[1]); tail = r.read()
idx = np.frombuffer(tail[:-4], dtype="<u8").reshape(n_inner, 2)
print(f"shard object: {total/1e6:.1f} MB; shard {a0.shards[2:]} = {n_inner} inner chunks of {a0.chunks[2:]}; index = {index_bytes} bytes")
print(f"inner chunk sizes: min {idx[:,1].min()/1e6:.1f} MB, max {idx[:,1].max()/1e6:.1f} MB, sum {idx[:,1].sum()/1e6:.1f} MB "
f"(raw {np.prod(a0.chunks) * 2 / 1e6:.1f} MB each -> zstd ratio {np.prod(a0.chunks) * 2 * n_inner / idx[:,1].sum():.1f}x)")
print("first four (offset, nbytes):", [tuple(int(v) for v in row) for row in idx[:4]])
t0 = time.time()
blk = a0[0, 0, 0:128, 0:256, 0:256] # the first inner chunk of that shard, through zarr
print(f"zarr partial-shard read of one inner chunk {blk.shape}: {time.time() - t0:.1f} s, max {blk.max()}, mean {blk.mean():.1f}")
Output:
shard object: 72.1 MB; shard (512, 512, 512) = 16 inner chunks of (128, 256, 256); index = 260 bytes inner chunk sizes: min 4.5 MB, max 4.5 MB, sum 72.1 MB (raw 16.8 MB each -> zstd ratio 3.7x) first four (offset, nbytes): [(0, 4512604), (4512604, 4505050), (9017654, 4506801), (13524455, 4499141)] zarr partial-shard read of one inner chunk (128, 256, 256): 1.0 s, max 1328, mean 1031.5
What the sample looks like
Level 3 of a 488 tile and level 2 of the matching 561 tile share the same voxel size (120 x 120 x 160 um), so a maximum projection of each is a like-for-like pair. Each is a single shard of a few megabytes.
def mip(name, level):
return np.asarray(zarr.open_group(f"{ROOT}/SPIM/{name}.ome.zarr", mode="r")[level][0, 0]).max(axis=0)
t0 = time.time()
pairs = [("tile_000000_ch_488", "3"), ("tile_000000_ch_561", "2"), ("tile_000001_ch_488", "3"), ("tile_000001_ch_561", "2")]
imgs = {(n, l): mip(n, l) for n, l in pairs}
print(f"four projections in {time.time() - t0:.1f} s; shapes {[v.shape for v in imgs.values()]}")
fig, axes = plt.subplots(1, 4, figsize=(6.8, 2.1), dpi=72)
for ax, ((n, l), im) in zip(axes, imgs.items()):
lo, hi = np.percentile(im, [1, 99.7])
ax.imshow(im, cmap="gray", vmin=lo, vmax=hi); ax.set_title(f"{n[5:11]} {n[-3:]} nm, L{l}", fontsize=8); ax.axis("off")
plt.tight_layout(); plt.show()
Output:
four projections in 3.8 s; shapes [(333, 444), (333, 444), (333, 444), (333, 444)]
Where to go next
- Neuroglancer reads Zarr v3 directly (zarr3:// source, sharding included); the link
below streams the tile from the public bucket.
- Positions. Both tiles record the same translation, so this store cannot be laid out
from its own metadata; treat it as two independent volumes.
- Scale. The voxel sizes are what the metadata says (15.04 um at level 0 for 488, 30.08 um
for 561); there is no acquisition.json here to cross-check them against.
- The whole prefix is 16.5 GB, so even level 0 of a 488 tile (7 GB on disk, 19 GB raw) should be
read a shard or a slab at a time in this pod.
from urllib.parse import quote
from IPython.display import HTML, display
state = {
"dimensions": {"x": [15.04e-6, "m"], "y": [15.04e-6, "m"], "z": [20e-6, "m"]},
"layers": [{"type": "image", "source": f"zarr3://{HTTPS}/SPIM/{TILE}.ome.zarr/", "name": TILE,
"shader": "#uicontrol invlerp normalized(range=[0,4000])\nvoid main(){emitGrayscale(normalized());}"}],
"layout": "4panel",
}
url = "https://neuroglancer-demo.appspot.com/#!" + quote(json.dumps(state))
display(HTML(f'<a href="{url}" target="_blank" rel="noopener">Open {TILE} in Neuroglancer (Zarr v3 source, streams from the public bucket)</a>'))
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.