EMPIAR: Electron Microscopy Public Image Archive: sample notebook
A worked notebook for EMPIAR: Electron Microscopy Public Image Archive, with 9 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- EMBL-EBI (EMPIAR)
- License
- Per-entry (CC0 / CC-BY) — see each EMPIAR entry
- Stored at
- ftp.ebi.ac.uk
- Access
- Free and public. No account with the data provider is needed.
- DOI
- 10.6019/EMPIAR
The notebook
3 - Working with a full dataset
Previewing streams from EBI, which is what you want for *looking*. To *reprocess* an entry (motion correction, CTF, particle picking, sub-tomogram averaging) you need the whole thing, and entries run from 100 GB to several TB.
Scigantic does not fetch those for you yet. EMPIAR's own bulk channels are the right route, and Globus is their preferred one:
import scigantic_empiar as se
from scigantic_empiar import preview, add_to_fast_workspace, EmpiarClient, EmpiarCatalog
cat, api = EmpiarCatalog(), EmpiarClient()
print('EMPIAR mounted at', se.MOUNT)
print(f'catalog: {len(cat.load()):,} entries indexed')
1 - Search every entry, by science
EMPIAR titles describe the *experiment* ("Cryo electron microscopy of ..."), not the molecule. The protein names, organism, resolution and per-chain molecular weight live in the linked EMDB entry, so the catalog pulls them in and indexes them together. That means you can search the way you think.
Queries expand known synonyms, so "GPCR" matches receptor-G protein complexes even though neither EMPIAR nor EMDB ever writes the word.
cat.search('GPCR', limit=8)
# Search by protein, organism, or accession - all indexed, no live API calls.
cat.search('ribosome', organism='Saccharomyces', limit=5)
Filter on what decides whether a dataset is usable
Picking a benchmark set is rarely "find me a receptor". It is closer to *a receptor under 100 kDa, better than 3.5 Å, with half-maps deposited so I can compute FSCs*. That is one call:
hits = cat.search('GPCR', max_chain_kda=100, max_res=3.5, half_maps=True, sort='resolution')
hits[['id', 'sample_name', 'resolution_a', 'max_chain_kda', 'complex_kda', 'size_gb', 'has_mask']]
max_chain_kda is the largest single polymer, which is what you mean by "a 50 kDa receptor". complex_kda is the assembled weight, and a receptor-G protein complex is essentially always over 100 kDa even when the receptor itself is small. Filtering the wrong one is the easiest way to get an empty table.
Other bounds: min_gb / max_gb, min_res / max_res, complex_kda_max, mask=, has_emdb=, method=, organism=, and sort='resolution' | 'size' | 'id'.
# A thumbnail gallery of any result set (entries render as they are indexed).
cat.gallery(cat.search('tomogram', limit=8))
2 - Preview any entry, live, in seconds
preview(id) streams just enough of an entry's raw MRC data, over 8-way parallel range reads, to render the actual micrograph and its power spectrum. Try the classic EMPIAR-10002, a *S. cerevisiae* 80S ribosome dataset (260 GB). You will pull ~67 MB, not 260 GB.
preview(10002) # a real cryo-EM micrograph + FFT, pulled in ~seconds
Change the ID to any of ~3,000 entries. A few worth a look:
- preview(10406) - motion-corrected micrographs + extracted particles (small, fast)
- preview(10164) - cryo-ET of immature HIV-1 particles (tomography)
- preview(10005) - TRPV1, the 'resolution revolution' dataset
For a cleaner (heavier) image, preview(10002, average=True) averages several movie frames.
preview(10406)
3 - Working with a full dataset
Previewing streams from EBI, which is what you want for *looking*. To *reprocess* an entry (motion correction, CTF, particle picking, sub-tomogram averaging) you need the whole thing, and entries run from 100 GB to several TB.
Scigantic does not fetch those for you yet. EMPIAR's own bulk channels are the right route, and Globus is their preferred one:
# Reports whether a mirrored copy already exists. It does not start a transfer.
se.add_to_fast_workspace(10289) # -> local path if mirrored, else None
# For the full dataset, use EMPIAR's bulk download (Globus preferred, then Aspera):
print('https://www.ebi.ac.uk/empiar/EMPIAR-10289/')
Provenance: raw to map to model to paper
Every EMPIAR entry links to the 3D EMDB map it produced and the publication behind it. The catalog already carries the cross-references, so the whole chain is one lookup.
row = cat.search('80S ribosome', limit=1).iloc[0]
print('title: ', row['title'])
print('EMDB maps: ', row.get('emdb_ids'))
print('resolution: ', row.get('resolution_a'), 'A')
# The paper comes from the EMPIAR record itself (`citation` is a list).
e = api.entry(row['id'])
for c in (e.get('citation') or []):
authors = ', '.join(a.get('name', '') for a in (c.get('authors') or [])[:3])
print(f"paper: {c.get('title')}\n {authors} et al., {c.get('journal')} {c.get('year')} ({c.get('doi')})")
print('data DOI: ', e.get('entry_doi'))
Make it yours
This is a data browser and a compute environment for all of EMPIAR. Fork this notebook, point it at any entry, pull the frames you need, and run your pipeline. Then do the same with your own data.
The AI panel on the right knows this archive and can drive the catalog for you. Try *"build me a table of GPCR datasets under 100 kDa with half-maps, sorted by resolution"* or *"what's the pixel size of 10164?"*.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.