National Reef Monitoring Network - Global mobile macroinvertebrate abundance
- Provider
- AODN
- License
- http://creativecommons.org/licenses/by/4.0/
- Size
- 438 objects, 9,265,408 bytes (9.3 MB)
- Records
- 314,801 rows (sum of all 437 footers, 2026-10-04); 2025 partition 15,255 rows, 2026 partition 4,386 rows
- Format
- Parquet (in sampled files)
- Stored at
- Amazon S3, s3://aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet (ap-southeast-2)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Life Sciences
- Source
- https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/48cf3cb9-caa9-4633-9baa-8bba3c4d904a
About National Reef Monitoring Network - Global mobile macroinvertebrate abundance
The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of mobile macroinvertebrates collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for sea stars, urchins, gastropods, lobsters and crabs. Recorded concurrently with the cryptobenthic fish observations, these make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. This store is the quality-controlled M2 invertebrate table as hive-partitioned parquet: 314,801 rows from 36,081 surveys at 4,853 sites, 1992-03-10 to 2026-07-19, 34,861 of them South Australian (abalone Haliotis rubra and the urchin Heliocidaris erythrogramma lead the species list).
How to load this dataset in Python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()Run in a Scigantic notebook on 2026-10-04 against _common_metadata: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].
From any machine
# Task: same SA subset from outside the pod over anonymous S3 with pyarrow's S3 client
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
& (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].min(), sa['survey_date'].max())Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: Parquet
Fields
TIME | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
area | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
geom | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
hour | string. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of… |
block | int32. int32. 50 x 1 m swathe: 1 (197,141 rows), 2 (117,658), 0 (2 rows). RLS surveys average 1.94 blocks, ATRC 1.02… |
class | string. string. Gastropoda 99,153 rows, Echinoidea 70,842, Asteroidea 47,633, Malacostraca 39,666, Crinoidea 27,916… |
depth | float. float32, metres, 0 to 42. |
order | string. string. Taxonomic order. |
realm | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
total | int32. int32. Individuals counted; 0 to 22,000. 228 zero rows, 207 of them the placeholder species_name "No species… |
family | string. string. Taxonomic family. |
method | int32. int32. Always 2 (M2 invertebrate/cryptic search) on all 314,801 rows. |
phylum | string. string. Echinodermata, Mollusca, Arthropoda, Chordata (ascidians), Platyhelminthes, Cnidaria, Echiura; null on… |
biomass | string. string in this table and NULL on all 314,801 rows. No invertebrate biomass is published; drop the column. |
country | string. string. 59 distinct; "Australia" is 81% of fish rows. |
polygon | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
program | string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH. |
filename | string. string. Source CSV name; a single constant value per table. |
latitude | float. float32, degrees_north. Site latitude (float32 here, double in the fish table). |
location | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |
and 13 more
Analyze this dataset without downloading it
On Scigantic, National Reef Monitoring Network - Global mobile macroinvertebrate abundance opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Life Sciences datasets
- Opioid Industry Documents Archive (OIDA) Data on AWS
- Pacific Ocean Sound Recordings
- Cell Painting Gallery cpg0026: Protein Mislocalization of 3,448 Missense Variants in HeLa
- BIOSCAN-5M: Multimodal Insect Images and DNA Barcodes
- OpenOrganelle jrc_ctl-id8-1: Cytotoxic T Lymphocyte Attacking an Ovarian Cancer Cell, FIB-SEM
- KABR: Kenyan Animal Behavior Recognition from Drone Videos
- Cell Painting Gallery cpg0021: PERISCOPE Genome-Wide Pooled Optical CRISPR Screens in A549 and HeLa
- Emotions and Courtship in Pair Bonds: Agent-Based Evolutionary Simulations of the Iterated Prisoner's Dilemma (Google DeepMind, PNAS 2023)