# National Reef Monitoring Network - Global mobile macroinvertebrate abundance

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 438 objects, 9,265,408 bytes (9.3 MB)
- Records: 314,801 rows (sum of all 437 footers, 2026-10-04); 2025 partition 15,255 rows, 2026 partition 4,386 rows
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/48cf3cb9-caa9-4633-9baa-8bba3c4d904a
- Page: https://scigantic.com/archive/90c67c2e-600c-5f2f-b7da-7cc5ecafdc90
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of mobile macroinvertebrates collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for sea stars, urchins, gastropods, lobsters and crabs. Recorded concurrently with the cryptobenthic fish observations, these make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. This store is the quality-controlled M2 invertebrate table as hive-partitioned parquet: 314,801 rows from 36,081 surveys at 4,853 sites, 1992-03-10 to 2026-07-19, 34,861 of them South Australian (abalone Haliotis rubra and the urchin Heliocidaris erythrogramma lead the species list).

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-04 against `_common_metadata`: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].

From any machine:

```python
# Task: same SA subset from outside the pod over anonymous S3 with pyarrow's S3 client
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_mobile_macroinvertebrate_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].min(), sa['survey_date'].max())
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `TIME` | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
| `area` | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
| `geom` | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
| `hour` | string. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of… |
| `block` | int32. int32. 50 x 1 m swathe: 1 (197,141 rows), 2 (117,658), 0 (2 rows). RLS surveys average 1.94 blocks, ATRC 1.02… |
| `class` | string. string. Gastropoda 99,153 rows, Echinoidea 70,842, Asteroidea 47,633, Malacostraca 39,666, Crinoidea 27,916… |
| `depth` | float. float32, metres, 0 to 42. |
| `order` | string. string. Taxonomic order. |
| `realm` | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
| `total` | int32. int32. Individuals counted; 0 to 22,000. 228 zero rows, 207 of them the placeholder species_name "No species… |
| `family` | string. string. Taxonomic family. |
| `method` | int32. int32. Always 2 (M2 invertebrate/cryptic search) on all 314,801 rows. |
| `phylum` | string. string. Echinodermata, Mollusca, Arthropoda, Chordata (ascidians), Platyhelminthes, Cnidaria, Echiura; null on… |
| `biomass` | string. string in this table and NULL on all 314,801 rows. No invertebrate biomass is published; drop the column. |
| `country` | string. string. 59 distinct; "Australia" is 81% of fish rows. |
| `polygon` | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
| `program` | string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH. |
| `filename` | string. string. Source CSV name; a single constant value per table. |
| `latitude` | float. float32, degrees_north. Site latitude (float32 here, double in the fish table). |
| `location` | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |

and 13 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/90c67c2e-600c-5f2f-b7da-7cc5ecafdc90
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/90c67c2e-600c-5f2f-b7da-7cc5ecafdc90
- Over MCP: `get_schema_card` with id `90c67c2e-600c-5f2f-b7da-7cc5ecafdc90` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
