National Reef Monitoring Network - Global cryptobenthic fish abundance

Provider
AODN
License
http://creativecommons.org/licenses/by/4.0/
Size
436 objects, 7,576,126 bytes (7.6 MB)
Records
168,994 rows (sum of all 435 footers, 2026-10-04); 2025 partition 13,587 rows, 2026 partition 2,355 rows
Format
Parquet (in sampled files)
Stored at
Amazon S3, s3://aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet (ap-southeast-2)
Access
Free and public. No account with the data provider is needed.
Category
Life Sciences
Source
https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/9efa25cd-4da4-47b5-9385-45e3cbd11705

About National Reef Monitoring Network - Global cryptobenthic fish abundance

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of cryptobenthic fishes collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for hidden fishes. These observations are recorded concurrently with the macroinvertebrate observations and together make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. It supersedes the RLS-specific 'Cryptic Fish' collection. This store is the quality-controlled M2 cryptic-fish table as hive-partitioned parquet: 168,994 rows from 23,239 surveys at 4,483 sites, 1992-03-10 to 2026-07-19, with size class and biomass on most rows; 7,587 rows are South Australian.

How to load National Reef Monitoring Network - Global cryptobenthic fish abundance in Python

import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()

Run in a Scigantic notebook on 2026-10-05 against _common_metadata: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].

From any machine

# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: Parquet

Fields

TIMEtimestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00…
areastring. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct…
geomstring. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in…
hourstring. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of…
blockint32. int32. 50 x 1 m swathe: 1 (92,281 rows) or 2 (76,713).
classstring. string. Actinopterygii 161,741, Teleostei 4,484, Chondrichthyes 2,390, Reptilia 81, Actinopteri 80; null 218.
depthfloat. float32, metres, 0 to 42.
orderstring. string. Taxonomic order.
realmstring. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct.
totalint32. int32. Individuals counted; 0 to 10,000; 207 zero rows.
familystring. string. Taxonomic family.
methodint32. int32. Always 2 (M2) on all 168,994 rows.
phylumstring. string. Chordata on all non-null rows; null on 218.
biomassfloat. float32, grams, for all `total` individuals in the row. Non-null on 155,800 rows (92%); null on 13,194 (7.8%).…
countrystring. string. 59 distinct; "Australia" is 81% of fish rows.
polygonstring. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is…
programstring. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH.
filenamestring. string. Source CSV name; a single constant value per table.
latitudefloat. float32, degrees_north. Site latitude.
locationstring. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221…

and 13 more

Analyze this dataset without downloading it

On Scigantic, National Reef Monitoring Network - Global cryptobenthic fish abundance opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Life Sciences datasets