National Reef Monitoring Network - Global off-transect species observations

Provider
AODN
License
http://creativecommons.org/licenses/by/4.0/
Size
328 objects, 3,856,309 bytes (3.86 MB)
Records
18,571 rows (sum of all 327 footers, 2026-10-04); 2025 partition 1,153 rows, 2026 partition 186 rows
Format
Parquet (in sampled files)
Stored at
Amazon S3, s3://aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet (ap-southeast-2)
Access
Free and public. No account with the data provider is needed.
Category
Life Sciences
Source
https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/a73907b7-3686-4dcf-8a1d-7efc4e5ffc05

About National Reef Monitoring Network - Global off-transect species observations

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains unstandardised occurrence records of bony fishes and elasmobranchs (and some invertebrates, turtles and mammals): any sightings by RLS and ATRC divers that were outside the quantitative survey limits of the transect but can be considered as occurring at that site on the date of the survey they are listed with. It supersedes the RLS-specific 'Global reef fish dataset' off-transect records. This store is the quality-controlled Method 0 table as hive-partitioned parquet: 18,571 rows from 6,338 surveys at 2,506 sites, 2006-02-20 to 2026-05-29, presence-style records (median two per survey) rather than densities; 2,074 rows are South Australian.

How to load this dataset in Python

import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()

Run in a Scigantic notebook on 2026-10-04 against _common_metadata: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:string', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].

From any machine

# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','class','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())
print(sa['class'].value_counts())

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: Parquet

Fields

TIMEtimestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00…
areastring. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct…
geomstring. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in…
hourstring. string "HH:MM:SS"; null on 5,358 rows (29%).
blockstring. string (not int) in this table: null on 13,422 rows (72%), "0" 2,852, "1" 1,210, "2" 1,087. Not meaningful for…
classstring. string. 31 classes: Actinopterygii 15,422, Elasmobranchii 1,151, Gastropoda 326, Malacostraca 310, Reptilia…
depthfloat. float32, metres, 0.5 to 38.
orderstring. string. Taxonomic order.
realmstring. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct.
totalint32. int32. Individuals sighted; 0 to 10,000; 44 zero rows.
familystring. string. Taxonomic family.
methodint32. int32. Always 0 (M0 off-transect sighting) on all 18,571 rows.
phylumstring. string. 10 phyla, Chordata 17,144; null 127.
biomassfloat. float32, grams; null on 7,438 rows (40%); 0 to 7,161,824 g. Unstandardised, no effort.
countrystring. string. 59 distinct; "Australia" is 81% of fish rows.
polygonstring. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is…
programstring. string. RLS 18,435, ATRC 133, RRH 2, FRDC 1.
filenamestring. string. Source CSV name; a single constant value per table.
latitudefloat. float32, degrees_north. Site latitude (-67.57 to 67.44).
locationstring. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221…

and 13 more

Analyze this dataset without downloading it

On Scigantic, National Reef Monitoring Network - Global off-transect species observations opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Life Sciences datasets