National Reef Monitoring Network - Global cryptobenthic fish abundance
- Provider
- AODN
- License
- http://creativecommons.org/licenses/by/4.0/
- Size
- 436 objects, 7,576,126 bytes (7.6 MB)
- Records
- 168,994 rows (sum of all 435 footers, 2026-10-04); 2025 partition 13,587 rows, 2026 partition 2,355 rows
- Format
- Parquet (in sampled files)
- Stored at
- Amazon S3, s3://aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet (ap-southeast-2)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Life Sciences
- Source
- https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/9efa25cd-4da4-47b5-9385-45e3cbd11705
About National Reef Monitoring Network - Global cryptobenthic fish abundance
The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of cryptobenthic fishes collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for hidden fishes. These observations are recorded concurrently with the macroinvertebrate observations and together make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. It supersedes the RLS-specific 'Cryptic Fish' collection. This store is the quality-controlled M2 cryptic-fish table as hive-partitioned parquet: 168,994 rows from 23,239 surveys at 4,483 sites, 1992-03-10 to 2026-07-19, with size class and biomass on most rows; 7,587 rows are South Australian.
How to load National Reef Monitoring Network - Global cryptobenthic fish abundance in Python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()Run in a Scigantic notebook on 2026-10-05 against _common_metadata: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].
From any machine
# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
& (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: Parquet
Fields
TIME | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
area | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
geom | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
hour | string. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of… |
block | int32. int32. 50 x 1 m swathe: 1 (92,281 rows) or 2 (76,713). |
class | string. string. Actinopterygii 161,741, Teleostei 4,484, Chondrichthyes 2,390, Reptilia 81, Actinopteri 80; null 218. |
depth | float. float32, metres, 0 to 42. |
order | string. string. Taxonomic order. |
realm | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
total | int32. int32. Individuals counted; 0 to 10,000; 207 zero rows. |
family | string. string. Taxonomic family. |
method | int32. int32. Always 2 (M2) on all 168,994 rows. |
phylum | string. string. Chordata on all non-null rows; null on 218. |
biomass | float. float32, grams, for all `total` individuals in the row. Non-null on 155,800 rows (92%); null on 13,194 (7.8%).… |
country | string. string. 59 distinct; "Australia" is 81% of fish rows. |
polygon | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
program | string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH. |
filename | string. string. Source CSV name; a single constant value per table. |
latitude | float. float32, degrees_north. Site latitude. |
location | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |
and 13 more
Analyze this dataset without downloading it
On Scigantic, National Reef Monitoring Network - Global cryptobenthic fish abundance opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Life Sciences datasets
- Cell Painting Gallery cpg0016: JUMP Cell Painting Production Dataset, 116,000 Compounds and 15,000 Genes
- Pacific Ocean Sound Recordings
- Human and Mammalian Brain Atlas
- Pacific Ocean Sound Recordings
- IDLE-OO Camera Traps: Balanced Five-Set Species Classification Benchmark from LILA BC Camera Trap Images (Imageomics, BioCLIP 2)
- Daikinrin Trait Circus: 5.8 Million Controlled-Vocabulary Fungal Morphology and Ecology Traits Extracted from Species Descriptions (Japanese and English)
- Sounds of Central African landscapes
- OpenOrganelle jrc_hela-2: Whole Interphase HeLa Cell, 4 nm Isotropic FIB-SEM with Organelle Ground Truth