# National Reef Monitoring Network - Global reef fish abundance and biomass

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 439 objects, 24,471,206 bytes (24.5 MB)
- Records: 1,913,820 rows (sum of all 438 file footers, 2026-10-04); 2025 partition 82,719 rows, 2026 partition 19,344 rows (year to 2026-07-19)
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/b273fafa-03d6-4fc2-9acf-39d8c06581e5
- Page: https://scigantic.com/archive/4de33273-e822-59eb-8841-277d3f6df5ff
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of bony fishes and elasmobranchs collected by RLS and ATRC divers along 50 m transects on shallow rocky and coral reefs: abundance for every species recorded within quantitative survey limits (50 x 5 m swathes either side of the transect line, each a 'Block'), with size class and estimated biomass where available. These observations form the Method 1 component of the surveys and supersede the RLS-specific 'Global reef fish dataset'. This store is the quality-controlled M1 table as hive-partitioned parquet: 1,913,820 rows from 36,339 transect surveys at 4,875 sites in 59 countries, 1992-03-10 to 2026-07-19, of which 96,876 rows are South Australian.

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-04 against `_common_metadata`: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:double', 'order:string', 'realm:string', 'total:int32'].

From any machine:

```python
# Task: the same SA-gulfs-since-2025-03 subset from outside the pod, anonymous S3 (pyarrow's own S3 client, no s3fs needed)
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_reef_fish_abundance_biomass_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','site_name','survey_date','species_name','size_class','total','biomass']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())
print(sa.groupby('species_name')['total'].sum().sort_values(ascending=False).head(10))
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `TIME` | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
| `area` | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
| `geom` | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
| `hour` | string. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of… |
| `block` | int32. int32. Which 50 x 5 m swathe of the transect: 1 (974,362 rows) or 2 (939,458 rows); 98% of surveys have both… |
| `class` | string. string. Actinopterygii 1,892,118 rows, Teleostei 10,078, Elasmobranchii 8,580, Cephalopoda 1,556, Reptilia 666… |
| `depth` | double. double (fish) / float (others), metres. Transect depth, 0 to 42 m. |
| `order` | string. string. Taxonomic order. |
| `realm` | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
| `total` | int32. int32. Individuals counted for this species x size class x block; 0 to 800,000 (one row); exactly one zero row. |
| `family` | string. string. Taxonomic family. |
| `method` | int32. int32. RLS/ATRC method code: 1 on 1,913,293 rows (M1 fish transect). 527 rows carry method 10 (50 ATRC surveys… |
| `phylum` | string. string. Taxonomic phylum. |
| `biomass` | float. float, grams. Estimated biomass of all `total` individuals in the row (length-weight on size_class): median 24.2… |
| `country` | string. string. 59 distinct; Australia 1,546,047 rows (81%), then Indonesia, Spain, United States, Panama. Null on 222… |
| `polygon` | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
| `program` | string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH. |
| `filename` | string. string. Source CSV name; a single constant value per table. |
| `latitude` | double. double (fish) / float (others), degrees_north WGS84. Site latitude. |
| `location` | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |

and 13 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/4de33273-e822-59eb-8841-277d3f6df5ff
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/4de33273-e822-59eb-8841-277d3f6df5ff
- Over MCP: `get_schema_card` with id `4de33273-e822-59eb-8841-277d3f6df5ff` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
