# National Reef Monitoring Network - Global cryptobenthic fish abundance

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 436 objects, 7,576,126 bytes (7.6 MB)
- Records: 168,994 rows (sum of all 435 footers, 2026-10-04); 2025 partition 13,587 rows, 2026 partition 2,355 rows
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/9efa25cd-4da4-47b5-9385-45e3cbd11705
- Page: https://scigantic.com/archive/c6438623-f358-59f3-affe-d0f485d57c26
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains records of cryptobenthic fishes collected by RLS and ATRC divers along 50 m transects: abundance for all species recorded within quantitative survey limits (50 x 1 m swathes either side of the transect line, each a 'Block'), with divers searching the reef surface including cracks for hidden fishes. These observations are recorded concurrently with the macroinvertebrate observations and together make up the 'Method 2' component; typically one block per transect for ATRC and two for RLS. It supersedes the RLS-specific 'Cryptic Fish' collection. This store is the quality-controlled M2 cryptic-fish table as hive-partitioned parquet: 168,994 rows from 23,239 surveys at 4,483 sites, 1992-03-10 to 2026-07-19, with size class and biomass on most rows; 7,587 rows are South Australian.

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-05 against `_common_metadata`: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:int32', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].

From any machine:

```python
# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_cryptobenthic_fish_abundance_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `TIME` | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
| `area` | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
| `geom` | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
| `hour` | string. string "HH:MM:SS", documented as local time of sample collection. Null on 32% (fish) to 29% (off-transect) of… |
| `block` | int32. int32. 50 x 1 m swathe: 1 (92,281 rows) or 2 (76,713). |
| `class` | string. string. Actinopterygii 161,741, Teleostei 4,484, Chondrichthyes 2,390, Reptilia 81, Actinopteri 80; null 218. |
| `depth` | float. float32, metres, 0 to 42. |
| `order` | string. string. Taxonomic order. |
| `realm` | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
| `total` | int32. int32. Individuals counted; 0 to 10,000; 207 zero rows. |
| `family` | string. string. Taxonomic family. |
| `method` | int32. int32. Always 2 (M2) on all 168,994 rows. |
| `phylum` | string. string. Chordata on all non-null rows; null on 218. |
| `biomass` | float. float32, grams, for all `total` individuals in the row. Non-null on 155,800 rows (92%); null on 13,194 (7.8%).… |
| `country` | string. string. 59 distinct; "Australia" is 81% of fish rows. |
| `polygon` | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
| `program` | string. string. Contributing program: RLS, ATRC, Parks Vic, FRDC, RRH. |
| `filename` | string. string. Source CSV name; a single constant value per table. |
| `latitude` | float. float32, degrees_north. Site latitude. |
| `location` | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |

and 13 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/c6438623-f358-59f3-affe-d0f485d57c26
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/c6438623-f358-59f3-affe-d0f485d57c26
- Over MCP: `get_schema_card` with id `c6438623-f358-59f3-affe-d0f485d57c26` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
