# National Reef Monitoring Network - Global off-transect species observations

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 328 objects, 3,856,309 bytes (3.86 MB)
- Records: 18,571 rows (sum of all 327 footers, 2026-10-04); 2025 partition 1,153 rows, 2026 partition 186 rows
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/a73907b7-3686-4dcf-8a1d-7efc4e5ffc05
- Page: https://scigantic.com/archive/3cb491e8-c468-5d0c-8d41-838bade449c0
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset contains unstandardised occurrence records of bony fishes and elasmobranchs (and some invertebrates, turtles and mammals): any sightings by RLS and ATRC divers that were outside the quantitative survey limits of the transect but can be considered as occurring at that site on the date of the survey they are listed with. It supersedes the RLS-specific 'Global reef fish dataset' off-transect records. This store is the quality-controlled Method 0 table as hive-partitioned parquet: 18,571 rows from 6,338 surveys at 2,506 sites, 2006-02-20 to 2026-05-29, presence-style records (median two per survey) rather than densities; 2,074 rows are South Australian.

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-04 against `_common_metadata`: Parquet 0 rows, 33 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'block:string', 'class:string', 'depth:float', 'order:string', 'realm:string', 'total:int32'].

From any machine:

```python
# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_off_transect_species_observations_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
flt = ((ds.field('timestamp') >= 1735689600) & (ds.field('latitude') >= -36.5) & (ds.field('latitude') <= -32.0)
       & (ds.field('longitude') >= 135.0) & (ds.field('longitude') <= 139.5) & (ds.field('survey_date') >= '2025-03-01'))
sa = dset.to_table(filter=flt, columns=['survey_id','site_code','survey_date','species_name','class','total']).to_pandas()
print(len(sa), sa['survey_id'].nunique(), sa['survey_date'].max())
print(sa['class'].value_counts())
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `TIME` | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
| `area` | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
| `geom` | string. string. WKB hex of a POINT with SRID 4326 (prefix 0101000020E6100000). AXIS ORDER IS (latitude, longitude) in… |
| `hour` | string. string "HH:MM:SS"; null on 5,358 rows (29%). |
| `block` | string. string (not int) in this table: null on 13,422 rows (72%), "0" 2,852, "1" 1,210, "2" 1,087. Not meaningful for… |
| `class` | string. string. 31 classes: Actinopterygii 15,422, Elasmobranchii 1,151, Gastropoda 326, Malacostraca 310, Reptilia… |
| `depth` | float. float32, metres, 0.5 to 38. |
| `order` | string. string. Taxonomic order. |
| `realm` | string. string. Spalding MEOW realm, e.g. "Temperate Australasia". 12 distinct. |
| `total` | int32. int32. Individuals sighted; 0 to 10,000; 44 zero rows. |
| `family` | string. string. Taxonomic family. |
| `method` | int32. int32. Always 0 (M0 off-transect sighting) on all 18,571 rows. |
| `phylum` | string. string. 10 phyla, Chordata 17,144; null 127. |
| `biomass` | float. float32, grams; null on 7,438 rows (40%); 0 to 7,161,824 g. Unstandardised, no effort. |
| `country` | string. string. 59 distinct; "Australia" is 81% of fish rows. |
| `polygon` | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
| `program` | string. string. RLS 18,435, ATRC 133, RRH 2, FRDC 1. |
| `filename` | string. string. Source CSV name; a single constant value per table. |
| `latitude` | float. float32, degrees_north. Site latitude (-67.57 to 67.44). |
| `location` | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |

and 13 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/3cb491e8-c468-5d0c-8d41-838bade449c0
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/3cb491e8-c468-5d0c-8d41-838bade449c0
- Over MCP: `get_schema_card` with id `3cb491e8-c468-5d0c-8d41-838bade449c0` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
