# National Reef Monitoring Network - Survey metadata

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 439 objects, 4,866,001 bytes (4.87 MB)
- Records: 36,974 rows = 36,974 unique survey_id (sum of all 438 footers, 2026-10-04); 2025 partition 1,234 rows, 2026 partition 605 rows (to 2026-07-19)
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/diver_survey_metadata_qc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/997c2e15-b345-438f-afac-49a4ac19be38
- Page: https://scigantic.com/archive/af6e6517-586f-5547-b06e-9f70c9ba0764
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS National Reef Monitoring Network (NRMN) sub-facility brings together shallow reef surveys conducted around Australia into a centralised database, collating, cleaning and publishing data from Reef Life Survey (RLS), the Australian Temperate Reef Collaboration (ATRC), Parks Australia, DBCA (WA), DEW (SA), DPI (NSW), Tasmanian Parks and Wildlife and Parks Victoria. RLS and ATRC coordinate surveys of rocky and coral reefs using standard underwater visual census methods, carried out by marine scientists and trained recreational divers, to establish national marine baselines and support the management of Commonwealth and State marine reserves. This dataset provides the metadata associated with individual 50 m transects surveyed by RLS and ATRC divers and partners, all undertaken using standard methods. ATRC surveys are predominantly collected as a group of 4 x 50 m transects per site; RLS surveys commonly have a minimum of 2 x 50 m transects per site on a date (typically at different depths), but this number varies. It supersedes the RLS-specific 'Survey Locations' collection. This store is the quality-controlled survey list as hive-partitioned parquet: one row per transect survey, 36,974 rows with unique survey_id, 4,892 sites in 59 countries, 1992-03-10 to 2026-07-19, with the methods done, depth, visibility, photoquadrat links and flags; it is the master key for the four NRMN observation tables and the only place the 2026 South Australian surveys currently appear.

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], '_common_metadata'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-05 against `_common_metadata`: Parquet 0 rows, 31 columns, 0 row groups; columns ['TIME:timestamp[ns]', 'area:string', 'geom:string', 'hour:string', 'depth:float', 'country:string', 'methods:string', 'polygon:string', 'program:string', 'surface:string'].

From any machine:

```python
# Task: same SA subset over anonymous S3 from outside the pod
import pyarrow.dataset as ds, pyarrow.fs as pafs
s3 = pafs.S3FileSystem(anonymous=True, region='ap-southeast-2')
dset = ds.dataset('aodn-cloud-optimised/diver_survey_metadata_qc.parquet', filesystem=s3, format='parquet', partitioning='hive')
m = dset.to_table(filter=(ds.field('timestamp') >= 1735689600) & (ds.field('area') == 'South Australia') & (ds.field('survey_date') >= '2025-03-01'),
                  columns=['survey_id','site_code','site_name','location','survey_date','methods']).to_pandas()
print(len(m), m['site_code'].nunique(), m['survey_date'].min(), m['survey_date'].max())
print(m.groupby(m['survey_date'].str[:7]).size())
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `TIME` | timestamp[ns]. timestamp[ns], naive (no tz). Equals survey_date + hour where hour is present, survey_date at 00:00… |
| `area` | string. string. State/territory or sub-national region, e.g. "South Australia", "Tasmania", "Catalonia". 153 distinct… |
| `geom` | string. string. WKB hex POINT SRID 4326 in (longitude, latitude) order here (the observation tables swap the axes).… |
| `hour` | string. string "HH:MM:SS", documented local time; null on 16,529 rows (45%). 8% of non-null Australian values are… |
| `depth` | float. float32, metres, transect depth 0 to 42. |
| `country` | string. string. 59 distinct; Australia 31,859 surveys (86%). Null on 3. |
| `methods` | string. string. Comma-separated RLS/ATRC method codes run on this transect, e.g. "1, 2" (9,734 surveys), "1, 2, 3"… |
| `polygon` | string. string, hive partition key (not stored in the files). WKB hex of the 20 x 20 degree spatial cell the row is… |
| `program` | string. string. RLS 20,660, ATRC 13,720, Parks Vic 2,430, FRDC 128, RRH 36. |
| `surface` | string. string. "Coral" 1,059, "Rock" 419, null on 35,496 (96%). |
| `filename` | string. string. Constant "ep_survey_list_public_data.csv". |
| `latitude` | float. float32, degrees_north. Site latitude. |
| `location` | string. string. Named region within the area, e.g. "Encounter", "Yorke Peninsula", "Adelaide", "Kangaroo Island". 221… |
| `direction` | string. string. Free-text transect heading: N/S/E/W/NE/SW ... but also lowercase, "N ", "West", "S.W.", "Sam G"… |
| `longitude` | float. float32, degrees_east. Site longitude. |
| `site_code` | string. string. Site identifier (e.g. "GSV135"); 4,892 distinct. |
| `site_name` | string. string. 4,864 distinct names. |
| `survey_id` | int32. int32. UNIQUE per row (36,974 distinct = row count). Primary key; join target for the four observation tables. |
| `timestamp` | int64. int64 per _common_metadata, read back as int32 by the hive partitioning. Partition key = epoch seconds of 1… |
| `pq_zip_url` | string. string. URL of the photoquadrat zip for the survey, e.g. http://rls.tpac.org.au/pq/923406716/zip/ ; null on… |

and 11 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/af6e6517-586f-5547-b06e-9f70c9ba0764
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/af6e6517-586f-5547-b06e-9f70c9ba0764
- Over MCP: `get_schema_card` with id `af6e6517-586f-5547-b06e-9f70c9ba0764` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
