# Understanding of Marine Imagery - Australian region - Aggregated kelp data product (2008 - ongoing)

## Facts

- Provider: AODN
- License: http://creativecommons.org/licenses/by/4.0/
- Size: 20 objects, 2.6 MiB
- Records: 123,852 rows (counted over all 19 partitions on 2026-10-04); 24 columns in the schema (22 in the files + the two hive keys timestamp and polygon that come from the directory names)
- Format: Parquet (in sampled files)
- Stored at: Amazon S3, s3://aodn-cloud-optimised/aggregated_kelp_nonqc.parquet (ap-southeast-2)
- Access: public, readable without an account
- Category: Life Sciences
- Source: https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/20be281d-dfb2-4868-b690-c889a23b5714
- Page: https://scigantic.com/archive/8ad8539a-a9db-53ba-971a-6a40c1cba80a
- Keywords: Life Sciences, Parquet, AWS Open Data

## About

The IMOS Understanding of Marine Imagery Sub-Facility provides a national repository for marine imagery and annotations, facilitating the sharing of quantitative data on species and habitat distributions. Squidle+ serves as the foundational software platform for this sub-facility, providing an end-to-end environment for the delivery, management and annotation of marine imagery. This product extracts the public kelp annotations from Squidle+ (filtered through a kelp label allow-list spanning the Australian Morphospecies Catalogue, CATAMI, the Reef Life Survey and Hunter extended RLS schemes and the BRUV/BOSS/DOV benthic habitat schema), aligns them to WoRMS taxonomy and publishes them as Darwin Core occurrence records; source imagery comes from numerous contributors including Reef Life Survey and Australian Temperate Reef Collaboration partners, and the aggregation was funded through NESP Marine and Coastal Hub Project 5.9. This store is the cloud-optimised Parquet copy: 123,852 presence-only kelp annotations from 2008-01-09 to 2026-08-13, one hive partition per year, 2.6 MiB in total, each row carrying a lat/lon point, a WoRMS scientific name and a link back to the annotated Squidle+ image.

## Load the data

```python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
```

Run in a Scigantic notebook on 2026-10-04 against `timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet`: Parquet 3,655 rows, 22 columns, 4 row groups; columns ['class:string', 'genus:string', 'order:string', 'family:string', 'phylum:string', 'h3Index:string', 'kingdom:string', 'label.id:int64', 'eventDate:timestamp[ms]', 'taxonRank:string'].

From any machine:

```python
# Task: read the schema (and the GeoNetwork uuid) without opening any data partition
import pyarrow.parquet as pq, s3fs
fs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})
with fs.open('aodn-cloud-optimised/aggregated_kelp_nonqc.parquet/_common_metadata', 'rb') as f:
    schema = pq.read_schema(f)
print(schema.names)
print(schema.metadata[b'global_attributes'])
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: Parquet

| Field | Type |
| --- | --- |
| `eventDate` | timestamp[ms] (pandas datetime64[ms], UTC per the technical document), not null; 2008-01-09T00:00:01 to… |
| `scientificName` | string, not null; WoRMS-aligned name at the lowest rank the label allowed: 18 values, e.g. 'Ecklonia radiata' (49,972)… |
| `scientificNameID` | string, not null; WoRMS LSID, e.g. 'urn:lsid:marinespecies.org:taxname:214344' (Ecklonia radiata) |
| `scientificNameAuthorship` | string, not null; e.g. '(C.Agardh) J.Agardh, 1848' |
| `taxonRank` | string, not null; Species 78,171 / Class 44,968 / Genus 470 / Family 243 |
| `kingdom` | string, not null; always 'Chromista' |
| `phylum` | string, not null; always 'Ochrophyta' |
| `class` | string, nullable; always 'Phaeophyceae' (no nulls observed) |
| `order` | string, nullable; Laminariales 50,687, Fucales 28,171, Sporochnales 26; NULL for the 44,968 class-level rows |
| `family` | string, nullable; NULL for 44,968 class-level rows |
| `genus` | string, nullable; NULL for 45,211 rows (class- and family-level records) |
| `verbatimIdentification` | string, not null; the original Squidle+ label text: 'Ecklonia radiata' 49,972, 'Brown' 44,939, 'Phyllospora' 18,893… |
| `label.id` | int64, nullable (no nulls observed); Squidle+ internal label id, 29 distinct values; note the dot in the column name |
| `occurrenceID` | string, not null; '<squidle annotation id>-<deployment id>', e.g. '1189945-4339'; the deployment id is the only… |
| `basisOfRecord` | string, not null; always 'MachineObservation' (hardcoded by the pipeline; the inputs are photographs annotated by… |
| `occurrenceStatus` | string, not null; always 'present' (presence-only product) |
| `associatedMedia` | string, not null; Squidle+ iframe URL to the annotated image, e.g. https://squidle.org/iframe/geodata/annotate/5362995/2… |
| `decimalLatitude` | double, EPSG:4326, not null; -43.71932 to -27.13051 (image pose latitude) |
| `decimalLongitude` | double, EPSG:4326, not null; 113.38759 to 153.48322 |
| `h3Index` | string, not null; H3 cell at resolution 5 derived from lat/lon, e.g. '85bf4c0bfffffff'; 122 distinct cells |

and 4 more fields

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/8ad8539a-a9db-53ba-971a-6a40c1cba80a
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/8ad8539a-a9db-53ba-971a-6a40c1cba80a
- Over MCP: `get_schema_card` with id `8ad8539a-a9db-53ba-971a-6a40c1cba80a` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Life Sciences datasets](https://scigantic.com/topics/life-sciences)
