Understanding of Marine Imagery - Australian region - Aggregated kelp data product (2008 - ongoing)

Provider
AODN
License
http://creativecommons.org/licenses/by/4.0/
Size
20 objects, 2.6 MiB
Records
123,852 rows (counted over all 19 partitions on 2026-10-04); 24 columns in the schema (22 in the files + the two hive keys timestamp and polygon that come from the directory names)
Format
Parquet (in sampled files)
Stored at
Amazon S3, s3://aodn-cloud-optimised/aggregated_kelp_nonqc.parquet (ap-southeast-2)
Access
Free and public. No account with the data provider is needed.
Category
Life Sciences
Source
https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/20be281d-dfb2-4868-b690-c889a23b5714

About this dataset

The IMOS Understanding of Marine Imagery Sub-Facility provides a national repository for marine imagery and annotations, facilitating the sharing of quantitative data on species and habitat distributions. Squidle+ serves as the foundational software platform for this sub-facility, providing an end-to-end environment for the delivery, management and annotation of marine imagery. This product extracts the public kelp annotations from Squidle+ (filtered through a kelp label allow-list spanning the Australian Morphospecies Catalogue, CATAMI, the Reef Life Survey and Hunter extended RLS schemes and the BRUV/BOSS/DOV benthic habitat schema), aligns them to WoRMS taxonomy and publishes them as Darwin Core occurrence records; source imagery comes from numerous contributors including Reef Life Survey and Australian Temperate Reef Collaboration partners, and the aggregation was funded through NESP Marine and Coastal Hub Project 5.9. This store is the cloud-optimised Parquet copy: 123,852 presence-only kelp annotations from 2008-01-09 to 2026-08-13, one hive partition per year, 2.6 MiB in total, each row carrying a lat/lon point, a WoRMS scientific name and a link back to the annotated Squidle+ image.

How to load this dataset in Python

import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()

Run in a Scigantic notebook on 2026-10-04 against timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet: Parquet 3,655 rows, 22 columns, 4 row groups; columns ['class:string', 'genus:string', 'order:string', 'family:string', 'phylum:string', 'h3Index:string', 'kingdom:string', 'label.id:int64', 'eventDate:timestamp[ms]', 'taxonRank:string'].

From any machine

# Task: read the schema (and the GeoNetwork uuid) without opening any data partition
import pyarrow.parquet as pq, s3fs
fs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})
with fs.open('aodn-cloud-optimised/aggregated_kelp_nonqc.parquet/_common_metadata', 'rb') as f:
    schema = pq.read_schema(f)
print(schema.names)
print(schema.metadata[b'global_attributes'])

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: Parquet

Fields

eventDatetimestamp[ms] (pandas datetime64[ms], UTC per the technical document), not null; 2008-01-09T00:00:01 to…
scientificNamestring, not null; WoRMS-aligned name at the lowest rank the label allowed: 18 values, e.g. 'Ecklonia radiata' (49,972)…
scientificNameIDstring, not null; WoRMS LSID, e.g. 'urn:lsid:marinespecies.org:taxname:214344' (Ecklonia radiata)
scientificNameAuthorshipstring, not null; e.g. '(C.Agardh) J.Agardh, 1848'
taxonRankstring, not null; Species 78,171 / Class 44,968 / Genus 470 / Family 243
kingdomstring, not null; always 'Chromista'
phylumstring, not null; always 'Ochrophyta'
classstring, nullable; always 'Phaeophyceae' (no nulls observed)
orderstring, nullable; Laminariales 50,687, Fucales 28,171, Sporochnales 26; NULL for the 44,968 class-level rows
familystring, nullable; NULL for 44,968 class-level rows
genusstring, nullable; NULL for 45,211 rows (class- and family-level records)
verbatimIdentificationstring, not null; the original Squidle+ label text: 'Ecklonia radiata' 49,972, 'Brown' 44,939, 'Phyllospora' 18,893…
label.idint64, nullable (no nulls observed); Squidle+ internal label id, 29 distinct values; note the dot in the column name
occurrenceIDstring, not null; '<squidle annotation id>-<deployment id>', e.g. '1189945-4339'; the deployment id is the only…
basisOfRecordstring, not null; always 'MachineObservation' (hardcoded by the pipeline; the inputs are photographs annotated by…
occurrenceStatusstring, not null; always 'present' (presence-only product)
associatedMediastring, not null; Squidle+ iframe URL to the annotated image, e.g. https://squidle.org/iframe/geodata/annotate/5362995/2…
decimalLatitudedouble, EPSG:4326, not null; -43.71932 to -27.13051 (image pose latitude)
decimalLongitudedouble, EPSG:4326, not null; 113.38759 to 153.48322
h3Indexstring, not null; H3 cell at resolution 5 derived from lat/lon, e.g. '85bf4c0bfffffff'; 122 distinct cells

and 4 more

Analyze this dataset without downloading it

On Scigantic, Understanding of Marine Imagery - Australian region - Aggregated kelp data product (2008 - ongoing) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Life Sciences datasets