Understanding of Marine Imagery - Australian region - Aggregated kelp data product (2008 - ongoing)
- Provider
- AODN
- License
- http://creativecommons.org/licenses/by/4.0/
- Size
- 20 objects, 2.6 MiB
- Records
- 123,852 rows (counted over all 19 partitions on 2026-10-04); 24 columns in the schema (22 in the files + the two hive keys timestamp and polygon that come from the directory names)
- Format
- Parquet (in sampled files)
- Stored at
- Amazon S3, s3://aodn-cloud-optimised/aggregated_kelp_nonqc.parquet (ap-southeast-2)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Life Sciences
- Source
- https://catalogue-imos.aodn.org.au/geonetwork/srv/eng/catalog.search#/metadata/20be281d-dfb2-4868-b690-c889a23b5714
About this dataset
The IMOS Understanding of Marine Imagery Sub-Facility provides a national repository for marine imagery and annotations, facilitating the sharing of quantitative data on species and habitat distributions. Squidle+ serves as the foundational software platform for this sub-facility, providing an end-to-end environment for the delivery, management and annotation of marine imagery. This product extracts the public kelp annotations from Squidle+ (filtered through a kelp label allow-list spanning the Australian Morphospecies Catalogue, CATAMI, the Reef Life Survey and Hunter extended RLS schemes and the BRUV/BOSS/DOV benthic habitat schema), aligns them to WoRMS taxonomy and publishes them as Darwin Core occurrence records; source imagery comes from numerous contributors including Reef Life Survey and Australian Temperate Reef Collaboration partners, and the aggregation was funded through NESP Marine and Coastal Hub Project 5.9. This store is the cloud-optimised Parquet copy: 123,852 presence-only kelp annotations from 2008-01-09 to 2026-08-13, one hive partition per year, 2.6 MiB in total, each row carrying a lat/lon point, a WoRMS scientific name and a link back to the annotated Squidle+ image.
How to load this dataset in Python
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()Run in a Scigantic notebook on 2026-10-04 against timestamp=1199145600/polygon=01030000000100000005000000000000000000000000000000008056C0000000000080664000000000008056C00000000000806640000000000080564000000000000000000000000000805640000000000000000000000000008056C0/kelp.parquet-0.parquet: Parquet 3,655 rows, 22 columns, 4 row groups; columns ['class:string', 'genus:string', 'order:string', 'family:string', 'phylum:string', 'h3Index:string', 'kingdom:string', 'label.id:int64', 'eventDate:timestamp[ms]', 'taxonRank:string'].
From any machine
# Task: read the schema (and the GeoNetwork uuid) without opening any data partition
import pyarrow.parquet as pq, s3fs
fs = s3fs.S3FileSystem(anon=True, client_kwargs={'region_name': 'ap-southeast-2'})
with fs.open('aodn-cloud-optimised/aggregated_kelp_nonqc.parquet/_common_metadata', 'rb') as f:
schema = pq.read_schema(f)
print(schema.names)
print(schema.metadata[b'global_attributes'])Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: Parquet
Fields
eventDate | timestamp[ms] (pandas datetime64[ms], UTC per the technical document), not null; 2008-01-09T00:00:01 to… |
scientificName | string, not null; WoRMS-aligned name at the lowest rank the label allowed: 18 values, e.g. 'Ecklonia radiata' (49,972)… |
scientificNameID | string, not null; WoRMS LSID, e.g. 'urn:lsid:marinespecies.org:taxname:214344' (Ecklonia radiata) |
scientificNameAuthorship | string, not null; e.g. '(C.Agardh) J.Agardh, 1848' |
taxonRank | string, not null; Species 78,171 / Class 44,968 / Genus 470 / Family 243 |
kingdom | string, not null; always 'Chromista' |
phylum | string, not null; always 'Ochrophyta' |
class | string, nullable; always 'Phaeophyceae' (no nulls observed) |
order | string, nullable; Laminariales 50,687, Fucales 28,171, Sporochnales 26; NULL for the 44,968 class-level rows |
family | string, nullable; NULL for 44,968 class-level rows |
genus | string, nullable; NULL for 45,211 rows (class- and family-level records) |
verbatimIdentification | string, not null; the original Squidle+ label text: 'Ecklonia radiata' 49,972, 'Brown' 44,939, 'Phyllospora' 18,893… |
label.id | int64, nullable (no nulls observed); Squidle+ internal label id, 29 distinct values; note the dot in the column name |
occurrenceID | string, not null; '<squidle annotation id>-<deployment id>', e.g. '1189945-4339'; the deployment id is the only… |
basisOfRecord | string, not null; always 'MachineObservation' (hardcoded by the pipeline; the inputs are photographs annotated by… |
occurrenceStatus | string, not null; always 'present' (presence-only product) |
associatedMedia | string, not null; Squidle+ iframe URL to the annotated image, e.g. https://squidle.org/iframe/geodata/annotate/5362995/2… |
decimalLatitude | double, EPSG:4326, not null; -43.71932 to -27.13051 (image pose latitude) |
decimalLongitude | double, EPSG:4326, not null; 113.38759 to 153.48322 |
h3Index | string, not null; H3 cell at resolution 5 derived from lat/lon, e.g. '85bf4c0bfffffff'; 122 distinct cells |
and 4 more
Analyze this dataset without downloading it
On Scigantic, Understanding of Marine Imagery - Australian region - Aggregated kelp data product (2008 - ongoing) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Life Sciences datasets
- Single-Cell Atlas of Human Blood During Healthy Aging
- Pacific Ocean Sound Recordings
- LungVis 1.0: light-sheet fluorescence microscopy (LSFM) of 78 cleared mouse lungs with AI airway segmentations and nanoparticle deposition
- Pacific Ocean Sound Recordings
- National Reef Monitoring Network - Global mobile macroinvertebrate abundance
- Opioid Industry Documents Archive (OIDA) Data on AWS
- Pacific Ocean Sound Recordings
- Cell Painting Gallery cpg0026: Protein Mislocalization of 3,448 Missense Variants in HeLa