MADIS METAR Surface Station Observations 2001-2023 (WeatherBench 2, Parquet)

Provider
NOAA MADIS; converted by Google Research (WeatherBench 2)
License
CC-BY-4.0 (source)
Format
Parquet (in sampled files)
Stored at
Google Cloud Storage, gs://weatherbench2/datasets/metar (us)
Access
Free and public. No account with the data provider is needed.
DOI
10.1029/2023MS004019
Category
Climate & Weather
Source
https://doi.org/10.1029/2023MS004019

About MADIS METAR Surface Station Observations 2001-2023 (WeatherBench 2, Parquet)

Global METAR and SPECI weather station reports from the NOAA Meteorological Assimilation Data Ingest System (MADIS), downloaded from the MADIS FTP server in NetCDF and converted by the WeatherBench 2 team to Parquet for verifying forecasts against point observations rather than reanalysis. Each row is one report with station name, location name, latitude, longitude, elevation, observation, nominal and receipt times, report type, visibility, present weather, sea level pressure, temperature, dewpoint, wind direction, speed and gust, altimeter, 24-hour min and max temperature, 1, 3, 6 and 24-hour precipitation, 3-hour pressure change and snow cover, with MADIS data-descriptor (DD) quality flags. The same data is laid out four ways, Hive-partitioned by year and month: by nominal time per month, per day and per hour, and by observation time per hour from 2001-07-01 to 2024-01-01. Measured 2026-11-14: the monthly layout holds 270 Parquet files totalling 26.8 GB. Read with pandas.read_parquet or pyarrow.dataset. The WeatherBench 2 LICENSE file states CC BY 4.0 and notes MADIS METAR data is publicly accessible without restrictions; cite NOAA MADIS and Rasp et al. 2024.

How to load this dataset in Python

import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'metar-timeNominal-by-day/year=2001/month=10/day=1/2001-10-01.parquet'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()

Run in a Scigantic notebook on 2026-11-15 against metar-timeNominal-by-day/year=2001/month=10/day=1/2001-10-01.parquet: Parquet 53,323 rows, 44 columns, 1 row groups; columns ['stationName:string', 'locationName:string', 'latitude:float', 'longitude:float', 'elevation:float', 'timeObs:timestamp[ns]', 'timeNominal:timestamp[ns]', 'timeReceived:timestamp[ns]', 'reportType:string', 'autoStationType:string'].

From any machine

import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('weatherbench2/datasets/metar/', detail=False)[:10]

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: Parquet

Files

Example file: datasets/metar/metar-timeNominal-by-day/year=2001/month=10/day=1/2001-10-01.parquet

Analyze this dataset without downloading it

On Scigantic, MADIS METAR Surface Station Observations 2001-2023 (WeatherBench 2, Parquet) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Climate & Weather datasets