StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind)

Provider
Google DeepMind (Liska, Kocisky, Gribovskaya, Terzi et al.)
License
CC-BY-4.0 (source)
Format
JSON Lines (in sampled files)
Stored at
Google Cloud Storage, gs://dm-streamingqa (us)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://github.com/google-deepmind/streamingqa

About this dataset

StreamingQA is a question answering benchmark for adaptation to new knowledge over time. Questions are tied to articles in the English WMT News Crawl corpus, and each carries the date it was asked (question_ts) and the publication date of its evidence article (evidence_ts), so a model can be evaluated on questions about events that happened after its training data. The bucket holds streaminqa_train.jsonl.gz (99,402 questions), streaminqa_valid.jsonl.gz (9,939) and streaminqa_eval.jsonl.gz (36,378), plus wmt_sorting_key_ids.txt.gz, a list of 11,393,471 SHA256-based document keys that reproduce the deduplicated WMT subset used in the paper with the repository's extraction script. Each record has the question, answers (one for train and valid, three references plus an additional fourth for eval), the evidence document key, a recent or past label, whether the question was written by annotators or generated, and Perspective API toxicity scores. The news articles themselves are not redistributed; they come from WMT News Crawl. Used to study temporal knowledge drift, continual learning, retrieval-augmented generation and knowledge updating in language models. License: CC BY 4.0 for the data (software Apache 2.0), stated in the repository. Cite Liska et al., StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (ICML 2022).

How to access this dataset in Python

import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('dm-streamingqa/', detail=False)[:10]

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: JSON Lines

Files

Example file: streaminqa_eval.jsonl.gz

Analyze this dataset without downloading it

On Scigantic, StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets