StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind)
- Provider
- Google DeepMind (Liska, Kocisky, Gribovskaya, Terzi et al.)
- License
- CC-BY-4.0 (source)
- Format
- JSON Lines (in sampled files)
- Stored at
- Google Cloud Storage, gs://dm-streamingqa (us)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://github.com/google-deepmind/streamingqa
About this dataset
StreamingQA is a question answering benchmark for adaptation to new knowledge over time. Questions are tied to articles in the English WMT News Crawl corpus, and each carries the date it was asked (question_ts) and the publication date of its evidence article (evidence_ts), so a model can be evaluated on questions about events that happened after its training data. The bucket holds streaminqa_train.jsonl.gz (99,402 questions), streaminqa_valid.jsonl.gz (9,939) and streaminqa_eval.jsonl.gz (36,378), plus wmt_sorting_key_ids.txt.gz, a list of 11,393,471 SHA256-based document keys that reproduce the deduplicated WMT subset used in the paper with the repository's extraction script. Each record has the question, answers (one for train and valid, three references plus an additional fourth for eval), the evidence document key, a recent or past label, whether the question was written by annotators or generated, and Perspective API toxicity scores. The news articles themselves are not redistributed; they come from WMT News Crawl. Used to study temporal knowledge drift, continual learning, retrieval-augmented generation and knowledge updating in language models. License: CC BY 4.0 for the data (software Apache 2.0), stated in the repository. Cite Liska et al., StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (ICML 2022).
How to access this dataset in Python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('dm-streamingqa/', detail=False)[:10]Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: JSON Lines
Files
Example file: streaminqa_eval.jsonl.gz
Analyze this dataset without downloading it
On Scigantic, StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- CourtListener Bulk Legal Data: US Court Opinions, Dockets, Citation Graph, Judges, Oral Arguments and Financial Disclosures as Quarterly PostgreSQL CSV Exports (Free Law Project)
- IndicGenBench Flores-IN: FLORES Machine Translation Sentences Human-Translated into 29 Indic Languages (Google Research)
- Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl)
- Multilingual Name Entity Recognition (NER) Datasets with Gazetteer
- ABEJA CC JA
- MT Bench Hi
- End of Term Web Archive Dataset
- OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative)