Natural Questions (NQ): 307,373 Real Google Search Questions with Long and Short Answers Annotated on Full Wikipedia Pages (Google Research)

Provider
Google Research (Kwiatkowski et al.)
License
CC-BY-SA-3.0 (source)
Size
303 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/google-research-datasets/natural_questions)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://doi.org/10.1162/tacl_a_00276

About this dataset

Natural Questions (NQ) is an open-domain question answering benchmark built from real, anonymized queries that users issued to Google Search. For each question an annotator was shown an entire English Wikipedia article and marked a long answer (usually a paragraph or table) and, where one exists, a short answer span or a yes/no answer; some questions have no answer on the page. Because systems must read the whole page rather than a selected passage, NQ is a standard test for reading comprehension, passage retrieval and long-document machine reading. This Hugging Face release has 307,373 training examples and 7,830 development examples, each with the full article HTML, tokens, the question, and the long and short answer annotations, stored as about 300 Parquet shards (about 58 GB) under default/ and dev/. Load it with the Hugging Face datasets library (load_dataset('google-research-datasets/natural_questions')) or read the Parquet files with pandas or pyarrow. Released by Google Research under Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0); cite Kwiatkowski et al., Natural Questions: a Benchmark for Question Answering Research, TACL 2019.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("google-research-datasets/natural_questions")
print(ds)

Data structure

File types in sampled files: Parquet

Files

Example file: default/train-00000-of-00287.parquet

Analyze this dataset without downloading it

On Scigantic, Natural Questions (NQ): 307,373 Real Google Search Questions with Long and Short Answers Annotated on Full Wikipedia Pages (Google Research) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets