DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI)

Provider
Allen Institute for AI and UC Irvine (Dua, Wang, Dasigi, Stanovsky, Singh, Gardner)
License
CC-BY-SA-4.0 (source)
Stored at
Amazon S3, s3://ai2-public-datasets/drop (us-west-2)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://allenai.org/data

About this dataset

DROP (Discrete Reasoning Over the content of Paragraphs) is an English reading comprehension benchmark in which a system must resolve references in a passage and then perform discrete operations over them, such as addition, subtraction, counting, comparison and sorting, to answer a question. The questions were written by crowd workers against baseline models. The folder holds drop_dataset.zip (8.3 MB) with drop_dataset_train.json, drop_dataset_dev.json and license.txt. Each JSON file maps a passage id to the passage text and its question-answer pairs; an answer is a number, a date or one or more text spans. The standard metrics are exact match and a numeracy-aware F1, implemented in the AllenNLP drop_eval script. DROP is a common evaluation for numerical reasoning in language models and is part of several LLM evaluation suites. Load the JSON with Python's json module, or through Hugging Face datasets and TensorFlow Datasets loaders. License: CC BY-SA 4.0, stated in the license.txt shipped inside the archive. Cite Dua et al., DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (NAACL 2019).

How to access this dataset in Python

import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-public-datasets', Prefix='drop/', Delimiter='/')

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

Files

Example file: drop/drop_dataset.zip

Analyze this dataset without downloading it

On Scigantic, DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets