DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI)
- Provider
- Allen Institute for AI and UC Irvine (Dua, Wang, Dasigi, Stanovsky, Singh, Gardner)
- License
- CC-BY-SA-4.0 (source)
- Stored at
- Amazon S3, s3://ai2-public-datasets/drop (us-west-2)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://allenai.org/data
About this dataset
DROP (Discrete Reasoning Over the content of Paragraphs) is an English reading comprehension benchmark in which a system must resolve references in a passage and then perform discrete operations over them, such as addition, subtraction, counting, comparison and sorting, to answer a question. The questions were written by crowd workers against baseline models. The folder holds drop_dataset.zip (8.3 MB) with drop_dataset_train.json, drop_dataset_dev.json and license.txt. Each JSON file maps a passage id to the passage text and its question-answer pairs; an answer is a number, a date or one or more text spans. The standard metrics are exact match and a numeracy-aware F1, implemented in the AllenNLP drop_eval script. DROP is a common evaluation for numerical reasoning in language models and is part of several LLM evaluation suites. Load the JSON with Python's json module, or through Hugging Face datasets and TensorFlow Datasets loaders. License: CC BY-SA 4.0, stated in the license.txt shipped inside the archive. Cite Dua et al., DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (NAACL 2019).
How to access this dataset in Python
import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-public-datasets', Prefix='drop/', Delimiter='/')Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
Files
Example file: drop/drop_dataset.zip
Analyze this dataset without downloading it
On Scigantic, DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- Mdd
- ARIES: Corpus of Scientific Paper Edits Made in Response to Peer Review Comments (OpenReview, Allen Institute for AI)
- Flores
- AdvancedIF
- Civil Comments: 2 Million Public News-Site Comments with Crowd-Rated Toxicity and Identity-Mention Labels (Jigsaw, Google)
- Community Alignment Dataset
- Nemotron RL Instruction Following MultiTurnChat V 1
- WikiDomains: 22,561 Wikipedia Term Definitions across 13 Academic Domains with Annotated Difficult Concepts for Targeted Concept Simplification (Google DeepMind)