# DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI)

## Facts

- Provider: Allen Institute for AI and UC Irvine (Dua, Wang, Dasigi, Stanovsky, Singh, Gardner)
- License: CC-BY-SA-4.0 (https://ai2-public-datasets.s3.amazonaws.com/drop/drop_dataset.zip)
- Stored at: Amazon S3, s3://ai2-public-datasets/drop (us-west-2)
- Access: public, readable without an account
- Category: Natural Language Processing
- Source: https://allenai.org/data
- Page: https://scigantic.com/archive/b10bc8de-ed02-5e7d-9eed-5b16d76d9b4a
- Keywords: Natural Language Processing, benchmark, AWS Open Data

## About

DROP (Discrete Reasoning Over the content of Paragraphs) is an English reading comprehension benchmark in which a system must resolve references in a passage and then perform discrete operations over them, such as addition, subtraction, counting, comparison and sorting, to answer a question. The questions were written by crowd workers against baseline models. The folder holds drop_dataset.zip (8.3 MB) with drop_dataset_train.json, drop_dataset_dev.json and license.txt. Each JSON file maps a passage id to the passage text and its question-answer pairs; an answer is a number, a date or one or more text spans. The standard metrics are exact match and a numeracy-aware F1, implemented in the AllenNLP drop_eval script. DROP is a common evaluation for numerical reasoning in language models and is part of several LLM evaluation suites. Load the JSON with Python's json module, or through Hugging Face datasets and TensorFlow Datasets loaders. License: CC BY-SA 4.0, stated in the license.txt shipped inside the archive. Cite Dua et al., DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (NAACL 2019).

## Load the data

```python
import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-public-datasets', Prefix='drop/', Delimiter='/')
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

Example file: `drop/drop_dataset.zip`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/b10bc8de-ed02-5e7d-9eed-5b16d76d9b4a
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/b10bc8de-ed02-5e7d-9eed-5b16d76d9b4a
- Over MCP: `get_schema_card` with id `b10bc8de-ed02-5e7d-9eed-5b16d76d9b4a` at https://api.scigantic.com/mcp

## Related topics

- [Benchmark datasets](https://scigantic.com/topics/benchmark)
- [Natural Language Processing datasets](https://scigantic.com/topics/natural-language-processing)
