BLiMP: Benchmark of Linguistic Minimal Pairs for English Grammar (67 Phenomena, NYU)

Provider
Warstadt, Parrish, Liu, Mohananey, Peng, Wang, Bowman (New York University)
License
CC-BY-4.0 (source)
Size
69 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/nyu-mll/blimp)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://arxiv.org/abs/1912.00582

About this dataset

The Benchmark of Linguistic Minimal Pairs (BLiMP) tests what language models know about English grammar. It has 67 sub-datasets of 1,000 minimal pairs each (67,000 pairs). A minimal pair is two sentences that differ in one point, one grammatically acceptable and one not, which isolates a phenomenon of syntax, morphology or semantics. Examples are subject-verb and anaphor agreement, argument structure, binding, control and raising, determiner-noun agreement, ellipsis, filler-gap dependencies, island effects, negative polarity item licensing, quantifiers and irregular forms. The sentences were generated automatically from grammars written by linguists, and human agreement with the labels is 96.4%. Each row has sentence_good, sentence_bad, the field (syntax, morphology, semantics), the linguistic term, the UID of the paradigm and flags. Psycholinguists and NLP researchers score a model by whether it assigns higher probability to the acceptable sentence. There is one Parquet file per phenomenon; load it with Hugging Face datasets by config name. BLiMP is distributed under CC BY 4.0. Cite Warstadt et al., BLiMP: The Benchmark of Linguistic Minimal Pairs for English, TACL 2020.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("nyu-mll/blimp")
print(ds)

Data structure

File types in sampled files: Parquet

Files

Example file: adjunct_island/train-00000-of-00001.parquet

Analyze this dataset without downloading it

On Scigantic, BLiMP: Benchmark of Linguistic Minimal Pairs for English Grammar (67 Phenomena, NYU) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets