# BLiMP: Benchmark of Linguistic Minimal Pairs for English Grammar (67 Phenomena, NYU)

## Facts

- Provider: Warstadt, Parrish, Liu, Mohananey, Peng, Wang, Bowman (New York University)
- License: CC-BY-4.0 (https://github.com/alexwarstadt/blimp#license)
- Size: 69 files
- Format: Parquet (in sampled files)
- Stored at: Hugging Face, https://huggingface.co/datasets/nyu-mll/blimp
- Access: public, readable without an account
- Category: Natural Language Processing
- Source: https://arxiv.org/abs/1912.00582
- Page: https://scigantic.com/archive/95452e86-5723-5d9f-a208-805d20f0dd59
- Keywords: Natural Language Processing, benchmark, Parquet, Hugging Face

## About

The Benchmark of Linguistic Minimal Pairs (BLiMP) tests what language models know about English grammar. It has 67 sub-datasets of 1,000 minimal pairs each (67,000 pairs). A minimal pair is two sentences that differ in one point, one grammatically acceptable and one not, which isolates a phenomenon of syntax, morphology or semantics. Examples are subject-verb and anaphor agreement, argument structure, binding, control and raising, determiner-noun agreement, ellipsis, filler-gap dependencies, island effects, negative polarity item licensing, quantifiers and irregular forms. The sentences were generated automatically from grammars written by linguists, and human agreement with the labels is 96.4%. Each row has sentence_good, sentence_bad, the field (syntax, morphology, semantics), the linguistic term, the UID of the paradigm and flags. Psycholinguists and NLP researchers score a model by whether it assigns higher probability to the acceptable sentence. There is one Parquet file per phenomenon; load it with Hugging Face datasets by config name. BLiMP is distributed under CC BY 4.0. Cite Warstadt et al., BLiMP: The Benchmark of Linguistic Minimal Pairs for English, TACL 2020.

## Load the data

```python
from datasets import load_dataset
ds = load_dataset("nyu-mll/blimp")
print(ds)
```

## Data structure

File types in sampled files: Parquet

Example file: `adjunct_island/train-00000-of-00001.parquet`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/95452e86-5723-5d9f-a208-805d20f0dd59
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/95452e86-5723-5d9f-a208-805d20f0dd59
- Over MCP: `get_schema_card` with id `95452e86-5723-5d9f-a208-805d20f0dd59` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Benchmark datasets](https://scigantic.com/topics/benchmark)
- [Natural Language Processing datasets](https://scigantic.com/topics/natural-language-processing)
