# DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research)

## Facts

- Provider: Google Research (Geva et al.)
- License: CC-BY-SA-3.0 (https://github.com/google-research-datasets/discofuse)
- Size: 49 files
- Format: Parquet (in sampled files)
- Stored at: Hugging Face, https://huggingface.co/datasets/google-research-datasets/discofuse
- Access: public, readable without an account
- Category: Natural Language Processing
- Source: https://arxiv.org/abs/1902.10526
- Page: https://scigantic.com/archive/7370e431-1f30-5375-b912-4f3070ea830f
- Keywords: Natural Language Processing, Parquet, Hugging Face

## About

DiscoFuse is a large-scale dataset for discourse-based sentence fusion: given two input sentences, a model must merge them into one fluent sentence, typically by inserting or adjusting a discourse connective (but, although, because), an anaphoric pronoun or a relative clause. Examples were created by applying a rule-based splitting method to single sentences, so each item pairs two separate sentences with the original fused sentence and labels for the discourse type and connective. The full release also has a sports-news part crawled from the web, which this row excludes. This row is the Wikipedia portion only (DiscoFuse-Wiki), built from English Wikipedia text: 16.6 million examples in random 98/1/1 train, development and test splits plus balanced versions of each, in 15 Parquet files (about 3.9 GB). It is used to train and evaluate text-editing and sentence-fusion models and to pre-train for discourse connective prediction. Load it with the Hugging Face datasets library (config discofuse-wikipedia) or read the Parquet files with pandas. Licensed by Google Research under Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0); cite Geva et al., DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion, NAACL 2019.

## Load the data

```python
from datasets import load_dataset
ds = load_dataset("google-research-datasets/discofuse")
print(ds)
```

## Data structure

File types in sampled files: Parquet

Example file: `discofuse-sport/test-00000-of-00001.parquet`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/7370e431-1f30-5375-b912-4f3070ea830f
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/7370e431-1f30-5375-b912-4f3070ea830f
- Over MCP: `get_schema_card` with id `7370e431-1f30-5375-b912-4f3070ea830f` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Natural Language Processing datasets](https://scigantic.com/topics/natural-language-processing)
