DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research)

Provider
Google Research (Geva et al.)
License
CC-BY-SA-3.0 (source)
Size
49 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/google-research-datasets/discofuse)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://arxiv.org/abs/1902.10526

About this dataset

DiscoFuse is a large-scale dataset for discourse-based sentence fusion: given two input sentences, a model must merge them into one fluent sentence, typically by inserting or adjusting a discourse connective (but, although, because), an anaphoric pronoun or a relative clause. Examples were created by applying a rule-based splitting method to single sentences, so each item pairs two separate sentences with the original fused sentence and labels for the discourse type and connective. The full release also has a sports-news part crawled from the web, which this row excludes. This row is the Wikipedia portion only (DiscoFuse-Wiki), built from English Wikipedia text: 16.6 million examples in random 98/1/1 train, development and test splits plus balanced versions of each, in 15 Parquet files (about 3.9 GB). It is used to train and evaluate text-editing and sentence-fusion models and to pre-train for discourse connective prediction. Load it with the Hugging Face datasets library (config discofuse-wikipedia) or read the Parquet files with pandas. Licensed by Google Research under Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0); cite Geva et al., DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion, NAACL 2019.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("google-research-datasets/discofuse")
print(ds)

Data structure

File types in sampled files: Parquet

Files

Example file: discofuse-sport/test-00000-of-00001.parquet

Analyze this dataset without downloading it

On Scigantic, DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets