DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research)
- Provider
- Google Research (Geva et al.)
- License
- CC-BY-SA-3.0 (source)
- Size
- 49 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/google-research-datasets/discofuse)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://arxiv.org/abs/1902.10526
About this dataset
DiscoFuse is a large-scale dataset for discourse-based sentence fusion: given two input sentences, a model must merge them into one fluent sentence, typically by inserting or adjusting a discourse connective (but, although, because), an anaphoric pronoun or a relative clause. Examples were created by applying a rule-based splitting method to single sentences, so each item pairs two separate sentences with the original fused sentence and labels for the discourse type and connective. The full release also has a sports-news part crawled from the web, which this row excludes. This row is the Wikipedia portion only (DiscoFuse-Wiki), built from English Wikipedia text: 16.6 million examples in random 98/1/1 train, development and test splits plus balanced versions of each, in 15 Parquet files (about 3.9 GB). It is used to train and evaluate text-editing and sentence-fusion models and to pre-train for discourse connective prediction. Load it with the Hugging Face datasets library (config discofuse-wikipedia) or read the Parquet files with pandas. Licensed by Google Research under Creative Commons Attribution-ShareAlike 3.0 (CC BY-SA 3.0); cite Geva et al., DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion, NAACL 2019.
How to load this dataset in Python
from datasets import load_dataset
ds = load_dataset("google-research-datasets/discofuse")
print(ds)Data structure
File types in sampled files: Parquet
Files
Example file: discofuse-sport/test-00000-of-00001.parquet
Analyze this dataset without downloading it
On Scigantic, DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- Wiki Movies
- XQuAD-R: Cross-lingual Answer Sentence Retrieval Benchmark, 11-Way Parallel Questions and Answers (Google Research LAReQA)
- Helpful Sentences from Reviews
- Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books)
- CourtListener Case Law Embeddings: Chunked US Court Opinion Text with 768-Dimensional ModernBERT Vectors for Semantic Legal Search (Free Law Project)
- MultiCoNER Datasets
- Belebele
- XQuAD (Cross-lingual Question Answering Dataset): 1,190 SQuAD Questions Professionally Translated into 11 Languages (Google DeepMind)