OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative)

Provider
Open Language Data Initiative (OLDI); Meta NLLB team and contributors
License
CC-BY-SA-4.0 (source)
Size
63 files
Format
JSON Lines (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/openlanguagedata/oldi_seed)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://aclanthology.org/2023.acl-long.154

About this dataset

OLDI Seed is a multi-way parallel corpus for starting machine translation in languages that lack training data. It has 6,193 sentences sampled from English Wikipedia, each translated into about 45 languages. Many are low-resource languages and scripts of Africa, Asia and the Americas, such as Acehnese in Arabic and Latin script, Moroccan and Egyptian Arabic, Bambara and Shan in Myanmar script. That gives about 576,000 sentence rows in total, one JSONL file per language and script under seed/, with per-language dataset cards and a bibliography of contributing papers. It updates the NLLB Seed data released with No Language Left Behind (Maillard et al., Small Data, Big Impact, ACL 2023). Linguists and MT researchers use it as aligned text in many languages for building translation models, studying cross-lingual transfer and documenting written varieties. Load it with Hugging Face datasets by language config. Because the source is Wikipedia, the data stays under CC BY-SA 4.0, managed by the Open Language Data Initiative. Cite the papers in bibliography.bib.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("openlanguagedata/oldi_seed")
print(ds)

Data structure

File types in sampled files: JSON Lines

Files

Example file: seed/ace_Arab.jsonl

Analyze this dataset without downloading it

On Scigantic, OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets