OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative)
- Provider
- Open Language Data Initiative (OLDI); Meta NLLB team and contributors
- License
- CC-BY-SA-4.0 (source)
- Size
- 63 files
- Format
- JSON Lines (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/openlanguagedata/oldi_seed)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://aclanthology.org/2023.acl-long.154
About this dataset
OLDI Seed is a multi-way parallel corpus for starting machine translation in languages that lack training data. It has 6,193 sentences sampled from English Wikipedia, each translated into about 45 languages. Many are low-resource languages and scripts of Africa, Asia and the Americas, such as Acehnese in Arabic and Latin script, Moroccan and Egyptian Arabic, Bambara and Shan in Myanmar script. That gives about 576,000 sentence rows in total, one JSONL file per language and script under seed/, with per-language dataset cards and a bibliography of contributing papers. It updates the NLLB Seed data released with No Language Left Behind (Maillard et al., Small Data, Big Impact, ACL 2023). Linguists and MT researchers use it as aligned text in many languages for building translation models, studying cross-lingual transfer and documenting written varieties. Load it with Hugging Face datasets by language config. Because the source is Wikipedia, the data stays under CC BY-SA 4.0, managed by the Open Language Data Initiative. Cite the papers in bibliography.bib.
How to load this dataset in Python
from datasets import load_dataset
ds = load_dataset("openlanguagedata/oldi_seed")
print(ds)Data structure
File types in sampled files: JSON Lines
Files
Example file: seed/ace_Arab.jsonl
Analyze this dataset without downloading it
On Scigantic, OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.