WikiDomains: 22,561 Wikipedia Term Definitions across 13 Academic Domains with Annotated Difficult Concepts for Targeted Concept Simplification (Google DeepMind)
- Provider
- Google DeepMind (Asthana, Rashkin, Clark, Huot, Lapata)
- License
- CC-BY-SA-4.0 (source)
- Format
- TSV (in sampled files)
- Stored at
- Google Cloud Storage, gs://wikidomains (us)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://github.com/google-deepmind/wikidomains
About this dataset
WikiDomains is an English dataset for targeted concept simplification: helping readers understand a hard concept inside a domain-specific definition without simplifying away the rest. Each row is an English Wikipedia article in one of thirteen domains (Food and Drink, Performing Arts, Business and Economics, Politics and Government, Biology, Chemistry, Computing, Earth and Environment, Mathematics, Medicine and Health, Physics, Engineering, Technology) with the term (article title), its definition (the first sentence of the lead), the difficult concept inside that definition, the domain, the Wikipedia page id and Wikidata id, the article's PageRank score, in-domain and out-of-domain term frequencies and the full lead section. The bucket holds three tab-separated files: wikidomains_train.tsv (15,873 rows), wikidomains_dev.tsv (3,384) and wikidomains_test.tsv (3,304), about 28 MB together. Used to evaluate large language models on explaining jargon, text simplification, definition generation and readability for lay readers. Read with pandas.read_csv(sep='\t'). License: CC BY-SA 4.0, covering both the Wikipedia-derived text and the other materials, stated in the repository. Cite Asthana et al., Evaluating LLMs for Targeted Concept Simplification for Domain-Specific Texts (EMNLP 2024, arXiv:2410.20763).
How to access this dataset in Python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('wikidomains/', detail=False)[:10]Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: TSV
Files
Example file: wikidomains_dev.tsv
Analyze this dataset without downloading it
On Scigantic, WikiDomains: 22,561 Wikipedia Term Definitions across 13 Academic Domains with Annotated Difficult Concepts for Targeted Concept Simplification (Google DeepMind) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- Common Crawl
- Japanese Tokenizer Dictionaries
- SMOL and GATITOS: Professional Translations into 221 Low-Resource Languages (Google Research)
- Wiki Dpr
- CantTalkAboutThis Topic Control Dataset NC
- Nemotron Content Safety Reasoning Dataset
- S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)
- Sudachi Language Resources