SciganticTopicsExplore

Natural Language Processing datasets

73 public Natural Language Processing datasets in the Scigantic catalog, from providers including Meta AI, NVIDIA and Amazon. 72 can be read without an account, and 57 carry an open license (8 are non-commercial or no-derivatives). Common formats: JSON Lines, Parquet and JSON. 4 have loading code that was run against a real file. Each opens in a hosted Jupyter notebook with the data mounted, so nothing has to be downloaded first.

DatasetProviderLicenseAccess
Common Screens
JPEGloading code checked
Common Screens[Attribution 4.0 International (CC BY…public
Low Context Name Entity Recognition (NER) Datasets with Gazetteer
loading code checked
Amazon[CC BY 4.0](https://creativecommons.org/licenses…public
Multilingual Name Entity Recognition (NER) Datasets with Gazetteer
loading code checked
Amazon[CC BY 4.0](https://creativecommons.org/licenses…public
Phrase Clustering Dataset (PCD)
JSONloading code checked
AmazonThis data is available for anyone to use under…public
ABEJA CC JA
JSON Lines
ABEJA inc.This data is available for anyone to use under…public
ARIES: Corpus of Scientific Paper Edits Made in Response to Peer Review Comments (OpenReview, Allen Institute for AI)
JSON Lines
Allen Institute for AI (Mike D'Arcy et al.)ODC-By-1.0public
Asset
Parquet
Meta AIcc-by-sa-4.0public
Babi Qa
JSON
Meta AIcc-by-3.0public
Belebele
JSON Lines
Meta AIcc-by-sa-4.0public
BFCL Hi
JSON
NVIDIAcc-by-4.0public
BLiMP: Benchmark of Linguistic Minimal Pairs for English Grammar (67 Phenomena, NYU)
Parquet
Warstadt, Parrish, Liu, Mohananey, Peng, Wang, Bowman (New YCC-BY-4.0public
BoolQ: 15,942 Naturally Occurring Yes/No Questions Paired with Wikipedia Passages (Google Research)
Parquet
Google Research (Clark et al.)CC-BY-SA-3.0public
CantTalkAboutThis Topic Control Dataset
JSON Lines
NVIDIAcc-by-4.0public
ChatRAG Hi
JSON
NVIDIAcc-by-sa-4.0public
Civil Comments: 2 Million Public News-Site Comments with Crowd-Rated Toxicity and Identity-Mention Labels (Jigsaw, Google)
Parquet
Jigsaw and Google (Borkan et al.); Civil Comments platformCC0-1.0public
Community Alignment Dataset
CSV
Meta AIcc-by-4.0public
Content Rephrasing
TSV
Meta AIcc-by-sa-4.0public
DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research)
Parquet
Google Research (Geva et al.)CC-BY-SA-3.0public
DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI)
Allen Institute for AI and UC Irvine (Dua, Wang, Dasigi, StaCC-BY-SA-4.0public
Flores
Parquet
Meta AIcc-by-sa-4.0public
Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books)
Google BooksCC-BY-3.0public
Helpful Sentences from Reviews
JSON
AmazonThis data is available for anyone to use under…public
IFEval Hi
JSON Lines
NVIDIAcc-by-4.0public
IndicGenBench Flores-IN: FLORES Machine Translation Sentences Human-Translated into 29 Indic Languages (Google Research)
JSON
Google Research (Singh et al.)CC-BY-SA-4.0public
IndicGenBench XorQA-IN: Cross-lingual Open-Retrieval Question Answering in 29 Indic Languages (Google Research)
JSON
Google Research (Singh et al.)MITpublic
IndicGenBench XQuAD-IN: Extractive Question Answering Translated into Indic Languages (Google Research)
JSON
Google Research (Singh et al.)CC-BY-SA-4.0public
Kilt Tasks
Parquet
Meta AImitpublic
Lama
Meta AIcc-by-4.0public
Md Gender Bias
Meta AImitpublic
Mdd
Meta AIcc-by-3.0public
Mlqa
Meta AIcc-by-sa-3.0public
MT Bench Hi
JSON Lines
NVIDIAcc-by-4.0public
MultiCoNER Datasets
Amazon[CC BY 4.0](https://creativecommons.org/licenses…public
MultiCoNER Datasets
Amazon[CC BY 4.0](https://creativecommons.org/licenses…public
MuP: Multi-Perspective Scientific Document Summarization Dataset from Peer Review Summaries (SDP 2022 Shared Task, Allen Institute for AI)
Allen Institute for AI and MuP 2022 organizers (Guy FeigenblODC-By-1.0public
Natural Questions (NQ): 307,373 Real Google Search Questions with Long and Short Answers Annotated on Full Wikipedia Pages (Google Research)
Parquet
Google Research (Kwiatkowski et al.)CC-BY-SA-3.0public
Nemotron Cascade RL Instruction Following
Parquet
NVIDIAodc-bypublic
Nemotron Content Safety Reasoning Dataset
JSON Lines
NVIDIAcc-by-4.0public
Nemotron Instruction Following Chat V 1
JSON Lines
NVIDIAcc-by-4.0public
Nemotron RL Instruction Following Calendar V 2
JSON Lines
NVIDIAcc-by-4.0public
Nemotron RL Instruction Following MultiTurnChat V 1
JSON Lines
NVIDIAcc-by-4.0public
Nemotron RL Instruction Following Structured Outputs
JSON Lines
NVIDIAcc-by-4.0public
Nemotron Safety Guard Dataset V 3
JSON Lines
NVIDIAcc-by-4.0public
Nemotron SFT Instruction Following Chat V 2
JSON Lines
NVIDIAodc-bypublic
OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative)
JSON Lines
Open Language Data Initiative (OLDI); Meta NLLB team and conCC-BY-SA-4.0public
Panda
JSON Lines
Meta AImitpublic
PASS: Perturb-and-Select Summarizer for Product Reviews
JSON Lines
AmazonThis data is available for anyone to use under…public
S2AND: Author Name Disambiguation Benchmark, Eight Bibliographic Datasets plus Semantic Scholar Production Models (Allen Institute for AI)
JSON
Allen Institute for AI, Semantic Scholar (Shivashankar SubraODC-By-1.0public
S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)
JSON
Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, SODC-By-1.0public
Shopping Humor Generation
AmazonDataset is published under…public
SMOL and GATITOS: Professional Translations into 221 Low-Resource Languages (Google Research)
JSON Lines
Google Research (Caswell et al.)CC-BY-4.0public
StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind)
JSON Lines
Google DeepMind (Liska, Kocisky, Gribovskaya, Terzi et al.)CC-BY-4.0public
Sudachi Language Resources
Works ApplicationsApache-2.0public
Wiki Movies
Meta AIcc-by-3.0public
WikiDomains: 22,561 Wikipedia Term Definitions across 13 Academic Domains with Annotated Difficult Concepts for Targeted Concept Simplification (Google DeepMind)
TSV
Google DeepMind (Asthana, Rashkin, Clark, Huot, Lapata)CC-BY-SA-4.0public
XQuAD (Cross-lingual Question Answering Dataset): 1,190 SQuAD Questions Professionally Translated into 11 Languages (Google DeepMind)
Parquet
DeepMind (Artetxe, Ruder and Yogatama)CC-BY-SA-4.0public
XQuAD-R: Cross-lingual Answer Sentence Retrieval Benchmark, 11-Way Parallel Questions and Answers (Google Research LAReQA)
Parquet
Google Research (Roy et al.)CC-BY-SA-4.0public
AdvancedIF
CSV
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
Anli
Parquet
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
CantTalkAboutThis Topic Control Dataset NC
JSON Lines
NVIDIAcc-by-nc-4.0 non-commercial or no-derivativespublic
COLD Cases: 8.3 Million US Court Decisions with Opinions and Metadata (Harvard Library Innovation Lab, CourtListener)
Parquet
Harvard Library Innovation Lab; Free Law Project (CourtListePublic Domainpublic
CourtListener Bulk Legal Data: US Court Opinions, Dockets, Citation Graph, Judges, Oral Arguments and Financial Disclosures as Quarterly PostgreSQL CSV Exports (Free Law Project)
CSV
Free Law ProjectPublic Domainpublic
CourtListener Case Law Embeddings: Chunked US Court Opinion Text with 768-Dimensional ModernBERT Vectors for Semantic Legal Search (Free Law Project)
JSON
Free Law ProjectPublic Domainpublic
Curiosity Dialogs
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
Empathetic Dialogues
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
End of Term Web Archive Dataset
End of Term Web ArchiveCustom (see documentation)public
ExploreToM
CSV
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
Google Books Ngrams
Not managedCreative Commons Attribution 3.0 Unported…public
Imppres
Parquet
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
Japanese Tokenizer Dictionaries
CotonohaCustom (see documentation)public
Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl)
ParaCrawlCreative Commons CC0 license ("no rights…public
Wiki Dpr
Parquet
Meta AIcc-by-nc-4.0 non-commercial or no-derivativespublic
Common Crawl
Common CrawlCommon Crawl Terms of Usecredentialed

Related topics