Common Screens JPEGloading code checked | Common Screens | [Attribution 4.0 International (CC BY… | public |
Low Context Name Entity Recognition (NER) Datasets with Gazetteer loading code checked | Amazon | [CC BY 4.0](https://creativecommons.org/licenses… | public |
Multilingual Name Entity Recognition (NER) Datasets with Gazetteer loading code checked | Amazon | [CC BY 4.0](https://creativecommons.org/licenses… | public |
Phrase Clustering Dataset (PCD) JSONloading code checked | Amazon | This data is available for anyone to use under… | public |
ABEJA CC JA JSON Lines | ABEJA inc. | This data is available for anyone to use under… | public |
ARIES: Corpus of Scientific Paper Edits Made in Response to Peer Review Comments (OpenReview, Allen Institute for AI) JSON Lines | Allen Institute for AI (Mike D'Arcy et al.) | ODC-By-1.0 | public |
Asset Parquet | Meta AI | cc-by-sa-4.0 | public |
Babi Qa JSON | Meta AI | cc-by-3.0 | public |
Belebele JSON Lines | Meta AI | cc-by-sa-4.0 | public |
BFCL Hi JSON | NVIDIA | cc-by-4.0 | public |
BLiMP: Benchmark of Linguistic Minimal Pairs for English Grammar (67 Phenomena, NYU) Parquet | Warstadt, Parrish, Liu, Mohananey, Peng, Wang, Bowman (New Y | CC-BY-4.0 | public |
BoolQ: 15,942 Naturally Occurring Yes/No Questions Paired with Wikipedia Passages (Google Research) Parquet | Google Research (Clark et al.) | CC-BY-SA-3.0 | public |
CantTalkAboutThis Topic Control Dataset JSON Lines | NVIDIA | cc-by-4.0 | public |
ChatRAG Hi JSON | NVIDIA | cc-by-sa-4.0 | public |
Civil Comments: 2 Million Public News-Site Comments with Crowd-Rated Toxicity and Identity-Mention Labels (Jigsaw, Google) Parquet | Jigsaw and Google (Borkan et al.); Civil Comments platform | CC0-1.0 | public |
Community Alignment Dataset CSV | Meta AI | cc-by-4.0 | public |
Content Rephrasing TSV | Meta AI | cc-by-sa-4.0 | public |
DiscoFuse Wikipedia: 16 Million Sentence Fusion Examples with Discourse Connectives from English Wikipedia (Google Research) Parquet | Google Research (Geva et al.) | CC-BY-SA-3.0 | public |
DROP: Discrete Reasoning Over Paragraphs Reading Comprehension Benchmark, Questions Requiring Addition, Counting and Sorting (Allen Institute for AI)
| Allen Institute for AI and UC Irvine (Dua, Wang, Dasigi, Sta | CC-BY-SA-4.0 | public |
Flores Parquet | Meta AI | cc-by-sa-4.0 | public |
Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books)
| Google Books | CC-BY-3.0 | public |
Helpful Sentences from Reviews JSON | Amazon | This data is available for anyone to use under… | public |
IFEval Hi JSON Lines | NVIDIA | cc-by-4.0 | public |
IndicGenBench Flores-IN: FLORES Machine Translation Sentences Human-Translated into 29 Indic Languages (Google Research) JSON | Google Research (Singh et al.) | CC-BY-SA-4.0 | public |
IndicGenBench XorQA-IN: Cross-lingual Open-Retrieval Question Answering in 29 Indic Languages (Google Research) JSON | Google Research (Singh et al.) | MIT | public |
IndicGenBench XQuAD-IN: Extractive Question Answering Translated into Indic Languages (Google Research) JSON | Google Research (Singh et al.) | CC-BY-SA-4.0 | public |
Kilt Tasks Parquet | Meta AI | mit | public |
Lama
| Meta AI | cc-by-4.0 | public |
Md Gender Bias
| Meta AI | mit | public |
Mdd
| Meta AI | cc-by-3.0 | public |
Mlqa
| Meta AI | cc-by-sa-3.0 | public |
MT Bench Hi JSON Lines | NVIDIA | cc-by-4.0 | public |
MultiCoNER Datasets
| Amazon | [CC BY 4.0](https://creativecommons.org/licenses… | public |
MultiCoNER Datasets
| Amazon | [CC BY 4.0](https://creativecommons.org/licenses… | public |
MuP: Multi-Perspective Scientific Document Summarization Dataset from Peer Review Summaries (SDP 2022 Shared Task, Allen Institute for AI)
| Allen Institute for AI and MuP 2022 organizers (Guy Feigenbl | ODC-By-1.0 | public |
Natural Questions (NQ): 307,373 Real Google Search Questions with Long and Short Answers Annotated on Full Wikipedia Pages (Google Research) Parquet | Google Research (Kwiatkowski et al.) | CC-BY-SA-3.0 | public |
Nemotron Cascade RL Instruction Following Parquet | NVIDIA | odc-by | public |
Nemotron Content Safety Reasoning Dataset JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron Instruction Following Chat V 1 JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron RL Instruction Following Calendar V 2 JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron RL Instruction Following MultiTurnChat V 1 JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron RL Instruction Following Structured Outputs JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron Safety Guard Dataset V 3 JSON Lines | NVIDIA | cc-by-4.0 | public |
Nemotron SFT Instruction Following Chat V 2 JSON Lines | NVIDIA | odc-by | public |
OLDI Seed: 6,193 English Wikipedia Sentences Translated into About 45 Low-Resource Languages (Open Language Data Initiative) JSON Lines | Open Language Data Initiative (OLDI); Meta NLLB team and con | CC-BY-SA-4.0 | public |
Panda JSON Lines | Meta AI | mit | public |
PASS: Perturb-and-Select Summarizer for Product Reviews JSON Lines | Amazon | This data is available for anyone to use under… | public |
S2AND: Author Name Disambiguation Benchmark, Eight Bibliographic Datasets plus Semantic Scholar Production Models (Allen Institute for AI) JSON | Allen Institute for AI, Semantic Scholar (Shivashankar Subra | ODC-By-1.0 | public |
S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI) JSON | Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, S | ODC-By-1.0 | public |
Shopping Humor Generation
| Amazon | Dataset is published under… | public |
SMOL and GATITOS: Professional Translations into 221 Low-Resource Languages (Google Research) JSON Lines | Google Research (Caswell et al.) | CC-BY-4.0 | public |
StreamingQA: Time-Stamped Questions over WMT English News for Testing How QA Models Adapt to New Knowledge (Google DeepMind) JSON Lines | Google DeepMind (Liska, Kocisky, Gribovskaya, Terzi et al.) | CC-BY-4.0 | public |
Sudachi Language Resources
| Works Applications | Apache-2.0 | public |
Wiki Movies
| Meta AI | cc-by-3.0 | public |
WikiDomains: 22,561 Wikipedia Term Definitions across 13 Academic Domains with Annotated Difficult Concepts for Targeted Concept Simplification (Google DeepMind) TSV | Google DeepMind (Asthana, Rashkin, Clark, Huot, Lapata) | CC-BY-SA-4.0 | public |
XQuAD (Cross-lingual Question Answering Dataset): 1,190 SQuAD Questions Professionally Translated into 11 Languages (Google DeepMind) Parquet | DeepMind (Artetxe, Ruder and Yogatama) | CC-BY-SA-4.0 | public |
XQuAD-R: Cross-lingual Answer Sentence Retrieval Benchmark, 11-Way Parallel Questions and Answers (Google Research LAReQA) Parquet | Google Research (Roy et al.) | CC-BY-SA-4.0 | public |
AdvancedIF CSV | Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
Anli Parquet | Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
CantTalkAboutThis Topic Control Dataset NC JSON Lines | NVIDIA | cc-by-nc-4.0 non-commercial or no-derivatives | public |
COLD Cases: 8.3 Million US Court Decisions with Opinions and Metadata (Harvard Library Innovation Lab, CourtListener) Parquet | Harvard Library Innovation Lab; Free Law Project (CourtListe | Public Domain | public |
CourtListener Bulk Legal Data: US Court Opinions, Dockets, Citation Graph, Judges, Oral Arguments and Financial Disclosures as Quarterly PostgreSQL CSV Exports (Free Law Project) CSV | Free Law Project | Public Domain | public |
CourtListener Case Law Embeddings: Chunked US Court Opinion Text with 768-Dimensional ModernBERT Vectors for Semantic Legal Search (Free Law Project) JSON | Free Law Project | Public Domain | public |
Curiosity Dialogs
| Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
Empathetic Dialogues
| Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
End of Term Web Archive Dataset
| End of Term Web Archive | Custom (see documentation) | public |
ExploreToM CSV | Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
Google Books Ngrams
| Not managed | Creative Commons Attribution 3.0 Unported… | public |
Imppres Parquet | Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
Japanese Tokenizer Dictionaries
| Cotonoha | Custom (see documentation) | public |
Provision of Web-Scale Parallel Corpora for Official European Languages (ParaCrawl)
| ParaCrawl | Creative Commons CC0 license ("no rights… | public |
Wiki Dpr Parquet | Meta AI | cc-by-nc-4.0 non-commercial or no-derivatives | public |
Common Crawl
| Common Crawl | Common Crawl Terms of Use | credentialed |