arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset)
- Provider
- Cornell University arXiv; mirrored by Hugging Face librarian-bots
- License
- CC0-1.0 (source)
- Size
- 13 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/librarian-bots/arxiv-metadata-snapshot)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Machine Learning & AI
- Source
- https://www.kaggle.com/datasets/Cornell-University/arxiv
About arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset)
Descriptive metadata for every paper on arXiv, mirrored weekly from the arXiv metadata snapshot that Cornell University publishes (the metadata portion of the Kaggle arXiv dataset). Each row is one e-print with its arXiv id, submitter, author list, title, comments (pages, figures), journal reference, DOI, report number, subject categories, licence, abstract, version history with dates and update date. Full texts are not included; arxiv.org/abs/{id} and arxiv.org/pdf/{id} resolve each paper. Listing data/ (measured 2026-11-14) shows 11 Parquet shards of about 0.17 to 0.28 GB, roughly 2.7 GB in total; the repository's stored history is much larger because each weekly sync adds a revision. The data support scientometrics and science-of-science studies, topic modelling and classification of papers into arXiv categories, author and collaboration analysis, citation-free trend tracking, literature search and recommendation, and building embeddings for retrieval. Load with Hugging Face datasets, pandas, polars or DuckDB. arXiv makes its descriptive metadata available under the CC0 1.0 Public Domain Dedication; paper full texts carry their own licences.
How to load this dataset in Python
from datasets import load_dataset
ds = load_dataset("librarian-bots/arxiv-metadata-snapshot")
print(ds)Data structure
File types in sampled files: Parquet
Files
Example file: data/train-00000-of-00011.parquet
Analyze this dataset without downloading it
On Scigantic, arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.