# arXiv Metadata Snapshot (weekly mirror of the Cornell arXiv OAI dataset)

## Facts

- Provider: Cornell University arXiv; mirrored by Hugging Face librarian-bots
- License: CC0-1.0 (https://info.arxiv.org/help/api/tou.html)
- Size: 13 files
- Format: Parquet (in sampled files)
- Stored at: Hugging Face, https://huggingface.co/datasets/librarian-bots/arxiv-metadata-snapshot
- Access: public, readable without an account
- Category: Machine Learning & AI
- Source: https://www.kaggle.com/datasets/Cornell-University/arxiv
- Page: https://scigantic.com/archive/4438a4e7-1990-52c8-aa77-2867a720b974
- Keywords: Machine Learning & AI, Parquet, Hugging Face

## About

Descriptive metadata for every paper on arXiv, mirrored weekly from the arXiv metadata snapshot that Cornell University publishes (the metadata portion of the Kaggle arXiv dataset). Each row is one e-print with its arXiv id, submitter, author list, title, comments (pages, figures), journal reference, DOI, report number, subject categories, licence, abstract, version history with dates and update date. Full texts are not included; arxiv.org/abs/{id} and arxiv.org/pdf/{id} resolve each paper. Listing data/ (measured 2026-11-14) shows 11 Parquet shards of about 0.17 to 0.28 GB, roughly 2.7 GB in total; the repository's stored history is much larger because each weekly sync adds a revision. The data support scientometrics and science-of-science studies, topic modelling and classification of papers into arXiv categories, author and collaboration analysis, citation-free trend tracking, literature search and recommendation, and building embeddings for retrieval. Load with Hugging Face datasets, pandas, polars or DuckDB. arXiv makes its descriptive metadata available under the CC0 1.0 Public Domain Dedication; paper full texts carry their own licences.

## Load the data

```python
from datasets import load_dataset
ds = load_dataset("librarian-bots/arxiv-metadata-snapshot")
print(ds)
```

## Data structure

File types in sampled files: Parquet

Example file: `data/train-00000-of-00011.parquet`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/4438a4e7-1990-52c8-aa77-2867a720b974
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/4438a4e7-1990-52c8-aa77-2867a720b974
- Over MCP: `get_schema_card` with id `4438a4e7-1990-52c8-aa77-2867a720b974` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Machine Learning & AI datasets](https://scigantic.com/topics/machine-learning-and-ai)
