S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)
- Provider
- Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, Sergey Feldman)
- License
- ODC-By-1.0 (source)
- Format
- JSON (in sampled files)
- Stored at
- Amazon S3, s3://ai2-s2-research-public/paper_clustering (us-west-2)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://github.com/allenai/S2APLER
About this dataset
S2APLER (Semantic Scholar Agglomeration of Papers with Low Error Rate) is the dataset and model Semantic Scholar uses to cluster records that describe the same scholarly paper, such as a preprint, a journal version and duplicate entries from different sources. The paper_clustering/ folder holds papers.json (about 3.3 GB of paper records with metadata used for pairwise features) and clusters.json (gold clusters of records that are the same work), about 3.1 GiB in total. The reported pairwise model reaches AUROC 0.999 and the final clustering B3 F1 0.9986 on the test set, about a 50 percent error reduction over the previous Semantic Scholar paper clustering. Load with the s2apler Python package from allenai/S2APLER. Some affiliation data comes from the Microsoft Academic Graph. Used for record linkage, deduplication of bibliographic databases and scholarly knowledge graph construction. License: dataset under ODC-BY. Cite the allenai/S2APLER repository (Rohatgi, Feldman et al., Allen Institute for AI).
How to access this dataset in Python
import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-s2-research-public', Prefix='paper_clustering/', Delimiter='/')Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: JSON
Files
Example file: paper_clustering/clusters.json
Analyze this dataset without downloading it
On Scigantic, S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- Sudachi Language Resources
- MuP: Multi-Perspective Scientific Document Summarization Dataset from Peer Review Summaries (SDP 2022 Shared Task, Allen Institute for AI)
- COLD Cases: 8.3 Million US Court Decisions with Opinions and Metadata (Harvard Library Innovation Lab, CourtListener)
- IndicGenBench XorQA-IN: Cross-lingual Open-Retrieval Question Answering in 29 Indic Languages (Google Research)
- ExploreToM
- Shopping Humor Generation
- Common Screens
- Phrase Clustering Dataset (PCD)