S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)

Provider
Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, Sergey Feldman)
License
ODC-By-1.0 (source)
Format
JSON (in sampled files)
Stored at
Amazon S3, s3://ai2-s2-research-public/paper_clustering (us-west-2)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://github.com/allenai/S2APLER

About this dataset

S2APLER (Semantic Scholar Agglomeration of Papers with Low Error Rate) is the dataset and model Semantic Scholar uses to cluster records that describe the same scholarly paper, such as a preprint, a journal version and duplicate entries from different sources. The paper_clustering/ folder holds papers.json (about 3.3 GB of paper records with metadata used for pairwise features) and clusters.json (gold clusters of records that are the same work), about 3.1 GiB in total. The reported pairwise model reaches AUROC 0.999 and the final clustering B3 F1 0.9986 on the test set, about a 50 percent error reduction over the previous Semantic Scholar paper clustering. Load with the s2apler Python package from allenai/S2APLER. Some affiliation data comes from the Microsoft Academic Graph. Used for record linkage, deduplication of bibliographic databases and scholarly knowledge graph construction. License: dataset under ODC-BY. Cite the allenai/S2APLER repository (Rohatgi, Feldman et al., Allen Institute for AI).

How to access this dataset in Python

import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-s2-research-public', Prefix='paper_clustering/', Delimiter='/')

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: JSON

Files

Example file: paper_clustering/clusters.json

Analyze this dataset without downloading it

On Scigantic, S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets