# S2APLER: Semantic Scholar Paper Clustering Dataset for Deduplicating Scholarly Paper Records (Allen Institute for AI)

## Facts

- Provider: Allen Institute for AI, Semantic Scholar (Shaurya Rohatgi, Sergey Feldman)
- License: ODC-By-1.0 (https://github.com/allenai/S2APLER)
- Format: JSON (in sampled files)
- Stored at: Amazon S3, s3://ai2-s2-research-public/paper_clustering (us-west-2)
- Access: public, readable without an account
- Category: Natural Language Processing
- Source: https://github.com/allenai/S2APLER
- Page: https://scigantic.com/archive/fad16ac1-cc25-5b62-84dd-42a68c36428f
- Keywords: Natural Language Processing, JSON, AWS Open Data

## About

S2APLER (Semantic Scholar Agglomeration of Papers with Low Error Rate) is the dataset and model Semantic Scholar uses to cluster records that describe the same scholarly paper, such as a preprint, a journal version and duplicate entries from different sources. The paper_clustering/ folder holds papers.json (about 3.3 GB of paper records with metadata used for pairwise features) and clusters.json (gold clusters of records that are the same work), about 3.1 GiB in total. The reported pairwise model reaches AUROC 0.999 and the final clustering B3 F1 0.9986 on the test set, about a 50 percent error reduction over the previous Semantic Scholar paper clustering. Load with the s2apler Python package from allenai/S2APLER. Some affiliation data comes from the Microsoft Academic Graph. Used for record linkage, deduplication of bibliographic databases and scholarly knowledge graph construction. License: dataset under ODC-BY. Cite the allenai/S2APLER repository (Rohatgi, Feldman et al., Allen Institute for AI).

## Load the data

```python
import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-s2-research-public', Prefix='paper_clustering/', Delimiter='/')
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

File types in sampled files: JSON

Example file: `paper_clustering/clusters.json`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/fad16ac1-cc25-5b62-84dd-42a68c36428f
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/fad16ac1-cc25-5b62-84dd-42a68c36428f
- Over MCP: `get_schema_card` with id `fad16ac1-cc25-5b62-84dd-42a68c36428f` at https://api.scigantic.com/mcp

## Related topics

- [JSON datasets](https://scigantic.com/topics/json)
- [Natural Language Processing datasets](https://scigantic.com/topics/natural-language-processing)
