S2AND: Author Name Disambiguation Benchmark, Eight Bibliographic Datasets plus Semantic Scholar Production Models (Allen Institute for AI)

Provider
Allen Institute for AI, Semantic Scholar (Shivashankar Subramanian, Daniel King, Doug Downey, Sergey Feldman)
License
ODC-By-1.0 (source)
Format
JSON (in sampled files)
Stored at
Amazon S3, s3://ai2-s2-research-public/s2and-release (us-west-2)
Access
Free and public. No account with the data provider is needed.
Category
Natural Language Processing
Source
https://api.semanticscholar.org/CorpusID:232233421

About this dataset

S2AND is a benchmark for author name disambiguation (AND): deciding which author mentions (signatures) on scholarly papers belong to the same real person. It unifies eight existing labeled AND datasets into one format, each in its own folder: aminer, arnetminer, inspire, kisti, medline, pubmed, qian and zbmath, each with signatures, papers and gold cluster files, organized into name blocks (for example all 'j smith' signatures). The release also holds full_union_seed_1-5 training pickles, name_counts.pickle (first and last name frequency counts), a fastText language-ID model (lid.176.bin), Semantic Scholar production clustering models (production_model_v1.0 to v1.2), and s2and_and_big_blocks_linker_dataset_20260513/. A newer Arrow layout is at s2and-release-arrow/ in the same bucket. Load with the s2and Python package (ANDData). Some affiliation data comes from the Microsoft Academic Graph. Used for bibliometrics, scholarly knowledge graphs and entity resolution research. License: dataset under ODC-BY. Cite Subramanian, King, Downey and Feldman, S2AND: A Benchmark and Evaluation System for Author Name Disambiguation, JCDL 2021.

How to access this dataset in Python

import boto3
from botocore import UNSIGNED
from botocore.client import Config
s3 = boto3.client('s3', region_name='us-west-2', config=Config(signature_version=UNSIGNED))
s3.list_objects_v2(Bucket='ai2-s2-research-public', Prefix='s2and-release/', Delimiter='/')

Lists the files with anonymous access. Runs on any machine, with no cloud account.

Data structure

File types in sampled files: JSON

Files

Example file: s2and-release/.feature_cache/aminer/51/all_features.json

Analyze this dataset without downloading it

On Scigantic, S2AND: Author Name Disambiguation Benchmark, Eight Bibliographic Datasets plus Semantic Scholar Production Models (Allen Institute for AI) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Natural Language Processing datasets