Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books)
- Provider
- Google Books
- License
- CC-BY-3.0 (source)
- Stored at
- Google Cloud Storage, gs://books/ngrams/books/20200217 (us)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Natural Language Processing
- Source
- https://books.google.com/ngrams
About this dataset
Version 3 of the Google Books Ngram datasets, generated in February 2020 and identified as 20200217: the raw counts behind the Google Books Ngram Viewer. For each n-gram (1-grams through 5-grams, with part-of-speech tagged forms such as _NUM and _NOUN) the files give, per publication year, the number of occurrences (match_count) and the number of distinct books it appeared in (volume_count). Each line is the n-gram followed by tab-separated year,match_count,volume_count triplets. Only n-grams occurring more than 40 times across the corpus are included; total_counts files give the full number of 1-grams per year for computing relative frequencies. The folder has one subfolder per corpus: eng (English), eng-us (American English), eng-gb (British English), eng-fiction (English fiction), chi_sim (simplified Chinese), fre (French), ger (German), heb (Hebrew), ita (Italian), rus (Russian) and spa (Spanish). Each corpus is sharded into gzip files named - -of- .gz (the English 1-grams are 24 shards, the larger orders hundreds of shards of about 0.5 GB each), with an HTML export page per order listing the shards. This is a newer release than the 2009 and 2012 versions mirrored on AWS. Used in culturomics, historical linguistics, lexicography and studies of word frequency change over centuries. License: Creative Commons Attribution 3.0 Unported, stated on the export page. Cite Michel et al., Quantitative Analysis of Culture Using Millions of Digitized Books (Science 2011).
How to access this dataset in Python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('books/ngrams/books/20200217/', detail=False)[:10]Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
Files
Example file: ngrams/books/20200217/chi_sim/0-00000-of-00007.gz
Path pattern: {id}/{seg1}/{YYYYMMDD}/{seg3}/{filename}
Analyze this dataset without downloading it
On Scigantic, Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Natural Language Processing datasets
- CourtListener Case Law Embeddings: Chunked US Court Opinion Text with 768-Dimensional ModernBERT Vectors for Semantic Legal Search (Free Law Project)
- MultiCoNER Datasets
- Belebele
- XQuAD (Cross-lingual Question Answering Dataset): 1,190 SQuAD Questions Professionally Translated into 11 Languages (Google DeepMind)
- Md Gender Bias
- S2AND: Author Name Disambiguation Benchmark, Eight Bibliographic Datasets plus Semantic Scholar Production Models (Allen Institute for AI)
- IndicGenBench XQuAD-IN: Extractive Question Answering Translated into Indic Languages (Google Research)
- BFCL Hi