# Google Books Ngram Viewer Datasets Version 3 (20200217): Yearly 1-gram to 5-gram Counts across Eight Languages and English Subcorpora (Google Books)

## Facts

- Provider: Google Books
- License: CC-BY-3.0 (https://storage.googleapis.com/books/ngrams/books/20200217/datasetsv3.html)
- Stored at: Google Cloud Storage, gs://books/ngrams/books/20200217 (us)
- Access: public, readable without an account
- Category: Natural Language Processing
- Source: https://books.google.com/ngrams
- Page: https://scigantic.com/archive/7d714642-3431-584d-b56c-708f80cb1ee0
- Keywords: Natural Language Processing, Google Cloud Storage

## About

Version 3 of the Google Books Ngram datasets, generated in February 2020 and identified as 20200217: the raw counts behind the Google Books Ngram Viewer. For each n-gram (1-grams through 5-grams, with part-of-speech tagged forms such as _NUM and _NOUN) the files give, per publication year, the number of occurrences (match_count) and the number of distinct books it appeared in (volume_count). Each line is the n-gram followed by tab-separated year,match_count,volume_count triplets. Only n-grams occurring more than 40 times across the corpus are included; total_counts files give the full number of 1-grams per year for computing relative frequencies. The folder has one subfolder per corpus: eng (English), eng-us (American English), eng-gb (British English), eng-fiction (English fiction), chi_sim (simplified Chinese), fre (French), ger (German), heb (Hebrew), ita (Italian), rus (Russian) and spa (Spanish). Each corpus is sharded into gzip files named - -of- .gz (the English 1-grams are 24 shards, the larger orders hundreds of shards of about 0.5 GB each), with an HTML export page per order listing the shards. This is a newer release than the 2009 and 2012 versions mirrored on AWS. Used in culturomics, historical linguistics, lexicography and studies of word frequency change over centuries. License: Creative Commons Attribution 3.0 Unported, stated on the export page. Cite Michel et al., Quantitative Analysis of Culture Using Millions of Digitized Books (Science 2011).

## Load the data

```python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('books/ngrams/books/20200217/', detail=False)[:10]
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

Example file: `ngrams/books/20200217/chi_sim/0-00000-of-00007.gz`

Path pattern: `{id}/{seg1}/{YYYYMMDD}/{seg3}/{filename}`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/7d714642-3431-584d-b56c-708f80cb1ee0
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/7d714642-3431-584d-b56c-708f80cb1ee0
- Over MCP: `get_schema_card` with id `7d714642-3431-584d-b56c-708f80cb1ee0` at https://api.scigantic.com/mcp

## Related topics

- [Natural Language Processing datasets](https://scigantic.com/topics/natural-language-processing)
