UCSF Industry Documents Library: metadata, documents and extracted text (tobacco, opioid, chemical, drug, food and fossil fuel industries)

The full UCSF Industry Documents Library as objects on S3: internal corporate records disclosed through litigation and regulatory action against the tobacco, opioid, chemical, pharmaceutical, food and fossil fuel industries, together with per-document metadata and machine-extracted text. This is the corpus behind much public-health policy research on industry conduct, and the extracted .ocr text makes it directly usable for text mining rather than only page-by-page reading. Documents are addressed by a hex-sharded path, four single-character directory levels then the document id, for example f/f/b/b/ffbb0000/ffbb0000.ocr, so any single directory lists quickly even though the whole corpus is very large. Measured 2026-10-27 over a 4,000-object sample: each document appears as a set of companions sharing one id, .ocr extracted text alongside the .pdf, a .tif page image and a _thumb.png thumbnail, with occasional .zip bundles; sixteen top-level shards were present (f/ g/ h/ j/ k/ l/ m/ n/ p/ q/ r/ s/ t/ x/ y/ z/). The object count was not enumerated exhaustively. This catalogue already holds the Opioid Industry Documents Archive, which is one collection within this library; this row is the whole library. Use is governed by the UCSF Industry Documents Library copyright and use terms, which vary by collection.

Category
Life Sciences
Provider
University of California, San Francisco (UCSF)
License
UCSF Industry Documents Library copyright and use terms (varies by collection)
Hosted on
Amazon S3, bucket ucsf-idl-dataset, us-east-1
Access
Free and public. No account with the data provider is needed.
Source
https://registry.opendata.aws/idl/
Data structure and loading code

Files

Example file: f/f/b/b/ffbb0000/ffbb0000.ocr

Path pattern: <c1>/<c2>/<c3>/<c4>/<docid>/<docid>.{ocr,pdf,tif} plus <docid>_thumb.png — four single-character hex shard levels, then one directory per document

Analyze this dataset without downloading it

On Scigantic, UCSF Industry Documents Library: metadata, documents and extracted text (tobacco, opioid, chemical, drug, food and fossil fuel industries) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

Related Life Sciences datasets