UCSF Industry Documents Library: metadata, documents and extracted text (tobacco, opioid, chemical, drug, food and fossil fuel industries)
The full UCSF Industry Documents Library as objects on S3: internal corporate records disclosed through litigation and regulatory action against the tobacco, opioid, chemical, pharmaceutical, food and fossil fuel industries, together with per-document metadata and machine-extracted text. This is the corpus behind much public-health policy research on industry conduct, and the extracted .ocr text makes it directly usable for text mining rather than only page-by-page reading. Documents are addressed by a hex-sharded path, four single-character directory levels then the document id, for example f/f/b/b/ffbb0000/ffbb0000.ocr, so any single directory lists quickly even though the whole corpus is very large. Measured 2026-10-27 over a 4,000-object sample: each document appears as a set of companions sharing one id, .ocr extracted text alongside the .pdf, a .tif page image and a _thumb.png thumbnail, with occasional .zip bundles; sixteen top-level shards were present (f/ g/ h/ j/ k/ l/ m/ n/ p/ q/ r/ s/ t/ x/ y/ z/). The object count was not enumerated exhaustively. This catalogue already holds the Opioid Industry Documents Archive, which is one collection within this library; this row is the whole library. Use is governed by the UCSF Industry Documents Library copyright and use terms, which vary by collection.
- Category
- Life Sciences
- Provider
- University of California, San Francisco (UCSF)
- License
- UCSF Industry Documents Library copyright and use terms (varies by collection)
- Hosted on
- Amazon S3, bucket ucsf-idl-dataset, us-east-1
- Access
- Free and public. No account with the data provider is needed.
- Source
- https://registry.opendata.aws/idl/
Data structure and loading code
Files
Example file: f/f/b/b/ffbb0000/ffbb0000.ocr
Path pattern: <c1>/<c2>/<c3>/<c4>/<docid>/<docid>.{ocr,pdf,tif} plus <docid>_thumb.png — four single-character hex shard levels, then one directory per document
Analyze this dataset without downloading it
On Scigantic, UCSF Industry Documents Library: metadata, documents and extracted text (tobacco, opioid, chemical, drug, food and fossil fuel industries) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
Related Life Sciences datasets
- OBIS species distribution models for marine species (MPA Europe, Bio-ORACLE v3.0 predictors)
- OpenOrganelle jrc_jurkat-1: Whole Jurkat T Cell, 4 nm FIB-SEM
- ASEM Pretrained Organelle Segmentation Models
- OpenFold3 Training Data
- National Reef Monitoring Network - Global reef fish abundance and biomass
- CZ Grand Challenges - Model Benchmarking
- Pacific Ocean Sound Recordings
- OpenOrganelle jrc_sum159-1: SUM159 Breast Cancer Cell with Induced Lipid Droplets, FIB-SEM