NovoMCP Open Corpus (lite): 122M PubChem compounds with precomputed descriptors and ADMET predictions

A precomputed cheminformatics substrate covering essentially all of PubChem: 122,454,458 compounds, one row per PubChem CID, stored as 21 Apache Parquet shards (Snappy, 36.8 GB) under the novomcp-open-corpus-lite/ prefix of the AWS Open Data bucket novomcp-open-corpus in us-east-2. Anonymous reads, no credentials needed. Each row carries 59 columns: the identifiers (cid, smiles, molecular_formula), RDKit physicochemical descriptors (molecular_weight, tpsa, complexity, fsp3, heavy_atom_count, aromatic_atom_count, aromatic_ring_count, rotatable_bonds, hydrogen-bond donor and acceptor counts under two different definitions, and two logP columns), drug-likeness scores (qed, synthetic_accessibility), roughly 30 ADMET and toxicity predictions from NovoMCP models trained on the public Therapeutics Data Commons benchmarks (absorption, distribution, CYP inhibition and substrate calls across six isoforms, hepatotoxicity, hERG, Ames mutagenicity, carcinogenicity, cardiotoxicity, LD50), and RDKit PAINS structural-alert flags (has_pains, pains_severity, pains_types). The point of the corpus is that these are already computed: a whole-PubChem property filter is one columnar scan rather than a 122-million-molecule RDKit and inference job, so virtual screening, QED filtering, chemical-space exploration, and assembling ADMET training sets run in minutes instead of machine-weeks. Every value in the ADMET and toxicity columns is a model prediction, not an experimental measurement, and carries no regulatory weight. Three operational details govern any query written against it. The shards are banded by molecular weight rather than partitioned by compound id: shards 00000 to 00007 hold everything under about 200 daltons, 00008 to 00011 hold the 200 to 400 dalton bulk (56 percent of all rows), 00012 to 00015 hold 400 to 600, 00016 to 00019 hold roughly 600 to 10,000, and 00020 holds the polymers and biologics above that. So a molecular weight predicate prunes whole shards, but no single shard is a sample of PubChem, and any statistic drawn from one is size-biased in a way that looks entirely plausible. Compound id is not sorted in any shard (each spans nearly the full CID range 1 to 176,730,695), so a single-compound lookup scans the whole corpus and there is no predicate pushdown on cid. And the shards are extremely uneven, with two of the 21 holding 53 percent of the rows, so per-file parallelism load-balances badly. The bucket also carries a second copy of the same 59-column data under the archive/lite-v1/ prefix as 1,409 small shards (27.4 GB), the pre-compaction layout, some shards holding only one or two rows. Published by NovoMCP, the open computational-chemistry engine that produced the corpus, under CC-BY-4.0, with a Zenodo DOI at 10.5281/zenodo.21710895 and a Kaggle mirror.

Category
Drug Discovery
Provider
NovoMCP (Ari Harrison)
License
CC-BY-4.0
Hosted on
Amazon S3, bucket novomcp-open-corpus/novomcp-open-corpus-lite/, us-east-2
Access
Free and public. No account with the data provider is needed.
Source
https://registry.opendata.aws/novomcp-open-corpus-lite/
Data structure and loading code

Fields

cidint64
smilesstring
molecular_formulastring
molecular_weightdouble
xlogpdouble
tpsadouble
complexitydouble
fsp3double
hbadouble
hba_countint32
hbddouble
hbd_countint32
heavy_atom_countint32
aromatic_atom_countint64
aromatic_ring_countint64
rotatable_bondsint32
qeddouble
drug_likenessdouble
synthetic_accessibilitydouble
bioavailability_probabilitydouble

and 39 more

Load the data

import os, glob, duckdb, pyarrow.parquet as pq
MNT = os.environ['SCIGANTIC_MOUNT_PATH']
FILES = sorted(glob.glob(os.path.join(MNT, 'pubchem_novomcp_lite_*.parquet')))
print(len(FILES), 'shards', round(sum(os.path.getsize(f) for f in FILES)/2**30, 2), 'GiB')
print(duckdb.sql(f"DESCRIBE SELECT * FROM '{FILES[0]}'").df()[['column_name','column_type']].to_string())
print('TOTAL ROWS:', sum(pq.read_metadata(f).num_rows for f in FILES))

Written for a Scigantic notebook, where /mnt/archive is the mounted dataset.

Analyze this dataset without downloading it

On Scigantic, NovoMCP Open Corpus (lite): 122M PubChem compounds with precomputed descriptors and ADMET predictions opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

Related Drug Discovery datasets