How to query 122 million PubChem compounds with ADMET predictions in a Jupyter notebook
The corpus is 21 Parquet shards, so DuckDB or pyarrow can query all 122 million rows by reading only the columns a query needs. In a Scigantic notebook the 34 GiB table is mounted at /mnt/archive with the shards directly at the mount root.
The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is s3://novomcp-open-corpus/novomcp-open-corpus-lite/ (AWS us-east-2, public, no account needed).
Open NovoMCP Open Corpus (PubChem with ADMET predictions) on SciganticOpen the sample notebook: Predicted everywhere, measured where it counts
Code
List the shards, show the schema and count rows
import os, glob, duckdb, pyarrow.parquet as pq
MNT = os.environ['SCIGANTIC_MOUNT_PATH']
FILES = sorted(glob.glob(os.path.join(MNT, 'pubchem_novomcp_lite_*.parquet')))
print(len(FILES), 'shards', round(sum(os.path.getsize(f) for f in FILES)/2**30, 2), 'GiB')
print(duckdb.sql(f"DESCRIBE SELECT * FROM '{FILES[0]}'").df()[['column_name','column_type']].to_string())
print('TOTAL ROWS:', sum(pq.read_metadata(f).num_rows for f in FILES))
The starter cell from this dataset's schema card. It ran without error on 2026-11-17 in the Scigantic notebook image against the mounted data.
Read one row group from the largest shard
import os
import pyarrow.parquet as pq
pf = pq.ParquetFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'pubchem_novomcp_lite_00010.parquet'))
print(pf.metadata)
df = pf.read_row_group(0).to_pandas()
df.head()
Run on 2026-11-17 in the Scigantic notebook image against the mounted data. It returned: Parquet 32,937,992 rows, 59 columns, 32 row groups; columns ['cid:int64', 'smiles:string', 'molecular_formula:string', 'molecular_weight:double', 'xlogp:double', 'tpsa:double', 'complexity:double', 'fsp3:double', 'hba:double', 'hba_count:int32']
What the files are
NovoMCP Open Corpus (lite) is a single table of 122,454,458 PubChem compounds with 59 columns: RDKit-style descriptors, QED, synthetic accessibility and drug-likeness scores, PAINS flags, and predicted ADMET, CYP and toxicity probabilities.
The ADMET values are model predictions, not measurements.
How the files are laid out
| Path under the mount | What it holds |
|---|---|
| pubchem_novomcp_lite_00000.parquet ... _00020.parquet | 21 Snappy-compressed shards at the mount root, with no subfolders and no manifest. |
| pubchem_novomcp_lite_00010.parquet | The largest shard: 32,937,992 rows in 32 row groups, 9.7 GB. |
Size, format and license
| Format | Parquet (Snappy) |
|---|---|
| Size | 21 files, 34.27 GiB, 122,454,458 rows by 59 columns |
| License | CC BY 4.0 |
| Provider | NovoMCP (Ari Harrison), on AWS Open Data |
| Public source | s3://novomcp-open-corpus/novomcp-open-corpus-lite/ (AWS us-east-2, public, no account needed) |
| Notebook image | Minimal (DuckDB and pyarrow preinstalled) |
Things to know before you read it
- Shard sizes are very uneven: two hold over 30 million rows each, and five are stubs of a few thousand rows.
- Shards are not partitioned by compound ID, so looking up one CID has to scan all 21 files. Select only the columns you need.
Run it
Start a notebook from the NovoMCP Open Corpus (PubChem with ADMET predictions) archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first. For a longer worked example, open the sample notebook Predicted everywhere, measured where it counts.
Related guides
- How to query ChEMBL bioactivity data as Parquet without installing the database
- How to open the OpenADMET CYP inhibition challenge data in a Jupyter notebook
- How to open the Alchemistry 2027 dengue binding free energy challenge data in a Jupyter notebook
- How to open Tahoe-100M single-cell perturbation data in a Jupyter notebook
Last checked 2026-11-21. All dataset guides.