PromoterZoo: 13.6 Million Multi-Species Promoter Sequences for Training the LOL-EVE Variant Effect Model (Harvard Marks Lab)

Provider
Marks Lab, Harvard Medical School
License
MIT (source)
Size
13 files
Format
Parquet, Arrow (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/Marks-lab/PromoterZoo)
Access
Free and public. No account with the data provider is needed.
Category
Genomics & Bioinformatics
Source
https://huggingface.co/datasets/Marks-lab/PromoterZoo

About this dataset

Training data for LOL-EVE (Language of Life, Evolutionary Variant Effects), a genomic language model from the Marks Lab at Harvard Medical School that scores the effect of indels and other variants in gene promoters. The dataset holds 13,611,373 promoter DNA sequences of 1,000 base pairs taken around transcription start sites (TSS) across many species grouped into evolutionary clades, giving the model evolutionary context for each gene's regulatory region. Each record has the sequence, gene symbol, species, clade, chromosome, strand, genomic start and stop, sequence length and direction (towards or away from the TSS), and both strands are included. Stored as one 6.4 GB data.parquet plus Arrow shards under dataset/, it loads with the datasets library or pandas and can be filtered by gene or species. Suited to pretraining DNA language models, comparative promoter analysis and cis-regulatory evolution studies. Released by the Marks Lab under the MIT License.

How to load this dataset in Python

from datasets import load_dataset
ds = load_dataset("Marks-lab/PromoterZoo")
print(ds)

Data structure

File types in sampled files: Parquet, Arrow

Files

Example file: data.parquet

Analyze this dataset without downloading it

On Scigantic, PromoterZoo: 13.6 Million Multi-Species Promoter Sequences for Training the LOL-EVE Variant Effect Model (Harvard Marks Lab) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Genomics & Bioinformatics datasets