How to query 1000 Genomes VCF files on S3 from a Jupyter notebook
Open the VCFs with pysam: pysam.VariantFile(path) reads a bgzipped, tabix-indexed VCF, and fetch(chrom, start, end) pulls one region without scanning the whole file. In a Scigantic notebook the 1000genomes bucket is mounted at /mnt/archive, so the multi-gigabyte files are read in place.
The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is s3://1000genomes (AWS us-east-1, public, no account needed).
Open 1000 Genomes on Scigantic
Code
Sample populations and allele frequencies for one region of chr22
import os
import pandas as pd
import pysam
M = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
HC = f"{M}/1000G_2504_high_coverage"
# Who is in this release: 3,202 samples (2,504 unrelated plus 698 relatives), GRCh38.
ped = pd.read_csv(f"{HC}/additional_698_related/20130606_g1k_3202_samples_ped_population.txt", sep=" ")
print(ped.groupby("Superpopulation").size())
# chr22 sites with allele counts per superpopulation. The file is tabix-indexed, so a
# region reads in seconds without scanning all 361 MB. It has no per-sample genotypes.
pysam.set_verbosity(0) # the header has a few non-standard tag names; skip the warnings
vcf = pysam.VariantFile(f"{HC}/working/20201028_3202_raw_GT_with_annot/"
"20201028_CCDG_14151_B01_GRM_WGS_2020-08-05_chr22.recalibrated_variants.annotated.vcf.gz")
pops = ["AFR", "AMR", "EAS", "EUR", "SAS"]
rows = []
for rec in vcf.fetch("chr22", 20_000_000, 20_100_000):
af = {p: rec.info[f"AC_{p}"][0] / rec.info[f"AN_{p}"] for p in pops if rec.info.get(f"AN_{p}")}
rows.append({"pos": rec.pos, "ref": rec.ref, "alt": rec.alts[0], **af})
df = pd.DataFrame(rows)
print(len(df), "variants in chr22:20.0-20.1 Mb")
df.head()
The starter cell from this dataset's schema card. It ran without error on 2026-11-15 in the Scigantic notebook image against the mounted data.
Read the first records of a VCF
import os
import pysam
with pysam.VariantFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'technical/working/20150617_old_phase3_svs/ALL.wgs.integrated_sv_map_v1_old.20130502.svs.genotypes.vcf.gz')) as vf:
print(list(vf.header.samples)[:5])
for i, rec in enumerate(vf):
if i >= 5: break
print(rec.chrom, rec.pos, rec.ref, rec.alts)
Run on 2026-11-15 in the Scigantic notebook image against the mounted data. It returned: VCF/BCF: 2504 samples, contigs ['1', '2', '3', '4', '5']
What the files are
The bucket mirrors the 1000 Genomes Project: sequencing reads, alignments and variant calls for more than 2,500 people from 26 populations.
The starter code reads the high-coverage release: 3,202 samples (2,504 unrelated plus 698 relatives) on GRCh38, with a sample table and per-chromosome annotated VCFs. The Phase 3 genotype VCFs most tools reference are under release/20130502/.
How the files are laid out
| Path under the mount | What it holds |
|---|---|
| 1000G_2504_high_coverage/ | The high-coverage GRCh38 release. additional_698_related/20130606_g1k_3202_samples_ped_population.txt lists every sample with its population and superpopulation. |
| 1000G_2504_high_coverage/working/20201028_3202_raw_GT_with_annot/ | Per-chromosome annotated VCFs (..._chr22.recalibrated_variants.annotated.vcf.gz), tabix-indexed. They carry allele counts per superpopulation and no per-sample genotypes. |
| release/20130502/ | Phase 3 genotype VCFs, one per chromosome (ALL.chr2.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz), with the sample panel integrated_call_samples_v3.20130502.ALL.panel. |
| technical/ | Reference files and working directories from the project, including structural variant call sets. |
Size, format and license
| Format | VCF (bgzip with tabix index), plus text sample tables; BAM and CRAM alignments elsewhere in the bucket |
|---|---|
| Size | 12 top-level folders. The chr22 annotated VCF read above is 361 MB. |
| License | Custom; see the provider documentation linked from the archive page |
| Provider | National Institutes of Health, on AWS Open Data |
| Public source | s3://1000genomes (AWS us-east-1, public, no account needed) |
| Notebook image | Genomics (pysam, cyvcf2, bcftools and plink2 preinstalled) |
Things to know before you read it
- Use
fetchwith a region. Iterating a whole-chromosome VCF from the start reads the entire file. - The annotated high-coverage VCFs hold site-level allele counts only. For per-sample genotypes use the Phase 3 files under
release/20130502/. - The Phase 3 files name chromosomes
1to22; the high-coverage GRCh38 files usechr1tochr22.
Run it
Start a notebook from the 1000 Genomes archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first.
Related guides
- How to open Tahoe-100M single-cell perturbation data in a Jupyter notebook
- How to open Sid Sijbrandij's osteosarcoma multi-omic dataset in a Jupyter notebook
- How to query ChEMBL bioactivity data as Parquet without installing the database
- How to open the Genomes to Fields (G2F) maize genotype by environment data in a Jupyter notebook
Last checked 2026-11-21. All dataset guides.