SciganticGuidesTopicsExplore

How to query 1000 Genomes VCF files on S3 from a Jupyter notebook

Open the VCFs with pysam: pysam.VariantFile(path) reads a bgzipped, tabix-indexed VCF, and fetch(chrom, start, end) pulls one region without scanning the whole file. In a Scigantic notebook the 1000genomes bucket is mounted at /mnt/archive, so the multi-gigabyte files are read in place.

The code on this page runs in a Scigantic hosted notebook, where the dataset is mounted read-only at /mnt/archive. The same path is in the SCIGANTIC_MOUNT_PATH environment variable and linked from ~/archive. The public source is s3://1000genomes (AWS us-east-1, public, no account needed).

Open 1000 Genomes on Scigantic

Code

Sample populations and allele frequencies for one region of chr22

import os
import pandas as pd
import pysam

M = os.environ.get("SCIGANTIC_MOUNT_PATH", "/mnt/archive")
HC = f"{M}/1000G_2504_high_coverage"

# Who is in this release: 3,202 samples (2,504 unrelated plus 698 relatives), GRCh38.
ped = pd.read_csv(f"{HC}/additional_698_related/20130606_g1k_3202_samples_ped_population.txt", sep=" ")
print(ped.groupby("Superpopulation").size())

# chr22 sites with allele counts per superpopulation. The file is tabix-indexed, so a
# region reads in seconds without scanning all 361 MB. It has no per-sample genotypes.
pysam.set_verbosity(0)  # the header has a few non-standard tag names; skip the warnings
vcf = pysam.VariantFile(f"{HC}/working/20201028_3202_raw_GT_with_annot/"
                        "20201028_CCDG_14151_B01_GRM_WGS_2020-08-05_chr22.recalibrated_variants.annotated.vcf.gz")
pops = ["AFR", "AMR", "EAS", "EUR", "SAS"]
rows = []
for rec in vcf.fetch("chr22", 20_000_000, 20_100_000):
    af = {p: rec.info[f"AC_{p}"][0] / rec.info[f"AN_{p}"] for p in pops if rec.info.get(f"AN_{p}")}
    rows.append({"pos": rec.pos, "ref": rec.ref, "alt": rec.alts[0], **af})
df = pd.DataFrame(rows)
print(len(df), "variants in chr22:20.0-20.1 Mb")
df.head()

The starter cell from this dataset's schema card. It ran without error on 2026-11-15 in the Scigantic notebook image against the mounted data.

Read the first records of a VCF

import os
import pysam
with pysam.VariantFile(os.path.join(os.environ['SCIGANTIC_MOUNT_PATH'], 'technical/working/20150617_old_phase3_svs/ALL.wgs.integrated_sv_map_v1_old.20130502.svs.genotypes.vcf.gz')) as vf:
    print(list(vf.header.samples)[:5])
    for i, rec in enumerate(vf):
        if i >= 5: break
        print(rec.chrom, rec.pos, rec.ref, rec.alts)

Run on 2026-11-15 in the Scigantic notebook image against the mounted data. It returned: VCF/BCF: 2504 samples, contigs ['1', '2', '3', '4', '5']

What the files are

The bucket mirrors the 1000 Genomes Project: sequencing reads, alignments and variant calls for more than 2,500 people from 26 populations.

The starter code reads the high-coverage release: 3,202 samples (2,504 unrelated plus 698 relatives) on GRCh38, with a sample table and per-chromosome annotated VCFs. The Phase 3 genotype VCFs most tools reference are under release/20130502/.

How the files are laid out

Path under the mountWhat it holds
1000G_2504_high_coverage/The high-coverage GRCh38 release. additional_698_related/20130606_g1k_3202_samples_ped_population.txt lists every sample with its population and superpopulation.
1000G_2504_high_coverage/working/20201028_3202_raw_GT_with_annot/Per-chromosome annotated VCFs (..._chr22.recalibrated_variants.annotated.vcf.gz), tabix-indexed. They carry allele counts per superpopulation and no per-sample genotypes.
release/20130502/Phase 3 genotype VCFs, one per chromosome (ALL.chr2.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz), with the sample panel integrated_call_samples_v3.20130502.ALL.panel.
technical/Reference files and working directories from the project, including structural variant call sets.

Size, format and license

FormatVCF (bgzip with tabix index), plus text sample tables; BAM and CRAM alignments elsewhere in the bucket
Size12 top-level folders. The chr22 annotated VCF read above is 361 MB.
LicenseCustom; see the provider documentation linked from the archive page
ProviderNational Institutes of Health, on AWS Open Data
Public sources3://1000genomes (AWS us-east-1, public, no account needed)
Notebook imageGenomics (pysam, cyvcf2, bcftools and plink2 preinstalled)

Things to know before you read it

Run it

Start a notebook from the 1000 Genomes archive page. The data is mounted when the notebook opens, so the code above runs as written and nothing is downloaded first.

Related guides

Last checked 2026-11-21. All dataset guides.