WIT Base: Wikimedia release of Wikipedia-based Image Text images

Provider
Wikimedia Research (images); Google Research (WIT dataset)
License
CC-BY-SA-4.0 (source)
Size
335 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/wikimedia/wit_base)
Access
Free and public. No account with the data provider is needed.
Category
Computer Vision
Source
https://arxiv.org/abs/2103.01913

About WIT Base: Wikimedia release of Wikipedia-based Image Text images

WIT Base is the Wikimedia Research team's release of the image files behind the Wikipedia-based Image Text (WIT) dataset, the multilingual image-text corpus of more than 37 million image-text associations from Wikipedia articles in 108 languages published by Google Research (SIGIR 2021). Google's WIT gives only image names and URLs; Wikimedia released the pixels so that models can be trained directly. This copy contains more than six million images from Wikipedia articles in 100+ languages, resized to 300 px, covering nearly all captioned images in WIT that have a non-null reference description. Images where a person's face covers more than 10% of the frame were excluded for privacy (detected with RetinaFace), as were images that were candidates for deletion on Wikimedia Commons. Each row pairs an image with its reference and attribution descriptions, page and section text and URLs in its language, and was released for the Wikipedia image caption matching Kaggle challenge. Uses: multilingual image captioning, image-text retrieval, contrastive vision-language pretraining. Measured 2026-11-14: 330 Parquet shards in data/, about 308 GB; open with the datasets library (streaming recommended), pandas or pyarrow. Wikimedia releases it under CC BY-SA 4.0; credit Wikimedia and the WIT paper (Srinivasan et al. 2021).

How to load WIT Base: Wikimedia release of Wikipedia-based Image Text images in Python

from datasets import load_dataset
ds = load_dataset("wikimedia/wit_base")
print(ds)

Data structure

File types in sampled files: Parquet

Files

Example file: data/train-00000-of-00330.parquet

Analyze this dataset without downloading it

On Scigantic, WIT Base: Wikimedia release of Wikipedia-based Image Text images opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Computer Vision datasets