# WIT Base: Wikimedia release of Wikipedia-based Image Text images

## Facts

- Provider: Wikimedia Research (images); Google Research (WIT dataset)
- License: CC-BY-SA-4.0 (https://huggingface.co/datasets/wikimedia/wit_base)
- Size: 335 files
- Format: Parquet (in sampled files)
- Stored at: Hugging Face, https://huggingface.co/datasets/wikimedia/wit_base
- Access: public, readable without an account
- Category: Computer Vision
- Source: https://arxiv.org/abs/2103.01913
- Page: https://scigantic.com/archive/3532b740-b1b6-503f-bae3-a55cb346a87d
- Keywords: Computer Vision, Parquet, Hugging Face

## About

WIT Base is the Wikimedia Research team's release of the image files behind the Wikipedia-based Image Text (WIT) dataset, the multilingual image-text corpus of more than 37 million image-text associations from Wikipedia articles in 108 languages published by Google Research (SIGIR 2021). Google's WIT gives only image names and URLs; Wikimedia released the pixels so that models can be trained directly. This copy contains more than six million images from Wikipedia articles in 100+ languages, resized to 300 px, covering nearly all captioned images in WIT that have a non-null reference description. Images where a person's face covers more than 10% of the frame were excluded for privacy (detected with RetinaFace), as were images that were candidates for deletion on Wikimedia Commons. Each row pairs an image with its reference and attribution descriptions, page and section text and URLs in its language, and was released for the Wikipedia image caption matching Kaggle challenge. Uses: multilingual image captioning, image-text retrieval, contrastive vision-language pretraining. Measured 2026-11-14: 330 Parquet shards in data/, about 308 GB; open with the datasets library (streaming recommended), pandas or pyarrow. Wikimedia releases it under CC BY-SA 4.0; credit Wikimedia and the WIT paper (Srinivasan et al. 2021).

## Load the data

```python
from datasets import load_dataset
ds = load_dataset("wikimedia/wit_base")
print(ds)
```

## Data structure

File types in sampled files: Parquet

Example file: `data/train-00000-of-00330.parquet`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/3532b740-b1b6-503f-bae3-a55cb346a87d
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/3532b740-b1b6-503f-bae3-a55cb346a87d
- Over MCP: `get_schema_card` with id `3532b740-b1b6-503f-bae3-a55cb346a87d` at https://api.scigantic.com/mcp

## Related topics

- [Parquet datasets](https://scigantic.com/topics/parquet)
- [Computer Vision datasets](https://scigantic.com/topics/computer-vision)
