GeckoNum: Numerical Reasoning Benchmark for Text-to-Image Models, Generated Images from Imagen, Muse and DALL-E 3 with Counting Prompts (Google DeepMind)
- Provider
- Google DeepMind (Kajic, Wiles, Albuquerque, Bauer, Wang, Pont-Tuset, Nematzadeh)
- License
- CC-BY-4.0 (source)
- Stored at
- Google Cloud Storage, gs://geckonum_t2i_benchmark (us)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Computer Vision
- Source
- https://github.com/google-deepmind/geckonum_benchmark_t2i
About this dataset
GeckoNum evaluates whether text-to-image generative models draw the right number of objects. Its text prompts probe different aspects of numerical reasoning, from a single counted noun to prompts with two or three counted entities that must add up, with numbers written as words or as digits, grouped into 12 prompt types. The bucket holds the generated images, one folder per model (dalle_3, imagen_a, imagen_b, imagen_c, imagen_d, muse_a, muse_b), with PNG files named geckonum_ _ .png for five seeds per prompt, plus geckonum.zip (18.8 GB) with all images and browsable index.html pages. The text prompts (prompts.csv) and the human annotations for three rating tasks (CSV per task and model) are in the GeckoNum GitHub repository and join to the images by prompt id. Used to benchmark object counting in image generation, compare automatic evaluation metrics with human judgments, and train or test text-image alignment scorers. License: CC BY 4.0 for the materials (software Apache 2.0), stated in the repository. Cite Kajic et al., Evaluating Numerical Reasoning in Text-to-Image Models (NeurIPS 2024).
How to access this dataset in Python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('geckonum_t2i_benchmark/', detail=False)[:10]Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
Files
Example file: css/
Analyze this dataset without downloading it
On Scigantic, GeckoNum: Numerical Reasoning Benchmark for Text-to-Image Models, Generated Images from Imagen, Muse and DALL-E 3 with Counting Prompts (Google DeepMind) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Computer Vision datasets
- Nemotron Image Training V 3
- SKZ 3D Printing Defect Detection, Platform Camera: YOLO-Labeled FFF Print Errors
- PLM Video Human
- Nemotron Personas France
- Common O
- MMLA The Wilds: YOLO-Annotated Drone Frames of Painted Dogs, Onagers, Giraffes and Grevy's Zebras
- ShapeR Evaluation
- WIT Base: Wikimedia release of Wikipedia-based Image Text images