# GeckoNum: Numerical Reasoning Benchmark for Text-to-Image Models, Generated Images from Imagen, Muse and DALL-E 3 with Counting Prompts (Google DeepMind)

## Facts

- Provider: Google DeepMind (Kajic, Wiles, Albuquerque, Bauer, Wang, Pont-Tuset, Nematzadeh)
- License: CC-BY-4.0 (https://github.com/google-deepmind/geckonum_benchmark_t2i)
- Stored at: Google Cloud Storage, gs://geckonum_t2i_benchmark (us)
- Access: public, readable without an account
- Category: Computer Vision
- Source: https://github.com/google-deepmind/geckonum_benchmark_t2i
- Page: https://scigantic.com/archive/047cb975-f8f3-5880-b538-8450520bf8a6
- Keywords: Computer Vision, benchmark, Google Cloud Storage

## About

GeckoNum evaluates whether text-to-image generative models draw the right number of objects. Its text prompts probe different aspects of numerical reasoning, from a single counted noun to prompts with two or three counted entities that must add up, with numbers written as words or as digits, grouped into 12 prompt types. The bucket holds the generated images, one folder per model (dalle_3, imagen_a, imagen_b, imagen_c, imagen_d, muse_a, muse_b), with PNG files named geckonum_ _ .png for five seeds per prompt, plus geckonum.zip (18.8 GB) with all images and browsable index.html pages. The text prompts (prompts.csv) and the human annotations for three rating tasks (CSV per task and model) are in the GeckoNum GitHub repository and join to the images by prompt id. Used to benchmark object counting in image generation, compare automatic evaluation metrics with human judgments, and train or test text-image alignment scorers. License: CC BY 4.0 for the materials (software Apache 2.0), stated in the repository. Cite Kajic et al., Evaluating Numerical Reasoning in Text-to-Image Models (NeurIPS 2024).

## Load the data

```python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('geckonum_t2i_benchmark/', detail=False)[:10]
```

Lists the files with anonymous access. Runs on any machine, with no cloud account.

## Data structure

Example file: `css/`

## Open it

- In a hosted notebook with the data mounted: https://scigantic.com/archive/047cb975-f8f3-5880-b538-8450520bf8a6
- Full record as JSON (schema card included): https://api.scigantic.com/api/archives/047cb975-f8f3-5880-b538-8450520bf8a6
- Over MCP: `get_schema_card` with id `047cb975-f8f3-5880-b538-8450520bf8a6` at https://api.scigantic.com/mcp

## Related topics

- [Benchmark datasets](https://scigantic.com/topics/benchmark)
- [Computer Vision datasets](https://scigantic.com/topics/computer-vision)
