MBPP (Mostly Basic Python Problems): 974 Crowd-Sourced Python Programming Tasks with Reference Solutions and Unit Tests (Google Research)
- Provider
- Google Research (Austin et al.)
- License
- CC-BY-4.0 (source)
- Size
- 10 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/google-research-datasets/mbpp)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Computer Science
- Source
- https://arxiv.org/abs/2108.07732
About this dataset
MBPP (Mostly Basic Python Problems) is a program synthesis benchmark of 974 crowd-sourced Python programming problems designed to be solvable by entry-level programmers, covering programming fundamentals and standard library functionality. Each problem has a short natural language task description, a reference code solution and three automated assert test cases, so generated code can be scored by execution (pass@k). A hand-verified subset of 427 problems is provided as the sanitized configuration. The authors' split uses task IDs 11 to 510 for testing, 1 to 10 for few-shot prompts, 511 to 600 for validation and 601 to 974 for training. The repository holds two folders of Parquet files, full/ and sanitized/, each with train, validation, test and prompt splits (well under 1 MB). It is one of the standard benchmarks for code generation by large language models, alongside HumanEval. Load with the Hugging Face datasets library (load_dataset('google-research-datasets/mbpp')). Licensed CC BY 4.0 by Google Research; cite Austin et al., Program Synthesis with Large Language Models, 2021.
How to load this dataset in Python
from datasets import load_dataset
ds = load_dataset("google-research-datasets/mbpp")
print(ds)Data structure
File types in sampled files: Parquet
Files
Example file: full/prompt-00000-of-00001.parquet
Analyze this dataset without downloading it
On Scigantic, MBPP (Mostly Basic Python Problems): 974 Crowd-Sourced Python Programming Tasks with Reference Solutions and Unit Tests (Google Research) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.