RxR (Room-Across-Room): Multilingual Vision-and-Language Navigation Instructions in English, Hindi and Telugu with Pose Traces for Matterport3D (Google Research)
- Provider
- Google Research (Ku, Anderson, Patel, Ie, Baldridge)
- License
- CC-BY-4.0 (source)
- Format
- NumPy (in sampled files)
- Stored at
- Google Cloud Storage, gs://rxr-data (us)
- Access
- Free and public. No account with the data provider is needed.
- Category
- Machine Learning & AI
- Source
- https://github.com/google-research-datasets/RxR
About this dataset
Room-Across-Room (RxR) is a multilingual vision-and-language navigation (VLN) dataset: people describe paths through indoor environments of the Matterport3D building scans, and an agent must follow the spoken-then-transcribed instruction to the goal. RxR is about ten times larger than Room-to-Room (R2R), covers English (en-IN and en-US), Hindi and Telugu, and has longer, more varied paths. The bucket (about 161 GB) holds guide annotations (gzipped JSON Lines with path, scan id, viewpoints, initial heading, instruction text, word-level timestamps from the recording and annotator ids, for train, val_seen, val_unseen and test splits), follower annotations, pose_traces/ per split (NumPy archives recording where each annotator looked in the panorama as they spoke, which grounds each word to pixels and surfaces), text_features/ (precomputed multilingual text encodings), grounded landmark annotations, model-generated instructions and Gibson navigation graphs. The Matterport3D imagery itself is not included and is governed by the Matterport3D Terms of Use. Used for embodied AI agents, instruction following, multilingual grounding and instruction generation. Download with gsutil -m cp -R gs://rxr-data. License: the RxR annotations are released under CC BY 4.0. Cite Ku et al., Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (EMNLP 2020).
How to access this dataset in Python
import gcsfs
fs = gcsfs.GCSFileSystem(token='anon')
fs.ls('rxr-data/', detail=False)[:10]Lists the files with anonymous access. Runs on any machine, with no cloud account.
Data structure
File types in sampled files: NumPy
Files
Example file: pose_traces/rxr_train/000000_follower_pose_trace.npz
Analyze this dataset without downloading it
On Scigantic, RxR (Room-Across-Room): Multilingual Vision-and-Language Navigation Instructions in English, Hindi and Telugu with Pose Traces for Matterport3D (Google Research) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.