DROID: In-the-Wild Franka Teleoperation (LeRobot port): sample notebook
A worked notebook for DROID: In-the-Wild Franka Teleoperation (LeRobot port), with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Remi Cadene (Hugging Face / LeRobot), ported from the DROID consortium release
- License
- apache-2.0
- Size
- 100,000 files
- Format
- Parquet (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/cadene/droid)
- Access
- Free and public. No account with the data provider is needed.
The notebook
DROID: in-the-wild Franka teleoperation
92,233 teleoperated episodes on a Franka Panda arm, recorded in real kitchens, offices and workshops. Three cameras per episode (two exterior, one wrist), a per-frame state/action table, and a free-text task instruction for about half the corpus.
The archive is FUSE-mounted read-only, so nothing downloads. Four things to know before you start:
| Layout | LeRobot v2.1: one parquet + three mp4 per episode, same index |
| Video | AV1 at 320x180, 15 fps. Use PyAV, not cv2.VideoCapture |
| Task text | 45.86% of episodes have none. Filter first |
| Pose | observation.state is [x, y, z, roll, pitch, yaw, 0, gripper]: Euler angles, not a quaternion, despite the labels in meta/info.json |
| Action | a normalized velocity command in [-1, 1] plus a gripper command equal to 1 minus the gripper state |
import json, os
import av
import numpy as np
import pyarrow.parquet as pq
import matplotlib.pyplot as plt
import matplotlib_inline
matplotlib_inline.backend_inline.set_matplotlib_formats('jpeg')
plt.rcParams['figure.dpi'] = 72
root = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
print(sorted(os.listdir(root)))
Output:
['.gitattributes', 'README.md', 'data', 'meta', 'videos']
1. Filter before you read pixels
meta/episodes.jsonl is one small file with every episode's task text and frame count. Reading it first is the difference between a clean training list and a loader that silently feeds empty strings. Nearly half this corpus has no instruction, and 358 episodes are under a second long.
eps = [json.loads(l) for l in open(f'{root}/meta/episodes.jsonl')]
has_text = [e for e in eps if any(t.strip() for t in e['tasks'])]
usable = [e for e in has_text if e['length'] >= 15]
print(f'total {len(eps):>7,}')
print(f'with task {len(has_text):>7,} ({100*len(has_text)/len(eps):.1f}%)')
print(f'usable {len(usable):>7,} (task text AND >= 1s)')
print()
print('a few instructions:')
for e in usable[:5]:
print(f' ep {e["episode_index"]:>6} {e["length"]:>4} frames {e["tasks"][0]}')
Output:
total 92,233 with task 49,935 (54.1%) usable 49,914 (task text AND >= 1s) a few instructions: ep 6 182 frames Pick the bottle on the table and put it in the lower compartment of the tray ep 7 141 frames Push down the button on the toaster. ep 8 286 frames Place the lid on the jug ep 11 181 frames Slide the lid on the wooden toy box to the right. ep 12 230 frames Move the pillow
2. One episode = one parquet + three mp4
Everything for an episode shares its zero-padded index, and chunk = episode_index // 1000. Frame N of the video is row N of the parquet, no offset table needed.
(HiFi-UMI, the companion archive on this portal, is not laid out this way. It concatenates episodes into one multi-GB file and you seek by timestamp.)
ep = usable[0]
idx = ep['episode_index']
chunk = f'chunk-{idx // 1000:03d}'
parquet = f'{root}/data/{chunk}/episode_{idx:06d}.parquet'
video = f'{root}/videos/{chunk}/observation.images.wrist_image_left/episode_{idx:06d}.mp4'
tbl = pq.read_table(parquet)
state = tbl['observation.state'].combine_chunks().flatten().to_numpy().reshape(-1, 8)
# [x, y, z (m), roll, pitch, yaw (rad), 0.0, gripper 0 = open .. ~1 closed]
print(f'episode {idx}: {ep["tasks"][0]}')
print(f' {len(state)} rows, state {state.shape}; frame 0 roll/pitch/yaw (deg):', np.degrees(state[0, 3:6]).round(1))
Output:
episode 6: Pick the bottle on the table and put it in the lower compartment of the tray 182 rows, state (182, 8); frame 0 roll/pitch/yaw (deg): [-175.2 -7. -2.3]
3. Decode the video with PyAV
These files are AV1. cv2.VideoCapture opens them and returns zero frames without raising, because the bundled OpenCV ffmpeg has no software AV1 decoder. PyAV carries its own libdav1d.
frames = []
with av.open(video) as c:
codec = c.streams.video[0].codec_context.name
for f in c.decode(video=0):
frames.append(f.to_ndarray(format='rgb24'))
print(f'codec {codec}: decoded {len(frames)} frames of {frames[0].shape}')
print(f'parquet rows: {len(state)} -> {"1:1" if len(frames) == len(state) else "MISMATCH"}')
Output:
codec libdav1d: decoded 182 frames of (180, 320, 3) parquet rows: 182 -> 1:1
4. Wrist view against the gripper trace
The gripper column is state[:, 7]. Plotted under the frames it shows you the grasp: a flat low stretch while approaching, a step up where the fingers close.
picks = np.linspace(0, len(frames) - 1, 4).astype(int)
fig, axes = plt.subplots(1, 5, figsize=(15, 2.8))
for a, i in zip(axes[:4], picks):
a.imshow(frames[i]); a.set_title(f'frame {i}', fontsize=8); a.axis('off')
axes[4].plot(state[:, 7], lw=1.2)
axes[4].set_xlabel('frame'); axes[4].set_ylabel('gripper'); axes[4].set_title('grasp', fontsize=8)
plt.tight_layout(); plt.show()
5. Play the episode
The source video is AV1, and browser support for AV1 is uneven (Safari in particular), so play_episode re-encodes the episode to H.264 and saves it to ~/clips/. The player appears under this cell.
This cell stands alone: it works even if you skip everything above it. For any other episode, call play_episode(12345) (or pass cam= for an exterior camera) in a new cell.
Two things that do not work, both verified: Video(..., embed=True) (JupyterLab strips embedded video from output it did not produce in your browser), and opening the .mp4 directly or with *Open in New Browser Tab* (Jupyter serves /files/ with a sandbox security policy that breaks the browser's own player).
import os, subprocess
from IPython.display import HTML, display
MOUNT = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
def play_episode(i, cam='observation.images.wrist_image_left'):
"""Re-encode episode i of one camera to H.264 and show a player."""
video = f'{MOUNT}/videos/chunk-{i // 1000:03d}/{cam}/episode_{i:06d}.mp4'
clips = os.path.expanduser('~/clips')
os.makedirs(clips, exist_ok=True)
name = f"droid_ep{i}_{cam.split('.')[-1]}.mp4"
out = os.path.join(clips, name)
# AV1 -> H.264 for browser support; +faststart puts the mp4 index first for streaming.
subprocess.run(['ffmpeg', '-v', 'error', '-i', video, '-c:v', 'libx264', '-crf', '28',
'-preset', 'veryfast', '-pix_fmt', 'yuv420p', '-movflags', '+faststart',
'-an', out, '-y'], check=True)
print(f'{os.path.getsize(video) / 1e3:.0f} KB AV1 -> {os.path.getsize(out) / 1e3:.0f} KB H.264')
# SCIGANTIC_PREEXEC is set only when this starter is pre-executed for publishing;
# the clip made then does not exist in your workspace, so ship a prompt instead.
if os.environ.get('SCIGANTIC_PREEXEC') == '1':
print('Run this cell to play the episode.')
return
# Jupyter serves the home directory at <server prefix>/files/.
src = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/') + 'files/clips/' + name
display(HTML(f'<p><b>Episode {i}</b>, {cam.split(".")[-1]}</p>'
f'<video src="{src}" controls width="480"></video>'))
play_episode(6) # the episode used above; any index 0..92232 works
Output:
549 KB AV1 -> 120 KB H.264 Run this cell to play the episode.
Where to go next. observation.state is absolute Cartesian pose (Euler angles), so differencing it gives you velocity. action[0:6] is already a normalized velocity command and action[6] is 1 minus the gripper state, so compare actions with the change in state, never with the state itself. Episode length is heavily skewed (median 220 frames, max 4,498), so bucket by length rather than padding to max.
Ask the assistant on the right for anything else: it has the full schema card, including the measured quirks that are easy to trip over.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.