DROID: In-the-Wild Franka Teleoperation (LeRobot port): sample notebook

A worked notebook for DROID: In-the-Wild Franka Teleoperation (LeRobot port), with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.

Provider
Remi Cadene (Hugging Face / LeRobot), ported from the DROID consortium release
License
apache-2.0
Size
100,000 files
Format
Parquet (in sampled files)
Stored at
Hugging Face (https://huggingface.co/datasets/cadene/droid)
Access
Free and public. No account with the data provider is needed.

The notebook

DROID: in-the-wild Franka teleoperation

92,233 teleoperated episodes on a Franka Panda arm, recorded in real kitchens, offices and workshops. Three cameras per episode (two exterior, one wrist), a per-frame state/action table, and a free-text task instruction for about half the corpus.

The archive is FUSE-mounted read-only, so nothing downloads. Four things to know before you start:

LayoutLeRobot v2.1: one parquet + three mp4 per episode, same index
VideoAV1 at 320x180, 15 fps. Use PyAV, not cv2.VideoCapture
Task text45.86% of episodes have none. Filter first
Poseobservation.state is [x, y, z, roll, pitch, yaw, 0, gripper]: Euler angles, not a quaternion, despite the labels in meta/info.json
Actiona normalized velocity command in [-1, 1] plus a gripper command equal to 1 minus the gripper state
import json, os
import av
import numpy as np
import pyarrow.parquet as pq
import matplotlib.pyplot as plt
import matplotlib_inline

matplotlib_inline.backend_inline.set_matplotlib_formats('jpeg')
plt.rcParams['figure.dpi'] = 72

root = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
print(sorted(os.listdir(root)))

Output:

['.gitattributes', 'README.md', 'data', 'meta', 'videos']

1. Filter before you read pixels

meta/episodes.jsonl is one small file with every episode's task text and frame count. Reading it first is the difference between a clean training list and a loader that silently feeds empty strings. Nearly half this corpus has no instruction, and 358 episodes are under a second long.

eps = [json.loads(l) for l in open(f'{root}/meta/episodes.jsonl')]

has_text = [e for e in eps if any(t.strip() for t in e['tasks'])]
usable  = [e for e in has_text if e['length'] >= 15]

print(f'total      {len(eps):>7,}')
print(f'with task  {len(has_text):>7,}  ({100*len(has_text)/len(eps):.1f}%)')
print(f'usable     {len(usable):>7,}  (task text AND >= 1s)')
print()
print('a few instructions:')
for e in usable[:5]:
    print(f'  ep {e["episode_index"]:>6}  {e["length"]:>4} frames  {e["tasks"][0]}')

Output:

total       92,233
with task   49,935  (54.1%)
usable      49,914  (task text AND >= 1s)

a few instructions:
  ep      6   182 frames  Pick the bottle on the table and put it in the lower compartment of the tray
  ep      7   141 frames  Push down the button on the toaster.
  ep      8   286 frames  Place the lid on the jug
  ep     11   181 frames  Slide the lid on the wooden toy box to the right.
  ep     12   230 frames  Move the pillow

2. One episode = one parquet + three mp4

Everything for an episode shares its zero-padded index, and chunk = episode_index // 1000. Frame N of the video is row N of the parquet, no offset table needed.

(HiFi-UMI, the companion archive on this portal, is not laid out this way. It concatenates episodes into one multi-GB file and you seek by timestamp.)

ep    = usable[0]
idx   = ep['episode_index']
chunk = f'chunk-{idx // 1000:03d}'

parquet = f'{root}/data/{chunk}/episode_{idx:06d}.parquet'
video   = f'{root}/videos/{chunk}/observation.images.wrist_image_left/episode_{idx:06d}.mp4'

tbl   = pq.read_table(parquet)
state = tbl['observation.state'].combine_chunks().flatten().to_numpy().reshape(-1, 8)
# [x, y, z (m), roll, pitch, yaw (rad), 0.0, gripper 0 = open .. ~1 closed]

print(f'episode {idx}: {ep["tasks"][0]}')
print(f'  {len(state)} rows, state {state.shape}; frame 0 roll/pitch/yaw (deg):', np.degrees(state[0, 3:6]).round(1))

Output:

episode 6: Pick the bottle on the table and put it in the lower compartment of the tray
  182 rows, state (182, 8); frame 0 roll/pitch/yaw (deg): [-175.2   -7.    -2.3]

3. Decode the video with PyAV

These files are AV1. cv2.VideoCapture opens them and returns zero frames without raising, because the bundled OpenCV ffmpeg has no software AV1 decoder. PyAV carries its own libdav1d.

frames = []
with av.open(video) as c:
    codec = c.streams.video[0].codec_context.name
    for f in c.decode(video=0):
        frames.append(f.to_ndarray(format='rgb24'))

print(f'codec {codec}: decoded {len(frames)} frames of {frames[0].shape}')
print(f'parquet rows: {len(state)}  ->  {"1:1" if len(frames) == len(state) else "MISMATCH"}')

Output:

codec libdav1d: decoded 182 frames of (180, 320, 3)
parquet rows: 182  ->  1:1

4. Wrist view against the gripper trace

The gripper column is state[:, 7]. Plotted under the frames it shows you the grasp: a flat low stretch while approaching, a step up where the fingers close.

picks = np.linspace(0, len(frames) - 1, 4).astype(int)
fig, axes = plt.subplots(1, 5, figsize=(15, 2.8))
for a, i in zip(axes[:4], picks):
    a.imshow(frames[i]); a.set_title(f'frame {i}', fontsize=8); a.axis('off')
axes[4].plot(state[:, 7], lw=1.2)
axes[4].set_xlabel('frame'); axes[4].set_ylabel('gripper'); axes[4].set_title('grasp', fontsize=8)
plt.tight_layout(); plt.show()

5. Play the episode

The source video is AV1, and browser support for AV1 is uneven (Safari in particular), so play_episode re-encodes the episode to H.264 and saves it to ~/clips/. The player appears under this cell.

This cell stands alone: it works even if you skip everything above it. For any other episode, call play_episode(12345) (or pass cam= for an exterior camera) in a new cell.

Two things that do not work, both verified: Video(..., embed=True) (JupyterLab strips embedded video from output it did not produce in your browser), and opening the .mp4 directly or with *Open in New Browser Tab* (Jupyter serves /files/ with a sandbox security policy that breaks the browser's own player).

import os, subprocess
from IPython.display import HTML, display

MOUNT = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')

def play_episode(i, cam='observation.images.wrist_image_left'):
    """Re-encode episode i of one camera to H.264 and show a player."""
    video = f'{MOUNT}/videos/chunk-{i // 1000:03d}/{cam}/episode_{i:06d}.mp4'

    clips = os.path.expanduser('~/clips')
    os.makedirs(clips, exist_ok=True)
    name = f"droid_ep{i}_{cam.split('.')[-1]}.mp4"
    out = os.path.join(clips, name)
    # AV1 -> H.264 for browser support; +faststart puts the mp4 index first for streaming.
    subprocess.run(['ffmpeg', '-v', 'error', '-i', video, '-c:v', 'libx264', '-crf', '28',
                    '-preset', 'veryfast', '-pix_fmt', 'yuv420p', '-movflags', '+faststart',
                    '-an', out, '-y'], check=True)
    print(f'{os.path.getsize(video) / 1e3:.0f} KB AV1 -> {os.path.getsize(out) / 1e3:.0f} KB H.264')

    # SCIGANTIC_PREEXEC is set only when this starter is pre-executed for publishing;
    # the clip made then does not exist in your workspace, so ship a prompt instead.
    if os.environ.get('SCIGANTIC_PREEXEC') == '1':
        print('Run this cell to play the episode.')
        return
    # Jupyter serves the home directory at <server prefix>/files/.
    src = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/') + 'files/clips/' + name
    display(HTML(f'<p><b>Episode {i}</b>, {cam.split(".")[-1]}</p>'
                 f'<video src="{src}" controls width="480"></video>'))

play_episode(6)   # the episode used above; any index 0..92232 works

Output:

549 KB AV1 -> 120 KB H.264
Run this cell to play the episode.

Where to go next. observation.state is absolute Cartesian pose (Euler angles), so differencing it gives you velocity. action[0:6] is already a normalized velocity command and action[6] is 1 minus the gripper state, so compare actions with the change in state, never with the state itself. Episode length is heavily skewed (median 220 frames, max 4,498), so bucket by length rather than padding to max.

Ask the assistant on the right for anything else: it has the full schema card, including the measured quirks that are easy to trip over.

Run it

Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.