HiFi-UMI-2K: High-Fidelity Robot-Free Bimanual Manipulation: sample notebook
A worked notebook for HiFi-UMI-2K: High-Fidelity Robot-Free Bimanual Manipulation, with 6 code cells and the output each one printed. On Scigantic it opens in a hosted JupyterLab with the dataset already mounted, so nothing is downloaded first.
- Provider
- Simple World Lab (simpleai.tech)
- License
- cc-by-4.0
- Size
- 5,176 files
- Format
- Parquet, JSON (in sampled files)
- Stored at
- Hugging Face (https://huggingface.co/datasets/simple-world-lab/HiFi-UMI-2K)
- Access
- Free and public. No account with the data provider is needed.
The notebook
HiFi-UMI-2K: robot-free bimanual manipulation
2,000 hours of manipulation captured from human hands, not a robot. The operator wears a head-mounted stereo rig and glove-like grippers with fiducial markers; offline SLAM recovers both end-effector poses to about 3 mm, with all sensors hardware-triggered to under 40 microseconds.
| Layout | LeRobot v3.0: 398 shards; each shard's episodes (140 to 1,779) are concatenated into one ~7 GB mp4 per camera |
| Cameras | 6 at 640x512 h264, 25 fps (stereo head + 2 fisheye per hand) |
| Reading one episode | seek by timestamp. There is no episode_000900.mp4 anywhere |
| Pose | 20-D [right_10d, left_10d], each [xyz(3), rotation_6d(6), gripper_rad(1)] |
| Gripper | an opening angle in radians: ~0 = closed, ~0.6 = open, so a grasp is a drop |
That rotation is a 6D representation (the first two rows of the rotation matrix). DROID on this portal stores Euler angles instead, and its gripper runs the other way (0 = open). Applying one's unpacking to the other gives wrong rotations and inverted grasps with no error.
import os
import av
import numpy as np
import pyarrow.parquet as pq
import matplotlib.pyplot as plt
import matplotlib_inline
matplotlib_inline.backend_inline.set_matplotlib_formats('jpeg')
plt.rcParams['figure.dpi'] = 72
root = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
shards = sorted(d for d in os.listdir(root) if d.startswith('chunk-'))
shard = f'{root}/{shards[0]}/part-0000'
CAM = 'observation.images.head_main'
print(f'{len(shards)} shards; each is a self-contained LeRobot v3 dataset')
print('shard contents:', sorted(os.listdir(shard)))
Output:
398 shards; each is a self-contained LeRobot v3 dataset shard contents: ['_SUCCESS', 'data', 'meta', 'videos']
1. The episode index holds the seek key
Because episodes are concatenated, the only way to find one is meta/episodes/, which carries from_timestamp and to_timestamp per camera. It is a small parquet: read it, filter it, and only then touch the video.
eps = pq.read_table(f'{shard}/meta/episodes/chunk-000/file-000.parquet').to_pylist()
print(f'{len(eps)} episodes in this shard')
print()
for e in eps[:5]:
t0 = e[f'videos/{CAM}/from_timestamp']
t1 = e[f'videos/{CAM}/to_timestamp']
print(f' ep {e["episode_index"]:>4} {e["length"]:>4} frames t={t0:>8.2f}-{t1:<8.2f} {e["tasks"][0]}')
tasks = sorted({t for e in eps for t in e['tasks']})
print(f'\n{len(tasks)} distinct tasks in this shard')
Output:
1125 episodes in this shard ep 0 453 frames t= 0.00-18.12 Take the disposable food container out of the microwave. ep 1 451 frames t= 18.12-36.16 Take the disposable food container out of the microwave. ep 2 472 frames t= 36.16-55.04 Take the disposable food container out of the microwave. ep 3 304 frames t= 55.04-67.20 Replace and open the tissue box. ep 4 441 frames t= 67.20-84.84 Take the disposable food container out of the microwave. 38 distinct tasks in this shard
2. Seek, do not scan
The head camera's file for this shard is about 7 GB and holds five hours of video. Seeking to one episode moves a few megabytes, not seven gigabytes. Measured over HTTP against the live repo: 3.8 seconds and ~4.9 MB to pull a 12-second episode out of the middle.
The file size is not the read cost. That is the single most useful thing to know about this archive.
ep = eps[0]
t0 = ep[f'videos/{CAM}/from_timestamp']
video = f'{shard}/videos/{CAM}/chunk-000/file-000.mp4'
print(f'file is {os.path.getsize(video) / 1e9:.1f} GB; seeking to t={t0:.2f}s')
frames = []
with av.open(video) as c:
st = c.streams.video[0]
c.seek(int(t0 / st.time_base), stream=st)
for f in c.decode(video=0):
frames.append(f.to_ndarray(format='rgb24'))
if len(frames) == 8:
break
print(f'decoded {len(frames)} frames of {frames[0].shape} ({st.codec_context.name})')
print(f'task: {ep["tasks"][0]}')
Output:
file is 7.2 GB; seeking to t=0.00s decoded 8 frames of (512, 640, 3) (h264) task: Take the disposable food container out of the microwave.
3. The matching state rows
data/ holds every frame for the shard in one 80 MB parquet with a single row group, so any read pulls the whole file. There is no need to filter it: meta/episodes already gives this episode's row range, dataset_from_index to dataset_to_index.
Column 9 is the right gripper angle in radians, column 19 the left: about 0 when closed, rising as the hand opens. The two hands are genuinely independent here, which is the point of the capture rig.
tbl = pq.read_table(f'{shard}/data/chunk-000/file-000.parquet', columns=['observation.state'])
S = tbl['observation.state'].combine_chunks().flatten().to_numpy().reshape(-1, 20) # 38 MB
state = S[ep['dataset_from_index']:ep['dataset_to_index']]
print(f'state {state.shape} = frames x 20 (episode length says {ep["length"]})')
print(' [0:3] right xyz ', np.round(state[0, 0:3], 3))
print(' [3:9] right rot6d ', np.round(state[0, 3:9], 3))
print(' [9] right gripper ', round(float(state[0, 9]), 3), 'rad')
print(' [10:13] left xyz ', np.round(state[0, 10:13], 3))
Output:
state (453, 20) = frames x 20 (episode length says 453) [0:3] right xyz [-0.051 0.351 0.373] [3:9] right rot6d [-0.347 -0.844 -0.41 -0.938 0.3 0.177] [9] right gripper 0.027 rad [10:13] left xyz [-0.115 0.325 0.368]
fig, axes = plt.subplots(1, 4, figsize=(14, 2.9))
for a, fr in zip(axes[:3], frames[::3]):
a.imshow(fr); a.axis('off')
axes[3].plot(state[:, 9], lw=1.2, label='right')
axes[3].plot(state[:, 19], lw=1.2, label='left')
axes[3].set_xlabel('frame'); axes[3].set_ylabel('gripper (rad)'); axes[3].legend(fontsize=7)
plt.tight_layout(); plt.show()
4. Play one episode
The 7 GB file cannot be streamed to your browser, so play_episode cuts one episode out by timestamp, exactly as above, and saves it to ~/clips/. The player appears under this cell.
This cell stands alone: it works even if you skip everything above it. For any other episode, call play_episode(729) (or pass cam= for another camera) in a new cell.
Two things that do not work, both verified: Video(..., embed=True) (JupyterLab strips embedded video from output it did not produce in your browser), and opening the .mp4 directly or with *Open in New Browser Tab* (Jupyter serves /files/ with a sandbox security policy that breaks the browser's own player).
import html, os, subprocess
import pyarrow.parquet as pq
from IPython.display import HTML, display
MOUNT = os.environ.get('SCIGANTIC_MOUNT_PATH', '/mnt/archive')
def play_episode(i, shard='chunk-0000/part-0000', cam='observation.images.head_main'):
"""Cut episode i of a shard out of its concatenated video and show a player."""
base = f'{MOUNT}/{shard}'
ep = pq.read_table(f'{base}/meta/episodes/chunk-000/file-000.parquet').to_pylist()[i]
t0, t1 = ep[f'videos/{cam}/from_timestamp'], ep[f'videos/{cam}/to_timestamp']
video = (f"{base}/videos/{cam}/chunk-{ep[f'videos/{cam}/chunk_index']:03d}"
f"/file-{ep[f'videos/{cam}/file_index']:03d}.mp4")
clips = os.path.expanduser('~/clips')
os.makedirs(clips, exist_ok=True)
name = f"hifi_{shard.replace('/', '_')}_ep{i}_{cam.split('.')[-1]}.mp4"
out = os.path.join(clips, name)
# +faststart moves the mp4 index to the front so playback can start while streaming.
subprocess.run(['ffmpeg', '-v', 'error', '-ss', str(t0), '-t', str(t1 - t0), '-i', video,
'-vf', 'scale=480:-2', '-c:v', 'libx264', '-crf', '28', '-preset', 'veryfast',
'-movflags', '+faststart', '-an', out, '-y'], check=True)
print(f'{os.path.getsize(out) / 1e3:.0f} KB clip, {t1 - t0:.1f} s, cut from a {os.path.getsize(video) / 1e9:.1f} GB file')
# SCIGANTIC_PREEXEC is set only when this starter is pre-executed for publishing;
# the clip made then does not exist in your workspace, so ship a prompt instead.
if os.environ.get('SCIGANTIC_PREEXEC') == '1':
print('Run this cell to play the episode.')
return
# Jupyter serves the home directory at <server prefix>/files/.
src = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/') + 'files/clips/' + name
display(HTML(f'<p><b>Episode {i}</b>: {html.escape(ep["tasks"][0])}</p>'
f'<video src="{src}" controls width="480"></video>'))
play_episode(0)
Output:
304 KB clip, 18.1 s, cut from a 7.2 GB file Run this cell to play the episode.
Where to go next. Try the same episode from a hand camera (observation.images.right_hand_up) to see the grasp close up, or compare the stereo pair. A handful of rows have valid.frame == False and are black frames in the video; filter on it before training.
One trap worth repeating: episode_index restarts at 0 in every one of the 398 shards. Key episodes by (shard, episode_index) or you will collapse 398 distinct episode 0s into one.
The assistant on the right has the full schema card, including the gripper direction, the 6D rotation layout, how to search tasks across all 398 shards, and what fits in the notebook's memory.
Run it
Open this notebook to run it, see the dataset page for the full description, or browse the full catalog.