Context
Dataset used for the papers Latent Plans for Task Agnostic Offline Reinforcement Learning and Grounding Language with Visual Affordances over Unstructured Data.
Content
Episodes of a 7-DoF robotic arm and parallel jaw gripper performing a variety of undirected manipulation tasks, with around 1% of the data is annotated with natural language embeddings. Episodes were gathered using teleoperation via VR controller, by telling users to teleoperate the robot without a specific task in mind. Each state-action pair is encoded in a Numpy npz file, and consist of RGB-D images for both a static and a gripper camera, proprioceptive state, and future actions of the robot corresponding to that state.
Use case
Last-meter item handling in a fulfilment center
| Example observations | 3 × RGB images 1280×720 (front, side, wrist) 3 × Depth maps (optional) Joint positions (q_t ∈ ℝ⁷) & velocities (v_t ∈ ℝ⁷) Gripper width (0–0.08 m) & force sensor (N) Previous action a_t-1 |
| Example actions | Δx, Δy, Δz position steps (mm) Δroll, Δpitch, Δyaw (rad) Gripper open/close velocity |
| Example reward | Can be engineered offline from logged data: +10 if barcode successfully scanned (vision classifier) +5 if object placed in correct bin (bin-ID matcher) -1 per time-step (latency penalty) -2 if object dropped (depth mask lost below table) |