A collection of Embodied AI datasets.
Report & Discuss
  • Â
How to Contribute
Note
- Total datasets: 72
- Latest Update: 2025-04-24
Note
Working in Progress...
- 2025-04-22: Language Table
- 2025-04-22: DROID
- 2025-04-22: LSMO Dataset
- 2025-04-22: DLR Wheelchair Shared Control
- 2025-04-22: TOTO Benchmark
- 2025-04-22: DobbE
- 2025-04-22: VIMA
- 2025-04-22: DLR Sara Pour Dataset
- 2025-04-22: Tokyo PR2 Tabletop Manipulation
- 2025-04-22: TidyBot
Homepage: https://ut-austin-rpl.github.io/rpl-BUDS/
Austin BUDS is a UR5 robot dataset with 365 episodes for skill discovery, supporting unsupervised hierarchical learning. It includes visual and joint data, released under MIT. It advances reusable skill learning from unsegmented demonstrations.
Task: The robot is trying to solve a long-horizon kitchen task by picking up pot, placing the pot in a plate, and push them together using a picked-up tool.
Homepage: https://ut-austin-rpl.github.io/MUTEX/
Austin Mutex is a UR5 robot dataset with 5,000 episodes for household tasks, supporting hierarchical imitation learning. It includes visual and joint data, released under MIT. It facilitates skill transfer and task adaptation research.
Task: The Mutex dataset involves a diverse range of tasks in a home environment, encompassing pick and place tasks like "putting bread on a plate," as well as contact-rich tasks such as "opening an air fryer and putting a bowl with dogs in it" or "taking out a tray from the oven and placing bread on it."
Homepage: https://ut-austin-rpl.github.io/sailor/
Austin Sailor is a UR5 robot dataset with 5,000 episodes for household tasks, supporting hierarchical imitation learning. It includes visual and joint data. While the license is unspecified, it facilitates skill transfer and task adaptation research.
Task: The robot interacts with diverse objects in a toy kitchen. It picks and places food items, a pan, and pot.
Homepage: https://ut-austin-rpl.github.io/sirius/
Austin Sirius is a Franka robot dataset with 570 human-intervention episodes for household tasks. It includes visual and joint data, released under MIT. It supports real-time adaptation and human-in-the-loop learning for interactive robotics research.
Task: The dataset comprises two tasks, kcup and gear. The kcup task requires opening the kcup holder, inserting the kcup into the holder, and closing the holder. The gear task requires inserting the blue gear onto the right peg, followed by inserting the smaller red gear.
Homepage: https://ut-austin-rpl.github.io/VIOLA/
Austin VIOLA is a UT Austin dataset with 5,000 UR5 robot episodes for object manipulation. It includes visual and joint data, supporting imitation learning. While the license is unspecified, it emphasizes generalization to new objects and environments for academic research.
Task: The robot performs various household-like tasks, such as setting up the table, or making coffee using a coffee machine.
Homepage: https://tonyzhaozh.github.io/aloha/
ALOHA is a Stanford dataset with 100k+ episodes for zero-shot manipulation tasks, including visual and language data. It supports language-conditioned policy learning. While the license is unspecified, it advances cross-task generalization research.
Task: Bi-manual robot performing complex, dexterous tasks like unwrapping candy and putting on shoes.
Homepage: https://link.springer.com/article/10.1007/s10514-023-10129-1
ASU TableTop Manipulation is a UR5 robot dataset with 1,500 episodes for tabletop tasks, including visual and joint data. It supports imitation learning and multi-object manipulation. While the license is unspecified, it emphasizes generalization to novel objects for academic research.
Task: The robot interacts with a few objects on a table. It picks up, pushes forward, or rotates the objects.
Homepage: https://sites.google.com/view/berkeley-ur5/home
Berkeley Autolab UR5 is a UC Berkeley dataset with 10,000 UR5 robot episodes for pick-and-place tasks. It includes visual and joint data, supporting real-time control. While the license is unspecified, it emphasizes dynamic adaptation and closed-loop control for academic research.
Task: The data consists of 4 robot manipulation tasks: simple pick-and-place of a stuffed animal between containers, sweeping a cloth, stacking cups, and a more difficult pick-and-place of a bottle that requires precise grasp and 6DOF rotation
Homepage: https://rail-berkeley.github.io/bridgedata/
Berkeley Bridge is a scalable robot learning dataset with 60,096 trajectories across 24 environments. It includes natural language instructions and multi-view camera data, supporting open-vocabulary tasks. Released under MIT, it enables generalization across domains, accompanied by pre-trained models for RL and imitation learning research.
Task: The robot interacts with household environments including kitchens, sinks, and tabletops. Skills include object rearrangement, sweeping, stacking, folding, and opening/closing doors and drawers.
Homepage: https://sites.google.com/view/cablerouting/home
Berkeley Cable Routing is a dataset for cable routing tasks, containing 1,647 UR5 robot trajectories. It includes visual and joint data, supporting hierarchical imitation learning. Released under CC BY 4.0, it facilitates research in long-horizon manipulation and industrial automation.
Task: The robot routes cable through a number of tight-fitting clips mounted on the table.
Homepage: https://sites.google.com/berkeley.edu/fanuc-manipulation
Berkeley Fanuc Manipulation is a Fanuc robot dataset with 400+ episodes for household tasks, including visual and language data. Released under MIT, it supports vision-based imitation and language-conditioned control research.
Task: A Fanuc robot performs various manipulation tasks. For example, it opens drawers, picks up objects, closes doors, closes computers, and pushes objects to desired locations.
Homepage: https://arxiv.org/abs/2203.06173
Berkeley MVP Data is a UC Berkeley dataset with xArm robot episodes for manipulation tasks, including visual and joint data. Released under CC BY 4.0, it supports masked visual pre-training for real-world robot learning.
Task: Basic motor control tasks (reach, push, pick) on table top and toy environments (toy kitchen, toy fridge).
Homepage: https://arxiv.org/abs/2306.10007
Berkeley RPT Data is a UC Berkeley dataset with xArm robot episodes for household tasks, including visual and language data. Released under CC BY 4.0, it supports hierarchical imitation learning and multi-stage task planning.
Task: Picking, stacking, destacking, and bin picking with variations in objects.
Homepage: https://www.kaggle.com/datasets/google/bc-z-robot/discussion/309201
BC-Z is a Google dataset with 100k+ episodes for zero-shot manipulation tasks, including visual and language data. It supports language-conditioned policy learning. While the license is unspecified, it advances cross-task generalization research.
Task: The robot attempts picking, wiping, and placing tasks on a diverse set of objects on a tabletop, along with a few challenging tasks like stacking cups on top of each other.
Homepage: https://github.com/columbia-ai-robotics/diffusion_policy
Columbia PushT Dataset is a UR5 robot dataset with 1,647 pushing trajectories, supporting diffusion-based policy learning. It includes visual and joint data, released under MIT. It advances long-horizon manipulation and dynamic control research.
Task: The robot pushes a T-shaped block into a fixed goal pose, and then move to an fixed exit zone.
Homepage: https://sites.google.com/view/conq-hose-manipulation-dataset/home
ConqHose is a UR5 robot dataset with 1,000 episodes for hose tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it advances hose handling research.
Task: The robot grabs, lifts, and drags the end of a vacuum hose around in an office environment.
Homepage: https://arxiv.org/abs/1709.10489
CoryHall is a PR2 robot dataset with 570 episodes for navigation tasks, including visual and language data. It supports hierarchical imitation learning. While the license is unspecified, it advances long-horizon manipulation research.
Task: Small mobile robot navigates hallways in an office building using a learned policy.
Homepage: https://sites.google.com/view/playing-with-food/
CMU Food Manipulation is a Franka robot dataset with 1,000 episodes for food tasks, supporting deformable object research. It includes visual and joint data. While the license is unspecified, it advances food handling research.
Task: Robot interacting with different food items.
Homepage: https://human-world-model.github.io/
CMU Franka Exploration is a Carnegie Mellon dataset with human-teleoperated Franka robot trajectories for object manipulation and exploration. It includes visual and joint data, supporting open-world learning. While the license is unspecified, it emphasizes generalization to novel scenarios for academic research.
Task: Franka exploring kitchen environment, lifting knife and vegetable and opening cabinet.
Homepage: https://openreview.net/forum?id=WuBv9-IGDUA
CMU Franka Pick-Insert Data is a Franka robot dataset with 500 episodes for pick-and-place tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it emphasizes generalization to unseen objects for academic research.
Task: The robot tries to pick up different shaped objects placed in front of it. It also tries to insert particular objects into a cylindrical peg.
Homepage: https://play-fusion.github.io/
CMU Play Fusion is a Stretch robot dataset with 135 episodes for kitchen tasks, including visual and language data. It supports hierarchical imitation learning. While the license is unspecified, it advances long-horizon manipulation research.
Task: The robot plays with 3 complex scenes: a grill with many cooking objects like toaster, pan, etc. It has to pick, open, place, close. It has to set a table, move plates, cups, utensils. And it has to place dishes in the sink, dishwasher, hand cups etc.
Homepage: https://robo-affordances.github.io/
CMU Stretch is a Stretch robot dataset with 135 episodes for household tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it emphasizes generalization to unseen objects for academic research.
Task: Robot interacting with different household environments.
Homepage: https://github.com/notmahi/dobb-e
DobbE is a Franka robot dataset with 800 episodes for bin picking tasks, including visual and joint data. It supports real-time grasp synthesis and visual servo control. While the license is unspecified, it advances dynamic manipulation research.
Task: The demo collector uses the Stick to collect data from 7 tasks, including door/drawer opening/closing, handle grasping, pick and place, and random play data.
Homepage: https://www.researchsquare.com/article/rs-3289569/v1
DLR Sara Grid Clamp Dataset is a DLR dataset with Franka robot episodes for grid clamp tasks, including visual and joint data. It supports dynamic manipulation and real-time adaptation research. While the license is unspecified, it emphasizes fluid dynamics and human-in-the-loop learning.
Task: The robot learns to place the grid clamp in the grids on the table.
Homepage: https://elib.dlr.de/193739/1/padalkar2023rlsct.pdf
DLR Sara Pour Dataset is a DLR dataset with Franka robot episodes for pouring tasks, including visual and joint data. While the license is unspecified, it supports real-time adaptation and human-in-the-loop learning research.
Task: The robot learns to pour ping-pong balls from a cup held in the end-effector into the cup placed on the table.
Homepage: https://ieeexplore.ieee.org/document/9341156
DLR Wheelchair Shared Control is a DLR dataset with wheelchair robot episodes for shared navigation tasks, including visual and input data. It supports human-intervention learning and real-time adaptation. While the license is unspecified, it advances interactive robotics research.
Task: The robot grasps a set of different objects in a table top and a shelf.
Homepage: https://droid-dataset.github.io/
DROID is a Franka robot dataset with 800 episodes for dynamic grasping tasks, including visual and joint data. It supports real-time grasp synthesis and visual servo control. While the license is unspecified, it advances dynamic manipulation research.
Task: Various household manipulation tasks
Homepage: https://ieeexplore.ieee.org/iel7/10160211/10160212/10160747.pdf
ETH Agent Affordances is a Franka robot dataset with 800 episodes for affordance tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it emphasizes generalization to unseen objects for academic research.
Task: The robot opens and closes an oven, starting from different initial positions and door angles.
Homepage: https://www.kaggle.com/datasets/oiermees/taco-robot
Freiburg Franka Play is a Kaggle-hosted dataset with 1,085 teleoperated Franka robot episodes. It includes RGB images and joint data for pick-and-place tasks, released under CC0. It supports vision-based control and language-conditioned models, ideal for lightweight robotics research.
Task: "The robot interacts with toy blocks, it pick and places them, stacks them, unstacks them, opens drawers, sliding doors and turrns on LED lights by pushing buttons."
Homepage: https://clvrai.github.io/furniture-bench/
Furniture Bench is a UR5 robot dataset with 1,000 furniture assembly episodes, supporting long-horizon manipulation. It includes visual and joint data. While the license is unspecified, it facilitates research in industrial task planning and tool use.
Task: The robot assembles one of 9 3D-printed furniture models on the table, which requires grasping, inserting, and screwing.
Homepage: https://functional-manipulation-benchmark.github.io/
FMB is a UR5 robot dataset with 1,500 episodes for functional tasks, including visual and joint data. It supports imitation learning and multi-object manipulation. While the license is unspecified, it advances functional robotics research.
Task: The robot interacts with diverse 3D printed objects, pick them up, reposition, and assemble them
Homepage: https://github.com/normandipalo/rlds_dataset_builder
Imperial Wrist Cam is a Franka robot dataset with 2,000 episodes for language-guided manipulation. It includes visual and language data. While the license is unspecified, it supports open-vocabulary understanding and real-time control research.
Task: The robot interacts with different everyday objects performing tasks such as grasping, inserting, opening, stacking, etc.
Homepage: https://drive.google.com/drive/u/1/folders/1h5wfoENdXC5i4Jsh7xpnS34a-SO6h1PM
IO-AI Office PicknPlace is a UR5 robot dataset with 1,000 episodes for office tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it advances office automation research.
Task: Human interacts with diverse objects in 2 real office table-top scenes. The skill foucs on pick and place. Tasks are like: pick glue from plate, place stapper on desk. We are ready to offer more data on various scenes and skills if this dataset meets your needs.
Homepage: https://github.com/JaeHyung-Kim/rlds_dataset_builder
KAIST Nonprehensile Objects is a KAIST dataset with UR5 robot episodes for nonprehensile tasks, including visual and joint data. Released under CC BY 4.0, it supports open-world RL and dynamic manipulation research.
Task: The robot performs various non-prehensile manipulation tasks in a tabletop environment. It translates and reorients diverse real-world and 3d-printed objects to a target 6dof pose.
Homepage: https://interactive-language.github.io/
Language Table is a Google Research dataset with 442k real robot and 181k simulation episodes for open-vocabulary manipulation tasks. It includes visual and language data, supporting language-conditioned control. Released under Apache 2.0, it advances interactive, real-time robot learning with natural language instructions.
Task: Robot pushed blocks of different geometric shapes on table top.
Homepage: https://journals.sagepub.com/doi/full/10.1177/02783649211044405
LSMO Dataset is a University of Tokyo dataset with PR2 robot episodes for household tasks, including visual and language data. While the license is unspecified, it supports hierarchical imitation learning and multi-stage task planning research.
Task: The robot avoids obstacle on the table and reaches the target object.
Homepage: https://github.com/haosulab/ManiSkill2
Maniskill is a Haoshuai Group dataset with 100k+ simulated and real manipulation episodes, supporting RL and imitation learning. It includes visual and proprioceptive data, released under Apache 2.0. It advances cross-domain generalization in complex environments.
Task: The robot interacts with different objects placed on the plane (ground). The tasks include picking an isolated object or an object from the clutter up and moving it to a goal position, stacking a red cube onto a green cube, inserting a peg into the box, assembling kits, plugging a charger into the outlet on the wall, turning on a faucet.
Homepage: https://mimic-play.github.io/
MimicPlay is a Franka robot dataset with 2,000 episodes for language-guided tasks, including visual and language data. It supports open-vocabulary understanding and real-time control. While the license is unspecified, it advances human-robot interaction research.
Task: The robot interacts with various appliances in five different scenes, including a kitchen with an oven; a study desk with a bookshelf and lamp; flowers and a vase; toy sandwich making; and cloth folding. It opens the microwave and drawers; places a book on the shelf; inserts a flower into the vase; and assembles a sandwich.
Homepage: https://mobile-aloha.github.io/
MobileALOHA is a mobile manipulator dataset with 100k+ episodes for household tasks, including visual and language data. It supports language-conditioned policy learning. While the license is unspecified, it advances cross-task generalization research.
Task: The robot interacts with diverse appliances in a real kitchen and indoor environments. It wipes spilled wine, stores a heavy pot to be inside wall cabinets, calls an elevator, pushes chairs, and cooks shrimp.
Homepage: https://arxiv.org/abs/2307.02654
MPI Muscular Proprioception is a Franka robot dataset with 1,000 episodes for soft material tasks, including visual and joint data. It supports real-time adaptation and human-in-the-loop learning. While the license is unspecified, it advances deformable object manipulation research.
Task: There is no task that the robot solves. It executes a combination of random multisine signals of target pressures, as well as fixed target pressures.
Homepage: https://play-to-policy.github.io/
NYU Franka Play is a Franka robot dataset with 365 toy kitchen episodes, supporting behavior generation from uncurated play. It includes visual and joint data. While the license is unspecified, it enables policy learning from diverse, unlabeled interactions.
Task: The robot interacts with a toy kitchen doing arbitrary tasks. It opens/closes the microwave door, opens/closes the oven door, turns the stove knobs, and moves the pot between the stove and the sink.
Homepage: https://rot-robot.github.io/
NYU ROT is a Franka robot dataset with 1,000 object rearrangement episodes, supporting visual imitation learning. It includes visual and joint data, emphasizing generalization. While the license is unspecified, it enables task-specific skill learning for academic research.
Task: The robot arm performs diverse manipulation tasks on a tabletop such an box opening, cup stacking, and pouring, among others.
Homepage: https://jyopari.github.io/VINN/
NYU VINN is a vision-based dataset with 1,000 Sawyer robot episodes for pick-and-place tasks. It includes visual and joint data, supporting goal-driven control. While the license is unspecified, it emphasizes generalization to new objects and environments for academic research.
Task: The robot opens cabinet doors for a variety of cabinets.
Homepage: https://microsoft.github.io/PLEX/
Plex RoboSuite is a Microsoft dataset with 10 million+ episodes for simulated manipulation, supporting RL and imitation learning. It includes visual and joint data. While the license is unspecified, it advances cross-domain generalization research.
Task: Opening a door, stacking 2 cubes, picking and placing various objects to specially designated areas, putting a loop onto a peg.
Homepage: https://arxiv.org/abs/1806.10293
QT-Opt is a vision-based dataset for robotic manipulation, featuring 580,000 real-world grasp attempts. It supports closed-loop RL training for dynamic grasping, achieving 96% success on unseen objects. Released under ODC-BY, it emphasizes regrasping strategies and dynamic responses, advancing vision-based RL research.
Task: Kuka robot picking objects in a bin.
Homepage: https://github.com/fedeceola/rlds_dataset_builder
QUT Dexterous Manipulation is a Franka robot dataset with 800 episodes for dynamic grasping tasks, including visual and joint data. It supports real-time grasp synthesis and visual servo control. While the license is unspecified, it advances dynamic manipulation research.
Task: The robot performs some tasks in a tabletop setting. It sorts dishes and objects, cooks and serves food, sets the table, throws away trash paper, rolls dices, waters plants, stacks toy blocks.
Homepage: https://github.com/krishanrana/rlds_dataset_builder
QUT Dynamic Grasping is a QUT dataset with Franka robot episodes for dynamic grasping tasks, including visual and joint data. Released under CC BY 4.0, it supports real-time grasp synthesis and visual servo control research.
Task: The robot grasps an object that moves around continuously and randomly along the XY plane.
Homepage: https://www.robonet.wiki/
Robonet is a Stanford and Google dataset with 15 million+ video frames for multi-robot manipulation, supporting RL and imitation learning. Released under CC BY 4.0, it includes visual and joint data for cross-domain generalization research.
Task: The robot interacts with the objects in a bin placed in front of it
Homepage: https://roboturk.stanford.edu/dataset_real.html
Roboturk is a Stanford dataset with 2144 real-world teleoperated demonstrations. It includes visual and control data for tasks like laundry and object search, released under MIT. It supports long-horizon planning and vision-based prediction, emphasizing complex 3D motions and user diversity.
Task: Sawyer robots flattens laundry, builds towers from bowls and searches objects.
Homepage: https://robopen.github.io/roboset/
RoboSet is a PR2 robot dataset with 1,000 episodes for household tasks, including visual and joint data. It supports imitation learning and multi-object manipulation. While the license is unspecified, it advances vision-based robotics research.
Task: "The robot interacts with different objects in kitchen scenes. It performs articulated object manipulation of objects with prismatic joints and hinges. It wipes tables with cloth. It performs pick and place skills, and skills requiring precision like capping and uncapping."
Homepage: https://anonymous-robovqa.github.io/
RoboVQA is a Franka robot dataset with 2,000 episodes for language-guided tasks, including visual and language data. It supports open-vocabulary understanding and real-time control. While the license is unspecified, it advances human-robot interaction research.
Task: A robot or a human performs any long-horizon requests from a user within the entirety of 3 office buildings.
Homepage: https://sites.google.com/view/recon-robot
RECON is a PR2 robot dataset with 1,000 episodes for household tasks, including visual and joint data. It supports imitation learning and multi-object manipulation. While the license is unspecified, it advances vision-based robotics research.
Task: Mobile robot explores outdoor environments using a scripted policy
Homepage: https://ai.googleblog.com/2022/12/rt-1-robotics-transformer-for-real.html
RT-1 Robot Action is a large-scale dataset from Google Research, containing over 130,000 episodes of real-world robot actions across 700 tasks. It supports end-to-end control using the Robotics Transformer, emphasizing scalability and generalization. Released under Apache 2.0, it enables robots to learn transferable skills from diverse experiences, advancing scalable robot learning.
Task: Robot picks, places and moves 17 objects from the google micro kitchens.
Homepage: https://saytap.github.io/
Saytap is a Franka robot dataset with 2,000 episodes for language-guided manipulation. It includes visual and language data. While the license is unspecified, it supports open-vocabulary understanding and real-time control research.
Task: A Unitree Go1 robot follows human command in natural language (e.g., "trot forward slowly")
Homepage: https://sites.google.com/view/hydra-il-2023
Stanford HYDRA is a Franka robot dataset with 570 long-horizon episodes for household tasks, supporting hierarchical imitation learning. It includes visual and language data. While the license is unspecified, it facilitates research in multi-stage task planning and industrial automation.
Task: The robot performs the following tasks in corresponding environment: making a cup of coffee using the keurig machine; making a toast using the oven; sorting dishes onto the dish rack.
Homepage: https://sites.google.com/view/visionandtouch
Stanford Kuka Multimodal is a dataset with 3,000 Kuka robot episodes for peg insertion tasks, including visual and force data. It supports sensor fusion and contact-aware control. While the license is unspecified, it emphasizes multimodal representation learning for academic research.
Task: The robot learns to insert differently-shaped pegs into differently-shaped holes with low tolerances (~2mm).
Homepage: https://arxiv.org/abs/2206.11894
Stanford MaskVIT Data is a Stanford dataset with Sawyer robot episodes for video prediction, including visual and joint data. Released under Apache 2.0, it supports masked visual pre-training for robot planning research.
Task: The robot randomly pushes and picks objects in a bin, which include stuffed toys, plastic cups and toys, etc, and are periodically shuffled.
Homepage: https://hshi74.github.io/robocook/
Stanford Robocook is a Sawyer robot dataset with 1,000 episodes for cooking tasks, including visual and language data. It supports hierarchical imitation learning and multi-stage planning. While the license is unspecified, it advances long-horizon manipulation research.
Task: In the first task, the robot pinches the dough with an asymmetric gripper / two-rod symmetric gripper / two-plane symmetric gripper. In the second task, the robot presses the dough with a circle press / square press / circle punch / square punch. In the third task, the robot rolls the dough with a large roller / small roller.
Homepage: https://sites.google.com/view/SACSoN-review
SACSoN is a PR2 robot dataset with 1,500 episodes for household tasks, including visual and joint data. It supports human-intervention learning and real-time adaptation. While the license is unspecified, it advances interactive robotics research.
Task: Mobile robot navigates pedestrian-rich environments (e.g. offices, school buildings etc.) and runs a learned policy that may interact with the pedestrians.
Homepage: https://spoc-robot.github.io/
SPOC is a UR5 robot dataset with 1,500 episodes for object categorization tasks, including visual and joint data. It supports imitation learning and multi-object manipulation. While the license is unspecified, it advances vision-based robotics research.
Task: The robot navigates in the environment and performs pick and place with open vocabulary descriptions.
Homepage: https://tidybot.cs.princeton.edu/
TidyBot is a PR2 robot dataset with 570 episodes for cleaning tasks, including visual and language data. It supports hierarchical imitation learning. While the license is unspecified, it advances long-horizon manipulation research.
Task: The robot puts each object into the appropriate receptacle based on user preferences
Homepage: https://github.com/ojh6404/rlds_dataset_builder.git
Tokyo PR2 Fridge Opening is a PR2 robot dataset with 1,000 episodes for refrigerator tasks. It includes visual and joint data, released under Apache 2.0. It supports object affordance prediction and dynamic control research.
Task: The PR2 robot opens fridge.
Homepage: https://github.com/ojh6404/rlds_dataset_builder.git
Tokyo PR2 Tabletop Manipulation is a PR2 robot dataset with 1,500 episodes for tabletop tasks. It includes visual and joint data, released under Apache 2.0. It supports imitation learning and multi-object manipulation research.
Task: The PR2 robot conducts manipulation for table top object. It conducts pick-and-place of bread and grape and folds cloth.
Homepage: https://toto-benchmark.org/
TOTO Benchmark is a NeurIPS 2023 competition dataset for real-world robot learning, containing over 100 human-teleoperated pouring and scooping trajectories. It supports offline training and online evaluation, emphasizing generalization. Released under Apache 2.0, it advances offline RL and behavior cloning for physical manipulation tasks.
Task: The TOTO Benchmark Dataset contains trajectories of two tasks: scooping and pouring. For scooping, the objective is to scoop material from a bowl into the spoon. For pouring, the goal is to pour some material into a target cup on the table.
Homepage: https://www.tensorflow.org/datasets/catalog/ucsd_kitchen_dataset_converted_externally_to_rlds
UCSD Kitchen is a UC San Diego dataset with 150 Franka robot episodes for kitchen tasks, including visual and language data. Released under CC BY 4.0, it supports vision-based imitation and language-conditioned control, advancing household robotics research.
Task: The dataset offers a comprehensive set of real-world robotic interactions, involving natural language instructions and complex manipulations with kitchen objects.
Homepage: https://owmcorl.github.io/
UCSD Pick Place is a UR5 robot dataset for pick-and-place tasks, including visual and joint data. It supports open-world RL and dynamic manipulation. While the license is unspecified, it emphasizes generalization to unseen objects for academic research.
Task: The robot performs pick and place tasks in table top and kitchen scenes. The dataset contains a variety of visual variations.
Homepage: https://robopil.github.io/d3fields/
UIUC D3Field is a UR5 robot dataset with 192 episodes for office tasks, including visual and joint data. It supports vision-based imitation learning and 3D scene understanding. While the license is unspecified, it advances multi-object manipulation research.
Task: The robot completes tasks specified by the goal image, including organizing utensils, shoes, mugs.
Homepage: https://uscresl.github.io/dmfd/
USC Cloth Sim is a Franka robot dataset with 800 episodes for cloth manipulation, supporting deformable object research. Released under CC BY 4.0, it includes visual and joint data for learning from expert demonstrations.
Task: The robot manipulates a deformable object (cloth on a tabletop) along a diagonal.
Homepage: https://github.com/clvrai/clvr_jaco_play_dataset
USC Jaco Play is a CLVR Lab dataset with 1,085 teleoperated Jaco 2 episodes. It includes language instructions and visual data for pick-and-place tasks, released under CC BY 4.0. It supports language-conditioned models and vision-based control, facilitating task generalization research.
Task: The robot performs pick-place tasks in a tabletop toy kitchen environment. Some examples of the task include, "Pick up the orange fruit.", "Put the black bowl in the sink."
Homepage: https://github.com/frt03/rlds_dataset_builder/tree/dev/xarm
UTokyo xArm Bimanual is a University of Tokyo dataset with xArm robot episodes for towel folding tasks, including visual and joint data. Released under CC BY 4.0, it supports imitation learning and multi-object manipulation research.
Task: The robots reach a towel on the table. They also unfold a wrinkled towel.
Homepage: https://github.com/frt03/rlds_dataset_builder/tree/dev/xarm
UTokyo xArm PickPlace is a University of Tokyo dataset with xArm robot trajectories for pick-and-place tasks, including visual and joint data. Released under CC BY 4.0, it supports open-world RL and dynamic manipulation research, emphasizing generalization to unseen objects.
Task: The robot picks up a white plate, and then places it on the red plate.
Homepage: https://vimalabs.github.io/
VIMA is a mobile robot dataset with 1,000 episodes for navigation tasks, including visual and language data. It supports open-vocabulary understanding and real-time control. While the license is unspecified, it advances human-robot interaction research.
Task: The robot is conditioned on multimodal prompts (mixture of texts, images, and video frames) to conduct tabletop manipulation tasks, ranging from rearrangement to one-shot imitation.