Bimanual Dexterity for Complex Tasks

Kenneth Shaw*         Yulong Li*        
Jiahui Yang         Mohan Kumar Srirama         Ray Liu         Haoyu Xiong        
Russell Mendonca†         Deepak Pathak†
Carnegie Mellon University
*Equal contribution. †Equal Advising.
CoRL 2024

Abstract

To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms ($16k USD).

Autonomous Results



Serving

Drill Operating
Pouring in the wild

Plate Pickup
Pouring

Wire Winding


Teleoperation Results



Table Rearragement

Card Pickup

Dining Rearrangement
Chopstick Pickup

Grape Plucking

Pour Chips
Scoop and Pour

Stack Bowl

Stack Cups




Stack Plate

Insert Brush

Cut Cucumber
Peel Cucumber

Pouring

Mix with Spatula
Spray and Clean with Brush

Whisking

Use Stapler




Screwdriver Pickup

Tool Cleanup
Use Drill

Hammering Dough
Rearrange and Insert Soldering Iron

Wire Winding

Hang Shirt
Fold Shirt

Teleoperation with Operator



Stack Plate
Stack Bowl
Grape Plucking


Mobile Teleoperation



Operator View of Box Transport

Clear Trash
Box Transport

Push Chair


LEAP Hand v1 Autonomous



Pouring
Pringles (Dyanmic)
Stack Cup

BibTeX

@inproceedings{shaw2024bimanual,
      title={Bimanual Dexterity for Complex Tasks},
      author={Shaw, Kenneth and Li, Yulong and Yang, Jiahui and Srirama, Mohan Kumar and Liu, Ray and Xiong, Haoyu and Mendonca, Russell and Pathak, Deepak},
      booktitle={8th Annual Conference on Robot Learning},
      year={2024}
    }

Acknowledgements

We thank Ankur Handa, Arthur Allshire, Toru Lin and Nancy Pollard for discussions about the paper. We also thank Maarten Witteveen and Sarah Shaban at Manus Meta for their assistance on their gloves. We thank Diya Dinesh and Lukas Kebuladze for helping with teleoperation. This work is supported in part by DARPA Machine Commonsense grant, AFOSR FA9550-23-1-0747, ONR N00014-22-1-2096, AIST CMU grant and Google Research Award.