Thesis Advisor: Prof. Tucker Hermans
Thesis Committee:
- Prof. Tucker Hermans
- Prof. John Hollerbach
- Prof. Srikumar Ramalingam
- Prof. Vivek Srikumar
- Prof. Kris Hauser
Abstract
In-hand manipulation is the skill of changing the state of a grasped object leveraging the dexterity available in a robot’s end-effector. To reach the desired state for a grasped object, the robot might need to change existing contacts on the object, impart forces through these contacts to change the object’s state while also maintaining a stable grasp on the object.
In-hand manipulation is challenging in the unstructured, real world as we do not accurately know the object’s dynamics or the dynamics model governing object-robot interactions. Lack of sensing at the contacts also makes perception of these interactions hard to observe and react. These challenges have limited analytic model based in-hand manipulation to lab environments with knowledge of the object’s dynamics or with hand tuned trajectory specific controllers. Learning based methods have shown to only learn object specific or trajectory specific models, thereby reducing their use in the real world.
In this thesis, we explore strategies to enable real-world in-hand manipulation of novel objects. We tackle some of the key bottlenecks preventing in-hand manipulation in the unstructured, real world. First, we explore trajectory optimization with kinematic constraints to perform in-grasp manipulation of novel objects without requiring extensive object dynamics knowledge. We extend this in-grasp optimization to include fingertip relocation and explore planning via alternating optimization. To improve perception at contacts, we learn a robust force estimation model for a biologically inspired fingertip tactile sensor leveraging large scale data collection with multiple robots. Finally, we formulate multi-contact dynamics as a physically consistent factor graph and perform inference to estimate object dynamics in-hand using the developed force estimation model. Our results show that dynamics of novel objects can be recovered from tactile perception at fingertips, while being kinematically tracked using our kinematic in-grasp constraints. Additionally, we develop benchmarking metrics and protocols to assess a robot’s in-hand manipulation capabilities.