DexHand: dexterous hand manipulation motion synthesis for virtual reality

Natural object manipulation is one of the important human skills. However, generating natural hand manipulation motions that are adaptive to object shapes and the tasks at hand in virtual reality is still a challenge. In this paper, we propose a neural network-based finger movement generation approach, enabling the generation of plausible hand motions interacting with objects. Given the object shape and movement features in the wrist coordinate system, the first network Generator infers the hand pose at the current frame that matches the object motion and shape. The second network Optimizer then fine-tunes the pose to improve the plausibility of hand-object interaction. Notably, a differentiable optimization module is proposed to generate the training dataset for Optimizer. Experimental results show that our approach can generate plausible dexterous hand manipulation motions for hand-object interaction without obvious delay.