PDF Paper arXiv arXiv github Project

UniTacHand

Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer


PKU    BeingBeyond

PDF Paper arXiv arXiv github Project

UniTacHand: Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer

Chi Zhang*1,2   Penglin Cai*1,2   Haoqi Yuan1,2   Chaoyi Xu2   Zongqing Lu§1,2


1PKU     2BeingBeyond


*Equal Contribution   §Corresponding Author

Paper arXiv Project

Abstract

    Enabling robotic hands to achieve human-level dexterous manipulation remains a significant challenge. While tactile sensing is crucial, especially in scenarios with visual occlusion, its application is often hindered by the difficulty of collecting large-scale, real-world robotic tactile data. On the other hand, learning from human demonstration (LfD) offers a promising alternative due to the high accessibility of rich and diverse human data. However, directly leveraging human demonstrations is often questionable due to the fundamental mismatch between human haptic gloves and robotic tactile sensors, which requires an alignment between human and robotic data. To bridge this gap, we propose UniTacHand, a novel unified representation to align robotic tactile information captured by dexterous hands with human hand touch obtained from gloves with sensors. In this paper, we project disparate tactile signals, from both the human haptic gloves and the robotic hands, onto a morphologically consistent 2D surface space of the MANO hand model. Such unification standardizes the heterogeneous data structures and inherently embeds the tactile signals with spatial context. We utilize a contrastive learning framework to align the two domains into a unified latent space. Based on this unified structure, we demonstrate that 10-minute paired data are sufficient to train this alignment. Our approach achieves a zero-shot tactile-based policy transfer, in which the policies are solely trained using human data, to a physical robot. Such policies can be deployed to perform complex downstream tasks, including identifying objects unseen during the paired training phase. We also demonstrate that co-training on mixed data including both human and robotic demonstrations via UniTacHand yields a superior capability, enabling policies to achieve better data efficiency with only one-shot demonstration on the real robot. UniTacHand provides a general, scalable, and data-efficient pathway for transferring human haptic intelligence to policies on tactile dexterous hands.

Method Overview

overview

    UniTacHand Framework Overview. An illustration of our two-stage pipeline: (Left) Stage 1: Projecting tactile data from a human haptic glove or a robotic hand onto a unified MANO UV map. (Right) Stage 2: The contrastive learning framework with reconstruction and adversarial losses to align the latent spaces. We align the tactility and hand gesture from both sources to the same latent space using a contrastive framework trained with paired data. The unified pressure UV maps serve as accurate prior knowledge to supervise the domain-specific encoders, thereby enriching such a latent space with tactile-grounded information.

UV mapping Results

overview

    UV mapping Results. When a human hand (or robotic dexterous hand) grasps an object, the activated tactile lattices on a MANO hand are highlighted in red, with the poses of the hand (actions of fingers and rotations of the wrist) rendered at the same time. In this way, we achieve the unification of hand actions and spatial tactility, as well as the alignment between human hands and dexterous hands in terms of tactile information.

Experimental Results

Detailed results of the tasks CompliantControl and ObjectClassification. We calculate the accuracy (%) for each task on each method.

Tasks PatchMatch UV-Direct UniTacHand (Ours)
CompliantControl 10.0 36.0 40.0
ObjectClassification Human Validation Set 43.2 71.6 59.5
Real Robot Test Set 15.7 18.9 38.6

One-shot manipulation task success rate (%). We compare our methods with both visual-only baselines and methods that are solely trained on real robot data.

Methods Success Rate
Robot Data Only R-Visual-Only 43.3
R-Visual-Tactile 50.0
Human Data w/
One-Shot Robot Data		 PatchMatch 56.7
UV-Direct 63.3
UniTacHand (Ours) 73.3

BibTeX

@article{unitachand,
  title={UniTacHand: Unified Spatio-Tactile Representation for Human to Robotic Hand Skill Transfer},
  author={Zhang, Chi and Cai, Penglin and Yuan, Haoqi and Xu, Chaoyi and Lu, Zongqing},
  journal={arXiv preprint arXiv:2512.21233},
  year={2025}
}