Learning Granularity-Aware Affordances from Human-Tool Interaction for Tool-Based Functional Grasping in Dexterous Grasping（GAAF-DEX）

September 2, 2025 · View on GitHub

To enable robots to use tools, the initial step is teaching robots to employ dexterous gestures for touching specific areas precisely where tasks are performed. Affordance features of objects serve as a bridge in the functional interaction between agents and objects. However, leveraging these affordance cues to help robots achieve functional tool grasping remains unresolved. To address this, we propose a granularity-aware affordance feature extraction method for locating functional affordance areas and predicting dexterous coarse gestures. We study the intrinsic mechanisms of human tool use. On one hand, we use fine-grained affordance features of object-functional finger contact areas to locate functional affordance regions. On the other hand, we use highly activated coarse-grained affordance features in hand-object interaction regions to predict grasp ges tures. Additionally, we introduce a model-based post-processing module that transforms affordance localization and gesture pre diction into executable robotic actions. This forms GAAF-Dex, a complete framework that learns Granularity-Aware Affordances from human-object interaction to enable tool-based functional grasping with dexterous hands. Unlike fully-supervised methods that require extensive data annotation, we employ a weakly supervised approach to extract relevant cues from exocentric (Exo) images of hand-object interactions to supervise feature extraction in egocentric (Ego) images. To support this approach, we have constructed a small-scale dataset, Functional Affordance Hand-object Interaction Dataset (FAH), which includes nearly 6𝐾 images of functional hand-object interaction Exo images and Ego images of 18 commonly used tools performing 6 tasks. Ex tensive experiments on the dataset demonstrate that our method outperforms state-of-the-art methods, and real-world localization and grasping.

Usage

1. Requirements

Code is tested under Pytorch 1.12.1, python 3.7, and CUDA 11.6

pip install -r requirements.txt

2. Dataset

You can download the FAH from Baidu Pan (3.23G). The extraction code is: ip6q.

3. Train and Test

Our pretrained model can be downloaded from Baidu Pan . The extraction code is: rwku. Run following commands to start training or testing:

python train_gaaf.py --data_root <PATH_TO_DATA>

python test.py --data_root <PATH_TO_DATA> --model_file <PATH_TO_MODEL>

Citation

@article{yang2025learning,
  title={Learning Granularity-Aware Affordances From Human-Object Interaction for Tool-Based Functional Dexterous Grasping},
  author={Yang, Fan and Chen, Wenrui and Yang, Kailun and Lin, Haoran and Luo, Dongsheng and Tang, Conghui and Li, Zhiyong and Wang, Yaonan},
  journal={IEEE Transactions on Neural Networks and Learning Systems},
  year={2025},
  publisher={IEEE}
}

Anckowledgement

This repo is based on Cross-View-AG , LOCATE Thanks for their great work!