gentlesjan / affordpose

AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose (ICCV 2023)

AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose

ICCV, 2023
Juntao Jian¹ · Xiuping Liu¹ · Manyi Li^2,☎ · Ruizhen Hu³ · Jian Liu<sup>4,☎

1</sup> Dalian University of Technology    ² Shandong University
³ Shenzhen University    ⁴ Tsinghua University
^☎ Corresponding author

Download Datasets

1. Download the AffordPose datasets from the AffordPose Project Page. You can download specific categories or all the data according to your needs. The data are saved with the path: AffordPose/Object_class/Object_id/affordance/xxx.json, look like:

    .
    └── AffordPose
        ├──bottle
        │   ├──3415
        │   │   ├──3415_Twist
        │   │   │   ├── 1.json
        │   │   │   ├── ...
        │   │   │   └── 28.json
        │   │   │
        │   │   └──3415_Wrap-grasp
        │   │       ├── 1.json
        │   │       ├── ...
        │   │       └── 28.json
        |   |
        |   └── ...
        |
        └── ...
   

2. The structure in xxx.json file as follows:

    .
    ├── xxx.json
        ├── rhand_mesh            # the hand mesh
        ├── dofs                  # the joint configurations of the hand
        ├── rhand_trans           # the translation of the paml
        ├── rhand_quat            # the rotation of the paml
        ├── object_mesh           # the object mesh, and the verts are annotated with affordance label
        ├── trans_obj             # with the default value: (0,0,0)
        ├── quat_obj              # with the default value: (1,0,0,0)
        ├── afford_name           # the object affordance corresponding to the interaction
        └── class_name            # the object class
   

Data visualization

• If you want to visualize the hand mesh, a feasible way is to save the value of "rhand_mesh" from the xxx.json as xxx.obj file and visualize it in MeshLab, which is also applies to object mesh.

• The hand model we use following the obman dataset, which ports the MANO hand model to GraspIt! simulator.

• We used GraspIt! to collect xxx.xml data and ran ManoHand_xml2mesh.py to obtain the hand mesh in 'mm'. Please note that you cannot obtain the correct hand mesh in 'm' by simply changing the 'scale' parameter in this python file.

  $ python ./ManoHand_xml2mesh.py --xml_path PATH_TO_DATA.xml --mesh_path PATH_TO_SAVE_DATA.obj --part_path DIRPATH_TO_SAVE_HAND_PARTS

Citation

If you find AffordPose dataset is useful for your research, please considering cite us:

@InProceedings{Jian_2023_ICCV,
  author    = {Jian, Juntao and Liu, Xiuping and Li, Manyi and Hu, Ruizhen and Liu, Jian},
  title     = {AffordPose: A Large-Scale Dataset of Hand-Object Interactions with Affordance-Driven Hand Pose},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month     = {October},
  year      = {2023},
  pages     = {14713-14724}
}

affordpose's Issues

Question about the affordance label

Hi,

Thanks for your great dataset. But may I ask how can I find the affordance label? It seems every object looks the same inside MeshLab regardless of the affordance. And the object_mesh vertexes look the same without any annotation about affordance. E.g., jar 4118 looks like this

I noticed every vertex is followed by 5 or 7. Does this means the affordance label?

Best,
Hui

Hand Model

Dear Authors,

Thanks for releasing the dataset!

What is the hand model used? Could you please provide more details? It does not look like a real hand like MANO.
Is this hand model replaceable by the MANO hand model?

What is the reason for not using the MANO hand model?

Thanks in Advance!

ETA?

Hi!

Do you have an ETA on the release of your dataset? I'd like to use it for my research :)

Thanks!

dataloader?

Thanks for the wonderful dataset!

Is there any formatted dataloader available for the dataset? Thanks!

Model from Section 5.2 Affordance-oriented Interaction Generation

Hello!

In Section 5.2 of the paper, you show results for hand-pose estimation trained on AffordPose data compared to GrabNet. Is this model available?