Object pose command about unrealcv

[qiuwch]

I think you can use vget /camera/[id]/location, vget /camera/[id]/rotation to get the camera 3D position. and use vget /object/[obj_name]/location to get object 3D location. Then use this information to compute object pose w.r.t to the camera. I am not an expert about 3D pose estimation, please correct me if I am wrong. I tend to keep the UnrealCV commands as simple as possible and leave some simple computation to the client side.

I think the 3D point cloud data is related to the discussion in #2 ? Is it easy for you to compute point cloud if the system returns the mesh data?

[hgaiser]

Ah apologies, I looked at the command list here and didn't see the /object/[obj_name]/location command. Is there also a /object/[obj_name]/rotation? If so, that would indeed be all I need.

As for point clouds, if I receive a mesh it is still relatively hard to get a proper point cloud out of this. Similarly to what you have for creating depth is what would be required for a point cloud. With the exception that in the depth image the value represents the distance to the camera, in a point cloud it would be the XYZ distance to the camera (or, if easier, XYZ position in world; but w.r.t. camera is preferred I'd say).

[qiuwch]

Sorry, the command list is not complete and accurate, I will update it. Currently, there is no vget /object/[objname]/rotation, but it is simple to implement, I will add it in the next release (in a few days).

If my understanding is correct, with mesh data and camera 3d location and rotation, we should be able to transform from the global coordinate to camera coordinate?

[hgaiser]

Hey, sorry for my late reply.

It should be possible with that information, but (and I am an UnrealEngine novice) isn't there a better way? Given the mesh data and camera pose you in theory have all the data you need, but you would need to raycast for each "pixel" in your 3D point cloud (using your camera pose and parameters). How is this currently done for the depth image which is generated? It seems to me that this is more or less the same problem.

Actually, perhaps if the depth image and camera parameters are available you would also have enough information. This seems easier to implement than having to raycast every pixel again.

[qiuwch]

@hgaiser Sorry for delay, I was quite busy last week. I add commands vget /object/[id]/rotation, vset /object/[id]/location, vset /object/[id]/rotation to the master branch. You can try it if you know how to compile the code.
I did NOT update the binaries on the website, and will update them later.

I am working on getting the camera parameters. This should be much easier than computing by raycasting.

[hgaiser]

@qiuwch thank you for your reply and thank you for adding those commands. I did indeed manage to compile the plugin a while back so that will be fine. Unfortunately I have some other project I need to take care of first so I have put this on hold. Hopefully I can pick this up again in a week or two.

I agree that using camera parameters and the depth image is easier. I'll have to look into converting this data to point clouds when I have the time. Thanks a lot for the feedback at least, I'll close this issue since it is resolved.

[qiuwch]

@hgaiser My pleasure. Let me know if you need further assistance.