[localization] Calibrated Camera RIG localization in an existing reconstruction using CCTag about alicevision

[fabiencastan]

Could you detail in the task what is missing?

[simogasp]

everything, I don't know why everything was already checked :-) . I'm actually working on this

[simogasp]

8d22c65 is a first stub for the localizer:

• for each camera the cctag are extracted, the N closest kframe retrieved and all these associations (2d feat -> 3D cctag point) are collected
• with all the collected associations, the 2D features are undistorted and put in its respective normalized camera coordinates (ie inv(K)*feat), the 3D points are placed in each camera sub-reference system (ie, they are multiplied by the sub-pose matrix)
• the pose is estimated and refined using these associations

for now we only return as output the pose of the rig, not the associations (it could be useful to have, or not?).

[simogasp]

Well, this is embarrassing but that's completely wrong, that's what happens when you do things without doing the math first (and thinking)...
The rig localization problem is not as easy as we put it there, if we want to use the associations from all the cameras to call a p3p algorithm, we actually need to model the rig as a non-central camera and use a generalized p3p algorithm, such as explained here (http://e-collection.library.ethz.ch/eserv/eth:7644/eth-7644-01.pdf) and implemented here:
https://github.com/laurentkneip/opengv
The simplest way to go for now is to localize each camera independently and then run a BA with the constraint on the camera subposes.
Just for the record, what I was trying to do (wrongly) was:

q1 ~ [I 0] Q  // the first camera
q2 ~ [R t] Q = C Q // a second camera, q1 and q2 in normalized coordinates 
then imagine that Q = [R1 t1] A = L A, where A is the pose of the rig we want to estimate, the idea was to compute it such as
q1 ~ L A // ok
q2 ~ L B // for some B that depends only on C (the relative pose) but the only possible B is B = inv(L) C L A... 

@lcalvet

[pmoulon]

I think you can also resect the camera (suppose that there is no subpose) and once the global pose is found multiply it by inv(subpose), so you will identify in this way tha combination of poses.

[simogasp]

There are certainly many ways to do the localization of the rig, what is interesting for us with a "global" approach is that you can overcome the problem of few features available, pushing to the limit of having only 1 or 2 features available per camera (or having a camera without features and 2 features on the other 2 cameras and so on) so that each camera cannot be localized by itself but globally the rig can.

[simogasp]

A naive version of the rig localizer is now available for testing. For now we just localize the cameras of the rig independently and we eventually run a BA in the end to refine the global pose of the rig

[simogasp]

Needs to be tested. Another issue?