bochenys / bpnp

Back-propagatable PnP

This repo provides the code used in the paper

End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization (CVPR 2020)

Watch our video demo

Install

bash requirements.sh

Back-propagatable PnP (BPnP)

Using BPnP is easy. Just add the following line in your code

import BPnP
bpnp = BPnP.BPnP.apply

Then you can use it as any autograd function in Pytorch.

Demo experiments

To see the demos presented in the paper, run

python demoPoseEst.py

or

python demoSfM.py

or

python demoCamCali.py

Cite this work

@inproceedings{BPnP2020,
    Author = {Chen, Bo and Parra, Alvaro and Cao, Jiewei and Li, Nan and Chin, Tat-Jun},
    Title = {End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization},
    Booktitle = {CVPR},
    Year = {2020}}

bpnp's Issues

Usage to train with only pose label

I have a thought that if I only have pose label and camera intristic, as for some tasks like gaze estimation(6DoF of eyeball) it's hard to get cannical model ,can I train a network using bpnp minimizing the pose loss to get the latent rigid 3D key points of the object and 2D key points.

HRNet-DSNT-BPnP

您好，我是一名学生研究物体姿态估计方向，读了您的论文感觉这是一项非常优秀的工作，其中由HRNet作为主干网络预测出热图然后使用DSNT得到关键点再使用BPnP的这个方法，我想参考一下您的代码，您可以发给我一份吗？谢谢您。

3d vs reprojection error

Thanks for the wonderful work.
One question on the implementation, bpnp.py, line 86
r = pts2d_i*Si-proj_i[0:2,:]
This seems to be a 3d error instead of reprojection error, which does not match eq.5 in paper. Could you please provide more details on this? Thanks

conda environment yml

thanks for the code.

I'm also a conda fan, glad you use it. why not use a conda environment file like this:
conda env export > bpnp.yml

mine is attached.
bpnp.zip

then replace the conda steps in your requirements.sh file with:
conda env create -f bpnp.yml

thanks.

adam

Tensor inversion RuntimeError and the possibility to use customized forward PnP solver

Hi Bo CHEN,

Thanks for sharing the wonderful code. I have some questions regarding the BPnP implementations and I would appreciate it if you could give me a hand:

1. torch.inverse() RuntimeError. I encounter this error when running the demoPoseEst.py. The other 2 demo scripts work perfectly though. The complete error message is:

(bpnp) qiyan@qiyan-qdlpc:~/Documents/BPnP$ python demoPoseEst.py
i:    0, loss:21164.001953125
i:    1, loss:21162.431640625
i:    2, loss:21160.376953125
i:    3, loss:21158.195312500
i:    4, loss:21156.015625000
i:    5, loss:21153.923828125
i:    6, loss:21151.771484375
i:    7, loss:21149.613281250
i:    8, loss:21147.451171875
i:    9, loss:21145.251953125
i:   10, loss:21143.001953125
Traceback (most recent call last):
  File "demoPoseEst.py", line 67, in <module>
    loss.backward()
  File "/home/qiyan/anaconda3/envs/bpnp/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/qiyan/anaconda3/envs/bpnp/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File "/home/qiyan/anaconda3/envs/bpnp/lib/python3.7/site-packages/torch/autograd/function.py", line 77, in apply
    return self._forward_cls.backward(self, *args)
  File "/home/qiyan/Documents/BPnP/BPnP.py", line 96, in backward
    inv_J_fy = torch.inverse(J_fy)
RuntimeError: inverse_cuda: U(6,6) is zero, singular U.

I tried using cpu instead of cuda and the problem remains. BTW, as you suggested in here, torch 1.4.0 and torchvision 0.5.0 are used. Accordingly, kornia 0.2.2 is installed for compatibility as shown in here, rather than using the latest version.

2. Can we use another customized PnP solver in the BPnP forward loop?
   

   BPnP/BPnP.py

   Lines 20 to 42 in 114f731

   	@staticmethod
   	def forward(ctx, pts2d, pts3d, K, ini_pose=None):
   	bs = pts2d.size(0)
   	n = pts2d.size(1)
   	device = pts2d.device
   	pts3d_np = np.array(pts3d.detach().cpu())
   	K_np = np.array(K.detach().cpu())
   	P_6d = torch.zeros(bs,6,device=device)
   	for i in range(bs):
   	pts2d_i_np = np.ascontiguousarray(pts2d[i].detach().cpu()).reshape((n,1,2))
   	if ini_pose is None:
   	_, rvec0, T0, _ = cv.solvePnPRansac(objectPoints=pts3d_np, imagePoints=pts2d_i_np, cameraMatrix=K_np, distCoeffs=None, flags=cv.SOLVEPNP_ITERATIVE, confidence=0.9999 ,reprojectionError=3)
   	else:
   	rvec0 = np.array(ini_pose[i, 0:3].cpu().view(3, 1))
   	T0 = np.array(ini_pose[i, 3:6].cpu().view(3, 1))
   	_, rvec, T = cv.solvePnP(objectPoints=pts3d_np, imagePoints=pts2d_i_np, cameraMatrix=K_np, distCoeffs=None, flags=cv.SOLVEPNP_ITERATIVE, useExtrinsicGuess=True, rvec=rvec0, tvec=T0)
   	angle_axis = torch.tensor(rvec,device=device,dtype=torch.float).view(1, 3)
   	T = torch.tensor(T,device=device,dtype=torch.float).view(1, 3)
   	P_6d[i,:] = torch.cat((angle_axis,T),dim=-1)
   	ctx.save_for_backward(pts2d,P_6d,pts3d,K)
   	return P_6d

   
   For the vanilla BPnP forward function, the output pose doesn't come with autograd and it seems fine to use alternatives to solve 2D-3D correspondences.
   So do you think it feasible to implement BPnP directly for other PnP solvers by replacing line 32, 36? For example, those based on P3P/EPNP which are also supported by the openCV.

Many thanks! Have a nice day & week-end.
Qi Yan

6D pose with BPnP

Hello,

 I am a graduate student, I have been studying your paper recently and trying to reproduce the results of 6DPOSE in the fifth part of the paper. 

 So I would like to learn a question: how do you normalize the coordinates of 15  2D projection key points (truth)?

Thanks for your time.

Normalisation of the key-point residual loss for integration with PVNET

Hello @BoChenYS and @jonbakerfish ,

First of all, Thanks for publishing code for your insightful paper.

I am trying to train pvnet along with the loss function defined in you demoPoseEst.py. However, the loss is way too high in value compared to the other two losses in PVNET. I tried to normalise it by dividing it by Σ(y - y_mean)² where y is the batch of ground truth key_points and y_mean is the mean Keypoint.
Normalized RSS = RSS / Σ(y - y_mean)²

However, the value is still too high in the range of 1 to 100. Meanwhile other losses are less than 0.01 generally.

How do I solve this? Or is there any other way I can incorporate BPNP in Clean-Pvnet.

Thanks!

RunTimeError Fixed

Hi,

First of all, I would like to congrats the authors due to such amazing work! I'm a post-graduate researcher.

Recently I faced the same RunTimeError here #4

However, I have to use the newest version of PyTorch and Torchvision. So, to fix the error, I've found that setting the saved Tensors as a contiguous tensor, the RunTimeError is fixed.

def backward(ctx, grad_output):
    pts2d, P_6d, pts3d, K = ctx.saved_tensors

    pts2d = pts2d.contiguous()
    P_6d = P_6d.contiguous()
    pts3d = pts3d.contiguous()
    K = K.contiguous()

I would like to know if is possible to do a Pull Request.

Thanks in advance!

Run error

Hi Bo,

Thanks for your work. I have a question when I run python demoPoseEst.py without any modification. The other demo scripts have similar error while running.

Regards

main contribution of paper seems to work worse

The main contribution i.e., your "invention" of "BPnP" in the table 1, loss l_p (backpropable pnp) seems to be considerably worse than l_h loss (already published in eccv'18 paper "Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimation").

Can you please tell the reason why is it worse than l_h. Using l_m = (\beta * l_p) + l_h together does not make sense as they both do the same thing. Also, you have set \beta to very small value of 0.0002, which mean l_m does not get affected by bpnp loss much.
The final results of l_m are not much better than l_h, and the change must be there just because of randomness.

Object pose estimation with BPnP

Hi Bo,

We are trying to reproduce your results presented in Section 5 of the paper so we would like to ask for some clarifications.

• Could you share some details about the configuration of HRNet (e.g. nr of stages, heatmaps resolution, etc.) and the training parameters (e.g. learning rate, optimizer, batch size, etc.)

• Regarding the heatmap loss have you tried others expect MSE (e.g. a KL loss instead)?. We've noticed better convergence with an L1 loss so far.

• Have you added any priors to the predicted heatmaps (e.g. softmax). Additionally, the gt heatmaps are in [0,W] x [0, H] format or normalised?

Thanks for your time.

Best,

Giorgos

Could you please elaborate on BPnP_fast?

Hi Bo,

I was wondering if you could elaborate more on the BPnP_fast method implemented in your code, which is not explained in your paper. In its comments, it says that higher order derivatives are dismissed for efficiency purpose and this sacrifice is 'negligible' for gradient accuracy.

BPnP/BPnP.py

Lines 211 to 225 in 114f731

	class BPnP_fast(torch.autograd.Function):
	"""
	BPnP_fast is the efficient version of the BPnP class which ignores the higher order dirivatives through the coefs' graph. This sacrifices
	negligible gradient accuracy yet saves significant runtime.
	INPUTS:
	pts2d - the 2D keypoints coordinates of size [batch_size, num_keypoints, 2]
	pts3d - the 3D keypoints coordinates of size [num_keypoints, 3]
	K - the camera intrinsic matrix of size [3, 3]
	OUTPUT:
	P_6d - the 6 DOF poses of size [batch_size, 6], where the first 3 elements of each row are the angle-axis rotation
	vector (Euler vector) and the last 3 elements are the translation vector.
	NOTE:
	This BPnP function assumes that all sets of 2D points in the mini-batch correspond to one common set of 3D points.
	For situations where pts3d is also a mini-batch, use the BPnP_m3d class.
	"""

Could you provide some insights for the application of this function? For example, what kind of practical consequences would you expect out of using BpnP_fast? In our task, I find regular BPnP is really slow and thus hardly applicable to joint training with other modules, while BPnP_fast is much better.

Best,
Qi Yan

DLT algorithm needs at least 6 points for pose estimation from 3D-2D point correspondences

Hello, I would like to ask if the following use of the BPnP algorithm requires a version of conda's environment. For example: pytorch, opencv-python and so on. I tried to use is eleven points for settlement and my inputs are correct, but he has the following error that suggests that I need to enter more than 6 points. I wondered if there was a problem with the input, so I ran the test code you gave me, demoPoseEst.py, and still get the problem. Looking forward to your recovery if you have experienced the same problem before.

Traceback (most recent call last):
File "/home/lab/LY/Space_6D_Pose_Estimation/spacecraft-uda/main.py", line 248, in
rt_source = pnp_fast(kpts_pred, kpts_world[0], k_mat_input[0]) # Bx6 [rot,pose]
File "/home/lab/LY/Space_6D_Pose_Estimation/spacecraft-uda/BPnP/BPnP.py", line 238, in forward
_, rvec0, T0, _ = cv.solvePnPRansac(objectPoints=pts3d_np, imagePoints=pts2d_i_np, cameraMatrix=K_np, distCoeffs=None, flags=cv.SOLVEPNP_ITERATIVE, confidence=0.9999 ,reprojectionError=3)
cv2.error: OpenCV(4.1.2) /io/opencv/modules/calib3d/src/calibration.cpp:1171: error: (-2:Unspecified error) in function 'void cvFindExtrinsicCameraParams2(const CvMat*, const CvMat*, const CvMat*, const CvMat*, CvMat*, CvMat*, int)'

DLT algorithm needs at least 6 points for pose estimation from 3D-2D point correspondences. (expected: 'count >= 6'), where
'count' is 5
must be greater than or equal to
'6' is 6

kornia integration

Hi guys ! this is a really nice paper !!

We would like to have BPnP integrated inside kornia, however there's a drawback with your code which is that uses opencv and we don't allow third-party libraries other than PyTorch.

Do you guys have any alternative so that we can collaborate ?

Are you going to release tensorflow version?

As tile.

Seems a wrong implementation of coef's gradient calculation

Hi，Bo Chen,

We are currently following your excellent work BPNP. And we have a question about your eq(17,18,20,21,22). In eq. 17, c_ij in f_j is calculated from y , so it must have derivative with respect to y. But in your code, you don't account for the derivative and just return a coef without computational graph.

We have tested that your implementation y_grad = torch.autograd.grad(proj[:,:,k],y,vec, retain_graph=True) will not give y_grad a grad_fn, unless change it into

y_grad = torch.autograd.grad(proj[:,:,k],y,vec, retain_graph=True,create_graph=True)

I wonder how you think about this question, have you tested it?

And we have tested our modified implementation and the results show that it has a significant improve on convergence rate in your PoseEstimation and CamCalib demo.

Thank you!

Cannot find training and testing scripts

Hi guys ! this is a really nice paper !!

I want to reproduce your paper now, but because you only published the demo without training and testing scripts, I can't reproduce it further.

Could you send me a copy of the training and test script?

Thank you!